Subject: [Nel] Network Connections

From: "Tom wright" <[email-masked]>
Date: Thu, 1 Mar 2001 08:48:29 -0000

Guys/Gals,

Having followed your arguements so far, i though i would chuck in my 2 p worth.

1) i think you are going to find it very difficult to scale a server to 5000 concurrent users, using any kind of scheme unless its a BIG server.  Look at EQ currently, they struggle with 2000.  But this is just my opinion :-) so feel free to ignore it.

2) On windows NT/2k, using async sockets with events is a very efficent way of handling socket comms, and its quite easy to demultiplex the events to work out which socket caused the event.  I would recommend this route, as one thread could then wait for events to be generated on say 30 sockets and then take appropriate action.  unfortunately i can not tell you how well this is going to scale :-( Just look up WSAEvents for READ, SEND and CLOSE.  It does make your sending more entertaining however :-)

3) Windows did support something called micro threads, which had a much lower over head than traditional threads.  These ( if i remember correctly ) were designed specifically for scalable IO ( as in IIS ).  Some time looking here may well yield some useful results.

4) Some OS's have limits to the size of the array to select, Windows is one of them, typical values seem to be 64.  However using values this large, you are going to spend time thrashing the array looking for the source of the select, with events this is much easier and quicker, tho not linux compatible.  Wrapping up the comms like this specific to each platform is not going to be to hard, just code to a common interface.

Regards

Tom

PS A binary of the next release for windows would be nice, takes HOURS to build :-)

---

From: Vincent Archer <[email-masked]>
Date: Thu, 1 Mar 2001 11:46:35 +0100

According to Tom wright:
> 1) i think you are going to find it very difficult to scale a server
> to 5000 concurrent users, using any kind of scheme unless its a BIG
> server.  Look at EQ currently, they struggle with 2000.  But this is
> just my opinion :-) so feel free to ignore it.

EQ doesn't struggle with 2000 because of server size. A "server" on EQ
consists (to the best of my knowledge) of about 15 or so quadriprocessor
RISC boards, with all nearly 100 zones (now) spread of these. They
struggle not because the server can't handle the load, but because the
content of the game can't handle the load (i.e. problems stem not from
too many people for CPU time, they stem from too many people for not
enough spawns).

No MMOG so far can use a single server, no matter how big it is. What
we term server, or shard, or whatever, typical technical litterature
terms 'cluster'. And using the right architecture can make a cluster
very large, until it collapses not from load, but when adding a new
note consumes more CPU spent in inter-node communications than the node
itself adds to the whole.

> 2) On windows NT/2k, using async sockets with events is...
...
> 3) Windows did support something called micro threads, which had...

All these optimisation techniques are nice. But they're heavily dependent
on Windows itself. We do want our server to run under windows, for debugging
purpose (and for the odd amateur out there who would want to run a
smallish server on his home machine), but we clearly do NOT intend our
production servers to run under windows.

Which is why we're looking at a reasonably portable architecture, and why
these suggestions, as nice as they are, cannot be used, because they tie
too much the architecture :(

> PS A binary of the next release for windows would be nice, takes HOURS to build :-)

As much as people would love one, it's right now a "marketing" decision,
so that we attract people that really want to look at the source, and not
just people who want to look at the finished product without caring about
how it's done :)

(However, some Nevrax people, and maybe external people as well, have set
 up binary archives on their own private home pages. Ask, and you'll
 probably find it. But officially, we do not support or provide these
 releases. Yet)

-- 
Vincent Archer                                         Email: [email-masked]

Nevrax France.                              Off on the yellow brick road we go!

---

From: "Michael Warnock" <[email-masked]>
Date: Thu, 1 Mar 2001 14:38:16 -0800

3) Windows did support something called micro threads, which had a much lower over head than traditional threads.  These ( if i remember correctly ) were designed specifically for scalable IO ( as in IIS ).  Some time looking here may well yield some useful results.
I come from a nix background, but I wouldn't doubt that windows also has had some implementation of microthreads.  The best current way to use microthreads afaik is stackless python (stackless.org) which I (as the game mechanics programmer) plan to use on the server side of the game my company (In-Orbit Entertainment) is working on using the Nel library.  Microthreads use an old and mostly forgotten programming concept called continuations- a continuation is, in brief, the rest of a program from any given point- that includes it's state and any branch of code it might follow from there.  Stackless Python provides a way to reference continuations- which requires it not to use the C stack for framing(a neat trick u should read about).  Writing microthreads is as simple as defining a bunch of functions and calling them at the same time with a special call- they are executed in parralel within the same thread and each requires almost no overhead above what they would if executed in serial.  EVE (a commercial mmorpg) also plans to use stackless as the basis for their game mechanics.  I dont have any benchmarks on its network performance or scalability, but I plan to test it soon.

Michael Warnock
In-Orbit Entertainment

---

From: "Michael Warnock" <[email-masked]>
Date: Thu, 1 Mar 2001 15:41:15 -0800

oops:)

It is actually http://www.stackless.com 

Here are more relavent urls:
http://world.std.com/~wware/uthread.html
http://www.eve-online.com/faq/faq_08.asp
http://pdos.lcs.mit.edu/exo/

We hope to use the MIT exokernel the last url refers to as a stripped down os for the servers.

Michael Warnock
In-Orbit Entertainment
  ----- Original Message ----- 
  From: Michael Warnock 
  To: [email-masked] 
  Sent: Thursday, March 01, 2001 2:38 PM
  Subject: Re: [Nel] Network Connections

  Michael Warnock
  In-Orbit Entertainment

---

From: Charles Dupont <[email-masked]>
Date: Fri, 02 Mar 2001 14:18:32 -0500

When you talk about network connections how are they going to work?  For 
example when say a player wants to find out what he/she has in his 
inivitroy how does the client get this information.  Does the client 
talk deriectly to the items sever or does it first contact another 
server, that it would contact for every data request, and which then 
goes asks the items server for the information which the server passes 
back to the client.  If it is the case that there is  a gateway server 
that talks to the data servers not the client talks directly to the data 
servers.  Why does there have to be a single gateway server.  You could 
set it up so that the gateways only had to control the max amount of 
connections the server can handel.  To do this you would need a another 
server to handle incomming connections and pass them off to the gateway 
servers.  Of course this system could bog down the intercluster network, 
but you would think that the total amount of trafic would not change.  
If it is the other way then this won't work. i.e. client talks directly  
to items server.

Charles Dupont