Subject: [Nel] NeL Network Engine

From: Olivier Cado <[email-masked]>
Date: Thu, 22 Feb 2001 16:41:12 +0100

We are very pleased that you are interested by NeL and our project.
Thanks to free software, we are now working in cooperation with the
community. So let's discuss some network issues.

Here is the purpose of this message:
- Present the future of the NeL Network Engine
- Ask for your input on a couple of points

At present, the Nel Network Engine is single-threaded. We plan to
rewrite the engine using multi-threading. The library will be made up of
five layers:
- Layer 0: socket wrapper (roughly, present CBaseSocket + listening
socket functionalities)
- Layer 1: multiple socket multi-threaded I/O mechanism
- Layer 2: adapted to CMemStream (allows serialization)
- Layer 3: adapted to CMessage (contains type information)
- Layer 4: using callbacks (as presently used in the services and
provided by CMsgSocket)

The main features of Layer 1 are as follows:

External view:
The user programmer will be able to send data (A), to check if some data
has been received (B), and if so to get a data block from the receive
queue.

Implementation:
(A) When the user requests to send data, his data block is put into a
send queue. The actual sending is triggered off by a time flush trigger,
a size flush trigger or an explicit flush trigger (at the user's
demand). Let's say the queue control is executed in the main thread, in
an update() method, called evenly.
(B) Each connection is handled by a separate thread that sleeps while
not receiving data so that no CPU time will be used if nothing is
received on a particular socket. When incoming data is actually
received, it is put into a global receive queue (synchronized with a
mutex of course) and popped when the user requests to receive a block.

This implementation still raises a few questions:
(A) If no buffer space is available within the transport system to hold
the data to be transmitted, the actual sending will block. Has anybody
come across this case ? When does this happen in practice ?
(B) As we are building a *massively* multiplayer game, we expect to have
a great number of connections (even if all clients won't be connected on
the same machine), therefore a great number of threads. Does anybody
know the scale limits on Linux systems (and on Windows BTW), i.e. the
optimum and maximum thread numbers per process and per system ?

I'm sure a lot of you are great Linux specialists, so you probably have
an idea about this issues.

Thanks.
Olivier Cado
--

---

From: Nicolas Hognon <[email-masked]>
Date: Thu, 22 Feb 2001 17:01:57 +0100

i've got the same problem for the project i work for  :
how many thread can i run under win nt/2000 and linux

i haven't made test about this yet ... but if some one answer to this
question it will be cool :))

-- 
Nicolas Hognon
[email-masked]

Virtools - The Behavior Company
Tel. (+33) 1 42 71 46 86 / Fax. (+33) 1 42 71 86 53
http://www.virtools.com/

---

From: "Ludovic LIEVRE" <[email-masked]>
Date: Thu, 22 Feb 2001 17:14:50 +0100

Hi all,

We also build a MM game.

Our servers now run on NT and Linux and will soon be ported to Solaris.

I've read that with LinuxThreads, after 100 threads the performances of the
system downgrade.

Another question about network engine: will your messages all be TCP or UDP
? If UDP is used will you add a mechanism that make UDP reliable ?
In the first stages of our game, TCP was used. We made some test and we saw
a terrific ping (due to ACK mechanism) time even on our LAN (200ms). So we
decide to use UDP instead...

> -----Message d'origine-----
> De : [email-masked] [mailto:[email-masked]]De la part de
> Nicolas Hognon
> Envoye : jeudi 22 fevrier 2001 17:02
> A : [email-masked]
> Objet : Re: [Nel] NeL Network Engine
>
>
>

---

From: Dim Segebart <[email-masked]>
Date: Thu, 22 Feb 2001 17:53:43 +0100

Although  this  site  http://www.linuxraid.org/ isn't directly related
to  NEL, it has a great number of tips and tricks on how to make Linux a little bit
stable  and  how to expand Linux system resources (see the bottom of their page).

Thursday, February 22, 2001, 5:01:57 PM, you wrote:

NH> i've got the same problem for the project i work for  :
NH> how many thread can i run under win nt/2000 and linux

NH> i haven't made test about this yet ... but if some one answer to this
NH> question it will be cool :))

-- 
Dim Segebart                         mailto:[email-masked]

---

From: Nicolas Hognon <[email-masked]>
Date: Thu, 22 Feb 2001 23:12:39 +0100

> come across this case ? When does this happen in practice ?

i'm not sure that i understand this part of your mail and i hope you'll
understand my answer because i'm not sure my english enough fluent :))

you're asking want happened if you call send or write on a socket
and there is not enough space in the tcp stack / transport layer buffer.
it depends you've got blocking or non blocking socket.
if it's a blocking socket you're thread will be blocked
until you're data can be send by the tcp stack.
if you're socket is in non blocking mode send/write
will return you an error an errno will be EWOULDBLOCK
or WSAEWOULDBLOCK under winsock.

but i'm not sure it's the answer to your question
because it is not a very huge problem.
i make the same mistake because when you make
some little test it's more common that receive failed
with a wouldblock than send. when i began
writting our network api (for the company i work)
i made this mistake. but when i made
some test with huge network messages and a poor modem
i encounter tihs problem.

i hope this help you
but can you answer me that i know if i understand
your problem :))

-- 
Nicolas Hognon
home : [email-masked]         / www.cblt.org
work : [email-masked] / www.virtools.com
#ICQ : 36044443
    
Enjoy the silence
(DM)

---

From: [email-masked] (Bernard Hugueney)
Date: Mon, 26 Feb 2001 17:49:39 +0100

> how many thread can i run under win nt/2000 and linux

Well, I'm as far as possible from a threads expert, but I think that
what makes LinuxThreads special is that they are system threads (vs user 
threads) maybe heavier, but making use of SMP, so basically, I think
the max nb of threads is very much system dependand (SMP?).

On a pessimistic note, I remember seeing in kernel traffic that Solaris
was more effective in handling huge number of threads :-(
Not only that, but Linus T. seemed not to recognize a defect is Linux,
but claimed that it was a matter of trade-off and that Solaris was
optimized for huge number of threads, and Linux more effective with
small number of threads.

But my memory is weak so I may be wrong...

On a completly unrelated note, I remember seeing a NeL coder seeking assistance
on a gcc mailing list for a trouble with exception causing an abort when
linking with C code (I think). I do not remember if there was a useful response
, but I saw a related message on boost mailing list:
<citation>
Hi,
I finally figured out why the regex++ shared library was crashing when
throwing an exception.  Somehow the shared library wasn't getting
created correctly by using ld to link it.  When using ld with c++ code,
some additional libraries need to be specified on the command line.  The
easy fix is to simply call g++ to link the files.  g++ then calls ld
with the appropriate commands.
</citation>

Maybe this can help...

Cheers,
Bernard

---

From: Vincent Caron <[email-masked]>
Date: Mon, 26 Feb 2001 19:50:35 +0100

> the max nb of threads is very much system dependand (SMP?).

Threads are indeed system tasks internaly, and they are scheduled as such.
The number of tasks and/or threads is only limited by NR_TASKS at compile
time in Linux. And yes, the scheduler is not really effective at managing
more than 1000~2000 tasks, and doesn't provide enough semantics to
optimize thread use. This is rather a matter of design, and you could
without great trouble design a Linux scheduler that handles a bunch of
threads the way you expect it, but as far as I known, only the real time
versions of Linux (such as RTlinux) have played with this part of the
Linux code. Now about SMP, Linux does spreads threads over the available
CPUs to optimize parallelism, but there's no limitation. And you'll only
get a speed up on SMP machines if you really have parallel threads (ie.
not a thread join every 10 lines !).

I guess we're raising the question of time sharing (or 'multiplexing', sounds
cool :)). You can let the OS do it for you and use threads everywhere : BeOS
is a good example since everything than can receive events is in a thread.
You can do it all by yourself : use one process and dispatch the incoming
events to functions (ie. WinProc). Why you would use one more than the other
is a matter of design, and also depends on the type of application and the
OS you use. The anwser is really not simple and won't certainly fit in a mail,
but I must admit I don't have good pointers to give ...

---

From: Dim Segebart <[email-masked]>
Date: Mon, 26 Feb 2001 20:17:11 +0100

I  have already posted this information to NEL list, but as
this topic have been arised it's worth to mention it once again.
At  SGI  web  site  there  is (IMHO) great article (and library) which
tries to solve common problem of networked application.
Look at it by the URL http://oss.sgi.com/projects/state-threads/docs/st.html
Project home page is http://oss.sgi.com/projects/state-threads/

---

From: Thierry Mallard <[email-masked]>
Date: Wed, 28 Feb 2001 08:31:06 +0100

On Mon, Feb 26, 2001 at 05:49:39PM +0100, Bernard Hugueney wrote:
> [ Linux and threads ]

In Linux 2.4.x, you may configure the maximum threads with
/proc/sys/kernel/threads-max, if i remember correctly.

But the default value (4096) is still much more that what the default kernel
can really handle nicely, as is it said in some other posts. We'll have to
look deeper into this...

	Shaman

-- 
Thierry Mallard              |              
GnuPG key on wwwkeys.pgp.net |
key 0xA3D021CB               |
http://thierry.mallard.com   |

---

From: "Vianney Lecroart" <[email-masked]>
Date: Wed, 28 Feb 2001 10:07:19 +0100

Hello,

In fact, we don't want to know the theorical limit. we know that we can
change this value in the kernel sources and recompile it.

What we don't know is the thread number limit after what the system uses too
much CPU. In the linuxthread faq, they
said that an application should not create more than 100 thread. In this
case, we have to forget the solution where each
socket is on a thread and use a blocked receive(). The problem is that
select() is quite slow and if we have only 100 thread,
each thread needs to manage, with a select(), around 50 players and we ll
lost lot of time to create the array for the select()
and check who have wakeup the select().

Vianney Lecroart
---
lead network programmer / nevrax.com
icq#: 6870415
homepage: http://ace.planet-d.net
www.geekcode.com: GCS/E d- s+++: a-- C+++$ UL++ P- L+++>+$ E+>- W++ N+ o? K-
w++$ O- M- V- PS- PE? Y PGP t 5? X+ R- tv++ b- DI D+ G e++ h+ r-- y?

----- Original Message -----
From: "Thierry Mallard" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, February 28, 2001 8:31 AM
Subject: Re: [Nel] NeL Network Engine

> _______________________________________________
> Nel mailing list
> [email-masked]
> /mailman/listinfo.cgi/nel

---

From: "Vianney Lecroart" <[email-masked]>
Date: Wed, 28 Feb 2001 10:48:50 +0100

Hello,

Yes, we didn't send a message to say what do we think about this library
(sate thread).
We currently look in deep the documentation and it seems a very interesting.
We just have to be sure that this lib fulfils our requirements.

Vianney Lecroart
---

----- Original Message -----
From: "Dim Segebart" <[email-masked]>
To: <[email-masked]>
Sent: Monday, February 26, 2001 8:17 PM
Subject: Re[2]: [Nel] NeL Network Engine

> Look at it by the URL
http://oss.sgi.com/projects/state-threads/docs/st.html

---

From: Nicolas Hognon <[email-masked]>
Date: Wed, 28 Feb 2001 10:57:11 +0100

> In this case, we have to forget the solution where each

I encouter the same problem : one thread must manage more than one 
socket on the server. the problem is how managing those socket
in the thread. the problem is I want to be portable between windows
and linux :((

under windows there is many solution :
  - select (the only one which is portable)
  - windows event and message
  - overlapped I/O

under linux the only solution I know is :
  - select / poll :  i'm not sure but i think the difference between
    select and poll is that with poll you don't have to fill your array
    before each call.

another problem when a thread maange more than one socket is
what your threads will do :

a first approch is something like that :

while(run)
{
   TestSokets();

   for each socket which received something
   do
     read the incoming data
     process the data
     send data
   done
}

but during one step of this loop your socket list must be locked
so if a new socket need to enter this thread the calling thread will be
blocked.

a solution is to have more than one thread type.
thread which test and read socket (perhaps wire in socket to)
thread which to process a network message queue.
(and perhaps thread which send network message)

but it increase the number of thread so the maximum
number of client is lesser.

but i'm not really used to programming with thread yet so perhaps
my problem are not really problems.

i hope this can help and you can understand me :))
and i think i'll take a look at http://oss.sgi.com/projects/state-threads/
to ...

good luck

-- 
Nicolas Hognon
[email-masked]

---

From: "Vianney Lecroart" <[email-masked]>
Date: Wed, 28 Feb 2001 11:33:10 +0100

There is a very very big problem in the State Threads library

The State Threads library is a derivative of the Netscape Portable Runtime
library (NSPR) and therefore is distributed under the Netscape Public
License (NPL).

And the NPL is not compatible with GPL licence so we could not use it in
your GPL project!!!

Another less important problem is that it doesn't work on windows and our
server must be able to run on windows (for debugging purpose for example)

Vianney Lecroart
---

---

From: Vincent Caron <[email-masked]>
Date: Wed, 28 Feb 2001 15:30:32 +0100

> and check who have wakeup the select().

Looking at Apache or Samba projects, it seems that a good compromise is to
set a 'maximum client requests by thread' and spawns threads accordingly.
Apache uses process forking and memory sharing, but the design remains the
same. You then just tune this max_request_by_thread for each OS, say 1 for
Solaris which is said thread-efficient, 10 for Linux ? Just a hint ...

---

From: Nicolas Hognon <[email-masked]>
Date: Wed, 28 Feb 2001 15:10:53 +0100

> Solaris which is said thread-efficient, 10 for Linux ? Just a hint ...

ok this is a solution
but as i said in my last mail the problem
is how managing more than one socket per thread efficiently ?

under windows i know other solution than select / poll but under
unixes i don't know other solutions.

some one knows other one ?

-- 
Nicolas Hognon
[email-masked]

---

From: "Vianney Lecroart" <[email-masked]>
Date: Wed, 28 Feb 2001 15:28:35 +0100

> is how managing more than one socket per thread efficiently ?

There's no 1000000 solutions :)
In fact, if you want to be portable, there s only one solution (if think),
it s select() that is available on all BSD socket system (all unix, linux
and windows)
The other solution is poll() that it should have better performance but is
conform to XPG4-UNIX (not available on windows)
And other solutions are OS specific (on unix, i don t know other solutions).

Vianney Lecroart
---

---

From: Vincent Caron <[email-masked]>
Date: Wed, 28 Feb 2001 16:30:55 +0100

I was more or less suggesting than the solution is not a matter of adopting
select() or not, but rather implies a deeper reflexion on the pb of service
scheduling... And being cross-platform doesn't mean being uniform, it would
be a shame to ignore each OS's specificity. An approach could be to get the
best implementation, say, for Linux and Windows, find out an abstract model
that unifies both and brings this into NeL. Here again, just a hint ...

---

From: Nicolas Hognon <[email-masked]>
Date: Wed, 28 Feb 2001 16:11:22 +0100

i'm aggry with you
at this time i'll use select for windows and linux (or poll for linux)
and design my server so that i can implement another better solution.

has i said before better solution under windows are
 - are windows message and event and i don't want use it
 - overlapped I/O but i don't really understand how it works
   so it's not easy to design a server which is able to use select and this
   technic. perhaps you already know it but you can take a look at
http://www.cyberport.com/~tangent/programming/winsock/articles/io-strategies.html

-- 
Nicolas Hognon
[email-masked]

---

From: "Zane" <[email-masked]>
Date: Wed, 28 Feb 2001 09:04:59 -0800

----- Original Message -----
From: "Vincent Caron" <[email-masked]>
Subject: Re: [Nel] NeL Network Engine

> that unifies both and brings this into NeL. Here again, just a hint ...

Would not the wisest choice be to research all the high performance web
servers out there?  It seems to me that this problem has been researched
quite extensively by a LOT of other GPL projects.  I doubt you'd be able to
find a better solution than what Apache or other heavy duty web servers have
already implemented.

-E.J. Wilburn
[email-masked]

---

From: Nicolas Hognon <[email-masked]>
Date: Wed, 28 Feb 2001 19:52:32 +0100

you're right
many people and project were confront to this problem
and they resolve it.

but the design of a web server or a ftpserver is another
thing than the design of a server for a MMORPG.

i think for a web server
the browser connect to the server when it need something
but in a game like the one developped by nevrax
you're client is always connected to the server
it's not exactly the same things.

    
Enjoy the silence
(DM)

---

From: elijah wright <[email-masked]>
Date: Wed, 28 Feb 2001 13:51:45 -0500 (EST)

> it's not exactly the same things.

as i recall, the jabber.org people have done some scalability work for
just this kind of thing- their IM client is an always-on/always-available
service.

might be fun to ask some of them what their ideas on this are.

elijah

---

From: "Zane" <[email-masked]>
Date: Wed, 28 Feb 2001 11:08:22 -0800

----- Original Message -----
From: "Nicolas Hognon" <[email-masked]>
Subject: Re: [Nel] NeL Network Engine

> it's not exactly the same things.

If nothing else this simply means that it has spent a great deal of time
optimizing opening and closing connections (and waiting on incoming
connections).  Perhaps combining their opening/closing optimizations with
the optimizations Jabber has made for always-on connections would give us
the best of both worlds.  I'd be willing to bet that Jabber has already
borrowed a lot of code from projects like Apache.

Anyway, at this point I don't think I have anything else usefull to add to
the discussion since I doubt I'll get around to doing the research myself so
back into lurker mode. :)

-E.J. Wilburn
[email-masked]

---

From: [email-masked] (Bernard Hugueney)
Date: Wed, 28 Feb 2001 23:13:12 +0100

> find a better solution than what Apache or other heavy duty web servers have
Indeed, this is a MUCH debated suject. But I'm afraid Apache is not
the fastest example to follow (they have special constraints such as modules
interface) Zeus and TUX Threaded linUX webserver or BOA come to mind.
http://www.kegel.com/c10k.html seems interesting but a nit out of date ...
Anyway, there was a discussion on lkml on how to overcome select()/poll()
shortcomings with a new API. If it made it to kernel 2.4, you should be
looking that way.

Bernard

---

From: Leighton Haynes <[email-masked]>
Date: Thu, 1 Mar 2001 10:45:11 +0800

> shortcomings with a new API. If it made it to kernel 2.4, you should be
<troll>
Hey! Why don't we shove the networking engine in kernel-space! Then we
can stop all these other pesky processes from pre-empting us and stuff.
</troll>
I'm not sure if this is a really important issue right now. As many
people have noted, premature optimization is the source of all evil. 
What we _need_ is a good api to the networking code, so that the 
underlying code can be optimized later. Actually optimizing the way it works
should be dealt with when you need to deal with it. After all, it may work out
that the control code behind it all ends up needing 1/4 of a cpu/thread, at 
which point this problem morphs into a completely different one. 
Someone might release a nice lightweight threads architecture in 4 months
meaning the point is unimportant. 

Leighton...

--

Part-time student. Full-time Programmer. 
Seeking the 36 hour day and the 10 hour working week.
(08) 9272 9058 (Home - like I'm ever there)
0401 335 136 (Mobile - like it's ever on)

---

From: Vincent Archer <[email-masked]>
Date: Thu, 1 Mar 2001 09:52:21 +0100

> people have noted, premature optimization is the source of all evil. 

It is somewhat important. There are optimisations that can be done further,
but there are also fundamental design issues that reflect how everything
else in the server code has to be written.

You do not program an event-driven system like you would on a thread-based
system. Converting from one model to another would be horribly painful,
which is why the "right" one has to be selected. Once we pick one, we'll
have to stick by it, and the more code is produced under a model, the
harder it becomes to switch.

Which is why the decision about this was deferred until it can no longer
be truly avoided.

-- 
Vincent Archer                                         Email: [email-masked]

Nevrax France.                              Off on the yellow brick road we go!

---

From: Leighton Haynes <[email-masked]>
Date: Thu, 1 Mar 2001 17:33:07 +0800

> Which is why the decision about this was deferred until it can no longer

*ponder*

I can't see the fundamental difference between an event-driven and
a thread-based model. In an event-driven model , some central arbiter
gets the input, grabs the data representing the 'entity', processes it, 
and sends a reply. In a thread-based model, each entitiy is a thread,
the entity receives a message , processes it, and sends a reply. 
The fundamental differences, are in the message addressing, and 
the context switching. 

What are the issues here? So far I've seen only discussion of
how many threads we can handle, which doesn't seem to be the 
central issue. What we really need, is a list of the requirements
for this particular sub-system. (Perhaps they exist somewhere, 
but I haven't seen them).

Leighton...

--

---

From: Dim Segebart <[email-masked]>
Date: Thu, 1 Mar 2001 12:34:59 +0100

Wednesday, February 28, 2001, 11:13:12 PM, you wrote:

>> find a better solution than what Apache or other heavy duty web servers have
I think what Slashdot.org is a right place to ask our "Network Engine"
question. I hope it'll gave us a lot useful ideas.