Subject: [Nel] TCP vs. UDP

From: "Vianney Lecroart" <[email-masked]>
Date: Thu, 5 Jul 2001 16:36:47 +0200

Hi everybody!

Let's talk a little bit about this great subject: TCP vs. UDP

Thanks to Valerio Santinelli and Olivier Lejade, there's some cool
discussion about this subject on the web and I would like to have your
thought and discuss about this.

On Anarchy Online game, they only use TCP protocol between client and server
(other commercial online game use UDP) and it seems that it causes lot of
problem like horrible ping or connection lost:
http://aoforums.funcom.com/showthread.php?threadid=26297

There's also a mail on the MUD-Dev mailing list talking about these problems
here:
https://www.kanga.nu/archives/MUD-Dev-L/2001Q3/msg00009.php

As you perhaps know, we only use TCP/IP (for server-server communication,
but also for the client-server connection) and we currently think more and
more about replacing the TCP client-server connection with a UDP one.

In a first step, we would like to know your opinion about this choice. And
in a next step, we'll do some test about ping/packet lost and so on to
compare TCP and UDP, and we'll need your help to do that.

Vianney Lecroart
---
lead network programmer / nevrax.com
icq#: 6870415
homepage: http://ace.planet-d.net
www.geekcode.com: GCS/E d- s+++: a-- C+++$ UL++ P- L+++>+$ E+>- W++ N+ o? K-
w++$ O- M- V- PS- PE? Y PGP t 5? X+ R- tv++ b- DI D+ G e++ h+ r-- y?

---

From: Dim Segebart <[email-masked]>
Date: Thu, 5 Jul 2001 17:00:13 +0200

Sorry, I'm not the network guru at all, so may be it looks childish ...
I think you should implement  minimum error checking protocol -
1. Client sends UDP data to the server
2. Client wait for confirmation from other end what data was received (use some
UID for packets to send back as confirmation)
3. If some timeout expired without confirmation, then client resends its data again

Thursday, July 05, 2001, 4:36:47 PM, you wrote:

VL> Hi everybody!

VL> Let's talk a little bit about this great subject: TCP vs. UDP

VL> Thanks to Valerio Santinelli and Olivier Lejade, there's some cool
VL> discussion about this subject on the web and I would like to have your
VL> thought and discuss about this.

VL> On Anarchy Online game, they only use TCP protocol between client and server
VL> (other commercial online game use UDP) and it seems that it causes lot of
VL> problem like horrible ping or connection lost:
VL> http://aoforums.funcom.com/showthread.php?threadid=26297

VL> There's also a mail on the MUD-Dev mailing list talking about these problems
VL> here:
VL> https://www.kanga.nu/archives/MUD-Dev-L/2001Q3/msg00009.php

VL> As you perhaps know, we only use TCP/IP (for server-server communication,
VL> but also for the client-server connection) and we currently think more and
VL> more about replacing the TCP client-server connection with a UDP one.

VL> In a first step, we would like to know your opinion about this choice. And
VL> in a next step, we'll do some test about ping/packet lost and so on to
VL> compare TCP and UDP, and we'll need your help to do that.

VL> Vianney Lecroart
VL> ---
VL> lead network programmer / nevrax.com
VL> icq#: 6870415
VL> homepage: http://ace.planet-d.net
VL> www.geekcode.com: GCS/E d- s+++: a-- C+++$ UL++ P- L+++>+$ E+>- W++ N+ o? K-
VL> w++$ O- M- V- PS- PE? Y PGP t 5? X+ R- tv++ b- DI D+ G e++ h+ r-- y?

VL> _______________________________________________
VL> Nel mailing list
VL> [email-masked]
VL> /mailman/listinfo.cgi/nel

-- 
Dim Segebart                         mailto:[email-masked]

---

From: Nicolas Hognon <[email-masked]>
Date: Thu, 05 Jul 2001 17:00:15 +0200

> more about replacing the TCP client-server connection with a UDP one.

You mean you are thinking about creating a tinny TCP over UDP.
If you want to do that i'm not sure it's a good idea.

in "effective TCP/IP Programming" Jon C. Snader says
in
Tip 7 : Don't Underestimate The Performance of TCP
Tip 8 : Avoid Reinventing TCP

Ok, i'm not very useful in this conversation but
i'm not (yet :)) a network guru ....

I'm interest by answer from other people.

> _______________________________________________
> Nel mailing list
> [email-masked]
> /mailman/listinfo.cgi/nel

-- 
Nicolas Hognon
[email-masked]

Virtools - The Behavior Company
Tel. (+33) 1 42 71 46 86 / Fax. (+33) 1 42 71 86 53
http://www.virtools.com/

---

From: "Valerio Santinelli" <[email-masked]>
Date: Thu, 5 Jul 2001 17:02:54 +0200

You already know my position :)

UDP for everything except datas that should get through reliably and that
aren't making a great impact on gaming speed, like chat, which should be
reliable (i don't really want to lose pieces of discussion throughout the
net) but that it's not critical to be delivered without lag.

--
Valerio Santinelli
http://www.digisoft-multimedia.it/

---

From: Dave Turner <[email-masked]>
Date: Thu, 05 Jul 2001 11:14:47 -0400

> compare TCP and UDP, and we'll need your help to do that.

There's often a lot of lag at the beginning of a UDP "connection" (yes,
I know UDP doesn't have those).  Then gateways, which consider UDP to
have higher priority, remember that there's a connection, and things
start going faster.  Your tests should be written with this in mind.  

You might also want to look into XUDP, 
http://www.gweep.net/~rocko/XUDP_Paper/paper.html

-- 
-[Dave Turner                              Stalk me:  (215)-545-2859]
---------------------------------------------------------------------
"I would say our forefathers would have been ashamed, but they were 
slave owners." - Slashdot Anonymous Coward.  Bring Pug back!

---

From: Dim Segebart <[email-masked]>
Date: Thu, 5 Jul 2001 17:45:39 +0200

Just facts:
We  run  relatively large chat server with our own server software. We
use  TCP  for all communications between Java clients and server-side.
At  peak-load  time we have 2000 opened TCP-sockets at once. Profiling
shows  up  what most of CPU time server performs write().
And finally SW and HW - we use SunOS 5.7 aka Solaris7
Our homemade software written in C++ utilize 2,5 CPU out of 4 in our
Sun Enterprise Server 450 with 4 CPU and 1G RAM
May  be  someone  have  the  similar  service on UDP and can share his
statistic ?

--
Dim Segebart

---

From: Vincent Archer <[email-masked]>
Date: Thu, 5 Jul 2001 17:50:31 +0200

> If you want to do that i'm not sure it's a good idea.

Hi. And first, I confirm I'm the same Vincent Archer that had @nevrax.com,
but I no longer work there. Just resubscribed in time to see the discussion.

I had a vulgarisation post on the topic on one yahoogroups (Dark Age of
Camelot), but you can't access it without subscribing, so I'll summarise
the thing.

> Tip 8 : Avoid Reinventing TCP

That's good advice... if you have the objectives of TCP in mind. You don't.
In theory.

Pros of TCP:

- Reliable
- Optimised over 20 years of programming
- Fast for development
- Most of the firewalls/telecom eqpt out hereare optimised for TCP protocols;
  I know several IP providers who drop anything non-TCP based first when they
  suffer congestion on their routers.

Cons of TCP:

- TCP is stream-based, the unit is the byte. You need to provide your own
  packet framing, as you may, with one read call, catch part of a packet,
  an entire packet, or multiple packets at the same time.
- TCP has its own model for retransmission of data; you have minimal to no
  input on that model's behaviour. And the model is geared to the transmission
  of bulk data. There is one bit to indicate "interactive data" in the TCP
  protocol; no common equipment I know out there uses or checks that bit.
- TCP is strictly sequential: there is no way to access data "as soon as
  received", or to ignore/discard "obsolete data": each data chunk must be
  received, processed (at least at the framing level 5 layer, if you follow
  the OSI layer model) before the next one.

The last item in the list is what makes everyone look toward UDP (or any
other IP-level protocol in fact, but routers and firewalls are usually
disposed to treat more kindly UDP protocols than pure IP ones) for any
application with real time constraints where:

1: The exact ordering of data isn't a strong requirement
2: The immediate processing of received data is highly desirable

The real trap, as you underline, is when people design their own protocol
over UDP and end up using the same - unconscious - model as TCP; this
often leads to a "poor man's TCP" protocol that address none of the cons
while providing none of the pros.

To be efficient, any UDP-based protocol needs to be highly aware of the
data semantics (not just syntax - i.e. format - but semantics - i.e. meaning)
of what it carries. Which makes it very complex and usage-specific; you
cannot plug in a "random UDP protocol" in your app (in this case, the
client-server part of NeL architecture). If you design the protocol in
complete isolation of the application (the MMOG) inside, better go directly
to TCP; it won't be worse than anything you'll design. And it's already
made and tested.

Then, there's the whole problem of kernel resources devoted to TCP stack.
I pass here; I'm not a kernel developper.

(but anyone who plays AO knows the problems you can have merely to *connect*
 to the game at times)

-- 
	Vincent Archer			Email:	[email-masked]

All men are mortal.  Socrates was mortal.  Therefore, all men are Socrates.
							(Woody Allen)

---

From: Vincent Archer <[email-masked]>
Date: Thu, 5 Jul 2001 18:04:42 +0200

> http://aoforums.funcom.com/showthread.php?threadid=26297

This post (and the rest of the discussion) do highlight the problems of
AO. However, only one post, in the whole thread, seems to be close to the
real "problem".

I have two friends who are playing AO together. They often experience
"bad lag" (i.e. 20-30s delays between a command and it's execution).
However, there's one strange thing during these periods of bad lag.

They still chat with each other.

And I've done checking with tcpdump: you do not talk directly between clients.
Everything goes thru the AO servers.

My guess is that their architecture is based on a front-end/zone service
model. Clients connect to a front end, and said front-end connects to a
zone service, depending on the zone you are in. This is further supported
by various analysis points during beta, notably when the zone service
crashed while I was in it (and the whole mission dungeon got resert and
randomly re-rolled), and the numerous problems people have for zoning
(zone... after a strangely fixed 45s - the default TCP connection timeout -
 you get "Area Change not initiated on server).

So you have:

Client ---- TCP ----> Front End ---- TCP ----> Zone server
                          ^                  /
                          |                 /
                          V                /
Client ---- TCP ----> Front End ---- TCP -/

which is probably the worst architecture I can imagine, specially as there
appears to be one front-end per client, and front ends closes and opens
communication to zone servers. :(

> compare TCP and UDP, and we'll need your help to do that.

Packet loss is a non-sequitur under TCP. You *cannot* lose packets under TCP :)
(you lose connection first)

---

From: "Zane" <[email-masked]>
Date: Thu, 5 Jul 2001 10:08:09 -0700

----- Original Message -----
From: "Vincent Archer" <[email-masked]>
Sent: Thursday, July 05, 2001 8:50 AM

> to TCP; it won't be worse than anything you'll design. And it's already

I don't believe that this is entirely true.  Let's look at what we need out
of a protocol for NeL:

1)  Know what data HAS to be delivered and what data doesn't really.
2)  Know what data can be overwritten by new data if we recieve a large
chunk of data
at once or out of order. (If the data is spread out, not read in one call to
the network object, then we don't need to know if it can be overwritten, and
in order  the game engine will overwrite it if it's the same class of data)

Problem one is very easy to solve, we simply use a priority argument when
sending data with our network object.  In this case high priority data could
be sent via TCP and low priority data could be sent via UDP.

Problem two really isn't all that hard to solve either.  Perhaps we have a
'class' argument whend sending data via our network object.  When a packet
is sent it includes it's data class and a serial number, both client and
server keep track of both.  When we recieve data we check the class and
serial number.  If the serial number is out of sequence and the data is
considered old for that class then it's discarded, otherwise it's accepted.
If the data has no class (class is set to 0) then all data is accepted and
the serial number isn't checked.

The only problem I see is mainting the large list of data classes.  Really I
don't think it's that much of a problem and wether or not you're using this
model if you do discarding of data you'll be doing class checking in some
form anyway.

I'm sure the rest of you will point out the many other flaws I don't see.
:P

-E.J. Wilburn
[email-masked]

---

From: "Zane" <[email-masked]>
Date: Thu, 5 Jul 2001 12:05:07 -0700

----- Original Message -----
From: "Vincent Archer" <[email-masked]>
Sent: Thursday, July 05, 2001 9:04 AM

> However, there's one strange thing during these periods of bad lag.

I also play AO but every time I've experienced real "bad lag" you cannot sit
or do any action that requires server-side confirmation including ALL chat
channels.  In fact, I frequently will say something or shout something so
that I know as soon as it shows up it's done lagging and this works like a
charm.  What they're experiencing is a different type of lag than is
discussed in the above post, that's when a particular zone they're in lags
(or the server hosting that zone) but not all servers are affected.  I
haven't experienced that type of lag since beta.

As a side note I've noticed that when I get the "bad lag" others around me
appear to get it too (at least sometimes) which lends weight to the packet
storm theory.

> communication to zone servers. :(

There doesn't need to be one front-end per client.  There can be several
load-balanced front-ends that handle multiple clients each.  The major
problem with this is if one front-end crashes all those clients get dropped
(although under UNIX (I haven't been able to get win32 to do this) you can
pull some funky sockets tricks and recover from a crash without dropping
most players, just majorly lagging them & loosing some updates).

The good side to this is you only need one connection per client per
protocol (so 2 connections if using both TCP and UDP).  Unfortunately with
TCP that's both a pro and a con.  With one TCP connection a dropped packet
on a chat message delays all other TCP traffic, but it also lessens
bandwidth and server resources over multiple connections (larger, more
efficient packets).  Also, with a single front end you can have as many
seperate services as you want without having to have a ton of different
connections to the client.

Regardless, we have no data as to wether or not AO is doing it that way.
Maybe tonight I'll run it in windowed mode and check netstat.  If we've got
more than one active TCP connection to Funcom servers than that model
probably isn't what they're using.

On a side note, using multiple TCP connections would eliminate some of the
packet-loss latency issues at the cost of increased bandwidth.  Say you have
one connection for chat channels, one for inventory & stat handling, one for
world actions and one for combat.  If connection 1 drops a packet its lag
won't affect the other connections as much.  But of course if they all drop
packets at the same time we get the packet storm problem again. :)

> (you lose connection first)

Yes but TCP has latency issues, UDP has packet-loss issues.  Why can't we
have the uber protocol that has neither??? :)

BTW, does anyone know if ipv6 has addressed this issue?  I'm aware of QoS
but not sure to what degree they've taken it.  Personally I think the only
way we could get garunteed delivery with low latency is to have each router
along the way garuntee a packet is delivered (if, of course, it's load
allows that packet to be accepted in the first place).  That way if a packet
is dropped by a router due to load (or some other issue) the previous router
expects a timely response and when it doesn't get one it resends or sends
via a different route.  (Of course I would expect a per-packet ack, probably
a CRC-ack for a certain amount of traffic)  The point being the original
sender should never have to resend as long as the first router gets all the
packets.

-E.J. Wilburn
[email-masked]

---

From: David Mentre <[email-masked]>
Date: 06 Jul 2001 09:54:59 +0200

Hi,

From all other posters, it seems that performance result is highly
dependent on implementation. So to make the Really Good Choice(tm), you
should have both and choose according to *figures*. So I would do that
in the following order :
 
 1. design a "packet sending" layer which allows messages
    sending/receiving with semantics in mind (see Vincent's post in the
    same thread). Maybe NeL has already that, I haven't check the
    source, sorry.

 2. implement this layer using TCP. Should be straightforward.

   => gives you a backup implementation

   => gives you basic performance figures

 3. design and implement UDP implementation_s_ with various
    requirements depending on the following parameters:

     - packet loss or not

     - in order delivery or not

    You could start from the easiest (packet loss && out of order <=>
    raw UDP) and incrementally develop more complicated ones.

 4. substitute parts of TCP implementation (chat, position packets, ...)
    with the corresponding UDP based protocol

 5. implement a benchmark and test, test, test, ... ; then go to step 4
    and loop

Ok. I don't propose any line of code so you could directly throw this
post to /dev/null but you asked for comments. ;)

BTW, what is your Linux configuration? I've been stuck with the need to
reinstall everything (X, STLport, freetype) on my debian machine. So it
may be easier to install the Linux distribution your are using at Nevrax
on some spare partition to try to "enter the loop".

Best regards,
d.
-- 
 [email-masked]
 Opinions expressed here are only mine.

---

From: "alfred" <[email-masked]>
Date: Fri, 6 Jul 2001 19:41:50 +1000

You *must* replace your transport mechanism with UDP :)

TCP provides (nah, ensures) reliable delivery of data, but it can spend a
long time transmitting (especially on clogged up links). Between packet
errors and backoffs due to packet drops it can easily take 2seconds plus to
get data down the "tcp pipe". And seeing its a pipe, everything behind it is
"backed" up. So, time critical data can't just jump past useless data (like
MOTD's,etc).

With UDP on the otherhand, you lose the reliable data, but you get your
timeliness. Because of the datagram nature of transmission, you can also
implement independant "streams", with reliablity on some streams (via extra
software layers) and timeliness on others (auto drop of old data for eg).

Btw, you cannot "design your own layer", because you need to use the
existing internet to send your data (which will only accept UDP and TCP).
However, they probably did add to UDP, by adding an extra software layer to
provide "reliable udp" (its almost trivial to do).

Any, UDP is the only option that is sensible for anything that has time
critical performance or for things with mutliple "data channels".

BTW, I do stuff like this for a living (I am a network researched for
www.dsto.defence.gov.au), with a fair bit of work in the voice arena, so I
have some experience backing me up here :)

---

From: Vincent Archer <[email-masked]>
Date: Fri, 6 Jul 2001 13:48:53 +0200

> channels.  In fact, I frequently will say something or shout something so

I wasn't referring to chat channels. I was referring to /tell commands. Who,
apparently, works (sometimes) while the rest is in limbo.

As there is no detectable client-client communication, this shows that the
lag problem is sometimes/often an intra-server lag, not a client-server
lag.

At other times, everything freezes up, as you experience.

> probably isn't what they're using.

I've done a tcpdump. There are 2 connexions I've seen used. One goes to the
login service (dimension load/character selection). The other goes to the
game service frontend. I haven't found out if you disconnect/reconnect from
the game service; I suspect not.

I'm maybe my graphics (love ansi art) should have put "client A", and "client
B". I was suggesting different clients, each connected to a separate process
that handle their frontend, not parallel TCP streams.

> have the uber protocol that has neither??? :)

RDP. Proposed 20 years ago, never ever used :)

> but not sure to what degree they've taken it.  Personally I think the only

QoS in IPV6 is a lot better than the V4 version. At least, it uses more than
two bits in the header :)

---

From: Vincent Archer <[email-masked]>
Date: Fri, 6 Jul 2001 13:57:24 +0200

> 1)  Know what data HAS to be delivered and what data doesn't really.

That's priority, and bandwidth management. Not really TCP-vs-UDP relevant.

> 2)  Know what data can be overwritten by new data if we recieve a large

The latter requires you to look at the semantics of the data, not just the
syntax. You have to posit specific types of data, and, for example, you
don't have to retransmit data type X about object A if it has been already
sent out, but you have to retransmit data type Y about the same object A.

This quickly require you to tie the protocol to the exact game being made.
A current 3D coordinate/movement vector does not require retransmission
(or, at least, not the retransmission of the same data; you may send back
 a more up-to-date version), but it automatically implies your game has
moving 3D objects.

The example above is simple and obvious, but you quickly have to tie a lot
of knowledge about the game being made *into* the protocol. Making the
protocol generic enough to handle an "unspecified" game is *hard*. Hard
enough that I contend that using TCP in that case might be a good
compromise.

Of course, the more specific (to a game design) your library is, the easier
it is to make it UDP-optimised.

---

From: Vincent Caron <[email-masked]>
Date: Fri, 06 Jul 2001 20:21:51 +0200

> provide "reliable udp" (its almost trivial to do).

But do you need 'reliable transmission' ? Lost or delayed packets are
part of the rules, couldn't you take this into account in your network
and game code design ? Have a look at the streaming pb and protocols
such as RTP or RTSP, they deal with the pb of 1) time sync and 2) data
loss. This is sometime perfectly acceptable to loose some info in
a game and replace it with interpolated data until getting back in
sync.

---

From: "Zane" <[email-masked]>
Date: Fri, 6 Jul 2001 11:59:26 -0700

----- Original Message -----
From: "Vincent Archer" <[email-masked]>
Sent: Friday, July 06, 2001 4:57 AM

> That's priority, and bandwidth management. Not really TCP-vs-UDP relevant.

Yes, but what I'm saying is knowing that is a feature of the network object
I was discussing further into the message.

> sent out, but you have to retransmit data type Y about the same object A.

I don't mean to be confrontational but did you even read the rest of my
message?  With the network object I was discussing you pass data to be sent
to the client with Priority and data class.  The user of the object sets up
the classes in a manner such that the network object is aware of the classes
(in this case a simple enum would be enough).  Here's an example:

You want to pass property X of object A to the client.  You don't care if it
gets there or not.  You then pass the data to the network object with a low
priority and a class corresponding to the type of data property X is.

You want to pass property Y of object A to the client and you need to ensure
that it gets there.  You then pass the data to the network object with a
high priority (and no class).

>  a more up-to-date version), but it automatically implies your game has

The network object I described doesn't need to know anything other than data
class and priority.  Both of those items are determined by the engine.  The
only special knowledge the network object needs is the list of data classes.

Please re-read my e-mail.

In case the original wasn't clear enough the object itself decides which
protocol to use based on priority.  High priority data would use TCP and low
would use UDP with of course an additional header in the packet.

-E.J. Wilburn
[email-masked]

---

From: "Sal" <[email-masked]>
Date: Fri, 6 Jul 2001 18:32:46 -0400

> compare TCP and UDP, and we'll need your help to do that.

    I've coded small scale 3d client/server systems in both TCP and UDP
while developing software for Worldforge, and have playtested both systems.
The end result was that UDP gameplay just seemed smoother, especially under
low latency/bad connections.  Even when I had inserted code to purposely
drop packets, gameplay was still surprisingly bearable.

    On my LAN or for persons with fast links to the server TCP was not
really noticeably slower, though.  So with very low latency, and little
packet loss TCP would be the better choice, but unfortunately that isn't the
case for the internet just yet.  A lot of people are behind slow links,
shared internet connections, or in unfortunate locations... for some people
overseas my TCP-based software was almost unplayable, where UDP was still at
least minimally usable.  If you guys do end up benchmarking TCP vs. UDP, be
sure to test over a simulated low-latency, and/or 'noisy' link.  Drop
packets randomly, simulate possible internet conditions.  I'm sure you will
come to the same results I did.

    The majority of problems I've had with UDP are firewall related. And
they were almost all serverside, ie a person trying to run a UDP server
behind a firewall and unable to map ports/etc. to let clients connect.
Clientside, however there was very little trouble, it worked through my
ipmasq firewall without problems, for one, and others had been testing
behind firewalls also.

    However, a bit of 'reinventing TCP' was needed for the reliable data.
This is unavoidable, some messages need to be sent and can't be
ignored/dropped. And I agree that the OS's TCP implementation is probably
more efficient than one drawn up for a video game.  But I don't think using
both UDP and TCP is worth the trouble to gain a reliable data transmission
stream.  I just don't think the performance improvement would be significant
enough to justify the implementation... Let me explain why I think so.

    Now, say you were sending a 10 MB file over the internet to a friend.
_Every_ packet must make it to the peer or the file would be corrupt. Every
packet must be acknowledged somehow, using some crc, sequence number... etc.
I haven't looked at any actual TCP implementation's source code, but I'll
bet it does all sorts of compromising between packet transmission and
acknowledgement frequency,  message sizes, etc. and over the course of
sending 10 Million packets (ok, maybe less for a 10 MB file)  it probably
saves a considerable amount of time over some dumb UDP code that
acknowledges say, every 10 packets or some other simplistic mechanism.

    Yes, in that situation, reinventing TCP would be a bad idea... and yes,
it is definately a better choice than UDP, which in fact is why almost every
internet protocol uses TCP.  But in our case we aren't sending a 10MB file,
we're sending short messages to tell the client/server something important.
ie a 'player X has died' message, which could be maybe a dozen bytes in
size. All the fancy optimization that an OS's TCP stack does over the course
of time wouldn't be of much use here.  Also, in sending a 10MB file you dont
care about latency, and you know what the next few thousand packets are
going to consist of beforehand, whereas in a video game this is not the
case. The optimizations TCP gives you would only take effect if you were
sending large, continuous blocks of data.

    So for the gameplay networking layer I suggest UDP.  The only exception
in my opinion is for things like automatic client-patching, or downloading
of media.  In that case,  having the client make an HTTP connection to a web
server, or something similair is a *much* better choice than using the
'gameplay networking' layer to send this data.

    The software I spoke of here is all in Worldforge's (www.worldforge.org)
CVS repository.  The software I tested with was XClient, and XServer, which
are both in CVS.

    The UDP networking library I created and used in XClient/XServer, which
I recommend taking a look at, is called 'XUDP' (not the library that Dave
posted about, but has the same name :-/) is also in Worldforge CVS. I
extracted the UDP networking code from the GPLed Quake1 src, and converted
it into a general C++ OO UDP networking library. It contains code for both
reliable and unreliable transmission of data.  Credit should go mostly to
John Carmack of Id software for the code. I just abstracted, and tweaked a
bit for use in an MMORPG setting. It might be helpful if you guys decide to
go with UDP for Nel.

    Anyway, sorry for the long winded rant. Hope this helps some, and keep
up the great work!

- Sal

---

From: "Daniel Miller" <[email-masked]>
Date: Sun, 8 Jul 2001 18:44:09 +0200

I've been reading this thread with great interest.  There's clearly a good
case for using UDP for low priority or short life-span data.  In this case
the app software on top of NeL must clearly be designed with the fact that
some data isn't going to get through in mind.

The problem is particularly interesting because:
1. The limiting factor on the amount of data we transmit looks like being
the cost of output bandwidth for the servers.
- as far as I can ascertain, at today's costs we're looking at something
like 8-16 kilobits per client per second maximum output from the servers
(including lost packets)
2. The complexity of the scenes that we are trying to convey to the clients
is rising fast.
- view distances are increasing, as are screen resolutions which means that
you can see more dynamic world content at any given time (characters,
creatures, objects...)
- It is clear that for MMORPGs we wil soon need to be able to display crouds
or armies of 100s or even 1000s of characters and creatures at once.
- Character animation is becoming more complicated to reflect a mix of
actions performed simultaneously (such as speaking with emotional facial
expression and manipulating an object in one's hands while sitting down) -
which means more information to describe each character's state.
- and so on...

These two points combined mean that, whichever way one looks at it, a lot of
scene information will have to be filtered out or sent at low frequency.  If
one isn't careful low frequency sends can obviously be very sensitive to
packet loss - which leads me to the point of this posting: In order to work
the problem we need to have a good understanding of true internet behaviour
and to have a good set of test data for simulating it.

The trouble right now is that I have no hard data to use to model packet
loss or delivery latency over time. If anybody knows of any studies that
have been done or has any of their own data I'd be very interested. In
particular I'm interested in moderately bad connections that exhibit both
good behaviour and bad behaviour over time.

Regards to all,
Daniel

---

From: "John Cosby" <[email-masked]>
Date: Sun, 8 Jul 2001 13:28:04 -0400

The US Dept. of Defense has been doing simulation for decades.  When I first
read this thread, I though of my years-ago experience in Distributed
Interactive Simulation (DIS) and its successor, the High-Level Artchitecture
for Simulation (HLA).  Although designed primarily for LANs, both have been
successfully been used over WANs.

DIS is a protocol defined atop UDP.  Derived from work in the 70s, it's a
bit-level definition of Ethernet packets, the most important of which is the
Entity State Protocol Data Unit (PDU), which describes in all detail
necessary for rendering one particular simulation entity and was rebroadcast
whenever something changed over a configurable threshold (direction change,
speed, etc.) or a max waiting time occurred (5 secs).  Members of the
simulation were responsible for receiving and filtering things they cared
about, dead-reckoning the motion based on PDU updates, and broadcasting the
ES PDUs they were responsible for whenever the dead-reckoning thresholds
were hit or the entities timed out.  Other PDUs describe weapons fire,
detonation, weather effects, etc.  DIS was originally broadcast UDP.

HLA was an OO approach to simulation where Entities published Events, and
subscribed to Events that other Entities published.  It's based on classes,
and some reference object models are available.

There's a lot of stuff published about these topics, and an experience base
that's studied many of the concerns brought up.  Check the US Defense
Modeling and SImulation office (www.dmso.mil) and their HLA site
(www.dmso.mil/hla).  Also check the Simulation Interoperability Workshop
(http://www.sisostds.org/siw/) for lots of interesting topics and papers -
these people have been working on creating worlds for a long time.

-John

---

From: alfred <[email-masked]>
Date: Mon, 09 Jul 2001 17:27:09 +1000

DIS and HLA are not solutions for a MMORPG's. They were designed around 
LAN topologies, and there is an implicit assumption about latencies and 
bandwidths available. HLA itself is pretty ugly, as all it describes is 
a framework for a simulation "federate", it still requires lots of 
specification to get things to talk (I am working with some DIS stuff 
right now, related to wargaming for the Aust. Defence Dept, and trying 
to intergrate some comms reality into the system).
The other problem with DIS is the amount of bytes it uses on the wire, 
its pretty heavy....

Gaming requires specific solutions to the problems of latency and 
bandwidths due to the adhoc nature of gaming (anyone can play, you can't 
pre-determine optimal toplogies) and because gamers always want the 
bleeding edge.

I think nevrax should roll their own protocol, with a mix of udp, r-udp 
and tcp based on some kind of "traffic class" scheme. Shouldn't be too 
hard to design a nice OO schema to make this easy to use :)

As for simulating bad links, I hear "The Cloud" (http://www.shunra.com/) 
can be a very useful tool. It can simulate WAN links between two 
ethernet segments. Useful to stress the networking sublayer to see what 
higher level effects you get. Other similar tools can be found on this 
page (http://www.topology.org/soft/sim.html).

I don't know of any papers that have "characterised" the performance of 
the net, and I suspect that this is because it is such an agile beast. 
 From my own experience I regularly get 2-3sec pings from AU to the US, 
and 10-30% packet loss is not unheard of. Perhaps some "VoIP" studies 
exist that provide a more analytical analysis? Or even some backbone 
providers stats pages?

John Cosby wrote:

>>These two points combined mean that, whichever way one looks at it, a lot
>>
> of
> 
>>scene information will have to be filtered out or sent at low frequency.
>>
> If
> 
>>packet loss - which leads me to the point of this posting: In order to
>>
> work
> 
>>the problem we need to have a good understanding of true internet
>>
> behaviour
> 

-- 
Alfred Reynolds
[email-masked]

---

From: Olivier Lejade <[email-masked]>
Date: Mon, 09 Jul 2001 13:00:11 +0200

At 06:44 PM 7/8/01 +0200, you wrote:
>If anybody knows of any studies that
>have been done or has any of their own data I'd be very interested.

You should absolutely take a look at CiteSeer:
http://citeseer.nj.nec.com/cs
There's always tons of interesting stuff in there.

Or, you could also try the old pfd search:
http://searchpdf.adobe.com/

---

From: "Vianney Lecroart" <[email-masked]>
Date: Tue, 10 Jul 2001 17:08:53 +0200

Hi David,

> on some spare partition to try to "enter the loop".

Ok, normally, it should works on all linux distribution with gcc 2.95 or
greater
We tested it on radhat, debian and it works well. But you have to install
external library like STLport and Freetype as explains in the INSTALL
file in the NeL root directory.
We plan to modify the configure.in to enable the compilation without STLport
but
we have to be sure that NeL works compile correctly without STLport!

Vianney Lecroart
---

---

From: David Mentre <[email-masked]>
Date: 11 Jul 2001 09:23:40 +0200

"Vianney Lecroart" <[email-masked]> writes:

> We plan to modify the configure.in to enable the compilation without
> STLport but we have to be sure that NeL works compile correctly
> without STLport!

STLport is not the most difficult software to install. Anyway, thanks
for your reply. I'll try to give it a new try.

Best regards,
d.

---

From: Ed Landa <[email-masked]>
Date: Wed, 11 Jul 2001 13:34:38 -0400

> We tested it on radhat, debian and it works well. But you have to install

I've tried using STLport 4.0 and 4.5, but I still end up with an error in
one section.  It's too much C++ for this C programmer to be able to debug, so
I thought I'd ask the list if anyone else is having this problem:

(this isn't pretty, but it looks to me like it is trying to make a STL call
that doesn't exist)

$ rpm -q gcc
gcc-2.96-85

make[1]: Entering directory `/home/elanda/NeL/code/nel/src/3d'
/bin/sh ../../libtool --mode=compile c++ -DHAVE_CONFIG_H -I. -I/home/elanda/NeL/
code/nel/src/3d -I../../include    -I/usr/include/freetype2 -O3 -I/usr/include/s
tlport   -I/home/elanda/NeL/code/nel/src -Wno-multichar -D_REENTRANT -I/usr/X11R
6/include -c tile_bank.cpp
rm -f .libs/tile_bank.lo
c++ -DHAVE_CONFIG_H -I. -I/home/elanda/NeL/code/nel/src/3d -I../../include -I/us
r/include/freetype2 -O3 -I/usr/include/stlport -I/home/elanda/NeL/code/nel/src -
Wno-multichar -D_REENTRANT -I/usr/X11R6/include -Wp,-MD,.deps/tile_bank.pp -c ti
le_bank.cpp  -fPIC -DPIC -o .libs/tile_bank.lo
../../include/nel/misc/stream.h: In method `void 
NLMISC::IStream::serialSTLContLen (T &, int) [with T = 
_STL::set<string, _STL::less<string>, _STL::allocator<string> >]':
tile_bank.cpp:616:   instantiated from here
../../include/nel/misc/stream.h:570: no matching function for call to 
`_STL::set<string, _STL::less<string>, _STL::allocator<string> 
>::insert (_STL::_Rb_tree_iterator<string, _STL::_Const_traits<string> 
>)'
/usr/include/stlport/stl/_set.h:134: candidates are: 
_STL::pair<typename _STL::_Rb_tree<_Key, _Key, _STL::_Identity<_Tp>, 
_Compare, _Alloc>::const_iterator, bool> _STL::set<_Key, _Compare, 
_Alloc>::insert (const _Key &) [with _Key = string, _Compare = 
_STL::less<string>, _Alloc = _STL::allocator<string>]
/usr/include/stlport/stl/_set.h:139:                 typename 
_STL::_Rb_tree<_Key, _Key, _STL::_Identity<_Tp>, _Compare, 
_Alloc>::const_iterator _STL::set<_Key, _Compare, _Alloc>::insert 
(typename _STL::_Rb_tree<_Key, _Key, _STL::_Identity<_Tp>, _Compare, 
_Alloc>::const_iterator, const _Key &) [with _Key = string, _Compare = 
_STL::less<string>, _Alloc = _STL::allocator<string>]
make[1]: *** [tile_bank.lo] Error 1

Also, I've put together a .spec file for making a STLport RPM.  If anyone wants
it, just drop me a line.

Ed

---

From: Vincent Caron <[email-masked]>
Date: Wed, 11 Jul 2001 23:45:21 +0200

> I thought I'd ask the list if anyone else is having this problem:

Be sure to use the latest beta labeled '0619', it fixes some nasty bugs
of the 4.5b7 (http://www.stlport.org/beta/STLport-0619.tar.gz).

---

From: "Cyril Corvazier" <[email-masked]>
Date: Thu, 12 Jul 2001 09:32:00 +0200

Hi,

This error is a NeL compilation error, not a STLport version problem.
This should be fixed now. Thanks for the report.

Regards,

Cyril Corvazier
Lead 3d programmer
Nevrax France

----- Original Message -----
From: "Ed Landa" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, July 11, 2001 7:34 PM
Subject: [Nel] Linux compiling (was: TCP vs. UDP)

> /bin/sh ../../libtool --mode=compile
c++ -DHAVE_CONFIG_H -I. -I/home/elanda/NeL/
code/nel/src/3d -I../../include    -I/usr/include/freetype2 -O3 -I/usr/inclu
de/s
tlport   -I/home/elanda/NeL/code/nel/src -Wno-multichar -D_REENTRANT -I/usr/
X11R
> 6/include -c tile_bank.cpp
c++ -DHAVE_CONFIG_H -I. -I/home/elanda/NeL/code/nel/src/3d -I../../include -
I/us
r/include/freetype2 -O3 -I/usr/include/stlport -I/home/elanda/NeL/code/nel/s
rc -
Wno-multichar -D_REENTRANT -I/usr/X11R6/include -Wp,-MD,.deps/tile_bank.pp -
c ti