Subject: [Nel] Suggestion for the NeL network library / architecture

From: Jean-Noel Moyne <[email-masked]>
Date: Fri, 8 Dec 2000 09:53:09 +0100

Just had a look at your site and your project because my brother is
going to work with you guys (as a graphic artist). I see that you have
need for fast reliable real-time asynchronous (and probably one to many)
network transport for your messaging. That need being for server to
server communication, not server to client.

You should have a look at PGM, which is an open reliable multicast
protocol designed by Cisco and TIBCO (the company I work for), which is
design to solve just this kind of problems in the most efficient way.

While TIBCO makes a complete messaging system that will use PGM and that
is a commercial product, we are also releasing an Open Source version of
the PGM protocol stack (which is probably all you need, and all you
would want to use, being an Open Source project yourself).

As a suggestion from a network professional with a lot of experience
designing large scale scalable real time oriented distributed systems,
you should take a look at it.

http://pgm.tibco.com/

    JNM

---

From: Dim Segebart <[email-masked]>
Date: Fri, 08 Dec 2000 11:39:18 +0100

Hello !
Just one question.

From http://pgm.tibco.com/draft-speakman-pgm-spec-04.txt
>>> PGM runs over a datagram multicast protocol such as IP multicast [5].

AFAIK, many ISP's ban multicast traffic and many users'll be unable to use
software based on it.
Am I wrong or missing something ?
TIA

--
Dim

Jean-Noel Moyne wrote:

> _______________________________________________
> Nel mailing list
> [email-masked]
> /mailman/listinfo.cgi/nel

---

From: Vincent Archer <[email-masked]>
Date: Tue, 12 Dec 2000 11:08:20 +0100

> design to solve just this kind of problems in the most efficient way.

Hmmm, about everyone around started talking about, then dismissing it
as unsuited to our needs, but that's chiefly due to one misunderstanding.

At first we didn't see the "server to server" sentence.

We still don't know how well multicasting can be useful in our server
architecture, because so far, we have very few one-to-many messages
across servers. Most services discuss with a specific target on a specific
service.

However, we still have lots of discussion for the "world service",
i.e. the service which "simulates" the world and run the various objects
and NPC agents. That one might be a candidate for a multicast protocol,
since we can expect replication of objects across multiple world services.

This tie in with another question which I didn't had time to reply to,
regarding how to "split" the world across the multiple world services.
If the split is done by geography (the UO model), then there is little
need of a PGM-based protocol, because most objects will be accessed by
2 world services (edges), or at most 3 (T-intersections). However, if the
split is done instead by reference (i.e. we cluster an agent with the
agents that refer to it more often), then we have a higher redundancy
of agents on servers, and a multicast is probably the best method to
send message to these agents.

-- 
Vincent Archer                                         Email: [email-masked]

Nevrax France.                              Off on the yellow brick road we go!

---

From: Vincent Archer <[email-masked]>
Date: Tue, 12 Dec 2000 11:11:10 +0100

> server communication, not server to client.

Note "server to server". What Jean-Noel was suggesting was not a protocol
to communicate with the clients, but a protocol to communicate between
the various servers that constitute a world cluster.

Since these clusters will be located in a single facility, on a dedicated
switch, the issue of IP filtering do not apply :)

---

From: MIGUEL ANGEL BLANCH LARDIN <[email-masked]>
Date: Tue, 12 Dec 2000 11:13:59 +0100 (MET)

> At first we didn't see the "server to server" sentence.

Ok, I have a dummy question, but when it is better to have multicast 
to a server delivering the messages to the rest of servers?

How portable/compatible is mulitcast technology?

---

From: Vincent Archer <[email-masked]>
Date: Tue, 12 Dec 2000 11:37:00 +0100

> to a server delivering the messages to the rest of servers?

Quite simple: When you need to send the same message to more than one
server. As soon as you need to send a message to multiple servers,
you will tie up bandwidth and CPU resending the same message over and over
to each server.

Consider the following model. Each agent (which represent an object, NPC,
PC, the door, the lamp, whatever entity in your world) is represented
typically by an object in a "world service process". Once your world
becomes large, you need multiple processes. What happens when an agent
want to "talk" to another agent on a different process. The agent, in an
ideal (and OO) world, merely talks to an object located in its process
space, said object being either the recipient agent itself, or merely a
replica, which will forward the message across the network to the real
agent.

You don't need multicast for that: its straight 1-to-1.

However, what if the message is "what is your current XYZ". That's a fairly
frequent question. If you need to query across the network, you'll quickly
end up loading the net, just to find the value three variables.

That's where PGM/multicast intervenes. In that model, the agent, when he
changes some of his variables, notifies the replica that "XYZ are now...".
I spoke about how the world services organise where an agent resides. In
some models, you have about one, maybe two replicas, which you can notify
using a standard TCP stream. Or RDP. But, if you want a load-balancing
system, you quickly end up with a replica of an object in most world
service processes. It becomes then more efficient to multicast the updates
as above to all processes, and update them all in one sweep.

> How portable/compatible is mulitcast technology?

Multicast is working, right now. However, to work across separate networks,
it requires specific support from IP routers, and resources, which most
providers aren't willing to dedicate (multi-network multicast is mostly
used for video/audio "broadcasts", video conferencing or shows).

As for PGM, well, the current library has lots of empty directories and
"TODO" notes in comments, so it's clearly a work in progress :)

---

From: MIGUEL ANGEL BLANCH LARDIN <[email-masked]>
Date: Tue, 12 Dec 2000 11:48:18 +0100 (MET)

> as above to all processes, and update them all in one sweep.

Ok, I always have though of load balancing as load-work division 
between servers.

I think that the above kind of working can have several inconsistences 
, just take a look to some distributed Mutual exclusion algos.

Just imagine that both copies of a object multicast different 
positions, you have to choose what is the best one, and whatever your 
choose is, it will be wrong, and servers can't be wrong.

Replycating work can be a real pain.
And for what you have said multicast isn't a solution for end-users.

I think that IPv6 will fix these things, by having a multicast in the 
standart, isn't it?

---

From: Vincent Archer <[email-masked]>
Date: Tue, 12 Dec 2000 12:11:26 +0100

> choose is, it will be wrong, and servers can't be wrong.

Ahem, no. I might have used object and agent in various contexts, but
an agent is the only one which is authorised to change its state. The
other objects are "replica", or passive objects. Each agent may exist
only as a single copy over the set of processes, and he is the entity
responsible for updating the local states of the replica objects in
the other processes.

If you want "what is your XYZ", the local replica answers. However, if
you want "teleport to XYZ", the message is transmitted across the network
to the unique agent, which updates its internal state, and then retransmits
to all replicas "set internal state XYZ". And the latter is when multicast
helps when there are many replicas to update.

> Replycating work can be a real pain.

Hey, if I could buy a terahertz processor and have everything run on a
single process, I'd be happy too :)

---

From: MIGUEL ANGEL BLANCH LARDIN <[email-masked]>
Date: Tue, 12 Dec 2000 12:20:32 +0100 (MET)

> helps when there are many replicas to update.

Ok, as far as I am thinking about agents, a replica only show info but 
can't process code. So all the modificators should be send to the 
main agent, that then update.

There is still the possibility of an object to get two contradictory 
updates at the same time. To avoid this you should order the messages, 
something that I feel is unefficient and ugly.

Just image agent is a ball, and then two user decide to kick the two 
replica of the agent, so the replicas get the kick commnand and send 
it to the agent.

> single process, I'd be happy too :)

If you need a terahertz processor to run the server, you better start 
again the design stage.

Just I want to note that replycanting the work can lead to ugly 
things and race conditions.