Subject: [Nel] RE: Compiling under Linux

From: Tels <[email-masked]>
Date: Sun, 18 Feb 2001 16:07:20 +0100 (CET)

-----BEGIN PGP SIGNED MESSAGE-----

Moin,

On 18-Feb-01 Tels tried to scribble about:
[snip]
>#checking whether c++ accepts -g... \c
>#yes
>#./configure: /dev/null: Permission denied
>#configure: error: can not run ./config.sub 

I accidentily descroyd my /dev/null (Dont ask). Sorry.

Tels

- --
 "Why do you go so slowly? Do you think this is some kind of game?"
 http://bloodgate.com/thief/     Thief - The Dark Project
 http://bloodgate.com/aifilter   Rewriting the HTML as we know it.
 http://freedomforlinks.de       Fight for your right to link. 
 PGP key available on http://bloodgate.com/tels.asc or via email.

The email address contained in the From/Reply of this email is NOT
to be displayed on webpages, entered into any mailing lists, given 
to other people, used to send me electronic greeting cards without
my written, signed permission. No SPAM nor UCE, either! Thank you!
And PLEASE refrain from forwarding me chain letters, massmailings,
or pyramid schemes.

---

From: Vincent Archer <[email-masked]>
Date: Mon, 19 Feb 2001 11:15:49 +0100

> I accidentily descroyd my /dev/null (Dont ask). Sorry.

Wow. And linux kept on working?

On many unix systems, destroying /dev/null and /dev/zero means a
straight go to single user, and system repair using static binaries,
as a lot of dynamic linking uses these files (donno for what exactly).

-- 
Vincent Archer                                         Email: [email-masked]

Nevrax France.                              Off on the yellow brick road we go!

---

From: Vincent Caron <[email-masked]>
Date: Mon, 19 Feb 2001 12:49:15 +0100

> I accidentily descroyd my /dev/null (Dont ask). Sorry.

As root :

# mknod /dev/null c 1 3
# chmod 666 /dev/null

... and it's back to life again !

---

From: "Glaze" <[email-masked]>
Date: Mon, 19 Feb 2001 15:37:25 +0100

Hello,

I have one "small" question and I hope this is the right place to ask it. I
was just wonder how (or if) you will solve the question of how to balance
the workload on the game servers, will you use some kind of dynamic load
balancing or have you opted for another approach.

Viktor Godaly

---

From: "EagleEye" <[email-masked]>
Date: Mon, 19 Feb 2001 09:30:26 -0600

I would love dynamic load balancing... just add another server to your
cluster, and assign it a piece of the game world to take care of... perhaps
that city area that just got really popular in the last few weeks...

----- Original Message -----
From: "Glaze" <[email-masked]>
To: <[email-masked]>
Sent: Monday, February 19, 2001 8:37 AM
Subject: [Nel] Dynamic load balancing?

> _______________________________________________
> Nel mailing list
> [email-masked]
> /mailman/listinfo.cgi/nel
>

---

From: Vincent Archer <[email-masked]>
Date: Mon, 19 Feb 2001 17:04:15 +0100

> that city area that just got really popular in the last few weeks...

That's not dynamic load balancing if you have to assign servers to a
specific area. That's merely 'reconfigurable static' :)

Ultima Online did this for their servers. They measured during beta
a lot of statistics, and created a map of areas for allocation to all
processors (server being ambiguous here, because their servers were
multiprocessor boxes). That map could be redone at will, depending on
the number of processors allocated to a shard. But once the shard was
started, that was it. The only way to add a processor would be to
split one area in two, and the whole migration process was hard
enough that it wasn't worth (their words) the development cost vs the
number of times it would be used.

True dynamic load would mean that each processor would adjust its
border (i.e. which objets it manages) according to its load, compared to
its neighbours. If it has too high a load, then its border would shrink
along its most lightly loaded neighbour, swapping objects between one
and the other.

Of course, that means a variable, almost fractal geometry of processors,
and a processor that, at boot time, was serving city A and its adjacent
areas could well, four days later, be serving chiefly the mountain range
5km from there, because of simple "pressure". It looks nice from a
strict research standpoint, but it's HELL for system management.

---

From: Vincent Archer <[email-masked]>
Date: Mon, 19 Feb 2001 17:33:33 +0100

> balancing or have you opted for another approach.

Hmm, I realise I've explained what is dynamic load balancing, but I
haven't answered this question.

Let's hope we have a good discussion on this :)

If you look at some of the basics in the... ok in the future design
documents, you see we're working with a kind of ORB approach, with
a naming service, which lets you discuss with specific services.

We're aiming for a functional approach, not the classic EQ/UO/AC
geographical approach. In that classic model, each processus is a
complete copy of the code. It contains everything, from spell effects
to pathfinding to combat code. Each process serves a specific "area",
which is defined by an X/Y/Z box (or, for EQ, a zone ID number, since
each area is strictly separate).

We're not going to use this approach. Chiefly because static allocation
fails when you put too many objects in the same area (be it because a
guild has decided to hold a meeting of all its 350 members there, or
because we want to have an event with an army of 100 mobs spawning).
And, as I explained somewhere else, dynamic allocation means:
1) A complete lack of control on what process is doing what, and
2) A need for a very fast way of finding who is near you and under
   which process control

What we're aiming for is a functional approach. That is, each service
provides specific functions. For example, all items in the game are
handled by a single service. Combat runs on a separate service.
And so on.

I'll let you imagine how this works, and then will further explain as
you try to shoot holes into that design. :)

---

From: Nahuel Greco <[email-masked]>
Date: Mon, 19 Feb 2001 15:46:01 -0300

On Mon, 19 Feb 2001 17:33:33 +0100

> a naming service, which lets you discuss with specific services.

You will use standard CORBA?, wich orb?

> handled by a single service. Combat runs on a separate service.

What are the "functional" parts that you will be planning? 

How do you plan to send the map to the user, i mean, if you divide the game
in areas, you can say the client to download the map for an area before
enter, but if you dont use the area-divided approach, then, you must send
the sorrounding terraing at each step that the client do?

There is no risks of overloading the internal network / get out of sync?

> you try to shoot holes into that design. :)

----------------------------------------------------------
Nahuel Greco                 Web Development - Open Source
http://www.codelarvs.com.ar  Game Programming - Research
Freelance coding / sysadmin  Networking. The answer is 42.
----------------------------------------------------------

---

From: "Jared Mark" <[email-masked]>
Date: Mon, 19 Feb 2001 15:43:59 -0600

Okay, so what you're saying is basically this...

By seperating the services, you can have a cluster of machines, each running
different services, and thus, keeping the load down across the board.  You
would have a single system, for example, handling just combat resolution,
and you would have another system that would keep track of the database...
or on an even more molecular level (assuming you have a very large world
with huge database demands) a database service that is strictly dedicated to
handling certain types of objects (like dynamic world objects such as rocks
and such, as opposed to character objects like their posessions in their
backpacks for example).

You would have other servers running the service managers for the AI of the
monsters, etc, etc, etc...

So instead of dividing up the world into geographical sectors and having a
single system handle each sector... you would divide the world up into
different processes, and have each system handle the specific process for
the entire world.

Let me know if I'm getting pretty close to what you're trying to say here.
;)  My only question on the whole thing would be... what happens if you run
into a problem where activity on a process gets to be too much for a
server... would you be forced to upgrade that server?  Or could you
theoretically split the process between two systems?  (running the same
serice twice in the same cluster, on different systems).

---

From: Thierry Mallard <[email-masked]>
Date: Tue, 20 Feb 2001 10:56:54 +0100

> and you would have another system that would keep track of the database...

We may have to add features concerning high-availability, as this scheme
doesn't allow it : if the computing holding the combat resolution fails, the
world will be a heaven of peace  ;-)

The second point is that if a service grow too much, the load-balancing won't
respond to the problem. You mentionned this point in your previous mail I
think.

One possible architecture is to write separate process, and give two or three
computer the responsiblity to distribute, by monitoring the farm, those
process. I dunno if such scheme is possible w.r.t. the current NeL objectives
(?)

Best regards,

	Shaman

-- 
Thierry Mallard              |              
GnuPG key on wwwkeys.pgp.net |
key 0xA3D021CB               |
http://thierry.mallard.com   |

---

From: Vincent Archer <[email-masked]>
Date: Tue, 20 Feb 2001 14:20:32 +0100

> > and you would have another system that would keep track of the database...

Basically, yes. It may be more complicated than this. For example, we must
strive to keep only a single service handling items, while we may run
multiple concurrent instances of AI services to have lots of smart bots.

> world will be a heaven of peace  ;-)

It's all a compromise. If a geographic server (in a geographic approach)
fails, suddendly, part of the world becomes unpassable. You can't log in
if you were in them, you can't enter them, and so on. Different failure
modes.

> process. I dunno if such scheme is possible w.r.t. the current NeL objectives

It is. In fact, a lot of services will be spread across different nodes,
but that's because these services are 'linear': each agent it manages
deals chiefly with itself. On the other hand, some are harder to split.
Item management for example, because of ownership and transfer issues:
if item A is owner by entity E, and all items of entity E are on server S1,
what happens when item is given to entity D, whose items are all on the
server S2. You break locality, which defeats the purpose of functional
services.

However, for load balancing purpose, one feature of functional split is
quite simple. If your item server, for example, manages 300,000 currently
present items, it doesn't matter if everyone currently on-line decides
to cluster in a single city: there are still 300,000 items to manage, and
the amount of item interactions haven't changed much.

---

From: Vincent Archer <[email-masked]>
Date: Tue, 20 Feb 2001 14:35:01 +0100

> You will use standard CORBA?, wich orb?

Not CORBA. As I said, 'a kind of ORB approach'. Not a straight lift
from existing solutions, who are nice from a development point of view,
but whose features are too generic and too heavy for optimisation.

> What are the "functional" parts that you will be planning? 

I don't have the exact split, which is deeply tied into game design,
but you'll have chiefly:
- access services (who translate between client protocol and service
  requests)
- PC manager services
- combat services
- item services
- databaser service (only one)
- pathfinding services
- bots service (AI)
- magic services (maybe, I'm not 100% sure if these aren't managed by the
  other services as exceptions to their rules)

then some geographic-based services

- ecology managers
- state tracking (the true geography service. That one knows where mobs,
  ground items, and PC are relative to each other... but that's about
  all it does)

and a few others, I think, who are related to higher-level game dynamics.

I might be off, there's still questions on which service handle which
agent for some cases.

> the sorrounding terraing at each step that the client do?

We use a 'mainly static terrain' approach. I.E. the client already has the
various maps parts on-line. We only send gross modifications (i.e. replace
this map part with that pre-patched one, so the rope bridge across the chasm
is no longer there) and local items information (some trees have grown
while you were away).

Sending the map on the fly requires substantial sacrifices in term of map
complexity (see Asheron's Call for a good example for this).

> There is no risks of overloading the internal network / get out of sync?

The functional approach is made by looking at the communication between
agents, and putting all agents that discuss with each other the most in
the same services, to avoid network load. Note that agents do *not*
migrate across the network, ever (i.e. a PC full state is always managed
by a specific PC service, regardless of what the PC does or where he
moves).

---

From: Nahuel Greco <[email-masked]>
Date: Tue, 20 Feb 2001 11:19:04 -0300

On Tue, 20 Feb 2001 14:35:01 +0100

> - access services (who translate between client protocol and service

I must switch to combat mode in the game?, i mean, when i combat, the scenary
and the gameplay changes, like in the PSX rpg games?

>   other services as exceptions to their rules)

If the bots service want to pathfind to some location, they must ask to the
pathfinding service?, and the pathfinding service must ask to the "state
tracking" ?, suposse that i have a bot script, running in the bot service, 
in some routine i must see if i can walk to some location, on the nort, on the
 south, etc, to decide later where i will walk.. then.. each bot script 
get_is_reachable() function will trigger a query to another service, and must 
wait the response of that service  to continue the code execution.

for thousands of scripts, that will be too slow, without thinking in the
"wait, is your query is in the queque" issue.

and btw, what scripting language do you plan to use for the code that will
execute in thousand of instances..?

> then some geographic-based services

That will instantiate, for example.. more gold to keep the economy ok ?

---

From: Nahuel Greco <[email-masked]>
Date: Tue, 20 Feb 2001 11:21:38 -0300

On Tue, 20 Feb 2001 14:20:32 +0100

> if you were in them, you can't enter them, and so on. Different failure

Then, you has the map divided in zones,... if a geographic server dies, other
will be taking the zone that belong to the died server? .. 
the zones are fixed or dynamic?

---

From: Thierry Mallard <[email-masked]>
Date: Tue, 20 Feb 2001 20:28:01 +0100

On Tue, Feb 20, 2001 at 11:21:38AM -0300, Nahuel Greco wrote:
> [... case of zone server failure ...]

The problem here will be persistency (is that english ? ;-). All the data
currently in memory on server A, which is failing, will be lost for server B
which is supposed to take over.

I'm still trying to get up to date with NeL design, so I dunno yet how the
data are/will be stored.

	Best regards,

		Shaman

---

From: Vincent Archer <[email-masked]>
Date: Wed, 21 Feb 2001 11:21:23 +0100

> which is supposed to take over.

That's what you get with a purely geographic design, like EQ/UO design.
When a zone fails, you get kicked out, everything you gained since your last
save is lost, and when you return, every single critter has respawned. All
state is lost, since all state was located in a single process.

With a functional approach, all game entities are "holographic"
regarding to storage. They're a kind of "cloud" of various agents located
on various services. If a geographic server dies, you get suddendly
a kind of fog for a while. Another service starts up, gets back information
about what critters are where from the AI services, where all players are
from the various player services, and what items are grown where from the
ecology services. It takes a little time, but you recover more
gracefully.

The same occurs if an AI server dies. Those critters that were managed
by that service freezes, but then another service takes them over. They
will probably forget a lot of what they were doing, but they don't "depop",
since the geography service knows that there's that kind of beast
that was located there.

Of course there are services that are more critical than other. If a login
service dies, all players using it are kicked out forcefully. If the
item manager dies, your inventory locks up completely (but you still look
like yourself, can move around, and talk).