Documentation

Author:

Olivier Cado

Date:

May 10, 2001

Server structure

The server provides a single receive queue (see "Receive FIFO Buffer" on the object diagram below), and one send queue per connection ("Send FIFO Buffer"). Internally, each connection is associated with a receive buffer to handle non-blocking receiving of uncomplete data blocks (in CServerBufSock). The actual receives and sends are done by CTcpSock (from layer 0).

Launching the server

The server (CBufServer) starts a listening socket (CListenSock), handled by a particular thread (CListenTask). It can then accept incoming connections.

Accepting a connection

When a connection is accepted, the server advertises the connection by pushing a connection event into the receive queue, and it dispatches the associated socket to a receive thread from the pool (or a new one).

Reading data

The user of layer 1 (note: it can be a higher NeL Net level) calls CBufServer::dataAvailable() to check for incoming data. If a connection or disconnection event is found at the top of the receive queue, the associated system callback is called. Then the socket is logically connected (CBufSock::connectedState() is true) in case of a connection event.

If dataAvailable() returned true, the user calls CBufServer::receive(data,&sockid). The second argument tells from which connection the data is coming. It can reply to it directly, by calling CBufServer::send(replydata,sockid).

The user must call CBufServer::update() for the system to work properly.

Sending data

The sending is buffered as well. The moment when the data is actually sent depends on the triggers specified. By default, the time trigger is enabled, with a value of 20 ms. It means the data will be actually sent after 20 milliseconds, provided update() is called evenly. The user can also specify a size trigger: when the size in the send buffer exceeds the specified size, the data is sent. Eventually, the user can force sending by calling flush().

How the non-blocking receiving works

Each receiving thread (CServerReceiveTask) performs a select() on its sockets. When incoming data is reported, the method CServerBufSock::receivePart() tries to read a block (which is made up of a length prefix and a payload buffer). If the actually received data is smaller than expected, it is retained in the receive buffer, for later completion. When it is complete, it is pushed into the main receive queue.

How the non-blocking sending works

All data from a send queue is copied into a buffer, then sent actually. If the sending was not done in its entirety (or it would block), part the buffer is kept for later sending.

Handling a disconnection

When a receive thread detects that a socket is disconnected, a disconnection event is pushed into the receive queue by the next CBufServer::update(). When the disconnection event is processed (at the top of the receive queue), the socket is added to the synchronized set of connections to remove of the thread. It will be effectively removed before its next select.

Wake-up pipes (Unix)

A Unix pipe is added in every select set, so that the select can be stopped when there is a new connection to add to its set or when the server is required to exit. Similarly, the listen thread performs a select on the listen socket and the wake-up pipe.

Under Windows, the wake-up mechanism is not implemented. Instead, the select timeouts are shorter.

Thread synchronization

Because of the sharing of data among different threads, some mutexes are used to synchronize data access/modification, on the receive queue, the thread pool, the connections, the set of connections to remove and the connected property of sockets (note: to ensure a fair access to the variables, the GNU/Linux implementation uses a semaphore instead of a pthread_mutex)

Gathering statistics

Several methods are provided in CBufServer to know how many bytes have been read and written.

Client (in relation to server)

The client provides one receive queue and one send queue. The receiving is done is a separate thread (CClientReceiveTask), but the actual sending flush() is done in CBufClient::update(). The socket is in blocking mode.

Unlike in the server, there is no connection advertisement. A disconnection event is pushed into the receive queue when the receive thread or the sending detects the disconnection. Besides, the client does not remove the socket after disconnecting, but at destruction time, because the user can reuse the same CBufClient object by calling again connect() after having disconnected.