[Nel] NeL Status Update?

Fri, 20 Apr 2001 19:06:03 +0200

Hi,
+
+> E> Is there any way I can get an ETA on that 3DS plugin, and detailed
+documentation of how to structure the data for my world, so NeL will work
+with it?
+
+What you said about the NeL file format makes a lot of sense,  and it match
+what we planned to do in a near future.
+
+Actually, i'm going to describe the current data management.
+
+Today, our export plug-ins generate NeL binary files using the NeL
+serialisation system.
+The serialisation system is described in the document inserted at the end of
+the mail. This document
+will be available on the web site in the Doxygen Related Pages during the
+next week.
+
+Here is the URL of another document that describes how to build NeL 3d data
+from your 3d editor and export them in NeL binary format:
+
+http://www.nevrax.org/docs/doxygen/nel/3d_data_howto.html
+
+---
+
+NeL Files and Serialisation
+
+* Introduction
+
+This is really quite a difficult subject to write about - so this file is an
+introduction which describes the basic features and principles of our
+system.
+
+
+* How our files work in NeL
+
+The NeL files are NOT designed to be man-readable.  Interpretation and
+generation of file contents is performed by the objects that are to be read
+and written using a standardised mechanism. This mechanism was inspired by
+the system provided by Java.
+
+We use the term 'serialisable' to describe a class  that can be read from/
+written to a NeL data file.
+Counter-intuitive as it may, at first, appear, each 'serialisable' class
+supplies a single method that is used for both reading and writing.
+
+Note that the files are encoded in little-endian and that the NeL library
+code deals with conversion of endian-ness for big-endian platforms
+
+
+* Serialisation beyond files
+
+The serialisation system can be used for generating binary data buffers in
+memory (without writing the result to a file) or for packing and unpacking
+data for transfer over a LAN.
+
+
+* How it works
+
+Technically, we define a 'serialisable' class as a class that can be passed
+to IStream::serial().
+In order for a class to be serialisable it is sufficient for it to include
+the following method:
+ void serial(IStream&).
+The fact that we use a template method definition means that a serialisable
+class does not have to be derived from any other class.
+All standard types are serialisable due to a non-template prototypes shown
+below.
+STL containers of serialisable types are serialisadble
+Pointers to non-polymorphic serialisable types are serialisable.
+
+The IStream class definition looks something like this:
+
+ class IStream
+ {
+ ...
+  void serial (int&);
+  void serial (float&);
+ ...
+  template <class T> void serial (T&t)
+  {
+   t.serial (*this);
+  }
+ };
+
+
+Example:
+To make the following class serialisable:
+
+ class myFirstClass
+ {
+  int a,b;
+ };
+
+you would need to extend the class as follows:
+
+ class myFirstClass
+ {
+  int a,b;
+  void serial (IStream&istream)
+  {
+   istream.serial(a);
+   istream.serial(b);
+  }
+ };
+
+The following example shows how to serialise a more complicated data
+structure
+
+ class myFirstClass
+ {
+  void serial (IStream&);
+ };
+
+ class myclass
+ {
+  int     BaseType;
+  myFirstClass    SerialisableClass
+  std::vector< myFirstClass>  STLContainerOfSerialisableClass;
+  myFirstClass    *PointerToSerialisableClass;
+  std::vector< myFirstClass*>  STLContainerOfPointersToSerialisableClass;
+
+  void serial (IStream&istream)
+  {
+   istream.serial(BaseType);
+   istream.serial(SerialisableClass);
+   istream.serialCont(STLContainerOfSerialisableClass);
+   istream.serialPtr(PointerToSerialisableClass);
+   istream.serialContPtr(STLContainerOfPointersToSerialisableClass);
+  }
+ };
+
+
+
+* Dealing with cross referenced or hierarchical data
+
+If an object contains a pointer to another object in memory then the
+serialPtr() method is used to read/ write the referenced object.
+The NeL library code writes a value corresponding to the pointer to the
+serialised data, followed by the data that the pointer points to (In the
+case of a NULL pointer the value 0 is written without any following data)
+The NeL library code automatically deals with the cases where two or more
+objects reference the same object or there is a circular reference.  Each
+time a pointer is de-referenced, for writing, NeL checks against a table of
+previous pointers;  if the pointer value already exists in the table then no
+data is written.  At read time the data structures are faithfully
+reconstructed.
+
+
+* Dealing with polymorphism within cross referenced data
+
+In a nut shell, in order to un-serialise a data record that one only has an
+interface type for, one needs to store an additional identifier with the
+data record that identifies it's real type.  The mechanism for doing this is
+best shown with an example:
+
+ class IBaseClass : public IStreamable
+ {
+  // This class is an interface. It is polymorphic.
+  virtual void foo ()=0;
+
+  // It must declare it's name
+  NLMISC_DECLARE_CLASS (MyClass);
+ };
+
+ class CClassToSerialise
+ {
+  IBaseClass  *PointerToAPolymorphicClass;
+
+  void serial (IStream& s)
+  {
+   s.serialPolyPtr (PointerToAPolymorphicClass);
+  }
+ };
+
+ void main ()
+ {
+ ...
+  // The polymorphic class must be registered in the registry
+  NLMISC_REGISTER_CLASS (MyClass);
+ ...
+ }
+
+
+* Dealing with file format evolution
+
+ void serial (IStream& s)
+ {
+  // At the begining of the serial process, read/ write the version number
+of the class implementation
+
+  // In the following example - at read time 'version' contains the version
+read from the stream. At
+  // write time version code '3' is written to the stream and to the
+variable 'version'.
+  int version=s.serialVersion (3);
+
+  // Now switch the version
+  switch (version)
+  {
+  case 3:
+   // The last field added in the class
+   s.serial (LastField);
+
+   // do some different stuff at read time and write time
+   if (s.isReading())
+   {
+    // at read time
+    ...
+   }
+   else
+   {
+    // at write time
+    ...
+   }
+
+  case 2:
+   // note that the code provided as of here allows for the reading of old
+versions of the class
+
+   s.serial (Toto);
+
+   // in the case where the evolution from my version 1 implementation to my
+version 2
+   // is not simply an extension of version 1 we need to break execution
+here
+   break;
+
+  case 1:
+   s.serial (Foo);
+  case 0:
+   s.serial (Truc);
+  }
+
+ }
+
+
+* NeL File Headers
+
+The objective of NeL file headers is to verify that a file is in the right
+format before attempting to interpret the contents.
+
+ // The NeL team use the following advise serialise a file this way:
+ void CFileRootClass::serial (IStream& s)
+ {
+  // First write / read-check the header
+  s.serialCheck ((uint32)'_LEN');
+  s.serialCheck ((uint32)'HSEM');
+
+  // This code write / read-check the header 'NEL_MESH' at the beginning of
+the file.
+  // If the check fails, serialCheck throws the EInvalidDataStream
+exception.
+ }
+
+* Good examples to look at:
+
+ include/nel/misc/stream.h   // Stream base classes
+ class CTileBank in src/3d/tile_bank.cpp // Good example of file format
+evolution
+ class CAnimation in src/3d/animation.cpp // Good example of polymorphism
+
+
+---
+Cyril Corvazier
+Lead 3d programmer
+Nevrax France
+
+
+