NLMISC::CI18N Class Reference

#include <i18n.h>

---

Detailed Description

Class for the internationalisation. It's a singleton pattern.

This class provide an easy way to localise all string. First you have to get all available language with getLanguageNames(). If you already know the number of the language (that is the index in the vector returns by getLanguagesNames()), you can load the language file with load(). Now, you can get a localised string with his association with get().

        // get all language names (you must call this before calling load())
        CI18N::getLanguageNames ();
        // load the language 1 (french)
        CI18N::load (1);
        // display "Salut" that is the "hi" string in the selected language (french).
        nlinfo (CI18N::get("hi").c_str ());
        // display "rms est un master", the french version of the string
        nlinfo (CI18N::get("%s is a master").c_str (), "mrs");
 *

If the string doesn't exist, it will be automatically added in all language files with a mention. If the language file doesn't exist, it'll be automatically create.

Update 26-02-2002 Boris Boucher

Language are now preferably handled via official language code. We use the ISO 639-1 code for language. Optionnaly, we can append a country code (ISO 3066) to differentiate between language flavor (eg chinese is ISO 639-1 zh, but come in traditionnal or simplified form. So we append the country code : zh-CN (china) for simplified, zh for traditionnal).

Author:

Vianney Lecroart

Nevrax France

Date:

2000

Definition at line 76 of file i18n.h.

Static Public Member Functions
std::string	encodeUTF8 (const ucstring &str)
const ucstring &	get (const std::string &label)
	Find a string in the selected language and return his association.
ucstring	getCurrentLanguageName ()
	Returns the name of the language in english (french, english...).
const std::vector< std::string > &	getLanguageCodes ()
const std::vector< ucstring > &	getLanguageNames ()
void	load (const std::string &languageCode)
	Load a language file depending of the language.
ucstring	makeMarkedString (ucchar openMark, ucchar closeMark, const ucstring &text)
void	readTextBuffer (uint8 *buffer, uint size, ucstring &result, bool forceUtf8=false)
void	readTextFile (const std::string &filename, ucstring &result, bool forceUtf8=false, bool fileLookup=true, bool preprocess=false)
	Temporary, we don't have file system for now, so we do a tricky cheat. there s not check so be careful!
void	remove_C_Comment (ucstring &commentedString)
void	setLoadProxy (ILoadProxy *loadProxy)
	Set the load proxy class. Proxy can be NULL to unregister.
void	writeTextFile (const std::string filename, const ucstring &content, bool utf8=true)
std::string	hashToString (uint64 hash)
void	hashToUCString (uint64 hash, ucstring &dst)
uint64	makeHash (const ucstring &str)
uint64	stringToHash (const std::string &str)
Category control
bool	parseLabel (ucstring::const_iterator &it, ucstring::const_iterator &last, std::string &label)
	Parse a label.
bool	parseMarkedString (ucchar openMark, ucchar closeMark, ucstring::const_iterator &it, ucstring::const_iterator &last, ucstring &result)
	Parse a marked string. NB : usualy, we use [ and ] as string delimiters in translation files.
void	skipWhiteSpace (ucstring::const_iterator &it, ucstring::const_iterator &last, ucstring *storeComments=NULL)
Private Types
typedef std::map< std::string, ucstring >	StrMapContainer
Static Private Attributes
const std::string	_LanguageCodes []
bool	_LanguagesNamesLoaded = false
ILoadProxy *	_LoadProxy = 0
const uint	_NbLanguages = sizeof(CI18N::_LanguageCodes) / sizeof(std::string)
const ucstring	_NotTranslatedValue
sint32	_SelectedLanguage = -1
StrMapContainer	_StrMap
bool	_StrMapLoaded = false

---

Member Typedef Documentation

typedef std::map<std::string, ucstring> NLMISC::CI18N::StrMapContainer [private]

Definition at line 185 of file i18n.h.

---

Member Function Documentation

string NLMISC::CI18N::encodeUTF8 (  const ucstring &  str  )  [static]

Encode a unicode string into a string using UTF-8 encoding. Definition at line 627 of file i18n.cpp. References ucstring::toUtf8(). Referenced by writeTextFile(). { return str.toUtf8(); /* string res; ucstring::const_iterator first(str.begin()), last(str.end()); for (; first != last; ++first) { //ucchar c = *first; uint nbLoop = 0; if (*first < 0x80) res += char(*first); else if (*first < 0x800) { ucchar c = *first; c = c >> 6; c = c & 0x1F; res += c | 0xC0; nbLoop = 1; } else if (*first < 0x10000) { ucchar c = *first; c = c >> 12; c = c & 0x0F; res += c | 0xE0; nbLoop = 2; } for (uint i=0; i<nbLoop; ++i) { ucchar c = *first; c = c >> ((nbLoop - i - 1) * 6); c = c & 0x3F; res += char(c) | 0x80; } } return res; */ }

const ucstring & NLMISC::CI18N::get (  const std::string &  label  )  [static]

Find a string in the selected language and return his association. Definition at line 140 of file i18n.cpp. References _LanguageCodes, _SelectedLanguage, _StrMap, and nlwarning. Referenced by getCurrentLanguageName(). { if (label.empty()) { static ucstring emptyString; return emptyString; } StrMapContainer::iterator it(_StrMap.find(label)); if (it != _StrMap.end()) return it->second; nlwarning("I18N: The string %s did not exist in language %s", label.c_str(), _LanguageCodes[_SelectedLanguage].c_str()); static ucstring badString; badString = ucstring(std::string("<NotExist:")+label+">"); return badString; }

ucstring NLMISC::CI18N::getCurrentLanguageName (   )  [static]

Returns the name of the language in english (french, english...). Definition at line 162 of file i18n.cpp. References get(). { return get("LanguageName"); }

const std::vector<std::string>& NLMISC::CI18N::getLanguageCodes (   )  [static]

Return a vector with all language code available. Code are ISO 639-2 compliant. As in getLanguageNames(), the index in the vector can be used to call load()

const std::vector<ucstring>& NLMISC::CI18N::getLanguageNames (   )  [static]

Return a vector with all language available. The vector contains the name of the language. The index in the vector is used in load() function

string NLMISC::CI18N::hashToString (  uint64  hash  )  [static]

Definition at line 705 of file i18n.cpp. References uint32, and uint64. { uint32 *ph = (uint32*)&hash; char temp[] = "0011223344556677"; sprintf(temp, "%08X%08X", ph[0], ph[1]); return string(temp); }

void NLMISC::CI18N::hashToUCString (  uint64  hash, ucstring &  dst )  [static]

Definition at line 716 of file i18n.cpp. References sint, ucchar, and uint64. { static ucchar cvtTable[]= {'0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F'}; dst.resize(16); for(sint i=15;i>=0;i--) { // Must decal dest of 8, cause of hashToString code (Little Endian) dst[(i+8)&15]= cvtTable[hash&15]; hash>>=4; } }

void NLMISC::CI18N::load (  const std::string &  languageCode  )  [static]

Load a language file depending of the language. Definition at line 67 of file i18n.cpp. References _LanguageCodes, _LoadProxy, _NbLanguages, _SelectedLanguage, _StrMap, _StrMapLoaded, NLMISC::CI18N::ILoadProxy::loadStringFile(), nlwarning, parseLabel(), parseMarkedString(), readTextFile(), remove_C_Comment(), skipWhiteSpace(), and uint. { // nlassert (lid < _NbLanguages); // nlassert (_LanguagesNamesLoaded); uint i; for (i=0; i<_NbLanguages; ++i) { if (_LanguageCodes[i] == languageCode) break; } if (i == _NbLanguages) { nlwarning("I18N: Unknow language code : %s, defaulting to %s", _LanguageCodes[0].c_str()); i = 0; } std::string fileName = _LanguageCodes[i] + ".uxt"; _SelectedLanguage = i; if (_StrMapLoaded) _StrMap.clear (); else _StrMapLoaded = true; ucstring text; // read in the text if (_LoadProxy) _LoadProxy->loadStringFile(fileName, text); else readTextFile(fileName, text); // remove any comment remove_C_Comment(text); ucstring::const_iterator first(text.begin()), last(text.end()); std::string lastReadLabel("nothing"); while (first != last) { skipWhiteSpace(first, last); std::string label; ucstring ucs; if (!parseLabel(first, last, label)) { nlwarning("I18N: Error reading label field in %s. Stop reading after %s.", fileName.c_str(), lastReadLabel.c_str()); return; } lastReadLabel = label; skipWhiteSpace(first, last); if (!parseMarkedString('[', ']', first, last, ucs)) { nlwarning("I18N: Error reading text for label %s in %s. Stop reading.", label.c_str(), fileName.c_str()); return; } // ok, a line read. std::pair<std::map<std::string, ucstring>::iterator, bool> ret; ret = _StrMap.insert(std::make_pair(label, ucs)); if (!ret.second) { nlwarning("I18N: Error in %s, the label %s exist twice !", fileName.c_str(), label.c_str()); } skipWhiteSpace(first, last); } // a little check to ensure that the lang name has been set. StrMapContainer::iterator it(_StrMap.find("LanguageName")); if (it == _StrMap.end()) { nlwarning("I18N: In file %s, missing LanguageName translation (should be first in file)", fileName.c_str()); } }

uint64 NLMISC::CI18N::makeHash (  const ucstring &  str  )  [static]

Definition at line 679 of file i18n.cpp. References uint, uint32, uint64, and uint8. { // on passe au moins 8 fois sur chaque octet de resultat if (str.empty()) return 0; const uint32 MIN_TURN = 8*8; uint64 hash = 0; uint8 *ph = (uint8*)&hash; uint8 *pc = (uint8*)str.data(); uint nbLoop = max(uint32(str.size()*2), MIN_TURN); uint roll = 0; for (uint i=0; i<nbLoop; ++i) { ph[(i/2) & 0x7] += pc[i%(str.size()*2)] << roll; ph[(i/2) & 0x7] += pc[i%(str.size()*2)] >> (8-roll); roll++; roll &= 0x7; } return hash; }

ucstring NLMISC::CI18N::makeMarkedString (  ucchar  openMark, ucchar  closeMark, const ucstring &  text )  [static]

Definition at line 590 of file i18n.cpp. References ucchar. { ucstring ret; ret.push_back(openMark); ucstring::const_iterator first(text.begin()), last(text.end()); for (; first != last; ++first) { if (*first == '\n') { ret += '\\'; ret += 'n'; } else if (*first == '\t') { ret += '\\'; ret += 't'; } else if (*first == closeMark) { // excape the embeded closing mark ret += '\\'; ret += closeMark; } else { ret += *first; } } ret += closeMark; return ret; }

bool NLMISC::CI18N::parseLabel (  ucstring::const_iterator &  it, ucstring::const_iterator &  last, std::string &  label )  [static]

Parse a label. Definition at line 264 of file i18n.cpp. Referenced by load(). { label.erase(); // first char must be A-Za-z@_ if (it != last && ( (*it >= '0' && *it <= '9') || (*it >= 'A' && *it <= 'Z') || (*it >= 'a' && *it <= 'z') || (*it == '_') || (*it == '@') ) ) label.push_back(char(*it++)); else return false; // other char must be [0-9A-Za-z@_]* while (it != last && ( (*it >= '0' && *it <= '9') || (*it >= 'A' && *it <= 'Z') || (*it >= 'a' && *it <= 'z') || (*it == '_') || (*it == '@') ) ) label.push_back(char(*it++)); return true; }

bool NLMISC::CI18N::parseMarkedString (  ucchar  openMark, ucchar  closeMark, ucstring::const_iterator &  it, ucstring::const_iterator &  last, ucstring &  result )  [static]

Parse a marked string. NB : usualy, we use [ and ] as string delimiters in translation files. Definition at line 297 of file i18n.cpp. References nlwarning, and ucchar. Referenced by load(), and readTextFile(). { // ucstring ret; result.erase(); // parse a string delimited by the specified opening and closing mark if (it != last && *it == openMark) { ++it; while (it != last && *it != closeMark) { // ignore tab, new lines and line feed if (*it == openMark) { nlwarning("I18N: Found a non escaped openmark %c in a delimited string (Delimiters : '%c' - '%c')", char(openMark), char(openMark), char(closeMark)); return false; } if (*it == '\t' || *it == '\n' || *it == '\r') ++it; else if (*it == '\\' && it+1 != last && *(it+1) != '\\') { ++it; // this is an escape sequence ! switch(*it) { case 't': result.push_back('\t'); break; case 'n': result.push_back('\n'); break; case 'd': // insert a delete result.push_back(8); break; default: // escape the close mark ? if(*it == closeMark) result.push_back(closeMark); // escape the open mark ? else if(*it == openMark) result.push_back(openMark); else { nlwarning("I18N: Ignoring unknown escape code \\%c (char value : %u)", char(*it), *it); return false; } } ++it; } else if (*it == '\\' && it+1 != last && *(it+1) == '\\') { // escape the \ char ++it; result.push_back(*it); ++it; } else result.push_back(*it++); } if (it == last || *it != closeMark) { nlwarning("I18N: Missing end of delimited string (Delimiters : '%c' - '%c')", char(openMark), char(closeMark)); return false; } else ++it; } else { nlwarning("I18N: Malformed or non existent delimited string (Delimiters : '%c' - '%c')", char(openMark), char(closeMark)); return false; } return true; }

void NLMISC::CI18N::readTextBuffer (  uint8 *  buffer, uint  size, ucstring &  result, bool  forceUtf8 = false )  [static]

Read the content of a buffer as a unicode text. This is to read preloaded unicode files. The method support 16 bits or 8bits utf-8 tagged buffer. 8 bits UTF-8 encofing can be reconized by a non official header : EF,BB, BF. 16 bits encoding can be reconized by the official header : FF, FE, witch can be reversed if the data are MSB first. Optionnaly, you can force the reader to consider the file as UTF-8 encoded. Definition at line 475 of file i18n.cpp. References buffer, ucstring::fromUtf8(), nlassert, size, src, uint, uint16, and uint8. Referenced by readTextFile(). { static uint8 utf16Header[] = {char(0xff), char(0xfe)}; static uint8 utf16RevHeader[] = {char(0xfe), char(0xff)}; static uint8 utf8Header[] = {char(0xef), char(0xbb), char(0xbf)}; if (forceUtf8) { if (size>=3 && buffer[0]==utf8Header[0] && buffer[1]==utf8Header[1] && buffer[2]==utf8Header[2] ) { // remove utf8 header buffer+= 3; size-=3; } std::string text((char*)buffer, size); result.fromUtf8(text); } else if (size>=3 && buffer[0]==utf8Header[0] && buffer[1]==utf8Header[1] && buffer[2]==utf8Header[2] ) { // remove utf8 header buffer+= 3; size-=3; std::string text((char*)buffer, size); result.fromUtf8(text); } else if (size>=2 && buffer[0]==utf16Header[0] && buffer[1]==utf16Header[1] ) { // remove utf16 header buffer+= 2; size-= 2; // check pair number of bytes nlassert((size & 1) == 0); // and do manual conversion uint16 *src = (uint16*)(buffer); result.resize(size/2); for (uint j=0; j<result.size(); j++) result[j]= *src++; } else if (size>=2 && buffer[0]==utf16RevHeader[0] && buffer[1]==utf16RevHeader[1] ) { // remove utf16 header buffer+= 2; size-= 2; // check pair number of bytes nlassert((size & 1) == 0); // and do manual conversion uint16 *src = (uint16*)(buffer); result.resize(size/2); uint j; for (j=0; j<result.size(); j++) result[j]= *src++; // Reverse byte order for (j=0; j<result.size(); j++) { uint8 *pc = (uint8*) &result[j]; std::swap(pc[0], pc[1]); } } else { // hum.. ascii read ? // so, just to a direct conversion std::string text((char*)buffer, size); result = text; } }

void NLMISC::CI18N::readTextFile (  const std::string &  filename, ucstring &  result, bool  forceUtf8 = false, bool  fileLookup = true, bool  preprocess = false )  [static]

Temporary, we don't have file system for now, so we do a tricky cheat. there s not check so be careful! Read the content of a file as a unicode text. The method support 16 bits or 8bits utf-8 tagged files. 8 bits UTF-8 encofing can be reconized by a non official header : EF,BB, BF. 16 bits encoding can be reconized by the official header : FF, FE, witch can be reversed if the data are MSB first. Optionnaly, you can force the reader to consider the file as UTF-8 encoded. Optionnaly, you can ask the reader to interpret #include commands. Definition at line 377 of file i18n.cpp. References ucstring::c_str(), file, nldebug, nlwarning, NLMISC::CIFile::open(), parseMarkedString(), readTextBuffer(), skipWhiteSpace(), ucstring::toString(), uint, and uint8. Referenced by load(). { std::string fullName; if (fileLookup) fullName = CPath::lookup(filename, false); else fullName = filename; if (fullName.empty()) return; // If ::lookup is used, the file can be in a bnp and CFile::fileExists fails. // \todo Boris bool isInBnp = fullName.find('@') != string::npos; if (!isInBnp && !CFile::fileExists(fullName)) { nlwarning("CI18N::readTextFile : file '%s' does not exist, returing empty string", fullName.c_str()); return; } NLMISC::CIFile file(fullName); // Fast read all the text in binary mode. std::string text; text.resize(file.getFileSize()); file.serialBuffer((uint8*)(&text[0]), text.size()); // Transform the string in ucstring according to format header if (!text.empty()) readTextBuffer((uint8*)&text[0], text.size(), result, forceUtf8); if (preprocess) { ucstring final; // parse the file, looking for preprocessor command. ucstring::size_type pos = 0; ucstring::size_type lastPos = 0; ucstring includeCmd("#include"); while ((pos = result.find(includeCmd, pos)) != ucstring::npos) { // copy the previous text final += result.substr(lastPos, pos - lastPos); // extract the inserted file name. ucstring::const_iterator first, last; first = result.begin()+pos+includeCmd.size(); last = result.end(); ucstring name; skipWhiteSpace(first, last); if (parseMarkedString('\"', '\"', first, last, name)) { string subFilename = name.toString(); nldebug("I18N: Including '%s' into '%s'", subFilename.c_str(), filename.c_str()); ucstring inserted; { CIFile testFile; if (!testFile.open(subFilename)) { // try to open the include file relative to current file subFilename = CFile::getPath(filename)+subFilename; } } readTextFile(subFilename, inserted, forceUtf8, fileLookup, preprocess); final += inserted; } else { nlwarning("I18N: Error parsing include file in line '%s' from file '%s'", result.substr(pos, result.find(ucstring("\n"), pos) - pos).c_str(), filename.c_str()); } pos = lastPos = first - result.begin(); } // copy the remaining chars ucstring temp = final; for (uint i=lastPos; i<result.size(); ++i) { temp += result[i]; } // final = final + temp; result.swap(temp); } }

void NLMISC::CI18N::remove_C_Comment (  ucstring &  commentedString  )  [static]

Remove any C style comment from the passed string. Definition at line 168 of file i18n.cpp. Referenced by load(). { { ucstring temp; temp.reserve(commentedString.size()); ucstring::const_iterator first(commentedString.begin()), last(commentedString.end()); for (;first != last; ++first) { temp.push_back(*first); if (*first == '[') { // no comment inside string literal while (++first != last) { temp.push_back(*first); if (*first == ']') break; } } else if (*first == '/') { // start of comment ? ++first; if (first != last && *first == '/') { temp.pop_back(); // one line comment, skip until end of line while (first != last && *first != '\n') ++first; } else if (first != last && *first == '*') { temp.pop_back(); // start of multiline comment, skip until we found '*/' while (first != last && !(*first == '*' && (first+1) != last && *(first+1) == '/')) ++first; // skip the closing '/' if (first != last) ++first; } else { temp.push_back(*first); } } } commentedString.swap(temp); } }

void NLMISC::CI18N::setLoadProxy (  ILoadProxy *  loadProxy  )  [static]

Set the load proxy class. Proxy can be NULL to unregister. Definition at line 61 of file i18n.cpp. References _LoadProxy. { _LoadProxy = loadProxy; }

void NLMISC::CI18N::skipWhiteSpace (  ucstring::const_iterator &  it, ucstring::const_iterator &  last, ucstring *  storeComments = NULL )  [static]

Skip the white space. You can optionnaly pass a ucstring pointer to receive any comments string that build the white space. This is usefull if you whant to keep the comments. NB : comments are appended to the comments string. Definition at line 220 of file i18n.cpp. Referenced by load(), and readTextFile(). { while (it != last && ( *it == 0xa || *it == 0xd || *it == ' ' || *it == '\t' || (storeComments && *it == '/' && it+1 != last && *(it+1) == '/') || (storeComments && *it == '/' && it+1 != last && *(it+1) == '*') )) { if (storeComments && *it == '/' && it+1 != last && *(it+1) == '/') { // found a one line C comment. Store it until end of line. while (it != last && *it != '\n') storeComments->push_back(*it++); // store the final '\n' if (it != last) storeComments->push_back(*it++); } else if (storeComments && *it == '/' && it+1 != last && *(it+1) == '*') { // found a multiline C++ comment. store until we found the closing '*/' while (it != last && !(*it == '*' && it+1 != last && *(it+1) == '/')) storeComments->push_back(*it++); // store the final '*' if (it != last) storeComments->push_back(*it++); // store the final '/' if (it != last) storeComments->push_back(*it++); // and a new line. storeComments->push_back('\r'); storeComments->push_back('\n'); } else { // just skip white space or don't store comments ++it; } } }

uint64 NLMISC::CI18N::stringToHash (  const std::string &  str  )  [static]

Definition at line 730 of file i18n.cpp. References nlassert, uint32, and uint64. Referenced by STRING_MANAGER::TGetWorksheetHashValue::operator()(). { nlassert(str.size() == 16); uint32 low, hight; string sl, sh; sh = str.substr(0, 8); sl = str.substr(8, 8); sscanf(sh.c_str(), "%08X", &hight); sscanf(sl.c_str(), "%08X", &low); uint64 hash; uint32 *ph = (uint32*)&hash; ph[0] = hight; ph[1] = low; return hash; }

void NLMISC::CI18N::writeTextFile (  const std::string  filename, const ucstring &  content, bool  utf8 = true )  [static]

Write a unicode text file using unicode 16 or UTF-8 encoding. Definition at line 557 of file i18n.cpp. References encodeUTF8(), file, uint, and uint16. { COFile file(filename); if (!utf8) { // write the unicode 16 bits tag uint16 unicodeTag = 0xfeff; file.serial(unicodeTag); uint i; for (i=0; i<content.size(); ++i) { uint16 c = content[i]; file.serial(c); } } else { static char utf8Header[] = {char(0xef), char(0xbb), char(0xbf), 0}; std::string str = encodeUTF8(content); // add the UTF-8 'not official' header str = utf8Header + str; uint i; for (i=0; i<str.size(); ++i) { file.serial(str[i]); } } }

---

Field Documentation

const std::string NLMISC::CI18N::_LanguageCodes [static, private]

Initial value: { std::string("en"), std::string("fr"), std::string("zh-TW"), std::string("zh-CN") } Definition at line 44 of file i18n.cpp. Referenced by get(), and load().

bool NLMISC::CI18N::_LanguagesNamesLoaded = false [static, private]

Definition at line 56 of file i18n.cpp.

CI18N::ILoadProxy * NLMISC::CI18N::_LoadProxy = 0 [static, private]

Definition at line 58 of file i18n.cpp. Referenced by load(), and setLoadProxy().

const uint NLMISC::CI18N::_NbLanguages = sizeof(CI18N::_LanguageCodes) / sizeof(std::string) [static, private]

Definition at line 52 of file i18n.cpp. Referenced by load().

const ucstring NLMISC::CI18N::_NotTranslatedValue [static, private]

sint32 NLMISC::CI18N::_SelectedLanguage = -1 [static, private]

Definition at line 57 of file i18n.cpp. Referenced by get(), and load().

CI18N::StrMapContainer NLMISC::CI18N::_StrMap [static, private]

Definition at line 53 of file i18n.cpp. Referenced by get(), and load().

bool NLMISC::CI18N::_StrMapLoaded = false [static, private]

Definition at line 54 of file i18n.cpp. Referenced by load().

---

The documentation for this class was generated from the following files:

• i18n.h
• i18n.cpp

---