When programming Windows games, do you use multibyte character settings or Unicode character settings? I'm under the impression that professional companies today use the Unicode setting to make future localization and translations as easy as possible. Is using wchar's a good practice for modern games?
While I'm thinking about the subject, what is the difference between multibyte character sets and Unicode sets in Windows? Is it UTF-8 vs. UTF-16?
I turn to the always-excellent Qt documentation regarding the QString class:
QString stores a string of 16-bit QChars, where each QChar corresponds one Unicode 4.0 character. (Unicode characters with code values above 65535 are stored using surrogate pairs, i.e., two consecutive QChars.)
In addition to QString, Qt also provides the QByteArray class to store raw bytes and traditional 8-bit '\0'-terminated strings. For most purposes, QString is the class you want to use. It is used throughout the Qt API, and the Unicode support ensures that your applications will be easy to translate if you want to expand your application's market at some point. The two main cases where QByteArray is appropriate are when you need to store raw binary data, and when memory conservation is critical (e.g., with Qt for Embedded Linux).
And this how I use the class in my games: QString for everything, unless I absolutely need the performance. It's very nice, because you don't have to worry about encodings at all. QString also provides methods to convert its internal representation to ASCII, UTF-8, Latin-1, and UCS-4.
But if you don't want to use Qt (understandable), my advice would be to do everything in UTF-8. It's the most flexible codec and it provides backwards-compatibility to ASCII. It is, however, a bit of a hassle to work with.
Further reading from the trenches: Wolfire talks about supporting Unicode in file paths They decided to go for UTF-32, because that's the common ground between their platforms.
I would avoid the use of wchar and tchar as they are Windows only, while you did specify Windows in your post I think it's always good to have portability.
The short answer is use UTF-8 everywhere you can.
If you really only want to deal with windows then use the Microsoft only characters.
UTF-16 is used in Windows, Java, Qt, and a few other places. But it's not because it's the best choice, it's actually a historical artifact. Originally there was a thing called UCS-2 which was 16-bit characters but a codeunit and a codepoint where the same thing. It only covered the Basic Multilingual Plane which was though to be enough back then and Windows and Java both jumped on board.
But since then we have needed to go beyond the BMP (for example China has thousands of characters, also they have come up with their own encoding system that is government mandated just to mess with things more GB 18030 but that's another issue).
In order to give backwards compatibility UCS-2 become UTF-16 which now meant that a codepoint could be represented by multiple codeunits (the 16 bit blocks).
What this means is that UTF-16 and UTF-8 must have every character analysed to check to see if it's 1 single code point or multiple (if you want to truncate the length of a name for example you can't just do it on bytes or you may get half a UTF codepoint and end up with gibberish). UTF-32 doesn't have that issue so some people incorrectly recommend just using UTF32 everywhere for 'efficacy', but they are wrong as every UTF encoding things must still be analysed as you now have glyphs that can be made up of multiple codepoints (for example adding a accent to a letter).
UTF-16 and UTF-32 both use more memory (UTF16 can save some if your encoding something like Chinese), now days it generally isn't such an issue but there's no sense in doubling the amount of memory you use. With video games though it's probably much more of an issue. Also with UTF-16/32 endianness is an issue.
UTF-8 is normally much smaller and much more widely supported. You don't need to use any special char's, but you do need to use special library functions when manipulating text (are you getting the size of the string in bytes, the number of codepoints or the number of visible glyphs?).
With video games, you normally control the text rendering. So using whatever you want internally should be fine, you only have to deal with conversions for external API's.
With Windows, Microsoft have their own non-portable functions that accept tchar/wchars so if your using them you will need to convert to UTF-16. For games, this is probably only an issue with things like setting the window title, message boxes and filenames (which have other issues such as slashdirection). I'm not sure how Direct X does text rendering though.
Generally, yes you should
#include <tchar.h>
and use wide-character strings where possible.
The TEXT() macro will prepend an L to a string if _UNICODE is defined in your compiler settings (which it always should be!)
TCHAR* str = TEXT( "some string" ) ;
Becomes
WCHAR* str = L"some string" ; // _UNICODE defined
Or
char* str = "some string" ; // _UNICODE NOT defined
I know it's annoying, but I have this function
wchar_t* getUnicode( const char* ascii )
{
int len = strlen( ascii ) ;
WCHAR * wstr = new WCHAR[ len+1 ] ;
MultiByteToWideChar( CP_ACP, 0, ascii, len, wstr, len ) ;
wstr[ len ] = 0 ; // add null terminator
return wstr ;
}
to convert an ASCII character string to WCHAR where required.
I know it's annoying, but you should probably get used to it and stick to TCHAR and WCHAR wherever possible.
