How to handle Japanese on Windows and Linux using SDL_ttf and C++?

Question

In SDL_ttf, there are a few functions provided that handles non-ASCII string literals, such as:

• TTF_RenderUNICODE_Solid
• TTF_RenderUTF8_Solid

and etc. I'll be using only the two functions as part of my question.

Currently, I'm fussing over whether to stick to using TTF_RenderUNICODE_Solid or TTF_RenderUTF8_Solid, mainly because I wanted my program to be able to work flawlessly on both Windows and Linux. (No plans for Mac, because of lack of hardware).

I know for a fact, on Windows, they used UTF-16 for handling wide characters. It's pretty straightforward to use Japanese for SDL, by using TTF_RenderUNICODE_Solid and the use the following code, that I'm currently using, to convert from wchar_t string literals to Uint16* type that the SDL_ttf uses:

static inline Uint16* Convert(wchar_t LString[]) {
    return reinterpret_cast<Uint16*>(&(LString[0]));
}

Thus, I can easily do:

SDL_Surface* textSurface = TTF_RenderUNICODE_Solid(font, Convert(L"い"), this->color);

But, given that any string literals, L"" is a wchar_t[] type, it makes things more complicated for me if I were to port this over to Linux. The only way I know of compiling code for Linux is to use GCC, and GCC clearly defines wchar_t to be of 32 bits long.

Of course, I can just straight up use the UTF8 versions:

• TTF_RenderUTF8_Solid

And just use string literals, such as u8"あ", or u8"いいえ", and they will all work in most cases. The problem here is, when I store these string literals and then inspect the variables in the debugger, I would not get the values I wanted. The stored data would suffer from integer overflow, like so:

Where at the top, I used u8"あ" and for bottom, I used L"あ". You can see how the value is no longer preserved when using UTF8 strings, but the value is preserved when using wchar_t type strings.

Thus, I really can't use TTF_RenderUTF8_Solid in SDL_ttf to help me with Japanese encoding.

May I ask what is the best way to go about this? And how I should correctly handle Japanese on both Windows and Linux platforms?

Thanks in advance.

Answer 1

I'm an idiot.

In C++, the compiler knows what a u8"" string literal is, and will convert that string literal appropriately to an UTF-8 compatible string.

And since it's UTF-8, it means each character is separated into 8-bits long bytes. With this in mind, you can just point a char* at the string literal, and it will work as if it's an ASCII char* string.

The Visual Studio's Immediate Window only displays ASCII characters, and will convert some parts of UTF-8 compatible characters into multiple bytes, then display each byte as if they are ASCII characters. This will create an allusion that the values are getting integer overflowed. I was merely tricked by this until now.

Thus, the best way to render text is to use the provided UTF-8 methods:

SDL_Surface* textSurface = TTF_RenderUTF8_Solid(font, u8"い", this->color);

And then it should work from there. No need to do any conversion whatsoever.