Given ASCII is 8 bits (1 byte), UNICODE is 32 bits (4 bytes).

An idea came to my mind to support multiple language by isolating certain UNICODE Letters to support multiple language for my chat(game) program.

It will be composed of [4 bit (Language trigger)] + [8 bits (255 possible characters) x 16]


trig|message per character, each 8 bit is character depending on trigger, language changes

Trigger for characters: (example purpose only)

0001 Normal Latin 
0010 European Latin
0011 Historic Latin
0100 Japanese
0101 Chinese

The characters at UNICODE not present at my list will make a special character... I'm thinking a Box (which will be present at the 0001 character list).

My in-built character table will supply the character depending on the trigger. Of course the User can only use one set of trigger and not mixed character using different triggers.

I also want to use this character for my Account Server Creation and Queries. Note that other character format do not support multi-language characters.

Result Comparison: Instead of sending 512 bits (32 bit x 16). I am actual sending 132 bits (4 bit + (8x16)) only.

Is this feasible and currently in use or am I doomed to use UNICODE? I am open for better ideas than the ones above.

  • 2
    \$\begingroup\$ Why are you using 32 bits for storing UNICODE instead of just using UTF-8? \$\endgroup\$ – Sean Middleditch Apr 10 '15 at 2:08
  • \$\begingroup\$ let me edit the question to give more example on what I want to achieve. \$\endgroup\$ – Ace Caserya Apr 10 '15 at 2:27
  • \$\begingroup\$ I doubt 8 bits will be enough for languages like Chinese/Japanese, they have at least 2000 characters for basic reading, let alone other less common ones. \$\endgroup\$ – Qantas 94 Heavy Apr 10 '15 at 4:11
  • \$\begingroup\$ @AlvinCaseria: this problem is already solved and the solution is called UTF-8. Use it over the wire. Use it on disk. Use it in memory. \$\endgroup\$ – Sean Middleditch Apr 10 '15 at 4:34
  • \$\begingroup\$ My concerns about UTF-8 is stability of character stored on database, since I will use it for Account Queries. \$\endgroup\$ – Ace Caserya Apr 10 '15 at 5:15

Here are 2 common ways games save space when communicating strings over the network or serializing to disk.

  1. Zlib the text or use some other compression algorithm (even just Huffman compression). You can compress the text with a pre-shared compression header so that the header doesn't add to the size.
  2. If there are common strings you have to send or serialize, have each game client have a built in string table of these strings and have it send the index into the string table (one uint32) instead of the string itself. Side benefit is that this sets you up nicely for localization as well (having your game run in multiple languages). Some games also have a dynamic table that can be added to at runtime and have ids starting at the last static string id, for the cases where new strings can be needed that are hard to predict in advance.

Also, like other folks have suggested, use utf-8 encoded strings! (:

| improve this answer | |
  • \$\begingroup\$ The number 1 totally slipped my mind. Thank bro. \$\endgroup\$ – Ace Caserya Apr 10 '15 at 5:29
  • \$\begingroup\$ No problem! Just to be clear, #2 is a way better solution if there's a set of commonly used strings, or if strings repeat themselves a lot \$\endgroup\$ – Alan Wolfe Apr 10 '15 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.