I'm wondering what's the approach for handling localization in a modern game engine. Typically I'd reference a string that would serve as a tag (say: AURA_POISON_CLOUD) in the code and then later in my UI I'd look it up in the language of my choice.

Considering that strapping a full-on string to my gameplay Status struct (ECS architecture) seems a bit inefficient, I could hash the string and then in my UI I would have to first get the original string and then do the localization look up. How would you handle the hashing though when it needs to be reversible? By definition it seems impossible, so an alternative would be two dictionaries: strings=>ints and ints=>strings for the reverse lookup. Is there something I'm missing here?

How do you approach this problem in your games?

  • \$\begingroup\$ I'm not sure about what you mean for "localization strings" in games; Could you expand on that? \$\endgroup\$
    – user115399
    Feb 24 '19 at 9:59
  • \$\begingroup\$ Strings that serve as identifiers for the localization. \$\endgroup\$
    – waxx
    Feb 24 '19 at 10:12

You don't need your hash to be reversible.

At runtime, all your lookups to recover an actual string should be done using the an integer ID value, so you might have a function called getLocalizedString to perform such a lookup. Given storage for all the strings for the active language that looks something like

std::unordered_map<int, std::string> localizedStrings;

that function can be simply

std::string getLocalizedString(int id) {
  return localizedStrings[id];

Naturally, you don't actually want to write code like getLocalizedString(892), you want to use a human-readable token like AURA_POISON_CLOUD in place of 892. The simple approach is just define all such tokens

constexpr int AURA_POISON_CLOUD = 892;

but that's also a maintenance headache, so a common approach is to hash the token itself into an integer value using a macro and/or constexpr function. For example:

#define LOCSTR(token) hash(#token)

constexpr int hash(const char* string) {
  return string % 0xFFFF; // replace with a real hash function though

The Fowler-Noll-Vo hash function is simple to implement as a constexpr function in C++ and works well for this purpose.

With this, you can call getLocalizedString(LOCSTR(AURA_POISON_CLOUD)) as well as using LOCSTR(AURA_POISON_CLOUD) all over your code to refer to the string. The hash value will be computed at compile-time; the runtime will still see a call to getLocalizedString(892).

Just like you don't want to write 892 all over the place in code, you don't want the translation team to see 892 anywhere. You want them to see AURA_POISON_CLOUD. The file you give them should be something that associates the original English strings with those identifiers, as a .csv or spreadsheet or text file. The latter might look like this:

UI_CHOOSE_CHAR Choose Your Character
UI_PLAYER_DIED You have died!

You give them the source file, they returning it to you having replaced the English text with whatever translations. Your game then has to load that file, and when it does, it iterates through every pair of (token, translation) and adds them to localizedStrings:

// pseudo-code
foreach(pair in the file) {
  localizedStrings.insert(hash(pair.token), pair.translation)

Having populated that database of translated text, getLocalizedString will now work fine. At no point will you need to expose the actual hashed ID to a translator or person writing code. They're all able to use the human-readable token.

(Note this approach still works if you don't have constexpr functions, you just pay the hash cost at runtime, so you might write your code a little differently, but it should generally be fine.)

(Also note that while you don't need to perform the reverse lookup ever, keeping the information for the reverse lookup around in debug builds can be helpful, but while stepping through the code in the debugger, you will see the raw hash results and having the ability to reverse the lookup in the watch window or something can be useful; you don't need this for shipping builds at all, though.)

You may also be interested in reading an approach to localization described on the Our Machinery blog.

  • \$\begingroup\$ The problem in the solution of having a 8921 => "Cloud of Poison" mapping is that it'd be difficult to then load the translations from a csv file or an xml file that a localization team would use. They definitely would like to know the original identifiers (AURA_POISON_CLOUD) as that gives them way more information than 8921. They'd also be able to add new keys themselves without having to use a random binary that would output the hashed integer for their entry or a custom editor. \$\endgroup\$
    – waxx
    Feb 24 '19 at 18:37
  • \$\begingroup\$ @waxx Yes, that's why I said you should not do that. Instead have (AURA_POISON_CLOUD, "Cloud of Poison") in that .csv file exactly as you're proposing. There's no reason to expose the actual hash value to the translation team at all. \$\endgroup\$
    – user1430
    Feb 24 '19 at 19:32
  • \$\begingroup\$ Oh sure, but then comes the original problem. In my game logic I have lots of different abilities that are able to apply different auras (data driven approach) and for logic purposes it'd be more efficient to pass the hash around to identify the exact aura type. The issue is going from that hash to the localization identifier in the UI system. \$\endgroup\$
    – waxx
    Feb 24 '19 at 19:55
  • \$\begingroup\$ I must be either misunderstanding you or not explaining myself clearly, why do you think you need the ID once you have the hash? The hash should be all you are looking up with at runtime. The IDs are only in the source or the content (like the .csv) for human convenience. The hash is used in memory for efficiency. You shouldn’t need a reverse lookup from hash to ID. Maybe I’ll expand my answer with pseudo code when I get back to a real computer; it would help if you can clarify why you think you need that reverse lookup though. \$\endgroup\$
    – user1430
    Feb 24 '19 at 20:26
  • 1
    \$\begingroup\$ Yes, exactly. I reworked the answer while you were typing that comment, does this new form make what I was saying clearer? \$\endgroup\$
    – user1430
    Feb 24 '19 at 22:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .