How can I improve this enemy database implementation?

Question

I'm developing an RPG and I'm at the point where I need to start building an enemy database. There's a couple challenges associated with this and a few solutions I've been considering.

Here's what I need to do in my enemy database:

I have two primary enemy classes I need to represent data about:

A base enemy class that includes the follow:

Base Stats
Status Resistance Table
Elemental Resistance Table
Steal Table
Drop Table
Level
Unique ID
Base XP
AI Hook
Name
Display Name

And a derived class that adds the ability to add equipment, adding the following fields:

Main Weapon
Secondary Weapon/Equipment
Armor
Accessories

I may add additional fields or additional classes in the future if it makes sense. I've considered two possible formats to database enemies.

XML Files

I'd basically do it like this:

<?xml version="1.0" encoding="utf-8"?>
<Enemies>
  <Enemy name="Red Dragon" type="BaseEnemy" level="56" displayname="Red Dragon">
    <Stats HP="55000" MP="2500" SP="2500" Strength="212" Vitality="125" Magic="200" Spirit="162" Skill="111" Speed="109" Evasion="100" MgEvasion="100" Accuracy="100" Luck="55"/>
    <StatusResistances>
      <Resistance name="Sleep" value="100" />
      <Resistance name="Stop" value="100" />
    </StatusResistances>
    <ElementResistances>
      <Resistance name="Fire" value="75" />
    </ElementResistances>
    <LootTable>
      <Item name="Elixir" rate="0.03" count="1"/>
    </LootTable>
    <DropTable>
      <Item name="Elixir" rate="0.03" count="1"/>
    </DropTable>
    <AIScript value="BasicBehaviour.py" />
    <BaseXP value="4800"/>
  </Enemy>
  <Enemy name="Gaverick 1" type="HumanoidEnemy" level="33" displayname="Gaverick">
  <!--Same Stuff as above here-->
    <Equipment>
      <Weapon name="Dark Eclipse"/>
      <Armor name="Terra Defense"/>
      <Accessory name="Ribbon"/>
    </Equipment>
  </Enemy>
</Enemies>

Advantages:

• Easy to extend if I need to add/rearrange parameters
• easy to assign default values
• I already have an XML parser (pugixml) included for config files, tiled maps and resource description loading

Disadvantages:

• potentially slow (my database will likely hit several hundred enemies)
• can't query for arbitrary enemies, so will likely need to keep all enemies in memory.
• This would mean I need to restart the game to load changed enemy data as well

SQLite

For this, I'd basically make a table with columns representing all the data I'd need and leave the unnecessary fields empty

Advantages

• Arbitrary querying can keep unnecessary enemy data out of memory
• Feels more structured
• Smaller file size

Disadvantages

• More difficult to extend/rearrange parameter orders
• Unnecessary overhead to to unused fields
• Will need to write a database interface wrapper for sqlite

With this in mind I was curious on getting some outside experience on what other people have done. I may be thinking about this totally wrong, and if so, please suggest an alternative to the two possibilities I have here.

Additionally, any suggestions on how to improve either of these possibilities would be appreciated. Really, I just want to know if I'm on the right track.

I'm open to using any free library and am already incorporating boost

Answer 1

I see no need for the complexity of a fully relational database. Relational databases exist to make complex searching operations easy, as well as handle searching over vast datasets. If all you're doing in-game is indexing one of these every time an enemy spawns, you are using an incredibly overcomplicated tool to do so.

In short, SQLite is overkill.

At which point, the question isn't SQLite vs. XML. It's how you're storing it on disk and how you want to represent the data in-game. If the only kinds of queries you do in game is to search by name, a simple std::map would be sufficient. Indeed, you could stick them in a std::vector and sort them after loading for fast searching.

For the on-disk representation, you should be most concerned with the format that makes it easiest to edit, not to read. Reading is something you can do easily enough in code; editing you have to do once per enemy. When you're designing new enemies, you'll have to edit their data a lot.

To me, the dichotomy is between XML and Lua, which is perfectly serviceable as a data-description language. For example, the example XML file could be represented in Lua as:

return
{
    {
        name = "Red Dragon", type="BaseEnemy", level=56, displayname="Red Dragon",
        stats = {
            HP=55000, MP=2500, SP=2500, Strength=212, Vitality=125,
            Magic=200, Spirit=162, Skill=111, Speed=109,
            Evasion=100, MgEvasion=100, Accuracy=100, Luck=55
        },
        status_resistances = {
            {name="Sleep", value=100},
            {name="Stop", value=100},
        },
        element_resistances = {
            {name="Fire", value=75},
        },
        loot_table = {
            {name="Elixir", rate=0.03, count=1},
        },
        drop_table = {
            {name="Elixir", rate=0.03, count=1},
        },
        script = "BasicBehaviour.py",
        xp = 4800,
    },
    {
        name="Gaveric 1", type="HumanoidEnemy", level=33, displayname="Gaveric",
        --Same Stuff as above here
        equipment = {
            weapon = "Dark Eclipse",
            armor = "Terra Defense",
            accessory = "Ribbon",
        }
    }
}

The biggest advantage to a Lua-script based approach is that... it's Lua. While it can be pure data, it does not have to be. So if you have some repetitions of stat blocks, you can easily have the Lua script generate these data for you. All this is is a Lua script that returns a table; the table is what you read into your in-memory data structures.

The disadvantage to this approach is mainly if you are not using a tool to write these files. If you have a tool that you use to edit these enemy files, then the Lua approach is fine. But if you're hand-editing them, then that means your loading code that walks the Lua table needs to verify the input. It needs to check that the necessary fields are there, that each number value is a valid number, etc.

And while that's not exactly difficult code to write, it is exceedingly tedious. The advantage with XML is that you can validate it with a RelaxNG or WXS schema. There are even XML editors that have schema-guided editing, so that it becomes impossible to write an invalid XML file.

If you need to add a new field, just adjust your schema, and you're fine. If you change the file structure, again, adjust your schema and revalidate the files, correcting errors where they appear. You can even write an XSLT tool to automatically convert files from one format version to another.

Of course, you have to know how to write schemas, and you have to have a schema-driven XML format. Otherwise, the Lua script will be no worse off, since you have to validate the data either way. And the Lua script is arguably easier to parse through, since it allows you to query the data directly. Given a Lua table that contains an enemy definition, you can query for specific "elements" by name. With an XML parser, you have to process the elements as they come to you, which is generally a bit more complex to write.

can't query for arbitrary enemies, so will likely need to keep all enemies in memory.

... so? The data structures, taken from your example XML, would consume (with no attempts at compression:

• 14 stats * 4 bytes per = 56 bytes.
• Internal name: 32-byte string.
• Display name: 64-byte string.
• type: part of the data structure; does not need to be stored.

Each weapon can also be a 32-byte string. So a grand total in the worst case of... 248 bytes. You can store more than 3500 of those in half of a MB of RAM. I wouldn't worry about it.

This would mean I need to restart the game to load changed enemy data as well

Why? That's up to you. There's nothing that says you can't dump the enemy data and reload it in the middle of a game. If you can't do that, it would only be because you did not properly structure your code to make that possible.

Answer 2

I have two primary enemy classes I need to represent data about. A base enemy class... and a derived class...

I'm not convinced that you need two classes for this; a single one should suffice. You could just as easily model this with composition rather than inheritance; modern software design techniques are trending away from deep class hierarchies because they tend to be brittle and hard to maintain. While your proposed hierarchy is not very deep, it is unnecessary (which is a first step towards "deep"). Your derived class adds only properties which could just as easily remain on the base class but left empty.

Not only would this allow you to better iterate on and tweak the behavior and properties of your individual enemies later, it allows your code to be simpler as you deal with only a single public API and the processing thereof.

I've considered two possible formats to database enemies.

Of these two, I would choose XML. In general, you may want to look at JSON. I don't think using an actual relational database here is warranted. See this question for some discussion about why: the short version is that it's complex and less direct to access or edit, and since you need none of the advantages a relational databases offers, you are better off sticking with something similar.

Both XML and JSON have the huge advantage of being trivially human-readable and human-editable without the need for specialized tools, which can really help increase your iteration times. XML allows you to use XSLT to to bulk transformation on your data should you ever drastically change the schema or need to do some other kind of large-scale migration.

Specifically for your three "advantages" to SQLite:

• You can just as easily keep the data out of memory when its in a file on a disk (possibly depending on your XML reader), but you probably don't want to do this as the disk is "slow" and you probably don't have enough data to really cause much memory pressure.
• It's only as structured as you make it, and designing really good, normalized database schemas isn't always a trivial task. You generally only get the maximum gains from a database when you have a proper schema design. It's much harder to change the schema, also.
• You can always transition to a binary file format for shipping your game that reduces or eliminates the inherent overhead of a more verbose, text-based format like XML or JSON. It's quite easy to do.

Answer 3

Here is another way to look at it, create a data structure fill it with your monster values then save the data struct to a binary file and put that in a pak file like:

ork_footsoldier.npcdat
ork_chief.npcdat
guard.npcdat
 Into..
NPCData.pak

Then load the pak to memory, parse the npcdat files, then parse the data in them at request. Its faster than XML and SQLLite, and takes up much less space.

Answer 4

The only advantage of SQLite I can think of, is duplication reduction, i.e. if falls into "Feels more structured" advantage part of yours.

But you can query XML with XPath, if you wish. But I think for hundreds (and even thousands) storing all data in memory is not critical. With this amount of data size of either db or XML file is not critical.

I vote for XML.