Would it be better to use XML/JSON/Text or a database to store game content?

Question

I'm considering how to implement a component-based game, as that seems to be the hot thing and I like the idea of such a flexible design. One of the features of such a design is that adding new things to the game can be done through data, often presented as loading content through text files such as XML. This has the advantage of being human readable and easily editable in any text editor. On the downside, text can be slower to deal with, and you have to manage a large collection of data files. Similar text-based formats like JSON or config files would have similar benefits.

On the other side, there are small, portable databases like SQLite or Tokyo Cabinet. While not directly human readable, these files are easy to interface with, and I'd imagine some kind of editing tool would be preferable for game content design anyway. Using a DB allows for a consistent storage of config information and easy retrieval. You could serialize data into a DB for save games as well.

Performance wise, I think generally XML is faster for small files but a database scales better to large amounts of data. I'd imagine any real game is going to have a whole lot of game objects.

So the question: Which is the preferable approach? I'm leaning towards the DB, but I want to know if there are hidden pitfalls or real strong advantages to text files. Or if there are other alternatives besides these (serialize to binary format I guess?)

Answer 1

It might not come up so much for a small personal game, but one hard problem when it comes to game data is multi-user editing/versioning. We use a lot of small text files that get baked down to a small number of binary blobs by a build process. This makes life easier for designers since they have a lot of flexibility in their workflow. CCP, as a counter example, uses a central editing database that all designers connect to. This makes the build step unneeded (or at least a lot simpler) but it means you need to implement all the versioning and workflow features yourself, so they are bound to be simpler than other tools out there. You can deal with performance in either case, so the real question is what do you want for a designer workflow and how can you get there?

Answer 2

Im late to the party here but I spent A-LOT of time researching this.

First why I don't use the following:

XML: Excessively verbose. Tons of redundancy. Repeating field names? GROSS

JSON: I think JSON is great for a UI layout but for a database, hell no. It will have the same problems as XML, redundancy, and deep nesting. GROSS.

SQL: This is a great option, if you manage the headaches of setting it up. I've made mobile games where we store the game data online in an SQL database. The table format is nice. The issue was we had to fetch the database from online and setting that up can be a hassle. But SQL is a decent solution. Unity doesn't support this natively, and the plugins I tried had major issues (esp when trying to get it to work with version control).

Finally, here is the solution I opted to use (and I love it).

CSV: Simple. Lets me use table format, and I have one easy point of reference when I need to update it. I can have a table for all my weapons, columns for all the attributes, and rows for each type of weapon.

You can use Microsoft Excel but it spits out these stupid warnings every time you need to save. You can use macros to override it but I say save yourself the trouble and get LibreOffice. Its free, supports CSV editing, and when you open the file it loads the column width to match the name (Excel doesnt do this and it drove me nuts).

Then you just need to parse the CSV files into your game. I use Unity so all I had to do was use this nifty C# CSV parser I found:

CSV parser example

You convert the CSV files into your in game data objects and you are good to go. With Unity you can turn these into ScriptableObjects. So my workflow is: Update CSV -> Parse CSV into Scriptable Objects -> Use data from ScriptableObjects for my game

Answer 3

JSON is very lightweight and easy to understand. I think it's better suited for a game. cJSON is really nice. it will also incorporate into your source in the same manner as SQLlite does.

XML files are harder for users to edit than you may think. if you go that route you may want to make an editor for folks to use to help them avoid common pitfalls.

Answer 4

The answer to this depends on the what language you are using for the game.

If you are using C++, you would be well advised to use one of the existing XML libraries (such as TinyXml or eXpat).

On the other hand, if you were using PHP, you could very easily use a database server such as MySQL or SQLite. (Keep in mind that if you go this route, you will need more resources because the database server will be running as a separate process and larger database applications like MySQL can consume a lot of RAM when running.)

JSON is gaining a lot of momentum and is certainly the best choice if your application is written using more than one language.

In short, it depends on what is easily usable in your language.

Answer 5

If you wanted to stay in the XML realm of things you can use Binary XML to help increase load performance.

For example Fast Infoset is an implementation of binary xml

One can think of Fast Infoset as gzip for XML, though FI aims to optimize both document size and processing performance, whereas gzip optimizes only the size. While the original formatting is lost, no information is lost in the conversion from XML to FI and back to XML.

Answer 6

I've been designing databases and played around with many game ideas for years now. My personal favorite at the moment, is some kinda textbased, human-readable format for configurations, but then parsing these "slow scripts" into something more runable. Much like JAVA and .NET is compiled into Run-time bytecode.

Same idea goes here. I have a "source" version of components and then a pre-compiler/parser runs through these. If they take up too much memory, you can put them in a fast SQL-datbase of some kinda or save them as binary files. But my point is, use the memory or SQL-database as your workspace/cache so that you dont have to parse the scripts over and over again. You could make the compiler, move the parsed files into a "source-lib" and this way let the precompiler do the versioning too (keeping the previous files for rollback purpose).

If its about "unlimited harddrive space", then sign up to something like Dropbox or Amazon S3 and sync with them.

So the plan for me is currently:

1. Config/scripts/xml (some human readable format) as textfiles (just like sourcecode)
2. Parser/validator that runs through new scripts and checkes they are following rules
3. Compiler that makes binary or SQL-row out of the original files, so that these are fast to retreive + fast to run.

Regarding stability, I'm currently building the database to be "co-location hosted" and autosync between multiple locations too, so that if there is a network/hardware failure one location, the other sites should be able to handle the same traffic/gamestates.

Answer 7

If you use couchdb you will essentially be saving everything as a JSON structure. Couchdb will take care of replicating your (multiple) user's data more or less painlessly.

Answer 8

You can find a procedure in Unity Wiki with PHP, MySQL and C#/JavaScript, and it works fine for its purpose.

It consists of three steps

1. Create a blank MySQL Database and a table.
2. Create a PHP server side script (This will connect to the MySQL table, receive data from a Unity script (step 3), and query the database (the examples given are either inserting data or selecting))
3. Create the Unity Controller Script (This will connect to the PHP script created in step 2)