How to store data in a data-driven RPG [duplicate]

Possible Duplicate:
Would it be better to use XML/JSON/Text or a database to store game content?

I'm working on a 2D data-driven RPG. I intend for it to be modabble; my intention is to expose all the major game data (eg. monster stats, item stats, pet stats) through some interface to allow tweaking by end-users with a tool.

First, my basic assumption is that there will be a "lot" of data (perhaps on the order of magnitude of megabytes). I have a few requirements that I'm trying to figure out.

• How to efficiently store and load the data. For example, a SQLite DB would be faster than reading text files on startup.
• How to make it version-control friendly (text-files wins here).
• How to maintain data integrity. If it's just a text-file, you can put invalid values in it.
• If possible, I would like to allow users to make their own tools to crunch my data.

Since maintaining version-control friendly data is important, I think having one or more text files with data in it will win. I can probably try to get people to use my tools to edit the data, and put lots of integrity checks in the actual code (eg. if totalHealth < 0, throw).

Are there better approaches to achieve these four goals?

I would also like input on the file format (eg. YAML vs. XML vs. JSON vs. whatever)

• Have you conciderd using XML? i havent personaly worked with it, but from what i read, it could be a realy neat solution to this. Jan 26 '13 at 13:01
• @Tordin I find XML very bloated. I personally try to avoid it if other options exist. Jan 26 '13 at 13:03
• @ashes999 How do you define bloated? only requiring the header <?xml version="1.0" encoding="UTF-8" ?> the rest is decided by the creator doesnt seem bloated to me Jan 26 '13 at 14:32
• @ashes999 OK I see your point on size, although I feel the benefit of having 100's of tools that can be used, advantages in validation and as a lot of languages have support for XML can out wiegh the cost. plus if you are truly concerned with it you could remove all white space and change the XML tags to letters, just as you would with JSON Jan 26 '13 at 15:14
• Jan 26 '13 at 17:11

For example, a SQLite DB would be faster than reading text files on startup.

That's an interesting assumption. Opening it might be faster, depending on the size of the text file. But reading it is another matter altogether. If all your startup code does is open the file and read a couple of entries out, then it would be faster. But if you're reading the entire thing into memory, then a DB is going to be slower than the text file.

Not unless you write your own parser, or use a bad parser framework.

And of course, once it's in memory, finding the data for monster X will be faster than an SQLite DB. Assuming you use an appropriate data structure.

This is the only requirement that would suggest using an SQLite DB, so you probably shouldn't use that.

Lastly, remember that SQL databases are designed and optimized for complex searching for stuff. If all you're searching for in your DB is a monster by name or something, you're really using an over-designed tool for the job.

How to maintain data integrity. If it's just a text-file, you can put invalid values in it.

If you are going to allow your users to directly modify game data without forcing them to use a specific tool, then you're going to have to accept that users can do the wrong thing. With great power, comes great opportunity to abuse that power.

So your options are to "force" users to use a specific tool (which would require some form of encryption on your data file. It won't stop dedicated hackers, but the hoi polloi will be warded off), or to do integrity testing at load time. Or you could split the difference.

Let users edit text files however they please, but you introduce a "build mod" step between the text and your game. They have to run some command line tool that converts their text files into your game data. That's where integrity testing goes. You could also optimize your data in such cases, if text parsing is really that big of a turn-off for you. Of course, this makes it a bit more difficult to use, thus slightly slowing down turnaround time between editing the game data and seeing a change.

If possible, I would like to allow users to make their own tools to crunch my data.

That's a lot less viable for SQLite DBs. Especially if you add integrity check expressions. And it becomes much harder if those integrity expressions require functions that only your code exposes; that last part makes it impossible for someone to use their own SQLite tool without linking to a library of yours for the interface.

I would also like input on the file format (eg. YAML vs. XML vs. JSON vs. whatever)

That's dealer's choice. However, XML has one significant advantage (beyond being the most prevalent and de facto lingua franca of textual data storage): XML schemas can do integrity testing for you, for simple cases. There are even schema-guided editors that can help prevent users from doing the wrong thing at edit time.

I'd use whatever is easier for you as a developer. Pick the syntax you like best; there will be tools for reading it. Personally, I'd use Lua, as I would also be using Lua for scripting, so I would have it handy.

• This is a good answer, although I'm not sure it answers my question. It seems like you're saying "keep everything in text files and create useful tools, XML or whatever format is up to you, although XML has integrity checking." Is that a valid summary? Jan 26 '13 at 13:48
• @ashes999, Doesn't writing everything in txt, or xml makes the data very big? Jan 26 '13 at 14:23
• @Sri: Define "very big"? Compared to what? And how important is data size in the grand scheme of things? Jan 26 '13 at 14:32
• The size of this sort of data is going to be tiny compared to the graphical and audio assets. There's no need to be fighting to save a handful of kilobytes. Jan 26 '13 at 15:09
• @Sri: "The thing very big is the data's size on disk." Compression files can make that effectively irrelevant, in cases where the size even matters. Yes, the data will be bigger than some raw binary format. But ease of use is far more important in the long-run. Also, an int in ASCII could take up 1 byte or 8, depending on the integer. Maintenance for a text format is a lot less complex than a binary one. Jan 26 '13 at 15:22