How to store data in a data-driven RPG [duplicate]

Question

Possible Duplicate:
Would it be better to use XML/JSON/Text or a database to store game content?

I'm working on a 2D data-driven RPG. I intend for it to be modabble; my intention is to expose all the major game data (eg. monster stats, item stats, pet stats) through some interface to allow tweaking by end-users with a tool.

First, my basic assumption is that there will be a "lot" of data (perhaps on the order of magnitude of megabytes). I have a few requirements that I'm trying to figure out.

• How to efficiently store and load the data. For example, a SQLite DB would be faster than reading text files on startup.
• How to make it version-control friendly (text-files wins here).
• How to maintain data integrity. If it's just a text-file, you can put invalid values in it.
• If possible, I would like to allow users to make their own tools to crunch my data.

Since maintaining version-control friendly data is important, I think having one or more text files with data in it will win. I can probably try to get people to use my tools to edit the data, and put lots of integrity checks in the actual code (eg. if totalHealth < 0, throw).

Are there better approaches to achieve these four goals?

I would also like input on the file format (eg. YAML vs. XML vs. JSON vs. whatever)

Answer 1

For example, a SQLite DB would be faster than reading text files on startup.

That's an interesting assumption. Opening it might be faster, depending on the size of the text file. But reading it is another matter altogether. If all your startup code does is open the file and read a couple of entries out, then it would be faster. But if you're reading the entire thing into memory, then a DB is going to be slower than the text file.

Not unless you write your own parser, or use a bad parser framework.

And of course, once it's in memory, finding the data for monster X will be faster than an SQLite DB. Assuming you use an appropriate data structure.

This is the only requirement that would suggest using an SQLite DB, so you probably shouldn't use that.

Lastly, remember that SQL databases are designed and optimized for complex searching for stuff. If all you're searching for in your DB is a monster by name or something, you're really using an over-designed tool for the job.

How to maintain data integrity. If it's just a text-file, you can put invalid values in it.

If you are going to allow your users to directly modify game data without forcing them to use a specific tool, then you're going to have to accept that users can do the wrong thing. With great power, comes great opportunity to abuse that power.

So your options are to "force" users to use a specific tool (which would require some form of encryption on your data file. It won't stop dedicated hackers, but the hoi polloi will be warded off), or to do integrity testing at load time. Or you could split the difference.

Let users edit text files however they please, but you introduce a "build mod" step between the text and your game. They have to run some command line tool that converts their text files into your game data. That's where integrity testing goes. You could also optimize your data in such cases, if text parsing is really that big of a turn-off for you. Of course, this makes it a bit more difficult to use, thus slightly slowing down turnaround time between editing the game data and seeing a change.

If possible, I would like to allow users to make their own tools to crunch my data.

That's a lot less viable for SQLite DBs. Especially if you add integrity check expressions. And it becomes much harder if those integrity expressions require functions that only your code exposes; that last part makes it impossible for someone to use their own SQLite tool without linking to a library of yours for the interface.

I would also like input on the file format (eg. YAML vs. XML vs. JSON vs. whatever)

That's dealer's choice. However, XML has one significant advantage (beyond being the most prevalent and de facto lingua franca of textual data storage): XML schemas can do integrity testing for you, for simple cases. There are even schema-guided editors that can help prevent users from doing the wrong thing at edit time.

I'd use whatever is easier for you as a developer. Pick the syntax you like best; there will be tools for reading it. Personally, I'd use Lua, as I would also be using Lua for scripting, so I would have it handy.