Game Development Stack Exchange is a question and answer site for professional and independent game developers. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Do people write their own DB for some reason?

share|improve this question

closed as too broad by Josh Petrie Oct 27 '13 at 15:55

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

10 Answers 10

up vote 35 down vote accepted

We did, hopefully Ben Z will give a better explanation of the reasons than I can since he was actually the original author of our database. The short version is that relational DBs are not very useful for games because they cannot efficiently store heavily structured hierarchical data, which makes up the vast majority of data an MMO needs for normal operation. We choose to build a custom object database and part of a distributed transaction processing system, which is now in production and powering our two live games.

As for if you should do the same? Probably not, at least not at first. I would still advocate avoiding SQL, but there are now many more options in the "NoSQL" world that didn't exist a few years ago.

That said, most MMOs do run on SQL (relational) databases. I know Eve runs on a SQL Server installation sitting on top of a few multi-TB RAMSANs (I will probably never in my life have as much money as those things cost).

EDIT: Hopefully he won't mind me posting this here, but this is a blog entry with some links and comments about the presentation we gave at GDC'08 about our database.

share|improve this answer
+1 NoSQL FTW... – Jonathan Fischoff Aug 6 '10 at 5:48
+1 I'd imagine that most people ended up writing there own database. But it's cool that the NoSQL world is growing now. – Noctrine Aug 6 '10 at 6:58
Even if you end up heavily customizing something later on, there is probably at least one DB out there you can prototype on. – coderanger Aug 6 '10 at 16:55
coderanger has it right. I wrote a custom database for an MMO. If I were to do it over again in todays world I would probably look more closely at adapting existing technology, because as he mentions that world is more active than it was 4 years ago. Many MMOs do in fact run on SQL, some of them well and some of them not very well. Max shard concurrency (as opposed to max zone or continent concurrency) is low for those games. – Ben Zeigler Aug 6 '10 at 17:54
"because they cannot efficiently store heavily structured hierarchical data, which makes up the vast majority of data an MMO needs for normal operation" - that's even more true in applications development than it is in game programming. Unfortunately, we're stuck with what we've got, and since the performance (and availability of DBAs) for NoSQL solutions currently can't match that for RDBMS's, we've got to smush and smudge our data into a form workable for RDBMS's. – BlueRaja - Danny Pflughoeft Jun 18 '11 at 4:40

Okay this answer is more of an observation.

Full disclosure I haven't worked on MMORPG. I have worked at what was one of the top 10 most visited sites back in 2009, and I have worked at game engine company that thought they were making MMORPG tech (I don't know if shipped).

If you look at companies that have achieved massive scale (Google, Facebook, Twitter, etc., I know they are not game companies, but they are worth looking at in this space) there are two things that I think are important.

  1. They started by grabbing whatever was cheap and easy to get themselves going
  2. They ultimately had to roll a lot of their tech on their own. (And sometimes hardware)

A big mistake is to grab some expensive turnkey thing and expect it magically scale for you. It doesn't work that way. The key to scaling up is dealing with each new challenge appropriately. You need flexible easy to use solutions that you can move in and out easily.

So my advice to you would be if your starting out, just get what seems least headache to deal with.

share|improve this answer

Basically, there's a whole range of approaches, and I think they're chosen based more on the experiences of their developers as much as the properties of the database. At one end, many MMOs are using standard relational databases - eg. Dark Ages of Camelot used/uses (are they still going?) MySQL, FreeRealms use a modified Postgresql, etc.

Moving along the scale, you have Guild Wars: they use SQL Server, but they stick their data in there as a single BLOB. This means they get the benefits of their usual binary formats with some of the benefits that a database provides. Needless to say, you can probably see some disadvantages of this approach too, but for a company used to working with flat files, it's probably still a step up.

Then there are probably some places using the various formalised NoSQL approaches, although it's still early days yet. Farmville uses membase, which they made for the purpose, based on memcached apparently. This shares many features with the likes of MongoDB, CouchDB, etc. Again with these approaches you're typically rolling your own storage formats but get the benefits of distribution, caching, redundancy, etc.

Some are rolling their own NoSQL entirely: Cryptic said they did something similar but also said they weren't using it in production. (EDIT: but see comments below.)

And then you come to people who are using flat text or binary files - as I understand it the older games like Ultima Online, Everquest, etc all went down this route, and for a company with game development experience but little database experience this works well too.

Note also that you almost never get around writing your own database in some loose sense of the term - in games you'll always have bespoke data that you need to manage either in-memory or on-disk in ways that don't lend themselves to generic software. You can't do SQL queries every time you want some data so you will need to be mirroring a lot of your information in the code no matter what back end you use.

share|improve this answer
Cryptic is now using it in production with extremely high shard concurrency compared to other MMOs (except Eve, but Cryptic did it on much more modest hardware than Eve). – user744 Aug 8 '10 at 11:20
Good to know. (Although, I don't think DB access is all that relevant to concurrency. The 'SQL considered harmful' presentation demonstrated that Cryptic wanted to hit the database for things most other MMOs just keep in memory instead, so it was a self-imposed bottleneck really.) – Kylotan Aug 8 '10 at 17:56
If you keep them in memory, you need to implement a database in-memory, or you risk item duplication / loss (as an example) when trading across zones. "Keep them in memory" only works as long as all your players are within the same memory space all the time. – user744 Aug 8 '10 at 19:59
XP very much needs to be transacted. One of the specific exploits that lead to the creation of the CrypticDB in the first place (trust me, I wrote it) involved XP gain on multiple servers at once and a failed syncing that was user interruptible. Anything that is critical to the progression of a character really needs to be transacted, otherwise players will find some way to exploit. – Ben Zeigler Aug 13 '10 at 18:54
@expiredninja: that's a very open question. But by 'flatfile' it just means "arbitrary serialisation to disk". Game developers would often write out each field with fprintf, for example. – Kylotan Jun 8 '14 at 20:22

One other thing to note is that if you have a (relatively) static data set, don't store it in a database! There's no really good reason to store your spell definitions, item definitions, maps, etc. in a database. You're rarely querying them except by name. I see a lot of people jump into game development storing these things right alongside persisted player data, and it's a totally different class of things.

You need a data store, like a filesystem, for these. Outside of development, you don't need ACID, you don't need CRUD, you don't need a database for that kind of data. Inside of development, you need a database, but you also need a VCS anyway, and your choice of VCS will make that choice of database for you.

(If you're working on a game where players can create custom spells or items or quests, well, then that data is the same sort as player data, and you do need a database again.)

share|improve this answer

On Pirates of the Burning Sea, we stared off with MySQL (though honestly I would've preferred Postgres) and when it started falling down under load, we switched to Microsoft SQL Server. Honestly, though, I probably won't go that route again in the future. Most of the PotBS data is pre-serialized before it's persisted anyway, so a large portion of what's in the DB is nothing more than an overgrown key/value store. On top of that, to get better performance out of our persistence layer, and better control of how data was cached in memory, we ended up writing a caching server that sat in front of the database.

So Kylotan's very right, you're not going to get around rolling your own on some level. SQL relational database servers aren't designed for the way games use data. Instead of assuming relational DB servers were the end-all of databases, we really should have thought more about what benefits we were looking for before choosing a tool.

Next time around, I'd look more into things like BerkleyDB or some of the other NoSQL-style DBs.

share|improve this answer

MMORPGs are intensive apps and are a huge undertaking. If you are starting off with game development I strongly recommend you do something smaller.

That said you will need two squeeze top performance out of your setup. You are obviously going to need a server farm/hive. Because network communication is relatively expensive (and most MMORPGs are realtime) you might want to replicate your central database to each of your game servers (the equivalent of realtime low-latency replication in your database of choice).

If each of your servers are serving a specific segment of the game world (like WoW works); something like Distributed Partitioned Views. DPVs allow you to segment rows of your database across different servers; according to the the constraints on the columns at each server. Both MSSQL and Oracle are smart enough to only talk to the server that contains the data that falls within your WHERE or ON clause. I am not sure if any of the open source offerings have DPVs.

The net result of this all is that it doesn't matter what DB you use, just use one with a good name: MSSQL, Oracle, MySQL, PostgreSQL. Write your database in ANSI SQL and see which system works the best - that's the only way to find out.

The NoSQL route might also be a good idea because it's designed for high concurrency. Pranny's suggestion might do the trick.

share|improve this answer

I've used MongoDB (NoSQL), it's a very fast document store that can be accessed over TCP. Give it a try! It's awesome!

Writing your own DB is a daunting task. I suggest you first explore what's already out there and if you can't find any thing that matches your specific set of criteria, build your own. Oh, and don't forget to open source it. :)

share|improve this answer

I guess it depends pretty heavily on what kind of data you need stored, and under which form.

I think most people out there use "standard" SQL databases for "dynamic" data like player informations, be it SQL Server, MySQL, PostgreSQL.

However, "internal" data (state of the world, content, ....) can be stored under any form, and my guess would be that most company write at least partially custom databases systems for that part, suited for their specific needs.

share|improve this answer

Well, it actually depends on your application - that's the bottomline. I work, where we develop social RPGs and we use Google App Engine as a backend

share|improve this answer

This question is rather old but my Idea of how to handle it is as valid as it was asked as it is today.

If you're using a relational database you need to reduce the load on it so this idea should help.

Store all public info on a local DB on each client system as well as in the servers DB.

Store all tables with items on the client database. Instead of the client looking to the server when you mouse over that weapon for stats, the client just checks the local database for stats. The server has its own database for pulling stats when calculating and just forwards damage and health remaining to the clients.

Worst case with this idea is that everyone can see the stats to all weapons if they manage to get into the clients database. it doesn't matter though cause even if they change the values, the database values on the server are what does the calculating in the game.

share|improve this answer
Once you are storing all that information locally like that, it need not be in a database at all. – Josh Petrie Oct 27 '13 at 16:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.