Best practice to implement persistent data when using Tiled TMX map files?

Question

A friend of mine and I are always talking about our current progress in learning game development. Recently, saving the game (data persistence) came up, and when I summarized how I tackled that for my project, he thought I should post a question here and see if anyone else had input on the scenario.

A little bit of background: I'm working on a top down 2d rpg and experimenting with different gameplay concepts here and there. I am using cocos2d for Python, and may eventually move to Kivy. Most importantly for the save portion, I'm using Tiled TMX files for my maps / levels, and sqlalchemy.

So, I wanted all the different objects in each map to maintain their states and positions when the player leaves the map, and when the players come back, everything is at or near where he left off...unless a great deal of time has passed or something. Now, I hadn't considered at all what to do to allow this at first. ...Another friend of mine decided to just 'pickle' his map objects to a file. I tried to do this but ran into a lot of problems – I was attempting to pickle an entire sprite entity instead of just the data bits, so I ran into issues of threading... it was a mess.

Then I went with a sqlite db for storing all the various data. The reason I had this trouble at all is because a tmx map file is just xml underneath – and I wouldn't want to have to make copies of the original map files and then commit the map objects' locations back to a separate xml file – that sounded messy.

My final plan was to create a tmx map object to db map object converter. The logic is, if the player hasn't been to this map yet, initialize the objects from tmx to the db, and mirror all their properties and value pairs as well... then get the actual map from tmx xml as usual, then load map objects from the db. On map exit, commit all the map objects back to db.

TL;DR: what would have been a better way to store persistent map object data if I'm using Tiled TMX maps? Was doing a db overkill? What's an example of best practice for map save data like this, where you want the objects on the map to remain as they were when you were last on the map?

Thanks, I hope that's clear enough!

Edit:

Here's the objectgroup of a tmx file I'm using

<objectgroup name="objects" width="30" height="30">
    <object name="Entrance1" type="start" x="159" y="326" width="32" height="32">
       <properties>
           <property name="start_name" value="Entrance1"/>
       </properties>
    </object>
    <object name="To hello1" type="portal" x="160" y="256" width="32" height="64">
       <properties>
          <property name="map_name" value="hello.tmx"/>
          <property name="portal_name" value="Entrance2"/>
       </properties>
    </object>
    <object name="Item1" type="item" x="480" y="288" width="29" height="30">
        <properties>
            <property name="image" value="ironbarstack.png"/>
            <property name="item" value="ironbar"/>
            <property name="quantity" value="3"/>
            <property name="requirements" value="0"/>
        </properties>
    </object>
    <object name="Item2" type="item" x="449" y="351" width="29" height="30">
        <properties>
            <property name="image" value="rope.png"/>
            <property name="item" value="rope"/>
            <property name="quantity" value="1"/>
            <property name="requirements" value="0"/>
        </properties>
    </object>
</objectgroup>

If I'm working on a level in Tiled (adding another item on the floor for instance) and then playtesting it, I need to check for updates each time I load up the map right now. So, I check to see if that tmx map name exists in the maps table of the save game db being used. If it exists, create a list of tmx objects and db objects, and iterate through the tmx objects, update/add the db counterparts with the key being their name value. Then, at the end if there are any db objects remaining from the db object list, remove them from the db. If the map doesn't exist in the db yet, then it's a straight conversion from tmx object to db object, no checks done.

At first when I went the db route, I was storing each type of map object in its own table with properties I expected to be required for each. That proved difficult, time consuming and rather unnecessary. I ended up just doing a one to one conversion from tmx map to db map, tmx object to db object, and tmx object properties to db object properties.

After the game content is completed, I plan on removing all of the tmx files and these sync checks, and just having a master db that the game content is copied from. As a bonus question, does this make sense, or am I making a lot of work by attempting to avoid the building of a proprietary map editor by using Tiled?

Answer 1

TMX Files are basically a XML file describing your map/level; You usually read it to initialize your world and forget about it.

Now, if the lib you use to manipulate/display them (or your own code if you did it by hand) is able to regenerate the TMX from your world after the player interactions modified it I'd say the way to go is to just serialize the new TMX, either in a file or in a DB's TEXT field.

It can seem a little overkill, but it has a few advantages:

• No need to change your code if you redesign your levels
• It can handle totally or partially procedural worlds (let say a part of your TMX is a labyrinth which changes every time a player starts a new game, or some bonus/resources are randomly placed depending of things like the xp/score of the player when he enters the level, or anything you can think of, when the player continue his game after saving, everything will be in place
• Your DB schema/file format won't have to change if you add a new kind of object/actor to your game

Of course, you may also save a few extra data just to make the player game easier (total time played when the save was performed, thumbnail image of current screen, etc...).

(and you probably don't need to save the actual tileset images (unless you change them dynamically but I didn't seen any game doing this yet)).

Bottom line: It will not be the most "optimized" way, but it will be the most flexible and easy to maintain way (this means less bug, and the kind of bugs players hate the most is the one that lost their progress after saving/reloading a game).

EDIT: The Bonus question:

As a bonus question, does this make sense, or am I making a lot of work by attempting to avoid the building of a proprietary map editor by using Tiled?

Well, developing a very crude level editor from scratch would take you at least a couple of weeks, and the result would be a lot less productive than using Tiled.

If you really want to get rid of TMX files (that's something I do most of the time), you can write a script which convert a TMX file into whatever format you'll be using (in your case, the script could output a list of SQL's INSERT statements that will add the new level to your DB).

This way, you can have the better of the two worlds (Great tilemap editor to design your levels (Tiled) and the data structure you want for your game).

If you're using some kind of Makefile, you can even treat your TMX files as source files (this way, whenever you want to change a level, you just have to save the file and rebuild your game (or at least the DB in your case; you may use a specific build target for the DB so the game itself won't have to be recompiled everytime you edit a level)).

Answer 2

This is an excellant question, I am developing an app for Android using AS3, Flash Builder and Tiled for the maps. I am running into complications with the shear amount of data taken up by the tmx files. Ive cut down a bit of the memory usage by creting bitmaps of the ground tiles and any canopy tiles, instead of parsing those. However when I load the map information they are still present. I am thinking I might develop a script which would convert my collision object layer and the interaction layer into sql information.

Now as per your initial question this is how I deal with persistance between objects and players. Lets take my chest object for example, this object gets instantiated each time there is a chest present in a map. At creation the chest will take on the stats of the chest from the tmx file. When the player travels to the map and therefore the chest has been added to the stage, I have the chest search for an element in my event array and return the data, which in my case is a secondary array that defines its locked and empty status. Ie. Game_Events[mapID][chestID] = new array(isLocked, isEmpty).

When the chest gets unlocked and opened (or opened if it exists as unlocked), then I will update my sqlite db to reflect the mapID, chestID and locked and emptied status.

I find this works the best and is the easiest method.