# Resource Files creation and quick parsing

I'm having to come up with the format of my Resource File and I'd like to ask about the ways they're made, for example, if I'm using the code:

ifstream fileReader;

{
char buffer[256];
{
//Here
}
}


When parsing .OBJ or .ASE files from 3DS Max, I could simply check for a...

if(strcmp("VERTEX:", buffer) == 0)
{
//Whatever I want to do with vertexes
}


...because each vertex value was separated by an empty space, but when parsing a .FBX there's no spaces until the end of the line, and thus I had to:

int lineLength = strlen(buffer);
for(int i = 0; i < lineLength; i++)
{
//buffer[i] has my current char
}


And get the values together by counting how many ',' I went through, etc. I think the 2nd way is much worse due complexity, but I don't know how it affects performance-wise.

What are the best practices for the creation of asset files? I plan on having all of them in a single file, but separated by something like "IMG:" (all image data defined by IDs) "OBJ:" (all model data) "WAV:" (all audio) etc.

Edit: I'm asking if there's any "best practices" when developing your own resource file, and it'd be binary of course, and I didn't meant to use the original resource files at all. Something like this:

Resource.dat (in ASCII)
IMG:
1 MyPicture.jpg [binary data composing only what I need from a .jpg file] ;
2 AnotherPicture.bmp [binary data composing only what I need from a .bmp file] ;
OBJ:
3 MyModel.fbx [fbx binary data] ;


etc I won't save all the file headers for example, and I probably won't have more than one format for my resources, I'll be converting them all to just the data I need into my resource, such as the imageWidth, imageHeight, and 4 values per pixel for a image resource. I have another code which parses these original files (.jpg, .fbx) and I'm going to save them into a new resource file (in a bloated .dat, organized by object index) to be used in the game engine, but I'd like to know any best practices while creating such a format so I don't code everything just to figure out after "Ops, I should've also saved [data] because I need it now"

'What are the best practices for the creation of asset files' is a huge, huge question, but as a fundamental, the sort of text-based parsing your question starts with is slow, unwieldy and clunky for anything more than trivial data. For example, how do you hope to include WAV data in a text file, since it is binary?

As a basic starting point, split your consideration into two parts. The first is knowing what format you need your individual assets to be in. Do your images need to be compressed textures, PNGs, BMPs? Is all of your audio going to be WAV format? Each format requires its own loader, and each format is going to be almost entirely different from every other format, with its own parser, its own versioning and its own way of working. More to the point, most pre-existing formats are massively complicated, which is why you generally re-use the loader/parsers other people have implemented. There's a reason why they are complicated, because they have to handle a lot of cases! You will burn an awful lot of time trying to re-invent those wheels, so you should ask yourself whether you really need or want to.

As a general rule, you want to be storing your assets in as near to their final form as possible, so that loading involves nothing more than getting them off disk and into memory, and then telling the system that they are their. Parsing OBJ files at runtime is going to be terribly slow, as they are generally intended as an intermediate form for your assets. Generally you would have a tool which reads .OBJ files that your modelling tool has exported, and it spits out a binary version of the asset in just the right format for your engine to load with a minimal amount of work. Worrying about whether your string parsing code is the most efficient sort of string parsing code is very much like re-arranging deck-chairs on the Titanic - even the best text asset parsing code is going to be horribly slow if you have to do it at run-time.

• I'm sorry I haven't asked properly, but I wanted to know what sort of "best rules" there are when creating your own resource file. Also, I forgot to mention it will be in binary, I'm just using text format so it's easier to develop and figure a best practice while doing so. I'm updating the question with an example of what I'm asking. – Danicco Jan 28 '13 at 19:16
• As I say in the answer - keep your composite file system ignorant of the meaning of the files it's managing, store metadata for each asset (usually in an index at the start of the file), and be aware of the complexities of how you address assets within that file (e.g. grouping assets into subdirectories, allowing search, etc.) It's worth looking at RIFF for an example of how assets of intederminate (even unknown at the time the composite file code is written) type can be packaged into a backwards-compatible / versioned stream. – MrCranky Jan 29 '13 at 10:40

Things like OBJ files should be used only as intermediates. Your production game should use custom binary formats for models and anims that directly represent how models are stored in your game's memory/VBOs

You should have a separate build step that converts your tool created intermediate files into your game files. It should also convert things like PNGs and such into efficient-for-GPU formats like DDS, convert all audio into a single compressed format like OGG Vorbis, and pack all your assets into bundles (Zip works ok, and you can use physicsfs to read it).

Custom text formats have little use in production. Sometimes they're useful as an intermediate format for custom data, though JSON/XML work fine (you should make a nice tool for the custom data, not expect users to edit the files directly).

• That's what I meant in the question, I've updated the question for better clarification. – Danicco Jan 28 '13 at 19:24
• The answer stands. Use model formats that represent in memory layouts. Use ZIP for bundles; it gives you both compression and an index, which are key. Making your own format is a time waster; there are other things your game probably needs you to work on. :) – Sean Middleditch Jan 28 '13 at 22:14
• And just use JSON/XML for any custom text data. 0 reasons to create your own format. – Sean Middleditch Jan 28 '13 at 22:22
• Using a ZIP is a good idea, but I think it'll conflict with my question for SceneGraph Management for OpenWorld, because so far I'm planning on opening the stream but not loading the objects and only loading them when needed, this means I'd have to uncompress the data every load I call. But I'll definitely use this for data which I won't be loading often, thanks for the tip! – Danicco Jan 29 '13 at 4:03
• You can compress individual files in the archive. So open, seek to file, then read it in. – Sean Middleditch Jan 29 '13 at 17:33