Resource Files creation and quick parsing

Question

I'm having to come up with the format of my Resource File and I'd like to ask about the ways they're made, for example, if I'm using the code:

ifstream fileReader;
fileReader.open("Test.fbx");

if(fileReader.is_open())
{
    char buffer[256];
    while(fileReader.eof())
    {
        fileReader >> buffer;
        //Here
    }
}

When parsing .OBJ or .ASE files from 3DS Max, I could simply check for a...

if(strcmp("VERTEX:", buffer) == 0)
{
    //Whatever I want to do with vertexes
}

...because each vertex value was separated by an empty space, but when parsing a .FBX there's no spaces until the end of the line, and thus I had to:

int lineLength = strlen(buffer);
for(int i = 0; i < lineLength; i++)
{
    //buffer[i] has my current char
}

And get the values together by counting how many ',' I went through, etc. I think the 2nd way is much worse due complexity, but I don't know how it affects performance-wise.

What are the best practices for the creation of asset files? I plan on having all of them in a single file, but separated by something like "IMG:" (all image data defined by IDs) "OBJ:" (all model data) "WAV:" (all audio) etc.

Edit: I'm asking if there's any "best practices" when developing your own resource file, and it'd be binary of course, and I didn't meant to use the original resource files at all. Something like this:

Resource.dat (in ASCII)
IMG:
1 MyPicture.jpg [binary data composing only what I need from a .jpg file] ;
2 AnotherPicture.bmp [binary data composing only what I need from a .bmp file] ;
OBJ:
3 MyModel.fbx [fbx binary data] ;

etc I won't save all the file headers for example, and I probably won't have more than one format for my resources, I'll be converting them all to just the data I need into my resource, such as the imageWidth, imageHeight, and 4 values per pixel for a image resource. I have another code which parses these original files (.jpg, .fbx) and I'm going to save them into a new resource file (in a bloated .dat, organized by object index) to be used in the game engine, but I'd like to know any best practices while creating such a format so I don't code everything just to figure out after "Ops, I should've also saved [data] because I need it now"

Answer 1

'What are the best practices for the creation of asset files' is a huge, huge question, but as a fundamental, the sort of text-based parsing your question starts with is slow, unwieldy and clunky for anything more than trivial data. For example, how do you hope to include WAV data in a text file, since it is binary?

As a basic starting point, split your consideration into two parts. The first is knowing what format you need your individual assets to be in. Do your images need to be compressed textures, PNGs, BMPs? Is all of your audio going to be WAV format? Each format requires its own loader, and each format is going to be almost entirely different from every other format, with its own parser, its own versioning and its own way of working. More to the point, most pre-existing formats are massively complicated, which is why you generally re-use the loader/parsers other people have implemented. There's a reason why they are complicated, because they have to handle a lot of cases! You will burn an awful lot of time trying to re-invent those wheels, so you should ask yourself whether you really need or want to.

The second is how you manage the structure of your assets. Composite files (collections of individual assets) are a useful tool, but they need proper indexing (the ability to find individual assets given only a name or unique ID). Generally you build a composite file system to be completely ignorant about the content of the individual assets - they are stored with some metadata (filename, type, unique ID, contents hash and perhaps some additional type-specific tags), but the contents are treated as an opaque binary blob, to be loaded into memory and passed to a parser appropriate to the asset type. Nothing in the composite file format needs to know anything about how to parse the assets it contains - it's rather more like a ZIP file. In fact, ZIP files are a pretty common form of composite file, since you already have libraries to load and handle them, and pre-written tools to generate them. But have you asked yourself why you need or want composite files? Have you asked how you intend to reference and arrange those files? How would you tell your code to look for 'shark.png' when 'shark.png' is actually an asset in 'fish.pak'. Does all of the code simply assume one single composite file? What happens when that file gets too large? What happens when you get naming collisions in the file because you've got two textures with the same name? Do you support subdirectories in the file? You'll find yourself needing to re-implement your own file system unless you are careful.

As a general rule, you want to be storing your assets in as near to their final form as possible, so that loading involves nothing more than getting them off disk and into memory, and then telling the system that they are their. Parsing OBJ files at runtime is going to be terribly slow, as they are generally intended as an intermediate form for your assets. Generally you would have a tool which reads .OBJ files that your modelling tool has exported, and it spits out a binary version of the asset in just the right format for your engine to load with a minimal amount of work. Worrying about whether your string parsing code is the most efficient sort of string parsing code is very much like re-arranging deck-chairs on the Titanic - even the best text asset parsing code is going to be horribly slow if you have to do it at run-time.

Answer 2

Things like OBJ files should be used only as intermediates. Your production game should use custom binary formats for models and anims that directly represent how models are stored in your game's memory/VBOs

You should have a separate build step that converts your tool created intermediate files into your game files. It should also convert things like PNGs and such into efficient-for-GPU formats like DDS, convert all audio into a single compressed format like OGG Vorbis, and pack all your assets into bundles (Zip works ok, and you can use physicsfs to read it).

Custom text formats have little use in production. Sometimes they're useful as an intermediate format for custom data, though JSON/XML work fine (you should make a nice tool for the custom data, not expect users to edit the files directly).