Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

How minimal should a games engine be? How much of the content pipeline should be embedded in the engine?

Some use cases where the super engine might be useful:

  • When loading user content, the user isn't required to package up his textures, the engine will do it at load time.

  • A script requests a font at a much larger size than was pre-generated, the engine could parse the ttf file ad build a new texture atlas.

  • Halo forge.

None of this is free of course. This requires your content pipeline tools to written in C++. Support libraries you use in the pipeline need to be compiled for use on the device. It requires the content generation to be not buggy. And it generally makes your engine larger and unwieldy.
What are some other pros and cons?
Do the pros outweigh the cons?

Some specific questions:

  • Should engine be capable of loading various image formats? A TGA only loader is pretty easy to hand code.

  • What about audio formats? Is it feasible to only support loading wav files? What about ambient music files which are often huge.

  • Should the engine be capable of dynamic TTF parsing and atlas generation?

  • Texture packing.

share|improve this question
add comment

2 Answers

up vote 9 down vote accepted

Noel Llopis's Games from Within blog touched on this recently in the "Remote Game Editing" post. The opening paragraph:

I’ve long been a fan of minimal game runtimes. Anything that can be done offline or in a separate tool, should be out of the runtime. That leaves the game architecture and code very lean and simple.

(The article is a highly recommended read, as with most of Noel's stuff, whether you agree 100% or not.)

I believe the key here is to keep the complexity outside the engine. You can still have flexibility, but it's flexibility in the content pipeline. And you get better performance by not spending time converting and moving data around.

Better performance may translate into lower iteration time, strangely, despite losing some of the in-engine editing abilities: it's easier to try something if you can load the game in a second.

Adopting some of the tenets of the "unix philosophy" will help you keep your toolchain flexible: a small modular pipeline.

My personal philosophy: bake as much of the data as much as possible offline, but provide engine support to receive new baked data at any time. (Note that this new data doesn't need to come into play until a convenient point: the "refresh" button is pressed, the next level begins, you transition to a new area, whatever. The key is finding the sweet spot that minimizes iteration time with minimum code complexity and coding effort.)

At our company most of our artist/designer-facing tools are focused on UI issues: ease of manipulation of single assets or batches thereof, etc. Sometimes they're just 3rd party tools like Photoshop or 3DS Max. These tools export to an intermediate format (often xml that references source binary data, but not always). The intermediate format is picked up by a backend "data make" tool, which bakes it into something useful and quick-loading for the target platform.

Portability is achieved by adding additional backend data make tools, or expanding the existing backend data make tools, which has the additional advantage of being invisible to the content creators.

Now, with a proper incremental data make, you can have changes in baked format within seconds; your engine can spider, or a tool can spider, and then these will appear in your resource system, ready for reload when it's convenient.

The tools -- especially the backend data make tools -- are often sloppier and buggier than the engine code. This is OK, because they're easier to refactor/rewrite, extend, and test; you have specs for their behavior and it's fairly easy to unit test them as compared to some engine code.

My opinions on your questions:

Should engine be capable of loading various image formats? A TGA only loader is pretty easy to hand code.

(Aside: even if you use a TGA decoder in-engine, don't handcode it. You're just asking for trouble -- there are a lot of subtleties with most image formats, and a lot of tools that don't adhere exactly to the probably-underspecified format. You're best off finding existing well-tested library code for image processing.)

I'd have the tool here convert from TGA to whatever your internal texture format is, plus metadata.

What about audio formats? Is it feasible to only support loading wav files? What about ambient music files which are often huge.

We use three formats here: tracked music (.xm), ADPCM (.wav), and Speex (.spx). This is mostly because we're on handhelds, and these formats are very lightweight to decode.

Should the engine be capable of dynamic TTF parsing and atlas generation? Texture packing.

Atlasing is a hard problem: see your recent question's answers. It's almost always worth doing offline.

Plus you can make the per-character metadata into a near-zero-load-code baked struct.

In closing, you can clean up and package this pipeline with your game, for the mod community. You can always add more source formats. And there's nothing preventing you from bridging the gap between the content creation tools and the engine in specific cases; hopefully your data baking code and spider/transfer code can be refactored into libraries that might eventually be used directly by the content creator tools in some cases. But I wouldn't make that my first goal, necessarily... Just be aware that it will be an eventual goal and let that influence your design a bit, and go for the low hanging fruit first.


As an update, you may want to consider using the KTX File Format for textures. It has the advantage of being mostly "read into struct and go" for most GL usecases (and from your comments it sounds like you were targeting GL) while still being flexible and well-defined.

The KTX header overhead may be a bit high for fully-baked data, depending on your target, and you may want to forgo endian swap support, depending on your usecase... but it's definitely at least worth a gander for design considerations.

share|improve this answer
    
Great stuff thanks. I had never considered building my own easy to load image format. Is there an already built minimal texture loading library. Then again it's basically a stream of bytes with a width, height, and encoding(ie GL_RGB555). –  deft_code Aug 21 '10 at 18:23
    
Don't so much build your own image format as get the image into the exact format that DirectX, GL, or whatever you're using wants. =) As far as libraries go: there are a ton out there. For tools, I tend to just use the ImageMagick ( imagemagick.org/script/index.php ) library stuff or even the programs... The ImageMagick code is old-school and a bit ugly, but pretty fast, flexible, and road-tested. I'm sure others will have lots of other suggestions; if you're using e.g. C# for your toolchain, a lot of this stuff will already be built into .NET libs... –  leander Aug 21 '10 at 18:40
1  
"I'm sure others will have lots of other suggestions" MFC! :) Or perhaps OpenIL. I think one of the important points about baking assets into platform-specific formats is that as you add more source formats and more platforms the number of combinations will explode. Converting the source assets into an intermediate format and then converting that into platform-specific formats will cut down on the number of conversion routes. Add another source format, just write a converter to the intermediate format. Add another platform, add a baker to that platforms target format. –  Chris Howe Aug 21 '10 at 19:31
1  
I would add that keeping the complexity outside of the engine doesn't necessarily mean that functionality isn't available to the engine. Keeping it separate though so it's easily divorced from the engine is key. I can't stress enough how useful it is to support hot loading of assets, i.e. reloading things on the fly. This too can bring a lot of complexity into your system though so you'll want to make sure it's built in such a way that the game doesn't need to care about where the assets are coming from. –  dash-tom-bang Aug 21 '10 at 21:48
add comment

Your questions sound very subjective and depend heavily on who your target audience is.

Let's take your font/ttf parsing question. If you're planning on modders adding their own font, then you probably want to make the import process as few steps as possible. If you're adding a lot of fonts/font sizes to the game internally, maybe you want a somewhat sophisticated tool that an artist can use with a little training. If you're going to be doing it once internally then the time you spend making a proper importer is probably going to be less than taking the time to write tools for the previous cases.

It's really a matter of scale with everything else. Embedded tools are worth the effort when you're doing something a lot of times and want to reduce the complexity/bugs that result from it. Every additional qualifier you add (i.e. TGA textures only instead of importing of, say, PSDs) results in more time spent by the end user.

Remember that content tools are usually used by the less technically inclined (read: artists). Personally, I really like the way Unity works where you can just drag in a source file (psd, 3ds, ttf, mp3, jpg, mov, whatever) and it will automatically convert it to its internal format. The internal format is mostly hidden from the end user. It'll also auto-re-convert when it detects the source file changing. But that's a lot of work.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.