# Protobuf vs Lua for custom file format

I'm defining a custom/simple file format for loading textures in my game. I narrowed my choice of file formats to two options:

I chose Lua, because my game is already using Lua, I wouldn't need to add protobufs to the build. Eventually, I'll use the engine to build a network game, at which point I'll add protobufs it's just that I don't need it yet and have other pressing needs in the engine.

However, I'm kinda stuck. Most (all?) of my support tools I write in python. I can't find a small clean Lua module for python. That means my python tools end up parsing the Lua syntax by hand. It's not too onerous as file format uses a very small subset of Lua, but still annoying.

So now I'm second guessing myself, Should I have picked protobuf instead?.
Then I promptly third guessed myself, Am I just afflicted with wanderlust and should stick with a solution that already kinda works?.

So which would you pick (or already did)?
And more importantly why?

My thoughts.

### Lua

texture
{
width=8;  --#256
height=7; --#128
format=GL_RGB;
pixels= [=[
<binary-data-here>]=];
}

• + "plain" text format
Technically only the meta data is plain text but base64 encoding just for the sake of plain text is silly.
• + parsing is trivial as the engine already supports Lua.
• -/+ easy (not trivial) to generate, and can be easily verified with luac.
• - python tools have difficulty parsing

### protobuf

message Texture {
required uint32 width = 1;
required uint32 height = 2;
enum Format {
GL_RGB = 0;
GL_RGBA = 1;
...
GL_LUMINANCE_ALPHA = N;
};
required Format format = 3;
required bytes pixels = 4;
}

• + protobufs are well tested and optimized
• + they will be needed eventually to do network communication
• - networking isn't part of the engine yet (and wont be for some time, the next few games don't need it).
• + trivial to parse
• + trivial to generate
• - complex binary format, impossible to edit, or verify by inspection.

Personally, I'd question why you're writing support tools in a different language than your game code. That seems like a dubious idea. :)

I've used both Lua and Protobufs rather extensively. I've been using Lua lately, mostly because it is really ridiculously easy to merge them into a Lua-based game. However this is just a preference. They'll both work fine for nearly all purposes - unless you have a strong reason to change, I'd just stick with whatever you have.

Two things to note:

• Gzipped Lua is surprisingly compact.
• Protocol buffers do have a text output option which is easy to edit and verify.
• Using different languages for different tasks is exactly what different languages were invented for. Python is perfect for tiny batch programs like this. – tenpn Oct 4 '10 at 7:29
• @tenpn: True, but you have to be careful with overuse. A project that uses 12 different languages for 12 different segments is going to be nearly unmaintanable for other programmers. Each language is a significant amount of extra complexity, and you need to carefully weigh whether that complexity is worth what you gain. In my experience, in games, it's rare that you want more than C++ and (choose your favorite scripting language). – ZorbaTHut Oct 6 '10 at 6:30
• that is my situation. C++ for the engine and game, and python (when it is easier) for tools. I'm not using Lua as my tools language because it's Python's standard library that makes it easy to use. I'm not embedding Python as I'm not insane. – deft_code Oct 11 '10 at 21:01

I would pick protobufs. The temptation to do clever things with Lua would be too strong. First I would cave and write basic arithmetic because I can just eval it in Python. Next I would let myself use intermediate variables because they're easy to parse out. Eventually I'd need Lua-in-Python for any tools I wrote in Python, and then I'd want to go back in time and stop myself.

Protobufs also support schemas, which can provide structural and semantic validation; luac will only provide syntactic validation.

However...

Protobufs are not needed for network communication, so don't choose them for that reason. You can gzip Lua and send that just as easily, if you have the code written to serialize/deserialize Lua objects.

(If I was doing this from scratch, I would pick neither, and use YAML. The protobuf syntax, especially the manual field numbering, annoys the crap out of me.)

• I see you argument against Lua (or any full language). However, comparing YAML to protobuf is like comparing JSON to XML:schema. Protobuf syntax is only a schema the actual data is highly efficient binary blobs. The proto files are compiled into your language of choice to provide access to the content of those binary blobs. – deft_code Oct 2 '10 at 21:26
• gzipped YAML is not very efficient binary blobs, but I bet it's more than compact enough to use for anything a game needs. – user744 Oct 2 '10 at 22:08

I choose Lua. Only having the possibility of insert some check functions in the last moment or to prototype some features is more than enough for me. However I'm assuming that performance isn't an issue because we are working in a network environment.