Binding c++ templated functions to lua

Question

I have started using Lua as a scripting language for my game engine. It works perfectly for things like input, audio etc. However, I'm now trying to bind my entity component system functions which make heavy use of templates.

For example, getting a component of an entity in c++ would look like this:

uint32 entity = 1;
Transform &transform = registry.get<Transform>(entity);

and adding a component to an entity:

registry.add<Velocity>(entity, 0.0f, 0.0f);

From what I can tell, Lua cannot handle templated functions - I would have to bind a new function for each component e.g getTransformComponent(entity). This obviously isn't great and becomes a huge pain for the end user. I've had a look around and can't find anything that is particually helpful for my problem. I see a few people suggesting things like getComponent("Transform") but I don't see how this would be implemented in c++ as my components are PODs, no base 'Component' class?

Answer 1

This is a general interop problem with C++ features. If you want to target other languages, you often have to forego many of them by the time you reach the other language calling your functions. Besides templates, another example is exceptions. You can't necessarily throw exceptions from C++ into another language to catch, so often you have to swallow (catch) any exceptions thrown and translate them into, say, error codes at your API boundaries.

There are two main strategies I would suggest for your case depending on the maturity of your engine and how difficult it is to change the central interface designs.

The first is to use a Lua binding library which covers the grunt work for you when binding template instantiations. The last one I used of this sort was many years back and there might be more popular ones now, but I used LuaBind in the past without too much pain. While Lua naturally can't generate C++ code, LuaBind will let you export a template instantiation like Registry::get<Transform> to a Lua function of your choice of name, like getTransform. It'll generate all the code to bind such a template instantiation to Lua for you automagically at compile-time. Since it's also a compile-time code generator written using C++ templates, it can understand C++ template instantiations and export them to Lua.

That still means you have to export every single template instantiation you use and would have to introduce new binding code every time you introduce a new component type, but it's the most direct way to, in essence, "export a template instantiation to Lua" while still preserving type safety (at least runtime type checking on the Lua side) and so forth. As a bonus LuaBind also lets you export function overloads. Normally Lua can't handle overloaded functions, but LuaBind lets you export multiple functions with the same name to Lua and effectively generates code which does type checking and branching for you based on the type of the parameter passed in to the Lua function. As yet another bonus, IIRC LuaBind will also generate the functions with their own catch blocks to swallow any exceptions and translate them to Lua errors with pcalls before reaching Lua.

The downside I found to this approach aside from the grunt work for binding was actually compile times. For big codebases with APIs that span several hundred header files, it wasn't unusual for me to find that the source file containing my Lua bindings took almost 15 minutes to build, and that build time is there any time you change anything in the entire SDK. On the plus side though, I was able to do things like export std::vector instantiations to Lua, like std::vector<int> as VectorI in Lua, std::vector<float> as VectorF in Lua, and so forth. It's pretty cool to be using instantiations of a standard library class template from another language without too much work.

I would have to bind a new function for each component e.g getTransformComponent(entity). This obviously isn't great and becomes a huge pain for the end user.

This is an unavoidable conceptual requirement with templates. After all, if we use std::vector<T, Alloc> for example, there are an infinite number of combinations of code it could generate based on what T and Alloc are. Same goes for your get<T> method. We can't generate an infinite amount of code to export, so you have to explicitly specify which template instantiations to generate and which to export at the very least. The above suggestion with libraries like LuaBind at least minimize the grunt work down to a minimum. You can export a finite amount of generated code to be called from a target language, but you can't export metacode to be generated against a code generator from your language to another (unless your target language has an FFI which can generate, compile, and link C++ template instantiations on the fly in a JIT-like fashion -- that actually seems conceptually feasible given how C++ JITs are cropping up these days... OMG, I just JIMPed thinking about how cool it would be).

C API

The second approach is similar to the existing suggestions. Forego the templates (static polymorphism) and type safety for your "base API". Make it more C-like with functions that accept some kind of component ID like a string or integer and make them return a uniform type regardless of the type of component we're retrieving, like an index or void pointer.

However, because this is dangerous and error-prone, you can write some wrappers on the Lua side to re-impose type safety, as well as wrappers on the C++ side to re-impose type safety (and you're free to use templates all you want for the wrappers).

This is actually the least painful way I've ever experienced after binding C++ code to multiple embedded languages (Lua, Python, and Java) using a variety of libs (SWIG, LuaBind, boost::python, JNI, Lua modules, Python modules, etc). Basically create a C API or C-like API and bind that to Lua. If you do that, you can even use LuaJIT which comes with an FFI that can import dylib-exported C functions and call them directly from Lua without any binding work whatsoever. It's also the most universally language-friendly, since many languages can interop with a C API but would choke on C++ features like function overloading, vtables, constructors, destructors, templates, exceptions, etc.

As opposed to:

But it would be painful to apply if your engine is already mature. And it's not free of its share of grunt work unless you want to work with raw C-like code in Lua and C++, e.g. You'll still most likely want to wrap the C API into idiomatic code in the target language (Lua tables and C++ classes in your case). But among all the alternative solutions I tried, the C API approach was the most painless one I've encountered so far. The point of exporting a C API is to aim for a "lowest common denominator" API that all languages of interest can understand and work with. C language features are often that "lowest common denominator".

Note that a C API doesn't mean you have to implement those API functions in C. You can use a header which only uses C language features and still use a C++ source file using all sorts of C++ language features to implement it.

Also just as a side note, I don't recommend trying to make Lua's garbage collector extend the lifetime of your components. I worked in code that tried to do that with Python's GC and it was some of the most error-prone code I ever encountered, prone to logical leaks and slicing and so forth by trying to mix and match two very different memory models and using two different garbage collectors against each other. Think of what Lua stores as a garbage-collected "handle" to your component (if you wrap it in tables as opposed to directly working with light userdata or indices) which could be invalidated before the GC collects it, not something that is actually extending the lifetime of a component, and your life should be much easier.

I see a few people suggesting things like getComponent("Transform") but I don't see how this would be implemented in c++ as my components are PODs, no base 'Component' class?

You'd still have to write some code, somewhere, that casts what getComponent returns which might be like an integer indicating a component index or void* or Component* (but still requiring a down-cast even in that case) before you can get at the actual component data.

The way I do it is that my API functions don't even work with, say, CMotion*. They work with ComponentIndex:

struct ComponentIndex
{
    int32_t type;
    int32_t index;
};

... which means I've completely lost compile-time type safety, since you can pass a component index pointing at a CSprite to a system that wants CMotion. However, it can still do runtime type-checking using that type field, and I've implemented a rich set of C++ libraries to make it easy to grab component data from these indices in a way that automates the type-checking for debug builds using a C++ template interface. When you're doing interop, it often helps to think of convenience and safety as a separate problem to be solved through wrappers instead of something to be achieved upfront, since a lot of the features relating to type safety and the most convenient and idiomatic usage are very language-specific and not understood by other languages.

The first goal is to find that lowest common denominator which all languages can understand which is usually neither safe nor convenient nor idiomatic. Then in the target language, build a set of wrappers which impose the safety and convenience and "idiomatic-ness".

Answer 2

I had this same issue when I was implementing my entity-component model.

In the end I settled on having separate methods for getting components, but using my lua bindings I access them like properties. My syntax looks like this:

local actor = linkActor() -- links script to the actor
local script = actor.controller -- get our script controller
local events = actor.components.events -- our event component, where unique events and callbacks happen
local animator = actor.components.animator -- our animation component, controls our animations

script.initialize = function()
    assert(events ~= nil)
    assert(animator ~= nil)

    -- upon being hit, this function will be called
    events:listen('on_hit', function(event)
        animator:play('hit')
    end)

end

This was the only solution I was happy with in the end - it's clear to understand and use, and that was my primary purpose.

Answer 3

Altough it is popular to access components via templates (most common implementation I stumbled across so far), only few developers consider implementing access like in the following snippet:

entitiy.component('transform');
// ... OR ...
// when you're not afraid of operator overloading.
entity['transform'];

// When you implement more magic, you can still access your 
// type via templates
entity.component<Transform>();

Using a clean approach, one would create a base class for components, note, however, that a plain void pointer should be sufficient, since we know, which objects we're trying to access.*

Following this pattern, it should be easy to bind the getter to Lua.

I'd recommend to wrap the function on the Lua side, so that casting is being handled automatically by the function and a global component registry.

• Yes, I know that this solution can lead to nasty bugs, when applied badly. But, when you know, what you're doing, it can be a great (and sometimes necessary) tool.