How can one implement hot-swappable C++ modules?

Question

Fast iteration times are key to developing games, much more so than fancy graphics and engines with truckloads of features in my opinion. No wonder many small developers choose scripting languages.

The Unity 3D way of being able to pause a game and modify assets AND code, then continue and have the changes take effect immediately is absolutely great for this. My question is, has anybody implemented a similar system on C++ game engines?

I have heard that some really super high end engines do, but I am more interested in finding out if there is a way to do it in a home-grown engine or game.

Obviously there would be tradeoffs, and I can't imagine that you can just recompile your code while the game is paused, have it reloaded and working under every circumstance or every platform.

But perhaps it's possible for AI programming and simple level logic modules. It's not as if I want to do that in any project in the short (or long term), but I was curious.

Answer 1

It's definitely possible to do, although not with your typical C++ code; you'll need to create a C-style library that you can dynamically link and reload at runtime. To make this possible, the library must contain all state within an opaque pointer which can be provided to the library upon reload.

Timothy Farrar discusses his approach:

"For development the code is compiled as a library, a small loader program makes a copy of the library, and loads the library copy to run the program. The program allocates all data at startup, and uses one single pointer to reference any data. I use nothing global except read only data. While the program is running the original library can be recompiled. Then in one key press, the program returns to the loader and returns that single data pointer. The loader makes a copy of the new library, loads the copy, and passes in the data pointer to the new code. Then the engine continues where it left off." _{(original source, now available through web archive)}

Answer 2

(You may want to know about the term "monkey patching" or "duck punching" if for nothing other than the humorous mental image.)

That aside: if your goal is decreasing iteration time for "behavior" changes, try some approaches that get you most of the way there, and combine nicely to enable more of this in the future.

(This will go out on a bit of a tangent, but I promise it'll return!)

• Start with data, and start small: reload at boundaries ("levels" or the like), then work your way up to using OS functionality to get file change notifications or simply poll regularly.
• (For bonus points and lower load times (again, decreasing iteration time) look into data baking.)
• Scripts are data, and allow you to iterate behavior. If you use a scripting language, you now have the notifications / ability to reload those scripts, interpreted or compiled. You can also hook your interpreter to an in-game console, a network socket, or the like for increased runtime flexibility.
• Code may be data too: your compiler may support overlays, shared libraries, DLLs, or the like. So you can now choose a "safe" time to unload and reload an overlay or DLL, whether manual or automatic. The other answers go into detail here. Do note that some variants of this may mess with cryptographic signature vecification, NX (no-execute) bit, or similar security mechanisms.
• Consider a deep, versioned save/load system. If you can save and restore your state robustly even in the face of code changes, you can shutdown your game and restart it with new logic at the exact same point. Easier said than done, but it is doable, and it's markedly easier and more portable than poking memory to change instructions.
• Depending on the structure and determinism of your game you may be able to do recording and playback. If that recording is just over "game commands" (think of a card game, for example), you can alter all the rendering code you want, and replay the recording to see your changes. For some games this is as "easy" as recording some starting parameters (e.g. a random seed) and then user actions. For some it is much more complicated.
• Make efforts to reduce compile time. In combination with the aforementioned save/load or record/playback systems, or even with overlays or DLLs, this may decrease your turnaround more than any single other thing.

Many of these points are beneficial even if you don't get all the way to reloading either data or code.

Supporting anecdotes:

On a large PC RTS (~120 person team, mostly C++), there was an incredibly deep state saving system, that was used for at least three purposes:

• A "shallow" save was fed not to disk but to a CRC engine to ensure that the multiplayer games remained in lock-step simulation one CRC every 10-30 frames; this ensured no-one was cheating and caught desync bugs a few frames later
• If and when a multiplayer desync bug occurred, an extra-deep save was performed every frame, and again fed to the CRC engine, but this time the CRC engine would generate many CRCs, each for smaller batches of bytes. In this manner, it could tell you exactly which portion of state had started to diverge within the last frame. We caught a nasty "default floating point mode" difference between AMD and Intel processors using this.
• A normal depth save might not save e.g. the exact frame of animation your unit was playing, but it'd get the position, health, etc of all of your units, allowing you to save and resume at any time during gameplay.

I've since used deterministic record/playback on a C++ and Lua card game for the DS. We hooked into the API we designed for the AI (on the C++ side) and recorded all of the user and AI actions. We used this functionality in game (to provide a replay for the player), but also to diagnose issues: when there was a crash or odd behavior, all we had to do was get the save file and play it back in a debug build.

I've also since used overlays more than a few times, and we combined it with our "automatically spider this directory and upload new content to the handheld" system. All we'd have to do is leave the cutscene/level/whatever and come back in, and not only the new data (sprites, level layout, etc) would load but also any new code in the overlay. Unfortunately that's getting much harder with more recent handhelds due to copy protection and anti-hacking mechanisms that treat code specially. We still do it for lua scripts though.

Last but not least: you can (and I have, in various very small specific circumstances) do a bit of duck punching by patching instruction opcodes directly. This works best if you're on a fixed platform and compiler, though, and because it's nearly unmaintainable, very bug-prone, and limited in what you can accomplish quickly, I mostly only use it to re-route code while debugging. It does teach you a hell of a lot about your instruction set architecture in a hurry, though.

Answer 3

Hot-swapping is a very, very difficult problem to solve with binary code. Even if your code is placed into separate dynamic link libraries, your code needs to ensure that there are no direct references to functions in memory (as the address of functions can change with a recompile). This essentially means not using virtual functions, and anything that uses function pointers do so via some sort of dispatch table.

Hairy, constraining stuff. It's better to use a scripting language for code that needs fast iteration.

At work, our current code base is a mix of C++ and Lua. Lua is no small part of our project, either -- it's almost a 50/50 split. We've implemented on-the-fly reloading of Lua, such that you can change a line of code, reload, and keep going. In fact, you can do this in order to fix crash bugs that occur in Lua code, without restarting the game!

Answer 4

You can do something like hot-swapping modules at runtime implementing the modules as dynamic link libraries (or shared libraries on UNIX), and using dlopen() and dlsym() to load functions dinamically from the library.

For Windows, the equivalents are LoadLibrary and GetProcAddress.

This is a C method, and has some pitfalls by using it in C++, you can read about that here.

Answer 5

If you are using Visual Studio C++ you can in fact pause and recompile your code under certain circumstances. Visual Studio supports Edit and Continue. Attach to your game in the debugger, make it stop at a breakpoint, and then modify the code after your breakpoint. If you are to save and then continue, Visual Studio will attempt to recompile the code, reinsert it into the running executable, and continue. If all goes well the change you made will live apply to the running game without having to do an entire compile-build-test cycle. However, the following will stop this from working:

1. Changing callback functions will not work properly. E&C works by adding a new chunk of code and changing the call table to point to the new block. For callbacks and anything using function pointers it will still be executing the old, unmodified calls. To fix this you can have a wrapper callback function that calls a static function
2. Modifying header files will almost never work. This is designed for modifying actual function calls.
3. Various language constructs will cause it to mysteriously fail. From my personal experience pre-declaring things like enums will often do this.

I have used Edit and Continue to dramatically improve the speed of things like UI iteration. Let's say you have a working UI built entirely in code except that you accidentally swapped the draw order of two boxes so you can't see anything. By changing 2 lines of code live you can save yourself a 20 minute compile/build/test cycle to check a trivial UI fix.

This is NOT a full solution for a production environment, and I've found the best solution there is to move as much of your logic as possible into data files and then making those data files reloadable.

Answer 6

As others have said, it's a hard problem, dynamically linking C++. But it is a solved problem - you might have heard of COM or one of the marketing names that have been applied to it over the years: ActiveX.

COM has a bit of a bad name from a developer point of view because it can be a lot of effort to implement C++ components that expose their functionality using it (although this is made easier with ATL - the ActiveX template library). From a consumer point of view it has a bad name because applications that use it, for example to embed an Excel spreadsheet in a Word document or a Visio diagram in an Excel spreadsheet, tended to crash quite a bit back in the day. And that comes down to the same issues - even with all the guidance that Microsoft offers, COM/ActiveX/OLE was/is difficult to get right.

I will emphasise that the technology of COM is not itself inherently bad. First of all, DirectX uses COM interfaces to expose it's functionality and that works well enough, as do a multitude of applications that embed Internet Explorer using it's ActiveX control. Secondly, it's one of the simplest ways to dynamically link C++ code - a COM interface is essentially just a pure virtual class. Athough it does have an IDL like CORBA, you're not forced to use it, especially if the interfaces you define are only used within your project.

If you are not writing for Windows, don't think that COM is not worth considering. Mozilla re-implemented it in their codebase (used in the Firefox browser) because they needed a way to componentize their C++ code.