# Are any scripting languages designed to be reloaded at runtime?

I'm currently working on my first game engine. So far, I have a cross-platform rendering framework and support for numerous file types, but game development is still a hard slog when I'm having to wait for resources to reload and then follow the game flow to the point I need to test.

Ideally, I'd like to be able to develop the game while it's running, through a scripting language that could be reloaded at runtime without compromising the game's state.

However, my biggest problem with this plan has been with pointers-to-functions appearing in the state. Whenever a script gets reloaded, these pointers either point to an old copy of the bytecode, or worse, if the old bytecode gets garbage-collected, they end up dangling!

I've found this in two of the most common (and appealing) scripting languages - Python and Lua, and I know that Javascript is similarly capable of this same issue. All the cases I've tested have been JIT-compiled - I've thought about running interpreted-only, but I've been worried about the performance hit. I've also considered trying a stateless language, but I like object-oriented syntax, especially if my game actors in the script are going to mirror entities in the renderer.

So my question is, is there any scripting language that gives me the best of all worlds? A scripting language with OO syntax that's designed to be reloaded at runtime, or at least limits the ability for pointers-to-code to get stored in the runtime state?

Right now, my best option looks like taking a batteries-not-included scripting engine, like TinyPy, and modifying it myself!!!

EDIT:

In one last vain appeal, I'm going to try a slightly different question...

Does anyone know of any scripting languages that DON'T treat functions as first-class objects???

• "Does anyone know of any scripting languages that DON'T treat functions as first-class objects???" No. Nor should they. Functions as first-class objects is like one of the main reasons to use a scripting language. It is not something you should engineer out of a language, and I'm fairly sure Python and Lua would be non-functional (in more ways than one) without it. With a properly-contained approach to scripting, using language mechanisms to sequestor each script, it should not be an impediment to script reloading. If it is, you're using scripting wrong. – Nicol Bolas Feb 1 '13 at 18:50

Where I come from, we see the upgrade problem as subsumed by the persistence problem. In this case, that means:

Most games need saved games; you will want some way to save the entire state of the game and resume it later. Get that implemented now, and define your hot-reload as follows:

1. Save game to a temporary buffer.
2. Discard all game state and loaded scripts.
3. Load new version of scripts.
4. Load game from buffer.

In this way, you don't have to implement any additional architectural features to support the development feature of hot reload: you only need the game-save feature that you will likely be writing anyway, and your development process will thus also give a thorough workout to the save feature — including loading old saves in new “versions” of the game, something which is tricky and likely to annoy players if done wrong!

• Note that there is a caveat here: to do this, your old scripts will have to write their appropriate data in a format that the new scripts can read and understand. If your new scripts change radically, that may not be possible. Obviously for a shipping game, you don't want to wreck everyone's save games for a patch, so you would be careful with how you change your AI or you implement versioned loading. But during development, you want to be agile, not burdened with having to load outdated versions of data. So there can be a downside to this. – Nicol Bolas Jan 31 '13 at 22:53
• You're right, persistence is the real issue here, and this has highlighted one of the major problems with TinyPy - it has a complete lack of any runtime type information, making deserialization extremely difficult. Perhaps now would be the time to switch back to full Python, and use a library like Pickle to marshal my state between script refreshes, and also between game sessions. – Richard Copperwaite Feb 2 '13 at 20:09

You don't have to throw the baby out with the bathwater. Your issue with these interpreters can easily be solved: Don't directly store pointers to the functions!

Either look them up dynamically every time you invoke, or cache the pointer and invalidate all of the pointers when you reload the scripts. All you have to do is add this one level of indirection.

• I was hoping to avoid imposing coding standards on myself or dipping into the guts of the scripting engine - I hoped that there might be some magical language out there that had done the appropriate work for me, or that simply disallowed the use of function pointers as part of the language definition - but it looks like I'm just going to have to bite the bullet. – Richard Copperwaite Jan 31 '13 at 20:19
• Someone has to add a level of indirection, and doing it in the scripting engine loses performance for those who don't need it. Consider what happens to your function pointer if the new script doesn't define a function of that name! What should happen then? – ccxvii Feb 1 '13 at 1:41

I suppose you only need hot-swap when developing a game; you probably could build a system which wraps pointers-to-functions in an object that behaves in two different ways:

1. If this is a release build, store a bytecode pointer and invoke it directly. No hot-swapping is possible.
2. If this is a development build, don't store a raw bytecode pointer; instead store the name (and other details), allowing you to to find it again. Every time you need to call the stored function, find it again (to get a temporary function pointer) - then you can be sure your code is correct. The behaviour is transparent to the users, at the cost of performance.

If the performance in development builds is too low, you might consider a hybrid approach - your wrapper objects should contain both a bytecode pointer and a function name. If the script changes and is hot-swapped, all your wrapper objects should refresh their bytecode pointers. You'll need to have a list or something or all the wrapper objects (some smart pointers are implemented in a similar fashion).

• I did actually have a go at this, both wrapping raw pointers and storing the function's name, but I found that having a separate "function pointer object" still required imposing coding standards on myself that went against the intuitiveness of Python. – Richard Copperwaite Jan 31 '13 at 20:16

I agree with Kevin Ried's answer in part: if you intend to be able to save and reload your game arbitrarily, then you'll need to solve this problem in addition to many more. However, if your game simply doesn't need this (for whatever reason), then you can employ less invasive measures.

The best way to solve it for such a game is to be disciplined in what is considered actual data and what is not.

For example, let us say you're making a top-down 2D space shooter game. And you want to test the AI of an entity. So you have a script that governs the control over that entity.

Your script should contain and manage all of the data needed to control it, but your actual code should only be calling the UpdateControllerAction function, which calls the script. This function will tell the game what the entity is doing: accelerating in direction X, firing in direction Y, etc. That is, there is a very rigid interface between external code and the script. The script can go out and detect things (being targeted by unit X, was hit last frame by unit Y, etc). But everything it does happens in here.

When it comes time to reload this script, you simply take the Controller object (the AI that knows about scripts, the thing that holds the script itself). And you tell it to reload its script, which will have the script do all of its necessary preamble work and return the new main function. From that point on, the UpdateControllerAction will call the new script function.

You let garbage collection take care of the old data. Since the script was responsible for keeping track of targets and such, reloading the script clears those. So the downside is that reloading will mean that the entity will basically be starting from a neutral state.

This mechanism works because of the separation of responsibility and having a minimal interface between the script and the outside world. The C++ world only keeps track of the main script function, not a bunch of script data and such. The script world can call into the C++ world (to find targets, etc), but all of its persistent data stays within the script world. It is therefore bound to the script function; when that dies, everything goes with it. New scripts, even new instances of the same script, can't access that data.

Here's how this would work in a language like Lua. This script would represent a minimal behavior script. It's just an example, so don't take it too seriously in terms of AI logic.

--Arguments to this script are this ship's information and a list of its allies.
local shipData, alliedList = ...

local targetList = {}

--Updates and prioritizes the targetList.
local function FindTargets()
--...
end

--The AI function.
return function()
FindTargets();

if(#targetList ~= 0) then
local currTarget = targetList[1]
AttackTarget(currTarget);
else
--No target.
FlyStraight();
end
end


Each time you run this script, you pass it a list of allies and data on it's ship. And it returns the function you will call from UpdateControllerAction.

Now, you could use multiple functions instead of one. Maybe an update controller function and a message processing function for when certain events happen to the entity (it gets hit, someone targets it, allies change, etc). But the point is to keep the interface both minimal and hidden from C++.

I guess you're going to have to take the bite and design such a system yourself.

Basically that means that you have to "watch" your files and react to events like a file has been updated etc. (via ReadDirectoryChangesW in Windows).

If one of your scriptfiles gets updated, you'll need to recompile / reinterpret it and refresh your function pointer.

• Not really an answer; the question indicates the author doesn't have a problem with monitoring changes, but with "you'll need to recompile / reinterpret it and refresh your function pointer" – Liosan Jan 31 '13 at 14:49