Is implementing your own scripting language viable?

Question

I am coding a beat 'em up game in C++ and the time has come to implement scripting for events, triggers, cutscenes etc. I have read around on the internet and gotten a fair bit of information. My first choice solution would be to implement my own scripting language like the one in Cave Story. I have seen this suggested but most poeple suggest lua but that does not seem to fit my type of programming.

Have you made your own scripting language? Why did you chose to roll your own instead of using an existing one? What resources did you consult during the development?

Answer 1

No. At least, probably not.

This is a very frequent case of reinventing the wheel is game development, a mistake that is still quite popular.

If you're asking this question, you're very likely to be influenced by what others do, so just look at what Epic Games just did with the Unreal Engine:

• UE3 had a custom, weird, non-optimized, hard-to-debug UnrealScript thing,
• If the rumor is true, its support is being removed in UE4, in favor of hot-reloadeable C++ DLLs.

Do you think you can do better than Epic?

Creating progamming languages belongs to programming language creators, not game engineers.

It takes years and years for a language to become fully mature, and its accompanying toolset (compiler, linker, interpreter, debugger..) usable. Nowadays you have a lot of available solutions at hand so there's absolutely no real reason to start a new thing from scratch, at least not if the goal is simply to make a game. Period.

To answer your side questions, no, for these very reasons I've never implemented my own scripting language. But I've suffered a lot with some half-baked ones. Because they were created with a very narrow feature set in mind, they always had this little insane quirks that make you crazy. Often, you'll find yourself spending an awful lot of time just trying to work around the language's limitation instead of just making your game.

If the reason you want to create a language is because it is intended for use by people that don't know programming very well, or if you believe you need it because you want something very domain-specific, let me tell you these are also bad reasons. You can write a very high-level API with functions that do_what_they_say_and_say_what_they_do(), and some very simple boilerplate code that exposes its basic usage. Your not-so-technical users will be glad to learn a bit of programming, and you will be glad not to be limited by some badly implemented language.

So, as this will sound a bit abrupt or even harsh, I'll say that there is one case where it could make sense: if you want to learn how a scripting language is made. But please, please, if you do so: don't force others to use it.

Edit

I just had a look at the Cave Story command list you've linked. Ouch:

<ECJx:y      [EC?] Jump           @ Jump to event Y if any npc with ID X is present

I don't want to show disrespect to the developer behind Cave Story, but this is a perfect example of a simple command list that mutated in a uncontrollable custom scripting language. This might be still usable for a single developer or a very small team, but at this stage I advice to switch to a proper Turing-complete and well-tried language (e.g. Lua), where you could do:

if (npc.id == x) then
    jump_to_event(y)
end

This will make things so much easier when, for instance, you'll need a more complex condition:

if (npc.id == x) or (npc.type == "enemy") then
    jump_to_event(y)
end

Answer 2

Have you made your own scripting language and why did you chose to roll your own instead of using an existing one?

I have, although I borrowed the syntax from other languages. All in all it was a great learning experience and in my case not that hard because the language was simple.

I did it mainly because I wanted to use a regular C-style syntax for my scripts, as everyone on the team was familiar with it.

I was also interested in learning a bit about implementing programming languages, and since I only needed a very simple subset of features (variables, arithmetic, conditionals, loops, and calling in-game functions or coroutines) I decided to give it a try.

What resources did you consult during the development?

My main resource was this book:

I managed to learn enough from this book to implement:

• A lexer: which was almost trivial using regular expressions, just using Regex.Match to identify and split tokens.
• A recursive descent parser: basically one function for each rule in the grammar and some helper methods to lookahead and consume tokens. I looked at the C# grammar specification for inspiration. Hardest part was dealing with arithmetic and relations operator precedences without going into infinite recursion.
• An AST interpreter: using the visitor design pattern, I traversed the tree generated from the parser,and executed each instruction node recursively.

I would have stopped there since almost everything I needed was already working, except for one thing - calling and yielding on Unity3D coroutines deep from within the recursive interpreter. In order to solve that problem, I had to get rid of the recursion, and turned once again to the book. This time I ended up adding:

• A compiler: another visitor, but instead of executing code, it generates a list of small, atomic instructions from each node of the AST (i.e. a simple stack-based custom assembly language). Operations like a while loop or an if condition, are converted into labels and goto-type instructions. At this point I also write the compiled script to disk as a binary file.
• A bytecode interpreter: simply iterates over a flat list of instructions. Without the recursion it was now easy to integrate with Unity3D coroutines.

The whole process took about 2 weeks from zero knowledge, and was a lot of fun :)

PS: At the end I also wanted to add syntax highlighting and intellisense for my custom language on our tools. Scintilla was a life saver in this regard. I used the following wrapper:

http://scintillanet.codeplex.com/

Answer 3

I have seen this suggested but most poeple suggest lua but that does not seem to fit my type of programming.

OK, let's try this from a different angle: what is it about Lua that you don't like? Is it something that is easily fixable, or is it something fundamental?

For example, take the use of keywords such as then/do/end to denote code block rather than good C/C++-style curly braces. If that's your problem... that's something you can fix. All you need to do is define your own little dialect of Lua, and write a simple transformation tool that will convert your curly-brace syntax into actual Lua.

Do you want += in some form? That's also easily something you can do in a pre-processing system. Just turn statements of the form expr1 += expr2 into expr1 = expr1 + expr2.

Granted, you're going to have to find a way to detect whether a pair of curly-braces represent a table or a do/end pair. And you'd have to hook your pre-processing system into Lua by overriding dofile, loadstring, and other standard Lua library functions. But it's all ultimately doable.

If small issues like that are your concern, and you're too wedded to one programming style to just change how you code (note: this is generally a terrible quality to have as a programmer), this is a far more viable alternative than simply writing your own language. This would take maybe a couple of week at most. Compare that to the years that would be spent on a proper language, with rich debugging support and the like.

If your issues are larger than this (globals being the default, thus requiring you to use local everywhere), some of these can be managed (simply make declaring new global an error, through the use of altered environments and metatables). If you hate functions as first class objects, coroutines, garbage collection, or other basic elements of Lua... well then, you're in a hell of your own making ;)

And if you really, really want to be hardcore about it, you can write your own language that you compile into Lua. So you will at least be able to leverage the very well-tested Lua runtime, the exceptional Lua API, and other basic Lua facilities, all from your !Lua language. This will take time, but it still won't be the years that you'd spend on something else.

Answer 4

To complement the other answers, this is not a strictly binary option. Within an existing scripting language, you can also create your own Domain-specific_language. The advantages of this approach are:

• You don't actually have to mess with formal grammars, parsers, implementing custom editors etc.
• You're getting most of the benefit of implementing a specialized scripting language, i.e. an additional abstraction, specific to your game.
• vs. homebrew: users that are acquainted with the scripting language of your choice will immediately be able to use your DSL.
• vs. plain existing language: users unacquainted with the scripting language will just only need knowledge of the DSL for modyfying your game.

The primary disadvantage is that you would be designing the DSL with the constraints of your "base" language.

Answer 5

It might make sense depending on the mechanics of your game. Some games with simple enough mechanics could use an interpreted language to save themselves from some over-complicated Lua/Python coding, but the complexity savings might not be worth too much. For instance, one of those Interactive Novel games could easily use a custom-written scripting engine.

If your game involves a physics engine, various game components, and varying complexity of characters and abilities, you should definitely consider looking at other existing scripting languages, so you're not scrambling to add necessary features to your custom one, or fixing bugs with it. While Lua is most likely the fastest, there are many others that you may like more for their syntax, and many of them brag about how easily they integrate with C. Python, Ruby, and Angelscript are a few. (Feel free to mention others in the comments)

If you ensure such languages are only used for "logic control" (ie handling a specific collision case for certain object types, like the flame blaster touching an ice block) then performance will hardly ever be an issue. Of course, on the other hand, if you use them for more routine code (making a custom collision check algorithm that runs each frame) it's more likely to bog your down.

Answer 6

I say, go for it. On a resume it would be an extra show of ability. Though, keep in mind, you need that ability first. It's not going to be easy, it's going to be pretty hard. There are book's on the subject, and online tutorials too, but ultimatly it's going to come down to you and your understanding of how a compiler work's, and how code is parsed and translated.

Make shure you start out simple, test frequently, and hold onto your ideal. But always remember, LUA is there for you.