# Is implementing your own scripting language viable?

I am coding a beat 'em up game in C++ and the time has come to implement scripting for events, triggers, cutscenes etc. I have read around on the internet and gotten a fair bit of information. My first choice solution would be to implement my own scripting language like the one in Cave Story. I have seen this suggested but most poeple suggest lua but that does not seem to fit my type of programming.

Have you made your own scripting language? Why did you chose to roll your own instead of using an existing one? What resources did you consult during the development?

• My personal rule of thumb is: if you need to ask other people whether it’s a good idea to roll your own _____, then it’s not a good idea. – sam hocevar Jun 17 '13 at 11:44
• Note that if you implement your own language, people must learn it and hope that there are no bugs in your interpreter, whereas languages like Lua are more popular and much less likely to have bugs. – Nick Caplinger Jun 17 '13 at 13:17
• @NickCaplinger hits it on the head. Rolling your own means there's no documentation, no community, no third-party tutorials, no StackExchange, etc. Whatever minor advances you get in syntax or integration will be outweighed by the lack of support unless you're a really well-known developer with millions of fans. – Sean Middleditch Jun 17 '13 at 20:28
• Also keep in mind tooling. This is my problem with D, Rust, Dart, etc. Your syntax is pointless on its own. You need a proper editor, highlighting, "Intellisense" (code-completion), good diagnostics, refactoring support, inline documentation support, etc. to really be worth anything. Not to mention all the actual language features, integration, efficiency, and so on. – Sean Middleditch Jun 17 '13 at 20:30
• If you are a C++ programmer, and you dislike LUA, try JavaScript instead. – zeel Jun 18 '13 at 4:01

No. At least, probably not.

This is a very frequent case of reinventing the wheel is game development, a mistake that is still quite popular.

If you're asking this question, you're very likely to be influenced by what others do, so just look at what Epic Games just did with the Unreal Engine:

• UE3 had a custom, weird, non-optimized, hard-to-debug UnrealScript thing,
• If the rumor is true, its support is being removed in UE4, in favor of hot-reloadeable C++ DLLs.

Do you think you can do better than Epic?

Creating progamming languages belongs to programming language creators, not game engineers.

It takes years and years for a language to become fully mature, and its accompanying toolset (compiler, linker, interpreter, debugger..) usable. Nowadays you have a lot of available solutions at hand so there's absolutely no real reason to start a new thing from scratch, at least not if the goal is simply to make a game. Period.

To answer your side questions, no, for these very reasons I've never implemented my own scripting language. But I've suffered a lot with some half-baked ones. Because they were created with a very narrow feature set in mind, they always had this little insane quirks that make you crazy. Often, you'll find yourself spending an awful lot of time just trying to work around the language's limitation instead of just making your game.

If the reason you want to create a language is because it is intended for use by people that don't know programming very well, or if you believe you need it because you want something very domain-specific, let me tell you these are also bad reasons. You can write a very high-level API with functions that do_what_they_say_and_say_what_they_do(), and some very simple boilerplate code that exposes its basic usage. Your not-so-technical users will be glad to learn a bit of programming, and you will be glad not to be limited by some badly implemented language.

So, as this will sound a bit abrupt or even harsh, I'll say that there is one case where it could make sense: if you want to learn how a scripting language is made. But please, please, if you do so: don't force others to use it.

Edit

I just had a look at the Cave Story command list you've linked. Ouch:

<ECJx:y      [EC?] Jump           @ Jump to event Y if any npc with ID X is present


I don't want to show disrespect to the developer behind Cave Story, but this is a perfect example of a simple command list that mutated in a uncontrollable custom scripting language. This might be still usable for a single developer or a very small team, but at this stage I advice to switch to a proper Turing-complete and well-tried language (e.g. Lua), where you could do:

if (npc.id == x) then
jump_to_event(y)
end


This will make things so much easier when, for instance, you'll need a more complex condition:

if (npc.id == x) or (npc.type == "enemy") then
jump_to_event(y)
end

• +1 "Creating progamming languages belongs to programming language creators, not game engineers." This quote could not be more correct. Having taken a programming language concepts class taught by a guy who was a whiz with programming languages, these people are a different breed. One class made me realize the amount of CS theory and mathematics that go into creating a real programming language. This made me appreciate these people and their skillset even more. They let me create the games, I stay out of their way and let them create the languages. – Dean Knight Jun 17 '13 at 14:09
• +1, i dont know lua or the custom scripting language, but it was evident exactly what was going on in lua and thats where the usability is important – RhysW Jun 17 '13 at 16:25
• There's nothing magical about one or the other making it impossible to learn both. Unreal is maybe removing UnrealScript but they're greatly enhancing and filling out Kismet to be a capable and complete visual scripting language. They didn't remove UnrealScript because it's a failure, they're removing it because it's outmoded by significantly more difficult features (hot-reloading C++, updated Kismit). Don't take this to mean I disagree with your point: writing your own language is a huge waste of time, a source of bugs, and a cause of large learning curves, etc. – Sean Middleditch Jun 17 '13 at 17:20
• @ott-- I'm not afraid either, my point is that expanding this notation will make it more and more complicated, whereas using a proper language in the first place will help in the long run. The overhead is not relevant compared to the ease of use in that case, IMHO. – Laurent Couvidou Jun 17 '13 at 17:41
• Note that UnrealScript is replaced by Blueprints (but I have bet my colleagues that it’ll be back). Hotloading DLLs is an orthogonal feature. – sam hocevar Jun 18 '13 at 9:19

I have, although I borrowed the syntax from other languages. All in all it was a great learning experience and in my case not that hard because the language was simple.

I did it mainly because I wanted to use a regular C-style syntax for my scripts, as everyone on the team was familiar with it.

I was also interested in learning a bit about implementing programming languages, and since I only needed a very simple subset of features (variables, arithmetic, conditionals, loops, and calling in-game functions or coroutines) I decided to give it a try.

What resources did you consult during the development?

I managed to learn enough from this book to implement:

• A lexer: which was almost trivial using regular expressions, just using Regex.Match to identify and split tokens.
• A recursive descent parser: basically one function for each rule in the grammar and some helper methods to lookahead and consume tokens. I looked at the C# grammar specification for inspiration. Hardest part was dealing with arithmetic and relations operator precedences without going into infinite recursion.
• An AST interpreter: using the visitor design pattern, I traversed the tree generated from the parser,and executed each instruction node recursively.

I would have stopped there since almost everything I needed was already working, except for one thing - calling and yielding on Unity3D coroutines deep from within the recursive interpreter. In order to solve that problem, I had to get rid of the recursion, and turned once again to the book. This time I ended up adding:

• A compiler: another visitor, but instead of executing code, it generates a list of small, atomic instructions from each node of the AST (i.e. a simple stack-based custom assembly language). Operations like a while loop or an if condition, are converted into labels and goto-type instructions. At this point I also write the compiled script to disk as a binary file.
• A bytecode interpreter: simply iterates over a flat list of instructions. Without the recursion it was now easy to integrate with Unity3D coroutines.

The whole process took about 2 weeks from zero knowledge, and was a lot of fun :)

PS: At the end I also wanted to add syntax highlighting and intellisense for my custom language on our tools. Scintilla was a life saver in this regard. I used the following wrapper:

http://scintillanet.codeplex.com/

I have seen this suggested but most poeple suggest lua but that does not seem to fit my type of programming.

OK, let's try this from a different angle: what is it about Lua that you don't like? Is it something that is easily fixable, or is it something fundamental?

For example, take the use of keywords such as then/do/end to denote code block rather than good C/C++-style curly braces. If that's your problem... that's something you can fix. All you need to do is define your own little dialect of Lua, and write a simple transformation tool that will convert your curly-brace syntax into actual Lua.

Do you want += in some form? That's also easily something you can do in a pre-processing system. Just turn statements of the form expr1 += expr2 into expr1 = expr1 + expr2.

Granted, you're going to have to find a way to detect whether a pair of curly-braces represent a table or a do/end pair. And you'd have to hook your pre-processing system into Lua by overriding dofile, loadstring, and other standard Lua library functions. But it's all ultimately doable.

If small issues like that are your concern, and you're too wedded to one programming style to just change how you code (note: this is generally a terrible quality to have as a programmer), this is a far more viable alternative than simply writing your own language. This would take maybe a couple of week at most. Compare that to the years that would be spent on a proper language, with rich debugging support and the like.

If your issues are larger than this (globals being the default, thus requiring you to use local everywhere), some of these can be managed (simply make declaring new global an error, through the use of altered environments and metatables). If you hate functions as first class objects, coroutines, garbage collection, or other basic elements of Lua... well then, you're in a hell of your own making ;)

And if you really, really want to be hardcore about it, you can write your own language that you compile into Lua. So you will at least be able to leverage the very well-tested Lua runtime, the exceptional Lua API, and other basic Lua facilities, all from your !Lua language. This will take time, but it still won't be the years that you'd spend on something else.

• To me, it seems like I could just code the functions if I were to use Lua. It feels like I'm just moving code. – skiperic Jun 17 '13 at 15:49
• @skiperic: I don't know what you mean by that. Can you give an example? – Nicol Bolas Jun 17 '13 at 15:51
• See Laurent's answer above. – skiperic Jun 17 '13 at 15:54
• @skiperic: That doesn't really explain what you're concerned with. Can you code in Lua? Yes. And the problem is... ? – Nicol Bolas Jun 17 '13 at 16:01
• @BartekBanachewicz: Even accepting that it's a C API, it's still superior to many C++-based scripting API's I've seen. It's the only scripting API I would actually consider using raw, without an automated binder some kind. – Nicol Bolas Jun 17 '13 at 20:05

To complement the other answers, this is not a strictly binary option. Within an existing scripting language, you can also create your own Domain-specific_language. The advantages of this approach are:

• You don't actually have to mess with formal grammars, parsers, implementing custom editors etc.
• You're getting most of the benefit of implementing a specialized scripting language, i.e. an additional abstraction, specific to your game.
• vs. homebrew: users that are acquainted with the scripting language of your choice will immediately be able to use your DSL.
• vs. plain existing language: users unacquainted with the scripting language will just only need knowledge of the DSL for modyfying your game.

The primary disadvantage is that you would be designing the DSL with the constraints of your "base" language.

It might make sense depending on the mechanics of your game. Some games with simple enough mechanics could use an interpreted language to save themselves from some over-complicated Lua/Python coding, but the complexity savings might not be worth too much. For instance, one of those Interactive Novel games could easily use a custom-written scripting engine.

If your game involves a physics engine, various game components, and varying complexity of characters and abilities, you should definitely consider looking at other existing scripting languages, so you're not scrambling to add necessary features to your custom one, or fixing bugs with it. While Lua is most likely the fastest, there are many others that you may like more for their syntax, and many of them brag about how easily they integrate with C. Python, Ruby, and Angelscript are a few. (Feel free to mention others in the comments)

If you ensure such languages are only used for "logic control" (ie handling a specific collision case for certain object types, like the flame blaster touching an ice block) then performance will hardly ever be an issue. Of course, on the other hand, if you use them for more routine code (making a custom collision check algorithm that runs each frame) it's more likely to bog your down.

• Lua is also a very popular scripting language in game development. – Philipp Jun 17 '13 at 14:37
• Yup, Lua is used all over the place, but the OP noted that he didn't really like it much. Preferences of coding style can be important. – Katana314 Jun 17 '13 at 15:24
• "For instance, one of those Interactive Novel games could easily use a custom-written scripting engine." You obviously haven't seen Inform. Really, even for text adventures, there's no point in making up your own language. – Nicol Bolas Jun 17 '13 at 15:34
• @Katana314: "Preferences of coding style can be important." Honestly, the world would be better off if programmers got off their high horses about "coding style". We would all be better programmers if we stopped thinking that <insert preferred language here> was fundamentally better than everyone else's <insert preferred language here>. Using languages that you may not like the syntax of builds character and helps avoid this kind of bug-thought. – Nicol Bolas Jun 17 '13 at 20:08

I say, go for it. On a resume it would be an extra show of ability. Though, keep in mind, you need that ability first. It's not going to be easy, it's going to be pretty hard. There are book's on the subject, and online tutorials too, but ultimatly it's going to come down to you and your understanding of how a compiler work's, and how code is parsed and translated.

Make shure you start out simple, test frequently, and hold onto your ideal. But always remember, LUA is there for you.

• I guess the question here is if the goal is to make a game, or to make something that looks good on a resume. – Tridus Jun 18 '13 at 1:42