# How can I make message passing between threads in a multithreaded engine less cumbersome?

The C++ engine I'm working on currently is split up into several large threads- Generation (for creating my procedural content), Gameplay (for AI, scripts, simulation), Physics, and Rendering.

The threads communicate with one another through small message objects, which pass from thread to thread. Before stepping, a thread processes all of its incoming messages- updates to transforms, adding and removing objects, etc. Sometimes one thread (Generation) will create something (Art) and pass it to another thread (Rendering) for permanent ownership.

Early in the process and I've noticed a couple things:

1. The messaging system is cumbersome. Creating a new message type means subclassing the base Message class, creating a new enum for its type, and writing logic for how the threads should interpret the new type of message. It's a speed-bump for development and is prone to typo-style errors. (Sidenote- working on this has me appreciating how great dynamic languages can be!)

Is there a better way to do this? Should I use something like boost::bind to make this automatic? I'm worried that if I do that I lose the ability to say, sort the messages based on type, or something. Not sure if that kind of management will even become necessary.

2. The first point is important because these threads do so much communicating. Creating and passing messages is a big part of making things happen. I'd like to streamline that system, but also be open to other paradigms that might be just as helpful. Are there different multithreaded designs I should think about to help make this easier?

For example, there are some resources which are infrequently written, but frequently read from multiple threads. Should I be open to the idea of having shared data, protected by mutexes, that all threads can access?

This is my first time designing something with multithreading in mind from the ground up. At this early stage I actually think it's going really well (considering) but I'm worried about scaling, and my own efficiency at implementing new stuff.

• There really isn't a single, directed question here, and as such this post isn't a good fit for the Q&A style of this site. I would recommend you break apart your post into distinct posts, one per question, and refocus the questions so they are asking about a specific problem you are actually having instead of a vague collection of tips or advice. – Josh Nov 7 '11 at 20:40
• If you want to engage in a more general conversation, I would recommend you try this post over in the forums at gamedev.net. As Josh said, since your "question" isn't a single specific question, it would be quite difficult to accomodate in the StackExchange format. – Cypher Nov 7 '11 at 20:49
• Thanks for the feedback guys! I was sort of hoping someone with more knowledge might have a single resource / experience / paradigm that might address several of my issues at once. I'm getting the feeling that one big idea could synthesize my various problems into one thing that I'm missing, and I was thinking someone with more experience than I have might recognize that... But maybe not, and anyhow- points taken! – Raptormeat Nov 7 '11 at 21:11
• I renamed your title to be more specific to message passing, since "tips" type questions imply that there isn't a specific problem to solve (and therefore these days I'd close as "not a real question"). – Tetrad Nov 7 '11 at 22:19
• Are you sure you need separate threads for physics and gameplay? Those two seem to be very intertwined. Also, it's hard to know how to offer advise without knowing how each of them communicate and with whom. – Nicol Bolas Nov 8 '11 at 0:28

To your broader problem, consider trying to find ways to reduce inter-thread communication as much as possible. It's better to avoid synchronisation issues altogether, if you can. This can be achieved by double buffering your data, introducing a single-update latency but greatly easing the task of working with shared data.

• Any recommendations for specific thread pooling libs to check out? – imre Nov 8 '11 at 13:31
• Nick- thanks so much for the response. As to your first point- I think thats a great idea and probably the direction I will move in. At the moment it's early enough that I don't yet know what would need to be double-buffered. I'll keep this in mind as that solidifies over time. To your second point- thanks for the suggestion! Yes, the benefits of thread tasks are clear. I'll read your links and think about it. Not 100% sure if it will work for me / how to make it work for me but I'll definitely give it serious thought. Thanks! – Raptormeat Nov 8 '11 at 17:32
• @imre check out the Boost library - they have futures, which are a nice/easy way of approaching these things. – Jonathan Dickinson Nov 8 '11 at 21:09

You asked about different multi-threaded designs. A friend of mine told me about this method that I thought was pretty cool.

The idea is that there would be 2 copies of every game entity (wasteful, I know). One copy would be the present copy, and the other would be the past copy. The present copy is strictly write only, and the past copy is strictly read only. When you go to update, you assign ranges of your entity list to as many threads as you see fit. Each thread has write-access to the present copies in the assigned range and every thread has read-access to the all past copies of the entities, and thus can update the assigned present copies using data from the past copies with no locking. Between each frame, the present copy becomes the past copy, however you want to handle the swapping of roles.

We've had the same problem, only with C#. After thinking long and hard about ease (or lack thereof) of creating new messages, the best we could do was to make a code-generator for them. It's a bit ugly, but usable: given only a description of message contents, it generates message class, enums, placeholder handling code etc - all this code that is almost the same each time, and really typo-prone.

I'm not entirely happy with this, but it's better than write all that code by hand.

Concerning shared data, the best answer is of course "it depends". But generally, if some data is read often, and needed by many threads, sharing it is worth it. For thread safety, your best bet is making it immutable, but if that's out of the question, mutex might do. In C# there's a ReaderWriterLockSlim class, specifically designed for such cases; I'm sure there is a C++ equivalent.

One other idea for thread communication, that probably solves your first problem, is to pass handlers instead of messages. I'm not sure how to work this out in C++, but in C# you can send a delegate object to another thread (as in, add it to some kind of message queue), and actually call this delegate from the receiving thread. This makes it possible to create "ad hoc" messages on the spot. I only toyed with this idea, never actually tried it in production, so it might turn out bad in fact.

• Thanks for all the great info! The last bit about handlers is similar to what I was mentioning about using binding or functors to pass functions. I kinda like the idea- I might try it out and see if it sucks or is awesome :D Might start with just creating a CallDelegateMessage class and dipping my toe in the water. – Raptormeat Nov 8 '11 at 17:36

I'm only in the design phase of some threaded game code, so I can only share my thoughts, not any actual experience. With that said, I'm thinking along the following lines:

• Most game data should be shared for read-only access.
• Writing data is possible using a kind of messaging.
• To avoid updating data while another thread is reading it, the game loop has two distinct phases: read and update.
• In the read phase:
• All shared data is read-only for all threads.
• Threads can calculate stuff (using thread-local storage), and produce update requests, which are basically command / message objects, placed in a queue, to be applied later.
• In the update phase:
• All shared data is write-only. Data is to be assumed in an unknown / unstable state.
• This is where update request objects are processed.

I think (though I'm not sure) in theory this should mean that during both the read and update phases, any number of threads may run concurrently with minimal syncing. In the read phase no one is writing the shared data, so no concurrency issues should arise. The update phase, well, that's trickier. Parallel updates on the same piece of data would be a problem, so some synchronization is in order here. However, I could still run an arbitrary number of update threads, as long as they are operating on different data sets.

In all, I think this approach would lend itself well to a thread pooling system. The problematic parts are:

• Syncing the update threads (make sure no multiple threads try to update the same data set).
• Making sure that in the read phase no thread can accidentally write shared data. I'm afraid there would be too much room for programming mistakes, and I'm not sure how much of them could be easily caught by debug tools.
• Writing code in a way such that you can't rely on your intermediate results being available for reading right away. That is, you can't write x += 2; if (x > 5) ... if x is shared. You need to either make a local copy of x, or produce an update request, and only do the conditional in the next run. The latter would mean a whole lot of additional thread-local state preserving boilerplate code.