Is video compositing feasible?

Question

I've been considering the idea of a game client that acted as a video compositor, only drawing, by itself, the terrain, the characters... deferring other graphic elements to separate processes. For instance, a health bar would be drawn by a separate process with its own address space and graphics context. Even if the mentioned bar could be drawn from another thread of the main process, or perhaps even from the renderer thread, this model would increase security (I believe it is already done in Chromium), ease development and the introduction of trivial and more complex addons, as well as make the whole game client, with all of its brethren, more balanced across threads/cores.

Please, note that the heavy rendering (i.e. terrain, characters,...) would be drawn from a thread of the main process on the entire window, so it wouldn't have to be composited. The compositor thread would then draw the contents of clients' graphics buffers on top of what the renderer thread drew, in their respective position.

Also, note that this is only a personal project, and that I have never developed a game. My target is Linux with Mesa. As for the IPC part, I had thought about Wayland.

---

Question(s)

• Would compositing, and having multiple graphics contexts, degrade performance seriously?
• Is this already done in any game client?

P.S.: I think World of Warcraft also uses addons, although I think they're in Lua and parsed from the main thread, so no compositing happens there.

Answer 1

A context in OpenGL represents an individual drawing surfaces, such as a viewport in an application. It is impossible to use multiple context to draw to the same surface. If you would like separate system, such as a terrain renderer and a GUI to draw overlapping each other you will need to use the same context.

Separating work into multiple threads has no benefit from a security context, and to be honest the only security context in games is related to online play which is often solved by letting the server make authoritative decisions. You can separate game code into threads, but this is a very complex process which needs to be synchronized each frame (as the GPU can only handle on request at a time). A novice or even moderately experienced game engine programmer will probably end up with a slower game engine and not a faster game engine when using multi-threading so I would advice against it. However, for a good reference see this article

What you might be looking for, and what you should do, is separating functionality and data from other functionality and data. In games we often use the entity-component paradigm for this. See this list of articles and the answer to this question for more information. (It helped me a lot).

Note that if you're not too experienced in game development I advice to use a very simple forward-rendering technique where you just let each system do its job in turn. Be pragmatic!

Answer 2

Games using Chromium Embedded Framework or Awesomium do something like these. The Chromium code runs all "renderers" in sandboxed separate processes and then composites the final view in the "host" process.

This isn't quite down to the level of having each element in a separate thread, but it does illustrate that compositing from separate threads can be done in real-time for games. One render thread vs many render threads shouldn't adversely affect performance much.

That said, I feel this is a fool's errand outside of the use case of using an existing tool like Chromium. Developing this kind of IPC system, sandboxing it properly, and making it all run efficient is hard. It also have very questionable utility. I contest that it'll make development easier; what's far more likely is that adding trivial changes is going to constantly require lots of reworking to the IPC layer, bug fixes, and complicated architectural changes.

There's also not likely going to be a lot gained from the "better parallelism." There's a point at which parallelism actually helps. The rendering of a health bar is not it. You want to render as much of your UI in a single call to the GPU as possible and parallelize the GPU hardware. Splitting up simple things into processes adds a ton of overhead at the CPU level and then negates your ability to parallelize the GPU well.

Modularizing your code and making good use of parallelism on the CPU where it's beneficial is achieved in far better ways than what you're proposing.