# Decoupling rendering pipeline (for UI responsiveness): Multithreading and multiple contexts?

Warning! Wall of text (see <TL;DR> paragraphs below for short version)

I have been noticing something in quite many games (most recently in cutting-edge RTS games such as Uber Entertainment's Planetary Annihilation, which is amazing by the way) that I think has room for improvement.

That is, the rendering system is synchronous in the sense that input event loops are apparently "bound" to the rendering loop. So the consequences are that e.g. mouse movements and keyboard events are usually queued up and processed only once a frame is finished rendering.

This may not be a completely accurate account of what's actually happening. What is certainly clear is that for example the mouse cursor movement updates no faster than the graphics in the game render, and obviously when keyboard events are fired, and their consequences are not visible in the game until the next frame renders.

The delay of that frame, if the framerate is very low, will very much be felt, and I want to emphasize that the same low frequency of response applied to input is a much bigger problem than if we are only talking about its impact on animation smoothness.

Of course Planetary Annihilation is a game that is still in beta, and is (as we are told) still rather unoptimized. But since this game makes a point of not having a unit cap, it is basically a fact of life that once a battle gets big enough and once there are enough units in existence, the sheer computational load will bring any computer, eventually, to 10, 5, or even 2 frames per second.

The interesting bit is that the game is still sort of playable since it isn't an FPS, but the first problem with a 2 fps rendering rate is not so much the choppiness of movement (which cannot really be helped without more dynamic rendering LOD or culling which is outside the scope of this question) but rather in the distinct lack of responsiveness that is caused by the synchronous nature of the mouse and keyboard input handling behavior, i.e. the mouse is rendered at the same limited frame rate. In fact what I very much expect to be happening is that the frame takes e.g. 500ms to render, and during that period the mouse event queue is piling up with about 500 mousemove events (due to my 1000Hz mouse) and perhaps a few keyboard events, and then this queue only gets a chance to flush and be handled when that frame finishes drawing, when the swapbuffers/glFinish/glFlush/etc. function returns.

That is the best case scenario. I imagine a lot of games could simply just ignore the entire queue altogether and drop a bunch of events, leading to even more unplayability at low loop frequencies. The reason for this is that upon light testing at a regular 60fps, most if not all events will still be properly handled with a faulty system like this.

One of the primary problems is that this can make certain fast actions impossible to perform. Suppose I want to drag to select a group of units in order to give them a command, but I complete the click, drag, release motion of the mouse within 500ms. If the event loop handler code does not update its state in a proper way then what effectively just happened is my entire selection command would just fail, and the best case scenario is the selection would be properly deducted but there still would never have been a frame drawn that showed the actual selection box itself (since it came into and went out of existence within the course of rendering a single frame).

Indeed what PA appears to do is only update the "I am now selecting" state once in the entire loop, so I have to click, drag, and release across a minimum of 3 separate frames in order to achieve an actual selection command. Consider this: It effectively physically limits the player to 20 APM (actions per minute).

There are a lot of gesture based controls that can exist especially in the world of tablets. In my opinion, being bound to the render loop will make for a severe responsiveness handicap.

I feel like I've over-explained the introduction by now. So the solution is clearly to decouple the UI from the render loop.

I want to render cursor position and key input and touches, with some reasonable set of UI elements, separately from my primary 3D scene. The question is, what is the best way to accomplish this?

I can think of two ways to approach it, and there may be more.

1. No multithreading. Main GL loop flushes input queue and draws UI, and does not necessarily update 3D scene. Design 3D rendering pipeline to draw fragments of the scene at a time into an alternate render buffer or texture, one piece at a time. This sounds a little impractical because it's not clear how the rendering should be split up. Tiles? scanlines? How big will they be? With a super heavy workload that normally renders at 2fps, I'd want to split the scene into 500ms/16.667ms = 30 chunks, but there's definitely no guarantee that each chunk would take the allotted 16.67ms. It also sounds like adjusting the chunk number will result in shuffling around resources on the GPU, and basically lead to a bunch of extra overhead.

2. <TL;DR #1> Two GL contexts, two threads. Thread #1 flushes input queue and draws UI, periodically updates texture on which 3D scene is drawn, draws 3D scene with full-screen-quad, and handles buffer swapping to ensure vsync smoothness. Thread #2 renders 3D scene to texture shared with Thread#1. Requires use of ping-pong scheme to facilitate resource sharing. Thread #2 flips a bit which Thread #1 will read on its next cycle to determine if it needs to flip the texture.

Option 2 sounds like the way to go here... Since option 1 still needs to read from the previously rendered frame in order to display something while still updating the UI responsively, it still needs to employ a ping-pong scheme with that texture. So I think option 2 wins on pretty much all fronts, speed and memory, and option 1 seems fraught with complexity.

However I am also seeing a few nontrivialities with option 2, because it's not clear how to make it perform properly when the 3D rendering is keeping up with the screen refresh rate -- thread #2 sort of has to wait for thread#1 to be done flipping the texture before it can start rendering to the new one. It seems like I have to dynamically switch back to a simpler "basic" pipeline if the rendering isn't taking too long.

The architecture that is emerging looks to me like an additional abstraction of a front and back buffer. The real front and back buffer are now decoupled from the heavy 3D render task so they are now continuously updated at refresh rate so long as the UI rendering can complete in time, and the flip rate of new buffer pair is the "real" framerate.

<TL;DR #2> My question is... can this be done in a cleaner way? Is there an engine design that can achieve this perhaps without requiring two contexts? I know that for example on iOS you are required to set up an OpenGL ES context for each thread, and as far as I can tell the only sane way to handle this is with threads.

I realize that one possible way "around" the problem is to just cheat by using whatever external interface system exists. For example, on Windows in an SDL application in windowed mode (or maybe even in fullscreen mode...) the default OS cursor is used by default and it should really stay responsive regardless of how quickly the buffer swaps in the GL window.

Or on iOS, you can have regular UI widget overlay views that go on top of the EAGLView which I would expect to be responsive as they bypass the GL rendering since they're not part of it.

But I am talking about a unified graphics pipeline that has UI integrated into it, where this UI is drawn with the same context used for drawing the 3D scene. This is for portability reasons. Even though I have labeled the question with OpenGL the topic obviously is applicable to DirectX for academic purposes.

• It is rather hard to extract a question from this wall of text. Could you shorten it up please? Or at least add a TL:DR section? – Kromster Feb 21 '14 at 13:50
• From my modest experience frameworks like e.g. XNA provide you with two "decoupled" methods Draw and Update. I put my game logic and input handling into the Update method and rendering occurs in the Draw method. I think it's a very typical approach. – Den Feb 21 '14 at 17:47
• @KromStern Agreed, the length is regrettable. I updated the post with markers for the two paragraphs that are most important. However you still need to skim the top section otherwise there is no context. This might be a question more suitable for StackOverflow as it may be too low-level for this site. – Steven Lu Feb 21 '14 at 17:49
• @Den Most frameworks -- whether it is XNA or not -- do indeed provide such entry points, however they are both implemented at specific points in a single event loop. It does not in any way address the problem that I am describing. – Steven Lu Feb 21 '14 at 17:51
• @StevenLu I understand your question better after the edit. I would probably render the scene to a separate texture and display as a quad but "less often" then the GUI. Should look like watching a slow-motion movie (or one dropping frames) in a very responsive video player :). – Den Feb 21 '14 at 17:58

I agree that option 2 (two render threads running at different frequencies) would be the most "logical" way to deal with this kind of situation, but it makes a big assumption: the GPU and driver have to support—or rather, simulate—pre-emptive multitasking. What you want is that the UI drawing commands from thread 1 should get high priority and get finished immediately even though thread 2 is in the middle of drawing. In other words, you don't want the GPU/driver to have to wait for thread 2's rendering to be finished before servicing thread 1.

However, you're at the mercy of the driver as to whether this works and how well. GPUs today are single-threaded devices at the top level: they only process one command buffer at a time, and they don't support pre-emption; you can't pause the GPU in the middle of a command buffer and switch to another one. So the driver pretty much has to do your option 1 internally. When you issue GL calls, it buffers up your commands then dispatches them to the GPU at some point, and it can switch between contexts only at the boundaries of these buffers.

The point here is that depending on how the driver decides to slice up command buffers, your UI commands from thread 1 might get stuck behind a long command buffer from thread 2.

My guess is that you'll find your option 2 works OK on some combinations of GPU and driver, and not on others. You might be able to help the situation by sprinking glFlush() calls into your thread 2 rendering, which hint to the driver to go ahead and finish up a command buffer, giving it the opportunity to switch contexts.

It's kind of like in the bad-old-days of non-preemptive CPU multitasking, where you'd have to explicitly yield to the OS by calling Sleep(0) or some such, or else a long-running operation would lock up the whole system. That's where we're at with GPUs today. Hopefully, GPUs (and graphics APIs) of the future will better support multitasking, but it's a really tough engineering problem to solve.

• Ah, so it's a much more difficult problem than I had imagined it. When I thought about it some more, I realized what I'm trying to do is quite like some "interrupt" system which allows me to (as you say) pre-empt and flush out a UI-update-frame on the regular vsync signal. Sounds like it's a hard problem, and I'll have to perhaps get even more creative lest we descend into driver implementation dependent behaviors. Thanks! – Steven Lu Feb 21 '14 at 19:11
• Just a heads up, feel free to head over to this related question and repost the community wiki answer as your own (it was copied from an identical question you answered). Comment on the community wiki, if you do, so i can remove it. – Gnemlock Sep 18 '16 at 0:16

It seems to me there may be a design issue in the game itself if the frame rate regularly drops to 2fps and that's considered normal. Some turn-based games might take a while to render and present the next screen as basically "static" plus some simple interactive rendering on top of it, but then that's the game style itself. Typically if your game is real-time interactive, then it has a minimum playable rate. Playability is not just animation rate, but also UI feedback rate. Of course you can always cut back on rendering detail if it's under load, but sounds like a lot to drop if starting from potentially 2fps.

Sorry if I misunderstood the question; it was fairly long :)

• Well sure, it's not ideal when things to slow down like this, but my idea was basically that it would be nice if I had two rendering contexts, one that dealt with really simple and fast rendering tasks (UI) while the other focuses on something more challenging. Basically the way that many RTS games these days get around the limitation (explained by Nathan Reed) is to use the default cursor which is being handled by the operating system at a driver level so the whole thing is separated from the hardware-accelerated graphics. – Steven Lu Apr 5 '15 at 18:44
• Hmm, let me try explaining another way. It usually takes a lot of development effort to make a game or other graphical application always scale levels of detail to maintain a desired framerate baseline. As a not-quite-alternative I wanted to know how feasible it is to just "run another thread" that's solely responsible for the user interface rendering, whose sole purpose is to maintain control responsiveness. If this could be done easily and transparently it makes for much improved usability at essentially no cost to the developer. Turns out, drivers today make no guarantees to support this. – Steven Lu Apr 5 '15 at 18:53