Warning! Wall of text (see <TL;DR> paragraphs below for short version)
I have been noticing something in quite many games (most recently in cutting-edge RTS games such as Uber Entertainment's Planetary Annihilation, which is amazing by the way) that I think has room for improvement.
That is, the rendering system is synchronous in the sense that input event loops are apparently "bound" to the rendering loop. So the consequences are that e.g. mouse movements and keyboard events are usually queued up and processed only once a frame is finished rendering.
This may not be a completely accurate account of what's actually happening. What is certainly clear is that for example the mouse cursor movement updates no faster than the graphics in the game render, and obviously when keyboard events are fired, and their consequences are not visible in the game until the next frame renders.
The delay of that frame, if the framerate is very low, will very much be felt, and I want to emphasize that the same low frequency of response applied to input is a much bigger problem than if we are only talking about its impact on animation smoothness.
Of course Planetary Annihilation is a game that is still in beta, and is (as we are told) still rather unoptimized. But since this game makes a point of not having a unit cap, it is basically a fact of life that once a battle gets big enough and once there are enough units in existence, the sheer computational load will bring any computer, eventually, to 10, 5, or even 2 frames per second.
The interesting bit is that the game is still sort of playable since it isn't an FPS, but the first problem with a 2 fps rendering rate is not so much the choppiness of movement (which cannot really be helped without more dynamic rendering LOD or culling which is outside the scope of this question) but rather in the distinct lack of responsiveness that is caused by the synchronous nature of the mouse and keyboard input handling behavior, i.e. the mouse is rendered at the same limited frame rate. In fact what I very much expect to be happening is that the frame takes e.g. 500ms to render, and during that period the mouse event queue is piling up with about 500 mousemove events (due to my 1000Hz mouse) and perhaps a few keyboard events, and then this queue only gets a chance to flush and be handled when that frame finishes drawing, when the swapbuffers/glFinish/glFlush/etc. function returns.
That is the best case scenario. I imagine a lot of games could simply just ignore the entire queue altogether and drop a bunch of events, leading to even more unplayability at low loop frequencies. The reason for this is that upon light testing at a regular 60fps, most if not all events will still be properly handled with a faulty system like this.
One of the primary problems is that this can make certain fast actions impossible to perform. Suppose I want to drag to select a group of units in order to give them a command, but I complete the click, drag, release motion of the mouse within 500ms. If the event loop handler code does not update its state in a proper way then what effectively just happened is my entire selection command would just fail, and the best case scenario is the selection would be properly deducted but there still would never have been a frame drawn that showed the actual selection box itself (since it came into and went out of existence within the course of rendering a single frame).
Indeed what PA appears to do is only update the "I am now selecting" state once in the entire loop, so I have to click, drag, and release across a minimum of 3 separate frames in order to achieve an actual selection command. Consider this: It effectively physically limits the player to 20 APM (actions per minute).
There are a lot of gesture based controls that can exist especially in the world of tablets. In my opinion, being bound to the render loop will make for a severe responsiveness handicap.
I feel like I've over-explained the introduction by now. So the solution is clearly to decouple the UI from the render loop.
I want to render cursor position and key input and touches, with some reasonable set of UI elements, separately from my primary 3D scene. The question is, what is the best way to accomplish this?
I can think of two ways to approach it, and there may be more.
No multithreading. Main GL loop flushes input queue and draws UI, and does not necessarily update 3D scene. Design 3D rendering pipeline to draw fragments of the scene at a time into an alternate render buffer or texture, one piece at a time. This sounds a little impractical because it's not clear how the rendering should be split up. Tiles? scanlines? How big will they be? With a super heavy workload that normally renders at 2fps, I'd want to split the scene into 500ms/16.667ms = 30 chunks, but there's definitely no guarantee that each chunk would take the allotted 16.67ms. It also sounds like adjusting the chunk number will result in shuffling around resources on the GPU, and basically lead to a bunch of extra overhead.
<TL;DR #1> Two GL contexts, two threads. Thread #1 flushes input queue and draws UI, periodically updates texture on which 3D scene is drawn, draws 3D scene with full-screen-quad, and handles buffer swapping to ensure vsync smoothness. Thread #2 renders 3D scene to texture shared with Thread#1. Requires use of ping-pong scheme to facilitate resource sharing. Thread #2 flips a bit which Thread #1 will read on its next cycle to determine if it needs to flip the texture.
Option 2 sounds like the way to go here... Since option 1 still needs to read from the previously rendered frame in order to display something while still updating the UI responsively, it still needs to employ a ping-pong scheme with that texture. So I think option 2 wins on pretty much all fronts, speed and memory, and option 1 seems fraught with complexity.
However I am also seeing a few nontrivialities with option 2, because it's not clear how to make it perform properly when the 3D rendering is keeping up with the screen refresh rate -- thread #2 sort of has to wait for thread#1 to be done flipping the texture before it can start rendering to the new one. It seems like I have to dynamically switch back to a simpler "basic" pipeline if the rendering isn't taking too long.
The architecture that is emerging looks to me like an additional abstraction of a front and back buffer. The real front and back buffer are now decoupled from the heavy 3D render task so they are now continuously updated at refresh rate so long as the UI rendering can complete in time, and the flip rate of new buffer pair is the "real" framerate.
<TL;DR #2> My question is... can this be done in a cleaner way? Is there an engine design that can achieve this perhaps without requiring two contexts? I know that for example on iOS you are required to set up an OpenGL ES context for each thread, and as far as I can tell the only sane way to handle this is with threads.
I realize that one possible way "around" the problem is to just cheat by using whatever external interface system exists. For example, on Windows in an SDL application in windowed mode (or maybe even in fullscreen mode...) the default OS cursor is used by default and it should really stay responsive regardless of how quickly the buffer swaps in the GL window.
Or on iOS, you can have regular UI widget overlay views that go on top of the EAGLView which I would expect to be responsive as they bypass the GL rendering since they're not part of it.
But I am talking about a unified graphics pipeline that has UI integrated into it, where this UI is drawn with the same context used for drawing the 3D scene. This is for portability reasons. Even though I have labeled the question with OpenGL the topic obviously is applicable to DirectX for academic purposes.
