OpenGL managing many textures smoothly

Question

I'm working on a game right now and running into some noticeable performance issues and limitations.

Basically, I have this:

• A few QGLWidgets, all sharing contexts with eachother (probably irrelevant that I'm using Qt [also: Windows, fwiw]).
• About 216000 total frames of 128x128 videos (with alpha, internal format GL_RGBA, data format GL_BGRA/GL_UNSIGNED_INT_8_8_8_8_REV, video codec is Quicktime RLE, decoded with libav), spread into approximately 120 videos of 1800 frame average length. This translates to roughly 216000*128*128*4 = 14.2 GB of raw data, not including overhead.

The game behaves as follows:

• There are many "modes". At any given time only one mode is active. Each mode uses a very small known subset of the available videos, say 5-10 videos (= about 9000 - 18000 frames) tops.
• There's a bunch of 2D quads flying around the screen, each displaying whatever videos depending on the current mode. Objects don't all display the same frame, the video time offsets are random. The videos loop.
• Every few minutes, the mode changes. Objects cross-fade to their new videos, so there is some overlap regarding when various sets of videos are needed.
• It is critical that this thing keep running smoothly, at 60 FPS, with no pauses for loading, etc.

My problem is this:

My initial approach was to simply preload all videos off the disk and into textures, in video memory (via glTexImage2D and friends). I didn't mind the slow startup. This worked great early on, when I didn't have a lot of videos. The game could run smoothly because all the textures were already loaded.

Now I have too many videos. I'm becoming extremely limited by available video memory, and the preload-everything strategy is no longer working. I haven't become limited by system memory yet but that's also inevitable. I suspected this was going to be an issue and kept my fingers crossed, but alas. I hit the limit way sooner than I expected.

I tried switching to a strategy where, every time a mode transition started I pre-loaded the videos for that mode (and every time a transition ended I unloaded the videos for the previous mode). This works as far as memory goes but it pauses everything for a very long time while loading (as in 3-8 seconds depending on the number of videos and the graphics hardware).

I've done the best searching that I know how to do, all of the texture memory management posts, blogs, tutorials etc. just talk about the basic technique of creating and destroying textures, rather than strategies for managing large numbers of them in an application. I went through every question in the auto-search on here that had a relevant-looking title. I did find out that OpenGL is definitely not thread-safe, which kind of kills my asynchronous background loading idea, unless there's some other way to implement that.

So my question is: What's a good strategy to use for swapping out tons of textures (200000+, 128x128 typical size) when at any given time only a small portion of those are needed, while maintaining a smooth frame rate? How is this typically done?

Answer 1

While OpenGL is thread unsafe, you can load such data asynchronously.

Say you know you will transition in about 20 seconds. You start loading the image data on another thread. The thread loads the image data from disk to memory, then sets a flag for the main thread(which owns the OpenGL context), to load it to the GPU.

In the main loop, perhaps in your update function, you will check if there is anything to load, but do it cleverly, so you will load an image, and check if you have some frame-time left. If you do, load another, if you don't then bail out, and continue uploading the video frames next frame/iteration.

If you start processing early enough, there will be no delay when the time to transition comes. You just need to make sure your program doesn't crash if the resources aren't loaded yet, and handle all the corner cases this solution imposes.

It definitely is a much more complex solution than just pre-loading, but to me it seems this is a necessary complication to solve your problem.