Skipping glReadPixels and reading iPhone's "VRAM" directly

Question

I'd like to rapidly capture "screenshots" of my OpenGL ES iPhone game and turn them into a video. I've tried the following approach (it works but degrades the game play to an unacceptable level):

• Use glReadPixels to copy the data from the framebuffer to a GLubyte array. Create a CGImageRef from that data. Then create a CVPixelBuffer from the CGImageRef. Finally append the CVPixelBuffer to an AVAssetWriter.

I've done a fair amount of additional research. Some people suggest copying the framebuffer and reading that piecewise with glReadPixels. This could work... but I don't know how to copy a framebuffer. (any help is appreciated!)

I just learned that the iPhone's "VRAM" is really part of the main memory. How can I read the contents of a framebuffer from "VRAM" without calling glReadPixels?

Hopefully this approach (or similar) will keep the frame rate of the game acceptable.

Answer 1

You can't. Only an OpenGL ES Extensions could allow it, but nothing is available. This item is on the Aras wishlist (Unity Engineer).

Why can't you use IOS emulator to do videos ? The emulator is often faster than real devices.

Answer 2

There are a number of ways to accelerate glReadPixels.

Firstly you can experiment with different values for the format and type parameters - getting these to match the layout of your framebuffer means that glReadPixels can do a direct transfer from the framebuffer; otherwise it will have to round-trip through software to convert from the framebuffer format/type to your specified format/type.

On PC hardware I've personally found that a format of GL_BGRA gives the best performance. On some drivers using a type of GL_UNSIGNED_INT_8_8_8_8_REV (instead of the more commonly seen GL_UNSIGNED_BYTE) is also required. It may be the same or may be different for the iPhone so experiment.

Secondly what you do with the resulting data can have an effect. Using an image format of TGA for example means that no further conversions need to happen (if your fastest format is GL_BGR or GL_BGRA then all the better - you don't need to swap the colour components). Just alloc a buffer of the appropriate size + 18 (which can be a one-time-only op), glReadPixels into &buffer[18], then fill in buffer[0] to buffer[17] with data for a TGA header - done.

10 or so years ago I used this method to export video from a PC game on a TNT2 and was able to get exports at maybe 30 FPS/640x480 resolution (including time spent in an AVI encoder).

Finally, glReadPixels will need to stall the pipeline in order to complete the readback. If you have PBOs available you should be able to read asynchronously into a PBO and avoid this stall. I've personally never used this method though, so can't comment any further on it.

Answer 3

Three answers:

First, figure out where your bottleneck is. You've described a very long process there. Where's the slowdown? If the slowdown is glReadPixels itself, then you've got a problem (see second answer). If the slowdown is everything between glReadPixels and AVAssetWriter, maybe you just need a more efficient way of passing it into AVAssetWriter. If the slowdown is AVAssetWriter itself, then maybe you just need a more efficient way of dumping the video to disk.

Second, if the slowdown is glReadPixels . . . well, if a much faster approach existed, don't you think glReadPixels would likely use it? It might just be slow. You might just be out of luck.

Third, I know I've seen videos of iPhone games. Contact an indie developer with an iPhone video. Ask them how they did it. Then post the answer here.

Answer 4

If you are just making a demo video then turn your game engine into a rendering engine.

Have a very set time step, no real time involved in this at all. Then instead of rendering the game to the screen, you render each frame to a file.. Now clearly this will require you to be able to script or direct the action that you are trying to capture, but this is a -very- common way demo videos are captured from yet unreleased games.

Again, this may require additional development work to accomplish the scripting aspects of it since you are pretty much removing real time from it.

Hope this helps.

Answer 5

To get the contents of the backbuffer, the fastest way is to render all your scene to an intermediate FBO, save the frame data to a file and draw onscreen.