3
\$\begingroup\$

I'd like to rapidly capture "screenshots" of my OpenGL ES iPhone game and turn them into a video. I've tried the following approach (it works but degrades the game play to an unacceptable level):

  • Use glReadPixels to copy the data from the framebuffer to a GLubyte array. Create a CGImageRef from that data. Then create a CVPixelBuffer from the CGImageRef. Finally append the CVPixelBuffer to an AVAssetWriter.

I've done a fair amount of additional research. Some people suggest copying the framebuffer and reading that piecewise with glReadPixels. This could work... but I don't know how to copy a framebuffer. (any help is appreciated!)

I just learned that the iPhone's "VRAM" is really part of the main memory. How can I read the contents of a framebuffer from "VRAM" without calling glReadPixels?

Hopefully this approach (or similar) will keep the frame rate of the game acceptable.

\$\endgroup\$

5 Answers 5

3
\$\begingroup\$

You can't. Only an OpenGL ES Extensions could allow it, but nothing is available. This item is on the Aras wishlist (Unity Engineer).

Why can't you use IOS emulator to do videos ? The emulator is often faster than real devices.

\$\endgroup\$
1
  • \$\begingroup\$ Simulator won't work for this unfortunately. \$\endgroup\$
    – MrDatabase
    Sep 15, 2011 at 20:40
3
\$\begingroup\$

There are a number of ways to accelerate glReadPixels.

Firstly you can experiment with different values for the format and type parameters - getting these to match the layout of your framebuffer means that glReadPixels can do a direct transfer from the framebuffer; otherwise it will have to round-trip through software to convert from the framebuffer format/type to your specified format/type.

On PC hardware I've personally found that a format of GL_BGRA gives the best performance. On some drivers using a type of GL_UNSIGNED_INT_8_8_8_8_REV (instead of the more commonly seen GL_UNSIGNED_BYTE) is also required. It may be the same or may be different for the iPhone so experiment.

Secondly what you do with the resulting data can have an effect. Using an image format of TGA for example means that no further conversions need to happen (if your fastest format is GL_BGR or GL_BGRA then all the better - you don't need to swap the colour components). Just alloc a buffer of the appropriate size + 18 (which can be a one-time-only op), glReadPixels into &buffer[18], then fill in buffer[0] to buffer[17] with data for a TGA header - done.

10 or so years ago I used this method to export video from a PC game on a TNT2 and was able to get exports at maybe 30 FPS/640x480 resolution (including time spent in an AVI encoder).

Finally, glReadPixels will need to stall the pipeline in order to complete the readback. If you have PBOs available you should be able to read asynchronously into a PBO and avoid this stall. I've personally never used this method though, so can't comment any further on it.

\$\endgroup\$
1
  • 1
    \$\begingroup\$ +1 for the stall. You may wish to do the glReadPixels after you've flushing the pipeline anyway, like near the page flip. \$\endgroup\$ Sep 16, 2011 at 8:34
2
\$\begingroup\$

Three answers:

First, figure out where your bottleneck is. You've described a very long process there. Where's the slowdown? If the slowdown is glReadPixels itself, then you've got a problem (see second answer). If the slowdown is everything between glReadPixels and AVAssetWriter, maybe you just need a more efficient way of passing it into AVAssetWriter. If the slowdown is AVAssetWriter itself, then maybe you just need a more efficient way of dumping the video to disk.

Second, if the slowdown is glReadPixels . . . well, if a much faster approach existed, don't you think glReadPixels would likely use it? It might just be slow. You might just be out of luck.

Third, I know I've seen videos of iPhone games. Contact an indie developer with an iPhone video. Ask them how they did it. Then post the answer here.

\$\endgroup\$
1
  • \$\begingroup\$ glReadPixels is the bottleneck when trying to copy the entire screen. If I copy part of the screen then AVAssetWriter's appendPixelBuffer is the bottleneck. Now I'm thinking about drawing directly into a framebuffer-attachable image. \$\endgroup\$
    – MrDatabase
    Sep 15, 2011 at 19:17
1
\$\begingroup\$

If you are just making a demo video then turn your game engine into a rendering engine.

Have a very set time step, no real time involved in this at all. Then instead of rendering the game to the screen, you render each frame to a file.. Now clearly this will require you to be able to script or direct the action that you are trying to capture, but this is a -very- common way demo videos are captured from yet unreleased games.

Again, this may require additional development work to accomplish the scripting aspects of it since you are pretty much removing real time from it.

Hope this helps.

\$\endgroup\$
1
\$\begingroup\$

To get the contents of the backbuffer, the fastest way is to render all your scene to an intermediate FBO, save the frame data to a file and draw onscreen.

\$\endgroup\$
5
  • \$\begingroup\$ How to efficiently save the frame data to a file? \$\endgroup\$
    – MrDatabase
    Sep 15, 2011 at 21:12
  • 1
    \$\begingroup\$ Here you are bound by the speed of your I/O hardware; the rule of thumb is saving to disk raw FBO dumps (same color format, no data format conversion) so you don't waste precious time. And try implementing @James solutions and running your game in non-interactive mode when recording video; this way the animations won't be jerky due to low framerate. \$\endgroup\$
    – r2d2rigo
    Sep 15, 2011 at 22:49
  • \$\begingroup\$ raw FBO dumps sound interesting. Formatting the data later will work. How would one save the contents of an FBO to disk? \$\endgroup\$
    – MrDatabase
    Sep 15, 2011 at 23:06
  • 1
    \$\begingroup\$ It seems the only way to read data back from GPU is using glReadPixels, so you should do it with your backbuffer/FBO. \$\endgroup\$
    – r2d2rigo
    Sep 16, 2011 at 0:11
  • \$\begingroup\$ Cool. What do you think about reading piecewise into a GLubyte buffer with glReadPixels... then dump the buffer to disk on another thread? The buffer would be alloc'd once and re-used. Suppose I might need two buffers since I'll be filling one while dumping the contents of the other to disk. \$\endgroup\$
    – MrDatabase
    Sep 16, 2011 at 0:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .