Instant replay system by saving frames to memory in Unity

Question

When designing an instant replay system that will playback the last n seconds of gameplay, I've seen people essentially record all of the game events and game object states and play it back. This seems like a very heavy weight solution. TowerFall achieves its instant replay by saving the last n seconds of the frame buffer and simply plays them back in order.

How would you achieve this in Unity? My initial attempt was to simply keep a Queue of Texture2D objects of the entire screen, ensuring the Queue never got larger then a few seconds worth of Texture2D objects.

However, I quickly discovered this was extremely inefficient as memory ballooned to several gigs and the frame rate dropped significantly. I know TowerFall wasn't developed in Unity, but it seems like it was able to efficiently achieve this, so I'm curious if my approach was just wrong?

Here is the code I used (from memory)

Queue<Texture2D> frames = new Queue<Texture2D>();

// 130 frames is the equivalent of ~5 seconds.
if (frames.Count > 130) {
    var trash = frames.Dequeue();
    trash = null;
}

var frame = new Texture2D (Screen.width, Screen.height, TextureFormat.RGB24, false);
// Read screen contents into the texture
frame.ReadPixels(Rect(0, 0, Screen.width, Screen.height), 0, 0);
frame.Apply();
frames.Enqueue(frame);

Answer 1

Texture2D in Unity are not garbage collected as readily as other content, because they live partly in graphics memory - so Unity can't easily verify they're not in use without a time-consuming sweep. Usually loose materials and textures are swept up when loading a new scene (and in-Editor, you'll get a warning about unreferenced assets left dangling)

So, it's better to explicitly reuse your textures - using a ring buffer to overwrite the contents of old ones rather than continually constructing new ones - and explicitly delete them with Destroy() when you're finished with them.

You can also reduce the performance impact by using RenderTextures instead of Texture2D. These live GPU-side, so you're not tying up CPU-GPU bandwidth every time you save a frame using Graphics.Blit to copy the framebuffer. You can still pull them down one at a time into a Texture2D if you need to read them CPU-side or store them to disc, but basic in-memory recording and playback can live entirely GPU-side.

Answer 2

In film 24fps is common. I see no reason why you can't use 20fps capping the Queue to 100 frames. Then you'll need to calculate the current fps and make sure you are skipping the appropriate number of frames. Example: if you have 60fps you are going to exceed your limit in not 5 seconds, but 2. So if the you have 60fps you'll need to record every 3rd frame instead.

Also, make sure to yield and WaitForEndOfFrame() before you begin your capture or weird shit will happen. Lastly there's no reason the texture needs to be full res, cut that bitch in half or even a quarter till you get the performance you desire.

Follow these steps and your framerate/memory issues will go away. A 640x320 jpg is roughly 50kb, which results in ~1mb per second you'd like to record.