High performance screenshots in LWRP

Question

I'm trying to capture some screenshots for my game. They need to be saved as jpgs to a folder, latency is not an issue, I don't care if it happens over 10 seconds. My number one priority is to keep the game at a steady framerate.

Some things I've tried:

• I've tried ScreenCapture.CaptureScreenshot but that is super slow and the game hangs for a split second.

• I've tried the old method of texture2d.ReadPixels() and EncodeToJPG() but that was unbelievably slow.

• I also found a relatively new feature: https://docs.unity3d.com/2018.3/Documentation/ScriptReference/Rendering.AsyncGPUReadback.html

  • I found a repo using this feature to capture screenshots asynchronously: https://github.com/keijiro/AsyncCaptureTest

  • This requires the use of OnRenderImage(), but I'm using LWRP and OnRenderImage() isn't supported in scriptable render pipeline.

• Pulling my hair out

Some metrics from my testing:

• ScreenCapture.CaptureScreenshot is ~200ms.
• ReadPixels() is 40-70ms.
• Texture.Apply() is ~5ms.
• EncodeToJPG() is 35-50ms.
• File.WriteAllBytes() is <1ms.

Also posted this question here and here:

Answer 1

There's two separate parts to this question. First is how to record screenshots without hitting your framerate too much, and second is how to capture screenshots at all in the new render pipelines. The first part is way easier, so let's get that out of the way first.

Capturing screenshots/GIFs during gameplay

You said you wanted these screenshots for bug testing, and that it would be even better to have video. Luckily, there's already a plugin for exactly this! I found Moments Recorder and have used it in my own project already and the quality is great. Key features:

• Runs a constant buffer so hitting a button exports the last x seconds, which is ideal for catching hard-to-reproduce bugs
• Encoding runs in a separate thread, so it shouldn't affect your framerate much
• Customizable resolution/quality, so you can reduce resolution or thread priority if it's slowing things down
• Well-commented source code makes it easy to modify

My project was very poorly coded at the time and barely made 15fps, with randomized colors that are terrible for GIF encoding, but I saw no major additional slowdown when exporting a GIF. I should note the comments in the code for GIF encoding do say it could tank the framerate for high resolution, but if you find that's the case, then it gives plenty of options to play with to speed things up. Otherwise you can try to modify it to just export jpgs.

Here's an example of it in my editor:

The moments recorder uses System.Threading, but you could also look into the Unity Jobs system for multithreading your own solution, which could potentially be safer/faster/easier. I haven't tried it, and it's still in preview, so it may still be slower. Either way, it's a good example of how to separate out the capturing/saving code if you write your own solution.

Also, if you want to record video clips while in the editor, Unity has an official Asset Store plugin for that.

Capturing screenshots in LWRP

This is a huge pain in the ass and I've been looking through solutions for a while. I'm still learning this stuff so I may have missed an obvious or easier solution. Also, all my testing has been very haphazard so I may be mixing up solutions and results, and I've crashed Unity a few times. Follow at your own risk.

Option 1 You said ScreenCapture.CaptureScreenshot() was taking 200ms, but have you tried CaptureScreenshotAsTexture()? It seems like that should be a lot faster and you could split off the file conversion/writing into a separate thread. I tried testing this both by modifying the Moments Recorder.cs and in a separate simple script, but I'm not getting any screenshot output at all and no errors, so I'm not sure what's going on there. If that method is working for you, that could be the fastest/easiest solution.

Option 2: This thread shows a way to capture an image from a camera in the new render pipelines, but the script linked doesn't show where/how it's called. Another post in the replies uses RenderPipeline.beginCameraRendering, so I modified the Moments recorder to use this callback and replaced the OnRenderImage method, but calling it at the beginning of camera rendering gives inconsistent/unhelpful results. It looks like it's moved out of experimental and a endCameraRendering event is added in 2019.1, but I don't have time to upgrade and test it. That might be the ideal solution. To do that with Moments, change the OnRenderImage() method in Recorder.cs to the following:

void CaptureCamera(Camera _camera)
    {
        if (State != RecorderState.Recording)
        {
            return;
        }

        m_Time += Time.unscaledDeltaTime;

        if (m_Time >= m_TimePerFrame)
        {
            // Limit the amount of frames stored in memory
            if (m_Frames.Count >= m_MaxFrameCount)
                m_RecycledRenderTexture = m_Frames.Dequeue();

            m_Time -= m_TimePerFrame;

            // Frame data
            RenderTexture rt = m_RecycledRenderTexture;
            m_RecycledRenderTexture = null;

            if (rt == null)
            {
                rt = new RenderTexture(m_Width, m_Height, 0, RenderTextureFormat.ARGB32);
                rt.wrapMode = TextureWrapMode.Clamp;
                rt.filterMode = FilterMode.Bilinear;
                rt.anisoLevel = 0;
            }

            _camera.targetTexture = rt;
            _camera.Render();
            _camera.targetTexture = null;

            m_Frames.Enqueue(rt);
        }
    }

Then add this in Init()

cam = GetComponent<Camera>(); //don't forget to create the cam var somewhere
RenderPipeline.EndCameraRendering(GetComponent<Camera>());
RenderPipeline.endCameraRendering += CaptureCamera;

Option 3: works in 2018.3, easy but weird results If you can't upgrade to 2019.x then this might work. I got this version working, but the resulting gif is washed out for some reason. Possibly shaders aren't being applied, not sure.

Modifying Recorder.cs in Moments, rename OnRenderImage to Update() or LateUpdate() and get rid of all the Graphics.Blit calls. Then at the end just throw this in:

Graphics.Blit(ScreenCapture.CaptureScreenshotAsTexture(), rt);
m_Frames.Enqueue(rt);

Answer 2

You should look into https://docs.unity3d.com/ScriptReference/Graphics.Blit.html.

Graphics.Blit is unity's API for copying from one texture to another. If you render your scene into a render texture (Which I assume the lightweight pipeline can do, correct me if I'm wrong), then blit that render texture using the scriptable pipeline API. Then render the render texture to the screen. And as crueltear has said save processing for another thread.

Answer 3

My response is based on taking screenshots in pure C++/OpenGL - I have no experience with Unity.

Most of the time needed for screenshots is for compression and saving the image.

Reducing the quality of saved image should decrease the time. Another option is to grab the screen pixels, and let another thread do the compression and saving.

If we're not speaking about 4K resolution, saving raw (BMP) images should be faster than JPEG/PNG (unless you're writing to a floppy).

Update: I took a look into Unity's API. If I were to add screenshot feature to Unity game, it would probably look like that:

1. Read screen's pixels: https://docs.unity3d.com/ScriptReference/Texture2D.ReadPixels.html
2. Launch separate thread (is it possible?)
3. Encode and save to file on separate thread https://answers.unity.com/questions/332335/saving-a-texture2d-as-a-png.html

Screenshots in C++ (SFML):

// Create texture
sf::Texture texture;
texture.create(windowSize.x, windowSize.y);
// Move window's pixels to texture
texture.update(window);
// VRAM -> RAM
sf::Image screenshot=texture.copyToImage();
// Encode+save to HDD
screenshot.saveToFile(ss.str().c_str());

Answer 4

For capturing video instead:

I found a github repo that can export mp4s using ffmpeg at runtime: https://github.com/keijiro/FFmpegOut

I had to make some modifications to make it save the last few seconds on demand, instead of a continuous recording. Specifically:

• Don't send each frame to the FFmpeg binaries as they come. Instead, save them in memory until further notice.
• Have a max number of frames to be saved in memory. When this number is exceeded, remove the oldest saved frame. (I actually calculated the approximate memory usage per frame and used this to dynamically set the max number of frames based on the max amount of ram I can budget to the recorder)
• When it's time to export the last few seconds of gameplay, send the entire queue to FFmpeg all at once. By this time, all the data is in a byte array already and so it can be multithreaded - important to ensure the main thread isn't blocked.

The final result of this was a steady 150 fps while recording at 1080p.