2
\$\begingroup\$

I'm trying to capture some screenshots for my game. They need to be saved as jpgs to a folder, latency is not an issue, I don't care if it happens over 10 seconds. My number one priority is to keep the game at a steady framerate.

Some things I've tried:

Some metrics from my testing:

  • ScreenCapture.CaptureScreenshot is ~200ms.
  • ReadPixels() is 40-70ms.
  • Texture.Apply() is ~5ms.
  • EncodeToJPG() is 35-50ms.
  • File.WriteAllBytes() is <1ms.

Also posted this question here and here:

\$\endgroup\$
1
+50
\$\begingroup\$

There's two separate parts to this question. First is how to record screenshots without hitting your framerate too much, and second is how to capture screenshots at all in the new render pipelines. The first part is way easier, so let's get that out of the way first.

Capturing screenshots/GIFs during gameplay

You said you wanted these screenshots for bug testing, and that it would be even better to have video. Luckily, there's already a plugin for exactly this! I found Moments Recorder and have used it in my own project already and the quality is great. Key features:

  • Runs a constant buffer so hitting a button exports the last x seconds, which is ideal for catching hard-to-reproduce bugs
  • Encoding runs in a separate thread, so it shouldn't affect your framerate much
  • Customizable resolution/quality, so you can reduce resolution or thread priority if it's slowing things down
  • Well-commented source code makes it easy to modify

My project was very poorly coded at the time and barely made 15fps, with randomized colors that are terrible for GIF encoding, but I saw no major additional slowdown when exporting a GIF. I should note the comments in the code for GIF encoding do say it could tank the framerate for high resolution, but if you find that's the case, then it gives plenty of options to play with to speed things up. Otherwise you can try to modify it to just export jpgs.

Here's an example of it in my editor:

Unity inspector window showing controls for Moments Recorder plugin

The moments recorder uses System.Threading, but you could also look into the Unity Jobs system for multithreading your own solution, which could potentially be safer/faster/easier. I haven't tried it, and it's still in preview, so it may still be slower. Either way, it's a good example of how to separate out the capturing/saving code if you write your own solution.

Also, if you want to record video clips while in the editor, Unity has an official Asset Store plugin for that.

Capturing screenshots in LWRP

This is a huge pain in the ass and I've been looking through solutions for a while. I'm still learning this stuff so I may have missed an obvious or easier solution. Also, all my testing has been very haphazard so I may be mixing up solutions and results, and I've crashed Unity a few times. Follow at your own risk.

Option 1 You said ScreenCapture.CaptureScreenshot() was taking 200ms, but have you tried CaptureScreenshotAsTexture()? It seems like that should be a lot faster and you could split off the file conversion/writing into a separate thread. I tried testing this both by modifying the Moments Recorder.cs and in a separate simple script, but I'm not getting any screenshot output at all and no errors, so I'm not sure what's going on there. If that method is working for you, that could be the fastest/easiest solution.

Option 2: This thread shows a way to capture an image from a camera in the new render pipelines, but the script linked doesn't show where/how it's called. Another post in the replies uses RenderPipeline.beginCameraRendering, so I modified the Moments recorder to use this callback and replaced the OnRenderImage method, but calling it at the beginning of camera rendering gives inconsistent/unhelpful results. It looks like it's moved out of experimental and a endCameraRendering event is added in 2019.1, but I don't have time to upgrade and test it. That might be the ideal solution. To do that with Moments, change the OnRenderImage() method in Recorder.cs to the following:

void CaptureCamera(Camera _camera)
    {
        if (State != RecorderState.Recording)
        {
            return;
        }

        m_Time += Time.unscaledDeltaTime;

        if (m_Time >= m_TimePerFrame)
        {
            // Limit the amount of frames stored in memory
            if (m_Frames.Count >= m_MaxFrameCount)
                m_RecycledRenderTexture = m_Frames.Dequeue();

            m_Time -= m_TimePerFrame;

            // Frame data
            RenderTexture rt = m_RecycledRenderTexture;
            m_RecycledRenderTexture = null;

            if (rt == null)
            {
                rt = new RenderTexture(m_Width, m_Height, 0, RenderTextureFormat.ARGB32);
                rt.wrapMode = TextureWrapMode.Clamp;
                rt.filterMode = FilterMode.Bilinear;
                rt.anisoLevel = 0;
            }

            _camera.targetTexture = rt;
            _camera.Render();
            _camera.targetTexture = null;

            m_Frames.Enqueue(rt);
        }
    }

Then add this in Init()

cam = GetComponent<Camera>(); //don't forget to create the cam var somewhere
RenderPipeline.EndCameraRendering(GetComponent<Camera>());
RenderPipeline.endCameraRendering += CaptureCamera;

Option 3: works in 2018.3, easy but weird results If you can't upgrade to 2019.x then this might work. I got this version working, but the resulting gif is washed out for some reason. Possibly shaders aren't being applied, not sure.

Modifying Recorder.cs in Moments, rename OnRenderImage to Update() or LateUpdate() and get rid of all the Graphics.Blit calls. Then at the end just throw this in:

Graphics.Blit(ScreenCapture.CaptureScreenshotAsTexture(), rt);
m_Frames.Enqueue(rt);
\$\endgroup\$
  • \$\begingroup\$ That looks perfect! This solves the issue I was having with the ffmpeg project I found. That is: a constant buffer that can be written to a file on demand. Just one thing tho, does it support LWRP? Also any chance of exporting as MP4 instead of gif? \$\endgroup\$ – mr-matt Mar 26 at 7:07
  • \$\begingroup\$ You'd have to find an encoder script and I don't think mp4 would be quite as easy as a gif due to the way video compression works, but I don't know, I've never tried! It should definitely work with LWRP, I'm in the process of upgrading my project right now and I needed similar code for my own issue. \$\endgroup\$ – Appleguysnake Mar 26 at 11:03
  • \$\begingroup\$ It looks like I may have misled you. This list says OnRenderImage is "Not Supported" in LWRP, but is in the new built-in render pipeline (which is what was giving me issues, so I got them mixed up, sorry) github.com/Unity-Technologies/ScriptableRenderPipeline/blob/… \$\endgroup\$ – Appleguysnake Mar 26 at 11:53
  • \$\begingroup\$ I have just tested it, I can confirm it does not work with LWRP. The FFmpeg one I found does work with LWRP though. Any ideas on how to make ffmpeg use a constant buffer like moments? \$\endgroup\$ – mr-matt Mar 26 at 18:46
  • \$\begingroup\$ That depends, how does ffmpeg grab the image? That's really the source of the problem here since the rest can be done asynchronously in a separate thread. I found this guide to writing post-processing effects with Command Buffers instead of OnRenderImage, but it seems a lot more complicated so I haven't been able to go through it yet github.com/Unity-Technologies/PostProcessing/wiki/… \$\endgroup\$ – Appleguysnake Mar 27 at 2:42
0
\$\begingroup\$

You should look into https://docs.unity3d.com/ScriptReference/Graphics.Blit.html.

Graphics.Blit is unity's API for copying from one texture to another. If you render your scene into a render texture (Which I assume the lightweight pipeline can do, correct me if I'm wrong), then blit that render texture using the scriptable pipeline API. Then render the render texture to the screen. And as crueltear has said save processing for another thread.

\$\endgroup\$
  • \$\begingroup\$ The problem is, to get data out of a render texture and into a jpg, I have to call ReadPixels() and EncodeToJpg(), both of which are ludicrously slow. To make things worse, none of unity is thread safe. So the only thing I can put on another thread is File.WriteAllBytes(), but that was always pretty fast anyway... \$\endgroup\$ – mr-matt Mar 25 at 16:41
  • \$\begingroup\$ @mr-matt remember you're not limited to Unity's methods here. You could use an external image compression library offered as source code, linked as a plugin, or sold as an Asset Store download, and run that on a thread of your choosing. \$\endgroup\$ – DMGregory Mar 25 at 17:41
  • \$\begingroup\$ @DMGregory Something to consider for sure, however, have a look at the performance metrics I took, I added them to the original question. ReadPixels was by far the slowest part, so even if I could shave the encoding down to 15ms, I would still be stuck with 40-70ms for reading the pixels from the screen. From what I could see, there is no way around ReadPixels. Even if I could asynchronously read the screen from the GPU with ASyncGpuReadback, command buffers or some SRP magic, I would still have to convert the RenderTexture to a Texture2D. \$\endgroup\$ – mr-matt Mar 25 at 18:23
  • \$\begingroup\$ The timing with which you call ReadPixels can be a factor, since part of the time taken could be waiting for a pipeline flush. Waiting until a moment when the GPU isn't busy may reduce this. Also, have you looked at reading just part of the render texture at a time? I'd be interested to see how much of the time is data transfer vs how much is waiting to sync. \$\endgroup\$ – DMGregory Mar 25 at 18:26
  • \$\begingroup\$ I just tested it with different resolutions. It's pretty clear that it scales with resolution. 50x50px was 2ms, 720p was 20ms, 1080p was 50ms, 2880x1800 (MBP) was 110ms, and 4k was 170ms. \$\endgroup\$ – mr-matt Mar 25 at 18:41
0
\$\begingroup\$

My response is based on taking screenshots in pure C++/OpenGL - I have no experience with Unity.

Most of the time needed for screenshots is for compression and saving the image.

Reducing the quality of saved image should decrease the time. Another option is to grab the screen pixels, and let another thread do the compression and saving.

If we're not speaking about 4K resolution, saving raw (BMP) images should be faster than JPEG/PNG (unless you're writing to a floppy).

Update: I took a look into Unity's API. If I were to add screenshot feature to Unity game, it would probably look like that:

  1. Read screen's pixels: https://docs.unity3d.com/ScriptReference/Texture2D.ReadPixels.html
  2. Launch separate thread (is it possible?)
  3. Encode and save to file on separate thread https://answers.unity.com/questions/332335/saving-a-texture2d-as-a-png.html

Screenshots in C++ (SFML):

// Create texture
sf::Texture texture;
texture.create(windowSize.x, windowSize.y);
// Move window's pixels to texture
texture.update(window);
// VRAM -> RAM
sf::Image screenshot=texture.copyToImage();
// Encode+save to HDD
screenshot.saveToFile(ss.str().c_str());
\$\endgroup\$
  • \$\begingroup\$ What platforms would this support? And can you provide any examples? I have very little experience in cpp so I wouldn't really know where to start. \$\endgroup\$ – mr-matt Mar 25 at 16:44
  • \$\begingroup\$ On another note, if I did this, can I target a specific window and only that window? I would be a bit concerned about the players privacy if it captures the whole screen even when the game is in windowed mode. \$\endgroup\$ – mr-matt Mar 25 at 16:46
0
\$\begingroup\$

For capturing video instead:

I found a github repo that can export mp4s using ffmpeg at runtime: https://github.com/keijiro/FFmpegOut

I had to make some modifications to make it save the last few seconds on demand, instead of a continuous recording. Specifically:

  • Don't send each frame to the FFmpeg binaries as they come. Instead, save them in memory until further notice.
  • Have a max number of frames to be saved in memory. When this number is exceeded, remove the oldest saved frame. (I actually calculated the approximate memory usage per frame and used this to dynamically set the max number of frames based on the max amount of ram I can budget to the recorder)
  • When it's time to export the last few seconds of gameplay, send the entire queue to FFmpeg all at once. By this time, all the data is in a byte array already and so it can be multithreaded - important to ensure the main thread isn't blocked.

The final result of this was a steady 150 fps while recording at 1080p.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.