5
\$\begingroup\$

I am writing an application in Unity which will be required to capture an image from a camera every frame (at ~60fps), and send the resultant data to another service running locally.

The issue is, I am aware that capturing the rendered data from the camera can cause massive frame rate drops (as explained in this article) when using the GetPixels() method. The article explains that "GetPixels() blocks for ReadPixels() to complete" and "ReadPixels() blocks while flushing the GPU" which is why the GPU and CPU have to sync up, resulting in a lag.

I have produced a sample project with a script attached which simply outputs frames to a file as a PNG to replicate the functionality of the program I wish to create. I have done my best to implement what is described in the article, namely allowing the GPU to render a frame, then wait a few frames before calling GetPixels() so as not to cause the GPU and CPU to forcefully sync up. However, I really haven't made any progress with it. The project still plays at about 10-15fps.

How can I achieve a realtime capture of 60 frames per second in Unity?

using System;
using System.Collections;
using System.IO;
using UnityEngine;

namespace Assets
{
    public class MyClass: MonoBehaviour
    {
        private const float reportInterval = 0.5f;
        private int screenshotCount = 0;
        private const float maxElapsedSecond = 20;
        private string screenshotsDirectory = "UnityHeadlessRenderingScreenshots";
        public Camera camOV;
        public RenderTexture currentRT;
        private int frameCount = 0;
        private Texture2D resultantImage;

        public void Start()
        {
            camOV.forceIntoRenderTexture = true;
            if (Directory.Exists(screenshotsDirectory))
            {
                Directory.Delete(screenshotsDirectory, true);
            }
            if (!Application.isEditor)
            {
                Directory.CreateDirectory(screenshotsDirectory);
                camOV.targetTexture = currentRT;
            }
        }

        // Update is called once per frame
        public void Update()
        {
            //Taking Screenshots
            frameCount += 1;
            if (frameCount == 1)
            {
                TakeScreenShot();
            }
            else if (frameCount == 3)
            {
                ReadPixelsOut("SS_"+screenshotCount+".png");
            }

            if (frameCount >= 3)
            {
                frameCount = 0;
            }
        }

        public void TakeScreenShot()
        {
            screenshotCount += 1;
            RenderTexture.active = camOV.targetTexture;
            camOV.Render();
            resultantImage = new Texture2D(camOV.targetTexture.width, camOV.targetTexture.height, TextureFormat.RGB24, false);
            resultantImage.ReadPixels(new Rect(0, 0, camOV.targetTexture.width, camOV.targetTexture.height), 0, 0);
            resultantImage.Apply();
        }

        private void ReadPixelsOut(string filename)
        {
            if (resultantImage != null)
            {
                resultantImage.GetPixels();
                RenderTexture.active = currentRT;

                byte[] bytes = resultantImage.EncodeToPNG();
                // save on disk
                var path = screenshotsDirectory + "/" + filename;
                File.WriteAllBytes(path, bytes);

                Destroy(resultantImage);
            }
        }
    }
}

The article implies that it is possible, but I haven't managed to get it to work.

Many thanks in advance for your help.

\$\endgroup\$
1
\$\begingroup\$

I've done something similar to this using compute shaders to convert a renderTarget to an array of Vector4s (or any other datatype).

I've never had any problems with performance. All the compute shader does is convert the texture into a buffer than can easily be read back.

#pragma kernel CSMain
Texture2D<float4> t;
RWStructuredBuffer<float4> pixels;

[numthreads(8,8,1)]
void CSMain (uint3 id : SV_DispatchThreadID)
{
    float4 temp = t[id.xy];
    pixels[twoToOne(id.xy)] = temp;
}

twoToOne is just a simple 2D to 1D conversion. So yea. If you plugin your renderTarget to t, and pull pixels using getData, you get an array of usable pixel data cpu side.

\$\endgroup\$
1
\$\begingroup\$

I've seen something like this done with asynchronous PBO read backs. Since it's asynchronous, you don't have to worry about blocking:

Conventional glReadPixels() blocks the pipeline and waits until all pixel data are transferred. Then, it returns control to the application. On the contrary, glReadPixels() with PBO can schedule asynchronous DMA transfer and returns immediately without stall. Therefore, the application (CPU) can execute other process right away, while transferring data with DMA by OpenGL (GPU).

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.