Streaming webcam via websocket performance issues (Code included)

Question

I finally figured out, how I can stream my webcam, but I'm getting not the best performance. In low resolutions it's working fine, but as soon as I turn the resolution up, the performance gets worse. I think Unity can't handle the decoding in that quality and speed so that the game is getting slowed down. Here is my Code :

Texture2D screenshot = new Texture2D(Screen.width, Screen.height, TextureFormat.ARGB32, false);
screenshot.ReadPixels(new Rect(0,0,Screen.width,Screen.height),0,0);
screenshot.Apply();
ws.Send(System.Convert.ToBase64String(screenshot.EncodeToJPG(quality)));

Hopefully there is something to improve.

Answer 1

There are 2 things that I think might make this run faster:

1) Limit the number of images you're attempting to send (e.g. the FPS) (Unity is single threaded, so everything you do has an impact on performance)

2) Make the resulting JPG smaller by scaling the image before Base64 encoding it

There are a number of ways to reduce the frame rate, but possibly the easiest is to use a Coroutine. If you also use WaitForSeconds then you'll be able to limit the frame rate to whatever you need. Something like the following will give you a stream of about 30FPS:

IEnumerator SendScreenshot()
{
    // Change this to affect frame rate
    yield return new WaitForSeconds(0.33f);

    Texture2D screenshot = new Texture2D(Screen.width, Screen.height, TextureFormat.ARGB32, false);
    screenshot.ReadPixels(new Rect(0,0,Screen.width,Screen.height),0,0);
    screenshot.Apply();
    ws.Send(System.Convert.ToBase64String(screenshot.EncodeToJPG(quality)));
}

You may find this still slows things down a little, as it will still be executing in the main Unity thread. If that's still not acceptable performance wise, you can try scaling the image down, before sending it over the network.

Hope this helps :-)