0
\$\begingroup\$

I finally figured out, how I can stream my webcam, but I'm getting not the best performance. In low resolutions it's working fine, but as soon as I turn the resolution up, the performance gets worse. I think Unity can't handle the decoding in that quality and speed so that the game is getting slowed down. Here is my Code :

Texture2D screenshot = new Texture2D(Screen.width, Screen.height, TextureFormat.ARGB32, false);
screenshot.ReadPixels(new Rect(0,0,Screen.width,Screen.height),0,0);
screenshot.Apply();
ws.Send(System.Convert.ToBase64String(screenshot.EncodeToJPG(quality)));

Hopefully there is something to improve.

\$\endgroup\$
  • \$\begingroup\$ That's no miracle. You aren't encoding a proper video-stream, you are encoding it frame-by-frame to jpeg. That results in far more bandwidth than a proper video codec would create. Try creating a proper video stream (depending on the webcam there might be an API which exposes the video input as one) and then use a movie texture on the Unity side. \$\endgroup\$ – Philipp Jun 9 '16 at 11:35
  • \$\begingroup\$ Makes sense to me, I got the webcam on a Texture, which is created by Vuforia... And I don't know how to stream a video via websocket, like I'm doing now with simple Base64. So I'm kinda lost in this video decoding section :/ \$\endgroup\$ – TobiasW Jun 9 '16 at 12:41
  • \$\begingroup\$ If you're making this for Web, you might want to look into WebRTC as a possible solution. There's a Unity package here that might help: assetstore.unity3d.com/en/#!/content/47846 and also a package mentioned here: forum.unity3d.com/threads/… \$\endgroup\$ – hobnob Jun 9 '16 at 13:27
  • \$\begingroup\$ It's cross-platform from Unity-Android to non-Unity-WebInterface via WebSocket. But I would be happy if it would work Unity to Unity via whatever :D \$\endgroup\$ – TobiasW Jun 9 '16 at 13:29
1
\$\begingroup\$

There are 2 things that I think might make this run faster:

1) Limit the number of images you're attempting to send (e.g. the FPS) (Unity is single threaded, so everything you do has an impact on performance)

2) Make the resulting JPG smaller by scaling the image before Base64 encoding it

There are a number of ways to reduce the frame rate, but possibly the easiest is to use a Coroutine. If you also use WaitForSeconds then you'll be able to limit the frame rate to whatever you need. Something like the following will give you a stream of about 30FPS:

IEnumerator SendScreenshot()
{
    // Change this to affect frame rate
    yield return new WaitForSeconds(0.33f);

    Texture2D screenshot = new Texture2D(Screen.width, Screen.height, TextureFormat.ARGB32, false);
    screenshot.ReadPixels(new Rect(0,0,Screen.width,Screen.height),0,0);
    screenshot.Apply();
    ws.Send(System.Convert.ToBase64String(screenshot.EncodeToJPG(quality)));
}

You may find this still slows things down a little, as it will still be executing in the main Unity thread. If that's still not acceptable performance wise, you can try scaling the image down, before sending it over the network.

Hope this helps :-)

\$\endgroup\$
  • \$\begingroup\$ Thanks! I will try that, I already got it in a coroutine but in a slightly other way : yield return new WaitForEndOfFrame(); \$\endgroup\$ – TobiasW Jun 9 '16 at 11:50
  • \$\begingroup\$ But I actually thought, that a coroutine would run in an extra thread? \$\endgroup\$ – TobiasW Jun 9 '16 at 11:55
  • \$\begingroup\$ As far as I understand Coroutines are still executed on the main thread (so you can instantiate game objects etc, which isn't possible outside the main thread). In terms of execution order, they happen after the Update call: docs.unity3d.com/Manual/ExecutionOrder.html \$\endgroup\$ – hobnob Jun 9 '16 at 12:02
  • \$\begingroup\$ Okay thanks for your help, the frame limit did improve the performance obviously. But I guess it should somehow be possible to stream in a better quality with some video encoding like @Philipp said already but I don't know where to start with that... \$\endgroup\$ – TobiasW Jun 9 '16 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.