I'm developing a software that aims to combine a keyed video with a 3D rendered one, synchronously frame by frame.

Each frame of video should be combined with a timely corresponding frame of 3D picture using a matching camera position parameter.

How can I perform this rendering and ensure it stays in synch?

  • \$\begingroup\$ Hi moslem yazdanpanah, and welcome to GameDev.StackExchange. We don't give tech/engine recommendations here, because software changes so fast and every developer has their own preferences for how they like to work, so there's often no "correct" answer. I've edited your question to remove this request. With a little work, you should be able to achieve this effect in any engine or framework you ultimately select. Can you tell us more about what particular problem you need to solve? Is it about detecting precisely which frame of video is being shown, associating camera info with each frame, etc? \$\endgroup\$ – DMGregory Apr 29 '18 at 11:28

One thing you'll need to contend with is making sure both the 2D video and the 3D scene produce a frame in a timely manner. Typically video is kept in compressed form on disk/network and is read a little at a time to decompress each frame and then display it. Likewise, 3D scenes may have what's currently visible in memory, and then load more geometry/textures/etc. from disk/network/etc. when needed. This could cause I/O contention if both geometry and video need to be loaded at the same time. If you can keep much of it in memory, you'll reduce I/O contention between the 2 tasks.

Additionally, video will need to be decompressed usually and that takes up CPU time. If you know your target machine, you may be able to optimize your threading strategy to have 1 (or a few) thread(s) doing video decompression while another thread (or threads) deals with your 3D needs. Furthermore, different codecs have different amounts of CPU requirements. You can experiment with different compression schemes to see which gives you a good trade-off of compressed size vs. decompression cost. Usually more highly compressed video will reduce IO overhead as fewer bytes need to be read from disk or network, but high quality, highly compressed video may be more expensive to decompress.

The best thing to do is come up with an implementation and profile it. Without having any specifics on type of video, complexity of 3D scene, etc., nobody can even begin to guess where your bottlenecks will be. Running real-world tests and measuring is your best bet for figuring this out.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.