6
\$\begingroup\$

In my project I would like to be able to play back video on surfaces in the world. I intend to do this by having the video frames rendered to a block of memory, then use this to update a texture each frame. Everything is in place - except for the part that actually gets the video.

I have looked on Google and found that the video library world is very expansive (and geared towards video processing), and am having trouble finding a suitable one.

FFMpeg is very comprehensive, but is an entire suite and would take a good amount of work to integrate. So far the most promising library I've found is the one based on the VLC player libraries - by virtue of it using the same resources as VLC Player it is known to be very capable; it also renders to blocks of memory, but the API (at least of the one on Github) is more of a port of the C++ API rather than a managed wrapper.

The 'solution' can be any wrapper/API/library, but with characteristics that make it suitable for use in a rendering engine, namely:

  1. Renders the video frame data to memory, so it can be picked up and passed to a texture on the GPU easily.
  2. Super simple - all that is needed is a way to load, jump and render a frame programatically - ideally it would use the systems codecs and not require an assortment of plugins.
  3. Permissive license (LGPL or more free-er)
  4. .NET bindings at least; all the better if it is natively managed

Can anyone suggest a lightweight, (.NET) library, that can take a video file, and spit out some frames into a byte[]?

\$\endgroup\$
1
  • 2
    \$\begingroup\$ It may break the 'lightweight' criteria, depending on your installation requirements, but XNA and its video player component may be worth investigating. \$\endgroup\$
    – lzcd
    Apr 29, 2012 at 23:41

2 Answers 2

6
\$\begingroup\$

(EDIT: The library in progress mentioned at the end of this post now has a project here - https://smallmedialibrary.codeplex.com/)

I have spent a few days looking at this and so I will share what I know (and as usual leave the Q. open for future additions).

As it turns out, doing video is really hard. It also turns out the only thing harder than doing video is navigating the licenses for the various libraries that do do video!

FFMpeg can play a number of roles in the solution. It is a suite but it modularised and there are different components that may be of assistance to game developers:

  1. libavcodec is a library providing access to FFMpeg's huge array of decoders. This can be utilised directly from a library but it will not 'unwrap' the container of whatever format the video is provided in.
  2. ffdshow is a DirectShow filter that can be installed independently of any solution and will be utilised by DirectShow to decode a range of types - i.e. you could I believe create a DirectShow library, install ffdshow and have the ability to decode all FFMpeg supported formats automatically, without needing to interact with FFMpeg directly.
  3. GStreamer also has an FFMpeg plugin, which brings me to...

GStreamer - this is an open source multimedia toolkit. It is a framework making use of graphs of filters and plugins, similar to DirectShow but from what I understand cleaner and simpler, which can only be a good thing! Unfortunately there isn't a repository of Windows binaries and building it will require MingW or the like.
That said, there are .NET bindings, which look very promising! What is not clear though is the extent of dependencies for solutions making use of GStreamer.

LibVLC has a number of .NET wrappers and none are really much help except for http://www.helyar.net/2009/libvlc-media-player-in-c/ - which is not a wrapper - but a blog bost demonstrating how easy it is to utilise libvlc via PInvoke. (Which is, truly, easier than learning to use a wrapper*)
LibVLC consists a large number of specialsed functions that do highly specific things. It does though utilise a graph of objects, such as a player object and a media object making it nice to work with. I have read posts from people complaining about the lack of 'design' in the API and that it should be remodelled to be generic.
I started writing a library with it and found it fine and easy to work with, and the documentation helpful. It operates at a higher level than the previous technologies though and it becomes apparent that if one of the specialised functions does not do what you want you are in for a struggle; when it does however it works wonderfully and I can see it being very productive.
(*one point about wrappers - until recently LibVLC was GPL, not LGPL and so many of the wrappers use contrary licences)

To add to the list of my incorrect assumptions, DirectShow also seems a good bet. I was put off by the Direct in DirectShow, as it suggested interop with DirectX was a requirement but it is not. The DX SDK is completely seperate so DirectShow libraries could be used in OpenGL games to render video in the way as suggested in the question.
DirectShow is being superceeded by the Windows Media Foundation (for Vista, 7 and up only) but I have read it is not as feature complete as DirectShow. Neither DirectShow or WMF have supported .NET bindings however there are third-party 'wrappers' for each. I say ''wrappers'' beacuse they are mostly thin layers for PInvoking the DS/WMF functions.
The official line from MS is that .NET bindings are not going to be produced because of performance concerns (neither DS or WMF was designed with a managed environment in mind)

I have decided to write my own library in C++/CLI using DirectShow.
I opted for DirectShow because there is a huge amount of material available for it (its been around for 6 years) - both in terms of documentation and codecs (and if those fail, ffdshow is a "50x encodings with one stone shot") - and using it without a wrapper means C&P examples. It also has the most permissive license of all of the above. Further its graph/filter based design means it doesn't need to be hacked to read video from a memory stream, or render to a block of memory like LibVLC - and the principles are the same as used by GStreamer for a Linux/Mac/Symbian version of the library should one be needed.

(@lzcd +1 I did look at XNA Video Player and it looks good but I think it uses DirectShow to draw directly to a DX9 surface, and so would be fairly inefficient if not using DX9 as it would require reading from the GPU! There is also XNA DirectShow Player which appears to do the same thing but still looks really nice so I will check it out esp. as they are doing pretty much what I want to, only for XNA.)

\$\endgroup\$
-1
\$\begingroup\$

While this is a bit over the top, XNA does have built in video playback. And, it will return the current frame in the form of a texture

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .