(EDIT: The library in progress mentioned at the end of this
post now has a project here - https://smallmedialibrary.codeplex.com/)
I have spent a few days looking at this and so I will share what I know (and as usual leave the Q. open for future additions).
As it turns out, doing video is really hard. It also turns out the only thing harder than doing video is navigating the licenses for the various libraries that do do video!
FFMpeg can play a number of roles in the solution. It is a suite but it modularised and there are different components that may be of assistance to game developers:
- libavcodec is a library providing access to FFMpeg's huge array of decoders. This can be utilised directly from a library but it will not 'unwrap' the container of whatever format the video is provided in.
- ffdshow is a DirectShow filter that can be installed independently of any solution and will be utilised by DirectShow to decode a range of types - i.e. you could I believe create a DirectShow library, install ffdshow and have the ability to decode all FFMpeg supported formats automatically, without needing to interact with FFMpeg directly.
- GStreamer also has an FFMpeg plugin, which brings me to...
GStreamer - this is an open source multimedia toolkit. It is a framework making use of graphs of filters and plugins, similar to DirectShow but from what I understand cleaner and simpler, which can only be a good thing! Unfortunately there isn't a repository of Windows binaries and building it will require MingW or the like.
That said, there are .NET bindings, which look very promising! What is not clear though is the extent of dependencies for solutions making use of GStreamer.
LibVLC has a number of .NET wrappers and none are really much help except for http://www.helyar.net/2009/libvlc-media-player-in-c/ - which is not a wrapper - but a blog bost demonstrating how easy it is to utilise libvlc via PInvoke. (Which is, truly, easier than learning to use a wrapper*)
LibVLC consists a large number of specialsed functions that do highly specific things. It does though utilise a graph of objects, such as a player object and a media object making it nice to work with. I have read posts from people complaining about the lack of 'design' in the API and that it should be remodelled to be generic.
I started writing a library with it and found it fine and easy to work with, and the documentation helpful.
It operates at a higher level than the previous technologies though and it becomes apparent that if one of the specialised functions does not do what you want you are in for a struggle; when it does however it works wonderfully and I can see it being very productive.
(*one point about wrappers - until recently LibVLC was GPL, not LGPL and so many of the wrappers use contrary licences)
To add to the list of my incorrect assumptions, DirectShow also seems a good bet. I was put off by the Direct in DirectShow, as it suggested interop with DirectX was a requirement but it is not. The DX SDK is completely seperate so DirectShow libraries could be used in OpenGL games to render video in the way as suggested in the question.
DirectShow is being superceeded by the Windows Media Foundation (for Vista, 7 and up only) but I have read it is not as feature complete as DirectShow. Neither DirectShow or WMF have supported .NET bindings however there are third-party 'wrappers' for each. I say ''wrappers'' beacuse they are mostly thin layers for PInvoking the DS/WMF functions.
The official line from MS is that .NET bindings are not going to be produced because of performance concerns (neither DS or WMF was designed with a managed environment in mind)
I have decided to write my own library in C++/CLI using DirectShow.
I opted for DirectShow because there is a huge amount of material available for it (its been around for 6 years) - both in terms of documentation and codecs (and if those fail, ffdshow is a "50x encodings with one stone shot") - and using it without a wrapper means C&P examples. It also has the most permissive license of all of the above. Further its graph/filter based design means it doesn't need to be hacked to read video from a memory stream, or render to a block of memory like LibVLC - and the principles are the same as used by GStreamer for a Linux/Mac/Symbian version of the library should one be needed.
(@lzcd +1 I did look at XNA Video Player and it looks good but I think it uses DirectShow to draw directly to a DX9 surface, and so would be fairly inefficient if not using DX9 as it would require reading from the GPU! There is also XNA DirectShow Player which appears to do the same thing but still looks really nice so I will check it out esp. as they are doing pretty much what I want to, only for XNA.)