4
\$\begingroup\$

I'm writing an application for the Oculus Rift using the SharpOVR library to access HMD position data. I decided it is reasonable to use SharpDX to also output webcamera streams.

Unfortunately, all SharpDX documentation is basically a copy paste from C++ documention and I only know basic C++. C# uses a different style, class names, etc. The MediaFoundation samples are also C++-only.

I'm trying to write a pretty simple part that renders 2 videos from 2 webcams. I've managed to get webcameras list with this code, but I'm failing to init one— it currently fails at source.Start(descriptor, null, time);

According to the exception, I need to bind all Stream events first. Or I am missing something I need to init?

Can one help with further actions from camera start till outputting to monitor? I know the code shouldn't be too hard, problem is adapting it to C#.

\$\endgroup\$
4
\$\begingroup\$

Sorry for not posting a complete solution, but I have done something relatively similar to what you're doing. You may be able to adapt this idea into your program.

What I did was output a video file into a Direct3D 9 (not 11) texture for rendering. I'm pretty sure it was possible to select a webcam as a video source instead of a video file, so in general this could be useful to you.

Notice that these are snippets from actual code, so there may be methods and other things that may not be included in these snippets. This is not meant for you to copy and paste, but so you can understand the general idea of what you have to do.

In general terms, the library you're interested in is SharpDX.MediaFoundation.

Reading a video source will give you a bunch of Sample elements, so you may want to create a Queue<Sample>, so on one side you're reading from the video source into the queue, and on the other side, you're converting those samples to textures and outputting them to your framebuffer.

Then, you create a MediaFactory and a SourceReader. Don't ask me what those parameters are, I wrote this a while ago, and I don't remember what each parameter is supposed to be (the docs are your friends):

var mediaAttributes = new MediaAttributes(IntPtr.Zero);
var SourceReader = new SourceReader(IntPtr.Zero);

MediaFactory.CreateAttributes(mediaAttributes, 16);
mediaAttributes.Set(SourceReaderAttributeKeys.EnableVideoProcessing, 1);
MediaFactory.CreateSourceReaderFromURL(url, mediaAttributes, SourceReader);
SourceReader.SetStreamSelection(SourceReaderIndex.AllStreams, false);

Right there where I'm doing CreateSourceReaderFromURL is where you want to change it to read from your webcam. After that, the rest of the program should be pretty similar for webcams and video files.

Now that you have the reader, you loop through the available streams. You should have one video stream, and one or more audio streams. At this point, we're just interested in the video stream:

int streamIndex = 0;
while (true)
{
    var nativeMediaType = SourceReader.GetNativeMediaType(streamIndex, 0);
    var currentMediaType = SourceReader.GetCurrentMediaType(streamIndex);
    var outputMediaType = new MediaType();

    SourceReader.SetStreamSelection(streamIndex, true);

    if (nativeMediaType.MajorType == MediaTypeGuids.Video)
    {
        VideoStreamIndex = streamIndex;

        VideoSubType = currentMediaType.Get<Guid>(MediaTypeAttributeKeys.Subtype);
        UnpackLong(currentMediaType.Get(MediaTypeAttributeKeys.FrameSize), out VideoWidth, out VideoHeight);
        UnpackLong(currentMediaType.Get(MediaTypeAttributeKeys.FrameRate), out VideoFrameRateNumerator, out VideoFrameRateDenominator);
        UnpackLong(currentMediaType.Get(MediaTypeAttributeKeys.PixelAspectRatio), out VideoAspectRatioNumerator, out VideoAspectRatioDenominator);
        VideoInterlaceMode = (VideoInterlaceMode)(currentMediaType.Get(MediaTypeAttributeKeys.InterlaceMode));

        MediaFactory.CreateMediaType(outputMediaType);
        outputMediaType.Set(MediaTypeAttributeKeys.MajorType, MediaTypeGuids.Video);
        outputMediaType.Set(MediaTypeAttributeKeys.Subtype, new Guid(22, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71));
        outputMediaType.Set(MediaTypeAttributeKeys.FrameSize.Guid, PackLong(VideoWidth, VideoHeight));
        outputMediaType.Set(MediaTypeAttributeKeys.FrameRate.Guid, PackLong(VideoFrameRateNumerator, VideoFrameRateDenominator));
        outputMediaType.Set(MediaTypeAttributeKeys.InterlaceMode, (int)VideoInterlaceMode.Progressive);
        outputMediaType.Set(MediaTypeAttributeKeys.PixelAspectRatio, PackLong(1, 1));

        SourceReader.SetCurrentMediaType(streamIndex, outputMediaType);

        VideoReady = true;
    }
    outputMediaType.Dispose();

    streamIndex++;
}

UnpackLong is simply a function I made to get two ints out of one long. The Guid there is the format of the video. I think I got this for H.264, so you're going to have to look online what is the Guid for the native format of your webcam.

And then it's just a loop in which you fill your queue, and then consume from the queue. To read one video frame from the queue, you do something like this:

int readStreamIndex;
SourceReaderFlags readFlags;
Sample sample;

sample = SourceReader.ReadSample(SourceReaderIndex.AnyStream, SourceReaderControlFlags.None, out readStreamIndex, out readFlags, out timestamp);
if (sample != null)
{
    if (readStreamIndex == VideoStreamIndex)
        VideoSampleBuffer.Enqueue(sample);
    else // Other stream
        sample.Dispose();
}
if ((readFlags & SourceReaderFlags.Endofstream) == SourceReaderFlags.Endofstream)
    EndOfStream = true;

Now, on the rendering side, first you have to create a texture in which you will write your frames:

VideoTexture = new Texture(device, videoWidth, videoHeight, 1, Usage.Dynamic, Format.X8R8G8B8, Pool.Default);
var lockData = VideoTexture.LockRectangle(0, LockFlags.None);
byte[] data = new byte[videoWidth * videoHeight * 4];
for (int k = 0; k < videoWidth * videoHeight * 4; )
{
    // Fill with black
    data[k++] = 0x00;
    data[k++] = 0x00;
    data[k++] = 0x00;
    data[k++] = 0xff;
}
Marshal.Copy(data, 0, lockData.DataPointer, videoWidth * videoHeight * 4);
VideoTexture.UnlockRectangle(0);

Now, for each frame we want to convert, we simply lock the source buffer, copy the bytes into the texture and we're done.

var sourceBuffer = sample.ConvertToContiguousBuffer();
var sourcePointer = sourceBuffer.Lock(out maxLength, out currentLength);
var destRect = texture.LockRectangle(0, SharpDX.Direct3D9.LockFlags.None);
var destPointer = destRect.DataPointer;

CopyMemory(destPointer, sourcePointer, (uint)(reader.VideoWidth * reader.VideoHeight * 4));
texture.UnlockRectangle(0);

sourceBuffer.Unlock();
sourceBuffer.Dispose();

This last step is pretty inefficient, since even though the video decoding is probably done in the GPU, and the texture is also in the GPU, we're bringing both these buffers to RAM to copy them.

There is a library called DXVA2, which allows you to convert Media Framework frames into Direct3D9 textures with a single pass, without going through RAM. This is pretty cool, but I could not get it to work. Additionally, when I wrote my video renderer, DXVA2 was not supported on SharpDX, so there was a lot of interop stuff in there. If you're interested, you may want to start reading up on DXVA2CreateDirect3DDeviceManager9

On a related note, I've gotten accelerated video rendering working on iOS using xamarin. The idea is exactly the same, although the API is certainly quite different.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.