Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

I'm trying to make an augmented reality game for advertisement, using the kinect sdk 1.6 and xna 4.0. I need to use the color image format rgb 1280X960, in spite of the fact that it runs at 12fps, because the quality of the image is important for this project. The problem is that when I call the function to build a color image frame, the fps of the game drops a lot. When I use only skeleton frame information to build a game scene, it runs at 50/60 fps, but when I call the function for the color frame the frames drop to 10/14 fps. The purpose of this function is the transformation of the stream to bgra format. If I comment out this loop, I get 30/40 fps. Is there any other way to make this more efficient?

public Texture2D Video() 
{
   try 
   {
      if (video == null) 
        video = new Texture2D(gd, Sensor.ColorStream.FrameWidth, Sensor.ColorStream.FrameHeight);

      ColorImageFrame framecolor = Sensor.ColorStream.OpenNextFrame(100);

      if (framecolor != null)
      {
        byte[] bgraPixelData = new byte[framecolor.PixelDataLength];
        framecolor.CopyPixelDataTo(bgraPixelData);

        //Convert RGBA to BGRA
        byte[] colorbuffer = new byte[framecolor.PixelDataLength];

        for (int i = 0; i < bgraPixelData.Length; i += 4)
        {
          colorbuffer[i] = bgraPixelData[i + 2];
          colorbuffer[i + 1] = bgraPixelData[i + 1];
          colorbuffer[i + 2] = bgraPixelData[i];
          colorbuffer[i + 3] = (byte)255; //The video comes with 0 alpha so it is transparent
        }

        video.SetData(colorbuffer);
      }

    }
    catch
    {
      video = null;
    }

    return video;
  }

-----How do i ressolve it?

I finally decide to use a shader to make the gpu deal with the process of tranform between formats, as it was suggested by @Seth Battin. this is the shader:

sampler sprite : register(s0);

/*
 * Convert from BGRX to RGBA
 */
float4 BGR2RGB(float2 texCoord : TEXCOORD0) : COLOR0
{
    float4 tex = tex2D(sprite, texCoord);
    return float4(tex.b, tex.g, tex.r, 1.0);
}

technique KinectVideo
{
    pass KinectVideo
    {
        PixelShader = compile ps_2_0 BGR2RGB();
    }
}

and the code i used before, as it no longer need to make the transform, now looks like this:

public RenderTarget2D Video()
        {
            try
            {
                if (Sensor != null)
                {
                    if (Sensor.ColorStream != null)
                    {
                        using (var frame = Sensor.ColorStream.OpenNextFrame(0))
                        {
                            if (frame != null)
                            {
                                if (buffer == null || buffer.Length != frame.PixelDataLength)
                                {
                                    buffer = new byte[frame.PixelDataLength];
                                }

                                frame.CopyPixelDataTo(buffer);
                                //GraphicsDevice.Textures[0] = null;

                                videoframe.SetData<byte>(buffer);

                            }
                        }
                    }
                }
            }
            catch
            {
            }
            return videoframe;
        }

I know that the try/catch its expensive, but now i have a good rate of fps, and its woorking fine. After that, i just apply the shader in the draw call, like this:

                sb.Begin(SpriteSortMode.Texture, null, null, null, null, kinectcolorvisualizer);
                sb.Draw(video, pantalla, null, Color.White, 0, Vector2.Zero, SpriteEffects.None, 1);
                sb.End();

where "sb" is spritebatch, "kinectcolorvisualizer" is the shader and "video" is the RenderTarget2D returned by the video function

Thanks to @Seth Battin and to @Mustafa Karakuş for their help.

share|improve this question

2 Answers 2

up vote 4 down vote accepted
  1. Use no try/catch in performance sensitive loops. You need to track down the source of what could cause an error, and prevent it from happening. Throwing and catching an exception is an extremely expensive operation.
  2. You are creating a Texture2D, and populating it from a stream. You could probably save some memory by not creating a new one for every call to your function.
  3. You grab a set of data from a stream, and then swizzle it on the CPU. Why not do that in a shader, so the GPU can do what it does best? It could correct for the alpha problem, too.
share|improve this answer
2  
Also allocating two new and potentially large buffers in each, the same would go for these that goes with the Texture2D, preallocate as much as possible. –  Evan Apr 27 '13 at 6:48
    
thanks, i will try all that. About the conversion between formats with a shader... i will need to study that in detail, i was already seeing something, but i dont stop in that. in the examples of the kinect sdk for xna, they use shaders to do that, so i suppose i can use that when i reallize how. all that must work. thanks again for the help. –  pablopp83 Apr 27 '13 at 14:52

Because of SetData, performance suffers a lot. SetData makes a transfer between GPU's ram and CPU's ram. Imagine doing this operation every frame. This causes performance issues.

I recommend using Task for seperating AllFramesReady method (in which you call your Video method). The images from kinect sensor may still be a bit laggy, but since they are not on the same thread anymore, game will not suffer that much performance issues. Like this:

Task.Factory.StartNew(() => checkFrameEvent());

.
.
.

private void checkFrameEvent()
{
    kinect.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(kinect_AllFramesReady);
}

I don't need to mention that the first line must be before starting kinect sensor.

I hope this helps.

share|improve this answer
    
thanks, i already resolve the issue making a shader for the convert process, and i gain a performance of 45fps. it is enough for the game, but if i can combine it with this, maybe it work better. i am going to give it a try. thanks again. –  pablopp83 Apr 30 '13 at 15:19
    
Glad if i could help. Can you share some code about how you did that process? –  Mustafa Karakuş May 7 '13 at 6:08
    
sure, i update the question to include the solution. –  pablopp83 May 7 '13 at 16:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.