Maya depth buffer plus OpenGL depth buffer

Question

I am trying to write an plugin for Maya, using -among other things- OpenGL. Everything works fine so far, but the final Maya scenario is not taking into account the OpenGL depth buffer (or OpenGL is not taking into account the Maya depth). Hence, OpenGL things always appear "over" the Maya native objects.

I looked at the Maya's documentation and I found out that you can get Maya's depth may through MStatus M3dView::readDepthMap function, which basically returns a float array with Maya's depth.

I am now in a situation where I have two float arrays, one with OpenGL depth's and another with Maya's depth, and I believe I should combine them both somehow to get a final render in Maya that would respect both depth maps.

So, the question is, is there anyway to "add" depth information into OpenGL's renderbuffer / depthbuffer? Or... I am trying a completely wrong approach and there's a much nicer and better way to do this?

Thanks a lot!

Answer 1

I am answering myself here, I hope it helps others when dealing with the same issue...

The trick here was to use MStatus M3dView::readDepthMap rather than MStatus M3dView::readDepthMap.

readDepthMap creates a 2D texture of the current Maya depth map. Note that you have to previously generate and bind this texture as follows

int mayaDepth;
glGenTextures(1, &mayaDepth);                           
glBindTexture(GL_TEXTURE_2D, mayaDepth);                    
GLint m_viewport[4];
glGetIntegerv( GL_VIEWPORT, m_viewport );   
glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_BORDER);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_BORDER);

glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT32, m_viewport[2], m_viewport[3], 0, GL_DEPTH_COMPONENT, GL_FLOAT, 0);
MStatus status = view.readBufferTo2dTexture(0, 0,m_viewport[2], m_viewport[3]);

glBindTexture(GL_TEXTURE_2D, 0);

This texture can be later on used in a GLSL fragment shader. Comparing the OpenGL depth accessible through gl_FragCoord.z and the depth information from Maya that can be passed to the GLSL shader through a normal sampler2D, you can decide whether to render your OpenGL stuff or Maya as follows

if(gl_FragCoord.z < maya_depth_value)
    gl_FragColor = col_acc;
else
    gl_FragColor = vec4(0,0,0,0);

Good luck! :)