You cannot do this directly. Both OpenGL and Direct3D operate under the assumption that they are the only things writing to the window.
Furthermore, most of their actual rendering is done in in-memory back buffers and not directly onto the presentation window. It's only once all rendering commands are complete for a particular frame that the back buffer contents is transferred, wholesale, to the front buffer (the presentation window). This transfer is done without regard to what currently exists in that front buffer, and so D3D and GL are totally ignorant of the render results of the other.
There are two ways to achieve what you want, and you can choose between them based on whether or not you want the respective scenes to ever overlap.
In the simple case, where you want only the D3D stuff on the left half of the screen and the OpenGL stuff on the right (for example), you can simply adjust the viewport of both APIs accordingly (using
glViewport in OpenGL or
RSSetViewports in D3D).
However, if you want the scenes to potentially overlap (as it sort of sounds like you do), you have to instead render your scenes to textures in both APIs (potentially outdated OpenGL example, D3D example). Then you'll need to recover the bits of those textures manually and composite them into a final scene.
You can do this composition and final presentation via a third graphics API (likely GDI), if you wish, but that would probably be unpleasantly inefficient. Instead, I would suggest you do the final composition with D3D or GL itself again. You can pick either, but for the sake of demonstration I'll assume you prefer GL.
That means you will, every frame:
- perform your OpenGL rendering into a texture
- perform your D3D rendering into a different texture
- copy the contents of the D3D texture into CPU memory and from there into GPU memory of a second OpenGL texture.
- render a fullscreen quad in OpenGL using both your OpenGL textures and a shader or with appropriate blend mode settings enabled to handle composition however you'd like