In my first attempt, I created a new projection matrix for the smaller window, and then tried to rotate the frustum using LookAt, this almost worked but had some distortion when tilting up or down.
However, using a different approach it turns out the problem was quite easy to solve in DirectX. The BoundingFrustum class that I used (while It can be constructed from a projection matrix) actually consists of a nearplane, farplane, and 4 different sloops. By multiplying the 4 sloops with the ratio between the sub window and parent window, I get a working subfrustum.
Here is a code sample in DirectX with some functionality hidden, but it should not be to hard to follow.
BoundingFrustum getSubFrustum( FloatRectangle& window, FloatRectangle& sub_window )
// Construct bounding frustum from camera projection
f.Origin = m_position;
f.Orientation = m_rotation;
// Translates the rectangle from "0 .. 2x", to "-x .. x"
// making center 0,0 in the new coordinate system
Float2 center = window.center();
// Adjust bounding slopes to match sub window
float left = sub_window.position.x/window.position.x;
float right = (sub_window.position.x + sub_window.size.x)/(window.position.x + window.size.x);
float bottom = sub_window.position.y/window.position.y;
float top = (sub_window.position.y + sub_window.size.y)/(window.position.y + window.size.y);
f.LeftSlope *= left;
f.RightSlope *= right;
f.TopSlope *= top;
f.BottomSlope *= bottom;
- It works! However, It feels like I have not understood the problem. I would still love to hear a solution using the projection matrix only, so it would be possible to use the projection while actually drawing.