# Arbitrary projection matrix from 6 arbitrary frustum planes

A projection matrix represent a tranformation from the camera view space to the rendering system clip space. In other words, it defines the transormation between a 6-sided frustum to the clip cube.

The glOrtho and glFrustum use only 6 parameter to define such a projection, but impose several constraints on the frustum that will get projected to the clip cube: the near and far planes are parallel, the left and right planes intersect on a vertical line, and the top and bottom planes intersect on a horizontal lines, both lines being parallel to the near and far planes.

I'd like to lift these restrictions. So, from the definition of the 6 frustum side planes (in whatever representation you see fit), how can I compute a general projection matrix?

• I'm curious to know what you could ever want to use something like that for. – Nicol Bolas Mar 5 '12 at 2:24
• I'm curious as well. If you're just trying to create custom clipping why use a frustum to do it? – Nic Foster Mar 5 '12 at 15:55
• Since the projection matrix is a projection transformation to [-1, 1]³ (a 6-sided figure), one of the “representations of the 6 side planes” is the projection matrix itself, so you should be more specific about what kind of input representation you want to start with. – Kevin Reid Mar 5 '12 at 17:18
• @Nicol: One of the uses is to compute warped shadow maps. I want the projection along Z to be orthographic (in the light direction), but the projection on x and/or y to be in perspective, so that the shadow map volume closely matches the camera frustum. – Doub Mar 7 '12 at 14:52
• @Nic: I'm not trying to do custom clipping, but rather to warp some arbitrary hexahedron to a shadow map clip space, to reduce wasted texels on parts of the scene I don't care, ie. whatever is not in the camera frustum (or between that frustum and the light source). – Doub Mar 7 '12 at 14:53

I'm going to go ahead and be the one to say that it's not possible to condense arbitrary contortions of space into a standard projection matrix which, by its nature, works with regular sides without being able to rotate them. Using your shadow map example, there's a reason the Digital/VFX industry hasn't done this already =)

You need a deeper set of equations than what a 4x4 matrix can represent. I'd suggest looking at implementing either a deformation matrix on top of the base projection frustum or a cage mesh deform.

deformation is complex

Blender visualization of a caged mesh deform

• You're totally right. I did some experimentations, and it appears I can control the frustum with only a center and three projections points, but two of them have to be at infinity (and make associated frustum edges parallel). But the real limitation was the distribution of space in the axis toward the point not at infinity. Resolution falls down too quickly, so a projection matrix is unusable as-is. Since I'm using shaders anyway in the normal and shadow rendering, I'll find some other projection method. – Doub Jun 5 '12 at 18:26
• The whole reason for Cascading Shadow Maps' existence is to try and cope with the distribution by brute force. You may find that progressively refining CSMs is faster than a general equation because they can use accelerated functions, though they have visual problems too. – Patrick Hughes Jun 5 '12 at 18:46

What you are asking is possible, but perhaps not in the way you had hoped (at least, to the best of my knowledge). Here is the problem, and a potential solution:

First of all, you cannot define six arbitrary planes with a single projection matrix, because the matrix really has nothing to do with planes. The matrix simply moves a given point around based on some matrix multiplication -- it transforms it, rotates it, and performs a perspective projection on it (expanding or shrinking its distance from the focal point). Thus, a projection matrix behaves in a more or less "symmetrical" manner - it is only capable of defining rectangular prisms, where one face of that prism is (potentially) larger than the other, along the axis of projection (not sure what you would call that shape?).

Now, here is what you could perhaps do to achieve what you want to do:

Rather than focusing on the 6 planes, focus on the eight corners of the view frustum. Give each corner its own projection matrix. Then, in the vertex shader, compute the new point eight times, one for each matrix. Then do several linear interpolations. Lets say the corners are set up like this:

Front:
A B
C D

Back:
E F
G H


Based on the x position of the point (scaled from 0 to 1), do a linear interpolation between A and B, then C and D, then E and F, then G and H. (Lets represent this with a "." character. You will now have this:

A.B C.D E.F G.H

Now take the y value (0 to 1) and do the following linear interpolation:

(A.B).(C.D)
(E.F).(G.H)


And last but not least:

((A.B).(C.D)).((E.F).(G.H)).


This will give you one final point. The interesting thing about this is that you could do some really crazy transformations, stuff that should not be physically possible. If you tied in some animation you could create really interesting effects, even though this is not the purpose of what you want this for.

This is more of a series of suggestions than an answer, but:

To my knowledge a 4x4 matrix is capable of representing this type of transformation, but the math escapes me.

Some things to consider

Even if that's not right I hope it leads you in the right direction!