Skewed: a rotating camera in a simple CPU-based voxel raycaster/raytracer

Question

TL;DR — in my first simple software voxel raycaster, I cannot get camera rotations to work, seemingly correct matrices notwithstanding. The result is skewed: like a flat rendering, correctly rotated, however distorted and without depth. (While axis-aligned ie. unrotated, depth and parallax are as expected.)

I'm trying to write a simple voxel raycaster as a learning exercise. This is purely CPU based for now until I figure out how things work exactly — fow now, OpenGL is just (ab)used to blit the generated bitmap to the screen as often as possible.

Now I have gotten to the point where a perspective-projection camera can move through the world and I can render (mostly, minus some artifacts that need investigation) perspective-correct 3-dimensional views of the "world", which is basically empty but contains a voxel cube of the Stanford Bunny.

So I have a camera that I can move up and down, strafe left and right and "walk forward/backward" — all axis-aligned so far, no camera rotations. Herein lies my problem.

Screenshot #1: correct depth when the camera is still strictly axis-aligned, ie. un-rotated.

Now I have for a few days been trying to get rotation to work. The basic logic and theory behind matrices and 3D rotations, in theory, is very clear to me. Yet I have only ever achieved a "2.5 rendering" when the camera rotates... fish-eyey, bit like in Google Streetview: even though I have a volumetric world representation, it seems —no matter what I try— like I would first create a rendering from the "front view", then rotate that flat rendering according to camera rotation. Needless to say, I'm by now aware that rotating rays is not particularly necessary and error-prone.

Still, in my most recent setup, with the most simplified raycast ray-position-and-direction algorithm possible, my rotation still produces the same fish-eyey flat-render-rotated style looks:

Screenshot #2: camera "rotated to the right by 39 degrees" — note how the blue-shaded left-hand side of the cube from screen #2 is not visible in this rotation, yet by now "it really should"!

Now of course I'm aware of this: in a simple axis-aligned-no-rotation-setup like I had in the beginning, the ray simply traverses in small steps the positive z-direction, diverging to the left or right and top or bottom only depending on pixel position and projection matrix. As I "rotate the camera to the right or left" — ie I rotate it around the Y-axis — those very steps should be simply transformed by the proper rotation matrix, right? So for forward-traversal the Z-step gets a bit smaller the more the cam rotates, offset by an "increase" in the X-step. Yet for the pixel-position-based horizontal+vertical-divergence, increasing fractions of the x-step need to be "added" to the z-step. Somehow, none of my many matrices that I experimented with, nor my experiments with matrix-less hardcoded verbose sin/cos calculations really get this part right.

Here's my basic per-ray pre-traversal algorithm — syntax in Go, but take it as pseudocode:

• fx and fy: pixel positions x and y
• rayPos: vec3 for the ray starting position in world-space (calculated as below)
• rayDir: vec3 for the xyz-steps to be added to rayPos in each step during ray traversal
• rayStep: a temporary vec3
• camPos: vec3 for the camera position in world space
• camRad: vec3 for camera rotation in radians
• pmat: typical perspective projection matrix

The algorithm / pseudocode:

// 1: rayPos is for now "this pixel, as a vector on the view plane in 3d, at The Origin"
rayPos.X, rayPos.Y, rayPos.Z = ((fx / width) - 0.5), ((fy / height) - 0.5), 0

// 2: rotate around Y axis depending on cam rotation. No prob since view plane still at Origin 0,0,0
rayPos.MultMat(num.NewDmat4RotationY(camRad.Y))

// 3: a temp vec3. planeDist is -0.15 or some such — fov-based dist of view plane from eye and also the non-normalized, "in axis-aligned world" traversal step size "forward into the screen"
rayStep.X, rayStep.Y, rayStep.Z = 0, 0, planeDist

// 4: rotate this too — 0,zstep should become some meaningful xzstep,xzstep
rayStep.MultMat(num.NewDmat4RotationY(CamRad.Y))

// set up direction vector from still-origin-based-ray-position-off-rotated-view-plane plus rotated-zstep-vector
rayDir.X, rayDir.Y, rayDir.Z = -rayPos.X - me.rayStep.X, -rayPos.Y, rayPos.Z + rayStep.Z

// perspective projection
rayDir.Normalize()
rayDir.MultMat(pmat)

// before traversal, the ray starting position has to be transformed from origin-relative to campos-relative
rayPos.Add(camPos)

I'm skipping the traversal and sampling parts — as per screenshot #1, those are "basically mostly correct" (though not pretty) — when axis-aligned / unrotated.

Answer 1

Your pseudo-code in Go language is a big mystery to me, so I`ll try to describe my approach :)

First of all you do not need any perspective-projective matrix. All you need to construct current view ray is:

• camera position
• camera up vector
• camera look at vector
• current pixel on the screen

Let`s assume that our free camera works right. Now we take our vectors and perform Gram–Schmidt process. This gives us orthogonal camera basis. In GLSL this step looks something like this:

vec3 cameraW = normalize(cameraPosition - cameraLookAt);
vec3 cameraU = normalize(cross(cameraUp, cameraW));
vec3 cameraV = cross(cameraW, cameraU);

Ok, we have basis. This basis is common for all pixels on the screen. Even more it`s invariant until we move or rotate camera. To compute view ray for current pixel one must do:

vec3 rayOrigin = cameraPosition;
vec3 rayDirection = normalize(p.x * cameraU + p.y * cameraV - cameraViewPlaneDistance * cameraW);

where p — is coord of your current pixel.

After this you have view ray. It`s a high time to do some raytracing or raymarching stuff.

Let`s go back to the free camera. To move forward or backward we simply shift camera position in the direction of target vector. To move left or right we extract right vector from camera rotation matrix and add it to the target and position vectors.

To rotate camera we use camera rotation matrix. First camera rotation matrix is identity matrix. We have three floats: yaw, pitch and roll. When user moves mouse this floats accumulates rotation angles. On update we must apply this rotations to our camera rotation matrix. Let`s take yaw which corresponds to rotations around up axis. First we extract up vector from rotation matrix and then construct new rotation matrix. After all we multiply camera rotation matrix and the new one. Here some C++ code that uses glm library:

vec3 up(cameraRotation[0][1], cameraRotation[1][1], cameraRotation[2][1]);
cameraRotation *= glm::rotate(mat4(1.0), yaw, up);

Hope this helps..