# Problem with Frustum Culling Right-Handed(LookAtRH GLM)

I am trying to do Frustum Culling based on Goemetric Approach, however don't matter how much i try, the frustrum rotate to opposite side, i tried to find a solution over all internet, but couldn't find any, really need help with this.

FOV is 35.0f (in Radians value) [fov = glm::radians(35.0f);] UP is (0.0f, 0.0f, 1.0f) Width : 1280 Height : 720 AspectRatio : 16:9

The API what i am using is DirectX 11, actually it works perfectly, the entire map is being rendered perfectly, if i disable frustum culling, however when i enable the frustum culling it's only rendering a part on the wrong direction.

i am doing this:

Calculate Camera View

UP = EVector3(0.0f, 0.0f, 1.0f);
view = glm::lookAtRH(CameraPosition, PlayerPosition, UP);


Calculate Frustum

EFloat tan = EMath::Tan(fov * 0.5f);
_NearWidth = _Near * tan;
_NearHeight = _NearWidth / _AspectRatio;
_FarWidth = _Far * tan;
_FarHeight = _FarWidth / _AspectRatio;

position = PlayerPosition; // Force frustum at player position for see where it points, if i put camera position when i move the camera the view frustum goes on different way, and it already rotates on opposite direction
direction = EMath::Normalize(PlayerPosition - _CameraPosition);
right = EMath::Normalize(EMath::Cross(direction, UP));
up = EMath::Cross(right, direction);
direction = -direction; // glm::LookAtRH does this inside, if i don't do this the frustum doesn't work, but it still going on opposite direction

EVector3 nearCenter = position + (direction * _Near);
EVector3 farCenter = position + (direction * _Far);

EVector3 upNear = up * _NearHeight;
EVector3 rightNear = right * _NearWidth;
EVector3 upFar = up * _FarHeight;
EVector3 rightFar = right * _FarWidth;

EVector3 nearTL = nearCenter + upNear - rightNear;
EVector3 nearTR = nearCenter + upNear + rightNear;
EVector3 nearBR = nearCenter - upNear + rightNear;
EVector3 nearBL = nearCenter - upNear - rightNear;

EVector3 farTL = farCenter + upFar - rightFar;
EVector3 farTR = farCenter + upFar + rightFar;
EVector3 farBR = farCenter - upFar + rightFar;
EVector3 farBL = farCenter - upFar - rightFar;

EVector3 nearTop =      nearTR  - nearTL;
EVector3 nearLeft =     nearBL  - nearTL;
EVector3 topLeft =      farTL   - nearTL;
EVector3 bottomRight =  nearBR  - farBR;
EVector3 farRight =     farTR   - farBR;
EVector3 farBottom =    farBL   - farBR;

_Planes[WP_NEAR]    = EMathPlane(EMath::Cross(nearTop, nearLeft), nearTL);
_Planes[WP_LEFT]    = EMathPlane(EMath::Cross(nearLeft, topLeft), nearTL);
_Planes[WP_TOP]     = EMathPlane(EMath::Cross(topLeft, nearTop), nearTL);
_Planes[WP_BOTTOM]  = EMathPlane(EMath::Cross(farBottom, bottomRight), farBR);
_Planes[WP_RIGHT]   = EMathPlane(EMath::Cross(bottomRight, farRight), farBR);
_Planes[WP_FAR]     = EMathPlane(EMath::Cross(farRight, farBottom), farBR);


EMathPlane:

EMathPlane(const EVector3 &normal, const EVector3 &point)
{
_Normal = EMath::Normalize(normal);
_Distance = -EMath::Dot(_Normal, point);
_AbsNormal = EMath::Abs(_Normal);
}


all functions of my math library are redirected to GLM, for some reason the frustrum points to different area, i am doing a quadtree frustum culling, and i have static boxes prepared for each test, for example:

// First bounding box to check
{0.0f, 0.0f, 0.0f, // min x, y, z
25600.0f, 25600.0f, 1000.0f} // max x, y, z

// Another bounding box
{12800.0f, 12800.0f, 0.0f, // min x, y, z
19200.0f, 19200.0f, 1000.0f} // max x, y, z


here is the (IsVisible) function:

    EVector3 center = (min + max) * 0.5f;
EVector3 half = (max - min) * 0.5f;
EVector3 absHalf = EMath::Abs(half);

EUInt32 isInside = 0x80000000;

for (EInt32 index = EMathFrustum::WP_NEAR;index < EMathFrustum::WP_MAX;index++)
{
EVector3 &normal = frustum._Planes[index]._Normal;
EFloat &dist = frustum._Planes[index]._Distance;
EVector3 tmpHalf = EVector3(EMath::ForceSign(absHalf.x, frustum._SignFlip[index]),
EMath::ForceSign(absHalf.y, frustum._SignFlip[index]),
EMath::ForceSign(absHalf.z, frustum._SignFlip[index]));

EVector3 corner = center - tmpHalf;
EFloat dot = EMath::Dot(corner, normal) + dist;

isInside &= RetrieveSign<EUInt32>(dot);
}

return isInside > 0;


it is based on Intel Software Culling sample, if the dot value is negative means the object is visible.

Resume of problem: It works(partially), the frustum is working well on the right player position(except when i put camera position, for some reason it goes to another way), however it rotates to wrong direction, if i rotate the camera to right, it rotates to left, i tried to find the problem in the code but i couldn't.

Really thanks everyone for your time, i am with this problem since 2 weeks, and tried everything for sure there is something what i am not doing well, but i don't know what it is, i am getting really stressed about it.

Well, at the end it was a mistake in my game data, after update the game data everything worked perfectly, but also did some changes, instead of keep geometric approach algorithm for get all six planes what I did is change to OpenGL Extract Planes from http://gamedevs.org/uploads/fast-extraction-viewing-frustum-planes-from-world-view-projection-matrix.pdf it works like charm.

If someone is stick with same problem as me, related to right-hand, just don't worry, use that pdf, it works for left-handed and right-handed, in my case I just used GLM library, and without transpose anything applied exactly what says that pdf, here is the code in case you need some example:

EMatrix4 matrix = MUVideoManager::_CurrentProjection * MUVideoManager::_CurrentView; // Don't transpose or inverse anything, just apply it exactly as you get after do lookAtRH and perspectiveFovRH :)
EVector4 tmp;
tmp = EVector4(matrix + matrix, matrix + matrix, matrix + matrix, matrix + matrix);
_Planes[WP_LEFT].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix - matrix, matrix - matrix, matrix - matrix, matrix - matrix);
_Planes[WP_RIGHT].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix - matrix, matrix - matrix, matrix - matrix, matrix - matrix);
_Planes[WP_TOP].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix + matrix, matrix + matrix, matrix + matrix, matrix + matrix);
_Planes[WP_BOTTOM].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix + matrix, matrix + matrix, matrix + matrix, matrix + matrix);
_Planes[WP_NEAR].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix - matrix, matrix - matrix, matrix - matrix, matrix - matrix);
_Planes[WP_FAR].SetPlane(EVector3(tmp), tmp.w, true);


Plane function:

EINLINE void SetPlane(const EVector3 &normal, const EFloat &distance, EBoolean normalize)
{
if (normalize)
{
EFloat length = EMath::Length(normal);
_Normal = normal / length;
_Distance = distance / length;
}
else
{
_Normal = normal;
_Distance = distance;
}

_AbsNormal = EMath::Abs(_Normal);
}


and Intersection Code:

EINLINE FrustumCullType Contains(EMathFrustum &frustum, EMathAABB &aabb)
{
EVector3 &min = aabb._Min;
EVector3 &max = aabb._Max;
EVector3 center = (min + max) * 0.5f;
EVector3 half = (max - min) * 0.5f;
EVector3 absHalf = EMath::Abs(half);

FrustumCullType cull = FR_INCLUDED;

for (EInt32 index = EMathFrustum::WP_BEGIN;index < EMathFrustum::WP_MAX;index++)
{
EVector3 &normal = frustum._Planes[index]._Normal;
EVector3 &absNormal = frustum._Planes[index]._AbsNormal;
EFloat &dist = frustum._Planes[index]._Distance;

EInt32 side = frustum.ComputeSide(index, center, half, absHalf);

if (side == EMathPlane::NEGATIVE_SIDE)
{
return FR_EXCLUDED;
}
else if (side == EMathPlane::NO_SIDE)
{
cull = FR_OVERLAP;
}
}

return cull;
}

EINLINE EFloat ComputeDistance(const EVector3 &v)
{
return EMath::Dot(_Normal, v) + _Distance;
}

EINLINE EInt32 ComputeSide(const EVector3 &center, const EVector3 &half, const EVector3 &absHalf)
{
// Calculate the distance between box centre and the plane
EFloat dist = ComputeDistance(center);

// Calculate the maximise allows absolute distance for
// the distance between box centre and plane
EFloat maxAbsDist = EMath::Dot(_AbsNormal, absHalf);

if(dist < -maxAbsDist)
{
return NEGATIVE_SIDE; // Outside! don't render
}

if (dist > +maxAbsDist)
{
return POSITIVE_SIDE; // Completely visible!
}

return NO_SIDE; // A part of it is visible
}


I hope it helps someone, wish you(who is reading) a nice day! :)