Problem with Frustum Culling Right-Handed(LookAtRH GLM)

Question

I am trying to do Frustum Culling based on Goemetric Approach, however don't matter how much i try, the frustrum rotate to opposite side, i tried to find a solution over all internet, but couldn't find any, really need help with this.

FOV is 35.0f (in Radians value) [fov = glm::radians(35.0f);] UP is (0.0f, 0.0f, 1.0f) Width : 1280 Height : 720 AspectRatio : 16:9

The API what i am using is DirectX 11, actually it works perfectly, the entire map is being rendered perfectly, if i disable frustum culling, however when i enable the frustum culling it's only rendering a part on the wrong direction.

i am doing this:

Calculate Camera View

UP = EVector3(0.0f, 0.0f, 1.0f);
view = glm::lookAtRH(CameraPosition, PlayerPosition, UP);

Calculate Frustum

EFloat tan = EMath::Tan(fov * 0.5f);
_NearWidth = _Near * tan;
_NearHeight = _NearWidth / _AspectRatio;
_FarWidth = _Far * tan;
_FarHeight = _FarWidth / _AspectRatio;

position = PlayerPosition; // Force frustum at player position for see where it points, if i put camera position when i move the camera the view frustum goes on different way, and it already rotates on opposite direction
direction = EMath::Normalize(PlayerPosition - _CameraPosition);
right = EMath::Normalize(EMath::Cross(direction, UP));
up = EMath::Cross(right, direction);
direction = -direction; // glm::LookAtRH does this inside, if i don't do this the frustum doesn't work, but it still going on opposite direction

EVector3 nearCenter = position + (direction * _Near);
EVector3 farCenter = position + (direction * _Far);

EVector3 upNear = up * _NearHeight;
EVector3 rightNear = right * _NearWidth;
EVector3 upFar = up * _FarHeight;
EVector3 rightFar = right * _FarWidth;

EVector3 nearTL = nearCenter + upNear - rightNear;
EVector3 nearTR = nearCenter + upNear + rightNear;
EVector3 nearBR = nearCenter - upNear + rightNear;
EVector3 nearBL = nearCenter - upNear - rightNear;

EVector3 farTL = farCenter + upFar - rightFar;
EVector3 farTR = farCenter + upFar + rightFar;
EVector3 farBR = farCenter - upFar + rightFar;
EVector3 farBL = farCenter - upFar - rightFar;

EVector3 nearTop =      nearTR  - nearTL;
EVector3 nearLeft =     nearBL  - nearTL;
EVector3 topLeft =      farTL   - nearTL;
EVector3 bottomRight =  nearBR  - farBR;
EVector3 farRight =     farTR   - farBR;
EVector3 farBottom =    farBL   - farBR;

_Planes[WP_NEAR]    = EMathPlane(EMath::Cross(nearTop, nearLeft), nearTL);
_Planes[WP_LEFT]    = EMathPlane(EMath::Cross(nearLeft, topLeft), nearTL);
_Planes[WP_TOP]     = EMathPlane(EMath::Cross(topLeft, nearTop), nearTL);
_Planes[WP_BOTTOM]  = EMathPlane(EMath::Cross(farBottom, bottomRight), farBR);
_Planes[WP_RIGHT]   = EMathPlane(EMath::Cross(bottomRight, farRight), farBR);
_Planes[WP_FAR]     = EMathPlane(EMath::Cross(farRight, farBottom), farBR);

EMathPlane:

EMathPlane(const EVector3 &normal, const EVector3 &point)
{
    _Normal = EMath::Normalize(normal);
    _Distance = -EMath::Dot(_Normal, point);
    _AbsNormal = EMath::Abs(_Normal);
}

all functions of my math library are redirected to GLM, for some reason the frustrum points to different area, i am doing a quadtree frustum culling, and i have static boxes prepared for each test, for example:

// First bounding box to check
{0.0f, 0.0f, 0.0f, // min x, y, z
25600.0f, 25600.0f, 1000.0f} // max x, y, z

// Another bounding box
{12800.0f, 12800.0f, 0.0f, // min x, y, z
19200.0f, 19200.0f, 1000.0f} // max x, y, z

here is the (IsVisible) function:

    EVector3 center = (min + max) * 0.5f;
    EVector3 half = (max - min) * 0.5f;
    EVector3 absHalf = EMath::Abs(half);

    EUInt32 isInside = 0x80000000;

    for (EInt32 index = EMathFrustum::WP_NEAR;index < EMathFrustum::WP_MAX;index++)
    {
        EVector3 &normal = frustum._Planes[index]._Normal;
        EFloat &dist = frustum._Planes[index]._Distance;
        EVector3 tmpHalf = EVector3(EMath::ForceSign(absHalf.x, frustum._SignFlip[index][0]),
                                    EMath::ForceSign(absHalf.y, frustum._SignFlip[index][1]),
                                    EMath::ForceSign(absHalf.z, frustum._SignFlip[index][2]));

        EVector3 corner = center - tmpHalf;
        EFloat dot = EMath::Dot(corner, normal) + dist;

        isInside &= RetrieveSign<EUInt32>(dot);
    }

    return isInside > 0;

it is based on Intel Software Culling sample, if the dot value is negative means the object is visible.

Resume of problem: It works(partially), the frustum is working well on the right player position(except when i put camera position, for some reason it goes to another way), however it rotates to wrong direction, if i rotate the camera to right, it rotates to left, i tried to find the problem in the code but i couldn't.

Really thanks everyone for your time, i am with this problem since 2 weeks, and tried everything for sure there is something what i am not doing well, but i don't know what it is, i am getting really stressed about it.

Answer 1

Well, at the end it was a mistake in my game data, after update the game data everything worked perfectly, but also did some changes, instead of keep geometric approach algorithm for get all six planes what I did is change to OpenGL Extract Planes from http://gamedevs.org/uploads/fast-extraction-viewing-frustum-planes-from-world-view-projection-matrix.pdf it works like charm.

If someone is stick with same problem as me, related to right-hand, just don't worry, use that pdf, it works for left-handed and right-handed, in my case I just used GLM library, and without transpose anything applied exactly what says that pdf, here is the code in case you need some example:

EMatrix4 matrix = MUVideoManager::_CurrentProjection * MUVideoManager::_CurrentView; // Don't transpose or inverse anything, just apply it exactly as you get after do lookAtRH and perspectiveFovRH :)
EVector4 tmp;
tmp = EVector4(matrix[0][3] + matrix[0][0], matrix[1][3] + matrix[1][0], matrix[2][3] + matrix[2][0], matrix[3][3] + matrix[3][0]);
_Planes[WP_LEFT].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix[0][3] - matrix[0][0], matrix[1][3] - matrix[1][0], matrix[2][3] - matrix[2][0], matrix[3][3] - matrix[3][0]);
_Planes[WP_RIGHT].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix[0][3] - matrix[0][1], matrix[1][3] - matrix[1][1], matrix[2][3] - matrix[2][1], matrix[3][3] - matrix[3][1]);
_Planes[WP_TOP].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix[0][3] + matrix[0][1], matrix[1][3] + matrix[1][1], matrix[2][3] + matrix[2][1], matrix[3][3] + matrix[3][1]);
_Planes[WP_BOTTOM].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix[0][3] + matrix[0][2], matrix[1][3] + matrix[1][2], matrix[2][3] + matrix[2][2], matrix[3][3] + matrix[3][2]);
_Planes[WP_NEAR].SetPlane(EVector3(tmp), tmp.w, true);
tmp = EVector4(matrix[0][3] - matrix[0][2], matrix[1][3] - matrix[1][2], matrix[2][3] - matrix[2][2], matrix[3][3] - matrix[3][2]);
_Planes[WP_FAR].SetPlane(EVector3(tmp), tmp.w, true);

Plane function:

EINLINE void SetPlane(const EVector3 &normal, const EFloat &distance, EBoolean normalize)
{
    if (normalize)
    {
        EFloat length = EMath::Length(normal);
        _Normal = normal / length;
        _Distance = distance / length;
    }
    else
    {
        _Normal = normal;
        _Distance = distance;
    }

    _AbsNormal = EMath::Abs(_Normal);
}

and Intersection Code:

EINLINE FrustumCullType Contains(EMathFrustum &frustum, EMathAABB &aabb)
{
    EVector3 &min = aabb._Min;
    EVector3 &max = aabb._Max;
    EVector3 center = (min + max) * 0.5f;
    EVector3 half = (max - min) * 0.5f;
    EVector3 absHalf = EMath::Abs(half);

    FrustumCullType cull = FR_INCLUDED;

    for (EInt32 index = EMathFrustum::WP_BEGIN;index < EMathFrustum::WP_MAX;index++)
    {
        EVector3 &normal = frustum._Planes[index]._Normal;
        EVector3 &absNormal = frustum._Planes[index]._AbsNormal;
        EFloat &dist = frustum._Planes[index]._Distance;

        EInt32 side = frustum.ComputeSide(index, center, half, absHalf);

        if (side == EMathPlane::NEGATIVE_SIDE)
        {
            return FR_EXCLUDED;
        }
        else if (side == EMathPlane::NO_SIDE)
        {
            cull = FR_OVERLAP;
        }
    }

    return cull;
}

EINLINE EFloat ComputeDistance(const EVector3 &v)
{
    return EMath::Dot(_Normal, v) + _Distance;
}

EINLINE EInt32 ComputeSide(const EVector3 &center, const EVector3 &half, const EVector3 &absHalf)
{
    // Calculate the distance between box centre and the plane
    EFloat dist = ComputeDistance(center);

    // Calculate the maximise allows absolute distance for
    // the distance between box centre and plane
    EFloat maxAbsDist = EMath::Dot(_AbsNormal, absHalf);

    if(dist < -maxAbsDist)
    {
        return NEGATIVE_SIDE; // Outside! don't render
    }

    if (dist > +maxAbsDist)
    {
        return POSITIVE_SIDE; // Completely visible!
    }

    return NO_SIDE; // A part of it is visible
}

I hope it helps someone, wish you(who is reading) a nice day! :)