Why doesn't TriangleTests::Intersects from DirectXMath work correctly?

Question

I am currently looking through the Introduction to 3D Game Programming with Direct X 11 by Frank Luna and I'm trying to implement the ray-triangle intersections. I already have a ray firing when the mouse is clicked and from what I can tell it is the correct direction as I have made the camera move to the direction I click. The book uses XNA math which from what I can tell got replaced by DirectXMath? The equivalent functions in the book are TriangleTests::Intersects found here https://msdn.microsoft.com/en-us/library/windows/desktop/hh855922(v=vs.85).aspx

But it doesn't seem to work in my application and it doesn't find any intersection.

Does anyone know why? Here is my attempt

for (int i = 0; i < terrain->terrainIndices.size() / 3; i++) {
    //indices
    int i0 = terrain->terrainIndices[i * 3 + 0];
    int i1 = terrain->terrainIndices[i * 3 + 1];
    int i2 = terrain->terrainIndices[i * 3 + 2];

    //vertices for single tri
    XMVECTOR v0 = XMVectorSet(terrain->terrainVertices[i0].Pos.x, terrain->terrainVertices[i0].Pos.y, terrain->terrainVertices[i0].Pos.z, 0.0f);
    XMVECTOR v1 = XMVectorSet(terrain->terrainVertices[i1].Pos.x, terrain->terrainVertices[i1].Pos.y, terrain->terrainVertices[i1].Pos.z, 0.0f);
    XMVECTOR v2 = XMVectorSet(terrain->terrainVertices[i2].Pos.x, terrain->terrainVertices[i2].Pos.y, terrain->terrainVertices[i2].Pos.z, 0.0f);

    //if ray intersects this triangle
    if (TriangleTests::Intersects(rayOrigin, rayDirection, v0, v1, v2, dist) || TriangleTests::Intersects(rayOrigin, rayDirection,
        v0, v2, v1, dist)) {
        stringstream ss;
        ss << "INTERSECTION FOUND" << endl;
        OutputDebugStringA(ss.str().c_str());
        return true;
    }
    else {
        stringstream ss;
        ss << "NO INTERSECTION" << endl;
        OutputDebugStringA(ss.str().c_str());
    }
}

Edit: I have also tried this. The loop is the same when I loop over my terrain triangles for calculating normals.

for (int i = 0; i < 512; i++) {

        for (int j = 0; j < 512; j++) {

            int index1 = (j * 512) + i;
            int index2 = (j * 512) + (i + 1);
            int index3 = ((j + 1) * 512) + i;

            int index4 = ((j + 1) * 512 + i);
            int index5 = ((j * 512 + (i + 1)));
            int index6 = ((j + 1) * 512 + (i + 1));

            //vertices of triangle
            Vertex vertex1 = terrain->terrainVertices.at(index1);
            Vertex vertex2 = terrain->terrainVertices.at(index2);
            Vertex vertex3 = terrain->terrainVertices.at(index3);

            Vertex vertex4 = terrain->terrainVertices.at(index4);
            Vertex vertex5 = terrain->terrainVertices.at(index5);
            Vertex vertex6 = terrain->terrainVertices.at(index6);

            //each vertex as XMVECTOR
            XMVECTOR point1 = XMVectorSet(vertex1.Pos.x, vertex1.Pos.y, vertex1.Pos.z, 0.0f);
            XMVECTOR point2 = XMVectorSet(vertex2.Pos.x, vertex2.Pos.y, vertex2.Pos.z, 0.0f);
            XMVECTOR point3 = XMVectorSet(vertex3.Pos.x, vertex3.Pos.y, vertex3.Pos.z, 0.0f);

            XMVECTOR point4 = XMVectorSet(vertex4.Pos.x, vertex4.Pos.y, vertex4.Pos.z, 0.0f);
            XMVECTOR point5 = XMVectorSet(vertex5.Pos.x, vertex5.Pos.y, vertex5.Pos.z, 0.0f);
            XMVECTOR point6 = XMVectorSet(vertex6.Pos.x, vertex6.Pos.y, vertex6.Pos.z, 0.0f);
            //if ray intersects this triangle
            if (TriangleTests::Intersects(rayOrigin, rayDirection, point1, point2, point3, dist) || TriangleTests::Intersects(rayOrigin, rayDirection,
                point1, point3, point2, dist)
                || TriangleTests::Intersects(rayOrigin, rayDirection, point4, point5, point6, dist)
                || TriangleTests::Intersects(rayOrigin, rayDirection, point4, point6, point5, dist)) {
                stringstream ss;
                ss << "INTERSECTION FOUND" << endl;
                OutputDebugStringA(ss.str().c_str());
                return true;
            }
        }
    }

Edit: The ray I'm firing is normalized, is this a problem for this function? Am I even using it correctly?

Edit: Nvm the function only works when the vector is normalized. Still can't seem to get any intersection. The rayOrigin and rayDirection seem to be fine. The origin is the cameras position when I click and the direction is a normalized vector of the direction i click in.

Edit: TriangleTests::Intersects is new for DirectXMath. This functionality is not available in XNAMath 2.x. Similar functionality for XNAMath can be found in the DirectX SDK Collision sample.

Does this have anything to do with the fact I'm not translating the object or rays into a different space? I.e. World space/object space

Edit: This is how I am getting calculating the ray direction

float pointX, pointY;
    XMMATRIX projectionMatrix, viewMatrix, inverseViewMatrix, worldMatrix, translateMatrix, inverseWorldMatrix;
    XMVECTOR direction, origin, rayOrigin, rayDirection;
    bool intersect, result;

    projectionMatrix = XMLoadFloat4x4(this->projectionMatrix);
    viewMatrix = XMLoadFloat4x4(this->viewMatrix);

    // Move the mouse cursor coordinates into the -1 to +1 range.
    pointX = ((2.0f * (float)mouseX) / (float)screenWidth) - 1.0f;
    pointY = (((2.0f * (float)mouseY) / (float)screenHeight) - 1.0f) * -1.0f;

    pointX = pointX / XMVectorGetX(projectionMatrix.r[0]);
    pointY = pointY / XMVectorGetY(projectionMatrix.r[1]);

    // Get the inverse of the view matrix.
    inverseViewMatrix = XMMatrixInverse(nullptr, viewMatrix);

    //click on screen and camera will rotate on the world Y axis to the clicked point.
    direction = 
        XMVectorSet((pointX * XMVectorGetX(inverseViewMatrix.r[0])) + (pointY * XMVectorGetX(inverseViewMatrix.r[1])) + XMVectorGetX(inverseViewMatrix.r[2]),
                    (pointX * XMVectorGetY(inverseViewMatrix.r[0])) + (pointY * XMVectorGetY(inverseViewMatrix.r[1])) + XMVectorGetY(inverseViewMatrix.r[2]),
                    (pointX * XMVectorGetZ(inverseViewMatrix.r[0])) + (pointY * XMVectorGetZ(inverseViewMatrix.r[1])) + XMVectorGetZ(inverseViewMatrix.r[2]), 0.0f);


    camera->turnCameraToPoint(XMFLOAT3(camera->getPosition().x + XMVectorGetX(direction),
        camera->getPosition().y + XMVectorGetY(direction),
        camera->getPosition().z + XMVectorGetZ(direction)));

    std::stringstream ss;

    // Get the origin of the picking ray which is the position of the camera.
    origin = XMVectorSet(camera->getPosition().x, camera->getPosition().y, camera->getPosition().z, 0.0f);

    // Get the world matrix and translate to the location of the sphere.
    worldMatrix = XMLoadFloat4x4(this->worldMatrix);
    //translateMatrix = terrain->getObjectWorld();
    //worldMatrix = XMMatrixMultiply(worldMatrix, translateMatrix);

    // Now get the inverse of the translated world matrix.
    //inverseWorldMatrix = XMMatrixInverse(nullptr, worldMatrix);

    // Now transform the ray origin and the ray direction from view space to world space.
    rayOrigin = XMVectorSet(camera->getPosition().x, camera->getPosition().y, camera->getPosition().z, 0.0f);
    //rayOrigin = XMVector3TransformCoord(origin, worldMatrix);
    //rayDirection = XMVector3TransformCoord(direction, worldMatrix);
    rayDirection = direction;

    // Normalize the ray direction.
    rayDirection = XMVector3Normalize(rayDirection);

    ss << " clicked direction: " << XMVectorGetX(rayDirection) << " y: " << XMVectorGetY(rayDirection) << " z: " << XMVectorGetZ(rayDirection) << endl;

    OutputDebugStringA(ss.str().c_str());


    // Now perform the ray-triangle intersection test.
    float distance = 0.0f;
    intersect = terrainIntersection(rayOrigin, rayDirection, distance);

    if (intersect == true)
    {
        // If it does intersect then set the intersection to "yes" in the text string that is displayed to the screen.
        std::stringstream ss5;

        ss5 << "INTERSECTION!! - " << "dist: " << distance << endl;
        OutputDebugStringA(ss5.str().c_str());
    }
    else
    {
        // If not then set the intersection to "No".
        std::stringstream ss5;
        ss5 << "NO INTERSECTION..." << endl;
        OutputDebugStringA(ss5.str().c_str());
    }

    return;

---

Edit: I have managed to get a valid intersection from the method, but there is something incorrect as it is now returning a valid intersection for anywhere I click on the screen, and the direction of the ray is producing incorrect results. I have tried to make sure that the ray, ray origin and the vertices of the triangle are all in the same space by doing

XMMATRIX translationMatrix = XMMatrixTranslation(camera->getPosition().x, camera->getPosition().y, camera->getPosition().z);
XMMATRIX finalWorld = translationMatrix * worldMatrix * viewMatrix * projectionMatrix;
rayOrigin = XMVector3TransformCoord(origin, finalWorld);

Each vertex is also in the same space

        finalWorld = XMMatrixIdentity();
        translationMatrix = XMMatrixTranslation(vertex3.Pos.x, vertex3.Pos.y, vertex3.Pos.z);
        finalWorld = translationMatrix * world * view * projection;
        point3 = XMVector3TransformCoord(point3, finalWorld);

This now produces a valid intersection

clicked direction: -0.799399 y: -0.0989474 z: -0.592597 INTERSECTION FOUND INTERSECTION!! - dist: -0

Although this allows me to click anywhere on the screen and the intersection is deemed valid, even when I am not clicking anywhere on the terrain. Does anyone have any idea why this is happening? The distance returned from the TriangleTests::Intersects method is always 0 or -0

Answer 1

I have actually fixed the issue I was having. I needed to transform the vectors into the same spaces using

XMVector3Transform(origin, ray->getObjectWorld());