I'm trying to get the mouse click position on a terrain geometry, but multiple times in each frame; doing it on the CPU was way too slow, so I'm trying to do it with a compute shader. I can't seem to get a valid result back. I guess it's somehow related to my raydirection/rayorigin vectors, because I tried 4 different written DirectX HLSL solutions, but none of them works for my case; though, I did have to modify them. Anyway, this is my code to generate the ray vectors:

XMFLOAT3 RayOriginFloat = m_Camera->GetPosition(); //The eye position of the camera. I use this vector in the compute shader.

XMVECTOR worldPos; 
XMVECTOR RayOrigin = XMLoadFloat3(&RayOriginFloat);
DirectX::BoundingFrustum frustum;

// I use the DirectXMath XMVector3Unproject method to project the mouse coord to worldspace.
ScreenToWorld(m_mouseX, m_mouseY, &worldPos); 

XMVECTOR RayDir = worldPos - RayOrigin; 
RayDir = XMVector3Normalize(RayDir); //And finally the normalized direction vector

And this is the HLSL code:

cbuffer RayBuffer
    float3 RayOrigin;
    float3 RayDir;
    matrix view;

struct ComputeInputType
    float3 position : POSITION;
    float2 tex : TEXCOORD;
    float3 normal : NORMAL;

struct ComputeOutputType
    float distance;
    float3 position;

static const uint ThreadSize = 32;

StructuredBuffer<ComputeInputType> BufferIn : register(t0);
RWStructuredBuffer<ComputeOutputType> BufferOut : register(u0);

[numthreads(ThreadSize, ThreadSize, 1)]
void PickingCSMain(uint3 dispatchThreadID : SV_DispatchThreadID)
    float result;
    float3 vertices[3];
    float3 edge[2];

    if (dispatchThreadID.x < 64 && dispatchThreadID.y < 64)
        vertices[0] = BufferIn[(dispatchThreadID.x + dispatchThreadID.y * ThreadSize * 2)].position;
        vertices[1] = BufferIn[(dispatchThreadID.x + (dispatchThreadID.y + 1) * ThreadSize * 2)].position;
        vertices[2] = BufferIn[(dispatchThreadID.x + 1 + (dispatchThreadID.y + 1) * ThreadSize * 2)].position;

        edge[0] = vertices[1].xyz - vertices[0].xyz;
        edge[1] = vertices[2].xyz - vertices[0].xyz;

        float3 normal = normalize(cross(edge[0], edge[1]));
        float b = dot(normal, RayDir);

        float3 w0 = RayOrigin - vertices[0].xyz;
        float a = -dot(normal, w0);
        float t = a / b;

        result = t;

        float3 p = RayOrigin + t * RayDir;
        float uu, uv, vv, wu, wv, inverseD;
        uu = dot(edge[0], edge[0]);
        uv = dot(edge[0], edge[1]);
        vv = dot(edge[1], edge[1]);

        float3 w = p - vertices[0].xyz;
        wu = dot(w, edge[0]);
        wv = dot(w, edge[1]);
        inverseD = uv * uv - uu * vv;
        inverseD = 1.0f / inverseD;

        float u = (uv * wv - vv * wu) * inverseD;
        if (u < 0.0f || u > 1.0f)
            result = -1.0f;

        float v = (uv * wu - uu * wv) * inverseD;
        if (v < 0.0f || (u + v) > 1.0f)
            result = -1.0f;

        BufferOut[(dispatchThreadID.x + dispatchThreadID.y * ThreadSize * 2)].distance = result;
        BufferOut[(dispatchThreadID.x + dispatchThreadID.y * ThreadSize * 2)].position = vertices[0].xyz;

The result is always -1.0f. I don't get it; I might have made a big mistake, but I can't find where.

Why does my triangle ray intersection test always return -1.0f?

  • \$\begingroup\$ What algorithm do you use? Möller-Trumbore? \$\endgroup\$
    – Bálint
    Jun 23 '17 at 16:07
  • \$\begingroup\$ Yes, Möller-Trumbore. \$\endgroup\$ Jun 23 '17 at 16:15

I've figured it out, finally. The first mistake i made was that i only tested every second triangle, and the second mistake was that the xyz parameters of the ray direction were offset in the buffer. Not sure how it happened, because I'm pretty sure it was fine earlier. (instead of e.g. (4,5,6) the direction vector on the GPU was (0,4,5). i've added a single float for padding)

It works like a charm now, and it's super fast!

If you took some time to try to help me, I'm sorry to bother you, and thank you for your time!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.