I'm doing some model view and projection transforms in the vertex shader and I want to determine whether the current vertex will end up on the viewport or not.

After searching a bit I found that after the projection transform the coordinates have not been "normalized". So the criteria for a vertex (after the transforms) to appear in the final viewport is the following:

$$-w<x<w$$ $$-w<y<w$$ $$0<z<w$$

where w is the 4th coordinate of the vertex after the transforms

Performing this check I had a great amount of vertices passing the test without being in the viewport and this caused many sorts of problems in whatever I was trying to implement.

However, passing the vertices in their local coordinates to the fragment shader and then doing the exact same process with the transforms and the tests everything worked out fine.


  • \$\begingroup\$ In what specific cases do the results differ from "seeming" correct? \$\endgroup\$ – DMGregory Dec 21 '18 at 19:25
  • \$\begingroup\$ I'm trying to implement shadow mapping creating 6 different shadow maps for 6 different views. In order to apply the correct shadow map to a vertex I check in which view of the 6 it lies in. To do that I apply the 6 model view projection transforms and after that I have to decide which one includes my vertex. When I say it is not correct I mean shadows move around unstable. Everything up to this point is correct though. For example, when testing two opposite faces only by the z value gives two successful shadow maps. Adding these comparisons above, even these two correct maps fail . \$\endgroup\$ – John Katsantas Dec 21 '18 at 19:31
  • 2
    \$\begingroup\$ Remember that a vertex that's out of frustum for this map might still be part of of a triangle that does cross into the frustum somewhere. So moving it might have side effects on in-frustum geometry. \$\endgroup\$ – DMGregory Dec 21 '18 at 20:05
  • \$\begingroup\$ That was a really good point. And it sort of led me to the solution. I was doing this process in the vertex shader. Doing this in the fragment shader did the trick. I haven't fully understood what happens between the fragment and the vertex shader and why it works now. Might upload a question tomorrow as this is not the first time that I've had this problem. It must be that I have to apply my ideas on every fragment and not every vertex. \$\endgroup\$ – John Katsantas Dec 21 '18 at 21:33
  • \$\begingroup\$ Can you elaborate on the specific symptoms by which you were able to diagnose "a great amount of vertices passing the test without being in the viewport" versus " everything worked out fine"? It sounds to me like there might be another issue at work here - not just false positives. \$\endgroup\$ – DMGregory Dec 22 '18 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.