0
\$\begingroup\$

(skip to edit for the actual problem)

I have a simple plane mesh generated via script (has arbitrary shape).

I want the shader to discard (set to black) all pixels that aren't passing through that mesh. Currently, the mesh is in MeshFilter but isn't rendered through Mesh Renderer, as I don't want the mesh itself to be visible - it's only purpose is to filter the camera output.

I could also send the mesh data to shader, but since it has arbitrary shape (and arbitrary number of vertices), I don't know how I can check if the pixel is running through that mesh or not. The mesh is made of few vertices (201) generated each fixedUpdate (currently, all the mesh vertices are changed in fixedUpdate (vertices set to empty array with fixed length, then filled). I didn't see low performance yet, so I didn't want to dive into optimizations just yet). All the vertices have same y axis (I don't care about y axis at all, so a vertical check can be omitted) and they don't overlap (I'm using simple raycasting to get visible area, something similar to this, but in 3D and the result is a plane-like mesh).

What I want is to show movable objects (like enemies) only when they're under that mesh (camera is from top, aiming down), else just show the map itself (which doesn't change), slightly darkened.

I found this page for creating a simple screen shader in Unity (I couldn't find how to apply shader to camera from Unity docs at all), but it didn't work (no errors, it just didn't show the screen with the black and white effect with intensity set to 1 (which should be fully black and white if I understand it correctly) or 0).

Edit:

As DMGregory pointed out, this is a job for stencil buffer.

What I thought of now is to check if the pixel is inside one of the triangles (not sure how fast that is), using for example this code from this question, then write that result to stencil buffer and filter out all points that aren't in that triangle. Then merge the result with the map 'background'.

I know how to merge them (at least I think so) and how to filter out pixels based on their stencil buffer value (direct example in docs).

\$\endgroup\$
7
  • 2
    \$\begingroup\$ This sounds like a job for the stencil buffer and stencil testing. \$\endgroup\$
    – DMGregory ♦
    Commented Jun 28, 2021 at 15:37
  • \$\begingroup\$ I found stencil buffers/testing too, but didn't see a way to somehow use it. I'm total newcomer to shaders, and have no idea where to start using them. What I believe is that you first need to write to that stencil buffer in that same shader, but then it still leaves the question on how to see if the pixel is running through that mesh (or is between given points). Let me edit the question a bit to account for newly learned material. \$\endgroup\$
    – Dan B.
    Commented Jun 29, 2021 at 11:45
  • \$\begingroup\$ You render the mesh with a shafer that just writes to the stencil, but not to the depth or colour buffers. Now you've laid an invisible mask over your scene. Then you draw the rest of your content with materials that include a stencil test, and draw only where the stencil was set to a particular value when you rendered the mesh. Although, to avoid modifying all your other shaders, you could accomplish a similar trick with depth: render an invisible quad at the camera's near plane, and punch a hole in it by rendering your mesh at max depth. Now everything outside the mesh will fail depth test. \$\endgroup\$
    – DMGregory ♦
    Commented Jun 29, 2021 at 11:50
  • \$\begingroup\$ Oh that makes sense, thanks a lot. Can I avoid modifying all other shaders by adding screen shader as 'post-processing effect'? They can be stacked, so I could just add one that filters based on the stencil test. I don't understand what you mean by punching a hole by rendering at max depth. Would simply adding the invisible quad and then rendering the mesh to contain all the relevant objects be enough (so in my case, adding invisible quad and adding height to the mesh)? \$\endgroup\$
    – Dan B.
    Commented Jun 29, 2021 at 12:13
  • \$\begingroup\$ Once you've rendered the rest of your scene, it's there in the colour buffer. You cannot retroactively "un-render" it as a post effect. You could draw a solid colour over the parts you want to cover though. Or render your scene to a RenderTexture, then render your mesh with that RT as its texture, so it acts like a "window" into the scene you drew before. These options are less efficient than cutting out the unwanted drawing at the source with depth or stencil tests though, because you end up still paying for pixels you don't want to see in the end. \$\endgroup\$
    – DMGregory ♦
    Commented Jun 29, 2021 at 12:16

1 Answer 1

Reset to default
0
\$\begingroup\$

What I was looking for all along is called fog of war. @DMGregory told me that, after a long discussion about the effects I was talking about.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .