1
\$\begingroup\$

I am working on a Unity project where I have 3 models: a stacked, rounded wall (think logs), a flat wall (think bricks), and a door.

The door has an animation to open / close itself, and when it's attached to a wall, there's one major problem -- when the door swings in, it reveals not the empty space one would expect, but the segment of wall it's layered on top of!

Since all 3 models are part of dynamic mesh structures, I can't "pre-calculate" what the model should look like to account for the empty space of the door -- what if the player puts the door on the next segment of wall instead?

I considered expanding the "door" model to a "door + wall segment" model (replacing that wall segment entirely), but because I already have two separate shapes of wall, it quickly becomes a combinatorics problem -- 2 models for each door, per door shape (not to mention windows, or a 3rd [or fourth!] wall shape!). In short, this doesn't seem like a tenable solution.

What would work most cleanly is if I could remove a "Door-shaped" segment of my Wall segment mesh, leaving a clear hole that the Door model could snugly fit.

In an equation: Final Mesh A = Mesh A Volume - (Mesh A Volume ∪ Mesh B Volume)

Is an operation like that feasible? My gut tells me that polyhedron intersection is hard, and there might be a better way to do this. I'm also equally interested if someone has a better suggestion on how to solve the initial problem of doors intersecting with wall segments.

\$\endgroup\$
  • \$\begingroup\$ Do you really want to know if mesh subtraction is possible? If so, the answer is obviously "yes" — any decent 3D modelling software implements tools for this, and it's been seen in various forms in games dating back decades. Does that answer your question, or do you really want to ask "How can I cut a doorway through my wall at runtime?" — mesh modification might not even be required for that. Shader clipping or stencil tricks may suffice. \$\endgroup\$ – DMGregory Nov 11 at 19:27
  • \$\begingroup\$ Do you have any limits on how many holes you need to cut in a single wall segment? \$\endgroup\$ – DMGregory Nov 11 at 21:24
  • \$\begingroup\$ Interested in solving for a single cut. (I'd think most solutions could be adapted to support disjointed holes if things were to get fancy.) \$\endgroup\$ – Sable Dreamer Nov 11 at 21:38
  • \$\begingroup\$ Shaders can definitely do this. It shouldn't even be hard. Sebastian Lague has made a video about this (among other things) that you can find here: youtu.be/Cp5WWtMoeKg?t=159. However, I don't know how this places with colliders. I suspect you're going to have to modify that separately; but it shouldn't prove too hard. \$\endgroup\$ – Demetre Saghliani Nov 11 at 22:31
  • \$\begingroup\$ Luckily, I'm not using mesh colliders at all, so no worries on that front! I'll check out the video. \$\endgroup\$ – Sable Dreamer Nov 11 at 22:45
3
\$\begingroup\$

Actually cutting this hole in the mesh can be challenging. It's doable - 3D modelling packages manage it all the time - but it can be computationally intensive to handle every potential case, and it requires topological information about a mesh that we normally don't use for rendering in games, which can be a pain to create/maintain.

It's easier with raymarching techniques like the one Demetre Saghliani linked to - if you're already using a raymarching renderer. But if you're using conventional polygonal rasterization, this too can be a big ask.

Fortunately, for simple tasks we can get by with a simple trick: don't actually compute the full intersection and internal surfaces or change the mesh in any way, just abort rendering any pixels inside the doorway when drawing your frame. Inside the wall's shader we can use some simple math to work out whether we're in the rectangle of the doorway, and if so, tell the blending pipeline "nevermind, forget this fragment was even trying to draw here".

It's basically an inversion of the shader shown here.

Note that this will leave a visible gap in the mesh - it's not generating new faces to cap the edges of the hole. But in the context of a doorway, that's not a big problem really. We can just add a door frame mesh that covers this raw edge, so the player never sees our cheat.

You can even cheat the collisions in a similar way. Place a trigger collider inside the doorway, small enough that a character outside the doorway won't touch it, but large enough that you can't pass through the door without touching it. Use this trigger collider to disable collisions between the character and the wall, and re-enable them when the character exits the trigger, allowing them to pass through the wall like there was nothing there.

\$\endgroup\$
  • \$\begingroup\$ So it sounds like such an algorithm hides based on vertex. I will see if I can implement this in Unity! \$\endgroup\$ – Sable Dreamer Nov 18 at 13:22
  • \$\begingroup\$ No, by fragment. The vertex culling in that answer is just an extra optimization for large, dense models which you won't need for this case. You can find a working Unity implementation at that link, you'll just need to invert the test (discard inside, keep outside) and change which axes you use. \$\endgroup\$ – DMGregory Nov 18 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.