I am using augmented reality to put a 3D object on top of a "camera detected" object.

The 3D object can be though of as leaves that go above the real world object but also fall in front / behind / on the sides of it.

The leaves are not uniform so they appear as arcs which let more of the "camera detected" object be seen in some parts and less in others.

Because of this I want to only show the leaves that would normally be seen if there was a real 3D object (instead of the camera detected one) in that place. But since the camera detected one is just a background image part of the leaves on the side or behind are seen when the object is rotated.

I was thinking it might be possible to use an "invisible" object that moves and rotates the same way as the "camera detected" one so it behaves as a placeholder for depth testing but is not actually rendered.

Is this possible? May I have some pointers on how to achieve this?

Thank you

  • \$\begingroup\$ Normally I'd say "Depth Mask" but I don't think that's going to cut it here. Depending on your AR setup, the camera input being displayed on the screen isn't done via object in the scene, but rather rendered directly to the screen on one pass with all 3D geometry rendered on another pass. \$\endgroup\$ May 16 '16 at 4:06
  • \$\begingroup\$ I'm confused can you provide some screenshots or something to further illustrate your problem ? \$\endgroup\$
    – Uri Popov
    May 16 '16 at 8:14
  • \$\begingroup\$ Imo: If you have just a video image with no depth information, you must re-construct the depth somehow. The goal would be to either render invisible objects that you somehow construct and fit into correct locations (eg. a wall), and let the GPU's z-buffer handle the testing. Alternatively you'd somehow construct a fullscreen depth texture from your bkg image, and do a depth test in a shader, to determine if to draw leave (= clip bkg image, let real world through) or draw bkg image. Fadeout & add fog to hide bad looking locations. Somehow you must resolve the depth, then anything is possible. \$\endgroup\$
    – Stormwind
    May 16 '16 at 10:50
  • \$\begingroup\$ Are you able to achieve the effect with a real 3D object that would represent the detected object? I would guess that the biggest issue will be in the positioning of the object and not the hiding of the leaves (as others have pointed out- a depth mask should work, if you have the obscuring object in place). \$\endgroup\$
    – Felsir
    May 19 '16 at 15:02
  • \$\begingroup\$ @Draco18s hey, i followed the link you posted and it did work out. I calculate the position of the dummy 3d object and attach this depth mask to it so it cuts off and lets my background be seen where the object would be. If you would like to post your comment as an answer i can mark it as correct. \$\endgroup\$
    – Pochi
    May 20 '16 at 1:46

Following Draco18s comment, I'm using a 3D object which is moved around the screen by tracking the live video object. Then I apply a depth mask to this object which behaves as if there was an actual object there since it allows my background to be seen wherever this object would be usually seen while still remaining invisible.

  • \$\begingroup\$ Exactly, just render an invisible object but still write to the dept mask so it is used for culling, if you render front-to-back. - If you render back-to-front (for transparent objects) you will have to use a texture trick. Just assign the camera-image as a texture to the object and use a custom shader, which will read the texture pixel which corresponds to the on-screen coordinates (instead of the object-specifiv u/v) - the same effect like a check-pattern in old comics. \$\endgroup\$
    – Falco
    Jul 25 '16 at 12:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .