# Sorting sprites above the surface of 3D mesh?

I'm having trouble getting sprites to display on top of a 3D mesh. My requirements are as follows:

1. Sprite should billboard
2. Sprite should not clip into the mesh
3. Sprite should be hidden when on the other side of the mesh, or blocked by a piece of the mesh that's "large/substantial"
4. Sprite should be not rely on placement above mesh to fix clipping
5. No raycasting should be used

4.Is needed as the sprite grows when zooming out whereas the mesh doesn't. Sprites can get pretty large. 5. Raycasting in this application is really expensive on the target platform so not really usable when considering 1000's of sprites.

It's a bit hard to explain so let me give an example. There's a bit of setup so ignore the first 100 lines or so. The shader code and other stuff is in the second section.

//=====================================
//=== SKIP DOWN TO THE NEXT SECTION ===
//=====================================
//This is just default THREE.js stuff
var getObjects = function(jCanvas){
var ret = {};
ret.renderer = new THREE.WebGLRenderer({canvas : jCanvas[0]});
ret.renderer.setSize(jCanvas.width(), jCanvas.height());
ret.renderer.context.getExtension("EXT_frag_depth");

ret.cam = new THREE.PerspectiveCamera(45, jCanvas.width() / jCanvas.height(), 0.1, 100);
ret.cam.position.set(0,2,6);
ret.cam.quaternion.multiply(new THREE.Quaternion().setFromAxisAngle( new THREE.Vector3( 1, 0, 0 ), -Math.PI/8 ));

ret.camControls = new THREE.TrackballControls(ret.cam, ret.renderer.domElement);
ret.camControls.target.set(0, 0, 0);

return ret;
};

var defaultScene = function(objs){
objs.scene = new THREE.Scene();
var geo = new THREE.BoxGeometry(1,1,1);
var mat = new THREE.MeshLambertMaterial({color : 0xff0000});
var msh = new THREE.Mesh(geo,mat);
var light = new THREE.HemisphereLight( 0xffffbb, 0x080820, 1 );
var directionalLight = new THREE.DirectionalLight( 0xffffff, 0.5 );
directionalLight.position.set( -2, 1, -1 );

var lastTime = Date.now();
objs.renderFunc = function() {
requestAnimationFrame( this.renderFunc.bind(this) );

var delta = Date.now() - lastTime;
this.camControls.update(delta);

this.renderer.render( this.scene, this.cam );
};

return objs;
};

var objs = getObjects(\$("#can"));
defaultScene(objs);

//=====================================
//=== THE CODE WE'RE INSTERESTED IN ===
//=====================================
var mat = new THREE.ShaderMaterial( {
transparent : true,
uniforms: {
depthPos: {
value : new THREE.Vector3()
}
},
extensions : {
fragDepth : true
},
vertexShader: 
uniform vec3 depthPos;
varying vec2 vUV;
varying float depthVal;
void main()
{
vUV = uv; //Output UV to fragment
gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
//Model maxtix not needed as depthPos is positioned in world coordinates already
depthVal = clamp((projectionMatrix * viewMatrix * vec4(depthPos, 1.0)).z, 0.0, 1.0);//Use depth pos to set depth
}
,
fragmentShader: 
varying vec2 vUV;
varying float depthVal;

void main() {
//Simply sample the passed UV and get the texture color
float depthVal2 = depthVal;// - 20.0;
gl_FragColor = vec4(depthVal2, depthVal2, depthVal2, 1.0);
gl_FragDepthEXT = depthVal2;
}
});

//
var thisPos = new THREE.Vector3(0.5,0.45,0.45);
var thisCam = objs.cam;
var thisNrm = new THREE.Vector3(1,0,0);

var geo = new THREE.PlaneGeometry(1, 1); //1 by 1 plane
var baseScale = 1/16;
var defectSprite = new THREE.Mesh(geo, mat);
defectSprite.scale.set(baseScale, baseScale, baseScale);
defectSprite.position.copy(thisPos);

geo = new THREE.SphereBufferGeometry(1,6,6);
mat = new THREE.MeshBasicMaterial({color : 0xFFFF00});
var defectBall = new THREE.Mesh(geo, mat);
defectBall.scale.set(1/64, 1/64, 1/64);
defectBall.position.copy(thisPos);

var clamp = function(a,b,c){ return Math.min(Math.max(a,b),c); }

var doDepthValue= function(){
window.requestAnimationFrame(doDepthValue);
var d = thisPos.distanceTo(thisCam.position);

//DefectSprite
//Set rotation
var r = thisCam.quaternion;
defectSprite.quaternion.copy(r);
//Set pos to depth test from
var depthNrm = new THREE.Vector3().copy(thisNrm).multiplyScalar(1/4);
defectSprite.material.uniforms.depthPos.value = depthPos; //Gets transforms applied to it so no further work need be done
};

doDepthValue();
objs.renderFunc();
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<script src="https://cdn.rawgit.com/mrdoob/three.js/dev/build/three.js"></script>
<script src="https://cdn.rawgit.com/mrdoob/three.js/dev/examples/js/controls/TrackballControls.js"></script>

<canvas id="can" width="400" height="300"></canvas>

Use left mouse drag to orbit, right mouse to pan, middle mouse/wheel to zoom. The sprite appears near the yellow ball as a white to black square representing the depth assigned to the sprite. The desired effect is that it appears on top of the cube unless it's turned around.

The most important parts are doDepthValue() along with the ShaderMaterial explained below.

• doDepthValue() will set the sprites rotation and calculate a position above the mesh using the normal and current sprite position to use in determining the final depth value (called depthPos). This is passed to through the uniforms of the shader.
• ShaderMaterial takes the depthPos and transforms it with the view and projection matrix (only those two because it's a world position already). After this is should be in clip space. If I take the z coordinate and then pass this to the fragment shader, applying it to the depth value (using gl_FragDepthEXT) should give a depth value for a position just a small amount above the normal of the original mesh.

However in the example this is not what happens. The sprite is draw above the mesh if you zoom in close enough but past a certain distance it is draw behind everything.

How do I accomplish what I want? What am I doing wrong?

• Okay, so minutes after posting this I fixed it (kind of). I forgot to divide by w in the vertex shader so now it's using homogenous coordinates, not clip space. Problem is, now it seems as though moving the depthPos by a fixed value (1/4 in this case) causes problems when for sprites far away from the camera when the mesh is also far away (maybe a depth buffer precision issue?) – Coburn Nov 21 '16 at 23:31
• Seems like I forgot to also do a transformation regarding glDepthRange (which in WebGL is by default depthVal / 2 + 0.5 (if you don't set the depth range. – Coburn Feb 16 '17 at 14:18

It seems I was just doing the transformation wrong and missed some steps in the transformation pipeline.

Here is my final transformation in ES6 javascript

let depthOffsetMagnitude = util.math.clamp(distToCam - 0.05, 0, 0.2);
let spriteDepthPos3 = new THREE.Vector3()
.copy(depthOffsetDir)
//Dynamically adjust offset with cam distance so that depth
//value doesn't get lost outside/behind cam otherwise the sprite
//will disappear.
.multiplyScalar(depthOffsetMagnitude)

After doing this I found out about THREE.Vector3.project which I probably could have used from the start.