I've been looking at creating some 2D rendering systems in D3D9, basically because I don't like ID3DXSprite. For the output of the vertex shader, what co-ordinate system does the run-time expect vertex co-ordinates in, like homogenous device co-ordinates? And secondly, does the depth of the resulting vertices have any effect except on the culling/depth buffering? I wanted to "reserve" a section of depth for use in 2D drawing, and then have a section for 3D geometry, to ensure that 3D geometry is always behind 2D.
For the output of the vertex shader, what co-ordinate system does the run-time expect vertex co-ordinates in?
You should output vertices in clip space from the vertex shader (this is generally where things end up after the projection matrix is applied). The hardware will take the output of the vertex shader, perform clipping, and then perform the perspective division by
w and continue along the pipeline.
Does the depth of the resulting vertices have any effect except on the culling/depth buffering?
It could also be used in early-Z optimizations, potentially, and possibly a few other things like that. And maybe fixed-function fog, but I don't think anybody uses that any longer anyhow.
So in general, no. You're probably fine trying to reserve a range in the depth buffer for 2D versus 3D from that standpoint, although I suspect it may be a touch cumbersome to give the 3D elements proper depth from the eye to ensure they land in the desired range (it may be easier just to disable the depth test for the 2D elements or clear the depth buffer to the desired "2D far" value after all the 3D elements are rendered, or something, depending on exactly what you need).
The only other thing to be aware of it is the non-linearity of the depth buffer, so you should probably use a much smaller range for the 2D stuff since it will presumably be closer and thus in the more granular region of the buffer.