I have gotten into quite a bit of trouble getting my rendering correct in my tile based 2D game.
This kind of shader is surprisingly annoying to write, I've heard (we used it in one of our 3D games and kept running into unsightly edge cases). One thing you might want to consider is encoding the outline directly into the sprite texture with a specific color like (0, 255, 255, 0), and just have the shader transform that color when the sprite is selected. Then the shader is a trivial color change, and you have a high level of artist control over the outline and any details you want. https://gamedev.stackexchange.com/a/3822/25297 by "Joe Wreschnig"
I'm trying to do as Joe Wreschnig suggests.
It works fine for the color part, for example if the HighlightState of an object is set to None, then I render the outline transparent so you can not see it.
If the HighlightState is Friendly then i render it as LighGreen, Enemy is Red and so on, the color switching is done with a parameter in the shader as I wanted to draw the objects in one batch (because I have to sort the sprites on Y position to render from TOP to BOTTOM to make sure that a tall object on a tile below another one is rendered on top of the one above).
However, this is where the trouble occurs. I need to change an Effect Parameter depending on the HighlightState of the object that is to be drawn. If I render with any SpriteSortMode other than SpriteSortMode.Immediate, every object will get the outline color of the last rendered object (since it is batched). Sadly, SpriteSortMode.Immediate is not a choice since there can be quite many objects on the screen at one time.
To avoid the issue with the same color upon rendering all the objects as one batch, I thought, why not split the object list into many, one for each HighlightState? This worked terrific, except for one small issue. The Y sorting. If I render them as separate batches, there is no way to tell that one object in one batch should be placed behind another in a different batch or vice versa, which makes me wonder if it feasible to render the outline using a shader or if I have to resort to two sprites per sprite, one slightly enlarged white sprite(color tinted) rendered behind the first one to fake a highlight..
How would I solve this issue in the best manner? If you need code, I will post it but the problem is not really the code I guess, more like the logic itself.
EDIT (SOLUTION):
Based on Nathan's(THANK YOU) answer I managed to implement a shader that can handle different color outlines depending on the state of the object, meaning if it is friendly, an enemy etc.
If you run into this problem yourself, trying to set parameters for each sprite drawn in a single batch and you notice that you only get the last set parameters for ALL the sprites in the batch, it probably be solved.
I for example, wanted to set the outline color individually for each sprite and it worked great using the SpriteBatch.Draw's Color parameter (which sets the vertex color in the VertexDeclaration I assume), I could then use that color in my pixel shader to set the correct outline color.
