The goal: issue least calls to d3d11 ImmediateContext and render advanced effects like toon / outline / anything that requires rendering the same vertex buffers with at least two different vertex shader and pixel shader programs efficiently by sorting opaque drawcalls front to back and transparent drawcalls back to front.
What I did: Abstract calls to d3d11 immediate context by a DrawCall struct that acts as a gpu command recorder. A call to DrawCall.SetIndexBuffer will create a c# Action that will issue a call on the ImmediateContext upon invoke call. The DrawCall internal gpu command list is sorted the following way: set shaders, set constantbuffers, set blendstate, set stencilstate, set vertexbuffers.... I am using sharpdx and c#.
My problem: I use a 64bit sortkey for my gpu commands. When rendering complex effects like outline or anything that requires state changes to outputmerger the whole idea with sorting doesn't seem right... example: I render opaque objects first and then transparent objects... I want to render an object opaque with a diffuse shader and it should have a red transparent outline. For this effect I have to modify stencil and blend states in between drawindexed calls. According to my sortkey those gpu commands are on different draw loops opaque<>transparent but the stencilstate requires both drawindexed calls to be one after another.
Should I implement a chained gpu command for multipass rendering or are there better ways to handle such effects?