Why does VertexPositionColor seem incompatible with Monogame?

Question

This question is a continuation of What causes this InvalidOperationException when drawing primitives?.

I've been dealing with HLSL shaders in Monogame over the past few days. It has been extremely time-consuming and frustrating trying to duplicate shader functionality I was able to achieve easily with vanilla XNA. As an example, rendering a fullscreen quad using VertexPositionColor elements fails with the following error.

Up until this morning, I had been using the BasicEffect class with as many configurations as I could think of without success. After it became clear that BasicEffect couldn't achieve what I wanted, I wrote a simple custom shader that looks like this.

struct VertexShaderInput
{
    float4 Color : COLOR0;
    float4 Position : SV_POSITION0;
};

struct VertexShaderOutput
{
    float4 Color : COLOR0;
    float4 Position : SV_POSITION0;
};

VertexShaderOutput VertexShaderFunction(VertexShaderInput input) 
{
    VertexShaderOutput output;
    output.Color = input.Color;
    output.Position = input.Position;

    return output;
}

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    return input.Color;
}

technique Technique0
{
    pass Pass0
    {
        VertexShader = compile vs_4_0 VertexShaderFunction();
        PixelShader = compile ps_4_0 PixelShaderFunction();
    }
}

The shader above seems reasonable because, as the name VertexPositionColor implies, I only need position and color. But using that effect in place of BasicEffect didn't change anything either. I finally started digging more deeply into vertex declarations (something I had no knowledge of until today) and came across the following source code for VertexPositionColor.

VertexElement[] elements = new VertexElement[]
{
    new VertexElement (0, VertexElementFormat.Vector3, VertexElementUsage.Position, 0),
    new VertexElement (12, VertexElementFormat.Color, VertexElementUsage.Color, 0)
};

Based on those lines, the vertex declaration is expecting 1) a Vector3 position and 2) a color. That seemed like the answer I needed, since in my shader, position is float4 rather than float3. But changing position to float3 produced the following error.

So, to recap and describe what I can see so far:

1. I'm trying to render a quad using four VertexPositionColor elements.
2. Rendering fails because the vertex shader input (defined in my shader code above) represents position using float4 rather than float3.
3. I can't change position to use float3 because the shader compiler version (vs_4_0) doesn't allow it (I think the shader version is relevant).

This whole this is just very confusing, especially since I've used shaders like this successfully in the past with XNA rather than Monogame. Is there a solution to this problem apart from creating my own custom VertexPositionColor replacement class with its own custom vertex declaration? That all seems overkill considering the simplicity of what I'm trying to do.

This issue echoes a general feeling I've gotten that HLSL shaders in Monogame are, for lack of better phrasing, messed up. Can anyone help clarify all this?

Thank you.

Answer 1

The actual root cause of this problem was identified by Felsir in this question. It turns out that an exception was thrown because no shader was applied when drawing primitives using GraphicsDevice. In other words, there was effectively nothing wrong with the shader itself (either using BasicEffect or a custom one). The exception message was misleading.

I'm not sure about StackExchange conventions on self-answering questions, but I wanted this question to have a correct accepted answer. If this kind of post doesn't fit conventions, please let me know.

Answer 2

struct VertexShaderInput
{

float3 Position : POSITION0;
float4 Color : COLOR0;
};

struct VertexShaderOutput
{
float4 Position : SV_POSITION;
float4 Color : COLOR0;

};

VertexShaderOutput VertexShaderFunction(VertexShaderInput input) 
{
VertexShaderOutput output;
output.Color = input.Color;
output.Position = float4(input.Position,1);

return output;
}

That is how it should be.

Input does not take SV_POSITION it has to be POSITION0 and must be first in struct same with output, SV_POSITION not SV_POSITION0 and it must be first in the struct.

If u look at the vertex struct in C# it has to match the one in HLSL.

---

I'm not sure if this will work in mono game but if u are talking full screen quad then u can do it like this:

don't set a vb or ib or inputlayout, call context.draw(3) and have the vertex shader like this...

// outputs a full screen triangle with screen-space coordinates
// input: three empty vertices
VertexShaderOutput VSQuad(uint vertexID : SV_VertexID)
{
VertexShaderOutput result;


result.TexCoord = float2((vertexID << 1) & 2, vertexID & 2);
result.Position = float4(result.TexCoord * float2(2.0f, -2.0f) + float2(-1.0f, 1.0f), 0.0f, 1.0f); 

return result;
}

I would have color set as a const so u can set it from C# per object

---

Edit: The name of the vertex element does not match the vertex input because in mono game on the back end the elements would look something like this:

  Public Shared VertexElements As InputElement() = New InputElement() {New InputElement("POSITION", 0, DXGI.Format.R32G32B32_Float, 0, 0, InputClassification.PerVertexData, 0),
                                                                     New InputElement("NORMAL", 0, DXGI.Format.R32G32B32_Float, InputElement.AppendAligned, 0, InputClassification.PerVertexData, 0),
                                                                     New InputElement("TEXCOORD", 0, DXGI.Format.R32G32_Float, InputElement.AppendAligned, 0, InputClassification.PerVertexData, 0)
                                                         }

Notice how the name of POSITION not SV_POSITION in XNA it didn't matter how u did ur vertex structs the runtime would sort it out but now all names must match so if you put SV_ in your shader it want match if u decompile the backend I'm sure it will look something like what I posted because that is how u do it in sharp dx.

But I don't even do it like that anymore because input layout building sucks by hand so I wrote this:

  Private Function InputLayoutBuilder(_shader As ShaderBytecode) As InputLayout
    Dim blah As New ShaderReflection(_shader)
    Dim Elements As New List(Of InputElement)
    For i As Integer = 0 To blah.Description.InputParameters - 1
        Dim nnn As ShaderParameterDescription = blah.GetInputParameterDescription(i)
        If nnn.SemanticName.Contains("SV_") Then
            Continue For
        End If
        Dim _InputElement As New InputElement
        Dim format As New DXGI.Format
        _InputElement.SemanticName = nnn.SemanticName
        _InputElement.SemanticIndex = nnn.SemanticIndex
        _InputElement.Slot = nnn.Stream
        _InputElement.InstanceDataStepRate = 0
        If i = 0 Then
            _InputElement.AlignedByteOffset = 0
        Else
            _InputElement.AlignedByteOffset = InputElement.AppendAligned
        End If
        If nnn.SemanticName = "WORLD" Then
            _InputElement.Slot = 1
            If nnn.SemanticIndex = 0 Then
                _InputElement.AlignedByteOffset = 0
            Else
                _InputElement.AlignedByteOffset = InputElement.AppendAligned
            End If
            _InputElement.InstanceDataStepRate = 1
            _InputElement.Classification = InputClassification.PerInstanceData
        Else
            _InputElement.InstanceDataStepRate = 0
            _InputElement.Classification = InputClassification.PerVertexData
        End If
        If nnn.SemanticName = "SHID" Then
            _InputElement.Slot = 1
            _InputElement.AlignedByteOffset = InputElement.AppendAligned
            _InputElement.InstanceDataStepRate = 1
            _InputElement.Classification = InputClassification.PerInstanceData
        End If
        If nnn.UsageMask = 1 Then
            If nnn.ComponentType = RegisterComponentType.UInt32 Then
                _InputElement.Format = Format.R32_UInt
            ElseIf nnn.ComponentType = RegisterComponentType.SInt32 Then
                _InputElement.Format = Format.R32_SInt
            ElseIf nnn.ComponentType = RegisterComponentType.Float32 Then
                _InputElement.Format = Format.R32_Float
            End If
        ElseIf nnn.UsageMask <= 3 Then
            If nnn.ComponentType = RegisterComponentType.UInt32 Then
                _InputElement.Format = Format.R32G32_UInt
            ElseIf nnn.ComponentType = RegisterComponentType.SInt32 Then
                _InputElement.Format = Format.R32G32_SInt
            ElseIf nnn.ComponentType = RegisterComponentType.Float32 Then
                _InputElement.Format = Format.R32G32_Float
            End If
        ElseIf nnn.UsageMask <= 7 Then
            If nnn.ComponentType = RegisterComponentType.UInt32 Then
                _InputElement.Format = Format.R32G32B32_UInt
            ElseIf nnn.ComponentType = RegisterComponentType.SInt32 Then
                _InputElement.Format = Format.R32G32B32_SInt
            ElseIf nnn.ComponentType = RegisterComponentType.Float32 Then
                _InputElement.Format = Format.R32G32B32_Float
            End If
        ElseIf nnn.UsageMask <= 15 Then
            If nnn.ComponentType = RegisterComponentType.UInt32 Then
                _InputElement.Format = Format.R32G32B32A32_UInt
            ElseIf nnn.ComponentType = RegisterComponentType.SInt32 Then
                _InputElement.Format = Format.R32G32B32A32_SInt
            ElseIf nnn.ComponentType = RegisterComponentType.Float32 Then
                _InputElement.Format = Format.R32G32B32A32_Float
            End If
        End If
        Elements.Add(_InputElement)
    Next

    If Elements.Count > 0 Then
        Dim newinputlayout As New InputLayout(MainDevice, _shader, Elements.ToArray)
        blah.Dispose()
        Return newinputlayout
    Else
        Return Nothing
    End If

End Function

That will pull it all out of the shader and spit out an input layout no need to do it by hand just set in your shader and let this do the rest it also checks for a WORLD tag so u can pass a world matrix to the vertex shader for instancing. Mono game may let you get at all that stuff but it may be hidden under all there abstraction but that bit of code saves all this tweaking for different input strucs.

Try this:

Texture2D colorMap;
sampler colorSampler = sampler_state
{

AddressU = Clamp;
AddressV = Clamp;
Filter = MIN_MAG_MIP_POINT;
};



struct VertexShaderOutput
{
float4 Position : SV_Position;
float2 TexCoord : TEXCOORD0;

};
struct PixelShaderOutput
{
float4 Color : SV_Target0;

};



// outputs a full screen triangle with screen-space coordinates
// input: three empty vertices
VertexShaderOutput VSQuad( uint vertexID : SV_VertexID )
{
VertexShaderOutput result;
result.TexCoord = float2((vertexID << 1) & 2, vertexID & 2);
result.Position  = float4(result.TexCoord  * float2(2.0f, -2.0f) + float2 -1.0f, 1.0f), 0.0f, 1.0f);
return result;
}


PixelShaderOutput PShader(VertexShaderOutput input) 
{
 PixelShaderOutput output = (PixelShaderOutput)1;
float4 Color = colorMap.Sample(colorSampler, input.TexCoord);
output.Color =Color;
output.Color.a=1;
 return output;
}


technique11 Render
{
pass P0
{

    SetVertexShader( CompileShader( vs_5_0, VSQuad() ) );
    SetPixelShader( CompileShader( ps_5_0, PShader() ) );
}
}