1
\$\begingroup\$

I am trying to update a shader from ps_1_3 inline assembly to ps_2_a HLSL. However the HLSL shader causes things to render differently when using ps_2_a. When using my HLSL conversion with ps_1_3 it works same way as with the inline assembly.

Original assembly

Sampler[0] = (DiffuseMapSampler);
pixelShader = asm
{
    ps_1_3
    tex t0
#ifdef _POINTLIGHT_
    add r0.rgb, v0, v0
    +mov r0.a, v0.a
    mul_x4 r0, t0, r0
#else
    mul_x4 r0, t0, v0
#endif
#if defined(OVERGROWTH) && HASALPHA2MASK
    mul_x2 r0.a, r0.a, t0.a
#endif
};

Sampler and VS_OUTPUT

struct VS_OUTPUT
{
    float4 Pos  : POSITION0;
    float2 Tex0 : TEXCOORD0;
#if _HASSHADOW_
    float4 TexShadow    : TEXCOORD1;
#endif
    float4 Color  : COLOR0;
    float Fog   : FOG;
};

texture DiffuseMap;
sampler DiffuseMapSampler = sampler_state
{
    Texture = (DiffuseMap);
    MipFilter = LINEAR;
    MinFilter = LINEAR;
    MagFilter = LINEAR;
    AddressU  = WRAP;
    AddressV  = WRAP;
    MipMapLodBias = 0;
};

My HLSL translation

float4 basicPixelShaderNoShadow(VS_OUTPUT VsOut) : COLOR
{
    float4 diffuseMap = tex2D(DiffuseMapSampler, VsOut.Tex0);
    float4 outCol;
#ifdef _POINTLIGHT_
    outCol.rgb = VsOut.Color.rgb + VsOut.Color.rgb;
    outCol.a = VsOut.Color.a;
    outCol = (diffuseMap * outCol) * 4;
#else
    outCol = (diffuseMap * VsOut.Color) * 4;
#endif

#if defined(OVERGROWTH) && HASALPHA2MASK
    outCol.a = (outCol.a * diffuseMap.a) * 2;
#endif
    return outCol;
}

And this is the result in game enter image description here For some reason when using ps_2_a the colours are oversaturated.

Edit: HLSL that fixed the issue:

float4 vertexColor = float4(VsOut.Color.rgb, VsOut.Color.a*2);
float4 diffuseMap = tex2D(DiffuseMapSampler, VsOut.Tex0);

float4 outCol = diffuseMap * vertexColor * 2;

#if defined(OVERGROWTH) && HASALPHA2MASK
outCol.a *= 2 * diffuseMap.a;
#endif

return outCol;

dont quite understand why, but it works :)

\$\endgroup\$
1
\$\begingroup\$

I haven't (yet) found definitive confirmation of this (and it's likely lost in the mists of time, in ancient documentation that no longer exists), but SM1.3 is clearly clamping intermediates to 1.0 whereas SM2 is not.

So, for example, where you have:

outCol.rgb = VsOut.Color.rgb + VsOut.Color.rgb;

And:

outCol = (diffuseMap * outCol) * 4;

If the result of any of these exceeds 1.0, SM1.3 is clearly clamping it. To replicate the SM1.3 behaviour in SM2 you could use e.g. the clamp or max intrinsics.

\$\endgroup\$
4
  • \$\begingroup\$ I tried using min, clamp and saturate to clamp it to a maximum of 1, but it always looked the same. The only thing I could find about clamping on the r registers is the range table on the bottom of this article. There it states those have minimum range of -1/+1 in 1.3 and -8/+8 in 1.4. I tested the HLSL in 1.4 and it works there, so dont think it is clamping due to range here \$\endgroup\$
    – Mats391
    Jan 23 at 22:35
  • \$\begingroup\$ @Mate391 - I suggest checking the generated assembly for the SM2 version. You're definitely getting an intermediate clamped, I'm 100% positive on that, but the trick is to find which intermediate. It's not always obvious from the HLSL. I'd pay particular attention to the HLSL that should generate those mul_x4 lines because it's a high chance it's in there. \$\endgroup\$ Jan 24 at 0:58
  • \$\begingroup\$ I played around with it some more and finally got a solution. I added that solution to the question. Still dont really understand why, but it works :D \$\endgroup\$
    – Mats391
    Jan 24 at 11:53
  • \$\begingroup\$ @Mats391 Solutuons should go in Answers, not in Questions. \$\endgroup\$
    – DMGregory
    Jan 24 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.