How can I replicate the color limitations of the NES with an HLSL pixel shader?

Question

So since 256 color mode is depreciated and no longer supported under Direct3D mode, I got the idea to use a pixel shader instead to simulate the NES palette of all possible colors so that fading objects and whatnot don't have smooth fade outs with alpha channels. (I know objects couldn't really fade out on the NES, but I have all objects that do fade in and out on a solid black background, which would be possible with palette swapping. Also, the screen fades in and out when you pause which I know also was possible with palette swapping as it was done in a few Mega Man games.) Problem is, I know next to nothing about HLSL shaders.

How do I do it?

Answer 1

In the pixel shader you could pass in a 256x256 Texture2D with the pallet colors all lined up horizontally in a row. Then your NES textures would be converted to direct3D Texture2Ds with ever pixel converted to a 0-255 index value. There is a texture format that only uses the red value in D3D9. So the texture would only take up 8bits per pixel, but the data that comes into the shader would be from 0-1.

// Pixel shader might look like this:

float4 mainPS() : COLOR0
{
    float4 colorIndex = tex2D(MainTexture, uv);
    float4 palletColor = tex2D(PalletTexture, float2(colorIndex.x, 0);
    return palletColor;
}

EDIT: A more correct way would be to add in all the blending pallet version you need aligned vertically in the texture and referencing them with your colorIndex's 'alpha' value:

float4 mainPS() : COLOR0
{
    float4 colorIndex = tex2D(MainTexture, uv);
    float4 palletColor = tex2D(PalletTexture, float2(colorIndex.x, colorIndex.a);
    return palletColor;
}

A third way would be to just fake the NES low fade quality by toon shading the alpha color:

float4 mainPS() : COLOR0
{
    float4 colorIndex = tex2D(MainTexture, uv);
    float4 palletColor = tex2D(PalletTexture, float2(colorIndex.x, 0);
    palletColor.a = floor(palletColor.a * fadeQuality) / fadeQuality;
    //NOTE: If fadeQuality where to equal say '3' there would be only 3 levels of fade.
    return palletColor;
}

Answer 2

If you don't really care about texture memory usage (and the idea of blowing an insane amount of texture memory to achieve a retro look has a kind of perverse appeal) you could build a 256x256x256 3d texture mapping all RGB combinations to your selected palette. Then in your shader it just becomes one line of code at the end:

return tex3d (paletteMap, color.rgb);

It may not even be necessary to go all the way to 256x256x256 - something like 64x64x64 may be sufficient - and you could even change palette mappings on the fly using this method (but at significant cost owing to a large dynamic texture update).

Answer 3

(both of my solution works only if you dont care about changing palletes on the fly using shaders)

You can use any type of texture and do just a simple computation on a shader. Trick is that you have more color information than you need, so what you will just get rid of information that you dont want to.

8bit color is in formar RRRGGGBB. Which gives you 8 shades of red and green and 4 shades of blue.

This solution will work for any RGB(A) color format textures.

float4 mainPS() : COLOR0
{
    const float oneOver7 = 1.0 / 8.0;
    const float oneOver3 = 1.0 / 3.0;

    float4 color = tex2D(YourTexture, uvCoord);
    float R = floor(color.r * 7.99) * oneOver7;
    float G = floor(color.g * 7.99) * oneOver7;
    float B = floor(color.b * 3.99) * oneOver3;

    return float4(R, G, B, 1);
}

note: i wrote that from the top of my head, but im really sure it will compile and work for you

---

Another possibility, would be to use D3DFMT_R3G3B2 texture format that is actually same as 8bit graphics. When you put data to this texture, you can use simple bit operation per byte.

tex[index] = (R & 8) << 5 + ((G & 8) << 2) + (B & 4);