Combining palette-swaps and smooth scaling in a pixel-art shader

Question

I am creating a pixel-art game in MonoGame, and have written a shader to perform various effects on a sprite when rendering it, such as performing palette swaps, highlighting outlines, etc.. Most of these effects work by comparing the color of a texture coordinate with a reference color, and if they match then the color is swapped for another. Simple stuff as long as you use nearest-neighbor filtering on the texture. (These effects are used in a very dynamic way, which is why they are done using a shader instead of, say, a separate texture-atlas for each color palette).

However, in the game there is quite a lot of floating-point camera scaling and movement going on, resulting in choppy-looking visuals when using nearest-neighbor filtering. Luckily there is a way of scaling pixel-art that looks much smoother without taking away the crispness, which is to use a texture with bilinear filtering enabled but only use interpolated colors on the boundary between pixels. Here and here are a couple of open-source shaders using this technique.

The problem is of course that you cannot do palette-swaps on a texture with bilinear filtering because the colors are all blended together. In other words the two aforementioned techniques seem to be completely incompatible with one another, yet I really need both of them.

As far as I can tell the logical solution would be to perform the palette-swapping and similar effects on the texture before scaling and filtering it, however I have no idea of how to actually do this from within a shader, and what the performance implications of doing so would be.

I am still very much a novice when it comes to shaders, so any help here would be deeply appreciated!

Answer 1

I'll be writing GLSL because I'm more familiar with it, and it is easier for me to test.

This is what I have been able to come up with:

1. Split the uv coordinates into integer and fractional and pixels:

   vec2 fuv = floor(uv / pixel_size);
   vec2 wuv = fract(uv / pixel_size);
   

   Here I'm assuming uv goes from (0.0, 0.0) to (1.0, 1.0), and pixel_size is the inverse of the size of the texture in pixels.

2. We are going to query the four pixels around the uv and apply pallet swapping to the colors we get:

   vec4 lox_loy = palette_swap(texture(TEXTURE, (fuv + vec2(0.0, 0.0)) * pixel_size));
   vec4 lox_hiy = palette_swap(texture(TEXTURE, (fuv + vec2(0.0, 1.0)) * pixel_size));
   vec4 hix_loy = palette_swap(texture(TEXTURE, (fuv + vec2(1.0, 0.0)) * pixel_size));
   vec4 hix_hiy = palette_swap(texture(TEXTURE, (fuv + vec2(1.0, 1.0)) * pixel_size));
   

   I'm, of course, assuming you have a palette_swap function that takes a vec4 and outputs a vec4. This would allow you to keep the approach you are currently using, or change if you need.

   I'm also assuming nearest-neighbor filtering. We are going to do bilinear filter later.

   For reference, this is how you would query at uv:

   vec4 color = palette_swap(texture(TEXTURE, uv));
   

   And I'm making new coordinates to query the texture (i.e. to replace uv in the above statement). I start with fuv, but since fuv is in pixels we need to multiply by pixel_size:

   fuv * pixel_size
   

   And I want to offset it by whole pixels, so I would something like this:

   fuv * pixel_size + vec2(off_x, off_y) * pixel_size
   

   As you can see pixel_size is a common factor, so we rewrite that:

   (fuv + vec2(off_x, off_y)) * pixel_size
   

3. Now we are going to apply the sharpen algorithm from Beefster09 that you linked, but only to the fractional part of the pixel coordinates:

   vec2 norm = (wuv - 0.5);
   vec2 norm2 = norm * norm;
   wuv = norm * vec2(pow(norm2.x, sharpness), pow(norm2.y, sharpness)) + 0.5;
   

   This code assumes sharpness is a uniform float, and I remind you that wuv is a vec2.

4. Then we apply simple bilinear interpolation using the result as weights:

   vec4 output =  mix(mix(lox_loy, hix_loy, wuv.x), mix(lox_hiy, hix_hiy, 
   wuv.x), wuv.y);
   

   Since we are interpolating the colors after the palette swap there should not introduce artifacts.

And that is our output color.

---

By the way, for testing this I came up with a different palette swap approach. I decided to use the corners of the RGB cube as reference (so, up to 8 colors), and apply a different color from the palette to each quadrant. Plus a cutoff for alpha. It looks like this:

vec3 hi = step(vec3(0.5), input.rgb);
vec3 lo = 1.0 - hi;
vec3 output = palette[0].rgb * lo.r * lo.g * lo.b
            + palette[1].rgb * lo.r * lo.g * hi.b
            + palette[2].rgb * lo.r * hi.g * lo.b
            + palette[3].rgb * lo.r * hi.g * hi.b
            + palette[4].rgb * hi.r * lo.g * lo.b
            + palette[5].rgb * hi.r * lo.g * hi.z
            + palette[6].rgb * hi.r * hi.g * lo.z
            + palette[7].rgb * hi.r * hi.g * hi.z;
return vec4(output.rgb, step(0.5, input.a));

Here hi will have components of value 0.0 or 1.0 depending if the components of input are lower or higher than 0.5. And lo will be the opposite (0.0 on the components where hi is 1.0, and viceversa). The products of the components of lo and hi allow me to make factors that are 1.0 in only one quadrant of the RGB cube, and by doing that I'm giving a different color from the palette to each quadrant.

Answer 2

I had a few more hours to think about the problem before Theraot provided their excellent answer, and came up with a different (albeit much more naive) solution that may also be worth describing. The idea is to create a RenderTarget2D with a resolution large enough to fit all sprites drawn in a frame. Then draw each sprite onto the render target at a 1:1 scale, and at the same time apply the palette-swapping and similar pixel-effects (of course keeping track of where you place them). Afterwards you upscale them and render them onto the screen one-at-a-time.

When compared to Theraot’s solution this approach has one obvious downside, which is that you need to draw every sprite twice. However, you only need to apply the palette swap effect one time per pixel as opposed to four times per fragment, which might add up if the palette-swap (etc.) function is slow enough. Another downside with this approach is that, unless your sprites all have similar dimensions, it will be a challenge to find an efficient way of packing them onto the render target.