# How can I use an HLSL pixel shader to peek at other pixels?

I think this is like a pixilation shader but I had a hard time trying to find one online, and one that I could manipulate specifically.

I want to create a pixel shader for my monogame project that splits the texture up into blocks. The width and height of the block may or may not match. Then on each block, I want to average out the color of all pixels in the block, and fill it in.

I'm not sure how I would do this in a pixel shader. I can do something like this in code pretty easily but that is because the code scans the entire image at once. A pixel shader though seems to run on each pixel, so I'm not sure how to do that.

I currently have a simple pixel shader where I can plug in code to test. It doesn't do anything right now but return the same color.

#if OPENGL
#define SV_POSITION POSITION
#else
#endif

// Our texture sampler
texture Texture;
sampler TextureSampler = sampler_state
{
Texture = <Texture>;
};

// This data comes from the SpriteBatch Vertex Shader
{
float4 Position : POSITION;
float4 Color : COLOR0;
float2 TextureCordinate : TEXCOORD0;
};

{
float4 color = tex2D(TextureSampler, input.TextureCordinate);
return color;
}

technique Technique1
{
pass Pass1
{
}
}


You could render your image to a lower resolution rendertarget with point sampling to achieve a pixelation effect. This way you could use a simple pass-through pixels shader but the sampling of the texture alone would make it pixelated.

An other way would be a compute shader. You could dispatch it with the number of desired blocks, and for each block average imageresolution/blockCount number of pixels in a sqare.

• Doing the lower resolution solution does work when you're using a full 3d scene, however with a SpriteBatch it doesn't. Since the sprites use a real screen pixel coordinate system, all sprites rendered are actually clipped out. – Thraka May 15 '16 at 19:42
• If you render the sprites to full res, than you can render the resulting image to a lower res target i suppose. – János Turánszki May 15 '16 at 23:02
• True. Thanks for the suggestion but I would love to find a shader way to do it. The problem with this way is that some of the blurriness of an edge pixel ends up getting lost when you don't use an average calculation. – Thraka May 16 '16 at 4:01

I would abandon the though of perfectly averaging all pixels in a block. Instead, sample the color from say 3x3 underlying (but not neighbouring) pixels, while using biliear filtering. This is only 9 texture lookups per final painted pixel - not at all a big job for the GPU. As opposite to per-painted-pixel sampling all pixels in a block and averaging. Say a block is 30x30 pixels. If the fullscreen picture is 2 million pixels, that would be 2mil x 30 x 30 = 1.8 billion lookups per painted frame. Too much.

Sampling 9 pixels 2 million times is only 18 million lookups. Feasible, and the final color of a block wouln't be too far away from the average.

You achieve this by "snapping UV-addresses onto grid". If your current U-address (for the pixel currently being painted) is 0.701, you'd first snap that into 0.7. Same with 0.699, 0.702, 0.703, etc., up to the snap-to-grid resolution.

Then you add (-0.01, 0, 0.01) to 0.7, do the lookup, calculate the average. Hence you always have a fixed place for the lookup, for all pixels in the current block -> all pixels in the block get the same color. The next block then does the lookup on 9 different pixcels, etc.

You round 0.709 to 0.7 by: x = 0.1 * floor(0.5 + x * 10)

You round 0.709 to 0.71 by: x = 0.01 * floor(0.5 + x * 100)

If you want to make the pixelation a bit "restless", you can do a small lookup adjustment from the color of the center pixel in the lookup. Dark = i bit to left, light = a bit to right. Or something similar.

If not using this technique, you need another help surface for storing the averages, and then looking up colors for the blocks from that surface. That surface would be 1px high and [number of blocks] wide. But that means much more work.

The 0.01, 0.701 etc are just sample values. You'd have to use 1/[number of blocks horizontally, vertically] to get the suitable snap resolution.

Maybe even a 2x2 px lookup would do. You must try.

If you want to pixelate only a specific area, you need a control texture, for example grayscale, preferrably with soft edged shapes. Then, in the pixelate shader, you do a lookup of the exact pixel from the source image, and separately calculate the average from the source, as mentioned above. Then you lookup the control texture, and lerp between to two earlier, by the control textures brightness, or some other element of it that controls the end result. Ie. using texture splattering.

• You can get a bit more averaging out of this technique by using bilinear filtering and snapping your sample coordinates to the corners between texels — then the texture filtering hardware does 4 texel averages for you with every sample. With just 4 samples you can get the average of 16 texels, typically with substantially better performance than 16 separate samples. :) – DMGregory Jun 17 '16 at 3:15
• Excellent greedy performance-oriented mostbangforleastbucks geek-thinking! :-) – Stormwind Jun 17 '16 at 5:01
• To get the average colour of a texture, just generate mipmaps and sample from the 1x1 miplevel. – Maximus Minimus Sep 30 '16 at 16:13
• Sure, but that's not what he asked. He wants to average the pixels per multiple blocks, not for the entire texture. The blocks need not necessarily be 2x, 4x, 8x ...[size/2]x a pixel of the texture, and the pixelated part may perhaps move (u know, like when scrambling someones face only). Additionally, it's not even given that it's an external texture, saved in a file - generating mipmaps is usually an off-line action, which consumes much processing. Hence i imagine this requires a bit more work :-). – Stormwind Oct 1 '16 at 17:15