Putting two R16_UNORM in a R32_UINT

Question

With my hardware limitation for CS5 I'm using R32_UINT in my shaders to store compacted colors and normasl as 555 bits each. My packing is using uint3 and the result is of poor quality. I'm planning to store the color as 565 bits and normals as 77 bits + 2 bits, one for a sign (of normal z) and one for a mask, but in a R16_UNORM each that gives me better quality and pack the two in my R32_UINT.

below the packing/unpacking funcs for float3 color to 565 bits as R16_unorm and back that works with correct quality in my other shaders. I also added the expected coding for the Normals to Nxy 77 bits

float C3_R16(float3 Color)
{
   uint3 uColor = uint3(min(Color,float3(1.0, 1.0, 1.0))*float3(31.0, 63.0, 31.0));
   return float(uColor.r | (uColor.g<<5) | (uColor.b<<11))/65535.0;
}

float3 R16_C3(float Pack)
{
    uint iPack = uint(Pack);
    return float3(iPack&0x1F, (iPack>>5)&0x3F, (iPack>>11)&0x1F)/float3(31,63,31);
}

float N2_R16(float3 Normal)//unchecked behavior
{
   uint2 uNormal = uint2(min(Normal.rg,float2(1.0, 1.0))*float2(127.0, 127.0));
   return float(uNormal.r | (uNormal.g<<7))/65535.0;
}

float3 R16_N3(float Pack)//unchecked
{
    uint iPack = uint(Pack*65535);
    float3 N; 
    N.xy = float2(iPack&0x7F, (iPack>>7)&0x7F)/float2(127,127)*2-1;//maybe optimze with /float2(63.5,63.5)-1;
    N.z = sqrt(1-dot(N.xy,N.xy));//in view space I should not need sign 
    return N;
}

The Normal 77 (N2_R16) enoding is to show how I will proceed for the Nxy encoding/decoding. The two final bits are added with a test and bitshift like uint(N.z<0)<<15.

Now to convert two R16 into a R32_UINT I'm planning to use asuint function this way:

uint ColorOut = (asuint(C3_R16(mycolor.rgb))&0x00EFFFFF)>>7;
//uint NormalOut = (asuint(C2_R16(mynormal.rg))&0x00EFFFFF)>>7;//this may be optimized to avoid latter <<16
uint MyFinalR32_UINT = ColorOut;//| NormalOut<<16; currently only one R16 is added for the color

In this code I'm storing only one R16 at the moment to check how it works.

For unpaking I'm doing this:

  uint Data = txDiffuseInt.Load(uint3(Input.Tex.x*960, Input.Tex.y*540,0));
  float3 C = R16_C3(float(Data&0x0000FFFF));
    

According to IEEE rules the *0x00EFFFFF mask considers that the 9 msb are the sign and 8 bit exponent and the lower 23 bit the mantissa. As the value reulting from C3_R16 must be in the range [0..1] I should not need the sign and exponent. The >>7 is to pick the 16 MSB bit of the mantissa (again if I'm correct these are the most significant bits). This is were things go wrong I think. My colors are mostly blue with some green on unpacking. Some objects should be full red.

Is my description of float32 well-used?

Answer 1

First answer

The definition of float 32 I found was not correct on 1 bit. Correct definition is Bit 32 sign Bit 31-25 exponent Bit 24-1 mantissa

Using asfloat/asuint functions you can have the float and uint version of your data like:

0x3f800000 =  1.000
0x3dcccccd =  0.100
0xbf800000 = -1.000

Regarding the exponent, as we manipulate bits it is not linear. So for instance below are some ranges of value and the corresponding exponent expressed as their value and binary:

63 0011 1111 1.0 - 0.5
62 0011 1110 0.4 - 0.2
61 0011 1101 0.1 - 0.04
60 0011 1100 0.03 - 0.008
59 0011 1011 0.007 - 0.002 
58 0011 1010 0.001 - 0.0005
57 0011 1000 0.0004 - 0.0003

If I consider using a unorm float in the range [0-1] and that values below 0.0003 are not useful, then I don’t need the bit sign 32 and the exponent bits 31-28 are constant. In the range we have always bit 31=0, 30=1, 29=1, 28=1. So I can decide to store only bit 27-25 for my packed data. In the range [0.008-1] you can even store only two bits.

Using a 16 bit format this leaves me 13 bits for the mantissa. The 32 float has 24 bits mantissa with higher bit being the most significant so we need to shift 11 bit left to store these 13 significant bits in our remaining R16 13 bits. Because the exponent bit and the mantissa part are on following bits 27-9 we end up with this single operation tha tremoves the sign and upper bit 31-28 exponent:

float My32Float = value;
uint Myuint32Value = asuint(My32Float);
uint MyPackedUint16Value = (Myuint32Value & 0x07FFFFFF)>>11;

Unpacking is like this:

uint MyPackData = value;//may come from texture 32_UNIT;
float MyUnpackedFloat = asfloat(0x38000000 | (MyPackData << 12));

The 0x38000000 mask restores the missing exponent bits that are constant in the range of values.

I have checked the presision that is good. Zero is restored as 0.0000031 so a test can be needed if zero is really wanted.

Second answer

I have below a couple of (un)packing methods that works quite fine but except the 565 packing the others have some artifacts depending on the POV (see picture). Some help wanted to solve that.

Except the 565 that stores the 3 normal components, the other methods store only Nxy and calculate back z with a sign for the Lambert and without sign for the 88 and 88as1F. In my view space the z sign appeared to be always negative on flat as well as on bumpy surfaces.

Some code to pack normal in R16 formats (select the method by moving the comment)

//#define SM_NPACK_565   //pack float3 as 565 bit format
//#define SM_NPACK_88    //pack float3 as xy 88 bit format
//#define SM_NPACK_871   //pack float3 as xy 87 bit format, 1 bit sign (if needed)
//#define SM_NPACK_88LEA //pack float3 as xy 88 bit format using Lambert Azimuthal Equal Area
#define SM_NPACK_88as1F_UNORM   //pack float3 as xy in single float as floor(x)+frac(y). Use the above uint32 to uint16 final packing

uint N3_R16(float3 Normal)
{
#ifdef SM_NPACK_565
    Normal = Normal*0.5+0.5;
    uint3 uColor =  uint3(Normal*float3(31,31,63));
    return uColor.r| (uColor.g<<5)| (uColor.b<<10);
#endif
#ifdef SM_NPACK_88
   uint2 uColor =  uint2(Normal.xy*127.5+127.5);
   return uColor.r| (uColor.g<<8);
#endif
#ifdef SM_NPACK_871
    uint2 uColor =  uint2((Normal.xy*0.5+0.5)*float2(255,127));
    return uColor.r| (uColor.g<<8)| ((Normal.z<0)<<15);
#endif
#ifdef SM_NPACK_88LEA
    float p = sqrt(Normal.z*8+8);
    uint2 uColor =  uint2(Normal.xy*255.0/p+127.5f);
    return uColor.r| (uColor.g<<8);
#endif
#ifdef SM_NPACK_88as1F_UNORM
    float2 Data = Normal.xy*0.5+0.5;
    float fValue = (floor(Data.x*255.0)+Data.y*0.99)/256.0;    //ensure that the final value is 255.9999 max with 255 max for x and 0.999 max for y.
    return (asuint(fValue) & 0x07FFFFFF) >> 11;
#endif
}

float3 R16_N3(uint Pack)
{
    float3 N;
#ifdef SM_NPACK_565
    N = float3( Pack & 0x001F, (Pack >> 5 )& 0x001F , (Pack >> 10 )& 0x003F )/ float3(15.5,15.5,31.5) -1;
#endif
#ifdef SM_NPACK_88
    N.xy = float2( Pack & 0x00FF, (Pack >> 8 )& 0x00FF)*INV255x2-1;
    /*if( N.x+N.y>1.95f) N.z = 1; else*/ N.z = -sqrt(1-dot(N.xy,N.xy));
#endif
#ifdef SM_NPACK_871
    N.xy = float2( Pack & 0x00FF, (Pack >> 8 )& 0x007F)/ float2(127.5,63.5) -1;
    int s = (Pack & 0x8000)!=0?-1:1;
    /*if( N.x+N.y>1.95f) N.z = 1; else*/ N.z =s*sqrt(1-dot(N.xy,N.xy));
#endif
#ifdef SM_NPACK_88LEA
    N.xy = float2( Pack & 0x00FF, (Pack >> 8 )& 0x00FF)*INV255;
    float2 fenc = N*4-2;
    float f = dot(fenc,fenc);
    N = float3( fenc*sqrt(1-f*0.25), 1-f*0.5);
#endif
#ifdef SM_NPACK_88as1F_UNORM
    float f = asfloat(0x38000000 | (Pack << 11))*256.0;
    N.x = floor(f);
    N.y = f-N.x;
    N.x *= 1.0/255.0;
    N.xy = N.xy*2-1;
    N.z = -sqrt(1-dot(N.xy,N.xy));
#endif
    return N;
}

The line test /if( N.x+N.y>1.95f) N.z = 1; else/ is used when I have unlit object. For those I set arbitrary normals to (1,1,1). As normalized normals are never >= 1.0 (or <=-1) I use it to skip lighting in my shaders. Whe nthis line aer not active the objects appear full black or full white depending on the packing method.

The picture with artifacts

After some search I found that clamping the normal to (-0.99, 0.99) solved the issue for method 88LEA. There might be a precision issue with these calculations. For other methods used to store only Nxy they are still not fully correct on any POV.