RGB to xyY color space conversion and luminance

Question

The luminance calculated by following GLSL functions (fragment shaders - tonemap) has different value:

float GetLuminance (vec3 rgb)
{
     return (0.2126 * rgb.x) + (0.7152 * rgb.y) + (0.0722 * rgb.z);
}


vec3 RGB2xyY (vec3 rgb)
{
     const mat3 RGB2XYZ = mat3(0.4124, 0.3576, 0.1805,
                               0.2126, 0.7152, 0.0722,
                               0.0193, 0.1192, 0.9505);
    vec3 XYZ = RGB2XYZ * rgb;

    return vec3(XYZ.x / (XYZ.x + XYZ.y + XYZ.z),
                XYZ.y / (XYZ.x + XYZ.y + XYZ.z),
                XYZ.y);
}

I used a glm library to calculate an example result. For glm::vec3(2.0f, 3.0f, 8.0f) GetLuminance returns 3.1484. RGB2xyY returns glm::vec3 which z component is equal 3.8144. What is wrong ?

Answer 1

Either the matrix is transposed (i.e. rows and columns swapped) in RGB2xyY, or the equation in GetLuminance uses the row vector instead of the column vector. The two values just happen to be close by accident.

Here is a Matlab trace showing whats going on. The quote is used to indicate matrix transposition. (The multiplication below is swapped around, of course you could also do s*v' and s'*v').

BTW, using tools like Matlab (or Octave) are great for quickly prototyping these types of algorithms and finding bugs like these fast.

>> s = [0.4124, 0.3576, 0.1805;
        0.2126, 0.7152, 0.0722;
        0.0193, 0.1192, 0.9505] %assign matrix to s

s =

    0.4124    0.3576    0.1805
    0.2126    0.7152    0.0722
    0.0193    0.1192    0.9505

>> v = [2 3 8] % assign vector to v

v =

     2     3     8

>> v*s         % multiply

ans =

    1.6170    3.8144    8.1816

>> v*s'        % multiply with transposed matrix

ans =

    3.3416    3.1484    8.0002