Infinite Seamless Background HLSL Shader in Monogame doesn't work with non square textures

Question

I'm trying to write a shader that draws a seamless background infinitely in Monogame.

sampler TextureSampler : register(s0);
float2 ViewportSize;
float4x4 ScrollMatrix;
struct VertexToPixel {
    float4 Position : SV_Position0;
    float4 TexCoord : TEXCOORD0;
    float4 Color : COLOR0;
};
VertexToPixel SpriteVertexShader(float4 color : COLOR0, float4 texCoord : TEXCOORD0, float4 position : POSITION0) {
    VertexToPixel Output = (VertexToPixel)0;

    // Half pixel offset for correct texel centering.
    position.xy -= 0.5;

    // Viewport adjustment.
    position.xy = position.xy / ViewportSize;
    position.xy *= float2(2, -2);
    position.xy -= float2(1, -1);

    // Transform our texture coordinates to account for camera
    texCoord = mul(texCoord, ScrollMatrix);

    //pass position and color to PS
    Output.Color = color;
    Output.Position = position;
    Output.TexCoord = texCoord;

    return Output;
}

technique SpriteBatch {
    pass {
        VertexShader = compile vs_2_0 SpriteVertexShader();
    }
}

I'm using Monogame.Extended so I can use their Camera2D class. I have this function to draw my texture:

public static void drawSeamlessBackground(SpriteBatch s, Texture2D t, GraphicsDevice gd, float parallax, Camera2D cam) {
    //TODO: Make this work with non square textures.
    Vector2 textureSize = new Vector2(t.Width, t.Height);
    Rectangle view = gd.Viewport.Bounds;
    Matrix m = Matrix.CreateTranslation(new Vector3(-cam.Origin / textureSize, 0.0f)) *
                Matrix.CreateScale(1f / cam.Zoom) *
                Matrix.CreateRotationZ(-cam.Rotation) *
                Matrix.CreateTranslation(new Vector3(cam.Origin / textureSize, 0.0f)) *
                Matrix.CreateTranslation(new Vector3((cam.Position * parallax) / textureSize, 0.0f));

    infiniteShader.Parameters["ScrollMatrix"].SetValue(m);
    //Normally this next line would not be here since the viewport doesn't usually change.
    infiniteShader.Parameters["ViewportSize"].SetValue(new Vector2(view.Width, view.Height));

    s.Begin(samplerState: SamplerState.LinearWrap, effect: infiniteShader);
    s.Draw(t, new Vector2(0, 0), view, Color.White);
    s.End();
}

This whole thing works fine for square textures, but when the width and the height are different, there's distortion that happens when I rotate my camera.

In the next image, there are two textures, one that is square and one that is a rectangle.

Here is what it looks like with no rotation:

Here is what it looks like with some rotation:

The red texture has square dimensions (500x500). The white texture has rectangular dimensions (1280x720).

I'm sure this is a simple fix, but I can't figure it out.

Answer 1

After working on this for a while, I found a solution. Just like someone else said in a comment, a shear happens when scaling is done before a rotation. It turns out my texCoord had the wrong ratio. It needed to use the texture's ratio. The shader is perfectly fine, but I need to apply the texture's ratio to my matrix before the rotation.

public static void drawSeamlessBackground(SpriteBatch s, Texture2D t, GraphicsDevice gd, float parallax, Camera2D cam) {
    Vector2 textureSize = new Vector2(t.Width, t.Height);
    Rectangle view = gd.Viewport.Bounds;

    Vector2 textureRatio = new Vector2(t.Width / t.Height, 1);
    if (t.Width < t.Height) {
        textureRatio = new Vector2(1, t.Height / t.Width);
    }

    Matrix m = Matrix.CreateTranslation(new Vector3(-cam.Origin / textureSize, 0.0f)) *
                Matrix.CreateScale(1f / cam.Zoom) *
                Matrix.CreateScale(textureRatio.X, textureRatio.Y, 1) *
                Matrix.CreateRotationZ(-cam.Rotation) *
                Matrix.CreateScale(1f / textureRatio.X, 1f /textureRatio.Y, 1) *
                Matrix.CreateTranslation(new Vector3(cam.Origin / textureSize, 0.0f)) *
                Matrix.CreateTranslation(new Vector3((cam.Position * parallax) / textureSize, 0.0f));

    infiniteShader.Parameters["ScrollMatrix"].SetValue(m);
    infiniteShader.Parameters["ViewportSize"].SetValue(new Vector2(view.Width, view.Height));

    s.Begin(samplerState: SamplerState.LinearWrap, effect: infiniteShader);
    s.Draw(t, new Vector2(0, 0), view, Color.White);
    s.End();
}

Because of rounding errors though, it's best to use textures that have nice ratios. For example, if I have a texture that is 1920x1080 in size (ratio of 1,777777777777778), I will still get shearing, but if I make that 2000x1000 (ratio of 2), I don't get any.