Aspect ratio of drawn quad messed up after rotating

Question

When I draw a quad that is rotating the aspect ratio of the quad gets messed up and the size changes.

Gif of what is happening

I am confident it has something to do with the way I calculate the size because that is relative to the width and height of the screen, I just don't know why the aspect ratio would change since the aspect ratio of the screen doesn't change.

Code of transformationMatrix creation:

        Matrix4f matrix = Maths.createTransformationMatrix(
                new Vector2f(
                        gui.getPosition().getX() / Display.getWidth() * 2 - 1 + gui.getSize().getX() / Display.getWidth(),
                        -gui.getPosition().getY() / Display.getHeight() * 2 + 1 - gui.getSize().getY() / Display.getHeight()
                ), gui.getRotation()
                ,new Vector2f(
                        gui.getSize().getX() / Display.getWidth(),
                        gui.getSize().getY() / Display.getHeight()
                )
        );
        shader.loadTransformation(matrix);

The x size, y size, x position and y position are in pixels, if you put in 1600 pixels for x position and the screen is 1600 pixels wide it will fill the whole screen. The calculation that is done here will translate those sizes to a value that the shaders will display on the screen.

I had tried calculating the sizes in a different way:

gui.getSize().getX() / (float) (Display.getWidth() * Math.abs(Math.cos((Math.toRadians(gui.getRotation())))) + Display.getHeight() * Math.abs(Math.sin((Math.toRadians(gui.getRotation()))))),

but this does not work at all, my thought process was that if it is sideways it should use Display.getHeight to calculate the x size and if it is vertical it should use Display.getWidth. This does work for 0, 90, 180 and 270 degrees rotation, but in between it gets smaller and weird. Of course I've also done this for the x position but then cos and sin switched around.

video of this method

And this is the createTransformationMatrix method:

public static Matrix4f createTransformationMatrix(Vector2f translation, float rotation, Vector2f scale) {
    Matrix4f matrix = new Matrix4f();
    matrix.setIdentity();
    Matrix4f.translate(translation, matrix, matrix);
    Matrix4f.rotate((float) Math.toRadians(rotation), new Vector3f(0, 0, 1), matrix, matrix);
    Matrix4f.scale(new Vector3f(scale.x, scale.y, 1f), matrix, matrix);
    return matrix;
}

Answer 1

Try this instead:

Matrix4f modelMatrix = new Matrix4f();
        modelMatrix.m00 =  2.0f / (float) Display.getWidth();
        modelMatrix.m11 = -2.0f / (float) Display.getHeight();
        modelMatrix.m30 = -1;
        modelMatrix.m31 =  1;

I have no idea what gui.getSize().getX() / Display.getWidth() was suppose to do, so I left it out from above. But if you intend to scale or translate some object after converting it to pixels, I recommend creating a second matrix with those transformation and multiplying them together to create a third matrix rather then modifying this one.

What I created above it an orthographic projection matrix which translates from pixel values e.i. [0,width] to [-1,1] and [0,height] to [1,-1]. This also means the origin is in the top-left corner. So if you now assume pixel coordinates and multiply any transformation with the projection matrix at the end, no distortions should occur.

Remember that the order you multiply matrices matter.

Example:

Matrix4f modelMatrix = new Matrix4f();
        modelMatrix.m00 =  2.0f / (float) Display.getWidth();
        modelMatrix.m11 = -2.0f / (float) Display.getHeight();
        modelMatrix.m30 = -1;
        modelMatrix.m31 =  1;

// No special trickery or dividing by width, just use pixel coordinates.
// The projection matrix takes care of converting the pixel coordinates.
Matrix4f modelMatrix = Maths.createTransformationMatrix(
    new Vector2f(
        gui.getPosition().getX(),
        gui.getPosition().getY())
    , gui.getRotation()
    , new Vector2f(
        1.0f,
        1.0f
    )
);

Matrix4f modelProjMatrix = Matrix4f.mul(projectionMatrix, modelMatrix, null);
shader.loadTransformation(modelProjMatrix);

The projectionMatrix can be shared by all of your geometry. Again I don't really understand what the gui.getSize().getX() is suppose to be doing in your code but I included it.