Rotate a plane defined by its normal and its distance

First apologies for the amount of pictures, it's a bit hard trying to explain my problem without pictures. Hope I've provided all the relevant code. If you feel you want to know about how I am doing something else, please tell me and I will include it.

I've been trying to work around rotating a plane in 3D space, but I keep hitting dead ends. The following is the situation:

I have a physics engine where I simulate a moving sphere inside a cube. To make things simpler, I have only drawn the top and bottom plane and moved the sphere vertically. I have defined my two planes as follows:

CollisionPlane* p = new CollisionPlane(glm::vec3(0.0, 1.0, 0.0), -5.0);
CollisionPlane* p2 = new CollisionPlane(glm::vec3(0.0, -1.0, 0.0), -5.0);

Where the vec3 defines the normal of the plane, and the second parameter defines the distance of the plane from the normal. The reason I defined their distance as -5 is because I have scaled the the model that represents my two planes by 10 on all axis, so now the distance from the origin is 5 to top and bottom, if that makes any sense.

To give you some reference, I am creating my two planes as two line loops, and I have a model which models those two line loop, like the following:

top plane:
std::shared_ptr<Mesh> f1 = std::make_shared<Mesh>(GL_LINE_LOOP);
std::vector<Vertex> verts = { Vertex(glm::vec3(0.5, 0.5, 0.5)), Vertex(glm::vec3(0.5, 0.5, -0.5)), Vertex(glm::vec3(-0.5, 0.5, -0.5)), Vertex(glm::vec3(-0.5, 0.5, 0.5)) };
f1->BufferVertices(verts);

bottom plane:
std::shared_ptr<Mesh> f2 = std::make_shared<Mesh>(GL_LINE_LOOP);
std::vector<Vertex> verts2 = { Vertex(glm::vec3(0.5, -0.5, 0.5)), Vertex(glm::vec3(0.5, -0.5, -0.5)), Vertex(glm::vec3(-0.5, -0.5, -0.5)), Vertex(glm::vec3(-0.5, -0.5, 0.5)) };
f2->BufferVertices(verts2);

std::shared_ptr<Model> faceModel = std::make_shared<Model>(std::vector<std::shared_ptr<Mesh>> {f1, f2 });

And like I said I scale the model by 10.

Now I have a sphere that moves up and down, and collides with each face, and the collision response is implemented as well. The problem I am facing is when I try to rotate my planes. It seems to work fine when I rotate around the Z-axis, but when I rotate around the X axis it doesn't seem to work. The following shows the result of rotating around Z: However If I try to rotate around X, the ball penetrates the bottom plane, as if the collisionplane has moved down: The following is the code I've tried to rotate the normals and the planes:

for (int i = 0; i < m_entities.size(); ++i)
{
glm::mat3 normalMatrix = glm::mat3_cast(glm::angleAxis(glm::radians(6.0f), glm::vec3(0.0, 0.0, 1.0)));

CollisionPlane* p = (CollisionPlane*)m_entities[i]->GetCollisionVolume();
glm::vec3 normalDivLength = p->GetNormal() / glm::length(p->GetNormal());
glm::vec3 pointOnPlane = normalDivLength * p->GetDistance();
glm::vec3 newNormal = normalMatrix * normalDivLength;

glm::vec3 newPointOnPlane = newNormal * (normalMatrix * (pointOnPlane - glm::vec3(0.0)) + glm::vec3(0.0));

p->SetNormal(newNormal);
float newDistance = newPointOnPlane.x + newPointOnPlane.y + newPointOnPlane.z;
p->SetDistance(newDistance);
}

I've done the same thing for rotating around X, except changed the glm::vec3(0.0, 0.0, 1.0) to glm::vec3(1.0, 0.0, 0.0)

m_entites are basically my physics entities that hold the different collision shapes (spheres planes etc). I based my code on the answer here Rotating plane with normal and distance

I can't seem to figure at all why it works when I rotate around Z, but not when I rotate around X. Am I missing something crucial?

You have at least one error. This is incorrect:

float newDistance = newPointOnPlane.x + newPointOnPlane.y + newPointOnPlane.z

float newDistance = glm::dot(newPointOnPlane, newNormal);

Because the "distance" of a plane is the distance along the normal to the plane origin, which is given by the dot product between the normal and the plane origin.

EDIT: Also:

glm::vec3 newPointOnPlane = newNormal * (normalMatrix * (pointOnPlane - glm::vec3(0.0)) + glm::vec3(0.0));

Is equivalent to:

glm::vec3 newPointOnPlane = newNormal * (normalMatrix * (pointOnPlane));

And further, it should just be:

glm::vec3 newPointOnPlane = normalMatrix * pointOnPlane;

Which is equivalent to:

glm::vec3 newPointOnPlane = newNormal * p->GetDistance();

Which means:

float newDistance = p->GetDistance();

Therefore, this is the correct code (you only have to rotate the normals!):

glm::vec3 normalDivLength = p->GetNormal() / glm::length(p->GetNormal());
glm::vec3 newNormal = normalMatrix * normalDivLength;

p->SetNormal(newNormal);

EDIT 2: Now, you need to rotate and translate the models so they align with the planes:

rotation = normalMatrix;
translation = newNormal * p->GetDistance();

Your meshes should both have exactly the same data:

std::shared_ptr<Mesh> f1 = std::make_shared<Mesh>(GL_LINE_LOOP);
std::vector<Vertex> verts = { Vertex(glm::vec3(-0.5, -0.5, 0.0)),   Vertex(glm::vec3(-0.5, 0.5, 0.0)), Vertex(glm::vec3(0.5, 0.5, 0.0)), Vertex(glm::vec3(0.5, -0.5, 0.0)) };
f1->BufferVertices(verts);
f2 = f1;
• Hmm, well that seems to make the whole sphere disappear if I rotate the plane on any axis. Feb 24 '15 at 20:39
• You may have other errors as well, but that was certainly one of your problems. Feb 24 '15 at 20:40
• Am I correct in rotating the plane around the origin with the following line : newNormal * (normalMatrix * (pointOnPlane - glm::vec3(0.0)) + glm::vec3(0.0)); ? Feb 24 '15 at 20:44
• No. I provided some more code to help you. You actually made it much more complicated than it has to be. You only need to rotate the normals. The distances are not affected at all. Feb 24 '15 at 20:46
• So you're saying I shouldn't change the distance at all, just rotate the normal? Feb 24 '15 at 20:48