2D Polygon Collision push direction with large polygons

I have a collision detection/resolution implementation that uses the Separating Axis Theorem. Detection and such works perfectly fine, as well as resolution for most objects/shapes in my game.

However, when it comes to sloped surfaces, there is a problem.

As the picture below in this post demonstrates, whenever an AABB polygon (i.e. the player) is tested against a large slope, the direction the player is pushed in will end up being incorrect. I understand why this phenomenon happens; my displacement calculation (which is used to determine if the push vector needs to be flipped) is giving improper results because the player is below the 'center' of the sloped shape, and thus the displacement calculation is reporting that +y (downward) is the direction to push away from the slope's polygon. This causes the player to get pushed into the ground, or through the slope entirely if there is no ground polygon underneath.

My displacement code looks something like this:

        MTV mtv(perpendicular, 0.f);

sf::Vector2f center1 = GetRectCenter(shape_1.getGlobalBounds());
sf::Vector2f center2 = GetRectCenter(shape_2.getGlobalBounds());
sf::Vector2f dir;
dir.x = center2.x - center1.x;
dir.y = center2.y - center1.y;
dir = unitVector(dir);
bool flipme = false;

float disp = dotProduct(mtv.axis, dir);
if(disp < 0)
flipme = true;


So what I am wondering, is there another solution to finding out if a polygon's displacement vector will end up towards the polygon or away from it? This displacement calculation works for rectangular shapes and smaller-sized slopes, but is obviously incorrect when the slope is large enough for its center to be navigated around.

I have also tried averaging the points together to find the centroid of the slope, but that still yields similar results.