Collision normal for sphere near edge of triangle face

Question

When my sphere (or bottom part of a capsule) moves with its bottom near the edge of a mostly horizontal triangle (a face of a heightmap), I am sometimes getting very bad (mostly horizontal) normals. How should a proper collision normal be calculated in this situation?

Here's my attempt to explain the current situation leading to these bad normals. Running the usual separating axis tests, it is determined that the best axis is mostly vertical. Based on this axis, the colliding features of the shapes are determined. The feature of the sphere is always a point on the bottom, and the feature of the triangle can be the whole face, an edge or a point. The bad normals here are arising from point-edge collisions. Namely, the best axis was determined to be mostly vertical, but it can happen that the point is extremely close to the edge, and vertically closer than horizontally (and so apparently a separating axis in the plane of the triangle was not considered or not the best), so that the contact pair consisting of the point of the sphere and the closest point on the edge has a normal which is mostly horizontal.

How should this problem be solved?

For reference: this is Godot issue #69683 which I would like to fix.

Answer 1

If possible, compute the collision with the triangle mesh, not a single triangle, as follows:

Any time the sphere-triangle collision test decides the sphere is colliding with the edge of the triangle, that means you should instead be colliding the sphere with the adjacent triangle across that edge.

If both triangles think the collision is on edge, then use a normal determined by the contact point on the sphere, or perhaps (this might be better for height fields) the average of the two triangles' normals.