In the wireframe you can see the tilemap collider for this hill. The red circle falls right through however. The circle has a 2D rigidbody attached. Everything is on the Default sorting layer and on layer order 0.
I forgot to put a circle collider on the red circle. Leaving this up for others who might run into the same problem.