I was about to post a link to the Asteroid Base blog on the Walker enemy when I noticed karmington already linked to the Gamasutra article in a comment (unfortunately, the images in that version are now broken, but the content is identical to the blog post above which still works as of 2024).
It's probably worth giving this a bit more visibility and explanation as an answer in its own right though.
The trick is raycasts. These let your code scan along a line for a collision.
To get the Walker to align nicely to surfaces, they fire two rays down at the terrain, one from the Walker's left side and one from its right (red and yellow lines above).
From each RaycastHit2D object, you can get a normal vector pointing perpendicular to the surface (magenta and green lines). By averaging the normals from the two hits, you get a suitable orientation for your character that's reasonably reflective of the underlying terrain (cyan line).
You can apply a force down along this orientation line to keep the character stuck to the terrain (though this may cause some sliding if the friction is low), or manually position the character at the average ray intersection point as is done with the Walker.
You'll want to blend the character's orientation gradually to keep your character from vibrating or looking robotic (see how the normal changes sharply when the Walker crosses the corner, but the body moves smoothly).
If you have sharp corners in your terrain, you'll probably want to angle these rays inward slightly - so that if the spider walks past the edge of a 90-degree cliff, the overhanging ray will hit the cliff face.
The Asteroid Base team also shoot a short ray in the direction of the Walker's motion (not shown), which overrides the ray on that side if it hits terrain. That ensures the walker doesn't get stuck when walking into a 90-degree concave corner.
According to the docs, the 2D raycast differs from the 3D one in that it detects colliders within which the ray starts. That sounds fishy to me, but if your testing confirms this you'll want to use a layerMask to limit the rays to only detect climbable terrain colliders.