Stuck on walls or sliding on top dilemma

I'm making a 2.5D platformer game. I earlier had problem with my character getting stuck on walls when holding the forward button. I solved this problem by adding a physics material with zero friction to my colliders on the wall. So now when hitting a wall the character slides down, which is correct.

My walls consits of 1 unit cubes tiled together, and box colliders are generated and merged at runtime.

Now I've created a ladder and by pressing a button the player will rotate 180 degrees, and I also add some force along the y- and z-axis:

var force = -transform.forward + Vector3.up;
force.y *= JumpOffForceY;
force.z *= JumpOffForceZ;