You would only restrict movement in that direction. In other words, let the player move left only he has not collided with a wall to his left. Let him move right only if there is no wall to his right. This way, if there is a wall on one side, the player can still back away.