Player clips into the ground when I press the restart button

Question

The title says it all. I have a subway surfers like game prototype, where I just added a restart button that can reload the scene. However whenever I press restart the charaters gets into a slightly lower position. After a few deaths the player is clipped into the ground and cannot jump at all. I looked up so many similar questions, all about checking colliders and rigidbodies but I personally couldn't find anything. It also has to do something with my gravity and stuff. Because I have a gravity modifier that if I set to 1, this issue never happens. The problem is, that then the player jumps around like he is on the moon. I also tried an experiment where I started my player from the air, not from the ground. And with every single restart the player fell faster and faster eventually falling trough the ground once again. So something can be with gravity. This is what I have in the character controller. I wasn't satisfied with my physics and I wanted it to make a bit more interesting.

I tried readding colliders, putting the same scripts on brand new objects, checking on and off rigidbody options.

void Start()
    {
        playerRb = GetComponent<Rigidbody>();
        Physics.gravity *= gravityModifier;
    }

    // Update is called once per frame
    void Update()
    {
        Vector3 xPos = new Vector3(xRange, 0, 0);
        if (Input.GetKeyDown(KeyCode.Space) && isOnGround)
        {
            playerRb.AddForce(Vector3.up * jumpForce, ForceMode.Impulse);
            isOnGround = false;
        }

Answer 1

The error happens because you do not only change the Physics once but each time you die. It is independent of your scene. Now you do not only change it each time you die, you use the old value and multiply it again. The fast fix would be to switch to the constant that your first change gave you.

Physics.gravity = newGravityValue;

What you wrote is

Physics.gravity = Physics.gravity * gravityModifier;

which is the same as

Physics.gravity *= gravityModifier 

The correct way would be to change the settings under the Project Settings