As Zibelas suggested in the comments, this is due to the wagon-wheel effect:
The wagon-wheel effect (alternatively called stagecoach-wheel effect or stroboscopic effect) is an optical illusion in which a spoked wheel appears to rotate differently from its true rotation. The wheel can appear to rotate more slowly than the true rotation, it can appear stationary, or it can appear to rotate in the opposite direction from the true rotation.
Here your ball is rotating so fast, the stripe that was facing the camera last frame rotates all the way around to the far side of the ball by the next frame, and what we see facing the camera is the other half of the stripe. But our eye will assume we're still looking at the same part of the stripe, which is not that far from where we last saw it, so we perceive a reduced speed of rotation based on that cue.
As the ball's rotation slows down, the other half of the stripe starts falling further and further behind where its counterpart was on the previous frame, which means we see a larger and larger visible change between frames, and paradoxically it looks like the ball is speeding up. But it's just an artifact of the mismatch between the strobing of the frame rate and the rotation period of the ball.
(One way to confirm this: write a script that logs the angular velocity of the ball, of the magnitude of that vector, on each frame. You'll see that numerically it's only decelerating, even if it looks like it's accelerating on screen)
Reducing your offset from the center can give it a more gentle spin, so you don't hit the extreme rotation rates where this effect appears.
Or, if you need high spin rates for your gameplay, you can use shader tricks to smear the visuals when they're moving too fast for a camera to capture them in a single frame's exposure. This motion blur will hide the cues our eyes are picking up on to perceive the wrong spin rate, so that the effect isn't distracting for your players.