I would suggest you look into Brackey's Rewind Tutorial and improve upon that. One thing you could do is you can store all the positions in a JSON file or your own format when you/player hits stop button and read from that format when replaying, also he is rewinding and not replaying so you need to store the position in reverse order than him, or you can read in different order. He also addresses some limitations with this method like it'd be silly to store game play of more than a few seconds as it takes up a lot of memory to track every object, another thing he addresses is that not every device runs on same frame rate so replays will be different on different devices.
Also here's a wild thought, if your game is not fast-paced than you could record your object's position every x frames and maybe Lerp between those values when replaying? It won't be as accurate but it'll be performant than storing each positional change. By doing this you can probably get around the "different frame rate" limitation cause it'll be same on every device as it's "recreating frames", and with this you can also record more game play as it won't need as much memory.