Turns out it was an issue with the way I was setting up the compute buffer.
Specifically, it was ComputeBufferMode.Dynamic that was incorrect.
https://forum.unity.com/threads/clarification-on-computebuffermode-immutable.1150250/
Immutable buffers should be used. It's a doc and naming bug. Immutable buffer is actually the most mutable of them all. Dynamic buffers are meant only for meshes etc, as they're CPU visible. Users shouldn't need to change the buffer mode at all. The default one works on C# side. We're delaying the doc fix for now.
Switching to ComputeBufferMode.Immutable [which isn't actually immutable] solved the problem.