Turns out it was an issue with the way I was setting up the compute buffer. 

Specifically, it was `ComputeBufferMode.Dynamic` that was incorrect.

https://forum.unity.com/threads/clarification-on-computebuffermode-immutable.1150250/

> Immutable buffers should be used. It's a doc and naming bug. Immutable buffer is actually the most mutable of them all. Dynamic buffers are meant only for meshes etc, as they're CPU visible. Users shouldn't need to change the buffer mode at all. The default one works on C# side. We're delaying the doc fix for now.


Switching to `ComputeBufferMode.Immutable` [which isn't actually _immutable_] solved the problem.