Let's add some colour coding to see if we can reveal some structure in the problem:
Here I've colour-coded the spiral into concentric rings of alternating colours. You'll notice the tile just before the bottom-left corner is always a perfect square. I'll dub this number the "Leader" of a given ring.
If we number our rings from 1 out, then the sequence of leader numbers is:
$$\begin{align} L(1) &= 1 &= 1^2\\ L(2) &= 9 &= 3^2\\ L(3) &= 25 &= 5^2\\ L(4) &= 49 &= 7^2\\ .&..\end{align}\\ L(ring) = (2\cdot ring - 1)^2$$
This will make it easy to number all the cells in a given ring, with reference to the corner. To reach...
- the row \$y = 0\$ in the left side of the ring, we subtract \$(ring - 1)\$
- the column \$x = 0\$ in the bottom of the ring, we add \$(ring)\$
- the bottom-right corner, we add \$(2 \cdot ring)\$
- the row \$y = 0\$ in the right side of the ring, we add \$(3 \cdot ring)\$
- the top-right corner, we add \$(4 \cdot ring)\$
- back to the column \$x = 0\$ along the top row, we add \$(5 \cdot ring)\$
So here's how we can translate that into a conversion method:
function spiralCellIndex(position) {
// First, we round our position to the nearest grid cell.
// Skip this if your input is already an integer pair grid coordinate.
var x = Math.round(position.x/gridSpacing);
var y = Math.round(position.y/gridSpacing);
// Compute a modified x value symmetric across the line x = 0.5
// (In the diagram I let this go negative,
// but doing it this way saves an unnecessary abs() )
var offCenterX = x > 0 ? x : -x + 1;
var ring = Math.max(offCenterX, Math.abs(y));
var leader = (2 * ring - 1);
leader *= leader;
// Top row of ring
if(y === -ring)
return leader + 5 * ring - x;
// Right side of ring
if(x === ring)
return leader + 3 * ring - y;
if(y === ring)
return leader + ring + x;
return leader - ring + 1 + y;
}