# Convert integer to float while also dividing down the scale, without data loss

I'm working on a game, and my intent is to avoid use of floating point for unit positions. To that end, I'm using 32-bit integers for all positions, with a millimeter scale.

However, for rendering, I know the GPU expects everything to be float, so I need to convert from millimeter-scale integer to meter-scale float. Sounds simple, but I'd like to know the most efficient way to perform these two conversions at the same time without losing something along the way.

I know I can convert my 32-bit integer to a double and know that it will all fit, before dividing by 1000 to get meters and converting the result to a float:

int i = 1234567890;
double d = i / 1000.0;
f = (float)d;


But I wonder if that's taking more steps than is necessary. The only other option I can think of is to do something like:

int i = 1234567890;
float f = (i % 1000) / 1000.0f;
f += i / 1000;


But that requires multiple divides and more steps than using a double.

Is there perhaps another existing method of performing this sort of conversion? Or are these my only real options?

• On cpu's there are sometimes hardware instructions to make this incredibly fast if you're willing to scale by powers of 2 (so maybe 1024ths of a meter instead of millimeters as the fixed point representation). On aarch64 for example VCVT takes the parameter fbits which determines (more or less) how big the divisor is in your conversion. Perhaps some the the GPU whiz's here might know of an equivalent instruction, and how to convince a toolchain to generate it – Steve Cox Jul 29 '20 at 22:29
• Anybody following the link take note that it's from back in 2006. Keep that in mind while reading... – leftaroundabout Jul 30 '20 at 9:22

## 3 Answers

I think you're making this too complicated. The trick to preserving precision in your floating point numbers is just to keep their magnitude small. No chaining through doubles or two-stage fractional/whole conversion required.

If you want millimetre accuracy, single-precision floats will keep that as long as your numbers are less than about 16 km.

(If your play space is even smaller than that, then you actually lose precision with your millimetre integer scheme, so you might as well stick with floats throughout)

"But my world is bigger than 16 km" — that's not a problem, because the immediate neighbourhood of your camera, where millimetre-sized errors might actually be visible, is not.

When you compute the matrices to send down the pipe to your GPU, all you need to do is form the translation component as a relative offset from the camera.

Vector3Float translation = (object.integerPosition - camera.integerPosition) / 1000f;


So as far as the GPU is concerned, your camera is always at (0, 0, 0) this way, giving you maximum precision in any direction you look. Objects a very long way from the camera can still suffer precision loss, but a 1 mm offset over a 16 km distance barely changes the direction of the vector at all — your rounding to the pixel grid of the screen is actually a much larger source of error than anything you'd get from floats in this scenario.

The other members of the object transformation matrices will already tend to be near the -1...1 range where floats have very high precision, so you don't have to do anything fancy with them.

This goes in general for other values too. If it's an absolute measure, like a position in the world or a time stamp, use integers so that you have the same precision no matter where/when you are. If it's a relative measure, like an offset/displacement, duration (like deltaTime), velocity, etc. then use a float. This gets you consistent relative precision, where the errors stay small in proportion to the quantity you're measuring.

• Wonderful answer, but I want to specifically point out your last paragraph. Integers for absolute, floats for relative, sounds like a great general rule to keep in mind. My attempt to use integers for positions in my game is somewhat on principal, after experiencing many games (even modern ones) that use floats/doubles for positions and suffer for it. This general rule should give me the best of both worlds. – Nairou Jul 29 '20 at 12:54
• Unless you're talking about early versions of Minecraft breaking down when you travel vastly far from the origin, the pains you experienced in those games are more likely due to modelling error than a fundamental limitation of the number format itself. – DMGregory Jul 29 '20 at 12:57
• The game that comes to mind is Space Engineers, and like Minecraft was due to being too far from origin. They "solved" the issue by switching from float to double, which didn't really fix it. "Too far from origin" is a special case, but I've already done many games using floats for position, so integers felt like a worthy cause to figure out. – Nairou Jul 29 '20 at 13:00
• Another example of a game that hasn't figured this out is Pokemon go. They keep the camera at your GPS coordinate and of you inspect a gym you left a Pokemon in after traveling a few hundred miles you'll notice that everyone in the gym has turned into a jagged Eldritch horror – Steve Cox Jul 30 '20 at 11:24

In a comment I mentioned in passing that you could switch to representing 1024ths of meters instead of millimeters. That also helps with representability generally. Your code would look like this:

int i = 1234567890;
float f = ((float)i) / 1024.f;


You don't need to worry about representability at all here. The temporary float value will already be the best representation of i possible in a single precision float, and the divisor 1024.f is not only a perfectly representable floating point number (feel free to write it with the wonky hexadecimal floating literal notation if you prefer) but will actually reduce to a trivial subtraction of the exponent of your floating point value when executed.

Of course you (and your compiler) will probably notice that a division isn't necessary at all at this point. The above code could we written with a similar multiplication (or even maybe some fancy hardware instructions):

int i = 1234567890;
float f = ((float)i) * 0x1p-10f;


(0x1p-10f is just a direct representation of the value 1.f / 1024.f)

• Where can I read about this effect? I'm curious how much one can rely on a compiler optimizing away a power-of-2 float division. – Nairou Aug 7 '20 at 1:10
• @Nairou good question. I found a previous stackoverflow question stackoverflow.com/questions/4125033/… . In a comment R.. confirms that compilers do this optimization (he's a trustworthy source) but I'm not really sure where else you could learn more about it outside of the clang/gcc mailing list. The optimization does fall right out of the format definition en.wikipedia.org/wiki/IEEE_754#Formats – Steve Cox Aug 7 '20 at 12:07

Since you mentioned GPU, I will assume you mean a modern one. In this case, a conversion within a shader per vertex is going to have a performance impact of a single drop falling into the ocean. You will be able to benchmark it, but not see it.

Out of curiosity, why do you think you need integers ? I ask, despite having written a fully-integer SW rasterizing pipeline, but that was on sub-100 MHz CPUs. It does make very little sense on current CPU+GPU combos, though.

• This is not a discussion site, so you should edit your answer to just contain the actual answer. If you need OP to clarify the question, you can add a comment to the question to let OP modify the question for you. – pipe Jul 29 '20 at 19:07
• Integer precision remains constant over the entire representable area, while floating-point precision varies. Using 32-bit integers, you can place an object 2100 km from the origin with millimeter precision, while 32-bit floats start losing precision just before the 17-km point. – Mark Jul 30 '20 at 2:22