Understanding floating point

You may have noticed this interesting behavior of floating point arithmetic:¹

It’s also true that very large integers can’t be represented, that small values can underflow to positive and negative zero, that a + (b + c) may not equal (a + b) + c,² that typeof NaN === "number" in JavaScript, and more!

While it may seem like floating point is just “broken”, these behaviors are unsurprising once you understand the IEEE-754 floating point format itself, and may start to feel like sensible tradeoffs. My goal for you in this tutorial is to develop that depth of understanding, one small motivating step at a time. By the end, you should be able to reason concretely about floating point’s strengths and limitations, enough to explain the 0.1 + 0.2 problem and a handful of others.

I do mean “explain”. This is an experiment in LLM integrated tutorials. To proceed, you will answer inline questions, seeking help too as needed. Each checkpoint has its own carefully designed prompt and grading rubric, and I’ve done my best to make them as sensible and educational as possible, so I sincerely hope your patience is rewarded!

Let’s say I have enough room on a card to write down two decimal digits. How many different numbers could I express?

With the simplest scheme possible, we can encode 0, 1, 2 all the way up to 99. Implicitly, we’re choosing to precisely represent every “natural number” (the ones we use to count) in the range 0 to 99, for 100 in total.

But, what if we want a wider range and are willing to compromise precision?

Let’s try a different convention, where the first digit is our “significant digit”³ and the second tells us how many zeros to add, so 32 would actually indicate 300, 91 would be 90, and 20 would be 2. Another way of thinking about this is that we write down two digits sig and exp, and read the pair as sig \times 10^{exp}. In the same amount of space, we have now achieved much more range. But how much range exactly, and at what cost?

With this scheme, we can represent 0-10 exactly, but there’s no way to represent 11. We can represent 20, 30, 40 and so on, but there’s no way to represent 110. The table below shows one row per exponent, with the significand n running from 0 to 9:

At each order of magnitude, we can represent nine numbers exactly.⁴ One interesting way to think about this, is that the number closest to 9 billion that we can represent in our scheme is 8 billion.

The big idea that we’re starting to play with here is that it’s a feature, not a bug, for the “gaps” between representable numbers to increase. If we want floating point to be as useful for microbiologists as it is for astrophysicists, we should aim for the same amount of precision at each order of magnitude.

But we have a few more steps to get there! Next, let’s map our thinking over to binary. Given 8 bits, what is the range of natural numbers we could represent exactly?

Using the standard encoding, we have \underbrace{2 \times 2 \times \cdots \times 2}_{8 \text{ twos}} = 256 possible values, from 0 to 255.

Just as we did above with our two decimal digits, we can trade some precision for much greater range, this time let’s designate the first 5 bits as “significand” and the other 3 as “exponent”. Our scheme can then be value = sig \times 10^{exp}.⁵

What is our range with this scheme, and what do you notice about precision?

With this scheme, we have 2 \times 2 \times 2 = 8 possible exponents (0 to 7) and 2^5 = 32 possible significands. As before, we have 256 possible bit patterns, but we’re spreading them over a range of up to 31 \times 10^7 = 310,000,000, with 32 expressible values for each exponent, with some redundancy as we’ll explore in a moment. Every number in the range 0 to 31 can be represented exactly, but 32 can’t, and again we get very large gaps at high orders of magnitude: the second largest number is 10 million smaller than the largest.

One unfortunate issue with our scheme is that two different bit patterns can produce the same value. Zero is the worst offender, since a significand of zero paired with any exponent is zero. But, other values also have redundant representations. For instance, can you find two different representations of 20? What about other numbers?

We can write 20 as 20 \times 10^0 or as 2 \times 10^1. Likewise 10 \times 10^1 and 1 \times 10^2 both give us 100. 1, 2 or 3, for all exponents n > 0, have duplicates of 10, 20 and 30 respectively, by using the exponent n - 1. And again we have an annoying representation of zero at every exponent!

To solve this problem, we need a way to avoid overlapping ranges, by having a canonical representation for each number.

If we think about how we do this with scientific notation, we start to approach an answer. Our convention there is to express say Avogadro’s number as 6.022 \times 10^{23}, not 602.2 \times 10^{21}: we keep just one nonzero digit in front of the point.

We could impose the same rule here, and normalize our numbers to have just one leading digit. 2.0 \times 10^1 would be our valid representation of 20, whereas 20.0 \times 10^0 would be illegal.

It’s one thing to deem certain bit patterns as “canonical”, and another to avoid non-canonical encodings in the first place. This is where it really pays off to switch to base 2 for our exponent, in other words to interpret our value as sig \times 2^{exp} rather than sig \times 10^{exp}. If we use the same rule of keeping exactly one nonzero digit in front of the point, that digit will always be 1 and we can normalize our significand as 1.fff \times 2^{exp}. If we then encode this normalized number, we can store just the fff bits and take the 1. as implicit. There is then no way to accidentally represent the same value in two ways.

Let’s play with this encoding now, and see where we’ve landed. Have we solved the duplication problem now? What are the smallest and largest possible numbers?

By switching bases, we’ve lost some range and gained some precision; we can rebalance this later by adjusting our allocation of bits. More importantly, we now have a coherent, canonical form of each value enforced by the encoding scheme itself. We have also lost our ability to encode zero, which we’ll again fix later.

As a next step, let’s support negative values using the simple approach of using one bit for the sign. What does this do to our range and precision?

By using one of our fraction bits for the sign, we have chosen to sacrifice some precision. We could have used one of our exponent bits instead, and it’s worth stopping to articulate the tradeoff here as it gets to one of the fundamental limitations of floating point.

The big idea with floating point is that with a finite number of bits, we spend some on the exponent (for range) and some on the significand (for precision). Integer and fixed point encodings commit entirely to the significand, whereas floating point strikes a balance!

A major missing piece of our toy float is that while we can represent negative values, we can’t yet represent negative exponents. For instance, there is no way for us to yet represent 0.5, or 0.125.

While we used a sign bit for the value itself, and it could have been reasonable to use something like ones’ complement or two’s complement, IEEE-754 instead uses a “bias”.⁶. We read the exponent field as an ordinary integer but then subtract a fixed constant, 3 in our example, to basically just “remap” each bit pattern to represent -3 to 4 instead of 0 to 7.

Using this scheme, we now have a way to encode 0.5. See if you can find it!

As a challenge question, see if you can figure out why IEEE-754 chooses to use a bias rather than one of the more familiar signed integer schemes. Two hints: I’ve moved the exponent field in front of the significand now to match the protocol, and given you two floats in the widget below, with a size comparison.

Using a bias, and moving the exponent field in front of the significand, the bigger bit pattern (interpreted as a plain integer) is always the bigger float, at least when comparing two positive floats.⁷ This allows the same integer comparison hardware to be used for comparing floats.

One major shortcoming still remains: we can’t encode zero, since every significand reads 1.ffff with an implicit leading 1.

The solution in IEEE-754 is to reserve the very bottom of the exponent range. When the exponent field is all zeros, the implicit leading 1 switches off. This gives us a representation of zero, but doesn’t quite stop there. Have a play and see if you notice any of its other effects.

With this scheme we now have both zero and negative zero, distinguished by the sign bit. These are distinct, but defined by the standard to be equal. While this may seem like a bug, it does have some value: a number which becomes too small (in magnitude) to represent in floating point will underflow to zero, but do so without losing directionality. As a mini exercise, you may want to repeatedly divide -1.0 by 2 in your language of choice, and observe underflow to -0.

Reserving one of our exponent values entirely for \pm 0 would waste many bit patterns, since any fraction bits would effectively be ignored. Instead, by switching off the leading 1, we end up with another range of valid representations below the normalized numbers, appropriately called the “subnormals”. I’ll leave the details of the subnormals for another time, but you may want to convince yourself that there’s no overlap between the normalized numbers and subnormals. It’s also worth noting that due to dramatically poorer performance (and associated timing attacks) the subnormals are often flushed to zero in practice.

There’s one last feature we should support, which is overflow to \pm \infty. In scientific applications, just as we’d like to underflow to \pm 0, we’d like excessively large numbers to make that clear, rather than wrapping around as happens with integers. The widget below implements the same scheme for infinities as IEEE-754. See if you can find them! While you’re there, you may also discover another class of supported values.

Just as the bottom of the exponent range is reserved for zero (and subnormals), the top is reserved for \pm \infty (and the NaNs).

Having a NaN value allows us to encode the result of an operation like 42/0 or \infty - \infty. Having two NaN types, differentiated by the high order fraction bit, allows us⁸ to optionally raise an exception instead of propagating NaNs. You may have also noticed that NaN is a whole range of bit patterns, intended to be used for payloads regarding the cause of the NaN, or for fun tricks like NaN boxing and being sneaky.

At this point, we have re-invented every structural feature of IEEE-754, in miniature form:

• Sign bit
• Normalized significand with implicit leading 1
• Biased exponent
• Reserved exponent value at the low end for zeros and subnormals
• Reserved exponent value at the high end for infinities and NaNs

All we need to do to get to a real float is to scale this up! More exponent bits for greater range, and more significand bits for greater precision.

Let’s start with the smallest format in common use: the 16-bit “half precision” also known as fp16, commonly used in graphics and machine learning inference.⁹ It uses 1 sign bit, 5 exponent bits with a bias of 15, and 10 fraction bits. See if you can figure out its range, and say a little about its precision.

Half precision is very useful where space is at a premium, but it certainly comes at a cost! Any number from 65,520 onwards rounds to +\infty, and hopefully you don’t need to represent the number 2,049.

Since that’s a little tight for general purpose use, let’s also briefly look at the 32-bit “single” and 64-bit “double”. Again, there is nothing structurally different about these, we just spend more exponent bits to buy range, and more significand bits to buy precision.

For the 32-bit float below, what is the range? What can you say about its precision?

With an allocation of 1 sign / 8 exponent / 23 fraction bits, a single precision float can reach as far as around 3.4 \times 10^{38}, allowing a broader range of uses. While it may feel alarming that the gap to the second largest fp32 is 2^{104} \approx 2.028 \times 10^{31} smaller, this is reasonable with such a large exponent. A more reassuring way to think about precision is that with 23 fraction bits we have 2^{23} = 8,388,608 representable values at each power of 2.

Once again, in choosing a float format, we’re paying with space¹⁰ and buying some amount of precision and range. While we’re focused here on IEEE-754 standard floats, there’s no reason we couldn’t push the tradeoff in other directions too. Let’s say for instance that you wanted the range of 32 bit floats but jammed into 16 bits for performance, and you’re willing to sacrifice a lot of precision. This would suggest 1 sign bit, 8 exponent bits and only 7 fraction bits, effectively just dropping the last 16 bits of a 32 bit float! While this may sound like an extreme tradeoff, it’s the chosen structure for bfloat16 and suits machine learning training.

If you’d like more of both precision and range, and are willing to use twice the space, you would opt for the 64-bit “double”, with 11 exponent bits and 52 fraction bits. This is the default float type in many languages, such as float in Python and Number in JavaScript¹¹ and provides tremendous precision all the way up to its maximum value of around 1.8 \times 10^{308}.

We’ll skip playing with 64-bit floats directly as they are a bit unwieldy, but they are central to our first question: why is 0.1 + 0.2 always 0.30000000000000004?

We now have everything we need to tackle this! Hopefully you have a sense that the problem must lie in the fraction bits: the exponent just scales the issue up to other orders of magnitude.

So, let’s focus on the range 0 to 1 and see what our fractional binary digits can build for us, starting with just 4:

With 4 fraction bits, we have sixteen bit patterns representing the multiples of \tfrac{1}{16}, from 0 up to \tfrac{15}{16}.¹²

What’s a number that you could represent exactly if you just had one more fraction bit? Think through it first, then check and explain your answer. What does each new bit give us?

Now let’s tackle the interesting case. Let’s say you had unlimited bits. Can you represent 0.1 exactly? Feel free to keep adding bits!

It may be easiest to think about this in reverse: given a finite string of bits, this must encode some integer divided by a power of 2. For instance 0.1011 is \frac{1}{2} + \frac{1}{8} + \frac{1}{16} = \frac{8 + 2 + 1}{16} = \frac{11}{16}

In the case of 0.1 = \tfrac{1}{10} there is no way to do this, since 10 = 2 \times 5 and no power of 2 has a factor of 5. However many bits we add, we just get a repeating bit pattern of 0.000110011001100.... A similar fate awaits 0.2 and 0.3, in fact any decimal fraction whose denominator is not a power of two (in lowest terms) can not be represented exactly, no matter how many bits.

Confronted with storing 0.1, the best a computer can do is to store the nearest value. This is what IEEE-754 mandates, so even with 52 fraction bits, the stored value for 0.1 is exactly 0.1000000000000000055511151231257827021181583404541015625 on every machine. 0.2 has the same significand with one higher exponent, so it simply scales the rounding error by 2!

If both 0.1 and 0.2 can’t be represented exactly, it may feel obvious that the sum is also rounded. However, this isn’t always the case! Consider an exactly representable target number such as 0.5. Can two imprecise inputs add to one precise output?

When reasoning through this, consider that rounding will occur not only on each stored input, but also in storing the result of an arithmetic operation. Try it below in half precision, and see if you can describe some of the rounding effects.

So in half precision, 0.2 + 0.3 gives us exactly 0.5 because the two errors exactly cancel. 0.1 + 0.4 is different: both inputs are stored low, and the exact sum of the stored values falls precisely halfway between two representable values. For ties like this, IEEE-754 specifies “round-to-even”: pick the neighbour whose last significand bit is 0.¹³ In this case, that neighbour is 0.5 itself.

Unfortunately, it’s also possible for two exact inputs to add to something unrepresentable above. Consider 2048 + 1 above,¹⁴ or this in JavaScript:

The most common case, though, is that both our inputs and our outputs will round, with behavior that is deterministic and comprehensible, if only you look closely enough!

So let’s do that now with 0.1 + 0.2. I’ll explain it in full detail for 64-bit doubles, and your final challenge will be to demonstrate your understanding by extrapolating it to half floats.

We’ve already seen that 0.1 is stored slightly high, in 64 bits: 0.1000000000000000055511151231257827021181583404541015625 and that the stored 0.2 is the same significand one exponent up, stored high by exactly twice as much.

The exact sum of those two stored values is 0.3000000000000000166533453693773481063544750213623046875 Which is not representable: doubles of this magnitude have gaps of 2^{-54}, and this sum lands exactly between two of them. Round-to-even then picks the neighbour whose last significand bit is 0, and this time we round up to 0.30000000000000004440.... Meanwhile a literal 0.3 rounds down, to 0.29999999999999998889... meaning that the comparison is consistently and predictably false:

Why then does the result print as 0.30000000000000004? Modern languages show the shortest decimal string that uniquely identifies the stored double.¹⁵ But now you know it’s not exactly that, either!

Since the effect is a function of rounding direction, different format float sizes behave differently. With 32-bit floats, the rounded sum coincides with a stored literal 0.3, so the same comparison comes out true:

This is not so with our old friend, the 16 bit half precision float. If you can extend our reasoning there and explain the behavior, this will serve as proof that you’ve really come to understand floating point at tremendous depth. Good luck!

So that’s it. In half precision, 0.1 and 0.2 are both stored slightly low, their sum lands exactly halfway across the gap at that magnitude, and round-to-even sends it down. Meanwhile a literal 0.3 rounds up, so in this case 0.1 + 0.2 < 0.3.

Of course, no single “fact” like this is individually meaningful. What every software engineer should know about floating point is that a finite number of bits must be shared between exponent and significand, buying us range and precision respectively.¹⁶ This works well for many applications, but not for exact representations of many numbers. The rounding behavior is deterministic and with care even understandable, and rarely affects typical use.

Add in some extra detail about reserved ranges for the zeroes, subnormals, infinities and NaNs, and you should know everything you need to explain the unique behavior of floating point, and perhaps you will agree with me that the juice is worth the squeeze.

Congratulations!