What's Up With Floating Point?
Tags: programming, tutorial, beginners, python
Published: 2020-09-03 11:23:43

---
title: What's Up With Floating Point?
published: true
tags: programming, tutorial, beginner, python
canonical_url: https://timroderick.com/floating-point-introduction/
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/qn84iawa6shqrnlvryyz.png
---

I presume most people reading this have used floating-point numbers at _some_ point, often not intentionally.

I'm also fairly sure a good number who _have_ encountered them did so after trying to discover why the result of a **simple computation is incorrect**. E.G.

```
0.1 + 0.2
// result: 0.30000000000000004
// Which I'm fairly sure
// should be 0.3 ...
```

The problem is that without understanding _what_ a floating-point number is, you will often find yourself frustrated when the result you expected is **<sup>ever-so-slightly</sup>** different.

My goal here is to clarify **what** a floating-point number is, **why** we use them, and **how** they work.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#why-do-we-even-need-floating-point-)Why do we even need floating point 🤔?

It's not a _bold_ statement to say that computers need to store numbers. We store these numbers in binary on our phones, laptops, fridges etc.

I hope that most people reading this are familiar with binary numbers in some form; if not, consider reading this blog post by [Linda Vivah](https://medium.com/@LindaVivah/learn-how-to-read-binary-in-5-minutes-dac1feb991e).

But what about decimal? Fractions, π, **real numbers**?

For _any useful_ computation, we need computers to be able to represent the following:

- Very<sup>very<sup>very<sup>very</sup></sup></sup> **SMALL** numbers,
- Very<sub>very<sub>very<sub>very</sub></sub></sub> **BIG** numbers,
- Everything in-between!

Starting with the very<sup>very<sup>very<sup>very</sup></sup></sup> _small_ numbers to take us in the right direction; how do we store them?

Well, thats simple. We store those using the equivalent to decimal representation in binary...

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#binary-fractions)Binary fractions

An example should help here. Let's choose a random binary fraction: `101011.101`

![Binary Fraction Example](https://dev-to-uploads.s3.amazonaws.com/i/e9h4qvw1y1npemr75th3.jpg)

This is very similar to how decimal numbers work. The only difference is that we have base `2` instead of base `10`. If you chuck the above in a binary converter of your choice, you'll find that it is **correct!**

**So how might we store these binary fractions?**

Let's say we allocate one byte (8 bits) to store our binary fraction: `00000000`. We then must choose a place to put our _binary separator_ so that we can have the fractional part of our binary number.

Let's try smack-bang in the middle!

```
0000.0000
```

What's the biggest number we can represent with this?

```
1111.1111 = 15.9375
```

That's... not very impressive. Neither is the smallest number we can represent, `0.00625`.

There is **a lot** of wasted storage space here alongside a poor range of possible numbers. The issue is that choosing any place to put our _point_ would leave us with _either_ **fractional precision** _or_ a **larger integer** range — not both.

If we could just _move_ the fractional _point_ around as we needed, we could get so much more out of our limited storage. If only the _point_ could _float_ around as we needed it, a _floating point_ if you will...

(I'm sorry, I had to 😅.)

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#so-what-is-floating-point)So what is floating point?

Floating point is exactly that, a _floating ( **fractional** ) point_ number that gives us the ability to change the relative size of our number.

So how do we mathematically represent a number in such a way that we,

1. store the significant digits of the number we want to represent **(E.G. the 12 in 0.00000012)**;
2. know where to put the fractional point in relation to the significant digits **(E.G. all the 0's in 0.00000012)**?

To do this, let's time travel (for some) back to secondary school...

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#standard-form-scientific-notation)Standard Form (Scientific Notation)

Anyone remember [mathsisfun](https://www.mathsisfun.com/numbers/scientific-notation.html)? I somehow feel old right now but either way, this is from their website:

![scientific notation explanation](https://dev-to-uploads.s3.amazonaws.com/i/m2cb64rioh9rv3c6q7r2.png)

You can do the _exact_ same thing with binary! Instead of {% katex inline %}7*10^2 = 700{% endkatex %}, we can write

{% katex %}
1010111100 * 2^0 = 700
\\\\
= 10101111 * 2^2 = 700
{% endkatex %}

Which is equivalent to {% katex inline %}175 * 4 = 700{% endkatex %}. This is a great way to represent a number based on the _significant digits_ and the _placement of the fractional point_ relative to said digits.

That's it! **Floating point is a binary standard-form representation of numbers.**

If we want to formalise the representation a little, we need to account for positive _and_ negative numbers. To do this, we also add a sign to our number by multiplying by ±1:

{% katex %}
(\text{sign}) * (\text{significant digits}) * (\text{base})^{(\text{some power})}
{% endkatex %}

And back to the example given by mathsisfun...

{% katex %}
700 = (1) * (7) * (10)^{2}
\\\\
= (1) * (10101111) * (2)^{2}
{% endkatex %}

If you are reading other literature, you'll find the representation will look something like...

{% katex %}
(-1)^s * c * b^{e}
{% endkatex %}

Where {% katex inline %}s{% endkatex %} is the sign bit, {% katex inline %}c{% endkatex %} is the significand/mantissa, {% katex inline %}b{% endkatex %} is the base, and {% katex inline %}e{% endkatex %} is the exponent.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#so-why-am-i-getting-weird-errors-)So why am I getting weird errors 😡?

So, we know what floating point is now. But why can't I do something as simple as add two numbers together without the risk of error?

Well the problem lies partially with the computer, but mostly with _mathematics itself_.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#recurring-fractions)Recurring Fractions

Recurring fractions are an interesting problem in number representation systems.

Let's choose any fraction {% katex inline %}\frac{x}{y}{% endkatex %}. If {% katex inline %}y{% endkatex %} has a _prime factor_ that isn't also a _factor of the base_, it will be a recurring fraction.

This is why numbers like {% katex inline %}1/21{% endkatex %} can't be represented in a finite number of digits; {% katex inline %}21{% endkatex %} has {% katex inline %}7{% endkatex %} and {% katex inline %}3{% endkatex %} as prime factors, neither of which are a factor of {% katex inline %}10{% endkatex %}.

Let's work through an example in decimal.

#### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#decimal)Decimal

Say you want to add the numbers {% katex inline %}1/3{% endkatex %} and {% katex inline %}2/3{% endkatex %}. We all know the answer is ~~42~~ {% katex inline %}1{% endkatex %}, but if we are working in decimal, this isn't as obvious.

This is because {% katex inline %}1/3 = 0.333333333333...{% endkatex %}

It isn't a number that can be represented as a _finite_ number of digits in decimal. As we can't store infinite digits, we store an _approximation accurate to 10 places_.

The calculation becomes...
{% katex %}
0.3333333333+0.6666666666=0.999999999
{% endkatex %}
Which is definitely not {% katex inline %}1{% endkatex %}. It's _real_ close, but it's not {% katex inline %}1{% endkatex %}.

The finite nature in which we can store numbers doesn't mesh well with the inevitable fact that we _can't easily represent all numbers in a finite number of digits_.

#### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#binary)Binary

This exact same problem occurs in binary, **except it's even worse!** The reason for this is that {% katex inline %}2{% endkatex %} has one less prime factor than {% katex inline %}10{% endkatex %}, namely {% katex inline %}5{% endkatex %}.

Because of this, recurring fractions happen more commonly in base {% katex inline %}2{% endkatex %}.

An example of this is {% katex inline %}0.1{% endkatex %}:

In decimal, that's easy. In binary however... `0.00011001100110011...`, it's another recurring fraction!

So trying to perform `0.1 + 0.2` becomes

```
  0.0001100110011
+ 0.0011001100110
= 0.0100110011001
= 0.299926758
```

Now before we got something similar to `0.30000000004`, this is because of things like _rounding modes_ which I won't go into here (but will do so in a future post). The same principle is causing this issue.

This number of errors introduced by this fact are lessened by introducing _rounding_.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#precision)Precision

The other main issue comes in the form of _precision_.

We only have a _certain number of bits_ dedicated to the significant digits of our floating point number.

#### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#decimal-1)Decimal

As an example, consider we have three _decimal_ values to store our significant digits.

If we compare {% katex inline %}333{% endkatex %} and {% katex inline %}1/3 * 10^3{% endkatex %}, we would find that in our system they are the _exact same_.

This is because we only have three values of precision to store the significant digits of our number, and that involves truncating the end off of our recurring fraction.

In an extreme example, adding {% katex inline %}1{% endkatex %} to {% katex inline %}1*10^3{% endkatex %} will result in {% katex inline %}1*10^3{% endkatex %}, the number _hasn't changed_. This is because you need _four_ significant digits to represent {% katex inline %}1001{% endkatex %}.

#### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#binary-1)Binary

This exact same issue occurs in binary with very<sup>very<sup>very<sup>very</sup></sup></sup> _small_ and very<sub>very<sub>very<sub>very</sub></sub></sub> _big_ numbers. In a future post I will be talking more about the limits of floating point.

For completeness, consider the previous example in binary, where we now have 3 _bits_ to represent our significant digits.

By using base 2 instead, `1 * 2^3`, adding `1` to our number will result in no change. This is because to represent `1001` (now equivalent to 9 in decimal) requires 4 bits, the _less significant_ binary digits are _lost in translation_.

There is no solution here, the limits of precision are defined by the amount we can store. To get around this, use a larger floating point data type.

- **E.G.** move from a _single_-precision floating-point number to a _double_-precision floating-point number.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#summary)Summary

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#tldr)TLDR

To bring it all together, floating-point numbers are a representation of binary values akin to [_standard-form_ or _scientific notation_](https://www.mathsisfun.com/numbers/scientific-notation.html).

It allows us to store **BIG** and **<sub>small</sub>** numbers with precision.

To do this we move the _fractional point_ relative to the significant digits of the number to make that number bigger or smaller at a rate proportional to the base being used.

Most of the **errors** associated with floating point come in the form of representing _recurring fractions_ in a _finite_ number of bits. Rounding modes help to reduce these errors.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#thanks-%F0%9F%A5%B0)Thanks 🥰!

**Thank you so much for reading!** I hope this was helpful to those that needed a little refresher on floating point and also to those that are new to this concept.

I will be making one or two updates to this post explaining the in-depths of the [IEEE 754-2008 Floating Point Standard](https://en.wikipedia.org/wiki/IEEE_754), so if you have questions like:

- "What are the biggest and smallest numbers I can use in floating point?"
- "What do financial institutions do about these errors?
- "Can we use other bases?"
- "How do we _actually_ perform floating-point arithmetic?"

Then feel free to follow to see an update! You can also follow me on twitter [@tim\_cb\_roderick](https://twitter.com/tim_cb_roderick) for updates.If you have any questions please feel free to leave a comment below.