Madness before UTF
Tags: unicode, utf
Published: 2023-07-27 13:36:05

Long time ago computers were using mostly [ASCII](https://en.wikipedia.org/wiki/ASCII) encoding. ASCII stands for **American Standard Code for Information Interchange** and was very early (first revision published in 1963) attempt to unify binary representation of a text.

Originally it used 7 bits, allowing to store `2^7=128` characters. Lowercase and uppercase Latin letters, Arabic numerals and punctuations were included. Because it was used at first for teleprinter and teletype machines it contained also a lot of control characters, which do not have graphical representation but have some effect like moving to new line or confirming transmission. Here is the full list:

![ASCII 7 bit](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dukpiqsiemh8q1pg506i.png)

But what if someone needed to write another character? This is where creativity kicked in. ASCII was using 7 bits, but computers were mostly using 8 bits as base registry size. Meaning that the smallest chunk of data interchanged between CPU, RAM and disk was much bigger (`2^8=256` combinations) than single ASCII character. This excessive space was used to store characters not included in base ASCII.

However that extra space in single byte could store only `128` additional characters. Way too few to write accented versions of Latin like `ę`, Cyrillic like `Д`, Kanji like `鰯`, Greek, Cherokee, math symbols, dead alphabets like Runic, etc. Various encodings were created as ASCII extensions, each held in this extra `128` space only characters needed for specific use case. What seemed like brilliant idea was actually a <u>poison that tormented computer industry for decades</u> with:

**Issue 1 - Operating system incompatibility**

Some encodings were standardized by ISO, but Microsoft and Apple went their own way. So the same character `ñ` had 2 different binary values: `0xF1` in [ISO-8859-1](https://en.wikipedia.org/wiki/ISO/IEC_8859-1) and [Microsoft CP-1252](https://en.wikipedia.org/wiki/Windows-1252) but `0x96` if you were using [Mac OS Roman encoding](https://en.wikipedia.org/wiki/Mac_OS_Roman). If you wanted to write `Ź` it was `0xAC` in [ISO-8859-2](https://en.wikipedia.org/wiki/ISO/IEC_8859-2) but `0x8F` in [Microsoft CP-1250](https://en.wikipedia.org/wiki/Windows-1250) and you were out of luck in [Mac OS Roman encoding](https://en.wikipedia.org/wiki/Mac_OS_Roman) which was not supporting it. It was truly a great time to receive document from a friend using different machine. Here are two encodings which allowed to write Polish characters, compared side by side. Above green line is base ASCII. Below green line is chaos with differences marked in red.

![ISO-8859-2 vs CP-1250](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xygozjrsh3l5eep3jxvg.png)

**Issue 2 - Encoding switching**

What if you wanted to write a Spanish name in your Russian text? If characters you needed were in two different encodings text had to contain some hidden instructions to switch encodings on the fly. Every office suite, every editor, every email client was using their own method back then. That led to common issues with copy-pasting, because those hidden instructions were not understood by another program. Copy-paste. Something we take for granted today was a painful experience in the past.

How bad was it? Well, I took liberty of searching in how many ways you could write Polish alphabet letters. I found whooping **26 encodings**:

![Polish alphabet](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zfbp7s4tj9kc2y78jmp7.png)

**Hint:** If you deal with retro tech `iconv` is your friend, allowing to convert between 140 encodings.
```
iconv -f CP1250 -t ISO-8859-2 windows_file.txt > iso_file.txt
```
**On a white horse**

Unification was an urgent need and when Unicode consortium announced "hey, we propose a common encoding for ALL characters to end this madness" it took computer world by a storm. Here is some interesting graphics from Wiki showing encodings popularity in years 2010-2021 and total UTF-8 domination we know today:

![Encodings popularity](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n3rmb2cz2z2pr0x5apz4.png)

**Coming up next:** [Variable encoding length to the rescue](https://dev.to/bbkr/utf-variable-length-to-the-rescue-1ga5)!