UTF-8 Byte Order Mark
Tags: unicode, utf, raku
Published: 2023-09-18 12:04:18

In [previous post of this series](https://dev.to/bbkr/utf-variable-length-to-the-rescue-1ga5) I explained that UTF is a multi byte encoding that also has few variants: `UTF-8`, `UTF-16` and `UTF-32`. To make things more complicated in `UTF-16` and `UTF-32` there are two ways to send bytes of single code point - in big endian or little endian order.

BTW: Endianness term is not related to Indians. It comes form Gulliver's Travels book. There was a law in Lilliputians world that forced citizens to break boiled eggs from little end. Those who rebelled and were breaking eggs from big end were called "big endians".

![Gulliver](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kx1n9stmvspw4ntvu8b7.jpg)

**What is Byte Order Mark?**

To notify which byte order is in processed file or data stream a special sequence of bytes at the beginning was introduced, called Byte Order Mark. Or BOM for short.

For example UTF-16 can start with `0xFE 0xFF` for big endian and `0xFF 0xFE` for little endian order. And UTF-32 can start with `0x00 0x00 0xFE 0xFF` for big endian and `0xFF 0xFE 0x00 0x00` for little one.

**Impact on UTF-8**

Here things gets weird. UTF-8 is [constructed in such a way](https://dev.to/bbkr/utf-8-internal-design-5c8b), that it has only one meaningful byte order, because first byte describes how many bytes will follow to get code point value.

However BOM specification **has** magic sequence for UTF-8, which is `0xEF 0xBB 0xBF`. It only indicates encoding type, therefore has no big endian / little endian variants.

**Implications**

BOM idea may sound weird today, because UTF-8 became prevalent and dominant. But remember that we are talking about year 2000, when [things were not that obvious](https://dev.to/bbkr/madness-before-utf-5f67).

Spec claims that if a protocol always uses UTF-8 or has some other way to indicate what encoding is being used, then it should not use BOM. So for example BOM should not appear in `*.xml` files:

```
<?xml version="1.0" encoding="UTF-8"?>
<tag>...
```

Or in MIME `*.eml` files:

```
--3e6ea2aa592cb31d47cefca38727f872
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"
```

Because those specify encoding internally. Unfortunately this is sometimes ignored, so if something broke your parser and you cannot find obvious error - look if file has UTF-8 BOM:

```
$ raku -e 'say "file.txt".IO.open( :bin ).read( 3 ) ~~ Buf.new(0xEF, 0xBB, 0xBF)'

True
```

**Security issues**

But what if BOM is not aligned with internal/assumed encoding? Let's create following file:

```
$ raku -e '
spurt "file.txt",
    Buf.new( 0xFE, 0xFF, 0x3c, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x3e )
'
```

Now you upload this to some service. This service has validator that <u>respects BOM</u> and should strip all HTML tags. Validator sees nonsense but perfectly legal content that passes validation:

![Trust BOM](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z0uef83phbolyb8el2cu.png)

Later this service opens and displays uploaded file, but it <u>ignores BOM and assumes UTF-8</u>:

![Assume UTF-8](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lmkivelqya13ljir73ao.png)

Oooops! If you trusted validator and displayed this file without proper HTML escaping then you have JavaScript injection. This happened because `㱳捲楰琾` in UTF-16 suggested by BOM has the same byte sequence as `<script>` in assumed UTF-8.

**Conclusions**

You should still be aware of existence of Byte Order Mark, even if it makes zero sense in UTF-8 dominated world today.

**Coming up next:** [UTF-8 in MySQL](https://dev.to/bbkr/utf-8-in-mysql-5e6f).