UTF-8 code point properties
Tags: unicode, utf, raku
Published: 2023-08-05 11:05:23

Having single encoding to express every intent in every language is awesome. But that comes at a cost of dealing with huge amount of characters you probably never seen before. Luckily every code point has a set of [properties](https://util.unicode.org/UnicodeJsps/character.jsp) that may help you with text processing. There are over 100 properties total, but the most useful are:

- Major/minor category.

```
$ raku -e '"a".uniprop.say'
Ll

$ raku -e '"3".uniprop.say'
Nd
```

When `uniprop` is called without params in [Raku](https://raku.org/) it returns acronym for major/minor category. `Ll` means `letter`/`lowercase`, `Nd` means `number`/`decimal digit`. Full list is available [here](https://en.wikipedia.org/wiki/Unicode_character_property#General_Category). Those can also be tested independently as shown below.

- Letter, Number, Punctuation, Separator.

Daily bread of text processing.

```
raku -e '
    my $text = "Is this 1970 Dodge?";
    say $text;
    for "Letter", "Number", "Separator", "Punctuation" {
        $text.uniprops( $_ ).join.say;
    }
'

Is this 1970 Dodge?
1101111000000111110 # Letters
0000000011110000000 # Numbers
0010000100001000000 # Separators
0000000000000000001 # Punctuation
```

Method `uniprops` is the same as `uniprop`, but returns property values for all characters in string. Both `uniprop` and `uniprops` methods can be given specific property to test against.

- Script

```
$ raku -e '"aГΦح日".uniprops( "Script" ).say'

(Latin Cyrillic Greek Arabic Han)
```

Script is a writing system. It should not be confused with alphabet - for example `a` and `ą` are both Latin script but the `ą` only belongs to Polish alphabet. And it should not be confused with language, despite the fact that it sometimes alignes with it - for example Greek.

How many scripts are there?

```
$ raku -e '.say for ( 1 .. 1_112_064 ).map( *.uniprop( "Script" ) ).unique'

Common
Latin
Bopomofo
Inherited
Greek
Unknown
Coptic
Cyrillic
Armenian
Hebrew
Arabic
Syriac
Thaana
Nko
...
```

The answer is 158.

- Casing

_Did you know_ that "lowercase" and "uppercase" terms come from printing press? Page was composed from metal stamps. Stamps with letters `a`, `b`, `c` were used more often than stamps with letters `A`, `B`, `C`. So `a`, `b`, `c` were stored in lower case on the desk, easier to reach. While `A`, `B`, `C` were stored in upper case above desk to save space.

What do we know about case of `A`?

```
$ raku -e 'say $_, " ", "A".uniprop( $_ ) for "Cased", "Lowercase", "Uppercase", "Lowercase_Mapping"'

Cased True
Lowercase False
Uppercase True
Lowercase_Mapping a
```

I won't go into rabbit hole of [titlecase vs uppercase](https://unicode.org/faq/casemap_charprop.html#4). Foldcase used for comparison will appear in another post of this series. But `Cased` is an interesting property, showing if given code point has upppercase/lowercase form. For example there is no concept of letter case in Kanji:

```
$ raku -e '"女".uniprop( "Cased" ).say'
False

$ raku -e 'say "女".lc.ord == "女".uc.ord'
True
```

![Dragon, source https://nohat.cc](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nokvo1u7tsw65o7q6dl6.png)

- Numeric value

For numbers Unicode also holds value.

```
$ raku -e '"4 Ⅴ ¾ 8️⃣ ㊷ 兆".uniprops( "Numeric_Value" ).grep( Int|Rat ).say'

(4 5 0.75 8 42 1000000000000)
```

This is super useful in text normalization. Although you must be aware of different numeric systems if you want to convert text to numeric type in your programming language. For example `ⅤⅠ` is not `51` but 6 in Roman numerals. Also Roman numerals are tricky - because of frequent use in the past on watches and clocks there are code points defined up to value of 12. So Roman `11` can be expressed by single code point `Ⅺ` or by two code points `ⅩⅠ`.

Just for completion - there are no numeric values defined for constants like `π` or `ℇ`. Despite the fact, that `ℇ` is `EULER CONSTANT` `U+2107` character that has no other purpose in life than being a numeric value.

[Raku](https://www.raku.org/) note: I added spaces between graphemes to increase readability, so later I had to filter out `NaN` values from the result by extracting only Integers and Rationals.

- more Punctuation 

Punctuation is **huge** in Unicode, there are 7 main categories for it - `Connector`, `Dash`, `Open`, `Close`, `Initial`, `Final`, `Other`. Full explanation is way beyond the scope of this series, but I want to show few examples that may help you working with text right away.

Regular ASCII? Ethiopic full stop? Double question mark? Extracting sentences from text has never been so easy:

```
$ raku -e '"!?.።⁇".uniprops( "Sentence_Terminal" ).say'
(True True True True True)
```

Well, until you get to articles about `J. F. Kennedy`, but that will be covered in future post about regular expressions.

Just as there are tons of sentence terminals there are also many dashes, 29 to be precise. Now you can find them easily:

```
$ raku -e '( "-", "—", "⸺", "⸻" ).map( *.uniprops( "Dash" ) ).flat.say'

(True True True True)
```

_Did you know_ that hyphen `‐` should not be slapped everywhere you mean dash? According to rules `semi-final` uses hyphen, inclusion `I was — as always — hungry` uses em-dash (named after width of letter `m`) and ranges `2020–2023` uses en-dash (named after width of letter `n`). Yeah, right...

For brackets Unicode brings one more tool to your toolbox - you can check if bracket is opening or closing and you can find matching one:

```
$ raku -e '.say for "(}".uniprops;'

Ps # Open_Punctuation
Pe # Close_Punctuation

$ raku -e '.say for "(}".uniprops( "Bidi_Mirroring_Glyph" );'

)
{
```

**Can you feel the power already?**

This post only scratched the surface of Unicode properties. But with this knowledge you can get any piece of text and be able to parse it without knowing all the letters and symbols used in different scripts.

**Coming up next:** [Composed / decomposed forms](https://dev.to/bbkr/utf-8-decomposition-1n3m).