If you say they should be considered 15 characters then software and devs should support optionally indexing and counting them as 15 characters. This is the most important point.

And, as a corollary, software devs should aspire to have and know about string functions in software that recognize that the text string I used is 15 characters long in contexts where that's the right way to view it. Furthermore, those functions should asap be as easily available for use as they are today for recognizing that the text 'What does "index" mean?' is 23 characters long.

This notion of software and devs properly indexing and counting characters was the ultimate point of my comment, as I will elaborate below. I hope that you will reply to confirm you understand the gist of what follows; that would make my day and leave this exchange on HN to hopefully shine light where it's sorely needed. :)

----

The OP title is "UTF-8 String Indexing Strategies". I could write that this begs the question What does "index" mean? Unfortunately it seems it still doesn't beg the question -- in 2019 -- for most western devs.

Last century devs generally assumed the index unit was bytes. So they created programming languages whose string type assumed indexing in bytes and functions and libraries that did the same. Nowadays they're starting to assume "codepoints", which is an equally broken assumption. (Codepoints are a Unicode notion and they're great for what they're great for. But being "characters" is, in the general case, something they're terrible for.)

Both these western devs and the OP are effectively ignoring the possibility that "इंडेक्स" का क्या अर्थ है? could be considered to be 15 characters (or 18). They're ignoring you, the half of the planet that are in a similar boat, and the whole of the planet that's coming together, sharing text like we are here.

----

bakery2k demonstrated the problem. They wrote:

> MicroPython ... indexing ... O(n) ... PyPy's ... O(1)

Neither of these deals with indexing characters, as one might expect based on an ordinary human's understanding of the word "characters". Instead they're myopically focused on indexing bytes and codepoints.

This goes hand-in-hand with Python's length function returning 26 for the text "इंडेक्स" का क्या अर्थ है?. It's counting codepoints, not characters, which is close to useless for that text.[1]

But you wouldn't have any clue about that from bakery2k's comment and it looks like bakery2k has no awareness of this:

> I wonder if new high-level languages would be better off using one of these [byte and codepoint] representations, or taking the Go/Julia approach of only indexing bytes.

Imo that's shockingly retrogressive given the lack of discussion of characters.

----

Chances are good that if you try to select the text I wrote one character at a time you will find that you can cursor across 18 units.

Why/how does software do this? It relies on part of the Unicode standard for indexing that builds on the concept of "what a user thinks of as a character".[2]

This mechanism allows the string to be indexed/counted as N characters, where N varies according to the definition of "character". Software is supposed to choose the definition with appropriate adherence to the Unicode standard, which includes customizing it as necessary to produce practical results. And, as I noted, most good modern software dealing with cursoring/editing text gets it right per the Unicode standard.

My guess the Unicode standard by default has software consider क्स to be 2 characters because the consonant is comprised of क् and स placed visually side-by-side whereas it has डे be considered 1 character because it's comprised of ड and ए somehow overlapping visually. (That's a pure guess. Please let me know if it sounds crazy. :))

For some other use cases, like a native speaker just reading text abstractly, what's a character changes. You say the text I wrote is 15 characters; therefore software should be able to index and count it as 15 characters.

I hope that all makes sense. Thank you for your comment, reading my reply, and TIA for any reply. :)

[1] https://tio.run/##K6gsycjPM/7/v6AoM69EIyc1T0Nd6cGS9gdLmh4sWf...

[2] https://unicode.org/glossary/#grapheme