Don’t Touch My Diacritics (2025)

Abstract: The common practice of preprocessing text before feeding it into NLP models introduces many decision points which have unintended consequences on model performance. In this opinion piece, we focus on the handling of diacritics in texts originating in many languages and scripts. We demonstrate, through several case studies, the adverse effects of inconsistent encoding of diacritized characters and of removing diacritics altogether. We call on the community to adopt simple but necessary steps across all models and toolkits in order to improve handling of diacritized text and, by extension, increase equity in multilingual NLP.

## introduction virtually all natural language processing workflows begin with the ingestion of text data (with or without annotations). this data is usually not a random sample of written language, but rather has been sampled or filtered for language, script, quality, or relevance, and may also have been subject to various substitutions (e.g., case-folding, removing markup, or excising encoding errors). yet preprocessing, as this process of text preparation is known, is generally regarded as more of a dark art than a topic for research, and as such has received minimal attention in the literature. while data preprocessing may not be the most important component of a speech or nlp system, any errors made at that stage are likely to propagate, and decisions made during preprocessing necessarily constrain what is possible downstream. in this paper we focus our attention on the consequences of decisions made while preprocessing text with diacritics, a notion we define below. using case studies, we show that failure to apply consistent unicode normalization, which provides a canonical representation of text with diacritics, leads to degradation in downstream performance. we then show that stripping diacritics, common when pre-training large neural network language models (llms), also leads to degradation. our recommendations, then, are simple: text preprocessing regimens should apply a consistent unicode normalization-any of the normalization forms will do-but in most cases, should not attempt to strip diacritics. ## defining diacritics written text consists of atomic units sometimes called glyphs. these glyphs act as the primary spatial units in text and their order mirrors the temporal ordering of the orthographically relevant linguistic units in the corresponding utterances @xcite . glyphs may also bear non-spacing marks appearing above, below, to the left or right, or even surrounding, the glyph; it is these marks we will call diacritics. it may be difficult to discern whether marks of these sorts are really "part" of a glyph or glyphs on their own, and judgments also seem to vary from language to language, reflecting sprachgefühl or conventions learned in school. 1 reflecting this ambiguity, unicode often provides multiple ways to encode diacritized glyphs. for example, an e with an acute accent can either be encoded either as a single character ⟨é⟩ (u+e9), or as an e (u+65) followed by a combining acute accent (u+301). similarly, consider the hindi word साड़ ी 'sari'. in this word, one character is marked with a dot (a nuqta) underneath. with this dot the character is read as [ɽiː]; without it, it is read as @xcite . in unicode one can encode ⟨ड़ ⟩ either as a single precomposed character ड़ (u+95c) or as a se- it is straightforward to apply unicode normalization (see appendix a for a brief tutorial) to convert between these two representations, but without normalization ⟨é⟩ and ⟨é⟩, and ⟨ड़ ⟩ and ⟨ड़⟩, are considered unequal by ordinary string comparison methods (e.g., the c standard library function strcmp, or @xmath0. at the same time, some characters that might naïvely appear to be diacritized forms of others are not regarded as such by unicode. for example, the "belted l" ⟨ł⟩ used in polish (among other languages) does not decompose into ⟨l⟩ and a diacritic as one might expect, nor does the "o with stroke" ⟨ø⟩ used in various languages of scandinavia decompose into ⟨o⟩ and a diacritic. one interesting comparison is between the "square script" used to write modern hebrew and the (perso-)arabic script used for modern standard arabic. in the former, consonant pointing (e.g., dagesh lene and the sin/shin dot) is considered optional and unicode regards the diacritic as a separate character. in the latter, consonant pointing (e.g., the dots distinguishing sīn and shīn, and ṣād and ḍād respectively) is mandatory and the points are part of the glyph in all normalization forms. unicode also defines a canonical order for sequences of diacritics for those scripts in which a single glyph may bear multiple diacritics. for instance, in arabic, in addition to the inherent consonant points, there are optional diacritics called tashkīl, including ones denoting quality of the following vowel (the ḥarakāt) and consonant gemination (shaddah). according to unicode's canonical order, the ḥarakāt precede shaddah. canonical order can be enforced by converting text to nfd or nfkd normalization forms. ## the case for unicode normalization applying unicode normalization to text data enforces consistency with respect to two dimensions. first, it ensures consistency in whether diacritics are precomposed or decomposed. secondly, in scripts where a glyph may bear multiple diacritics, it ensures that these diacritic sequences are in a consistent order. the normalization algorithms are deterministic, conceptually simple, computationally efficient, and available in the standard library of nearly all modern programming languages. foot_1 yet even some professionally developed corpora lack consistent normalization; it is not difficult to show that the failure to apply a consistent normalization would have negative consequences for nlp systems. for example, the example in panel (a) of ## the case for preserving diacritics many preprocessing regimens have no need to canonicalize diacritics for the simple reason that they remove them altogether. diacritics recognized as such by unicode can be stripped from text by converting to either of the decomposition normalization forms (nfd or nfkd), which causes the diacritics to be treated as separate characters, and then removing all characters in unicode's "mark, nonspacing" (mn) category. this procedure is, for instance, used by the normalizers in hugging face's tokenizers library. others may choose also to remove characters in the "control" (c) or "separator" (s) categories. an even more aggressive method for stripping pseudodiacritics (like the polish belted l) is proposed by @xcite . for instance, in their method the lower-case belted l ("latin small letter l with stroke") is mapped onto lower-case l ("latin small letter l") because the latter's full unicode name is a proper prefix of the former's. the effect of stripping are evident in not all languages are the same. it is fairly obvious why one might choose to strip diacritics. first, if diacritics are inconsistently encoded, it might be better to simply remove them. however, as just discussed above, it is trivial to apply a consistent encoding to text. secondly, there are many scripts which are only rarely written with diacritics. in arabic and hebrew, tashkīl and niqqud diacritics, respectively, are omitted except in certain pedagogical and religious materials. the same is true of the diaeresis used in russian text to distinguish ⟨ё⟩ [jo] from ⟨е⟩ [je], or the acute accent used to indicate stress. in many other scripts, diacritics are ordinarily present but occasionally omitted due to haste or technical challenges. for example, the contrast between ⟨l, ł⟩ is of some importance in polish, but it is easy to imagine a polish speaker typing ⟨l⟩ in place of ⟨ł⟩ because of limitations in the available text entry system. 4 in the presence of inconsistent diacritization, it might make sense to strip diacritics and thus use undiacritized word forms as equivalence classes for subsequent processing including tokenization. while there are likely some applications where this is a sensible decision, in many others it comes with measurable costs. our colleagues in the social sciences (p.c.) report to us that off-the-shelf hebrew speech recognition systems output undiacritized text. when they attempt to feed the resulting text into nlp analysis pipelines (e.g., pos taggers and dependency parsers), even state-ofthe-art systems struggle with the ambiguity introduced by the absence of diacritics and often produce incorrect outputs that could be avoided if the recognizer produced diacritized text and the nlp pipelines accepted it. modern greek writing makes consistent use of acute accents to mark primary stress. without these accents, ambiguities arise; e.g., νόμος 'law, ordinance' vs. νομός 'county, district'. yet greekbert @xcite , a pre-trained language model for greek, is trained on text stripped of all diacritics. 5 perhaps as a result, the model performs poorly when used for morphological analysis @xcite . @xcite found that biomistral @xcite , when asked to define medical terms given in romanian, not only performs better on undiacritized text (which is more common in everyday usage), but also tends to generate the diacritized form in a parenthetical remark preceding the definition. @xcite study transliteration in twelve languages of south asia. they experiment with a number of pre-trained language models, and find that one of them, mt5 @xcite , has poor vocabulary coverage in certain languages. this is because malayalam and telugu use the zero-width non-joiner character (u+200c) and marathi and sinhala use the zero-width joiner (u+200d), both to block inappropriate formation of conjunct (i.e., consonant cluster) characters, but these characters are absent from the mt5 vocabulary, presumably removed from the training data by an overzealous preprocessing routine. ## automatic diacritization one way to mitigate the effects of stripped diacritics or inconsistent treatment of diacritics may come from diacritization, the task where undiacritized text is annotated with the correct marks. this is now a well-established task in nlp, particularly pertaining to consonantal scripts like those used for arabic and hebrew. however, this is something of an unsolved problem, as systems achieve just over 10% word error rate (wer) in hebrew @xcite . similar wers are reported for arabic; 6 náplava et al. (2018) report that wer exceeds 40% for vietnamese. it may also seem that pre-trained neural language models would be quite effective of this task-in a zero-shot, few-shot, or fine-tuning scenario-even if they have been pre-trained on stripped text. while this certainly has been attempted, many of the state-of-the-art diacritization systems instead use randomly-initialized (rather than pre-trained) neural models. these models are robust, but are outmoded in most other nlp tasks, and one might be surprised to learn that a large amount of undiacritized text is less useful than relatively small amounts of in-domain diacritized data. we believe a major cause of llms' reduced ability to handle these tasks, demonstrated in figure 1 , is the fact that existing preprocessing ourtines prevent models from being exposed to diacritized training data. ## conclusion in this opinion paper, we argue and provide evidence that decisions about how diacritics are treated during text preprocessing may have detrimental downstream effects on model performance. these effects can largely be mitigated by preserving diacritics and by using simple, deterministic methods for ensuring they are encoded consistently. we believe the examples we have reviewed only scratch the surface. many other issues may reside unseen deep within large, black-box neural models where they are difficult to detect. mitigation of these effects, while not particularly hard to implement, does however require the attention and effort of many stakeholders. let us consider the case of large pre-trained models, used either for feature extraction (i.e., as pre-trained encoders) or prompting. given a pre-trained model, even one with detailed documentation (e.g., model cards) it is often difficult to determine what preprocessing steps were used to prepare data for the model. furthermore, given such a model, it is not obvious how one might modify an existing checkpoint-or its tokenizer and vocabulary-of a pre-trained model to improve support for diacriticized text it has not yet been exposed to. thus we are beholden to developers at institutions with sufficient resources to train such models and are in some sense stuck with the preprocessing decisions they have made. foot_6 it is for this reason that we appeal to the community at large, rather than quietly modeling what we have argued to be best practices. developers of llms can increase their overall utility, particularly for languages other than english, simply by applying a consistent normalization and resisting the urge to strip diacritics. we note that this preprocessing issue is orthogonal to the ongoing debate concerning the appropri-ate representation levels in language models. bytelevel and character-level models (e.g., @xcite are not immune to inconsistent encodings of diacritics, though it may be that they are better suited to represent inconsistent encoding than models with larger tokens. visiontransformer processing of rendered text may help address invisible or near-invisible differences in rendered text (e.g., @xcite , but even here caution must be taken regarding any preprocessing performed before the rendering phase. one might have expected that llms, given their impressive performance on a variety of tasks, would be robust to the use of diacritized inputs-or use in diacritization tasks-even when they themselves are not trained on diacritized text, but sadly this does not seem to be the case.