A rabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization (2024)

Abstract: The widespread absence of diacritical marks in Arabic text poses a significant challenge for Arabic natural language processing (NLP). This paper explores instances of naturally occurring diacritics, referred to as “diacritics in the wild,” to unveil patterns and latent information across six diverse genres: news articles, novels, children’s books, poetry, political documents, and ChatGPT outputs. We present a new annotated dataset that maps real-world partially diacritized words to their maximal full diacritization in context. Additionally, we propose extensions to the analyze-and-disambiguate approach in Arabic NLP to leverage these diacritics, resulting in notable improvements. Our contributions encompass a thorough analysis, valuable datasets, and an extended diacritization algorithm. We release our code and datasets as open source.

## introduction arabic orthography is infamous for its high degree of ambiguity due to its infrequently used optional diacritical marks. while other semitic languages like hebrew and syriac use similar systems, arabic has a richer inflectional space with case endings and other orthographic choices that make arabic more complex. interestingly, diacritical marks in arabic are common in limited contexts where correct reading is a goal: holy texts, poetry and children's books, as well as books for adult literacy and non-native learners. but in general reading contexts, for literate arabic native speaker adults, diacritical marks are used frugally: ∼1-2% of words are partially diacritized @xcite . we refer to these as diacritics in the wild (wilddiacs). in this paper, we follow in the footsteps of other researchers to investigate whether such precious occurrences can be exploited to help improve the quality of arabic nlp tools @xcite @xcite . we specifically focus on the arabic diacritization task, which is at once (a) a final target downstream application given the sparse and variable use of diacritics in arabic; and (b) an enabling technology for other applications such as speech synthesis @xcite . while the percentage of wilddiacs is small, our guiding intuitions are that on large scales, these are objects worthy of study, and given the extra information provided in such contexts, we assume the writers who added them wanted to provide hints to support optimal reading, e.g., to avoid garden path sentences. for a well-rounded exploration, we analyze and compare the diacritization patterns across six genres: news of multiple agencies, novels, children's books, poetry, political/legal documents of the un, and chatgpt output (which sometimes introduces diacritics unprompted). furthermore, we examine the diacritization patterns and choices in two commonly used datasets for evaluating arabic diacritization: the penn arabic treebank @xcite and wikinews @xcite . we also develop a new annotated dataset that includes instances of partially diacritized words along with their full diacritization, which we define carefully, acknowledging different practices. and finally, we propose an extension to the analyzeand-disambiguate approach (habash and @xcite @xcite to improve the quality of its choices, and we evaluate on the data we annotated. our contributions are the following: our code and datasets will be open-source and publicly available. 1 the paper is organized as follows. section 2 discusses arabic diacritics. section 3 reviews related work. section 4 covers the datasets, including our annotated datasets, and wilddiacs usage statistics. section 5 details the approach we build on and how we extend it. section 6 presents the experimental setup, evaluation, and error analysis. ## arabic diacritics we present an overview of arabic diacritics in terms of their form, function, and use. ## exploiting diacritics in the wild our basic approach to exploiting wilddiacs is based on the intuition that they are simply hints that we need to take @xcite . we build on a hybrid (neuro-symbolic) analyze-anddisambiguate solution for arabic diacritization implemented in camel tools @xcite . while this open-source toolkit reports competitive results @xcite , it has a number of limitations connected to its dependence on the-less-than-ideal atb diacritization (training and tools). we present the basic approach followed by our modifications and extensions. ## evaluation 6.1 experimental setup we evaluate our enhanced model in various settings to assess the impact of different extensions. regarding the ranking algorithm, we compare the camel tools baseline (ct) with our extended re-ranking approach (ct++). additionally, we conduct an oracle evaluation to determine the upper limits of both versions by selecting the analysis closest to the gold diacritization for each word. for context modeling, we examine three scenarios: no modeling (none), modeling as standalone utterances (solo), and full context (full ## conclusions and future work in this paper, we conducted a detailed analysis of wilddiacs patterns across six different genres of arabic text. we developed wild2maxdiacs, a new annotated dataset designed for evaluating models to exploit wilddiacs for maximal diacritization. additionally, we extended the wikinews dataset to wikinewsmaxdiacs, aligning it more closely with our definition of maximal diacritization. we also enhanced an open-source toolkit, camel tools, to effectively utilize partial diacritization, resulting in improved performance. in the future, we plan to enhance the morphological analyzer's coverage, expand our research to include less explored genres like classical arabic texts, and investigate other wild text signals such as hamza usage and dialectal spelling variations. all of the modifications and extensions will be integrated in a future version of camel tools. we also plan to study the effect of our system on downstream applications such as text-to-speech @xcite and arabic romanization @xcite .