Uncertainty-driven Partial Diacritization for A rabic Text (2025)

Abstract: We present an uncertainty-based approach to Partial Diacritization (PD) for Arabic text. We evaluate three uncertainty metrics for this task: Softmax Response, BALD via MC-dropout, and Mahalanobis Distance. We further introduce a lightweight Confident Error Regularizer to improve model calibration. Our preliminary exploration illustrates possible ways to use uncertainty estimation for selectively retaining or discarding diacritics in Arabic text with an analysis of performance in terms of correlation with diacritic error rates. For instance, the model can be used to detect words with high diacritic error rates which tend to have higher uncertainty scores at inference time. On the Tashkeela dataset, the method maintains low Diacritic Error Rate while reducing the amount of visible diacritics on the text by up to 50% with thresholding-based retention.

## introduction arabic script relies on diacritics, commonly referred to in arabic as tashkeel ( ) to mark short vowels, gemination, and other phonemic distinctions that may not be represented by the base letters. fully-diacritized text eliminates ambiguity and supports precise pronunciation, which is helpful for applications such as text-to-speech (tts) synthesis, machine translation, and language learning @xcite . however, when every letter carries its diacritic, the resulting text becomes visually dense and can slow down readers @xcite . partial diacritization can be employed to balance disambiguation and readability and to optimize performance in downstream nlp applications. state-of-the-art transformer-based diacritization models can achieve diacritic error rates (der) below 2% on standard benchmarks @xcite , but their performance may degrade in out-of-domain data @xcite . these models often operate as "black boxes," where model outputs are accepted regardless of confidence scores. their outputs are often fully diacritized, which increases visual complexity and can slow down reading speed and reduce clarity. furthermore, many diacritics are redundant, particularly in common words where pronunciation is intuitive or easily inferred. these factors make fully diacritized (fd) text less practical for general application, motivating the need for partially diacritized (pd) text in such settings. prior studies proposed computational approaches that rely on heuristics and morphological analysis to perform partial diacritization @xcite . others have proposed neural networks @xcite with some success. however, research on partial diacritization remains limited, largely due to the difficulty of evaluating performance; optimal partial diacritization is an illusive concept with no standard evaluation framework or metrics. in this paper, we explore an uncertainty-driven framework for pd and provide a preliminary intrinsic evaluation of this framework through error analysis. we evaluate three uncertainty metrics: softmax response, bayesian active learning by disagreement (bald) via monte carlo dropout, and mahalanobis distance in latent feature space. at inference time, the predicted diacritic of each character is compared with a chosen threshold θ. we may keep the diacritic if the uncertainty score is above or below said threshold, allowing for flexibility in application. to mitigate the well-known overconfidence of deep networks on rare or ambiguous inputs, we experiment with a lightweight, simplified confident error regularizer (cer) that penalizes high-confidence mistakes during fine-tuning. we summarize our contributions as follows: ## related work early approaches to arabic diacritization employed hidden markov models and morphological analyzers such as mada and madamira @xcite , @xcite , @xcite , achieving good accuracy by leveraging lexical features and large amounts of data. with the advent of neural networks, recurrent architectures and encoder-decoder transformers improved der to below 5%. the characterbased arabic tashkeel transformer (catt) (?) is one such innovation, achieving very good results on both its variants, encoder only (eo) and encoder-decoder (ed). partial diacritization has been studied from linguistic and machine learning perspectives with the aim of improving both nlp systems as well as improving text readability. rule-based schemes target case endings or homograph disambiguation, while supervised methods train classifiers to decide which positions to diacritize. @xcite investigated the impact of various diacritization schemes on statistical machine translation (smt) from arabic to english. the authors explored different levels of partial diacritization and found that partial diacritization could improve translation quality by reducing ambiguity without significantly increasing vocabulary size or out-of-vocabulary rates. @xcite demonstrated improvements in machine translation by employing partial diacritization strategies targeting syntactic clarity. building on these findings, the authors also employed selective diacritic restoration specifically for homograph disambiguation @xcite . @xcite further advanced pd research by achieving state-of-the-art results and seamless integration into machine translation workflows. @xcite introduced regularized decoding and adversarial training to improve diacritization robustness and accuracy. recently, @xcite analyzed naturally occurring instances of partial diacritics across diverse text genres, creating practical datasets for enhanced real-world applications. ## methodology our methodology for partial diacritization is to use model uncertainty to guide the removal or retention of diacritics based on target criteria. for example, if we have a fully diacritized text and wish to minimize the diacritics for improved readability, uncertainty scores may be helpful in identifying which diacritics to retain by keeping the ground truth diacritics in places with high model uncertainty. in applications where a diacritic restoration model is used directly to annotate undiacritized text, we may wish to remove predicted diacritics with high uncertainty and maintain low diacritic error rates in the resulting text. our methodology and preliminary analysis enable both types of application by exploring the relationship between uncertainty scores and diacritic error rates. in the following sections, we describe the base model, uncertainty metrics, and the calibration scheme used to improve uncertainty estimation for diacritic restoration. ## experiments this section presents exploratory and experimental analysis of our approach. we analyze the performance of different uncertainty estimation methods, with a particular focus on sr as our primary score, a choice we justify below. we also evaluate the impact of confidence calibration through cer, not to be confused with character error rate, with different regularization strengths. the analysis addresses several key aspects: the relationship between uncertainty thresholds and diacritization coverage, error rates at different thresholds of uncertainty, the effectiveness of uncertainty in identifying difficult words, and the calibration quality of the model with and without cer. ## discussion and conclusions we explored the application of uncertainty estimation for partial diacritization of arabic text. we experimented with various uncertainty estimation methods, and established the suitability of softmax response for this task. the other two metrics we explored had some drawbacks that made them less suitable for our task. while the mahalanobis distance exhibited some correlation with diacritic error rates, the effect is weaker than the other two methods, resulting in higher error rates at the same coverage points. bald achieved sim-ilar correlation to sr, but it is less suitable for practical diacritc restoration models that involve sequence labeling due to its higher computational cost. among the three metrics, sr provides optimal performance and efficiency, making it suitable for additional calibration and practical deployment. however, our experiments show that better calibration results in higher der, so additional work is needed to develop calibrated models that retain base accuracy. nevertheless, the calibrated model shows potential for identifying ambiguous words, which we define as words with high der, in terms of precision. this indicates that calibration may still be useful for some target application, where identifying ambiguous words with high precision (albeit with low recall) is desired. this preliminary analysis illustrates that additional work is needed to identify suitable calibration methods that optimize uncertainty estimation while maintaining the performance of the base diacritic restoration model. in terms of application, the sr-based approach is straightforward to integrate into any neural-based diacritic restoration model, and our experiments show that we can reduce the relative der in partial diacritization with various coverage thresholds. such approach can be used in user-facing applications where automatic diacritization is used to annotate undiacritized text, leading to a partially diacritized text that is more accurate than the baseline. however, such methods do not address the issue of ambiguous words, which are likely to rethe words that are identified as high-uncertainty could retain their diacritics, while diacritics on lowuncertainty words can be dropped. examples of sentences and their partial diacritics using each of these proposed schemes are shown in our exploratory analysis provides a starting point for such applications; further evaluation and analysis are needed to verify the effectiveness of such approaches in practical applications like readability enhancement, machine translation, and textto-speech synthesis.