Evaluating Calibration of A rabic Pre-trained Language Models on Dialectal Text (2025)

Abstract: While pre-trained language models have made significant progress in different classification tasks, little attention has been given to the reliability of their confidence scores. Calibration, how well model confidence aligns with actual accuracy, is essential for real-world applications where decisions rely on probabilistic outputs. This study addresses this gap in Arabic dialect identification by assessing the calibration of eight pre-trained language models, ensuring their predictions are not only accurate but also reliable for practical applications. We analyze two datasets: one with over 1 million text samples and the Nuanced Arabic Dialect Identification dataset(NADI-2023). Using Expected Calibration Error (ECE) as a metric, we reveal substantial variation in model calibration across dialects in both datasets, showing that prediction confidence can vary significantly depending on regional data. This research has implications for improving the reliability of Arabic dialect models in applications like sentiment analysis and social media monitoring.

## introduction arabic pre-trained language models (plms) have advanced significantly in dialect identification and classification, with most research focusing on improving accuracy and dataset development. however, these efforts often overlook calibration-how well a model's confidence scores align with the true probability of correct predictions @xcite . calibration is crucial for arabic dialect applications, where nuanced regional variations in language can lead to significant social and cultural implications if predictions are unreliable. in realworld applications like sentiment analysis, social media monitoring, and policy-making, accurate yet calibrated predictions are essential to support informed decision-making. this study addresses this gap by evaluating the calibration of existing arabic pre-trained models on dialectal text. using 1 million text samples automatically annotated and nadi-2023 datasets, we conduct calibration analysis exclusively on cases where all eight models unanimously agree on dialect labels, focusing on high-confidence predictions. we employ metrics such as expected calibration error (ece) to measure the alignment between model confidence and accuracy, assessing the trustworthiness of these models in dialect classification. by focusing on calibration, this work goes beyond accuracy metrics to highlight the reliability of model predictions. calibration evaluation not only aids in model selection for high-stakes applications but also informs areas for improvement, ensuring that arabic dialect models are both accurate and dependable in practice. ## result analysis and discussion the calibration analysis across models and dialects reveals distinct trends in model reliability on both automatically and manually annotated datasets. models 1 and 8 demonstrate more consistent calibration across dialects and datasets, suggesting they are better suited for varied dialectal data and annotation styles. in contrast, models 4 and 5 show higher calibration errors, especially on manually annotated data, indicating a greater sensitivity to the complexities introduced by human annotations. this difference underscores the potential need for fine-tuning or recalibration when applying these models to manually annotated datasets to enhance their predictive confidence. additionally, the calibration differences across dialects reveal that certain dialects, such as palestinian and sudanese, are more challenging for the models to interpret consistently, displaying higher calibration errors. this pattern suggests that these dialects might require additional data or targeted adjustments to improve model alignment. overall, these findings emphasize the importance of considering both annotation type and dialect specificity when evaluating model calibration, as these factors can significantly impact model reliability in multilingual and multi-dialectal applications. ## limitation this work has some notable limitations that could impact the generalizability and comprehensiveness of the findings. first, while the analysis provides insights into the calibration of eight pre-trained language models, it is constrained by the choice and availability of these models. each model has been pre-trained on varying datasets, which may lack consistent or comprehensive coverage of specific arabic dialects, thereby limiting our ability to capture the full linguistic diversity within arabic dialects. consequently, the calibration results might reflect biases inherent in the pre-training datasets rather than purely dialectal features. second, the study relies solely on expected calibration error (ece) as the calibration metric, which, while informative, provides only a single perspective on model calibration quality. ece does not capture all aspects of prediction reliability, such as miscalibration at different confidence levels or the potential impacts of class imbalance in dialect distribution. integrating additional calibration metrics, like brier score or maximum calibration error (mce), might provide a more comprehensive evaluation of model performance across dialects. additionally, the study does not consider the contextual or pragmatic nuances present in realworld dialectal arabic, as these models may not account for complex language variations or codeswitching phenomena commonly seen in arabic dialects. this limitation may impact the reliability of model predictions when applied to more dynamic or informal arabic text data, such as social media posts, which often contain non-standard dialectal expressions. finally, the study focuses on calibration without incorporating linguistic or sociolinguistic factors that could influence model performance across dialects. factors such as geographical proximity, historical language influences, and sociolinguistic prestige of certain dialects could affect model calibration in ways that ece alone cannot capture. future research could benefit from a more interdisciplinary approach that considers these factors, potentially enhancing model calibration for specific dialectal groups. ## conclusion and future work the ece analysis demonstrates considerable variability in model calibration performance across both automatically and manually annotated datasets for arabic dialect prediction. models 1, 6, and 8 exhibit relatively lower ece scores, suggesting they maintain more reliable calibration across different dialects and annotation types. conversely, models 4 and 5 display notably higher calibration errors, particularly with manually annotated data, which highlights the impact of annotation style on calibration outcomes. this variability suggests that certain models are better suited to dialectal arabic tasks, though a one-size-fits-all approach may not be feasible given the complexity of the data. since the data in the automatically annotated dataset was randomly sampled without balancing dialect distribution, future work can explicitly address this by exploring techniques like resampling or re-weighting to assess their impact on the reliability of the findings. we also plan to improve model calibration in arabic dialect prediction with focus on dialect-specific calibration techniques, with a particular emphasis on dialects that exhibit higher calibration errors, such as palestinian and sudanese arabic. approaches such as fine-tuning models with dialect-specific data or applying posthoc calibration methods may enhance model reliability for these challenging dialects. additionally, investigating why certain models like models 1, 6, and 8 perform better could yield insights into ar-chitectural or pre-training factors that contribute to calibration efficacy. incorporating domain-specific knowledge on linguistic features unique to each dialect may further enhance calibration, especially for dialects with distinct phonological or lexical characteristics.