A ra D ia WER : An Explainable Metric For Dialectical A rabic ASR (2023)

Abstract: Linguistic variability poses a challenge to many modern ASR systems, particularly Dialectical Arabic (DA) ASR systems dealing with low-resource dialects and resulting morphological and orthographic variations in text and speech. Traditional evaluation metrics such as the word error rate (WER) inadequately capture these complexities, leading to an incomplete assessment of DA ASR performance. We propose AraDiaWER, an ASR evaluation metric for Dialectical Arabic (DA) speech recognition systems, focused on the Egyptian dialect. AraDiaWER uses language model embeddings for the syntactic and semantic aspects of ASR errors to identify their root cause, not captured by traditional WER. MiniLM generates the semantic score, capturing contextual differences between reference and predicted transcripts. CAMeLBERT-Mix assigns morphological and lexical tags using a fuzzy matching algorithm to calculate the syntactic score. Our experiments validate the effectiveness of AraDiaWER. By incorporating language model embeddings, AraDiaWER enables a more interpretable evaluation, allowing us to improve DA ASR systems. We position the proposed metric as a complementary tool to WER, capturing syntactic and semantic features not represented by WER. Additionally, we use UMAP analysis to observe the quality of ASR embeddings in the proposed evaluation framework.

## introduction state-of-the-art (sota) asr systems such as wav2vec2 xlsr-53 @xcite , hu-bert @xcite , and whisper @xcite are designed to perform on a wide range of languages, including arabic speech. to benchmark these models, wer and character error rate (cer) metrics are used to calculate the number of words inserted, substituted, and deleted in transcribed speech. wer then calculates the error per- * corresponding author centage by dividing by the total number of words in the predicted transcript. this yields an error rate that quantifies a basic comparison without any language-specific analysis. wer is not designed to consider any form of syntactical or semantic differences in the reference transcript and the predicted transcript, but rather to compare them word by word. this form of calculation poses a gap in the evaluation methodology used for benchmarking asr systems that deal with multiple languages and dialects of the same language, particularly dialectical arabic (da), which imposes a multitude of morphological and orthographic variations. in the evaluation landscape, metrics present themselves as the source of truth for the quantities they report. however, most metrics used in research today do not give researchers enough insight into the reasoning behind the results and the methodologies used within the metric. this poses a critical issue for explaining results when a system deals with a multitude of morphological variations in speech. to improve the explainability of the results, our research work focuses on proposing a transparent method that provides a new metric named aradiawer that is based on wer with a new explainable identity, allowing the metric to report additional semantic and syntactic scores. it is well established that wer could be used as a benchmark metric for most speech recognition tasks, and in most languages, it works fairly well. however, the challenges imposed by synthetic languages and the lack of syntactic and semantic context of wer, as shown in @xcite , have required researchers to explore methods designed around the language itself. @xcite @xcite @xcite ) are five sota systems that propose supervised, unsupervised, and objective-based evaluation of arabic asr systems. semdist @xcite , flores @xcite , and the study of lexical distance @xcite provide a semantic distance component combined with ner tags and intent recognition to improve the evaluation of asr systems. our proposed aradiawer metric incorporates semantic and syntactic scoring, fluency scoring, and a umap analysis to better explain the performance of da asr, while also keeping the traditional metrics (i.e., wer) intact and available for benchmarking purposes. ## aradiawer methodology to introduce additional syntactic and semantic variances to the existing wer metric and enhance its explainability, we proposed the aradiawer metric. we used a weighted sum approach to capture more differences in utterances while maintaining the integrity and distribution of wer. we designed aradiawer to depend on the semantic and syntactic weight generated by other models through a factor we call the error weight or w err . the portable dependency on other lms for the semantic and syntactic scores provides flexibility for other researchers to improve aradiawer or adjust the sota models used for the syntactic and semantic components to their specific use cases. to assess the performance of the proposed metric, we fine-tuned a wav2vec2-based model with a connectionist temporal classification scorer on a large arabic speech dataset with more than ten dialects. we compared the performance of the finetuned model with five other state-of-the-art asr systems. ## results and analysis tables 4 and 5 summarize the results for each system. the averaged results of aalto showed that the best performance is observed when a system is trained and tested in a fully supervised approach on the same distribution and language. the results of tdnn show that even legacy systems can perform well when it comes to capturing syntactic and semantic patterns. this is further proven in the ablation studies for aalto and tdnn, where the number of syntactic tags captured is negatively correlated with the penalty-reducing error weight w err (see tables 1 and 2 ). linking this to the correlation analysis in aalto, it is possible to deduce that capturing additional syntactic tags can lead to improved syntactic scores and better overall capture of dialectical variations in utterances, decreasing the error weight and aradiawer value. higher semantic scores indicate a better contextual understanding of the utterance, allowing for more accurate prediction of words that are similar and reducing the rmse between wer and aradiawer. in addition, less complex utterances observe higher syntactic and semantic scores. lastly, certain outliers in the dataset still achieve high semantic and syntactic scores but fail at fluency; this shows the metric's ability to pinpoint low-quality utterances that are not intelligible. the inclusion of rmse in the calculation of the results serves as a means of quantifying the differences between wer and ara-diawer. by measuring the rmse, we can assess the degree of agreement between the two metrics and determine the extent to which aradiawer captures variations in dialectal arabic speech that are not fully represented by the traditional wer. this additional analysis provides further insight into the strengths and limitations of aradiawer, enabling researchers and practitioners to better understand the implications of adopting this new metric in the context of da asr systems evaluation. the experimental study reveals that the use of aradiawer brings an average improvement of 18.65% in error rate compared to wer. this improvement does not necessarily suggest that our metric is a direct replacement for wer or that it outperforms it in all aspects. rather, our approach offers a transparent and traceable method that utilizes language models to evaluate da asr systems in a more comprehensive and interpretable manner. ## conclusion the focus of this paper is to propose an explainable evaluation metric, aradiawer, that complements wer and is designed to assess the performance of automatic speech recognition (asr) systems for dialectal arabic speech. this metric combines three different scoring systems, namely syntactic, semantic, and fluency, by utilizing state-of-the-art models. the main objective of aradiawer is to provide a more detailed and inclusive assessment of the performance of asr systems in the context of dialectal arabic speech, which is a significant improvement compared to the conventional word error rate (wer) metric alone. this work can be considered a resource tool to capture the dialectal variations in speech, where the addition of syntactic features, such as parts of speech tags and lemmas, is helpful for improving the overall performance of the metric. moreover, the incorporation of semantic features allows the asr to be evaluated based on meaning, thus ensuring a more holistic assessment of the asr system. therefore, we do not seek to undermine the importance of wer but to offer a complementary tool that enables a more extensible evaluation of da asr systems the aradiawer framework relies on language models (lms) to extract both syntactic and semantic features from the text. while lms are primarily trained for syntactic features, they also contain information about semantic features. the syntactic component assigns morphological and lexical tags to the text using the embeddings of camelbert-mix lm. the semantic component uses the embeddings of minilm and cosine similarity to calculate the semantic similarity between the reference and predicted transcripts. the embeddings of each lm are used to extract explainable correlations between errors made by the asr and aradiawer's two scores (semantic, and syntactic). this allows us to capture both semantic and syntactic features and make more comprehensive and interpretable assessments of the asr systems. additionally, the proposed evaluation framework uses a umap analysis to evaluate the semantic patterns in a low-dimensional space and the gpt-2 generated perplexity score to determine the fluency of an utterance. in conclusion, while there is still room for improvement, our proposed aradiawer metric represents a step forward in the comprehensive evaluation of asr systems, especially in the context of dialectal variations. in future work, we plan to further improve the metric by incorporating multilingual language models to capture additional morphological and orthographic patterns in the transcripts, target a wider range of diverse datasets, and use modern lms like gpt-4, lamda, and llama to interpret perplexity and aradiawer results even further for a more detailed analogy.