What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations (2024)

Abstract: This paper explores the pitfalls in evaluating multilingual automatic speech recognition (ASR) models, with a particular focus on Indic language scripts. We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta’s MMS, Seamless, and Assembly AI’s Conformer, and their unintended consequences on performance metrics. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, by removing inconsistencies such as variations in spelling, punctuation, and special characters, are fundamentally flawed when applied to Indic scripts. Through empirical analysis using text similarity scores and in-depth linguistic examination, we demonstrate that these flaws lead to artificially improved performance metrics for Indic languages. We conclude by proposing a shift towards developing text normalization routines that leverage native linguistic expertise, ensuring more robust and accurate evaluations of multilingual ASR models.

## introduction automatic speech recognition (asr) systems have become increasingly relevant in various applications, ranging from voice assistants and transcription services to accessibility tools for the disabled population. the performance and usability of asr models are evaluated in terms of their error rates. recent advancements in open asr models pretrained in self-supervised @xcite @xcite manner or weakly supervised @xcite manner are capable of handling various languages and scripts. these models can be fine-tuned for improved performance in domains or languages of inter-est. this capability has revolutionized speech recognition in ultra low resource languages and scenarios @xcite . many of these models have brought down state of the art (sota) word error rates (wers) on popular benchmarks. evaluation of the performance of asr models are often affected by the prediction differing from the ground truth in letter casing, punctuation, spelling variants etc. leading to inflated wers. to mitigate this, a text normalization routine is employed @xcite . a proper text normalization routine is required to minimize penalization of non-semantic differences by aligning the predicted output more closely with the ground truth. the study presented in this paper examines the pitfalls in the current normalizations routines employed in the latest asr models on the banchmarking of non-english languages, specifically on many asian languages that use indic scripts foot_0 . our empirical analysis reveals that the current normalization practices can result in significant errors, particularly in many low-resource languages, by boosting the model performance on many benchmarks and misleading the research community. we propose for the development of linguistically informed normalization routines that account for the unique characteristics of each language, ensuring a fair and reasonable evaluation and benchmarking process for multilingual asrs. ## background and related works prior to the introduction of openai's whisper model @xcite , most asr systems were trained on normalized text transcripts and produced output without punctu- whisper's normalization routine for english extends beyond basic casing and punctuation, incorporating transformations such as converting contracted abbreviations to expanded forms and expanding currency symbols. however, this approach would require a languagespecific set of transformations for non-english text. due to the lack of linguistic knowledge to develop such normalizers for all languages, the whisper's normalization relies on a basic datadriven approach, which includes replacement of characters in the mark class with spaces and removes successive whitespace characters to a single instance @xcite . the non-english normalization routine employed by whisper, inadvertently removes vowel signs (matras), that belong to the the mark class of unicode characters. these vowel signs, essential for correct word formation and pronunciation, are removed along with other punctuation marks, leading to significant distortions in the text in languages such as hindi, bengali, tamil, and others (o'connell, 2023; @xcite . this results in words being broken down into consonants without their associated vowels, causing a loss of meaning and intelligibility. this also leads to incorrect wer calculations for languages written in indic scripts. additionally, thai, which does not use spaces between words but relies on spaces to delimit sentences, is also affected. the normalization process inserts spaces instead of vowel signs, effectively distorting the nature of the language. see this normalization routine has been adopted by various later models, including meta's mms, seamless series @xcite and assem-blyai's conformer-1 (assembyai, 2023) for evaluation and benchmarking and is integrated into huggingface transformers foot_1 , thus amplifying its impact. ## methodology in this study, we present two complementary empirical evaluations to assess the impact of whisper's normalization routine on different languages. first, we conduct an intrinsic evaluation by comparing the similarity of example sentences from various languages before and after normalization, using the meteor score as a text similarity metric. second, we perform an extrinsic evaluation by measuring the wer on a multilingual benchmark dataset for the same set of languages, with and without the application of whisper's normalization. for our case study, we used both the baseline and fine-tuned whisper asr because their outputs include punctuation, unlike other asr models in the literature. this allowed us to demonstrate the impact of normalization on asr outputs with punctuation. all the datasets and the models used in this experiments are available under permissive licenses in huggingface repositories and listed in appendix a. all the evaluations were run on a single nvidia a100 gpu. ## recommendations findings from our empirical evaluation underscore the importance of language-specific normalization routines to ensure accurate text representation and reliable performance evaluation in many underrepresented indic languages. building up on our findings, we propose a collaborative approach, leveraging the collective efforts of native speakers and linguistic experts to develop effective normalization routines for diverse linguistic contexts. ## conclusions the empirical evaluation conducted in this study highlights that the current practice of normalization severely affects the text representation across languages, resulting in artificially boosted wer and sota performance. by adopting a more tailored approach to evaluations, we can enhance the reliability of multilingual asr models, making them truly inclusive and effective across diverse linguistic landscapes.