P oly WER : A Holistic Evaluation Framework for Code-Switched Speech Recognition (2024)

Abstract: Code-switching in speech, particularly between languages that use different scripts, can potentially be correctly transcribed in various forms, including different ways of transliteration of the embedded language into the matrix language script. Traditional methods for measuring accuracy, such as Word Error Rate (WER), are too strict to address this challenge. In this paper, we introduce PolyWER, a proposed framework for evaluating speech recognition systems to handle language-mixing. PolyWER accepts transcriptions of code-mixed segments in different forms, including transliterations and translations. We demonstrate the algorithms use cases through detailed examples, and evaluate it against human judgement. To enable the use of this metric, we appended the annotations of a publicly available Arabic-English code-switched dataset with transliterations and translations of code-mixed speech. We also utilize these additional annotations for fine-tuning ASR models and compare their performance using PolyWER. In addition to our main finding on PolyWER’s effectiveness, our experiments show that alternative annotations could be more effective for fine-tuning monolingual ASR models.

## introduction code switching (cs) refers to the common phenomenon of mixing two or more languages within a single conversation. when the language switch happens within the same sentence or utterance, it is called intrasentential cs. while language switches between utterances (intersentential) are relatively easy to detect using language identification models, intrasentential code-switching is far more challenging. in automatic speech recognition (asr), intrasentential cs is particularly difficult to detect and accurately transcribe. moreover, evaluating the performance of asr systems in the presence of cs can be tricky due to script differences, transliteration, or non-standard spelling of foreign words. our exploration of multilingual pre-trained asr models such as whisper @xcite and massively multilingual speech (mms) @xcite reveals inconsistencies in cs transcriptions, such as transcribing in the source script, transliterating into the target script, or even translating into the target language. traditional asr metrics, namely word error rate (wer) and character error rate (cer), are intolerant to such variations, which can affect the ability to accurately compare the performance of various models. an example of this phenomenon is presented in in this work, we propose a novel variant of the wer algorithm designed to address the shortcomings outlined above. in particular, the algorithm allows the specification of different variants for each word in the reference transcription, including transliteration and translation, resulting in a more tolerant treatment of the variations in cs transcriptions for languages of different scripts. for transliteration, we use cer to account for the nonstandard spelling of transliterated words; for translation, we utilize a bert model and the cosine distance metric to match possible translations. to make the algorithm consistent with the logic of wer, which ensures that the output is in fact a correct transcription, we apply tight cutoff points for cer and cosine distance. while the algorithm is flexible to account for all kinds of spelling variations, we implement it specifically for handling predictable variations that arise in cs transcription. in light of this flexibility to handle multiple languages, we refer to the algorithm as polywer foot_1 . while the proposed polywer algorithm provides the needed flexibility, we lack cs data sets that include these variations in their reference annotations. to enable the application of this method, we manually annotate one of the available cs datasets to provide additional variants. namely, we selected the mixat corpus (al ali and aldarmaki, 2024), which consists of ∼15 hours of speech, roughly half of which includes arabic-english code-switching. the original annotations use the arabic and english scripts, making it easy to identify code-switching points in the text. we hire native speakers to provide transliterated and translated annotations for the cs segments. using this data, we evaluate three large pre-trained asr models that support the arabic language: whisper @xcite , @xcite , and @xcite . as reported in al @xcite , while these models support the arabic language more generally, their performance on this dataset is very poor for two reasons: the speech is in emirati arabic, which is a low-resource dialect, and these models were not trained to transcribe in this variant; the performance degrades even more as a result of code-switching. yet, some observed translated and transliterated outputs, as shown in evaluation metrics for code-switching. most works on code-switching in asr still rely on standard asr metrics like word error rate (wer) and character error rate (cer). while these metrics gt do you go back? lat gt video editing lit, lat csub ← d[i, j] + min(cost, costtr) 27: ## experimental settings to effectively test our algorithm, we select three different asr models and generate predictions on the mixat dataset under various settings. ## asr fine-tuning & evaluation in this section, we report the final performance of the various systems trained on mixat training set (part 1), and evaluated on mixat testing set (part 2) using polywer and other metrics. for each model, we evaluate three variants: ## conclusion in this paper, we introduced polywer, a holistic evaluation framework for code-switching in speech recognition. the algorithm accepts multiple references, including original transcriptions, transliterations, and translations. using special annotations to identify code-switched segments in references, polywer applies suitable metrics for the transliterated (cer) and translated segments to maximize flexibility without compromising its integrity as an error metric. to that end, tight thresholds are applied for accepting transliterations based on cer and translations based on bert cosine similarity. our evaluation against human judgement shows that a variant of polywer correlates well with human scores by balancing faithfulness and meaning preservation. at the same time, it maintains finegrained discriminative ability, unlike automatic machine translation metrics like wer or bertscore that are biased towards strict faithfulness or semantic similarity, respectively. the algorithm is flexible as it incorporates hyperparameters to adjust depending on the desired feature. we find that, using the arabic bert model used in our evaluation, including translation in the evaluation results in inferior performance. manual inspection reveals that the semantic similarity scores between arabic and english in this model are unreliable, so further analysis is needed to demonstrate the potential of including translations with a more reliable crosslingual similarity model. our implementation of polywer, alongside the additional annotations for mixat, are publicly available for research.