Universal- WER : Enhancing WER with Segmentation and Weighted Substitution for Varied Linguistic Contexts (2024)

Abstract: Word Error Rate (WER) is a crucial metric for evaluating the performance of automatic speech recognition (ASR) systems. However, its traditional calculation, based on Levenshtein distance, does not account for lexical similarity between words and treats each substitution in a binary manner, while also ignoring segmentation errors. This paper proposes an improvement to WER by introducing a weighted substitution method, based on lexical similarity measures, and incorporating splitting and merging operations to better handle segmentation errors. Unlike other WER variants, our approach is easily integrable and generalizable to various languages, providing a more nuanced and accurate evaluation of ASR transcriptions, particularly for morphologically complex or low-resource languages.

## introduction automatic speech recognition (asr) is now ubiquitous in our daily lives, facilitating translation, video transcription, note-taking, and interactions with voice assistants. while advances in deep learning models have significantly improved asr system accuracy, challenges remain, particularly for underrepresented and morphologically complex languages. despite these advancements, evaluating the performance of asr systems remains essential to ensure their accuracy and reliability. word error rate (wer) is still the benchmark metric for evaluating transcription quality, but it relies on the levenshtein distance, which does not account for lexical imprecision or segmentation errors, limiting its relevance in the face of linguistic diversity. low-resource languages often exhibit complex morphological structures @xcite , making them particularly vulnerable to segmentation errors, especially in the case of agglutinative languages. wer applies a double penalty to these errors, artificially inflating the error rate. furthermore, these languages are often characterized by high dialectal diversity, leading to inappropriate penalties for variations that are not inherently errors. lastly, wer treats all lexical substitutions in a binary manner, overlooking minor variations that could be considered acceptable. these limitations highlight the need for a more precise evaluation, better suited to linguistic diversity. two main approaches stand out to improve wer calculation: on the one hand, models incorporating weightings based on word meaning, such as the weighted word error rate (wwer) @xcite , and on the other hand, methods like the phoneme error rate (per) @xcite , which assess recognition at the phonemic level. recently, evaluation methods based on language models have also emerged. however, these solutions have limitations, particularly in terms of implementation complexity and generalization to all languages. our work proposes an improved version of wer, tailored to the specificities of asr transcriptions, by introducing weighted substitution based on lexical similarity measures, as well as splitting and merging operations to better handle segmentation errors. this approach aims to ensure adaptability to various languages and different usage contexts. ## introducing wer and the levenshtein distance the word error rate (wer), the main metric used to evaluate the performance of asr systems, cal-culates an error rate: he lower the rate (with a minimum of 0), the better the recognition. the maximum rate is unbounded and can exceed 1 (wikipédia, 2023). the wer formula is given by: w @xmath0, an algorithm that measures the similarity between two sequences by counting the minimum number of operations required to transform one sequence into another @xcite . the algorithm recognizes three operations: the algorithm works by constructing a matrix where each cell represents the alignment cost (by insertion, deletion, or substitution) of a segment from the input sequence (transcription) with a segment from the target sequence (reference text). the cost calculation is performed iteratively, comparing the elements of the two sequences. ## challenges of wer based on levenshtein distance this cost calculation method is effectively used in the character error rate rate (cer), where the comparison units are individual characters. in this context, each character is compared to another, and the substitution decision is naturally binary: either the units are identical (cost of 0), or they differ (cost of 1). however, the wer, which compares entire words, has significant limitations, as highlighted by @xcite . due to its binary approach, the wer mainly compares orthographic forms rather than the words themselves, which penalizes minor variations, such as "advisor" and "adviser." these orthographic variations also include space insertions, as in "doghouse" and "dog house," which are double-counted in wer calculation. this type of situation is treated as a segmentation error. these limitations affect all languages, but they are particularly pronounced in languages with complex morphology, minority languages, and those with limited resources. ## conclusion and future directions this study has highlighted the limitations of wer, especially its inability to account for lexical nuances and segmentation errors, making it unsuitable for morphologically complex or low-resource languages. we proposed an improved version of wer, which introduces weighted substitution based on lexical similarity, as well as splitting and merging operations. experimental results show that uwer improves evaluation accuracy across several languages. our approach aims to ensure wer's adaptability to the vast linguistic diversity while providing a simple-to-implement solution a simple-toimplement solution, fully interchangeable with wer, without requiring changes to current practices. by increasing the precision of this metric, we provide a more rigorous evaluation tool capable of revealing the true performance of models, especially for morphologically complex and low-resource languages. furthermore, this approach can also be leveraged as a loss function to optimize asr model training. although wer is not differentiable, adaptations such as differentiable approximation, reinforcement learning, or optimization via minimum bayes risk (mbr) can be considered to overcome this limitation.