Match ‘em: Multi-Tiered Alignment for Error Analysis in ASR (2025)

Abstract: We introduce “Match ‘em”: a new framework for aligning output from automatic speech recognition (ASR) with reference transcriptions. This allows a more detailed analysis of errors produced by end-to-end ASR systems compared to word error rate (WER). Match ‘em performs the alignment on both the word and character level; each relying on information from the other to provide the most meaningful global alignment. At the character level, we define a speech production motivated character similarity metric. At the word level, we rely on character similarities to define word similarity and, additionally, we reconcile compounding (insertion or deletion of spaces). We evaluated Match ‘em on transcripts of three European languages produced by wav2vec2 and Whisper. We show that Match ‘em results in more similar word substitution pairs and that compound reconciling can capture a broad range of spacing errors. We believe Match ‘em to be a valuable tool for ASR error analysis across many languages.

## introduction metrics like word error rate (wer) provide a simple, automated way of understanding how well an automatic speech recognition (asr) system is performing. however, this simplicity fails to capture the nuance regarding the severity of transcription errors, both in terms of spellings and semantics. efforts have been made to improve wer. these include adding new metrics around information lost by mistranscriptions @xcite and weighting kewyords more heavily in wer @xcite . attempts to optimize the alignment between transcriptions have utilized articulatory features @xcite as well as semantic distances @xcite . additionally, new metrics such as semdist @xcite and aligned semantic distance @xcite have been developed to utilize the embedding vector space, instead of aligning the words themselves, to calculate the severity of errors. however, as all these metrics only aim to summarize the quality or utility of an asr output, they do not provide details on the types or severity of commonly made errors. understanding the types of errors that asr systems make has been of interest for many years. the goals are both understanding how wrong a transcription really is, as well as identifying specific areas for improvement. in @xcite , the authors create individual word error rate to determine which words are frequently missed and which factors account for misrecognitions. in @xcite , the authors compare the ability of humans and automatic transcriptions to disambiguate homophonic or nearhomophonic words that are frequently missed by asr. words that are frequently missed in conversational speech for dutch, english, and german are analyzed in @xcite . the authors in @xcite ; @xcite manually classify asr errors for both their severity and type to understand how asr is performing on german and latvian speech, respectively. despite the benefits of metrics and error analysis, there are several factors that can be limiting to these tools. for semantic metrics, knowledge of the language (semantic embeddings, word importance) is crucial. however, access to such resources is not readily available for certain languages. similarly, analysis of asr errors is often reliant on manual efforts to label the errors made, thus limited by the amount of human hours in this paper, we propose a new framework, that we call match 'em, for aligning asr generated and reference text that operates both at the word and character level. the goal is to provide a better match between words and characters thus allowing for a detailed analysis of the common mistakes produced by asr systems. additionally, this method opens the possibility to use foundation asr models trained on massive amounts of data to study phenomena related to variability in speech production by analyzing the asr errors in detail; such phenomena include dialectal variation or pronunciation variation in second language learners or in speakers with speech sound disorder. the contributions of the paper can be summarized as: ## the match 'em framework the standard levenshtein alignment considers three edit operations (insertion, deletion, and substitution) when transforming the hypothesis text into the reference text @xcite . the edit costs (that is, the penalty for any of the three edits) are also fixed before alignment occurs. this method is traditionally used to separately compute either word error rate (wer) at the word level or character error rate (cer) at the character level. the match 'em framework we propose operates both at the word and the character level simultaneously. the alignment at each level is influenced by information coming from the other, as illustrated by figure 1 . in the figure, we can see that words that are spelled similarly are aligned, the characters within the words are aligned, and the breaking up of a compound word is accounted for. details on how each of these components was achieved follows in the subsections below. ## experiments to evaluate the impact of this new alignment method, audio in three different european languages was transcribed using two state-of-the-art asr model architectures. the three languages (norwegian, italian, and english) were chosen for a variety of reasons. firstly, match 'em re-quires languages with alphabets for which character articulatory vectors can be defined-thus excluding languages that use syllabaries or logographies, such as japanese or chinese, respectively. also, these three languages cover multiple language families (germanic and romance), orthographic depths (norwegian and italian spellings being largely phonetically written as opposed to english being irregular @xcite ), and dialectal variations (both norwegian and italian contain a large amount of dialectal variation compared to english @xcite ). additionally, norwegian bokmål allows for multiple legal spellings of words (e.g., vet and veit both being legal spelling for the present tense of å vite ("to know"). lastly, norwegian utilizes compounding of words (again with common, but perhaps less legal, variations to spellings) to a higher degree than english or italian, which gives us an opportunity to test how match 'em performs on this aspect. ## conclusions we propose the new match 'em framework for creating better alignment between reference and asr-generated transcriptions both at the word and character level. we show that match 'em allows for a deeper understanding of asr performance compared to wer, by supporting detailed analysis of common errors. by using word dissimilarity metrics and by reconciling compound errors, match 'em alignment results in word substitution pairs that are more similar compared to standard levenshtein alignment. we show that analysis of these substitution pairs can yield insights into the potential semantic impacts of these errors. our claims are verified across three european languages (english, italian and norwegian) and two state-of-the-art asr architectures (wav2vec 2.0 and whisper). we believe the match 'em framework to be a useful tool for other asr researchers for gaining insights into their own models' performances and, more generally, for speech researchers to gain linguistic insights by analyzing asr errors on large annotated speech corpora.