On the Robust Approximation of ASR Metrics (2025)

Abstract: Recent advances in speech foundation models are largely driven by scaling both model size and data, enabling them to perform a wide range of tasks, including speech recognition. Traditionally, ASR models are evaluated using metrics like Word Error Rate (WER) and Character Error Rate (CER), which depend on ground truth labels. As a result of limited labeled data from diverse domains and testing conditions, the true generalization capabilities of these models beyond standard benchmarks remain unclear. Moreover, labeling data is both costly and time-consuming. To address this, we propose a novel label-free approach for approximating ASR performance metrics, eliminating the need for ground truth labels. Our method utilizes multimodal embeddings in a unified space for speech and transcription representations, combined with a high-quality proxy model to compute proxy metrics. These features are used to train a regression model to predict key ASR metrics like Word Error Rate (WER) and Character Error Rate (CER). We experiment with over 40 models across 14 datasets representing both standard and in-the-wild testing conditions. Our results show that we approximate the metrics within a single-digit absolute difference across all experimental configurations, outperforming the most recent baseline by more than 50%.

## introduction automatic speech recognition (asr) models have made significant advancements in recent years, achieving near-human performance on several standard evaluation benchmarks @xcite @xcite . these models are typically evaluated using metrics like word error rate (wer) and character error rate (cer) @xcite , which are essential for assessing model performance. however, these metrics are dependent on ground truths, which are often scarce in resourceconstrained environments, and human labeling is both costly and time-consuming. to mitigate this challenge, several reference-free evaluation methods are proposed @xcite @xcite @xcite . while these approaches eliminate the reliance on labeled data, they primarily offer relative assessments of transcription quality, rather than providing precise error counts or rates. as a result, their applicability in real-world scenarios, where actionable performance metrics are crucial for further model refinement and deployment, is limited. given the limitations of both methods, approximating asr metrics has emerged as a promising alternative for label-free evaluation (chowdhury and @xcite @xcite . this approach typically involves training regression @xcite and/or classification models @xcite on top of speech and text encoders. while this method offers a close approximation of error metrics, several important questions remain unresolved. specifically, an approximation model trained on dataset sampled from d to predict asr metrics for a source model m must be evaluated under diverse conditions: 1) on test data that is iid (independent and identically distributed) sampled from d; 2) on out-of-distribution (ood) data representing diverse domains and recording conditions; 3) on iid data, but transcription from a target model t ; and 4) on ood data with transcriptions from a target model t . most prior works (chowdhury and @xcite focus primarily on the first condition. moreover, recent advancements in multimodal foundation models offer new opportunities to directly train regression models on unified speech and text embeddings. to address these critical research gaps, we pro-pose a novel framework for approximating the performance of a wide range of asr models, both on standard benchmarks and in-the-wild scenarios. specifically, we compute the similarity between speech and text embeddings in a unified space, capturing the semantic alignment between the two modalities. additionally, we incorporate a highquality reference model as a proxy, based on the intuition that agreement with a reliable proxy correlates with transcription quality, as shown in prior works @xcite . these features are then used to train a regression model to predict key asr metrics, such as wer, cer, and absolute word and character error counts. in summary, our work represents one of the most comprehensive studies to date on approximating asr metrics at scale, in terms of both data and model coverage. our proposed approach serves as a reference-free evaluation particularly suited for label-scarce scenarios. beyond evaluation, our method is especially valuable for tasks such as pseudo-labeling, where high-quality transcriptions are essential for downstream applications like knowledge distillation @xcite . our contributions are as follows: outline. the remainder of this paper is organized as follows: section 2 reviews related work. section 3 presents our proposed methodology. sections 4 and 5 detail our experimental setup, results, and ablation study, respectively. section 6 concludes the paper and outlines future directions. ## related work automatic speech recognition (asr) has witnessed significant advancements in recent years, primarily due to the scaling of both data and model size @xcite . transformer @xcite based models, in particular, have significantly contributed to these developments by effectively capturing long-range dependencies and contextual nuances in speech, achieving state-of-the-art (sota) performance across diverse benchmarks @xcite @xcite . while traditional evaluation metrics like word error rate (wer) and character error rate (cer) are de-facto evaluation metrics in benchmarking asr systems @xcite , scenarios where ground truth transcriptions are unavailable have caught interest in reference-free asr evaluation methods @xcite @xcite . reference-free asr evaluation methods aim to estimate asr performance without requiring ground truth transcriptions @xcite . earlier approaches rely on heuristic features or metadata such as speaker demographics, background noise, and linguistic characteristics @xcite , limiting their applicability across varied contexts. however, recent advancements focus on deep learning-based frameworks, such as convolutional neural networks (cnns) @xcite and contrastive learning methods @xcite , to predict asr quality directly from encoded speech and text. for instance, methods like norefer @xcite use siamese architectures fine-tuned on asr hypotheses, achieving high correlation with traditional metrics and improving wer by optimizing hypothesis ensembling @xcite . efforts to approximate asr metrics explore hybrid approaches that combine traditional and reference-free methods, such as leveraging word confidence scores, linguistic embeddings, or postprocessing adaptations to estimate wer and cer without explicit references @xcite @xcite . however, these approaches often suffer from reliance on specific asr models or domain characteristics, limiting their generalizability. unlike existing methods, our work addresses these limitations by introducing a robust, model and data-agnostic frame-work that evaluates asr outputs across diverse datasets and configurations, emphasizing adaptability to unseen domains and variations. ## methodology we present a scalable and robust method to approximate asr performance metrics using multimodal unified embeddings, proxy references, and regression models. the primary goal is to eliminate reliance on ground-truth labels, enabling performance evaluation in label-scarce scenarios. the pipeline consists of three components: representation similarity in a unified speech-text embedding space, agreement with a high-quality proxy reference, and a regression model trained on these features to predict asr metrics. our pipeline diagram is shown in figure 1 . ## experiments in this section, we present the experimental setup to evaluate our asr metrics approximation tool. we describe the datasets, models, and regression pipeline used in our experiments, highlighting the diversity of asr systems and testing conditions. ## results we conduct experiments using two dataset categories: standard benchmarks and in-the-wild, as described in section 4.1. for each asr model, a leave-one-out strategy is used, training the regression model on 9 benchmark datasets and testing it on the remaining benchmark dataset and all four in-the-wild datasets to ensure comprehensive evaluation on out-of-domain data. additionally, in-domain testing is included in ablation studies, as detailed in section 5.4. the regression model is trained to predict absolute error counts (word and character levels), which are normalized by the reference length to compute approximate error rates (aw er and acer). we also train regression models to directly predict wer and cer. we provide results for fine-grained metrics in ## conclusion we present a framework for approximating asr metrics, demonstrating its effectiveness in generalizing to unseen, in-the-wild, and challenging conditions. our results show that the model performs well with absolute error counts, consistently outperforming strong baseline, with error rates remaining relatively low. we show that our proposed method achieves consistent performance across 40 asr models and 14 evaluation setups, including both standard benchmarks and domain-specific conditions. the trained regression model can be efficiently used to approximate asr metrics, particularly in data-constrained environments, such as critical domains with limited labeled data. in summary, our work bridges the gap between theoretical advancements and real-world applications, paving the way for more reliable and scalable asr systems. while in this work, we evaluate monolingual and cross-lingual generalization, future work will focus on extending this framework to support a multilingual setting and exploring language-agnostic asr metric approximation.