Retrieval Augmented Generation based context discovery for ASR (2025)

Abstract: This work investigates retrieval augmented generation as an efficient strategy for automatic context discovery in context-aware Automatic Speech Recognition (ASR) system, in order to improve transcription accuracy in the presence of rare or out-of-vocabulary terms. However, identifying the right context automatically remains an open challenge. This work proposes an efficient embedding-based retrieval approach for automatic context discovery in ASR. To contextualize its effectiveness, two alternatives based on large language models (LLMs) are also evaluated: (1) large language model (LLM)-based context generation via prompting, and (2) post-recognition transcript correction using LLMs. Experiments on the TED-LIUMv3, Earnings21 and SPGISpeech demonstrate that the proposed approach reduces WER by up to 17% (percentage difference) relative to using no-context, while the oracle context results in a reduction of up to 24.1%.

## introduction automatic speech recognition (asr) systems have gained considerable development over the last few years and provide high transcription accuracy over an extensive spectrum of tasks and benchmarks @xcite @xcite . even though asr systems have grown tremendously, asr systems' performance continues to degrade under conditions of low-occurrence or out-of-vocabulary words like names, specialized domain names, and userpersonal references @xcite . to address this, several strategies have emerged, including asr personalization @xcite ), text injection @xcite , and contextual biasing (cb) @xcite . among these, cb has received significant attention due to its effectiveness in improving recognition of rare and user-specific terms. one of the core challenges of contextual asr is the definition and extraction of context. the success of biasing depends not only on model architecture but also on how well contextual material aligns to audio semantics @xcite . several works have explored cb by tightly integrating with the asr model through adapters or attention mechanisms over entity catalogs. @xcite propose a retrieval-based personalization method, where asr encoder representations query an infrequent entity catalog to retrieve phonetically similar candidates via a contextual adapter. tong et al. @xcite extend this by using slot-specific catalogs and combining entity embeddings with asr outputs through a cross-attention mechanism. these methods generally rely on fine-tuning the asr backbone and assume access to structured entity catalogs, often annotated by domain/slot type. recent research has explored large language models (llms) for contextual asr tasks. xiao et al. @xcite build a contextual knowledge base from custom vocabularies and documents, using a first-pass asr output to prompt an llm for error correction. sun et al. @xcite fine-tune an llm for token prediction and entity classification, applying it to second-pass hypothesis rescoring. these approaches, while effective, typically involve fine-tuning and curated entity prompts, which can limit adaptability across domains or resource-constrained settings. finally, mathur et al. @xcite propose doc-rag, a domain-sensitive framework that builds domain-specific corpora and uses cooccurrence matrices to estimate next-word probabilities for asr rescoring. retrieval-based approaches such as those in @xcite also leverage similarity search mechanisms, but their retrieval is conditioned on either learned audio embeddings or manually annotated slot labels, making them audio input llm-driven context generation transcript refinement via llm transcript y t s 1 s t .. .. context words c t or s 2 s3 s t y t-1 y t-2 y t-k ... y t-1 or s t y 1 ... query q t sentence transformer embeddingbased retrieval y t-1 previous transcripts concat asr model vad c t context retrieval contextual asr word vocabulary sentence transformer dependent on supervised signals and asr model internals. the majority of existing methods assume structured catalogs @xcite , require architectural modifications to asr models @xcite , or depend on domain-specific supervision @xcite , limiting their flexibility in general or black-box settings. inspired by prior work, this study proposes a lightweight, modular approach that avoids such constraints. specifically, it proposes an embedding-based retrieval strategy using pre-trained minilm embeddings over a large vocabulary, entirely decoupled from the asr model architecture. this method is evaluated alongside two llm-based alternatives: a prompt-based context generator and a post-asr transcript corrector. earlier context-biasing techniques are deeply integrated into the asr model itself -requiring custom layers, retraining, or access to model internals -so can't be run on the black-box recognizer used in this work. consequently, we benchmark only "plug-and-play" methods in a common pipeline that mirrors real-world deployment scenarios (identical audio inputs, metrics, and latency limits). this paper has three main contributions: the rest of the paper is organized as follows: section 2 presents the proposed methodology, sections 3 and 4 the experimental setup and results, and finally the paper concludes in section 5. ## methods an overview of the proposed approach is presented in figure 1 , and explained in the next subsections. ## results although llm-based approaches achieve the best relative wer improvement in spgispeech (16.96%), cb-rag configurations closely follow (16.52%). as shown in the results indicate that the proposed cb-rag approach is a competitive and effective method for automatic context construction without the use of user-specific historical text data. it is also a better alternative compared to llm-based context creation and transcript correction. although cb-rag has lower contextual overlap scores, the wer is better with significantly lower latency, depicting its utility in actual scenarios. its flexibility in selecting word context size extraction by each query (c) and number of segments in the past to consider (k) enables flexible application across domains with varied requirements for latency and performance. these findings suggest that cb-rag would be particularly appropriate for resource-constrained or real-time applications for which llm-based alternatives would be computationally infeasible. ## conclusion this paper explored different approaches to incorporating contextual information into asr pipelines, with an emphasis on automatic context extraction. among the approaches examined -embeddingbased retrieval, context generated by llm, and post-asr correction -the proposed cb-rag framework achieved the best overall performance, yielding the lowest wer (up to a 17% relative reduction) in test sets at the lowest computational cost and latency. despite a substantially lower context overlap, up to 47.3% absolute difference, cb-rag retrieved significantly higher number of contextual tokens, while operating, on average 83.5%, lower latency than llm-based alternatives. post-asr correction improved slightly but still remained less than that of cb-rag. these findings highlight cb-rag as the most scalable and adaptable solution, combining high accuracy with efficiency.