RECAST : Retrieval-Augmented Contextual ASR via Decoder-State Keyword Spotting (2025)

Abstract: Contextual biasing in ASR systems is critical for recognizing rare, domain-specific terms but becomes impractical with large keyword dictionaries due to prompt size and latency constraints. We present RECAST–a lightweight retrieval-augmented approach that repurposes decoder states of a pretrained ASR model to retrieve relevant keywords without requiring audio exemplars. RECAST introduces a contrastively trained retriever that aligns decoder-state embeddings with textual keyword representations, enabling fast token-level retrieval over large dictionaries. Retrieved keywords are ranked and formatted into a prompt to guide a downstream speech language model. Trained solely on LibriSpeech and evaluated on out-of-domain benchmarks covering up to 4,000 keywords across diverse domains, RECAST consistently outperforms full-list prompt biasing and strong phonetic/text baselines. It achieves up to 54.3% relative reduction in entity WER and 41.3% overall WER improvement over the baseline, along with up to 2.5x higher recall in challenging settings. Furthermore, RECAST remains effective for diverse languages such as Hindi, demonstrating its scalability, language-agnostic design, and practicality for real-world contextual ASR.

## introduction contextual biasing in asr via domain-specific keywords improves recognition of rare terms but fails to scale: large keyword inventories degrade transcription quality and exceed prompt-size constraints @xcite @xcite . while recent advances in speech language models @xcite @xcite and llm-based error correction @xcite enable effective keyword prompting @xcite , domains like medicine or finance involve thousands of rare entities, making full-list inclusion infeasible. this raises a central challenge: how can we efficiently identify a small, relevant subset of keywords for any given utterance? unlike prior contextual biasing methods that struggle to scale with large keyword dictionaries, such as attention based selection over the full list @xcite , or keyword spotting approaches that require audio exemplars and perform independent searches for each keyword @xcite , we propose a unified and scalable solution: re-cast (retrieval-augmented contextual asr via decoder-state keyword spotting). recast leverages the final decoder state of a pretrained encoder-decoder asr model as a query to retrieve relevant keywords directly from a large text-only inventory, without modifying the asr backbone or relying on audio examples. beyond the challenge of large keyword lists, realworld utterances are often long, with target entities appearing anywhere in the audio. this makes it crucial not only to retrieve relevant keywords but also to localize their aligned positions. to this end, recast introduces two lightweight modules: a unidirectional lstm-based keyword encoder and a feed-forward decoder-state projector, trained with a contrastive loss to align subword-level decoder states with keyword token embeddings in a shared retrieval space. at inference time, each decoder state is treated as a fine-grained retrieval query into a precomputed keyword index. retrieved tokenlevel matches are aggregated into candidate keyword spans and ranked using a position-aware scoring function to form a top-k shortlist. this shortlist is then incorporated into the decoder prompt of a downstream speech language model, enabling fluent and grounded contextual biasing without modifying the asr model. we validate recast by training the retriever solely on librispeech and evaluating it on diverse out-of-domain benchmarks with up to 4,000 keywords across domains such as locations, names, and medical terms. despite no exposure to these entities during training, recast consistently improves retrieval and asr performance. it outperforms strong retrieval-style baselines-common in information retrieval but underexplored in asr-which rely on asr hypotheses and apply fuzzy matching (e.g., bm25) or phonetic algorithms like soundex @xcite and double metaphone @xcite . in contrast, recast operates directly over decoder states and text keywords, enabling seamless generalization to languages such as hindi, where phonetic methods tailored for english like soundex and double metaphone, are ineffective. it integrates with existing speech language models, offering low-latency inference and practical gains for real-world contextual asr. our contributions are threefold: (1) a contrastively trained retriever that aligns asr decoder states with keyword token embeddings for efficient large-vocabulary retrieval; (2) a token-level tracking and ranking algorithm that builds contextually relevant shortlists in real time; and (3) robust generalization across domains and languages, improving accuracy and latency without modifying the underlying asr backbone. however, most methods still process keywords independently, leading to linear scaling in computation and latency with large dictionaries. ## recast we formalize the retrieval problem as follows. let @xmath0, . . . , ỹn } denote a large dictionary of keywords, where n is large. given an audio utterance x to be transcribed, the goal is to efficiently retrieve a small subset of at most k keywords from d that are likely to appear somewhere in x. the key insight behind recast is that the decoder of a pretrained encoder-decoder asr model uses cross-attention to focus on relevant segments of the input audio when generating each text token. these decoder states implicitly encode fine-grained audio-text alignment. recast leverages this property by introducing a lightweight extension to the decoder that repurposes these intermediate states to retrieve relevant keywords from d. the architecture and overall workflow of this retrievalaugmented framework are illustrated in figure 1 . overview of recast the two new trainable components we introduce are: a keyword-encoder ke ϕ and a decoder-state projector dp ϕ with parameters ϕ. the keyword-encoder ke ϕ converts any keyword k to a sequence of vectors s(k) = z 1 , . . . , z l k where l k denotes the number of tokens into which the asr tokenizer would decompose k. the decoder-state projector dp ϕ converts at each decoder step t, the last hidden vector of the decoder d t into an audio snippet embedding a t . more details about estimating ϕ are in section 3.1. during domain-specific deployment, first given any arbitrary list of keywords d in that domain, we first input each keyword @xmath1. this vector sequence is inserted into a vector search data struc-ture i. more details about constructing such an index appear in section 3.2. once the index is created, for each subsequent transcription task on an audio x, we retrieve a short list of at most k keyword matches from d. this retrieval is performed auto-regressively where at each step t, the decoder state is used as search key to probe into the i for potential matches. the matches across multiple consecutive decoding steps are stitched together to obtain the short list k. more details of this retrieval process appears in section 3.3. finally, the retrieved shortlist is used for contextual biasing the transcription of x into the text as described in section 3.4. ## experimental setup models and implementation details. we use whisper large-v2 (1.5b parameters) as the frozen asr backbone. our retrieval model consists of two lightweight modules: a single-layer lstm keyword encoder ke ϕ , which projects 1024-dimensional token embeddings into a 512dimensional space, and a feedforward decoderstate projector dp ϕ , which maps 384-dimensional decoder states into the same embedding space. combined, these modules introduce only 6.5m additional parameters. keyword embeddings are precomputed and indexed to enable efficient knnbased retrieval during inference. training and implementation details are provided in appendix a. 1 metrics. we evaluate recast primarily using recall-based metrics to assess keyword retrieval quality from large candidate pools. recall@50 measures retrieval effectiveness, while keyword recovery rate (krr) captures the proportion of keywords missed by the baseline asr but recovered through retrieval, highlighting gains in contextual recall. to assess downstream asr impact, we also report word error rate (wer) and entity-wer (e-wer), the latter computed over dictionary entity spans in test utterances 1 code: https://github.com/ashishmittal/recast retrieval baselines. for these baselines, we use the 1-best asr hypothesis and perform keyword retrieval via fuzzy text matching. while many such methods exist, we focus on scalable, index-based approaches supported by elastic search (and its phonetic analysis plugin) foot_0 . the five baselines are: (1) soundex. matches exact soundex @xcite codes between transcript and keywords. ## experiments and results as shown in these gains, summarized in effective with smaller asr models, balancing quality and efficiency. all wer gains are statistically significant under the mapsswe test (@xmath3.01). while wfst rescoring yields moderate improvements over the baseline, it remains significantly behind recast in both wer and e-wer. its effectiveness is limited by the lack of diversity in beam search outputs, where n-best lists often contain only minor variations of the same prediction. prism improves over wfst by injecting tts-synthesized exemplars, but scalability is a concern since synthesizing and indexing thousands of entities is costly, and decoding can be misbiased when exemplars are acoustically mismatched. thus, while contextual baselines provide useful biasing signals, they remain constrained by limited hypothesis diversity or reliance on audio exemplars, explaining their gap with recast 's retrieval-augmented framework. as shown in ## conclusion we introduced recast, a retrieval-augmented framework for contextual asr that leverages decoder states of a pretrained encoder-decoder model to query large text-only keyword dictionaries without audio exemplars. at the core of recast is a contrastively trained retriever and a token-level span aggregation algorithm that constructs and ranks keyword hypotheses using contextual similarity, length-based scaling, and positional diversity. recast achieves state-of-the-art results on in-domain and out-of-domain benchmarks for both retrieval and contextual asr, with substantial gains in recognition quality. it maintains low latency despite large vocabularies, owing to its lightweight design and efficient knn retrieval. these results position recast as a scalable, accurate, and practical solution for keyword-guided speech recognition even in languages where traditional phonetic baselines like soundex and double metaphone are ineffective. while the current setup is languagespecific, future work may explore multilingual extensions with a shared retriever.