Deferred NAM : Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASR (2024)

Abstract: Contextual biasing enables speech recognizers to transcribe important phrases in the speaker’s context, such as contact names, even if they are rare in, or absent from, the training data. Attention-based biasing is a leading approach which allows for full end-to-end cotraining of the recognizer and biasing system and requires no separate inference-time components. Such biasers typically consist of a context encoder; followed by a context filter which narrows down the context to apply, improving per-step inference time; and, finally, context application via cross attention. Though much work has gone into optimizing per-frame performance, the context encoder is at least as important: recognition cannot begin before context encoding ends. Here, we show the lightweight phrase selection pass can be moved before context encoding, resulting in a speedup of up to 16.1 times and enabling biasing to scale to 20K phrases with a maximum pre-decoding delay under 33ms. With the addition of phrase- and wordpiece-level cross-entropy losses, our technique also achieves up to a 37.5% relative WER reduction over the baseline without the losses and lightweight phrase selection pass.

## introduction automatic speech recognition (asr) applications often succeed or fail based on their ability to recognize words that are relevant in context, but may not be common, or even present, in the training data. for example, an assistant user may speak contact names from another language or titles of media entities which were released after the asr system was trained, or the speaker may use domain-specific jargon, like legalese or medical terms, which are not common in the more-typical speech used for training. asr contextual biasing @xcite aims to account for this domain shift between training and inference. in which the context is encoded into dense embeddings attended to during recognition, is one of the leading approaches for contextualizing end-toend (e2e) asr systems. as a fully-end-to-end method, it does not require separate biasing components which must be separately trained and whose integration with the core asr system must be optimized. however, recognition cannot begin until the inference-time context has been encoded, and this context may consist of tens of thousands of items which may not be cached, for privacy or system design reasons or, more simply, because the context may change at any point up until the beginning of recognition. thus, the context encoder must be able to efficiently handle very large contexts, as delays in context encoding translate directly into user-visible delays in asr transcription. in this work, we optimize context encoding by splitting it into two passes. we make the simplifying assumption that our asr system is non-causal and can access the entire audio input; we argue that this choice is not overly restrictive, as two-pass asr systems combining a causal asr system to produce streaming results with a non-causal asr system for producing the final result have become common, e.g., as in @xcite . with the speaker's entire audio available before context encoding begins, we can split context encoding into two phases. first, we encode all contextual phrases with an extremely lightweight encoder, and use the resulting encodings to determine the k phrases that are most likely to occur in the audio. we embed only the k most-likely phrases with a more powerful -but more expensive -encoder, and use the output of this "deferred" encoder for biasing. ## related work conventional asr contextualization relies on discrete contextualization components like (classbased) language models (lms) @xcite , combined with the base asr system through on-the-fly lm rescoring @xcite @xcite , shallow fusion @xcite , or lattice rewriting @xcite and contextual spelling correctors @xcite . the use of discrete contextualization components requires the implementor of an asr system to separately train the asr and contextualization components, to separately optimize their combination, and to take care that all relevant signals, like the input audio, are forwarded from asr to the contextualization system. attention-based biasing @xcite @xcite , in which the asr network learns to use inferencetime context through attention, and thus requires no separate contextualization components and can be optimized end-to-end by standard backpropagation, has become a popular method for contextualization of end-to-end asr models. the core technique has spawned several distinct lines of complementary research. there are threads of work on improving data efficiency by adapter-style training @xcite or improving training on synthetic audio derived from text-only data @xcite . work on precision improvements seeks to lower the rate of over-biasing through hierarchical or gated attention @xcite @xcite @xcite or through slot triggering @xcite ; often, such techniques can improve not only quality but also per-step inference run time @xcite . notably, tong et al @xcite improve wer through an auxiliary slot-level cross-entropy loss. we do not use slots for selecting context to bias, as we have found it possible to get high performance with hierarchical attention alone; however, in this work, we do apply a phrase-level, rather than slot-level, crossentropy loss during training to improve wer. further we extend the technique to a novel wordpiecelevel cross-entropy loss. further work aims to improve quality on biased utterances by providing the context encoder with more information than just the graphemic biasing phrases, including phonemelevel features @xcite @xcite , and augmenting the context encoder with semantics-aware embeddings from a pretrained bert model @xcite . attention-based biasing can also be complemented by shallow fusion with (contextualized) language models @xcite or combinations of language models and contextualized rescoring @xcite for further quality improvements. we distinguish the current work from most of the above by focusing on the inference run time of the context encoder which, as noted in section 1, is critical to the usability of contextualized asr. our work builds on dual-mode nam (neural associative memory) @xcite and its shared query encoder variant from section 3.4.2 of @xcite , which we use as a baseline. dual-mode nam computes both phrase encodings e p and wordpiece (wp) encodings e w through a fine-grained context encoder, where e p corresponds to the cls embedding and e w corresponds to the bias phrase wp embeddings of z; @xmath0; w n,1 , ..., w n,l } n @xmath1, n is the number of bias phrases associated with the utterance and l is the number of wps per bias phrase. ## conclusion we proposed a low-latency attention-based contextual asr system, augmented with phrase-and wplevel cross-entropy losses, which can handle thousands of bias phrases within milliseconds while achieving up to 37.5% relative average wer reduction. this demonstrates the potential for enhancing asr in real-world applications requiring fast and accurate contextual speech recognition.