Speech-enriched Memory for Inference-time Adaptation of ASR Models to Word Dictionaries (2023)

Abstract: Despite the impressive performance of ASR models on mainstream benchmarks, their performance on rare words is unsatisfactory. In enterprise settings, often a focused list of entities (such as locations, names, etc) are available which can be used to adapt the model to the terminology of specific domains. In this paper, we present a novel inference algorithm that improves the prediction of state-of-the-art ASR models using nearest-neighbor-based matching on an inference-time word list. We consider both the Transducer architecture that is useful in the streaming setting, and state-of-the-art encoder-decoder models such as Whisper. In our approach, a list of rare entities is indexed in a memory by synthesizing speech for each entry, and then storing the internal acoustic and language model states obtained from the best possible alignment on the ASR model. The memory is organized as a trie which we harness to perform a stateful lookup during inference. A key property of our extension is that we prevent spurious matches by restricting to only word-level matches. In our experiments on publicly available datasets and private benchmarks, we show that our method is effective in significantly improving rare word recognition.

## introduction rapid advancements in automatic speech recognition (asr) technologies have broadened their application in real-world settings. however, state-ofthe-art asr systems often fail to recognize named entities or critical rare words. for example, a chatbot we deployed in the medical domain backed by the state-of-art whisper asr model, consistently misrecognized in patient to impatient. such situations call for contextual biasing whereby the transcription from asr model is biased by a userprovided list of proper nouns (e.g., names from contact lists) or technical terms (e.g., drug names from medical dictionaries). the accurate transcription of such tokens could have a disproportionately high impact on the asr output's value. e.g., a spoken language understanding (slu) module that uses asr outputs might carry out the desired action as long as named entities in the spoken utterance are correctly transcribed. existing contextual biasing techniques typically demand additional parameterization, either via attention layers jointly trained with the asr model @xcite or via dictionary modules trained with the asr model frozen @xcite . inference-time adaptation to dictionaries has primarily been via the shallow fusion of an external lm @xcite . however, such a late fusion overly biases the model to the external lm and incurs high decoding latency especially for streaming asr. in our work, we address this gap and propose a new lightweight inference-time technique prism (short for "adaptation of asr at inference-time using speech-enriched memory") to adapt to predefined dictionaries at test-time with no additional training. prism is employed during beam-search decoding and used to adapt two state-of-the-art asr architectures, encoderdecoder models @xcite and transducers @xcite models. first, we use a text-to-speech (tts) system to synthesize speech for each dictionary entry. to the best of our knowledge, we are the first to use synthetic speech for contextual biasing and find that it facilitates improved matches compared to the use of text alone. utilizing both speech and text in each dictionary entry, we create a memory mapping speech and text representations (keys) to textual tokens (values). for encoder-decoder models, the decoder states are a natural choice for the memory keys. however, for transducer models, implementing memory keys requires more deliberation and careful handling of the alignment of acoustic and textual context (elaborated in §3.1). during inference, we perform a k-nearestneighbor (knn) search over the memory to retrieve the k most similar dictionary entries. we define a knn probability distribution over the next token that assigns higher probabilities to nearby memory entries and finally bias the model predictions via a linear interpolation with the knn probability distribution. the memory-based matching is implemented using an efficient trie-based search. notably different from prior work @xcite , we avoid triggering spurious matches in the memory after only a prefix match, by maintaining a backoff beam score that keeps track of the score assigned by the model to a suffix if it were not found in the trie. we show that such accounting is essential to avoid worsening baseline performance in the presence of a dictionary. prism can be used with any existing pretrained encoder-decoder or transducer asr model without any retraining and incurs close to no delays during inference. using prism, we obtain significant performance improvements with both transducer and encoder-decoder asr over diverse dictionaries, with up to 36% relative reductions in entity-specific error rates on the well-known librispeech asr benchmark. ## related work and background prior work on contextual biasing of asrs can be organized into three categories: dictionary as external language model. in the first category, the dictionary is viewed as an external lm. this is integrated with the asr's output either via shallow fusion where asr scores are combined with scores from an external lm during beam-search @xcite @xcite or deep fusion where the model has access to an external lm during training @xcite . while shallow fusion is a popular scheme that allows for contextual information to be combined with any trained end-to-end asr, it is also very sensitive to the weight assigned to the external lm and often helps asr of contextual entities at the expense of overall asr performance @xcite . training-time adaptation. the second category consists of approaches that modify the asr model during training to incorporate contextual information, often relying on attention-based mechanisms @xcite @xcite @xcite @xcite . such a direct integration of contextual information is usually more accurate than shallow fusion, but it comes with the added overhead of retraining the asr model for every new dictionary to be integrated. inference-time adaptation using dictionary. the last category of approaches make use of a dictionary during decoding @xcite @xcite . prism falls in the last category with two key differences from prior work. we also synthesize speech, using tts, for every dictionary entry, unlike prior work that only uses text from biasing lists/dictionaries. this facilitates improved matches to the dictionary terms at test time. we enforce word-level matches using a trie and additional beam state to avoid spurious prefix matches that could otherwise seriously distort baseline performance. ## contextual biasing in prism we are given a trained asr model m and a list of isolated word phrases @xmath0, . . . , ỹk } at inference time. prism uses d to first create a memory structure n . subsequently, for each subsequent input speech x, we perform a memory augmented inference to bias its text to the words in d. as stated earlier, the challenge is that recognition of words not in d should not worsen in the process. in order to not adversely impact the streaming and latency performance of m, prism integrates with existing asr decoding algorithms at the tokenlevel, by just modifying the token distribution at each step of decoding. an efficient mechanism proposed in the nlp literature for sequence to sequence models is knearest neighbor (knn) language models @xcite @xcite . a key insight of knn lms is that similar decoder contexts lead to similar next token distribution. this motivates the creation of memory in the form of a list of key-value pairs with context as key and next-token as value. however, applying this principle to asr models raises several challenges. first, we are given as input only a list of isolated word phrases, and context is missing both in terms of the missing audio input, and an overall sentence context in which the phrase lies. we partly offset for the lack of audio input by resorting to tts to generate synthetic audio for each phrase. a second challenge is that for transducer models, the design and extraction of context to form the memory keys is significantly harder, since they do not follow an encoder-decoder architecture. finally, a flat memory in the unit of tokens causes several spurious matches with pieces of a whole phrase during decoding. for example, a dictionary entry for "intuit" could spurious matches to the "in" or "it" tokens of this word. we therefore restrict to allow only whole phrase matches. to implement such whole phrase matches over multiple steps in existing token-level decoding algorithms, we organize the memory as a trie and maintain state in the beam on the prefix of the trie matched so far. additionally, we maintain a backoff beam score that keeps track of the model score for retraction unless the match completes a full word. we first present how we create the memory and then discuss our method of interpolating default model token distribution p θ with lookups from the memory. ## experiments we present an extensive comparison of prism with existing methods on both state-of-the-art transducer @xcite and encoderdecoder @xcite asr models spanning three different dataset settings. models and implementation details. for the encoder-decoder models, we use two publiclyavailable whisper models @xcite : whisper-base(74m parameters) and whisper-small (244m parameters). whisper was trained on over 600k hours of weakly labeled training data mined from the web. we do not do any finetuning of the whisper model, and only alter the beam-search decoder to incorporate prism. we used a beamwidth of 10 in all the whisper experiments. the transducer model (56m parameters) is trained on a proprietary dataset of 56k hours of labeled speech. this model takes 240-dimensional log-mel filterbank audio features as its input and predicts probability distributions over an output vocabulary of 45 english characters and a blank character. we elaborate in appendix a. memory. we use the faiss library @xcite to index and efficiently search through embeddings. recall that we index embeddings from both the audio and text encoder for the transducer model. we use a state-of-the-art vits @xcite system from coqui tts (coqui.ai, 2023) to synthesize audio for dictionary entries. we set the thresholds for the audio match to 0.4 and text match to 0.05. note that the same threshold is used across all datasets spanning 9 different dictionary lists. the text-based threshold is strict, thereby only allowing for matches that the language model readily permits. however, the audio-based threshold is more relaxed allowing for more variability in the audio matches to accommodate varying pronunciations, noise in the audio, etc. with the whisper models, we use an unnormalized embedding-based matching with a distance threshold of 250 and 1000 for the base and small models respectively. (normalized distances did not perform as well.) we further apply a min-max normalization to these unnormalized distances to use as a dynamic knn coefficient for the interpolation (outlined in section 3.2). for all the experiments, we use the four nearest neighbors while searching the index. ## conclusion we present prism, an efficient inference-time dictionary-based asr adaptation technique that uses synthesized speech for dictionary entries to facilitate improved matches. prism includes an efficient trie-based memory lookup algorithm, with effective fallback scoring mechanisms to prevent spurious matches. prism can augment standard beam-search decoders in both transducer and whisper encoder-decoder asr models. we demonstrate the effectiveness of prism with experiments across diverse dictionaries containing rare words and named entities. we obtain significant relative wer reductions of up to 36% on entities, yielding an overall wer reduction close to 10%, on the challenging test-other split in librispeech benchmark. we also achieve superior performance on evaluation sets drawn from an enterprise setting, with overall and entity error rates almost reduced by a factor of 2 (i.e., relative reductions of 40% to 55% over the baseline asr systems).