Retrieve and Copy: Scaling ASR Personalization to Large Catalogs (2023)

Abstract: Personalization of automatic speech recognition (ASR) models is a widely studied topic because of its many practical applications. Most recently, attention-based contextual biasing techniques are used to improve the recognition of rare words and/or domain specific entities. However, due to performance constraints, the biasing is often limited to a few thousand entities, restricting real-world usability. To address this, we first propose a “Retrieve and Copy” mechanism to improve latency while retaining the accuracy even when scaled to a large catalog. We also propose a training strategy to overcome the degradation in recall at such scale due to an increased number of confusing entities. Overall, our approach achieves up to 6% more Word Error Rate reduction (WERR) and 3.6% absolute improvement in F1 when compared to a strong baseline. Our method also allows for large catalog sizes of up to 20K without significantly affecting WER and F1-scores, while achieving at least 20% inference speedup per acoustic frame.

## introduction end-to-end asr models based on connectionist temporal classification (ctc) @xcite and transducers @xcite are widely popular. although these models have shown outstanding improvements over hybrid models, they often struggle to recognize uncommon domain-specific words. this is further exacerbated for streaming asr models due to limited audio context @xcite . to tackle this problem, attention-based contextual adapters (ca) have been proposed to boost a list of custom entity words (called 'catalog') and have showcased to work well with catalogs up to hundreds of catalog items @xcite . however, many industrial applications have larger * these authors contributed equally to this work catalogs that can comprise of tens of thousands of words for asr personalization. for example, a catalog of products sold by a business, a list of customer names or a search in video-on-demand platforms. in this work, we identify two main challenges in scaling the existing methods to larger lists: (1) computing attention scores for each catalog item can significantly increase the latency of the system (and often redundant!), which is prohibitively critical for any streaming application, (2) large catalogs have more phonetically similar words which makes it hard for the ca models to disambiguate the correct entity for boosting. to address these challenges, we propose novel inference and training strategies. through our inference method called "retrieve and copy", we first retrieve a smaller subset of relevant entities using approximate nearest neighbor (ann) search from the large catalog and then use only the retrieved entities for contextual biasing. our best model leverages fast ai similarity search (faiss) @xcite that is designed for fast retrieval at scale. further, we introduce a fine-tuning strategy using hard negatives for the ca models. we use clustering to identify phonetically similar words from the training data and help the model learn to disambiguate between them. overall, the contributions of our work are: ## related work attention-based contextual biasing modules have widely been used by asr systems to personalize towards a catalog of a few hundred custom entities @xcite @xcite @xcite . however, @xcite showed that inference latency increases significantly even with a few thousand catalog items. similar to our approach, they propose to filter a small set of entities using maximum inner product. however, their method reduces dependency of phrase-length in the attention computation for associative memory based biasing modules @xcite . in contrast, we use a single vector to represent an entity and hence do not have this dependency. also, their experiments are limited to catalogs of size 3k, while we scale to 20k custom entities because of retrieval methods like faiss. further, we introduce a fine-tuning strategy that specifically tackles the challenges of large catalog size on accuracy. @xcite previously used difficult examples for asr contextualization but their methods relied on generating fuzzy alternatives using phonetic similarity metric, while @xcite used an ann search with audio features. on the other hand, we use a simple clustering based strategy that allows us to easily use the elements belonging to the same cluster as phonetically similar entities. ## background a ctc encoder takes in an audio, passes it through multiple conformer blocks @xcite , and generates a sequence of word piece posteriors. contextual adapters (ca) is a separate module that is added to the ctc encoder for boosting custom entities for personalization. let x 1:t denote t output audio feature vectors from the ctc encoder. let w 1:n be a list of n custom entity words. ca comprises of two main components: (i) catalog encoder: an lstm @xcite ) that encodes word-piece sequences of custom entities into vectors (denoted by c 1:n ) (ii) biasing adapter: an attention module @xcite that uses x t for each audio frame @xmath0, t ] as query and c 1:n as keys to generate biasing vectors b t . b t is then added back to x t , thereby boosting any relevant custom entity. let θ q , θ k , θ v represent the query, key and value matrices of the biasing adapter respectively. then for each time frame @xmath1, t ], attention operation is equivalent to finding a score of each entity word w n , @xmath2, n ] using the inner product s n @xmath3, θ k c n ⟩ and then b @xmath4. for training this module, each audio-text pair (x, y) is augmented with a list of boosting words @xmath5, w ′ }, wherein the word w is from the ground truth transcript y that has the least term-frequency in the entire training data and w ′ is a random subset of other low term-frequency words present in the training data but not in y. in this way, ca learns to distinguish word w from the rest of words w ′ and boost its probability in the output sequence. we choose words with lower term-frequency as they are the hardest ones to be recognized by the un-adapted ctc encoder model. ## methodology in many practical applications, the number of custom entities at inference time (n ) can be substantially large and can contain up to 20k entities. as highlighted in section §1, this creates challenges for both inference speed and performance. following are our proposed inference-and training-side strategies designed to tackle these challenges respectively. ## conclusions in this work, we identified the challenges with the use of contextual biasing methods for an industrial use case of biasing towards large catalogs. as the size of the catalog increases, we see a significant increase in latency and a corresponding drop in accuracy, making it practically infeasible to use existing approaches at such scale. to mitigate these challenges, we propose a "retrieve and copy" inference that leverages efficient ann search methods like faiss to selectively choose a small subset of relevant entities per audio, thereby improving inference latency by at least 20%. additionally, to improve the accuracy, we propose a fine-tuning strategy that uses phonetically similar words as hard negatives to train the model. it yields up to 6% more wer reduction and up to 3.6% absolute increase in f1 scores on one of our datasets.