P ersona LM : Language Model Personalization via Domain-distributed Span Aggregated K-Nearest N-gram Retrieval Augmentation (2023)

Abstract: We introduce PersonaLM - Domain-distributed Span-Aggregated K-nearest N-gram retrieval augmentation to improve language modeling for Automatic Speech Recognition (ASR) personalization. PersonaLM leverages contextually similar n-gram word frequencies for recognizing rare word patterns associated with unseen domains. It aggregates the next-word probability distribution based on the relative importance of different domains to the input query. To achieve this, we propose a Span Aggregated Group-Contrastive Neural (SCAN) retriever that learns to rank external domains/users by utilizing a group-wise contrastive span loss that pulls together span representations belonging to the same group while pushing away spans from unrelated groups in the semantic space. We propose ASAP benchmark for ASR LM personalization that consists of three user-specific speech-to-text tasks for meetings, TED talks, and financial earnings calls. Extensive experiments show that PersonaLM significantly outperforms strong baselines with a 10-16% improvement in perplexity and a 5-8% reduction in Word Error Rates on popular Wikitext-103, UserLibri, and our ASAP dataset. We further demonstrate the usefulness of the SCAN retriever for improving user-personalized text generation and classification by retrieving relevant context for zero-shot prompting and few-shot fine-tuning of LLMs by 7-12% on the LAMP benchmark.

## introduction language modeling is a core task in nlp with important applications in automatic speech recognition (asr) @xcite @xcite . pre-trained lms @xcite ) memorize a surprising amount of knowledge from their training corpora in the underlying neural network parameters @xcite . however, this makes it difficult to personalize them for text generation, nonstreaming asr re-scoring, and on-device streaming asr models for unseen users and domains due to the existence of user-preferred rare word patterns, facts, proper names, and other domainspecific tail words not seen frequently in the training data @xcite @xcite . retrieval augmentation @xcite (see fig. 1 ) can help personalize lms by explicitly exposing them to external world knowledge during inference @xcite . lms leverage the retrieval mechanism to select contextually relevant users/domains from an external corpus and then attend over that knowledge to inform their predictions @xcite . ## related work language modeling for rare words prediction: earliest works explored the use of lstm with auxiliary pointer networks to predict rare words and long-term dependencies in language modeling @xcite . neural cache augmentation @xcite stored past hidden activations in cache memory to predict out-of-vocabulary words. implicit cache memorization @xcite ) used cache to store past word occurrences as an alternative to the attention-based pointer mechanism. for the asr re-scoring task, cross-sentence neural lms proposed to use word usage in preceding sentences to re-rank n-best hypotheses of upcoming sentences @xcite . retrieval augmentation for language modeling: knn-lm @xcite interpolated pre-trained lms with contexts extracted from an external data store using the knn algorithm. realm @xcite proposed a neural retriever to leverage external knowledge during lm pre-training. @xcite augmented gpt-2 with a large episodic memory for a zero-shot reduction in perplexity. retrieval-enhanced transformer (retro) @xcite retrieved document chunks similar to preceding tokens using a cross-attention module. our work is the first to use domain-distributed n-gram representations over document spans to retrieve rare word patterns from the most relevant external knowledge domains. figure 2: personalm: at inference, we compute the relevance score p (di|q) between the query and domains di as the dot product of their scan retriever representations. we construct a data store for each n-gram frequency matrix. k-most similar n-gram contexts wrt to the input query are retrieved and their weighted summation based on the domain's relevance score is computed to get probability distribution over targets pp ersonalm (wt|q) and interpolated with lm probabilities plm (wt|q). over the retrieved documents as: ## conclusion we introduce personalm retrieval augmentation for asr personalization using a scan retriever trained via group-contrastive learning to rank textual documents from an external knowledge corpus based on their semantic similarity with the input query. we aggregate the probability distribution of the next word prediction by utilizing domainspecific n-gram word frequency representations weighted by the relative importance of the external domains to the input query. experiments on our proposed asap benchmark and the userlibri dataset show that our method achieves sota perplexity and wer. we show that the scan retriever is also useful for in-context llm augmentation for zero-shot prompting and few-shot fine-tuning.