L a MP : When Large Language Models Meet Personalization (2024)

Abstract: This paper highlights the importance of personalization in large language models and introduces the LaMP benchmark — a novel benchmark for training and evaluating language models for producing personalized outputs. LaMP offers a comprehensive evaluation framework with diverse language tasks and multiple entries for each user profile. It consists of seven personalized tasks, spanning three text classification and four text generation tasks. We additionally propose two retrieval augmentation approaches that retrieve personal items from each user profile for personalizing language model outputs. To this aim, we study various retrieval models, including term matching, semantic matching, and time-aware methods. Extensive experiments on LaMP for zero-shot and fine-tuned language models demonstrate the efficacy of the proposed retrieval augmentation approach and highlight the impact of personalization in various natural language tasks.

## introduction the recent development of large language models (llms) has revolutionized natural language processing (nlp) applications. as the use of llms, such as @xcite , in real-world applications evolves, personalization emerges as a key factor in meeting the user's expectations for tailored experiences that align with their unique needs and preferences @xcite . personalization has been widely studied by various communities, including the information retrieval (ir) and human-computer interaction (hci) communities, often with applications to search engines and recommender systems @xcite @xcite . recent work has also highlighted the impact and concerns associated with personalizing llms and tying it to ongoing work on alignment @xcite . despite this and the importance of personalization in many real-world problems, developing and evaluating llms for producing personalized responses remain relatively understudied. to bridge this gap, this paper underscores the importance of personalization in shaping the future of nlp systems and takes a first step towards developing and evaluating personalization in the context of large language models by introducing the lamp benchmark foot_0 -a comprehensive and diverse benchmarks of personalized text classification and generation tasks. while many existing well-known nlp benchmarks, such as glue @xcite , super-glue @xcite , kilt @xcite , and gem @xcite have led to significant progress in various nlp tasks, they have often taken the dominant nlp approach of "one-size-fits-all" to modeling and evaluation, and do not allow the development of models that adapt to the specific needs of end users -limiting extensive research on personalization in nlp tasks. in contrast, lamp offers a comprehensive evaluation framework incorporating diverse language tasks that require personalization. lamp consists of three personalized text classification tasks: (1) personalized citation identification (binary classification), (2) personalized movie tagging (categorical classification with 15 tags), and (3) personalized product rating (ordinal classification from 1 to 5-star rating for e-commerce products). further, lamp includes four text generation datasets: (4) personalized news headline generation, (5) personalized scholarly title generation, (6) personalized email subject generation, and (7) personalized tweet paraphrasing. for these seven tasks, we explore the two dominant settings in personalization: (a) personalization for new users with a userbased data split and (b) personalization for future interactions of existing users with a time-based data split. therefore, lamp provides a rich environment for developing personalized nlp models. to foster research in this area, we release the lamp benchmark, the data construction, evaluation scripts, and a leaderboard. for personalizing the language model outputs, a straightforward solution is to incorporate the user profile into a language model prompt. however, user profiles are often large and exceed the length limitations of large language models. even as such limitations are relaxed with evolving technology, the cost of processing large input sequences is considerable. therefore, we propose two retrieval augmentation solutions for llm personalization, in which for each test input, we retrieve items from the user profile to be included in the llm prompt for personalization. the first approach uses in-prompt augmentation (ipa) for personalization, and the second approach encodes each personal item separately and integrate them later in the decoder using the fusion-in-decoder model of @xcite . we demonstrate that using this approach, the performance of language models improves on all datasets in the lamp benchmark. based on this retrieval augmentation solution, we evaluate different retrievers for personalized prompt construction and establish benchmark results for fine-tuned and zero-shot language models. the empirical findings of our research reveal that the process of fine-tuning a language model utilizing our personalized augmentation technique yields a noteworthy relative average enhancement of 23.5% across the benchmark. even in zero-shot settings where an off-theshelf llm without fine-tuning (e.g., flant5-xxl) is used, utilizing our proposed method results in a relative average improvement of 12.2% across the tasks. finally, this paper smooths the path towards developing advanced user-centric nlp systems. ## retrieval augmentation for personalizing llms to personalize a language model two broad strategies may be explored: (1) fine-tuning the lm for each user and (2) prompting a shared lm with user specific input or context. the former approach necessitates substantial computational resources, especially for fine-tuning larger llms. moreover, accommodating personalized llms for each user in industry-scale systems encompassing millions or billions of users necessitates a significant storage and serving capacity. therefore, we focus on developing strategies for training models personalized via user-specific inputs. each task in lamp, each user profile consists of a potentially large collection of data points. given the inherent context length constraint of many llms and the cost of processing long sequences, we incorporate a subset of these data points as input prompts. further, not all entries within a user profile are necessarily relevant to the specific input at hand. to do this, we propose solutions based on retrieval augmentation (see figure 1 ). this framework selectively extracts pertinent information from the user profile that is relevant to the current unseen test case and generates model predictions conditioned on this information. specifically, for a given sample (x i , y i ) for user u, we employ three primary components: (1) a query generation function ϕ q that transforms the input x i into a query q for retrieving from the user u's profile, (2) a retrieval model r(q, p u , k) that accepts a query q, a user profile p u and retrieves k most pertinent entries from the user profile, and (3) a prompt construction function ϕ p that assembles a personalized prompt for the user u based on input x i and the retrieved entries. for retrieval augmentation, we explore two strategies: (1) in-prompt augmentation (ipa) and (2) fusion-in-decoder (fid) (izacard and grave, 2021). the input to both approaches constructs inputs, xi , using r to select k items from the user profile p u : note that, ipa and fid offer different tradeoffs. fid necessitates training of the language model while ipa may be applied without training. further, while fid can only be used with encoder-decoder models, ipa can be used across architectures. however, fid allows us to incorporate more items from the user profile into the llm's input. we explore various choices for the retrieval model r. in our experiments, we study a strong term matching model, bm25 @xcite , a state-of-the-art pre-trained dense retrieval model, contriever @xcite , a retrieval model that returns most recent profile entries in descending order (i.e., recency), and a random document selector from the user profile. the prompt construction function ϕ p concatenates the instruction for each task, the input sequence, and the user profile. the specific prompts are presented in ## experiments this section describes our experiments, results, and findings on the lamp benchmark. ## research problems enabled by lamp lamp can facilitate research in several areas, including but not limited to: prompting language models for personalization. the integration of user profiles into language models can be approached using hard prompts, but their limited context size makes it difficult to include lengthy user profile entries. exploring different prompts for personalization could be interesting. an alternative solution is generating personalization prompts based on the user profile, instead of relying on retrieved entries. furthermore, the use of soft prompts @xcite can be helpful for personalizing language models. ## conclusion this paper presented a novel benchmark named lamp for training and evaluating language models for personalized text classification and generation. lamp consists of seven datasets: three classification and four generation datasets. we proposed retrieval augmentation solutions for personalizing llms. notably, we studied two augmentation approaches: in-prompt augmentation (ipa) and fusion-in-decoder (fid). we performed extensive experiments using various llms and retrieval techniques for selecting user profile entries for producing personalized prompts. we demonstrated that our llm personalization approaches can lead to 12.2% average performance improvements across datasets on zero-shot setting, and 23.5% with finetuning. finally, we underscore the paramount importance of personalization in the current era dominated by large language models. we firmly believe that the future of natural language processing systems lies in a user-centric approach, tailoring solutions to individual needs for optimal effectiveness. this work was supported in part by the center for intelligent information retrieval, in part by nsf grant #2143434, in part by the office of naval research contract number n000142212688, in part by google, microsoft, and lowe's. any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.