Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning (2024)

Abstract: Personalization in large language models (LLMs) is increasingly important, aiming to align the LLMs’ interactions, content, and recommendations with individual user preferences. Recent advances have highlighted effective prompt design by enriching user queries with non-parametric knowledge through behavior history retrieval and textual profiles. However, these methods faced limitations due to a lack of model ownership, resulting in constrained customization and privacy issues, and often failed to capture complex, dynamic user behavior patterns. To address these shortcomings, we introduce One PEFT Per User (OPPU), employing personalized parameter-efficient fine-tuning (PEFT) modules to store user-specific behavior patterns and preferences. By plugging in personal PEFT parameters, users can own and use their LLMs individually. OPPU integrates parametric user knowledge in the personal PEFT parameters with non-parametric knowledge from retrieval and profiles, adapting LLMs to user behavior shifts. Experimental results demonstrate that OPPU significantly outperforms existing prompt-based methods across seven diverse tasks in the LaMP benchmark. Further studies reveal OPPU’s enhanced capabilities in handling user behavior shifts, modeling users at different activity levels, maintaining robustness across various user history formats, and displaying versatility with different PEFT methods.

## introduction personalization refers to mining users' behavior history, and therefore tailoring and customizing a system's interactions, content, or recommendations to meet specific needs, preferences, and characteristics of individual users @xcite . by adapting to each user's preferences, personalization systems enhance user experience, increasingly getting vital in areas like content recommendation @xcite @xcite , user simulation @xcite , personalized chatbots @xcite , user profiling @xcite , healthcare @xcite , and education @xcite . large language models (llms) display emergent abilities not seen in smaller models @xcite , as they have billions of parameters and are trained on vast corpora. however, existing llms predominantly follow the "one-sizefits-all" paradigm. they are generally trained on extensive, domain-agnostic datasets, which limits their effectiveness in meeting the specific needs and preferences of individual users @xcite . therefore, the challenge of integrating the strong generative capabilities of llms with the tailored requirements of individual users has emerged as a significant area of research @xcite . existing works on personalizing llms have predominantly concentrated on developing prompt templates, which fall into three categories: vanilla, retrieval-augmented, and profile-augmented personalized prompts. the vanilla personalized prompt approach leverages the in-context learning capability of llms, utilizing the user's entire or randomly sampled history as contextual examples @xcite . considering the growing length of user behavior history and the limited llm context length, some studies applied retrieval methods to select the most relevant part of user behavior history to enhance llm personalization @xcite . besides the retrieval, some techniques explicitly generate user preferences and profiles in natural language to augment llms' input @xcite . despite much research progress has been made in llm personalization, existing methods face ownership and behavior shift challenges (fig. 1 ): in light of these challenges, we propose one peft per user (oppu), equipping each user with a personalized, parameter-efficient fine-tuning (peft) module. characterized by peft's plugand-play functionality and the minimal weight of updated parameters (typically less than 1% of the base llm), oppu facilitates llm ownership and enhances generalization in scenarios of user behavior shifts. by fine-tuning the peft module with the user's personal behavior history, the personalized peft parameters encapsulate behavior patterns and preferences. this process, when integrated into base llms, allows users to obtain their private llms, ensuring llm ownership and enhancing model customization. furthermore, as is revealed by @xcite , fine-tuning llms is more effective than retrieval augmentation when the retrieved instances are not highly relevant to the query. the fine-tuned personal llms in oppu are adept at capturing complex behavior patterns and thus capable of understanding new behaviors with less reliance on highly relevant history data. experimental results show that oppu outperforms all baselines on seven public tasks in the language model personalization (lamp) benchmark @xcite . additional studies emphasize the importance of integrating non-parametric user knowledge from retrieved history with parametric knowledge from personal peft parameters. in scenarios of user behavior shifts, where history is less relevant, oppu significantly outperforms retrieval-based methods. moreover, oppu is resilient to varying user history formats and demonstrates versatility across different peft methods, among other advantages. to summarize, the contribution of oppu lies in its pioneering approach to peft-based llm personalization. each user (or user cohort) benefits from a personal peft module, which not only ensures llm ownership but also significantly improves the model's ability to adapt to shifts in user behavior. the superiority of oppu is evidenced by state-of-the-art performance across seven tasks in the lamp benchmark. by introducing this innovative parametric-based personalization technique, oppu opens up new opportunities in democratizing personalized llms. ## one peft per user (oppu) once the base model for task adaption is obtained, users can only access the base model parameters and their personal behavior history data, controlling privacy risks. this section introduces personalized llms for target users through parametric peft and integrates non-parametric knowledge such as retrieval and profile augmentation. for each user, we plug a personal trainable peft module (lora by default) ∆θ we then use the user data h u for llm fine-tuning to learn the personalized peft parameters. the training objectives for user u under base, retrievalaugmented, and profile-augmented settings are: by optimizing personal peft parameters with the objectives mentioned above, oppu comprehensively capture the user behavior patterns in peft parameters ∆θ ## experimental settings datasets we use data from the large language model personalization (lamp) benchmark @xcite , which includes seven public language model personalization tasks: four classification tasks and three generation tasks. 2 to promote llm ownership, we emphasize the need for users to contribute extensive historical data for personalizing their model. therefore, we focus on the most active users, selecting 100 users with the longest history logs from the time-based dataset version as the test set, while using all other users for base llm training. dataset statistics are presented in baselines we compare our proposed oppu with the non-personalized baseline and the retrievalaugmented (rag) and profile-augmented (pag) llm personalization methods. for all baselines and oppu, we choose one of the most widely adopted open-source llm llama-2-7b @xcite as our base llm and take bm25 @xcite for all retrieval operations to ensure efficient and fair comparison. 3 evaluation metrics following lamp @xcite , we use accuracy and f1-score for classification tasks (lamp-1, lamp-2n, and lamp-2m), mae and rmse for lamp-3, and adopt rouge-1 and rouge-l @xcite ) for text generation tasks (lamp-4, lamp-5, lamp-7). note that all metrics are the higher the better, except for rmse and mae used for the lamp-3. ## results oppu brings universal improvement. models equipped with oppu outperform all baseline personalization methods across all seven tasks. notably, in personalized classification tasks, oppu achieves an average relative improvement of 17.38% in mae and 8.89% in rmse for personalized product rating prediction. additionally, it shows an 11.87% improvement in accuracy and 7.56% in f1-score for personalized movie tagging. for personalized text generation tasks, oppu enhances rouge-1 and rouge-l scores by 3.42% and 3.87%, respectively, in personalized scholarly title generation. integrating non-parametric and parametric knowledge performs the best. combining oppu's parametric knowledge stored in peft parameters and the non-parametric in retrieved items and user profiles, results in notable performance gains. for instance, averaging across all seven tasks, combining retrieval in oppu will bring 1.93% and 2.48% relative improvement compared with the non-retrieval and non-oppu yet retrieval version model, respectively. moreover, integrating oppu with user profiles would also bring 4.56% and 7.18% performance gain against non-profile and non-oppu versions, respectively. overall, combining non-parametric retrieval and profile knowledge with parametric peft knowledge in oppu delivers the best performance. performance w.r.t. difference between task and history format. in tasks like personalized citation identification, there is a notable discrepancy between the user history format and the task itself. here, the user history comprises the user's publication history, while the task involves binary classification to identify the correct citation paper. this disparity is also seen in the personalized tweet paraphrasing task. in these cases, oppu significantly enhances performance. specifically, for personalized citation identification, oppu increases accu- ## analysis performance under user behavior shift recent studies have shown that retrieval-augmented generation methods tend to underperform when the retrieved corpus does not contain highly relevant documents @xcite . this problem is common in personalization contexts where the user's behavior history does not closely match their current queries. to simulate this scenario, we use deberta-v3 @xcite to extract features from the user's historical behaviors and current query, computing cosine similarity to assess relevance. we then rank the historical behaviors and select the top 100 items with the lowest relevance scores as irrelevant user history. ## conclusion we introduced oppu, equipping each user with a personal peft module that facilitates model ownership and generalization under behavior shifts. by tuning these parameters with a user's history, oppu captured personalized behavioral patterns. it integrated non-parametric user knowledge via retrieval and user profiles, showing superior performance across all seven lamp benchmark tasks. additional experiments demonstrated oppu's versatility, robustness, and effectiveness for users with varying activity levels. our framework paved the way for new opportunities in peft-based llm personalization, enhancing llm modularity for effective and democratized personalization.