Few-shot Personalization of LLM s with Mis-aligned Responses (2025)

Abstract: As the diversity of users increases, the capability of providing personalized responses by large language models (LLMs) has become increasingly important. Existing approaches have only limited successes in LLM personalization, due to the absence of personalized learning or the reliance on shared personal data. This paper proposes a new approach for a few-shot personalization of LLMs with their mis-aligned responses (Fermi). Our key idea is to learn a set of personalized prompts for each user by progressively improving the prompts using LLMs, based on user profile (e.g., demographic information) and a few examples of previous opinions. During an iterative process of prompt improvement, we incorporate the contexts of mis-aligned responses by LLMs, which are especially crucial for the effective personalization of LLMs. In addition, we develop an effective inference method to further leverage the context of the test query and the personalized prompts. Our experimental results demonstrate that Fermi significantly improves performance across various benchmarks, compared to best-performing baselines.

## introduction the recent development of large language models (llms) has significantly accelerated progress in various nlp tasks, and yielded real-world applications used by millions of users, such as coding assistants and chatbots @xcite @xcite . as the use of llms by diverse users in real-world applications increases, personalization of llms, i.e., steering llms' responses towards the unique needs or preferences of individual users becomes progressively important @xcite . however, recent studies show that llms' responses are often biased toward certain groups but not suited for other diverse groups of users, and such biases cannot be fixed by providing simple instructions @xcite . recent work in steering the responses of llms can be roughly divided into two categories. one category is prompt engineering, which heuristically incorporates the user's information into the input prompts of llms @xcite . the other category focuses on learning from other users' data @xcite . both categories have their limitations: prompt engineering for each user would be too costly as it requires exploring the extensive search space of all possible prompts to find the personalized prompt. also, the learning-based category relies on the unrealistic assumption that personal data can be shared without violating privacy considerations. this paper addresses those limitations by introducing a new approach, namely few-shot personalization of llms with mis-aligned responses (fermi). our high-level idea is to use llm to progressively improve its input prompts based on a few examples of previous user opinions and profiles (e.g., demographics) in an iterative process. in addition to the current prompts' scores measured on given few-shot user opinions @xcite , fermi incorporates the misaligned responses (i.e., llm's responses with those prompts, which are inconsistent with given user opinions) as additional context. the contexts of mis-aligned responses include useful learning signals to update prompts such as the types of wrong predictions with the current prompts (see the empirical evidence in section 4). specifically, the iterative process of fermi consists of three steps: (1) scoring the initial or current prompts with llm, (2) updating the memory with highscored prompts in the form @xmath0, score, @xmath1, and (3) generating new improved prompts with llm based on the updated memory. in addition, we propose retrieval-or-prompt, a method to improve the inference on a given test query. retrieval-or-prompt selectively uses one of the personalized prompts obtained from the optimization, based on the context of the test query. an overview of fermi is presented in figure 1 . we demonstrate the effectiveness of fermi for few-shot personalization of llms, through extensive evaluations on various tasks including question-answering (qa), classification, regression, and generation. for example, we observe that fermi exhibited 6.8% and 4.1% average accuracy improvements on two multiple-choice qa datasets, constructed to evaluate the personalization of llms, compared to the previous state-of-the-art heuristic and optimization approaches, respectively. we also found that the personalized prompts produced with one llm are also effective on other llms, including both api-based and open-sourced ones, which is crucial for efficient deployment in practice. in addition, our in-depth analyses reveal why fermi is more effective than other prompting methods and what are the important features of prompts for effective personalization of llms. we hope our work provides useful insights for the research on llm personalization, which becomes increasingly emerging and important for the future success of llms in real-world applications. prompt optimization with llm. as the prior works for prompt-tuning, relying on the gradientbased update @xcite @xcite , become inapplicable to the recent api-based llm due to their black-box nature, other approaches have been recently explored for gradient-free prompt optimization, such as a progressive improvement using heuristic rules or llms @xcite @xcite . for example, @xcite receives text feedback on how to update the prompts by instructing llm. also, after generating initial prompts with llms, @xcite generates a semantically similar variant of the prompts with the highest accuracies. @xcite iter-ates evaluation and generation of prompts with two llms, to solve the black-box optimization such as prompt optimization; @xcite incorporates the past generated prompts with their scores to enable the llm for the optimization to construct new improved prompts. however, only providing the scores on training examples is insufficient to optimize the prompt for few-shot personalization of llms, as the context with mis-aligned responses such as the types or patterns within recursively wrong predictions can't be captured in scores. therefore, we propose an efficient way to incorporate such context, along with an additional method to improve the inference by considering the context of the given query. ## fermi: few-shot personalization via learning from mis-aligned responses in this section, we present our framework proposed for few-shot personalization of llms from misaligned responses (fermi). we first present our problem setup in section 3.1. then, in section 3.2, we present our core component that optimizes the input prompt with a given user information, by using llm as a black-box optimizer along with the additional contexts from mis-aligned responses. lastly, we introduce an efficient inference scheme after optimizing prompts with fermi, by utilizing the context of a test query (section 3.3). ## experiments in this section, we design our experiments to investigate the following questions: