Drift: Decoding-time Personalized Alignments with Implicit User Preferences (2025)

Abstract: Personalized alignments towards individual users have been a long-standing goal in large language models (LLMs). We introduce Drift, a novel framework that personalizes LLMs at decoding time with implicit user preferences. Unlike traditional Reinforcement Learning from Human Feedback (RLHF), which relies on vast annotated datasets and expensive gradient updates, Drift operates in a training-free manner by steering a frozen LLM through few-shot preference modeling. Our approach represents user preferences as a composition of interpretable and predefined attributes, and employs a zero-shot rewarding mechanism based on contrastive system prompts. Experiments on both a synthetic persona dataset Perspective and a real human-annotated dataset PRISM demonstrate that Drift achieves performance comparable to standard RLHF methods while using only 50–100 examples. Our results show that Drift delivers not only computationally efficient but also interpretable personalization.

## introduction large language models (llms) have rapidly become integral to a wide range of applications, driven by advances in reinforcement learning from human feedback (rlhf) @xcite . traditionally, rlhf aligns llms with general preferences by leveraging large-scale annotations from diverse users. building on these successes, an important question naturally arises: can we align llms with individual users' personal preferences? llm personalization, however, presents several challenges. first, collecting extensive, user-specific annotations is prohibitively expensive and impractical. second, training and maintaining separate llms per user is computationally infeasible, which motivates the need for a training-free approach. third, while user-specific system prompts have been proposed as an alternative @xcite , most users struggle to explicitly articulate their complex preferences @xcite . this often leads to misalignment between stated and actual implicit preferences. to address these challenges, we propose drift, an algorithm for few-shot personalization of llms that requires no gradient updates. our key contributions are as follows: drift approximation: we first decompose complex personal preferences into a composition of simpler and predefined attributes (e.g., "emotional," "concise," "technical"). in this process, we theoretically demonstrate how to convert the rlhf objective @xcite into a drift optimization problem, enabling efficient preference modeling with minimal data. zero-shot rewarding: we leverage a differential prompting approach to reward various attributes in a zero-shot manner. specifically, we modify a base system prompt with attribute cues and com-pute the difference in log likelihood between the modified and base prompts. this differential signal acts as a surrogate reward, eliminating the need for specialized datasets or attribute-specific training. drift decoding: finally, using the composite preferences obtained via drift approximation, we derive a principled method to steer the decoding process of a frozen llm. we prove that by integrating weighted, attribute-specific rewards into the logit space, we can achieve personalized generation without any model updates or gradient computations. we evaluate drift on two fronts: (i) efficient fewshot preference modeling and (ii) personalized generation. as shown in figure 1 , unlike traditional reward models, drift reaches a test-set accuracy of 70% with only 50 examples and even outperforms a reward model trained on 500 examples. by aligning this effective preference model at decodingtime, drift consistently produces outputs that better reflect individual users' implicit preferences. extensive analysis and discussion further validate the robustness and practical benefits of drift. contributions. 1) we propose the first few-shot implicit personalization and training-free algorithm for llms that decomposes implicit preferences into diverse, interpretable attributes and aligns their composition at decoding-time. 2) we introduce the differential prompting technique to capture numerous aspects of personal preferences, demonstrating strong empirical results without the need for dedicated dataset construction. 3) we empirically show that drift achieves robust few-shot preference modeling and personalized generation on both synthetic and real-world datasets, providing significant practical benefits with extensive analysis. ## preliminaries before describing drift in detail, we review the standard rlhf pipeline and recent decoding-time alignment methods that motivate our approach. ## drift algorithms drift overcomes data scarcity and computational inefficiency by decomposing a user's complex personal preferences as a linear combination of simpler attributes. as figure 2 , we describe two key components: drift approximation, which efficiently estimates attribute weights from a few dozen examples, and drift decoding, which integrates these weights into the llm's decoding process. ## related works explicit personalization. as humans express their own preferences, recent works explored aligning llms with individual values through explicit cues. multifacet @xcite has focused on designing diverse and detailed system prompts for llm control. pad @xcite and metaaligner @xcite have leveraged fine-grained rm-such as helpsteer @xcite -to construct specific policies and guide model behavior toward system prompts. others allow users to directly specify attribute importance weights, either for training @xcite or decoding-time alignments @xcite . ## conclusion we introduce drift, a training-free framework for personalizing llms via decoding-time alignment with implicit user preferences. by decomposing implicit personal preferences into a weighted combination of interpretable attributes, drift enables few-shot personalization that is both computationally efficient and interpretable. in the personalized ai domain-where obtaining sufficient annotations is challenging and training individual, user-specific llms is impractical-drift represents a significant advance. ultimately, drift holds promise for democratizing personalized ai services and fostering human-aligned ai systems that are adaptable to individual needs.