Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment (2025)

Abstract: Aligning language models with human preferences presents significant challenges, particularly in achieving personalization without incurring excessive computational costs. Existing methods rely on reward signals and additional annotated data, limiting their scalability and adaptability to diverse human values. To address these challenges, we introduce Persona-judge, a novel discriminative paradigm that enables training-free personalized alignment with unseen preferences. Instead of optimizing policy parameters through external reward feedback, Persona-judge leverages the intrinsic preference judgment capabilities of the model. Specifically, a draft model generates candidate tokens conditioned on a given preference, while a judge model, embodying another preference, cross-validates the predicted tokens whether to be accepted. Experimental results demonstrate that Persona-judge, using the inherent preference evaluation mechanisms of the model, offers a scalable and computationally efficient solution to personalized alignment, paving the way for more adaptive customized alignment. Our code is available here.

## introduction aligning large language models (llms) has shown tremendous potential in following human instructions and reflecting human preferences @xcite @xcite . however, aligning with a unified preference often overlooks the need to accommodate individuals, as individual preferences can vary significantly due to factors such as cultural, educational, religious, and political backgrounds @xcite @xcite . personalized alignment @xcite addresses this gap by tailoring language models to individual human base lm harmless dataset helpful dataset harmless and helpful preferred pre-trained rm predict reward (b) reward-guided lm policy pre-trained rm label rewards sample (a) policy-based preference dataset preference dataset maximum likelihood base lm harmless preference harmless preferred lm helpful preference helpful preferred lm draft model judge model predict verify (c) persona-judge preferences and values, which is crucial for human-ai interaction and user-focused applications @xcite . policy-based methods @xcite , such as reinforcement learning from human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite , as illustrated in figure 1(a) , use training signals from explicit or implicit rewards to optimize policy models. however, these methods struggle to scale to changing personalized preferences, with challenges including the construction of high-quality preference datasets and the substantial computational costs associated with optimizing the policy. to achieve personalized alignment efficiently, existing works utilize reward signals to guide @xcite , linearly combine the predictive distributions of different base models to generate the harmless prefix helpful prefix instruction : why is it that only proteins, carbohydrates and fats are deemed to have caloric value? preference p: , helpful harmless base language model harmless preferred lm helpful preferred lm round 0 draft model draft model judge model @xmath0.9 @xmath1.3 judge model forward thank you for the question , thank you for the question , thank you for the question , thank you for your round 1 round 2 round 3 accept reject next iteration thank you for thank you for the thank you for the question thank you for the question , @xmath2.8 @xmath3.3 @xmath4.2 @xmath5.5 figure 2 : an illustration of the inference phase for persona-judge. given the target preferences and the current context, different preference prefixes are first assigned to the same base model. in each sequence-length iteration, the two models alternate in the roles of the draft and judge models, which allows for the calculation of the likelihood of accepting the next token based on distinct probability distributions. in this paper, we introduce persona-judge (figure 1(c )), a generalizable and training-free personalized alignment framework. persona-judge leverages the intrinsic ability of the model to discern preferences by directly employing existing llms as judges, thus eliminating the need for external reward signals. inspired by speculative decoding @xcite , persona-judge adopts a draft-and-judge pipeline, where the same base model serves both as the draft model and the judge model. specifically, persona-judge interprets the preference representation of the output as the acceptance of the predicted token. different preferences are used to prompt the same base model, with the base model taking turns serving as the draft model to generate a sequence, the judge model to verify whether the predicted tokens are accepted. experimental results demonstrate that personajudge, by exclusively utilizing the model's intrinsic preference judgment capabilities, achieves performance comparable to training-based methods in a completely training-free setting. furthermore, it exhibits remarkable generalization ability across diverse and unique preferences, highlighting its exceptional scalability. ## persona-judge inspired by related works in llm-as-a-judge (appendix b.2), we aim to effectively implement such judgment-like behavior without introducing additional external training while retaining the advantages of prompt-based approaches, to ensure fast, efficient, and scalable alignment. ## conclusion this paper presents persona-judge, a novel approach for personalized alignment that eliminates the need for external reward signals or policy finetuning. by leveraging the model's inherent capability for preference judgment, persona-judge effectively aligns multidimensional preferences in the prediction of the next token, making it a promising solution for adaptive personalized alignment.