Unsupervised Human Preference Learning (2024)

Abstract: Large language models demonstrate impressive reasoning abilities but struggle to provide personalized content due to their lack of individual user preference information. Existing methods, such as in-context learning and parameter-efficient fine-tuning, fall short in capturing the complexity of human preferences, especially given the small, personal datasets individuals possess. In this paper, we propose a novel approach utilizing small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization. Our method involves a small, local “steering wheel” model that directs the outputs of a much larger foundation model, producing content tailored to an individual’s preferences while leveraging the extensive knowledge and capabilities of the large model. Importantly, this personalization is achieved without the need to fine-tune the large model. Experimental results on email and article datasets, demonstrate that our technique significantly outperforms baseline personalization methods. By allowing foundation models to adapt to individual preferences in a data and compute-efficient manner, our approach paves the way for highly personalized language model applications.

## introduction large language models have revolutionized various domains with their impressive capabilities in reasoning, knowledge retrieval, and content generation. however, a crucial aspect where they often fall short is the ability to adapt their outputs to individual user preferences. while llms excel at generating content that caters to a broad audience, they struggle to produce outputs that resonate with the unique stylistic and contextual needs of individual users @xcite . this limitation stems from the fact that llms are typically trained on massive, generic datasets, promoting neutrality and hindering their capacity to learn and adapt to individual nuances @xcite . existing personalization techniques, such as incontext learning (icl) @xcite and parameter-efficient fine-tuning (peft) @xcite , have shown promise in adapting llms to specific tasks. however, these methods encounter significant challenges when applied to the domain of human preference learning. icl, while effective in providing general guidance, struggles to capture the intricate and often contradictory nature of human preferences, especially when relying on limited in-context examples @xcite . similarly, peft methods, while efficient in terms of compute and storage, face difficulties in generalizing to new preferences, particularly when users only possess small, personal datasets @xcite . this raises a fundamental question: how can we efficiently and effectively align powerful llms to individual user preferences, especially when personalized data is scarce? to address this challenge, we propose a novel approach based on the concept of preference agents. these agents are small, locally trainable language models designed to learn and encode individual user preferences into concise natural language rules. these agents act like a small "steering wheel," guiding the output of a much larger, generic llm towards the desired personalized style and content. this modular architecture decouples the task of preference learning from the generic llm, allowing users to efficiently fine-tune a small agent on their personal data without the need for modifying the weights of the larger model. our approach represents a significant departure from conventional preference learning methods, offering a potential solution for unsupervised human preference learning. we evaluate our method across three diverse datasets encompassing human-generated content: emails, news articles and product reviews. our results demonstrate that preference-guided llms significantly outperform both fine-tuning baselines and standard prompting techniques, based on automatic metrics, gpt-4o evaluation, and human judgments. specifically, our contributions include: ## method in this section, we detail our approach for aligning language models to personalized user preferences using small preference agents. our method involves two key components: generating natural language rules that capture user preferences and utilizing these rules to guide a larger, pre-trained language model. this modular architecture allows for efficient personalization without extensive retraining. ## results we evaluated the performance of our fine-tuned preference agents against several baselines using gpt-4o and human evaluation. we report the win rates -the percentage of instances where our method outperforms the baseline -in large model baseline. while the baseline produced by the large model improves, the improvement is most noticeable in domains with lower degrees of available personalization (such as articles and product reviews), and less noticeable when direct preferences are involved (such as email writing) few shot. while our method consistently outperforms the few-shot baseline across all datasets, the performance gap is more pronounced in the new yorker dataset, compared to the lamp3u or enron datasets. we hypothesize that this difference stems from the nature of the tasks. few-shot examples are likely more effective for email writing or product review writing, a relatively structured and concise format, than for long-form article writing, where capturing stylistic nuances requires more than a few examples. naive finetune. human evaluators exhibited a stronger preference for preference agent outputs over naive fine-tuning compared to gpt-4o. post-annotation interviews revealed that naive finetuning often resulted in hallucinations of crucial information, a phenomenon not effectively penalized by automated metrics but disliked by humans. no baseline agent. the preference agent trained without access to zero-shot baseline information exhibits competitive performance, particularly when considering the marginal reduction in in-ference cost it offers. this makes it a viable alternative in scenarios where minimizing inference cost is a priority, even if it comes with a slight compromise in performance compared to the distillation-based approach. both automated and human evaluations confirm that our preference agent significantly improves the alignment of llm outputs with individual user styles and preferences, as discussed in section 5.1. the strong and consistent performance across diverse datasets and llms highlights the generalizability utilizing preference agents. qualitative examples and human annotation samples of the results are provided in appendix j. ## related work traditional methods of alignment. aligning language models to human preferences often employs techniques like reinforcement learning from human feedback (rlhf) @xcite and its variant, reinforcement learning from ai feedback (rlaif) @xcite , which leverages fine-tuned llms as annotators. while effective, rlhf requires substantial human annotation and complex distributed training. direct preference optimization (dpo) @xcite offers an alternative by using preference pairs, reducing computational complexity. however, dpo's reliance on contrasting pairs may not fully capture the nuances of overlapping human preferences. in-context learning methods @xcite , while showing promise, are limited by context length restrictions, hindering their ability to generalize effectively. agent-based alignment. to address the computational demands of training large models, agentbased architectures have emerged as a promising avenue for compute-constrained environments. for instance, @xcite utilize a fine-tuned t5 policy model to guide large models via stimulus prompting. however, this approach necessitates full-parameter supervised fine-tuning (sft) and rl optimization, introducing computational overhead and yielding limited performance gains in tasks like dialogue generation. similarly, aligner @xcite employs full-parameter sft and relies on a substantial custom dataset for preference learning, posing challenges in terms of data requirements and vram usage. @xcite propose parameter-efficient fine-tuning (peft) methods to personalize agents based on user history and preference retrieval. while computationally efficient, this approach is constrained by the reasoning capabilities of the smaller fine-tuned agent. these approaches often rely on automatic metrics like bleu and rouge, which predominantly capture lexical similarity without fully encapsulating the nuances of human preferences. @xcite introduce an agent trained on human edits to align zero-shot outputs. however, this approach requires multiple inference rounds for each query, increasing latency and computational costs. moreover, human edit history may not consistently reflect genuine preferences, and relying solely on edit distance as a measure of alignment can be unreliable. @xcite propose a framework for aligning llms through multi-perspective user preference ranking-based feedback. this approach, however, involves an initial sft phase, along with multi-perspective ranking aggregation (mpra) and reward imitation learning (ril), leading to significant training overhead and the use of metrics like bleu that may not accurately capture human preferences. comparison with aligner. while both aligner @xcite and our method utilize a small model trained with full-parameter sft, our approaches differ significantly. aligner focuses on correcting model outputs post-generation, while our preference agent proactively generates rules to steer the large model's initial output. this allows us to leverage the large model's reasoning capabilities by providing preference information upfront, rather than correcting its output afterwards. while aligner demonstrates strong performance on tasks like text summarization and dialogue generation, its design is geared towards making smaller adjustments to large model outputs. our task, on the other hand, often requires more substantial changes to align with user preferences, potentially necessitating complete rewrites of emails or articles. an aligner-style approach or naive fine-tuning would face challenges in our setting, as a small model might struggle to accurately make drastic changes while preserving the large model's knowledge and coherence. this would also complicate the finetuning objective, as the patterns to be learned would be less well-defined and vary significantly across examples. ## conclusion this work introduces a novel paradigm for aligning large language models (llms) with individual user preferences using limited data. we leverage small, locally trainable "preference agents" to efficiently guide larger llms without resource-intensive finetuning. our approach generates natural language rules that encapsulate user preferences, acting as a "steering wheel" to direct the larger model's output towards desired styles and content. this framework introduces a new preference fine-tuning objective: learning from implicit preference information found in the differences between a baseline llm output and the user's desired output. this allows the agent to distill user preferences into actionable rules, enabling efficient personalization without modifying the larger model's weights. our empirical findings across diverse datasets demonstrate that preference agents significantly improve alignment with user preferences compared to existing methods in a compute-efficient manner, highlighting the potential for building highly personalized llm applications at scale.