C om PO : Community Preferences for Language Model Personalization (2025)

Abstract: Conventional algorithms for training language models (LMs) with human feedback rely on preferences that are assumed to account for an “average” user, disregarding subjectivity and finer-grained variations. Recent studies have raised concerns that aggregating such diverse and often contradictory human feedback to finetune models results in generic models that generate outputs not preferred by many user groups, as they tend to average out styles and norms. To address this issue, we draw inspiration from recommendation systems and propose ComPO, a method to personalize preference optimization in LMs by contextualizing the probability distribution of model outputs with the preference provider. Focusing on group-level preferences rather than individuals, we collect and release ComPRed, a question answering dataset with community-level preferences from Reddit. This dataset facilitates studying diversity in preferences without incurring privacy concerns associated with individual feedback. Our experiments reveal that conditioning language models on a community identifier (i.e., subreddit name) during preference tuning substantially enhances model performance. Conversely, replacing this context with random subreddit identifiers significantly diminishes performance, highlighting the effectiveness of our approach in tailoring responses to communities’ preferences.

## introduction language models have become ubiquitous in userfacing applications, offering an opportunity for refinement through user feedback @xcite . a common approach to this refinement process is preference tuning, where users indicate their preference between two model-generated outputs, which is then used to adjust the model weights with the aim of generating outputs generally preferred by humans @xcite . however, annotating preferences is inherently subjective @xcite . current methodologies typically aggregate these (often contradicting) preferences without considering individual differences, resulting in models tuned towards a hypothetical "average" user @xcite . previous research has explored different methods to address this issue. some approaches focus on value pluralism @xcite -aiming to present balanced views for inputs where different users may hold different values, such as subjective or opinion-based questions. work on system instructions @xcite allows users to verbalize their preferences as part of the input, such as preferred content and style. additionally, there have been efforts to categorize human feedback into underlying factors such as demographics, culture, and stylistic preferences that advocate for factor-specific fine-tuning @xcite . however, preferences can be nuanced and often implicit. verbalizing them or factorizing them into interpretable dimensions may not always be feasible or practical in user-facing applications @xcite . in contrast to prior work which tends to separate the preference provider from the preferences, in this work, we propose to model the provider directly. inspired by research in recommender systems @xcite , we propose to contextualize lms with information about the users providing the preferences. that is, instead of p(y | x) where x and y are user input and model output, respectively, we propose to model p(y | x, user) where user provides the preference. given a preference dataset marked with this additional information, we make simple modifications to the supervised finetuning and preference tuning stages to incorporate 1 8246 this context ( §3). since gathering extensive user-specific preference data is infeasible for academic research, we validate our methodology by training models customized to reddit communities (i.e., subreddits). reddit's post-comment format naturally supports building question-answering models where posts can serve as model inputs and comments as outputs, respectively. additionally, subreddit users vote on others' comments, allowing us to use collective upvotes as a proxy for community preference. reddit consists of thousands of subreddits often discussing similar topics but different participants and different upvoting patterns. as a result, this data offers a readily available resource of diverse preferences to learn from. we refer to this approach as community preference optimization (compo). our aim is to study whether incorporating the community's context during preference tuning results in models generating responses tailored towards the communities' preferences. to that end, we collect and release compred, a preference dataset comprising five different groups of subreddits (covering themes related to gender, politics, history, science, and finance). each group constitutes subreddits that discuss similar topics but differ from each other in values or norms and thus diverge in preferences (e.g., r/askliberal vs. r/conservative under politics). with experiments using direct preference optimization (dpo; @xcite , we find that, across all domains, adding subreddit context indeed leads to more tailored responses, which are preferred both by human annotators and automated metrics. conversely, conditioning on the wrong subreddit yields inferior results, indicating that generating outputs preferred by a different user is detrimental to the current user's experience. ultimately, our work introduces a novel dataset and method to personalize lms to fine-grained communities, paving the way towards personalized and adaptive ai assistants. personalization in recommender systems our work draws parallels to recommender systems where personalization serves to match the right types of services, products, or content to the right users intended to improve user engagement whereas showing every user the same recommendation is undesirable. collaborative filtering, a widely adopted technique in recommender systems, matches different users with similar tastes @xcite . while early work in this space relied on learning linear separable or interpretable features for each user and product @xcite , those approaches have been surpassed by neural network-based methods to learn user and product embeddings as a means to capture the underlying preferences and intrinsic characteristics of users and items @xcite . we draw inspiration from this line of work to model preference providers directly in language models. preference tuning datasets high-quality datasets have been the primary driver of advances in nlp. while several human-annotated preference datasets are publicly available @xcite 2022; 2 q. how should the community handle immigration? a. we should welcome all immigrants help them integrate successfully. b. we should prioritize support for legal immigrants who contribute to the community. nakano et al., 2021; kaufmann et al., 2023, inter alia), they have been primarily focused on aspects such as helpfulness and harmlessness and are collected with a goal of obtaining high agreement among annotators or between researchers and annotators (ziegler et al., 2019; stiennon et al., 2020) rather than embracing subjectivity. our work is related to jang et al. (2023), who collect a synthetic data of factor-based preferences using gpt-4 as a judge; we rely on natural existing community-based preferences. most closely related to our work is ethayarajh et al. (2022, shp) who also collect a reddit based preference dataset. they largely focus on knowledge-seeking subreddits aimed at improving general purpose alignment. our goal is modeling diversity in preferences; hence, we curate a different set of subreddits where divergences are more prevalent. ## community preference optimization to build conversational or instruction following language models p θ (y | x), a typical pipeline starts with pretraining, followed by supervised finetuning (sft) to obtain p sft , and finally preference tuning. sft requires a dataset with instances of the form (x, y) where x is the input query in natural language, and y is the expected output. preference tuning is done with a dataset comprising prompts, preferred answers, and dispreferred answers, (x, y j , y k ). while the standard recipe for preference tuning relies on reinforcement learning (rlhf), in recent work, @xcite proposed direct preference optimization (dpo), which implicitly optimizes the same objective as rlhf but offers higher stability. hence, we adopt this framework for all experiments in this work. foot_0 dpo's training objective is as follows our goal is to build a community-personalized model, p θ (y | x, r), where r is the subreddit to which the model will be personalized towards. to train this model, we collect a preference dataset from reddit along with the subreddit: tuples of the form (r, x, y j , y k ). the subreddit r is represented by its name (such as r/conservative). 3 the same pair of answers (y j , y k ) may have different preferences depending on the subreddit r (see figure 1 ). to incorporate r into the pipeline, we concatenate the subreddit name in front of the input text x for both the sft and dpo stages. we refer to this approach as community preference optimization. in what follows, we detail our data collection strategy. ## compred: a dataset of diverse community preferences from reddit the reddit platform is divided into community forums known as subreddits. the majority of subreddits are created based on discussion topics; however, new subreddits are often formed where a subset of the community diverges from the existing ones. these divergences stem from multiple factors, including topic specialization, user demographic differences, value disparities, and different community norms @xcite . as a result, the preferences of the communities for what discussions they encourage are different. for instance, as illustrated in figure 1 , the subreddits r/askliberal and r/conservative have different values and the same two comments posted in both forums to the same post will receive vastly different upvotes. note that while some subreddits explicitly encode their rules and norms, these preferences are implicit and more nuanced; the members generally do not verbalize their reasons for upvoting or downvoting a comment. this setup mirrors a chat-based lm where a user may not be able to or do not explicitly verbalize their preferences. our work capitalizes on these factors to collect diverging preference datasets across different subreddits. each instance in our dataset consists of the subreddit name, a question, a preferred answer, and a dispreferred answer. we begin with a collection of threads. each thread consists of a main post q posted in the subreddit r along with @xmath0: {(a 1 , u 1 , t 1 ), . . . , (a n , u n , t n )} where a i denotes the comment text posted at time t i with u i being the number of upvotes it receives. 4 for each pair of comments a i and a j in this thread, we add to our dataset a preference instance (r, q, a i , a j ) if t @xmath1. that is, the comment a i is preferred over a j if it was posted after a j and still received more upvotes than the latter. we adopt this strategy following the stanford human preference dataset @xcite which argues that using the post time along with the upvote counts provides a more robust signal of preference than just using upvotes as it avoids recency bias. early comments receive higher upvotes on average due to more exposure time, but if a comment has received more upvotes despite being posted later, it is a clearer signal of preference. notably, preferences are conditional on the subreddit in which the post was made. this means that answering the same prompts with same answers may elicit different preferences in distinct subreddits. our approach aims to capture general trends in preferences rather than specific instances, aligning with real-world scenarios where users provide feedback on model-generated responses. ## motivational experiment: context aware reward training the driving motivation and the primary hypothesis in this work is that subreddit context aware models should generate answers that the respective community will prefer over the ones that are not context-aware. to solidify this motivation, we first train reward models using the preference data with and without providing the model with community context and measure changes in their preference prediction accuracy. although we do not use the reward models directly in our model training experiments, prior work has shown that reward accuracy is a good indicator of preference data quality @xcite . we train the models using a binary classification objective following the bradley-terry approach (bradley and terry, 1952) as used by prior work @xcite . here, f θ is a parameterized reward function (llama 2 7b @xcite with a classification head in our case) which takes as input the concatenation of the subreddit name r, question x, and a response y and predicts a scalar. training f θ with this loss leads to the preferred response getting a higher reward than the dispreferred one. in this experiment, we use this model as binary classifier to measure classification accuracy for our held out evaluation set, i.e., we compute, given a set of instances containing the preferred and the dispreferred responses, what fraction of them get a higher score for the preferred one. for comparison, we train another version the reward model where the subreddit information is not provided (no-context). we report the average accuracies for each of our datasets in ## main experiments: compo we first describe our experimental setup and baselines. next, we present results of large-scale automatic evaluations followed by human evaluation. at the end, we present analysis studying the impact of training dataset size and subreddit predictibality on the degree of personalization. implementation details we use llama-2 7b as the base model @xcite and conduct all finetuning experiments using low-rank adapters (lora; @xcite to keep memory requirements low. for each of the datasets, we first conduct sft on this model to generate preferred answers conditioned on the subreddit name and the question to obtain p sft . we then continue training the lora weights using the dpo objective to obtain our final models. training stage hyperparameters are provided in appendix b.2. we generate outputs for the evaluation sets using top-p sampling @xcite with @xmath2.95 with a temperature of 0.7 following @xcite . baselines we seek to understand the benefits of community-contextualized preference tuning (we refer to our final model as compo). hence, we compare this model with baselines that are noncontextualized, are contextualized but not preference tuned, or neither. our baselines are sft-nc, where we finetune the base model only on (question, preferred answer). dpo, where we preference tune sft-nc but without providing the subreddit information. sft-c, where we finetune the base model only on (question, preferred answer) contextualized with the subreddit identifier. note that we do not specify the target subreddit for other baselines except for sft-c. in our initial exploration, including the target subreddit in other baselines did not lead to improvements and, in some cases, resulted in disfluent outputs for certain subreddits. given the resource and cost intensity of gpt-4 evaluations, we decided not to include this version as a baseline. additionally, as we discuss in §4, we aim to isolate and study nuanced implicit preferences expressed in upvote patterns which are not verbalized or factorized along explainable dimensions. hence, existing methods such as factor-based personalization approaches @xcite or simply providing preferences in the (system) prompt are not applicable to our setup. foot_4 ## conclusions we present a framework for personalizing language models to diverse human preferences. to facilitate training such models, we create a dataset of community-level preferences from reddit spanning five domains covering a diverse range of topics, demographics, values, and community norms. we propose preference tuning models by contextualized them with the preference provider (i.e., the community). experiments reveal that our approach results in more personalized model outputs which are preferred by human and language model judges. our work raises several important directions for future work. for example, how much data is needed for effective personalization or how to solve the cold start problem? how can models continually learn from personalized feedback? can we build hybrid models that consider both explicit and implicit preferences?