Aligning Language Models to User Opinions (2023)

Abstract: An important aspect of developing LLMs that interact with humans is to align models’ behavior to their users. It is possible to prompt an LLM into behaving as a certain persona, especially a user group or ideological persona the model captured during its pertaining stage. But, how to best align an LLM with a specific user and not a demographic or ideological group remains an open question. Mining public opinion surveys (by PEW research), we find that the opinions of a user and their demographics and ideologies are not mutual predictors. We use this insight to align LLMs by modeling relevant past user opinions in addition to user demographics and ideology, achieving up to 7 points accuracy gains in predicting public opinions from survey questions across a broad set of topics. Our work opens up the research avenues to bring user opinions as an important ingredient in aligning language models.

## introduction personality is a defining feature of human beings, shaped by a complex interplay of demographic characteristics, moral principles, and social experiences @xcite . in turn, a person's personality has a significant influence on their ability to make decisions @xcite . owing to the wide-scale adaptation of the large language models (llms) for assisting individuals in their decision-making process @xcite , it becomes increasingly critical to ensure that these models are aligned with the unique personalities of their users. with lower barriers to entry, several recent works focused on prompting llms with persona or rolebased prompts such as pretend you are a democrat (deshpande et al., 2023; santurkar et al., 1 project page: https://github.com/eujhwang/personalized-llms 2023). however, the extent to which these approaches align language models with users remains unclear due to the subjective nature of defining user personas. users have nuanced opinions that can change over time and vary depending on context. while alignment with normalized user groups like religion or political inclination may be easier, llms continue to struggle to align with individual users or the long tail of user groups. additionally, llms tend to form opinions based on their pretraining data and feedback collected from crowd workers and model designers. as a result, they exhibit low steerability, even with user groups that have major representation @xcite . aligning llms to individual and long-tail opinions has received less attention, while mostly focusing on aligning to user groups. in our analysis over pew surveys, we found that people can share all of their demographic traits but still exhibit a large variance in their opinions, rendering the current group-based llm alignment insufficient. this paper investigates the relationship between demographic traits and individual opinions in llm alignment. specifically, we seek to answer the following research question: what do we need to align an llm to a user: demographic traits, fine-grained opinions, or both? the majority of the past work in nlp literature focused on aligning llms with normalized user groups @xcite @xcite . in social science studies, however, it has been shown that all users are unique even if they belong to the same broader user group, and normalizing user groups is not a true representative of a user's opinion @xcite . we apply insights from these social science studies to an empirical setting where we try to model individuals' opinions based on their various persona information such as demographic traits, ideological inclinations, and past opinions. in this paper, we give a thorough analysis of public survey responses in the opinionqa dataset @xcite with respect to their demographics, ideology, and implicit opinions and present comprehensive experimental results using the gpt-3 model with various combinations of inputs (i.e., demographic, ideology, user's past opinions). through our dataset analysis, we found that users' opinions and demographics do not necessarily correlate with each other. our experimental results show incorporating both user opinions, demographics, and ideology, results in significant gains of up to 7 points in qa accuracy for certain topics, and utilizing the most relevant past opinions helps the model to pinpoint the more accurate answers for the users. we present a study on various components that makes a personality (in short, persona) of a user. we use the opinionqa dataset, which contains 15 topics, and each topic contains an average of 100 questions and 5k users @xcite . ## aligning llms with persona in this section, we detail our task, possible modeling approaches, and evaluation protocols in section 3.1 and discuss how to select the most relevant past opinions of a user in section 3.3. ## results and analysis here, we first analyze our model variants (section 4.1) to validate hypotheses that we gather from analyzing the dataset (in section 2.4). we also provide our model's performance when we use similar modeling setup to predict group-level opinions in sections 4.2 and 4.3. ## related work personalization past works that focused on modeling individual users were from the pre-llms era and mainly hail from the recommender systems literature @xcite @xcite . however, these systems were trained on domain-specific annotated datasets or using latent information about the users (e.g. their previously written reviews). the recent llms have seen less content from the long tail of user groups during their pre-training phase, and there has been a lack of large-scale datasets of individual opinions until recently @xcite . thus it remains an open problem whether llms can be aligned effectively with individual user persona and how different user information (e.g., demographic traits vs. past opinions) influences how well an llm can model individual's opinions. for a comprehensive comparison among all previous work, see role of demographics and ideology there have been several studies investigating the correlation between ideological attitudes and psychological traits @xcite @xcite . @xcite found that normative political ideology is central to understanding shopping as a manifestation of social and political connections. @xcite found that the cognitive decision-making strategies of individuals reflect their ideological attitudes. differently, we show that ideology is not the only important factor in predicting the user's opinion using an llm. llms with retrieval-based approach extensive prior work has used retrievals from a text corpus to aid qa @xcite , or retrievals of prior qa pairs for nearestneighbor qa @xcite . @xcite uses a memory of user opinions to retrieve past relevant data points for the prompt. @xcite showed the effectiveness of the nearest neighbor search for language modeling by extending a pre-trained language model (lm). differently from work on llms and grouplevel personalization, we show that llms can be tuned for individual users with their opinions. ## conclusion and outlook this paper offers a new insight that aligning llms to users is best done by modeling user demographics, ideologies, and the most relevant past opinions. large-scale experiments on pew surveys present in the opinionqa dataset show an approximately 7% absolute qa accuracy over strong demographybased baselines. we proactively offer suggestions to avoid personalized llms from becoming echo chambers (see ethics statement ). an exciting future direction is to continuously store user opinions and grow the memory of opinions. an aligned llm offers the benefit to offer personalized perspectives that align with a user's values and cultural beliefs. however, there exist circumstances when llms can become an amplifier for unethical and biased views. our work lays the foundation for a robust llm alignment approach. by using memory-based personalization and recording interactions saved in a growing memory, the model can inform future instances of the most relevant past opinions.