The steerability of large language models toward data-driven personas (2024)

Abstract: Large language models (LLMs) are known to generate biased responses where the opinions of certain groups and populations are underrepresented. Here, we present a novel approach to achieve controllable generation of specific viewpoints using LLMs, that can be leveraged to produce multiple perspectives and to reflect the diverse opinions. Moving beyond the traditional reliance on demographics like age, gender, or party affiliation, we introduce a data-driven notion of persona grounded in collaborative filtering, which is defined as either a single individual or a cohort of individuals manifesting similar views across specific inquiries. As individuals in the same demographic group may have different personas, our data-driven persona definition allows for a more nuanced understanding of different (latent) social groups present in the population. In addition to this, we also explore an efficient method to steer LLMs toward the personas that we define. We show that our data-driven personas significantly enhance model steerability, with improvements of between 57%-77% over our best performing baselines.

## introduction in the recent past, large language models (llms; @xcite @xcite have shown exceptional generation capabilities across a range of tasks @xcite that have led to their adoption in applications pertaining to multiple highstakes fields such as healthcare @xcite , education @xcite and finance @xcite . given this context, it is of utmost importance to avoid biases towards specific underrepresented populations, and leverage llms in a way that enables generation across a broad spectrum of viewpoints in a balanced way. * * these authors contributed equally to this work in practice, llms have been shown to generate responses that represent a wide range of opinions, with tendencies to over-represent the opinions of certain populations while under-representing those of others. for example, @xcite showed that llms under-represent the opinions of individuals aged 65 and over, mormons, and the widowed, which constitute significant portions of the us population. the reason for this is that the typical fine-tuning of an llm, done across datasets at hand, leads to a model gaining a randomized viewpoint based on the nature of the dataset. instead of fine-tuning towards such a randomized viewpoint, it is desirable to enable llms to have controllable generation that can be steered towards specific viewpoints. this can be leveraged to produce multiple perspectives and in turn encourage diversity through the curated inclusion of a broad spectrum of viewpoints. such a diverse set of perspectives, enabled via controllable generation, can be extremely helpful in diminishing polarization and preventing the marginalization of the voices of minority groups. prior work has attempted to control llm generation by aligning models toward demographic groups that are defined based on features such as age, gender, political party affiliation @xcite @xcite @xcite . however, we argue that a simple group definition based on demographic features might not be sufficient to represent the nuances of the underlying different social groups present in a given population. in this paper, we present a new approach to achieve controllable generation of specific viewpoints using llms. we hypothesize that for a given dataset comprising of responses provided by a population of individuals to a set of questions, there is a space of differing characteristic opinions and beliefs. we map this space to an embedding or the centroid of a group of embeddings, referred to as a cluster persona (denoted by the circled clusters). to steer the llm we pass an embedding to a soft-prompting model (spm), which maps the embedding to a set of persona-specific virtual tokens. finally we prepend these virtual tokens to the tokenized input sequence and pass this into the llm to obtain a persona-specific response. given these data-driven personas, we then propose an efficient algorithm to steer llms towards both an individual and a cluster persona. in particular, we learn a soft-prompting model (spm) which maps the embedding of a persona to a set of virtual tokens. these virtual tokens are prepended before tokens mapping to the actual input text, to steer the responses of the llms. we conduct a number of experiments using the opinionqa dataset @xcite to evaluate the efficacy of our persona definition and the llm steering algorithm. our experiments show that llms steered via personas can align to the opinions of individuals and groups better than baseline methods. in particular, the personas defined using our approach result in more accurate prediction of opinions when compared to the use of personas based on demographic-traits. we summarize our contributions as follows: steering llms: in the recent past, several studies have investigated opinions expressed by llms @xcite @xcite . for example, @xcite curated the opin-ionqa dataset based on public opinion polls and evaluated the alignment of llm opinions with 60 u.s. demographic groups. they found a substantial misalignment between the views reflected by current llms and those of u.s. demographic groups (we mention a cross section of these in section 1). in another case, @xcite designed a survey comprising both high-ambiguity and lowambiguity moral scenarios to study the moral beliefs encoded in different llms. a variety of methods have also been proposed to steer the generation of llms toward specific opinions @xcite @xcite @xcite @xcite . for instance, @xcite added demographic information (such as democratic or republican affiliations) to prompts in order to steer the opinions of llms toward those groups. @xcite worked towards aligning llms with individuals by incorporating their past opinions. @xcite constructed four prompts based on combinations of liberal and conservative political identities, as well as moral and immoral stances, to steer the moral beliefs of llms. @xcite finetuned llms using tweets authored by democrats and republicans to improve the model's alignment with the opinions of these groups. in contrast to this prior work, which steer llms based on demographic traits, our approach using data-driven personas allows for a more expressive and nuanced understanding of the different social groups present in the overall population. this, in turn, enhances the applicability of model steerability. parameter-efficient fine tuning: given the rapid increase in size of llms, fine-tuning the full model has become costly, giving rise to various methods that only tune a portion of model parameters @xcite @xcite @xcite @xcite . in one such approach, hu et al. (2021) utlize low-rank decomposition to represent weight updates with two smaller matrices (called update matrices), and then only fine-tune the decomposition matrix. several approaches have been proposed where virtual tokens can be tuned to align a model towards a specific task. @xcite trains a sequence of virtual tokens that have been prepended to the input whereas li and liang (2021) optimize a task-specific vector of virtual tokens called the prefix. @xcite address the instability at the initial training phase using a learnable gate to control the effect of virtual tokens over generation. ## framework in this section, we propose the notion of data-driven personas (section 3.1) and an efficient algorithm for steering an llm towards them (section 3.2). an illustration of our process is shown in figure 1 . ## experimental results in this section we present our data, experimental setup and empirical results on data-driven persona definition and llm steerability. ## conclusion in this work, we present an approach for steerable generation using llms, where we utilize a datadriven notion of a persona to modulate the generation process. we proposed an simple model and efficient algorithm to align llms with these personas (both individuals and clusters). we validate the efficacy of our algorithm using the opinionqa dataset. for a select set of llms we show that our method out-performs traditional steering mechanisms supporting our hypothesis that llms align with individuals' opinions better when leveraging our data-driven personas.