Systematic Biases in LLM Simulations of Debates (2024)

Abstract: The emergence of Large Language Models (LLMs), has opened exciting possibilities for constructing computational simulations designed to replicate human behavior accurately. Current research suggests that LLM-based agents become increasingly human-like in their performance, sparking interest in using these AI agents as substitutes for human participants in behavioral studies. However, LLMs are complex statistical learners without straightforward deductive rules, making them prone to unexpected behaviors. Hence, it is crucial to study and pinpoint the key behavioral distinctions between humans and LLM-based agents. In this study, we highlight the limitations of LLMs in simulating human interactions, particularly focusing on LLMs’ ability to simulate political debates on topics that are important aspects of people’s day-to-day lives and decision-making processes. Our findings indicate a tendency for LLM agents to conform to the model’s inherent social biases despite being directed to debate from certain political perspectives. This tendency results in behavioral patterns that seem to deviate from well-established social dynamics among humans. We reinforce these observations using an automatic self-fine-tuning method, which enables us to manipulate the biases within the LLM and demonstrate that agents subsequently align with the altered biases. These results underscore the need for further research to develop methods that help agents overcome these biases, a critical step toward creating more realistic simulations.

## introduction the emergence of large language models @xcite has opened up exciting possibilities for computational simulations that aim to accurately replicate human behavior @xcite . current research suggests that llm-based agents become increasingly human-like in their performance and that they possess the remarkable ability to seamlessly adopt personas of different characters @xcite . the typical paradigm for such simulations involves selecting an llm, such as the widely used chatgpt @xcite , as a base model and crafting individual agents' identities through natural language prompts. for instance, by prepending the prompt, "john lin is a pharmacy shopkeeper," to an agent's context, the agent is expected to act as if his name is john and he works as a shopkeeper @xcite . if sufficiently reliable, these simulations could serve as invaluable tools for exploring the intricacies of human interactions and decision-making processes. this would allow scientists to conduct their research with speed and efficiency, substantially lowering the considerable resources usually needed for recruiting and analyzing human subjects. consequently, a range of studies have demonstrated the promise of these simulations across various disciplines, including human psychology @xcite , social dynamics @xcite , and economics @xcite . however, llms are complex statistical learners that do not depend on straightforward deductive rules. despite exhibiting impressive emerging skills that challenge our current understanding of cognition @xcite , their indeterminate nature leaves them susceptible to unintended behaviors. one example is their manifestation of inherent biases, including gender bias @xcite , ethnic bias @xcite , and social identity bias @xcite . given their undefined nature, it is vital to exercise caution when using llms, particularly in multiagent environments aimed at simulating complex, large-scale social phenomena. in this study, we explore the behavior of llm agents within simulations. our experiments are focused on the realm of attitude change @xcite and specifically on the extensively studied interactions between political partisans @xcite . this domain is susceptible to numerous prejudices @xcite , making it an ideal candidate for investigating the effect of llm biases on simulations. we facilitate debates on polarizing american topics between llm agents representing republican and democrat perspectives. the selected topics involve important aspects of people's day-to-day lives and decision-making processes. they are relevant to economic outcomes and markets, sociological and psychological phenomena, and for issues related to ethics. during every debate, we continuously monitor the agents' attitudes by asking them to rate their agreement with the debate's topic. to assess the believability of the agents' behavior, we compare the dynamics of their attitude shifts with known patterns seen in human interactions @xcite . in addition, we have developed a fine-tuning mechanism for agents, leveraging training data produced by the agents themselves. the data is generated by using a set of questions crafted to elicit the agents' political views, and the agents' responses are then used to train the base llm. we use this process to conduct controlled intervention studies, by manipulating the llm biases and analyzing the subsequent impact on the agents' behaviors. our results reveal that llm agents generally conform to the inherent social biases of their base models, even if these biases conflict with their assigned identities. consequently, this causes the simulations to diverge from well-established human social behaviors. moreover, when we employ our fine-tuning method to change the llms' viewpoints, we observe that the agents, despite retaining their original contexts, modify their behavior to be in line with the newly introduced bias. these insights underline the need to investigate ways to help agents circumvent these biases, a crucial step in developing simulations that more accurately reflect real human behavior. ## related work believable llm simulations recent studies show that llms possess human-like reasoning skills @xcite , and that llms are able to adopt personas of diverse characters @xcite . leveraging these abilities, park et al. (2023) developed a sandbox environment, populated it with 25 llm-based agents, and showed that the agents convincingly mimic human behaviors such as sharing news and forming relationships. the transformative potential of such simulations in areas like human psychology @xcite and economics (horton, 2023) was a significant motivator for our work. nonetheless, our findings indicate that inherent biases in llms pose substantial challenges in ensuring the reliability of agents to generate believable human behavior. ## problem definition our study delves into the impact of inherent biases within llms on their ability to accurately emulate diverse characters @xcite . we explore this relationship by facilitating political debates between llm agents. section 4 outlines our simulation methodology, including the criteria for selecting debate topics (4.1), how we crafted agents' identities (4.2), and techniques for managing and evaluating interactions between the agents (4.3). section 5 introduces a novel fine-tuning technique for agents, utilizing self-created training data. we have developed this method to adeptly adjust the llm's perspective, and it is applied in the controlled intervention experiments discussed within this research. in section 6, we present the primary findings of our work. through a sequence of experiments, we establish a strong connection between the inherent biases of llms and the patterns of attitude change observed in our simulations. lastly, section 7 offers a complimentary analysis aimed at evaluating and enhancing the robustness of our fine-tuning process against standard benchmarks. ## fine-tuning methods in the preceding section, we outlined our approach for simulating debates and tracking the attitude changes of the agents involved. in addition, our work offers a controlled intervention study designed to investigate the relationship between llm biases and role-played agents. for this purpose, we have developed an automated fine-tuning technique for the agents, which allows us to manipulate their underlying llm biases effectively. our method depends solely on training data generated by the agents, without the need for external data inputs. we commence the process by constructing a series of 100 questions intended to encourage agents to reveal their opinions and sentiments regarding their political views. we start with an initial set of 10 sample questions of different styles, for example, 'could you discuss your perspective on significant political issues facing america today?' or 'how do you believe the u.s. should handle immigration and border security?' (the remaining questions are detailed in the supplementary we intentionally craft the questions in a neutral way and ask a broad spectrum of questions, not only questions concerning the debated topics directly. we do this for two reasons: (1) we want our fine-tuning approach to be generic so it can be easily adjusted to other scenarios without needing to rewrite the questions. for instance, we used the same set of questions to generate both a republican-oriented dataset and a democraticoriented dataset. (2) a broader variety of training data helps the agent to maintain its conversational skills and avoid over-fitting. next, we initialize an agent with a context similar to the template used to create agents in section 4.3. we query this agent with the 100 previously generated questions. for every question, we gathered 20 responses by operating the agent at a tem-create a background story of a passionate democrat partisan with the following ideology: -believes climate change is a big problem. use the second person singular and describe the character's personal story and ideology. this story was generated automatically by feeding the llm with the prompt described in (a). we opted to develop comprehensive identities for each agent across all topics simultaneously rather than creating an individual agent for each topic. this strategy simplified our experimental design and provided a complete representation for each agent. a diagram of the procedure and additional technical details are provided in the appendix section a.2. all the reported scores for fine-tuned models included in this paper are the average of three independent fine-tuning runs with random seeds. ## fine-tuning robustness in section 5, we describe our multi-stage self-finetuning method that is shown to effectively alter the model's perspective toward a designated viewpoint. we designed our approach to be streamlined and easily replicable, focusing on ensuring the robustness of the process without resorting to localized optimizations. as a result, we made the following design choices: (1) solely using self-generated data, avoiding external dataset sources. (2) finetuning a comprehensive model applicable across all debate topics, rather than training individual models for each topic. (3) employing a simple nextword prediction task, in contrast to more complex reinforcement learning techniques. (4) using the efficient qlora method @xcite , which enabled training the model in minutes. the r, α lora hyper-parameters, which respectively control the number of trainable weights and the scale of weight updates, had a significant impact on our results. by increasing these hyperparameters, we observed a marked change in the political orientation of the default agent, which serves as a reflection of the llm's built-in bias. although our study primarily aims to modify the political viewpoint of the model, exploring how such adjustments impact the overall abilities of the llm is intriguing. in finally, we present an incremental optimization to our fine-tuning process, which enables us to manipulate the model's perspective more aggressively while mitigating the negative effects on its general performance. this optimization is based on the cutting-edge dpo method @xcite , which can be divided into two phases: first, a next-word-prediction phase that acclimates the model to the intended data distribution, followed by a contrastive learning phase aimed at teaching the model to differentiate between preferred and non-preferred outputs. as detailed in section 5, our models undergo fine-tuning through a nextword-prediction task, alongside the creation of selfgenerated datasets encapsulating republican and democrat viewpoints. this groundwork allows us to directly employ the dpo's second phase on the pre-fine-tuned models and leverage our partisan datasets as input to the contrastive learning task, training a republican model to prefer a response from the republican dataset and vice-versa. again, we train for a single epoch using the qlora. the results of this process are also included in ## discussion in our simulations of debates involving agents representing republicans and democrats, a persistent pattern emerged: agents' opinions consistently align with the llm's inherent social biases. in particular, when the model exhibits a strong bias in favor of one partisan agent, the opposing agent, which initially holds a differing view, often moderates its stance, gravitating significantly towards the position of its counterpart. this leads to a skewed pattern that appears to depart from the typical dynamics observed in human interactions. furthermore, using our self-fine-tuning process, we perform a controlled intervention study, demonstrating that it is possible to alter the llms' biases, and the agents will subsequently adjust their positions and align with the new biases. this highlights the strong influence of the llms' biases on agents behavior. it also implies that simulations by different llms, each with its unique set of biases, could yield vastly different portrayals of "authentic" human behavior. remarkably, even when agents engaged in debates with others of the same political orientation, they tended to adopt more moderate views over the course of interaction, increasingly mirroring the llm's default bias. this pattern is intriguing because it deviates from the well-documented realworld phenomenon called echo chambers @xcite , where like-minded individuals often reinforce and escalate their beliefs when interacting with each other. in an analogous real-life study, hobolt et al. (2023) divided labour and conservative supporters in england into groups to discuss government policies. contrary to our agent-toagent simulations, they found that echo chambers in homogenous groups intensified polarization. our findings thus highlight limitations of large language model agents as accurate representations of real-life humans. the political landscape, as well as the specific topics that we chose (section 4.1), are an important aspect of the day-to-day life of people and their decision-making processes, relevant to economic outcomes and markets, sociological and psychological phenomena, and for issues related to ethics. hence, the limitations we identified should be acknowledged as major factors in the usage and interpretation of large-scale simulations that aim to represent human behavior more accurately, such as in @xcite . in summary, despite llms being supposedly renowned for their ability to emulate human behavior @xcite , our research uncovers the constraints imposed by their intrinsic biases on their ability to simulate diverse agents with convincing personalities. this pivotal concern should be studied, addressed, and taken into consideration. our fine-tuning methodology demonstrates the possibility of modifying agents to adhere to specific perspectives consistently across simulations, unlike the temporary effects seen when defining agents' identities through prompts. we advocate for future research aimed at helping agents transcend the inherent biases of the model, potentially leveraging our fine-tuning processes and other alignment techniques, paving the way for more accurate and human-like simulations for both research and practical applications.