LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models (2024)

Abstract: Agent interaction has long been a key topic in psychology, philosophy, and artificial intelligence, and it is now gaining traction in large language model (LLM) research. This experimental study seeks to lay the groundwork for our understanding of dialogue-based interaction between LLMs: Do persona-prompted LLMs show consistent personality and language use in interaction? We condition GPT-3.5 on asymmetric personality profiles to create a population of LLM agents, administer personality tests and submit the agents to a collaborative writing task. We find different profiles exhibit different degrees of personality consistency and linguistic alignment in interaction.

## introduction from hegel's claim that complex understanding emerges because two conscious agents are confronted with each others perspective @xcite to marvin minsky's positing that networked interactions of numerous simple processes, known as "agents", together create complex phenomena like consciousness and intelligence @xcite , agent interaction has long been a topic of interest within and across scientific disciplines, including philosophy, cognitive science, and artificial intelligence. recently, research in machine learning and natural language processing has taken up a novel focus on interaction in the context of large language models (llms), with experimental frameworks progressively moving away from focusing solely on individual models @xcite @xcite . on the one hand, by exploiting language as an efficient interface for information exchange, populations of llms are proving as effective engineering solutions that outperform solitary llms in a wide variety of tasks @xcite . on the other hand, building on the increasing reliability of neural models as simulations of human interactive language use @xcite , populations of llm agents show potential as scientific tools to study the emergence of collective linguistic behaviour @xcite . for llms to be successfully deployed in agent interaction studies as simulations of populations of language users, it is important to (1) develop methods that efficiently induce, from a single or a few llms, desired levels of behaviour variability (i.e., akin to the variability observed in human populations) as well as to ( 2 ) validate whether interactions between agents give rise to human-like behaviour change. previous work has explored techniques for personalising language models, text generators and dialogue systems, for example by conditioning them on a personality type @xcite , on community membership (noble and bernardy, 2022), or on profile information @xcite , thus inducing population-level variability from individual systems. this study focuses on the problem of conditioning interactive llms on personality profiles, or personas. while evidence that llm behaviour can be successfully conditioned on personality profiles is increasingly strong when it comes to monologic language use @xcite , it is yet unascertained whether this holds true when llm agents interact with other agents @xcite . in particular, it is unclear whether llm agents adhere to their assigned personality profiles throughout linguistic interactions or whether they adapt towards the personality of their conversational partners. in this paper, we report exploratory work that addresses the following two research questions: rq1: can llm behaviour be shaped to adhere to specific personality profiles? rq2: do llms show consistent personalityconditioned behaviour in interaction, or do they align to the personality of other agents? we bootstrap a population of language agents from a single llm using a variability-enhancing sampling algorithm, and we condition each agent on a personality profile via natural language prompts. we then simulate interactions between agents and assess their adherence to the specified personality profile-before, during, and after interaction. using questionnaires (big five personality tests; @xcite and quantitative analysis of language use in an open-ended writing task, we assess agents' consistency to their assigned personality profile as well as their degree of linguistic alignment @xcite to their conversational partners. in brief, our experiments show that consistency to personality profiles varies between agent groups and that linguistic alignment in interaction takes place yet is not symmetric across personas. for example, agents in the creative group give more consistent responses to bfi questionnaires than those in the analytical group, both in interactive and non-interactive experimental conditions. at the same time, the degree of linguistic alignment of the creative persona to agents of the other group is higher than that of the analytical persona. all in all, this study provides a first insight into the impact of dialogue-based interaction on the personality consistency and linguistic behaviour of llm agents, highlighting the importance of robust approaches to persona conditioning. as such, it contributes to our better understanding of the workings of interaction-based llms and shines a new light on the philosophical and psychological theme of interaction. ## experimental approach to address our research questions we conduct two main experiments. in experiment 1, we test whether personality-conditioned llm agents show behaviour consistent to their assigned personality profiles, in terms of their responses to personality tests as well as language use in a writing task. this is a non-interactive experimental condition, which will serve as a reference against which to compare llm behaviour in interaction. in experiment 2, we assess whether the personality-conditioned behaviour of llm agents changes as a result of a round of interaction with a conversational partner. this interactive experimental condition allows us to test whether agents' behaviour remains consistent or whether agents align to their partners. in this section, we present the main components of our experimental approach, which consists of bootstrapping a population of agents from a single llm ( § 2.1), conditioning agents on a personality profile via prompting ( § 2.2), assessing their personality with explicit tests ( § 2.3), and analysing their language use in a writing task ( § 2.4). foot_0 ## conclusion do persona-conditioned llms show consistent personality and language use in interaction? in this study, we explore the capability of gpt-3.5 agents conditioned on personality profiles to consistently express their assigned traits in interaction, using both explicit and implicit personality assessments. the explicit personality tests are conducted via bfi questionnaires, whereas the implicit assessment is performed by quantitative linguistic analysis of model generated stories. our experiments show that the behaviour of llm agents can be shaped to mimic human personality profiles, but that agents' consistency varies depending on the assigned profile more than on whether the agent is engaged in linguistic interaction. the creative persona, in particular, can more consistently express its bfi traits than the analytical one both in the interactive and the non-interactive experimental condition. furthermore, while non-interactive language use reflects assigned personality profiles, agents exhibit linguistic alignment towards their conversational partner and, as a result, the language of the two agent groups becomes more similar after interaction. alignment, however, is not necessarily symmetric: the creative persona adapts more towards the analytical one, perhaps due to analytical agents' low degree of openness to experience induced through persona prompting. we plan to continue this line of work by introducing more diverse and fine-grained personality profiles in our experimental setup (see, e.g., @xcite , making interactions between agents multi-turn, and measuring alignment at varying levels of abstraction-such as lexical, syntactic, and semantic-in line with the interactive alignment framework @xcite . future research should also focus on designing methods (e.g., different prompting strategies) that offer better guarantees on personality consistency and more control on the degree of linguistic adaptation.