Investigating the Personality Consistency in Quantized Role-Playing Dialogue Agents (2024)

Abstract: This study explores the consistency of personality traits in quantized large language models (LLMs) for edge device role-playing scenarios. Using the Big Five personality traits model, we evaluate how stable assigned personalities are for Quantized Role-Playing Dialog Agents (QRPDA) during multi-turn interactions. We evaluate multiple LLMs with various quantization levels, combining binary indexing of personality traits, explicit self-assessments, and linguistic analysis of narratives. To address personality inconsistency, we propose a non-parametric method called Think2. Our multi-faceted evaluation framework demonstrates Think2’s effectiveness in maintaining consistent personality traits for QRPDA. Moreover, we offer insights to help select the optimal model for QRPDA, improving its stability and reliability in real-world applications.

## introduction role-playing dialogue agents (rpda) are large language models (llms) equipped with assigned personas. these personas can represent various groups, such as teachers, famous characters, historical figures, or individualized personas constructed from specific user profiles and personality traits. describing the behaviors of dialogue agents in terms of role-play allows us to escape the trap of anthropomorphism and provides a conceptual framework to investigate llm's behaviours @xcite . rpda has recently gained attention in both academic @xcite @xcite and industry settings (hello history; character ai; replika), while its applications range from emotional companions @xcite , interactive video games @xcite , and personalized assistants to digital clones @xcite . understanding the consistency of personality traits in rpda's applications is crucial for predictable and coherent user interactions, establishing trust and satisfaction. it is also crucial for responsible ai development as it helps minimize the risk of unintended consequences resulting from unpredictable responses due to personality inconsistency. on the other hand, given the increasing privacy concerns associated with chatbots, locally deployed rpdas have become more attractive. these agents operate directly on users' devices, minimizing data transmission and enhancing privacy. due to resource constraints on edge devices, optimization approaches like quantization are necessary when deploying models on the edge. while several recent studies @xcite @xcite have examined the personalities of llms, none have specifically investigated the impact of quantization on the behavior of locally deployed rpda. this study investigates the consistency of rpda personality constructed from locally deployed quantized versions of llms, i.e., qrpda. by focusing on quantized models, we aim to ensure efficient performance while maintaining the integrity of the assigned personas. this addresses both performance and privacy concerns in the deployment of dialogue agents. more specifically, we want to address the following research questions (rq): ## experimental results in our experiments, the ollama framework (ollama) was adopted to deploy the selected llms. we selected four models as candidates: llama3 8b instruct, mistral 7b instruct v0.3, gemma 7b instruct v1.1, and gemma2 9b instruct. these models were evaluated under three target quantization levels: fp16, q8_0, and q4_0. to thoroughly examine personality consistency, we used 16 pairs of opposite personalities. for each pair, we conducted 20 turns of interactions and repeat each experiment for 15 times. our analysis proceeded in three stages. first, we examine the ocean scores to identify any notable trends or shifts in personality traits across the 20 turns. next, we conduct regression analysis on the linguistic features extracted from the narratives to explore how these features reflected the rpdas' personalities. finally, we perform a correlation analysis between the ocean scores and the linguistic features. this multi-faceted analysis framework enables us to thoroughly investigate the impact of model quantization and the effectiveness of the proposed think2 approach in maintaining personality consistency in on-device qrpdas. ## conclusions for the rpda created from quantized llms, our experiments discovered that personality consistency decreases at higher quantization levels. we proposed a non-parametric method named think2, which effectively mitigates this issue, maintaining stability across interactions. specifically, gemma2 with think2 in q4_0 and llama3 with think2 in q8_0 emerge as optimal choices for preserving personality traits. our multi-faceted analysis framework demonstrates think2's potential to improve qrpda reliability for on-device deployments with critical resource constraints.