Kiss up, Kick down: Exploring Behavioral Changes in Multi-modal Large Language Models with Assigned Visual Personas (2024)

Abstract: This study is the first to explore whether multi-modal large language models (LLMs) can align their behaviors with visual personas, addressing a significant gap in the literature that predominantly focuses on text-based personas. We developed a novel dataset of 5K fictional avatar images for assignment as visual personas to LLMs, and analyzed their negotiation behaviors based on the visual traits depicted in these images, with a particular focus on aggressiveness. The results indicate that LLMs assess the aggressiveness of images in a manner similar to humans and output more aggressive negotiation behaviors when prompted with an aggressive visual persona. Interestingly, the LLM exhibited more aggressive negotiation behaviors when the opponent’s image appeared less aggressive than their own, and less aggressive behaviors when the opponent’s image appeared more aggressive.

## introduction large language models (llms) exhibit a high degree of alignment with human behavior based on their robust capabilities for natural language understanding and generation @xcite . specifically, when conditioned with personality traits, llms demonstrate human-like outputs, including conversations, contextual understanding, and coherent relevant responses @xcite . studies have explored whether llms, when endowed with personality traits such as demographic information @xcite @xcite and psychological characteristics @xcite @xcite , exhibit behaviors comparable to those observed in human subjects. despite these advancements, the most studies have focused on text-based persona assignments @xcite . however, state-of-theart llms such as gpt-4 and claude 3 at the time of this study are not limited to text modalities but can also comprehend and generate responses based on visual inputs @xcite . similar to humans, vision modality could enhance how llms perceive and interpret assigned personas, potentially enabling them to generate more human-aligned outputs @xcite @xcite . in line with this perspective, we investigated whether assigning virtual appearances to llms through images could alter their outputs. to the best of our knowledge, this is the first study to explore whether llms can understand and behave according to their visual persona. inspired by @xcite , we assigned virtual avatar images to llms and analyzed their behavior through negotiation games. the negotiation game serves as an experimental frame-work that allows a quantitative measurement of individual behavior @xcite . previous studies have also examined whether llms output align with the personalities assigned to them in these negotiation games @xcite @xcite . considering that one's appearance and perceived aggressiveness are crucial factors in explaining negotiation behavior @xcite , we analyzed the negotiation behavior of the llms based on the aggressiveness depicted in the assigned visual persona. to explore this, we first created a novel dataset of fictional avatar images and investigated how llms assessed and perceived the aggressiveness of these avatars. we investigated whether llms exhibit negotiation behaviors aligned with the aggressiveness depicted in their assigned visual personas. as shown in figure 1 , we also explored whether llms comprehend the differences in aggressiveness between themselves and others, as represented in images, and adjust their behaviors accordingly. in summary, our research contributes in the following ways: 1) we developed a novel dataset comprising 5,185 fictional avatar images and analyzed how llms understand and perceive these images compared to humans. 2) we explored the capability of llms to modify their behavior based on virtual appearance traits assigned to them. 3) we investigated whether llms can understand and interact based on both their own and others' visual traits simultaneously. ## visual persona we curated a novel image dataset to assign virtual avatar images as personas to the llms and investigated how llms perceive the aggressiveness of these avatars' appearances. data we constructed a dataset consisting of 5,185 fantasy-like fictional avatar images. these full-body avatar images were generated using the stable diffusion model, styled to resemble 3d models. detailed information on dataset construction and examples are provided in appendix a.1. ## experiment setup we assigned each avatar image in our dataset to llms as a visual persona and asked to participate in a negotiation game. we adopted the negotiation we employed state-of-the-art multi-modal llms, specifically openai's gpt-4o (openai, 2024) and anthropic's claude 3 haiku, for costeffectiveness. given the stochastic nature of llms, all experiments were conducted at a temperature of 1.0 and repeated five times. all prompts used in the games are documented in appendix b. ## conclusion this study is the first to explore whether large language models (llms) can embody and behave according to a visual persona based on appearance characteristics provided through images. we developed a novel dataset of virtual avatar images to assign visual personas to llms and found that llms interpret these image-based characteristics in a manner similar to humans. our results revealed that llms recognize aggressiveness based on appearance and that these traits influence their behaviors. notably, gpt-4o not only understands its own appearance traits but also those of its opponents, adjusting its behavior based on these relative differences-mirroring human tendencies to dominate less aggressive counterparts and submit to more aggressive ones.