Exploring the Choice Behavior of Large Language Models (2025)

Abstract: Large Language Models (LLMs) are increasingly deployed as human assistants across various domains where they help to make choices. However, the mechanisms behind LLMs’ choice behavior remain unclear, posing risks in safety-critical situations. Inspired by the intrinsic and extrinsic motivation framework within the classic human behavioral model of Self-Determination Theory and its established research methodologies, we investigate the factors influencing LLMs’ choice behavior by constructing a virtual QA platform that includes three different experimental conditions, with four models from GPT and Llama series participating in repeated experiments. Our findings indicate that LLMs’ behavior is influenced not only by intrinsic attention bias but also by extrinsic social influence, exhibiting patterns similar to the Matthew effect and Conformity. We distinguish independent pathways of these two factors in LLMs’ behavior by self-report. This work provides new insights into understanding LLMs’ behavioral patterns, exploring their human-like characteristics.

## introduction large language models (llms) are increasingly being adopted across numerous domains and often encounter practical scenarios where a choice needs to be made. for example, recommending some books for users without any explicit user preferences @xcite , analyzing open questions that different cultures have different viewpoints @xcite , and so on @xcite . despite the growing reliance on llms in these scenarios, the mechanisms behind their choice behavior remain unclear, raising questions about how llms make their choices and the influencing factors behind. therefore, uncovering their behavioral patterns and discovering * both authors contributed equally. psychology has long served as a powerful tool in uncovering the intricacies of human cognition and behavior @xcite , such as scenario simulation @xcite , neuroscience (uttal, 2011) and psychometrics @xcite . in recent years, the growing complexity and interpretability challenges of llms have spurred interdisciplinary approaches from artificial intelligence and psychology. by leveraging psychological methodologies, researchers are gaining deeper insights into the human-like behavioral characteristics and underlying mechanisms exhibited by these models @xcite @xcite . in terms of evaluation, psychometric insight has made it possible to assess human-like psychological traits in llms @xcite , such as personality @xcite , theory of mind @xcite . in terms of evaluation eliciting capabilities, advances in psychological research on reasoning, emotion, and motivation have enabled improvements in response quality through techniques such as the generation of multiple chains of thought @xcite . serving as a fundamental theory of human behavior, self-determination theory (sdt) distinguishes intrinsic and extrinsic motivation @xcite . the intrinsic motivation focuses on exploratory, playful, and curiosity-driven behaviors while extrinsic motivation focuses on instrumental value @xcite . in this paper, we analyze the choice behavior of llms from the perspective of sdt. we first propose three questions: (1) will llms also focus too much on one part of the options and ignore the other due to the influence of intrinsic factors when faced with choices? we use attention bias to describe this situation. (2) will llms change their choice behavior to some extent due to extrinsic factors when faced with choices? social influence is one of the important sources of extrinsic influence. to address these questions, we design an experiment platform inspired by @xcite , which explored social influence patterns through virtual music web pages. we begin by collecting two question sets spanning various domains, such as education, culture and ecology. these questions are then presented on a virtual qa platform modeled after quora (quo), where llms can view, like, and answer them. on this qa platform, we employ a controlled variable approach to observe the choice behaviors of four models from gpt and llama series across two distinct question sets under three experimental conditions. each experiment is replicated three times to ensure the reliability of the results. after analyzing the results, our key findings are: to summarize, our contributions are three-fold: (1) we pioneer applying self-determination theory to analyze decision-making mechanisms of llms. ## experiments we conduct experiments under three conditions. in the independent condition, we focus on the attention bias (3.1). under the si condition, we explore how social influence affects llms' choice behavior (3.2.1). under the induced-si condition, we examine the choice behavior of llms when the intrinsic bias and extrinsic social influence are in conflict (3.2.2). finally, we investigate the significance of attention bias and social influence on behavioral pattern of llms at the mechanistic level (3.3). ## discussion exploring the roots of attention bias identified in our research-model architecture, pre-training data, and alignment-offers insights into llms' mechanisms. analyzing distribution of views presents a simple method to quantify bias, suggesting avenues for future research into llms' inputoutput patterns. some social science papers which study the choice behavior of human conduct experiment in setting that similar with ours. their conclusion can be used to compare the similarity of llms and human. for example, an artificial "music market" was created to study user choice @xcite . in this market, people can select music to listen and download. then the authors control the visibility and strength of social bias (indicated by download count). and they found the people choices can be significantly affected by social bias and its strength. this conclusion is consistent with our findings in section 3.2. additionally, our study provides insights into assessing llms' human-like capabilities by examining their behavioral patterns. llms integrate internal and external influences, displaying emergent behaviors that mirror human intrinsic and extrinsic motivations, arising from language distribution in training. this finding advances llms' interpretability research. future research could investigate opaque aspects of training processes, such as pre-training data and fine-tuning datasets, to better understand bias formation. additionally, examining differences between humans and llms could help create ai systems that enhance human abilities rather than replicate weaknesses. the degree of human-like phenomena in llms, such as the matthew effect, warrants further exploration to understand discrepancies and implications for ai development. ## conclusion building on psychological methodologies, we introduce a qa platform to study llms' behavioral patterns. by observing the accumulation process of attention metrics across different questions and topics, and quantifying their relationship with selfreported intrinsic and extrinsic factors, we identify patterns of social behavior in llms that resemble human behavior and provide an internal mechanism-based explanation. in summary, our study offers valuable insights for future efforts to deepen the understanding of llms' behavioral pat-terns, guide alignment and fine-tuning processes, and establish stronger and more robust ai.