N egotiation T o M : A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding (2024)

Abstract: Large Language Models (LLMs) have sparked substantial interest and debate concerning their potential emergence of Theory of Mind (ToM) ability. Theory of mind evaluations currently focuses on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations, which lacks evaluation of machine ToM ability in real-world human interaction scenarios. This poses a pressing demand to develop new real-world scenario benchmarks. We introduce NegotiationToM, a new benchmark designed to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states (i.e., desires, beliefs, and intentions). Our benchmark builds upon the Belief-Desire-Intention (BDI) agent modeling theory and conducts the necessary empirical experiments to evaluate large language models. Our findings demonstrate that NegotiationToM is challenging for state-of-the-art LLMs, as they consistently perform significantly worse than humans, even when employing the chain-of-thought (CoT) method.

## introduction theory of mind (tom) was introduced as an agent's capacity to infer the mental states of others, such as desires, beliefs, and intentions @xcite . numerous scenarios involving human cognition and social reasoning rely on the tom modeling of others' mental states @xcite @xcite , such as comprehending and forecasting others' actions @xcite , planning over others' beliefs and subsequent actions @xcite , and various forms of reasoning and decision-making @xcite . some previous research believes that llms already exhibit a high 1 the dataset is available at https://github.com/ hkust-knowcomp/negotiationtom agent 1: hi there.. we're looking forward to this camping trip.. hope you are, too! is there anything in particular you guys need /want out of the extra stockpile we've got? agent 2: hello! which item do you need the most? agent 1: well, we actually really need all three items.. but i know you might, too. so, i want to make a deal that's fair for you.. is there anything in particular your family needs for the trip? agent 2: well, there has to be an item that you need the most. if you tell me what you need the most, we can try to make a deal around it. agent 1: since the forest is nearby enough, i think we'd be more interested in ensuring there's enough food instead of firewood for my people, i think. agent 2: hmm. i really need food too. i don't care much for water either. how about i take all 3 firewood, 2 food, and 1 water? agent 1: yeah, that's not going to work for me.. i can give you two food if you really need it, but in exchange i'm going to need two of the water and all the firewood. agent 2: i will just take 3 firewood, 1 food, and 1 water. that work for you? you're close to the forest, you won't need firewood. ## related work theory of mind benchmarks the existing tom evaluation benchmarks for large language models are primarily synthetic template-based data generated @xcite or derived from the sally-anne false belief test @xcite @xcite @xcite , which assesses model ability from a theoretical perspective and may inherently suffer from shortcuts and spurious correlations @xcite @xcite . other works, such as @xcite build benchmarks based on the faux pas test @xcite . the most related work to ours is the bigtom benchmark proposed by @xcite , which presents a framework for designing a tom benchmark from synthetic templates for evaluating different aspects of llms' tom capabilities (e.g., desire and belief). however, this work and other theoretical benchmarks may not reflect the tom capabilities of large language models in real-world scenarios. moreover, most of these prior works are concentrated on the belief aspects of the theory of mind. therefore, this work introduces negotiationtom, which is a multi-category mental state benchmark in realistic negotiation scenarios. negotiation negotiation is an expanding area of research in the natural language processing field, and @xcite conducted an impressive survey of existing literature on dialogue systems for negotiation. @xcite train recurrent neural networks to generate natural language dialogues in negotiations. @xcite proposed a modular generative model that is based on dialogue acts. various disciplines have explored bilateral bargaining from diverse perspectives and employing different methodologies. economic theory has examined the influence of incomplete information @xcite and emphasized the significance of explicit communication @xcite . @xcite and pruitt (2013) present a comprehensive overview of the psychology research on negotiation. these previous studies generally neglect the content of communication, although there are a few noteworthy exceptions @xcite @xcite @xcite . one intriguing work by @xcite introduces a probabilistic formulation method to encapsulate the opponent's personality type during learning and inference, drawing inspiration from the idea of incorporating a theory of mind (tom) into machines. however, distinct from this approach, our work presents a benchmark integrating a theory of mind (tom) into the negotiation surroundings. ## negotiationtom theory of mind (tom) describes the ability as humans have to ascribe and infer the mental states of others, and to predict which likely actions they are going to take @xcite . therefore, it is critical to acquire negotiation strategies based on one's own desire and temporary belief built upon the in-formation presented in conversation. nevertheless, understanding the theory of mind (tom) inherent in a negotiation dialogue is challenging due to its intricate linguistic features and complex reasoning attributes. therefore, some considerations have to be taken into account when constructing the nego-tiationtom.