It Takes Two to Lie: One to Lie, and One to Listen (2020)

Abstract: Trust is implicit in many online text conversations—striking up new friendships, or asking for tech support. But trust can be betrayed through deception. We study the language and dynamics of deception in the negotiation-based game Diplomacy, where seven players compete for world domination by forging and breaking alliances with each other. Our study with players from the Diplomacy community gathers 17,289 messages annotated by the sender for their intended truthfulness and by the receiver for their perceived truthfulness. Unlike existing datasets, this captures deception in long-lasting relationships, where the interlocutors strategically combine truth with lies to advance objectives. A model that uses power dynamics and conversational contexts can predict when a lie occurs nearly as well as human players.

## introduction a functioning society is impossible without trust. in online text interactions, users are typically trusting @xcite , but this trust can be betrayed through false identities on dating sites @xcite , spearphishing attacks @xcite , sockpuppetry @xcite and, more broadly, disinformation campaigns @xcite . beyond such one-off antisocial acts directed at strangers, deception can also occur in sustained relationships, where it can be strategically combined with truthfulness to advance a long-term objective @xcite . we introduce a dataset to study the strategic use of deception in long-lasting relationships. to collect reliable ground truth in this complex scenario, we design an interface for players to naturally generate and annotate conversational data while playing a negotiation-based game called diplomacy. ## diplomacy the diplomacy board game places a player in the role of one of seven european powers on the eve of world war i. the goal is to conquer a simplified map of europe by ordering armies in the field against rivals. victory points determine the success of a player and allow them to build additional armies; the player who can gain and maintain the highest number of points wins. 1 the mechanics of the game are simple and deterministic: armies, represented as figures on a given territory, can only move to adjacent spots and the side with the most armies always wins in a disputed move. the game movements become publicly available to all players after the end of a turn. because the game is deterministic and everyone begins with an equal amount of armies, a player cannot win the game without forming alliances with other players-hence the name of the game: diplomacy. conquering neighboring territories depends on support from another player's armies. after an alliance has outlived its usefulness, a player often dramatically breaks it to take advantage of their erstwhile ally's vulnerability. ## engaging a community of liars this dataset requires both a social and technical setup: finding a community that plays diplomacy online and having them use a framework for annotating these messages. ## detecting lies we build computational models both to detect lies to better understand our dataset. the data from the user study provide a training corpus that maps language to annotations of truthfulness and deception. our models progressively integrate information-conversational context and in-game power dynamics-to approach human parity in deception detection. receiver's perception truth lie sender's intention truth straightforward salut! just checking in, letting you know the embassy is open, and if you decide to move in a direction i might be able to get involved in, we can probably come to a reasonable arrangement on cooperation. bonne journee! cassandra i don't care if we target t first or a first. i'll let you decide. but i want to work as your partner. . . . i literally will not message anyone else until you and i have a plan. i want it to be clear to you that you're the ally i want. lie deceived you, sir, are a terrific ally. this was more than you needed to do, but makes me feel like this is really a long term thing! thank you. caught so, is it worth us having a discussion this turn? i sincerely wanted to work something out with you last turn, but i took silence to be an ominous sign. ## qualitative analysis this section examines specific messages where both players and machines are correctly identifying lies and when they make mistakes on our test set. most messages are correctly predicted by both the model and players (2055 of 2475 messages); but this is because of the veracity effect. the picture is less rosy if we only look at messages the sender marks as actual lie: both players and models are generally wrong ( both models and players can detect lies when liars get into specifics. in diplomacy, users must agree to help one another through orders that stipulate "i will help another player move from x to y". the in-game term for this is "support"; half the messages where players and computers correctly identify lies contain this word, but it rarely occurs in the other quadrants. models seem to be better at not falling for vague excuses or fantastical promises in the future. players miss lies that promise long-term alliances, involve extensive apologies, or attribute motivation as coming from other countries' disinformation (model correct). unlike our models, players have access to conversations with other players and accordingly players can detect lies that can easily be verified through conversations with other players (player correct). however, ultimately most lies are believable and fool both models and players (both wrong). for example, all messages that contain the word "true" are predicted as truthful by both models and play- model prediction correct wrong player prediction correct both correct not sure what your plan is, but i might be able to support you to munich. player correct don't believe turkey, i said nothing of the sort. i imagine he's just trying to cause an upset between us. wrong model correct long time no see. sorry for the stab earlier. i think we should try to work together to stop france from winning; if we work together we can stop france from getting 3 more centers, and then we will all win in a 3, 4, or 5 way draw when the game is hard-capped at 1910. both wrong i'm considering playing fairly aggressive against england and cutting them off at the pass in 1901, your support for that would be very helpful. finally, we investigate categories from the harbingers @xcite word lists. lies are more likely to contain subjectivity and premises while true messages include expansion phrases ("later", "additionally"). we also use specific words in the bag of words logistic regression model. the coefficient weights of words that express sincerity (e.g., "sincerely", "frankly") and apology (e.g., "accusation", "fallout", "alternatives") skew toward actual lie prediction in the logistic regression model. more laid back appella-tions (e.g., "dude", "man") skew towards truthfulness, as do words associated with reconnaissance (e.g., "fyi","useful", "information") and time (e.g., "weekend", "morning"). contested areas on the diplomacy map, such as budapest and sevastopol, are more likely to be associated with lies, while more secure ones like berlin, are more likely to be associated with truthful messages. ## related work early computational deception work focuses on single utterances @xcite , especially for product reviews @xcite . but deception is intrinsically a discursive phenomenon and thus the context in which it appears is essential. our platform provides an opportunity to observe deception in the context in which it arises: goaloriented conversations around in-game objectives. gathering data through an interactive game has a cheaper per-lie cost than hiring workers to write deceptive statements @xcite . other conversational datasets are mostly based on games that involve deception including werewolf @xcite , box of lies @xcite , and tailor-made games @xcite . however, these games assign individuals roles that they maintain throughout the game (i.e., in a role that is supposed to deceive or in a role that is deceived). thus, deception labels are coarse: an individual always lies or always tells the truth. in contrast, our platform better captures a more multifaceted reality about human nature: everyone can lie or be truthful with everyone else, and they use both strategically. hence, players must think about every player lying at any moment: "given the evidence, do i think this person is lying to me now?" deception data with conversational labels is also available through interviews @xcite , some of which allow for finer-grained deception spans @xcite . compared with game-sourced data, however, interviews provide shorter conversational context (often only a single exchange with a few follow-ups) and lack a strategic incentive-individuals lie because they are instructed to do so, not to strategically accomplish a larger goal. in diplomacy, users have an intrinsic motivation to lie; they have entertainmentbased and financial motivations to win the game. this leads to higher-quality, creative lies. real-world examples of lying include perjury @xcite , calumny @xcite , emails from malicious hackers @xcite , and surreptitious user recordings. but real-world data comes with realworld complications and privacy concerns. the artifice of diplomacy allows us to gather pertinent language data with minimal risk and to access both sides of deception: intention and perception. other avenues for less secure research include analyzing dating profiles for accuracy in self-presentation (toma and hancock, 2012) and classifying deceptive online spam @xcite . ## conclusion in dante's inferno, the ninth circle of hell-a fate worse even than that reserved for murderers-is for betrayers. dante asks count ugolino to name his betrayer, which leads him to say: but if my words can be the seed to bear the fruit of infamy for this betrayer who feeds my hunger, then i shall speak-in tears @xcite similarly, we ask victims to expose their betrayers in the game of diplomacy. the seeds of players' negotiations and deceit could, we hope, yield fruit to help others: understanding multi-party negotiation and protecting internet users. while we ignore nuances of the game board to keep our work general, diplomacy is also a rich, multi-agent strategic environment; paquette et al. (2019) ignore diplomacy's rich language to build bots that only move pieces around the board. an exciting synthesis would incorporate deception and language generation into an agent's policy; our data would help train such agents. beyond playing against humans, playing with a human in the loop (hitl) resembles designs for cybersecurity threats @xcite , annotation @xcite , and language alteration @xcite . likewise, our lie-detection models can help a user in the moment better decide whether they are being deceived @xcite . computers can meld their attention to detail and nigh infinite memory to humans' grasp of social interactions and nuance to forge a more discerning player. beyond a silly board game, humans often need help verifying claims are true when evaluating health information @xcite , knowing when to take an e-mail at face value @xcite , or evaluating breaking news @xcite . building systems to help information consumers become more discerning and suspicious in low-stakes settings like online diplomacy are the seeds that will bear the fruits of interfaces and machine learning tools necessary for a safer and more robust internet ecosystem. roberto gonzález-ibáñez, smaranda muresan, and nina wacholder. 2011. identifying sarcasm in twitter: a closer look. in proceedings of the association for computational linguistics. stephan greene and philip resnik. 2009. more than words: syntactic packaging and implicit sentiment. in conference of the north american chapter of the association for computational linguistics. naeemul hassan, fatma arslan, chengkai li, and mark tremayne. 2017. toward automated factchecking: detecting check-worthy factual claims by claimbuster. in knowledge discovery and data mining. david hill. 2014. got your back. this american life podcast. shuyuan mary ho, jeffrey t hancock, and cheryl booth. 2017. ethical dilemma: deception dynamics in computer-mediated group communication. journal of the association for information science and technology. sepp hochreiter and jürgen schmidhuber. 1997. long short-term memory. neural computation, 9(8):1735-1780. tom n jagatic, nathaniel a johnson, markus jakobsson, and filippo menczer. 2007. social phishing. communications of the acm, 50(10):94-100. nathalie japkowicz and shaju stephen. 2002. the class imbalance problem: a systematic study. intelligent data analysis, 6(5):429-449. david jurgens and roberto navigli. 2014. it's all fun and games until someone annotates: video games with a purpose for linguistic annotation. in transactions of the association for computational linguistics. mary e kaplar and anne k gordon. 2004. the enigma of altruistic lying: perspective differences in what motivates and justifies lie telling within romantic relationships. personal relationships, 11(4):489-507. srijan kumar, justin cheng, jure leskovec, and v.s. subrahmanian. 2017. an army of me: sockpuppets in online discussion communities. in proceedings of the world wide web conference, republic and canton of geneva, switzerland. srijan kumar and neil shah. 2018. false information on web and social media: a survey. in social media analytics: advances and applications. crc. vivian lai, han liu, and chenhao tan. 2020. "why is 'chicago' deceptive?" towards building modeldriven tutorials for humans. in international conference on human factors in computing systems. timothy r. levine, hee sun park, and steven a. mc-cornack. 1999. accuracy in detecting truths and lies: documenting the "veracity effect". communication monographs, 66(2):125-144. sarah ita levitan, angel maredia, and julia hirschberg. 2018. linguistic cues to deception and perceived deception in interview dialogues. in conference of the north american chapter of the association for computational linguistics. max louwerse, david lin, amanda drescher, and gun semin. 2010. linguistic cues predict fraudulent events in a corporate social network. in proceedings of the annual meeting of the cognitive science society. james edwin mahon. 2016. the definition of lying and deception. in the stanford encyclopedia of philosophy, winter 2016 edition. metaphysics research lab, stanford university. matthew l newman, james w pennebaker, diane s berry, and jane m richards. 2003. lying words: predicting deception from linguistic styles. personality and social psychology bulletin, 29(5):665-675. vlad niculae, srijan kumar, jordan boyd-graber, and cristian danescu-niculescu-mizil. 2015. linguistic harbingers of betrayal: a case study on an online strategy game. in proceedings of the association for computational linguistics. daniela oliveira, harold rocha, huizi yang, donovan ellis, sandeep dommaraju, melis muradoglu, devon weir, adam soliman, tian lin, and natalie ebner. 2017. dissecting spear phishing emails for older vs young adults: on the interplay of weapons of influence and life domains in predicting susceptibility to phishing. in international conference on human factors in computing systems. myle ott, claire cardie, and jeff hancock. 2012. estimating the prevalence of deception in online review communities. in proceedings of the world wide web conference. myle ott, yejin choi, claire cardie, and jeffrey t hancock. 2011. finding deceptive opinion spam by any stretch of the imagination. in proceedings of the association for computational linguistics. bo pang, lillian lee, et al. 2008. opinion mining and sentiment analysis. foundations and trends in information retrieval, 2(1-2):1-135. philip paquette, yuchen lu, seton steven bocco, max smith, satya ortiz-gagné, jonathan k. kummerfeld, joelle pineau, satinder singh, and aaron c courville. 2019. no-press diplomacy: modeling multi-agent gameplay. in proceedings of advances in neural information processing systems. curran associates, inc. verónica pérez-rosas, mohamed abouelenien, rada mihalcea, yao xiao, c. j. linton, and mihai burzo. 2016. verbal and nonverbal clues for real-life deception detection. in proceedings of empirical methods in natural language processing.