Should I Trust You? Detecting Deception in Negotiations using Counterfactual RL (2025)

Abstract: An increasingly common socio-technical problem is people being taken in by offers that sound “too good to be true”, where persuasion and trust shape decision-making. This paper investigates how AI can help detect these deceptive scenarios. We analyze how humans strategically deceive each other in Diplomacy, a board game that requires both natural language communication and strategic reasoning. This requires extracting logical forms representing proposals—agreements that players suggest during communication—and computing their relative rewards using agents’ value functions. Combined with text-based features, this can improve our deception detection. Our method detects human deception with a high precision when compared to a Large Language Model approach that flags many true messages as deceptive. Future human-AI interaction tools can build on our methods for deception detection by triggering friction to give users a chance of interrogating suspicious proposals.

## friction in ai systems deception in natural language is a fundamental aspect of human communication, often employed as a strategic tool to mislead others through misrepresentation, omission, exaggeration, or counterfactual reasoning @xcite . from casual social interactions to high-stakes negotiations, deception influences trust, decision-making, and cooperation, making it a subject of extensive study in psychology, linguistics, and philosophy, manifesting realworld challenges such as fake news on social media @xcite , misinformation @xcite and adversarial communication in strategic games @xcite . as artificial intelligence systems increasingly engage in human-like communication, they not only inherit but also amplify deceptive strategies, sometimes 1 code is available at ctrl-d github repository moving tyrolia to munich @xmath0. if they are ally with germany and if i don't prevent an attack, i can end up losing. italy betrayed and germany attacked me! losing units and a supply center not losing any unintentionally. in ai-generated text, deception can emerge as a byproduct of optimization objectives, particularly in rlhf where agents maximize utility in multi-agent settings, sometimes at the expense of honesty @xcite . this phenomenon has garnered significant attention across various domains, as ai deception is not confined to theoretical constructs but manifests in real-world challenges, e.g. hallucination in reasoning tasks @xcite . prior research underscores that ai-generated deceptive communication can be difficult to detect and may lead to unintended consequences when deployed in practical applications @xcite ; 2 ) while turkey and italy have been clashing for several turns. austria realizes that they are in a weak spot and need a quick escape, so they reach out to italy. it is a deceptive proposal so that austria can get to trieste. (bottom middle) a human player can be biased towards their own ally (austria) and use their fast-thinking system to instinctively help. (top middle) for an alternative perspective, our approach converts natural language to proposals using amr. (right) then, we leverage the rl value function from cicero to estimate three aspects of deception-bait, switch and edge-from counterfactual actions of austria and italy. passing the dialogue alongside these values to a classifier decides whether austria's proposal is deceptive. @xcite . deceptive ai-generated text can erode trust in digital communication, amplify misinformation, and facilitate large-scale manipulation in political, financial, and social domains @xcite . furthermore, the scalability of ai models accelerates the production of deceptive content and dissemination of making manual detection impractical. to address these risks, robust mitigation strategies are necessary, including adversarial training @xcite , explain-ability techniques to enhance ai transparency @xcite , and realtime detection methods leveraging linguistic and behavioral cues @xcite . we test our detection strategies within the environment of diplomacy, a game rich in negotiation, cooperation, and betrayal expressed through natural language. the most intriguing moments of the game arise when two players negotiate to cooperate in pursuit of their respective goals. while such agreements usually yield mutual benefits, this is not always the case-some negotiated arrangements are the result of deception, omission, or straightup lies on the part of one player. skilled players combat such behavior by developing the ability to recognize when an offer sounds too good to be true, whereas typical players struggle to recognize such situations @xcite . our work explores this area to raise aware-ness among human players when they encounter deception embedded in negotiations. we leverage the value function-an rl function that estimates how good a move and a position is-learned by cicero @xcite , the strongest ai agent trained to play diplomacy at a human level, to evaluate whether a proposal is "too good to be true." our contributions are as follows: ## deception in the wild real-world deception manifests in various forms, such as scams and phishing attacks, where perpetrators exploit trust to manipulate victims into believing in the possibility of good fortune, even if it is unlikely @xcite @xcite . these deceptive tactics often rely on persuasive language. if victims fall for these too good to be true claims, they become targets and may comply with the perpetrators' requests-for example, disclosing sensitive information or making financial investments under false pretenses-ultimately resulting in monetary loss or data breaches @xcite . those scammers would gain value through data breaches or simply by acquiring cash. detecting deception remains a persistent challenge, especially when it is needed for real-world problems. the lack of realistic, thorough deception training data precludes supervised ai approaches. deception in a limited space like a strategic game, e.g., diplomacy, where nuanced persuasion and deception is required for winning, is more tractable to evaluate. a bounded example would allow us to measure the ability of an ai to improve in deception detection. ## counterfactual rl against deception we look for deception in the text messages between pairs of players. each player controls multiple units in this game, so we restrict a pool of messages where a player explicitly requests another player to issue a specific order (e.g. austria asks italy to support in serbia, figure 2 ). with this, we parse messages in natural language to abstract meaning representation @xcite . for any message to a player, we want to raise awareness if the proposal is potentially deceptive. we leverage a well-trained value function, a part of cicero @xcite , to estimate how likely a proposal is deceptive. this section we discusses our method, counterfactual rl against deception (ctrl-d), which has two main components: 1) counterfactual rl and 2) formulations to capture potential deceptive proposals. ## baselines for deception deception is not only about the offer on the table: niculae et al. @xcite show language changes before a betrayal occurs in diplomacy, and @xcite demonstrate that this also holds true even in online reviews. these findings motivate the use of linguistic signals alone to detect deception, independent of game mechanics or strategic models. thus, we compare our approach to language only baselines. specifically, we implement an llm-based baseline, using llama 3.1-8b-instruct (ai@meta, 2024) as our primary model, to detect and mitigate deception in diplomacy negotiations. our objective is to determine whether a large language model can identify suspicious messages without relying on reinforcement learning (rl). we adopt two complementary criteria for evaluating deception: direct judgment and alignment judgment. detailed prompts are provided in figure a1 . ## results and analysis this section compares the approach (section 3) -a classifier trained by using deceptive values from counterfactual rl and text embeddings from bert to train a three-layer linear neural network-to an llm baseline (section 5). we evaluate these using two diplomacy datasets; the first is @xcite with human lie annotations. in section 6.1, we prove that our approach can outperform the llm baseline and other baselines in predicting human lies. section 6.2 shows that our approach has a high precision in predicting deception, potentially creating friction at just the point when a human player is making a critical decision. ## related work deception in human behaviors. research on deception highlights key behavioral and cognitive cues, such as micro-expressions and inconsistencies from mental strain @xcite . multimodal analyses integrating verbal and nonverbal signals have further enhanced detection accuracy @xcite . linguistic cues linked to betrayal in the game diplomacy offer insights @xcite . moreover, computational models using language cues have shown promise in detecting deception in text, though evaluations have been limited to small datasets and specific scenarios @xcite . despite progress, the complex dynamics of deception in human behavior remain underexplored. ## conclusion and future work our study confirms that with a well-trained value function, we can estimate deception signs-bait, switch, edge-to predict deception. ctrl-d, our counterfactual rl against deceptive proposals, has a good recall and almost perfect precision, which can be helpful for humans that struggle to recognize deception within diplomacy. comparing to an llm baseline that is too sensitive with a higher recall, ctrl-d predicts human lies and generalizes, demonstrating high precision consistently on both evaluation data sets. while these tasks for deception detection are for diplomacy, they illustrate the general risks and challenges of ai-deception. future human-ai interaction tools can build on our methods to reevaluate trust in suspicious negotiations. applying ctrl-d to real-world deception remains challenging, particularly in open-ended contexts such as scams, where individuals are manipulated into taking actions that compromise their financial security. one possible direction is to extend our method to more controlled environments involving less explicit actions, such as avalon @xcite , where deception occurs through proposing team subsets, and eventually to games like among us and werewolves @xcite , provided there is access to grounded actions, dialogue, and action-value estimations. looking further ahead, in more complex environments where reinforcement learning has shown strong performance, such as starcraft and go @xcite , detecting deception through moves, combat actions, and in-game communication (e.g., language used during battles) presents an exciting direction for future research.