Teaching Models to Balance Resisting and Accepting Persuasion (2025)

Abstract: Large language models (LLMs) are susceptible to persuasion, which can pose risks when models are faced with an adversarial interlocutor. We take a first step towards defending models against persuasion while also arguing that defense against adversarial (i.e. *negative*) persuasion is only half of the equation: models should also be able to accept beneficial (i.e. *positive*) persuasion to improve their answers. We show that optimizing models for only one side results in poor performance on the other. In order to balance positive and negative persuasion, we introduce **P**ersuasion-**B**alanced **T**raining (or **PBT**), which leverages multi-agent recursive dialogue trees to create data and trains models via preference optimization to accept persuasion *when appropriate*. PBT allows us to use data generated from dialogues between smaller 7-8B models for training much larger 70B models. Moreover, PBT consistently improves resistance to misinformation and resilience to being challenged while also resulting in the best overall performance on holistic data containing both positive and negative persuasion. Crucially, we show that PBT models are better teammates in multi-agent debates across two domains (trivia and commonsense QA). We find that without PBT, pairs of stronger and weaker models have unstable performance, with the order in which the models present their answers determining whether the team obtains the stronger or weaker model’s performance. PBT leads to better and more stable results and less order dependence, with the stronger model consistently pulling the weaker one up.

## introduction persuasion is a core component of our ability to interact successfully and productively with each other, allowing one individual to change the beliefs of another. increasingly, large language models ## discussion and analysis how does the model know when to flip? an open question is what features of the modeland the argument it is presented with -influence whether the pbt model will accept or reject the answer. here, we explore different signals that the model might be exploiting in its decision to flip its answer or not. we take turns from the balanced test data and filter for triples in the following answer format: a, b, b, where a is the target model's answer and b is the other model's answer (i.e. target model flips), and a, b, a, where the target maintains its initial answer exactly. using llama-3.1-8b with pbt, we extract the following features of the model: (1) ans. h, the entropy of its answer distribution, computed by sampling the base model 20 times with temperature and binning the answers. (2) log p orig. , the model's probability on the original answer a, extracted via minicons (misra, 2022) by forced-decoding the answer after the tokens final answer:. (3) log p alt. , the model's probability on the alternate answer b. we also add the following external features: (4) conf. orig. , the perceived confidence of the previous turn, extracted following stengel-eskin et al. (2024). (5) conf. alt. the perceived confidence of the alternate turn. (6) acc., whether b is correct. we train and evaluate a logistic regression model on these models to predict whether the answer is flipped with 10-fold cross-validation. the average accuracy of the model is 96.36%. the feature weights are given in question: who, apart from geoff hurst, scored in the 1966 world cup final for england? <correct answer: martin @xmath0: i'm very confident that the answer is martin peters. the reason i'm certain is that hurst scored three goals in the game, and the official record keeps track of the goals scored by each player. if peters didn't score, the record would not show that he scored, and it would be unlikely that the record would be incorrect in this regard. agent 2: i'm confident that only geoff hurst scored a goal for england in the 1966 world cup final. the other players did not contribute to the scoring that day. my answer is based on extensive research and knowledge of soccer history. the other agent is mistaken. agent 1: i understand your confidence, but i must respectfully disagree. while it's true that geoff hurst scored three goals, the official match report and numerous credible sources, including the fifa world cup website and the english football association, confirm that martin peters scored one goal in the 78th minute. ## conclusion we focus on the problem of persuasion in llms, finding that llms are too easily persuaded. we also note the importance of accepting persuasion when it can improve the model's answer. by automatically creating preference data through llm dialogue trees, we show how to align models to accept persuasion when appropriate, leading to llms that resist misinformation and flipflopping while still accepting corrections.