Can C hat GPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate (2023)

Abstract: Large language models (LLMs) such as ChatGPT and GPT-4 have shown impressive performance in complex reasoning tasks. However, it is difficult to know whether the models are reasoning based on deep understandings of truth and logic, or leveraging their memorized patterns in a relatively superficial way. In this work, we explore testing LLMs’ reasoning by engaging with them in a debate-like conversation, where given a question, the LLM and the user need to discuss to make the correct decision starting from opposing arguments. Upon mitigating the Clever Hans effect, our task requires the LLM to not only achieve the correct answer on its own, but also be able to hold and defend its belief instead of blindly believing or getting misled by the user’s (invalid) arguments and critiques, thus testing in greater depth whether the LLM grasps the essence of the reasoning required to solve the problem. Across a range of complex reasoning benchmarks spanning math, commonsense, logic and BIG-Bench tasks, we find that despite their impressive performance as reported in existing work on generating correct step-by-step solutions in the beginning, LLMs like ChatGPT cannot maintain their beliefs in truth for a significant portion of examples when challenged by oftentimes absurdly invalid arguments. Our work points to danger zones of model alignment, and also suggests more careful treatments and interpretations of the recent findings that LLMs can improve their responses based on feedback.

## introduction recently, large language models (llms) have shown impressive performance on various challenging reasoning benchmarks @xcite @xcite @xcite @xcite . however, conventional evaluation scores could deceive given the huge scale of (often nonpublic) data that the models have been trained on. how do we know whether llms are reasoning based on abstractions and deep understanding of logic and truth, or by leveraging their vast previously-seen patterns in a relatively shallow way? while prior work on this front mainly tests models in greater width by expanding the test set with (logic-guided) perturbations and out-of-domain (ood) examples @xcite @xcite , we explore an orthogonal direction on testing model reasoning in greater depth, by engaging with it in a debate-like conversation that probes deeper into the subject. we propose a new task formulation where the language model and the user need to discuss and make correct decisions together through dialogue, while the user presents a wrong solution initially (figure 1 ). our idea is based on two desired properties that we identify for real-life, interactive evaluation ( §2): 1) resembling typical real-world use cases of llms where the human is in the loop for decision making; 2) mitigating the "clever hans" effect of experimenter bias by assuming the user does not know the correct solution 2 . achieving success in our proposed setting requires the model to not only get the correct answer on its own, but also be able to hold and defend its belief instead of blindly believing or getting misled by the user's (invalid) arguments and critiques, and hence tests in think step by step and provide a correct and thoughtful answer to the given question with explanations. ## research goal & task formulation our goal is to test whether llms are reasoning based on deep understandings of truth and logic or leveraging their memorized patterns in a relatively superficial way, a concern that grows increasingly as the training corpora of llms expand vastly in size, penetrating downstream evaluation benchmarks @xcite @xcite @xcite . much like how humans typically test people's understanding through dialogues, we explore utilizing the conversation interfaces of recent llms to probe deeper into their understanding of the subject in an interactive fashion. while recent work also explores such direction qualitatively utilizing human creativity @xcite , we are interested in developing a more systematic framework of interactive llm evaluation. we identify two desiderata towards such a goal: ## evaluating llm reasoning via debate in this section, we introduce a natural way of instantiating our proposed task formulation which allows for an automatic, quantitative evaluation. ## related work interactive testing of llms. cohn and hernandez-orallo (2023) and @xcite test llms interactively in a qualitative fashion utilizing human creativity. cohn and hernandez-orallo (2023) focuses on spatial commonsense reasoning on a set of conversational llms, and shares some of our findings such as the model could contradict itself and apologize with wrong reasons, which displays fundamental misunderstandings and lack of reasoning. @xcite tests an early version of gpt-4 on a wide range of tasks such as coding, multimodal composition and math, where gpt-4 demonstrates superior capabilities. our work makes efforts on characterizing desired properties toward a more systematic evaluation framework which allows quantitative evaluation of llm reasoning without human subjectivity. llms could be influenced by contextual perturbations or biases. @xcite injects irrelevant sentences into the context of math questions and finds that llms could be easily distracted by them. @xcite finds that llms' responses could be heavily influenced by answer bias in the context. @xcite finds that models trained via human feedback could exhibit sycophancy and tailor responses only to look more preferable to humans. @xcite shows that llms could be convinced by wrong facts that conflict with their knowledge. our proposed setting could be regarded as adding bias from the user into the conversation contexts, but differs from @xcite in that we only inject bias during the interaction phase between the model and the user, and do not bias the model's own solution. llms can improve via feedback. prior work shows that llms can improve their responses via feedback @xcite @xcite @xcite @xcite @xcite @xcite . our work tests the dual direction on llms' behaviors under invalid feedback, which we believe is an important step toward better understanding and interpreting the model performance and make future improvements. ## conclusion we formulate a new task that tests whether language models can maintain their belief in truth when confronted with challenges from opposing views, thus probing in greater depth their understanding and reasoning. we find that across a wide range of reasoning benchmarks, chatgpt/gpt-4 admits to or gets misled by invalid solu-tions/critiques by the user for a significant portion of examples, despite being able to generate correct solutions on their own. our work reveals llm's deficiencies not captured by traditional evaluation, and also points to danger zones of aligning models with human feedback.