Self-contradictory reasoning evaluation and detection (2024)

Abstract: In a plethora of recent work, large language models (LLMs) demonstrated impressive reasoning ability, but many proposed downstream reasoning tasks only focus on performance-wise evaluation. Two fundamental questions persist: 1) how consistent is the reasoning, and 2) can models detect unreliable reasoning? In this paper, we investigate self-contradictory (Self-Contra) reasoning, where the model reasoning does not support answers. To answer 1), we define and assess the Self-Contra rate across three datasets and delve into finer-grained categories of Self-Contra reasoning. We find that LLMs often contradict themselves in reasoning tasks involving contextual information understanding or commonsense. The model may generate correct answers by taking shortcuts in reasoning or overlooking contextual evidence, leading to compromised reasoning. For 2), we task the state-of-the-art model GPT-4 with identifying Self-Contra reasoning and finer-grained fallacies. We find that finer-grained aided detection can improve GPT-4’s ability to detect Self-Contra. However, it is only able to detect Self-Contra with a 52.2% F1 score, much lower compared to 66.7% for humans. Our results indicate that current LLMs lack the robustness necessary for reliable reasoning and we emphasize the urgent need for establishing best practices in comprehensive reasoning evaluations beyond pure performance-based metrics.

## introduction large language models (llms) have shown impressive performance in many nlp tasks, such as question answering @xcite , and math reasoning @xcite @xcite . llms can achieve high accuracy on reasoning datasets such as 1 the code and dataset are available at https://github.com/uscnlp-lime/self-contradictory . the driver offered a ride to the teacher but he refused the offer. who is 'he'? ## self-contra reasoning we begin by defining self-contra reasoning and then introduce the methods to probe such problematic reasoning in llms. ## automatic detection previous evaluations rely solely on human annotation. exploring whether models can detect self-contra reasoning could significantly reduce human workload and benefit the community. in this section, we explore the capability of llms in detecting self-contra. ## conclusion our study focuses on self-contra reasoning in llms for question-answering tasks. we conduct self-contra reasoning evaluation across 3 datasets and 3 llms. next, we employ the wino-bias and winogrande datasets for an in-depth analysis. we analyze specific errors, such as models generating incomplete reasoning or using shortcuts, contributing to self-contra reasoning. we also find that llm is still not capable of detecting self-contra reasoning, with a lower performance compared to humans. this work represents the first comprehensive study of self-contra reasoning, offering a nuanced evaluation and a new task-automatic detection of self-contra.