Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension (2023)

Abstract: To precisely evaluate a language model’s capability for logical reading comprehension, we present a dataset for testing the understanding of the rationale behind critical reasoning. For questions taken from an existing multiple-choice logical reading comprehension dataset, we crowdsource rationale texts that explain why we should select or eliminate answer options, resulting in 3,003 multiple-choice subquestions that are associated with 943 main questions. Experiments on our dataset show that recent large language models (e.g., InstructGPT) struggle to answer the subquestions even if they are able to answer the main questions correctly. We find that the models perform particularly poorly in answering subquestions written for the incorrect options of the main questions, implying that the models have a limited capability for explaining why incorrect alternatives should be eliminated. These results suggest that our dataset encourages further investigation into the critical reasoning ability of language models while focusing on the elimination process of relevant alternatives.

## introduction critical reasoning, a type of logical reasoning not tied to formal logic, is a core ability of humans that is required for thoughtful reading of text. it involves not only understanding what a passage explicitly says but also comprehending its underlying assumptions, argument structure, and supported conclusions. developing systems capable of critical reasoning as reliably as humans is one of the ultimate goals of natural language processing. recent studies have proposed datasets that evaluate logical reasoning including critical reasoning ability @xcite in reading comprehension. owing to the recent development of large language models (llms; @xcite , the performance of the stateof-the-art models is nearing that of humans @xcite . ## baseline performance on rule we measure the baseline performance of recent state-of-the-art models on our dataset. because the main purpose of our dataset is to perform an extensive evaluation of the models tested on reclor, we use all of our main questions and subquestions as a test set. our hypothesis is that if the models can effectively generalize to understand the rationale behind the correct answer, they should exhibit a similar degree of performance on both the main questions and subquestions. ## analysis to qualitatively investigate the models' behavior observed in section 4, we aim to answer the following research questions. ## conclusion we construct a dataset to evaluate the models' ability of critical reasoning in logical reading comprehension. we crowdsource free-form rationale for main questions taken from an existing dataset and use an llm to generate subquestion texts. resulting questions ask about the underlying rationales for why a certain answer option should be selected and the others should be eliminated. we find that llms are particularly bad at answering eliminative subquestions, highlighting that those models do not necessarily have the comprehensive ability of critical reasoning. for future work, we will develop a more efficient pipeline for data collection and facilitate better rationale generation by llms.