Analytical Reasoning of Text (2022)

Abstract: Analytical reasoning is an essential and challenging task that requires a system to analyze a scenario involving a set of particular circumstances and perform reasoning over it to make conclusions. However, current neural models with implicit reasoning ability struggle to solve this task. In this paper, we study the challenge of analytical reasoning of text and collect a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016. We analyze what knowledge understanding and reasoning abilities are required to do well on this task, and present an approach dubbed ARM. It extracts knowledge such as participants and facts from the context. Such knowledge are applied to an inference engine to deduce legitimate solutions for drawing conclusions. In our experiments, we find that ubiquitous pre-trained models struggle to deal with this task as their performance is close to random guess. Results show that ARM outperforms pre-trained models significantly. Moreover, we demonstrate that ARM has better explicit interpretable reasoning ability.

## introduction transformer-based pre-trained language models including bert @xcite , @xcite and roberta @xcite have achieved state-of-the-art performance on a variety of nlp tasks @xcite @xcite . however, they still struggle to perform deep reasoning beyond shallow-level semantic understanding of literal clues. for example, @xcite show that pre-trained models fail completely on half of eight reasoning tasks that require symbolic operations. we hope to challenge current systems and take a step further towards analytical reasoning. from the passage, we first understand conditions (i.e., participants and positions) and comprehend rules and facts. then, we check each option to see whether it satisfy all the rules and select the most plausible one. ## related works there is an increasing trend on machine reasoning research in recent years. the reasoning ability investigated are partitioned into several major aspects, including (1) logical reasoning; logical reasoning the task of natural language inference (nli) @xcite @xcite @xcite @xcite @xcite requires the models to detect the logical entailment relationship of two sentences. there have been machine reading comprehension (mrc) works @xcite @xcite @xcite that examine the ability of logical reasoning. logiqa @xcite and reclor @xcite are sourced from examination in realistic scenario and examine a range of logical reasoning skills. commonsense reasoning there are many recent benchmarks that assess the commonsense reasoning capabilities from different aspects, like social @xcite , physics @xcite @xcite , or temporal @xcite aspects. there exist several mrc datasets that require commonsense knowledge @xcite @xcite . ## task and dataset in this section, we describe the task of analytical reasoning and introduce the dataset ar-lsat we collected from the law school admission test. ## approach we describe how our system, the analytical reasoning machine (arm), comprehends the knowledge, performs reasoning over the knowledge, and makes conclusions. figure 3 gives an overview of our approach. our system operates in four steps: ## experiments we make experiments on the ar-lsat dataset and evaluate our system with label accuracy. the data split is (train/dev./test) = (1, 585/231/230) we first compare our system with powerful neural baselines and conduct analysis. moreover, case study illustrates the reasoning process of our system by an explicit example. lastly, we make error analysis to point out challenges in this task. ## conclusion in this paper, we study the challenging task of analytical reasoning and introduce a dataset ar-lsat to facilitate research on analytical reasoning. we analyze the knowledge understanding and reasoning ability required for this task and present a system, analytical reasoning machine (arm), which can comprehend the knowledge, including participants, facts and rules mentioned in the context and extract logically equivalent logical functions from the rules. afterwards, it performs deep reasoning to find all the legitimate solutions to the problem posed and finally makes a prediction. experiments show that our system outperforms strong transformer-based baselines, which indicates that knowledge understanding and deep reasoning is essential for this task. results show that this task is very challenging for current neural-based models. wanjun zhong, duyu tang, nan duan, ming zhou, jiahai wang, and jian yin. 2020a. a heterogeneous graph with factual, temporal and logical knowledge for question answering over dynamic contexts. arxiv preprint, abs/2004.12057. wanjun zhong, jingjing xu, duyu tang, zenan xu, nan duan, ming zhou, jiahai wang, and jian yin. 2020b. reasoning over semantic-level graph for fact checking. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 6170-6180, online. association for computational linguistics. ben zhou, daniel khashabi, qiang ning, and dan roth. 2019. "going on a vacation" takes longer than "going for a walk": a study of temporal commonsense understanding. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3363-3369, hong kong, china. association for computational linguistics.