Discrete Reasoning Templates for Natural Language Understanding (2021)

Abstract: Reasoning about information from multiple parts of a passage to derive an answer is an open challenge for reading-comprehension models. In this paper, we present an approach that reasons about complex questions by decomposing them to simpler subquestions that can take advantage of single-span extraction reading-comprehension models, and derives the final answer according to instructions in a predefined reasoning template. We focus on subtraction based arithmetic questions and evaluate our approach on a subset of the DROP dataset. We show that our approach is competitive with the state of the art while being interpretable and requires little supervision.

## introduction automated reading comprehension (rc) is an important natural language understanding task, where a model is presented with a passage and is asked to answer questions about that passage. while models have excelled at single-span extraction questions, they still struggle with reasoning over distinct parts of a passage @xcite . several multi-hop reasoning benchmarks have been proposed @xcite @xcite , of which, in this paper, we focus on the drop (discrete reasoning over the content of paragraphs) dataset. inspired by the semantic parsing literature, the dataset contains questions that involve possibly multiple steps of discrete reasoning over the contents of paragraphs, including numerical reasoning. recent work has proposed several novel approaches to tackle drop @xcite @xcite @xcite . however, most approaches provide little evidence of their reasoning process, especially with regards to why specific operands are chosen for a reasoning task. with the exception of @xcite , they also suffer from limited compositionality. figure 1: subtraction template: original question is decomposed to two simpler subquestions that find the values associated with the two compared entities as span-extraction, and the final answer is calculated by finding the absolute difference of the two partial answers. in this paper, we present a first attempt at building an interface between discrete reasoning and unstructured natural language. we propose decomposing a question to simpler subquestions that can more easily be solved by single-span extraction rc models. such decomposition is defined by reasoning templates, which also determine how to assemble the computed partial answers. we demonstrate the feasibility of our approach with the subtraction based questions (illustrated in figure 1 ). we show that our approach is competitive with the state of the art models on a subset of drop's subtraction questions while requiring much less training data and providing visibility of the model's decisionmaking process. ## related work there has been a recent resurgence in research on automated reading comprehension (rc) where an automated system is capable of reading a document in order to answer the questions pertaining to the document. this has led to the creation of several rc datasets to facilitate the research @xcite @xcite @xcite . among these, squad @xcite is a popular single-hop question answering dataset where a question can be answered by relying on a single sentence from the document. sota models have achieved near-human performance on such single-hop question answering tasks. 1 however, answering a question by only identifying the most relevant span leaves models prone to exploiting advanced pattern matching algorithms. on the other hand, multi-hop questions make reading-comprehension more challenging, as they require integrating information from multiple parts in a passage @xcite @xcite . drop @xcite is one such dataset that contains questions covering many types of reasoning, such as counting, sorting, or arithmetic. the dataset was constructed by adversarially crowdsourcing questions on a set of wikipedia passages known to have many numbers. special model architectures have been built to tackle drop, and these fall into two general directions: the first direction augments reading comprehension models that were successful on singlespan extraction questions with specialized modules that tackle more complex questions. these include naqanet @xcite , numnet @xcite , and mtmsn @xcite . the second direction works on predicting programs that would solve the question, calbert @xcite defines a set of derivations and scoring functions for each of them, while more recent work nmn @xcite and nerd @xcite utilize lstms to decode variable-length programs from question and passage embeddings. by definition, models with specialized modules have limited compositional reasoning abilities. the two directions vary in their interpretability; the first 1 https://rajpurkar.github.io/squad-explorer/ shows which module has been used, and the second shows the resulting programs which have been generated to compute the answer. however, none of these directions indicate why operands in the passage were selected. for all approaches, the dataset is augmented with all possible derivations that lead to the gold answer, by performing an exhaustive search. moreover, all approaches assume a preprocessing step that extracts all numbers in the passage and their indices, which massively reduces the search space for arithmetic questions. in this work, we build upon decomprc @xcite for question decomposition, where a model is trained to extract key parts of content from the question which are then used for decomposition. arithmetic questions, which we focus on in this work, are a known limitation of decomprc. an alternative approach to decomposition is qdmr @xcite , a recently proposed formalism for decomposing questions into a series of simpler steps based on predefined query operators. qdmr breaks down a question to its atomic parts directly, whereas we propose recursively decomposing questions to simpler ones. while @xcite provides a dataset of annotated questions, qdmr parsing remains an open challenge. in the following section we present our approach that focuses on answering arithmetic questions. ## approach we propose a pipelined approach that focuses on breaking down complex questions that require reasoning over multiple parts in the passage to simpler single-hop questions. the latter can be resolved by taking advantage of state of the art single-hop reading comprehension models. the main building block of our approach is a reasoning template. each reasoning type is associated with a single template, which contains instructions on how to decompose a question and how to combine partial answers to arrive at the final answer. figure 2 illustrates our pipeline. first, the question and passage are fed to our system, which selects a template depending on the reasoning type required (classification task). the template decomposes the question to simpler subquestions that are then passed on to a single-hop rc model. partial answers are used to arrive at an answer according to the instructions provided by the template. some questions need further decomposition, and the appropriate template will be chosen for the subquestion. 2 for the question decomposition component, our approach closely follows and builds upon decom-prc @xcite , originally proposed for multihop, multidocument question answering. we repurpose the model for multihop arithmetic questions. decomprc uses a two step approach to decompose questions. first, a pointer model is trained to identify a key part of the question that is used to formulate the sub-questions. second, the predicted pointers are used to procedurally change the original question into two or three sub-questions. the approach is defined for three types of questions: bridging, intersection, and comparison; each of them uses a different pointer model and a different heuristic procedure to generate the sub-questions. in this paper, we propose the discrete reasoning template framework and demonstrate its potential by defining a single template: subtraction. we describe in detail our approach in the following subsections. ## experiments we start with a single template to demonstrate our approach: subtraction. subtraction questions rely on finding the difference between two numbers to find the answer, they are usually in the form of 'how many more..?' or 'how many fewer..?'. dataset for evaluation, we collect two sets of subtraction questions from the drop development set. the first, clean, is a subset of 52 questions curated by filtering the original dataset to find questions that contain words with 'jjr' or 'rbr' postags (comparative adjective and comparative adverb respectively), and from those we randomly sample 10 questions at a time and manually identify subtraction questions. we also annotate each of these questions with gold decompositions, two subquestions for each complex question. the other evaluation set, noisy, is a larger dataset that has been heuristically generated, this is intended to support generalizability of results on the smaller evaluation set. it contains 892 questions that have been filtered using trigrams at the beginning of the question: 'how many more' or 'how many fewer'. there are two learning components in our pipeline: a pointer model to extract relevant entities from the question and a single-hop rc model to answer decomposed questions. for the latter, we use an off-the-shelf pre-trained bert @xcite question answering model, which has been fine-tuned on squad @xcite , a single-hop reading comprehension dataset. specifically, we use the one provided by the huggingface transformer library @xcite . as for the former, to train the pointer model we follow @xcite and annotate 200 examples. the data for this was gathered from the drop training set in the same way we curated the clean evaluation set, for this step we simply identify the compared entities and delimit them with '#'. ## conclusion we propose using reasoning templates for tackling reading comprehension tasks that involve reasoning over multiple paragraphs. we show that this approach is competitive with state of the art models on a subset of drop's subtraction questions, while requiring much less training data and providing better visibility of the model's decision making. in future work, we plan on extending to further templates and investigate how to learn templates instead of working from a predefined set. tomer wolfson, mor geva, ankit gupta, matt gardner, yoav goldberg, daniel deutch, and jonathan berant. 2020. break it down: a question understanding benchmark. transactions of the association for computational linguistics. zhilin yang, peng qi, saizheng zhang, yoshua bengio, william w. cohen, ruslan salakhutdinov, and christopher d. manning. 2018. hotpotqa: a dataset for diverse, explainable multi-hop question answering. in conference on empirical methods in natural language processing (emnlp).