Multi-Step Inference for Reasoning Over Paragraphs (2020)

Abstract: Complex reasoning over text requires understanding and chaining together free-form predicates and logical connectives. Prior work has largely tried to do this either symbolically or with black-box transformers. We present a middle ground between these two extremes: a compositional model reminiscent of neural module networks that can perform chained logical reasoning. This model first finds relevant sentences in the context and then chains them together using neural modules. Our model gives significant performance improvements (up to 29% relative error reduction when combined with a reranker) on ROPES, a recently-introduced complex reasoning dataset.

## introduction performing chained inference over natural language text is a long-standing goal in artificial intelligence @xcite . this kind of inference requires understanding how natural language statements fit together in a way that permits drawing conclusions. this is very challenging without a formal model of the semantics underlying the text, and when polarity needs to be tracked across many statements. for instance, consider the example in figure 1 from ropes @xcite , a recently released reading comprehension dataset that requires applying information contained in a background paragraph to a new situation. to answer the question, one must associate each category of flowers with a polarity for having brightly colored petals, which must be done by going through the information about pollinators given in the situation and linking it to what was said about pollinators and brightly colored petals in the background paragraph, along with tracking the polarity of those statements. prior work addressing this problem has largely either used symbolic reasoning, such as markov logic networks @xcite and integer linear programming @xcite , or blackbox neural networks @xcite . symbolic methods give some measure of interpretability and the ability to handle logical operators to track polarity, but they are brittle, unable to handle the variability of language. neural networks often perform better on practical datasets, as they are more robust to paraphrase, but they lack any explicit notion of reasoning and are hard to interpret. we present a model that is a middle ground between these two approaches: a compositional model reminiscent of neural module networks that can perform chained logical reasoning. the proposed model is able to understand and chain together free-form predicates and logical connectives. the proposed model is inspired by neural module networks (nmns), which were proposed for visual question answering @xcite . nmns assemble a network from a collection of specialized modules where each module performs some learnable function, such as locating a question word in an image, or recognizing relationships between objects in the image. the modules are composed together specific to what is asked in the question, then executed to obtain an answer. we design general modules that are targeted at the reasoning necessary for ropes and compose them together to answer questions. we design three kinds of basic modules to learn the neuro-symbolic multi-step inference over questions, situations, and background passages. the first module is called select, which determines which information (in the form of spans) is essential to the question; the second module is called chain, which captures the interaction from multiple statements; the last one is called predict, which assigns confidence scores to potential answers. the three basic modules can be instantiated separately and freely combined. in this paper, we investigate one possible combination as our multi-step inference on ropes. the results show that with the multi-step inference, the model achieves significant performance improvement. furthermore, when combined with a reranking architecture, the model achieves a relative error reduction of 29% and 8% on the dev and test sets in the ropes benchmark. as ropes is a relatively new benchmark, we also present some analysis of the data, showing that the official dev set is likely better treated as an in-domain test, while the official test set is more of an out-of-domain test set. foot_0 ## model we first describe the baseline system, a typical qa span extractor built on roberta @xcite , and then present the proposed system with multi-step inference. furthermore, we introduce a reranker with multi-step inference given the output of the baseline system. following the standard usage of roberta, we concatenate the background, the situation and question with two special determiners [s:] and [q:] to be a long passage @xmath0:] s [sep] [sep] [q:] q [sep], where the background b and situation s are regarded as the first segment and the question q is the second segment, and [cls] and [sep] are the reserved tokens in roberta. ## data bias in ropes we experiment with ropes @xcite , a recently proposed dataset which focuses on complex reasoning over paragraphs for document comprehension. we noticed a very severe drop in performance between the ropes dev and test sets during initial experiments, and we performed an analysis of the data to figure out the cause. ropes used an annotator split to separate the train, dev, and test sets in order to avoid annotator bias @xcite , but we discovered that this led to a large distributional shift between train/dev and test, which we explore in this section. in light of this analysis, we recommend treating the dev set as an in-domain test set, and the original test set as an out-of-domain test. answer types our analysis is based on looking at the syntactic category of the answer phrase. we use the syntactic parser of @xcite to obtain constituent trees for the passages in ropes. we take the constituent label of the lowest subtree that covers the answer span foot_2 as the answer type. the four most frequent answer types in ropes are noun phrase (np), verb phrase (vp), adjective phrase (adjp) and adverb phrase (advp). ## experiments in this section, we evaluate the performance of our proposed model relative to baselines on ropes. section 3), we additionally set up an experiment using the dev set as an in-domain test set, by partitioning the training set into train (9,824 questions) and train-dev (1,100 questions). training following the settings of prior work @xcite , we fine-tune the roberta-large pre-trained transformer. the hidden sizes of all layers are set to 1024 which is the same to the output dimension of roberta-large, and the number of heads on multi-step attentions is 8. all the models share the same hyperparameters that are shown in ## related work neural module networks were originally proposed for visual question answering tasks @xcite , and recently have been used on several reading comprehension tasks @xcite @xcite , where they specialize the module functions such as find and compare to retrieve the relevant entities with or without supervised signals for hot-potqa @xcite or drop @xcite . as ropes is quite different from these datasets, the modules that we choose to use are also different, focusing on chained inference. multi-hop reasoning there are several datasets constructed for multi-hop reasoning e.g. hotpotqa @xcite @xcite @xcite , qangaroo @xcite @xcite and wikihop @xcite @xcite which aims to get the answer across the documents. the term "multi-hop" reasoning on these datasets is similar to relative information retrieval, where one entity is bridged to another example 1 background: ... for many of the works, the price goes up as the edition sells out... ## conclusion we propose a multi-step reading comprehension model that performs chained inference over natural language text. we have demonstrated that our model substantially outperforms prior work on ropes, a challenging new reading comprehension dataset. we have additionally presented some analysis of ropes that should inform future work on this dataset. while our model is not a neural module network, as our model uses a single fixed layout instead of different layouts per question, we believe there are enough similarities that future work could explore combining our modules with those used in other neural module networks over text, leading to a single model that could perform the necessary reasoning for multiple different datasets.