OPERA : Operation-Pivoted Discrete Reasoning over Text (2022)

Abstract: Machine reading comprehension (MRC) that requires discrete reasoning involving symbolic operations, e.g., addition, sorting, and counting, is a challenging task. According to this nature, semantic parsing-based methods predict interpretable but complex logical forms. However, logical form generation is nontrivial and even a little perturbation in a logical form will lead to wrong answers. To alleviate this issue, multi-predictor -based methods are proposed to directly predict different types of answers and achieve improvements. However, they ignore the utilization of symbolic operations and encounter a lack of reasoning ability and interpretability. To inherit the advantages of these two types of methods, we propose OPERA, an operation-pivoted discrete reasoning framework, where lightweight symbolic operations (compared with logical forms) as neural modules are utilized to facilitate the reasoning ability and interpretability. Specifically, operations are first selected and then softly executed to simulate the answer reasoning procedure. Extensive experiments on both DROP and RACENum datasets show the reasoning ability of OPERA. Moreover, further analysis verifies its interpretability.

## introduction machine reading comprehension (mrc) that requires discrete reasoning is a valuable and challenging task @xcite , especially involving symbolic operations such as addition, sorting, and counting. the examples in existing approaches for this task can be roughly divided into two categories: the semantic-parsingbased @xcite and the multi-predictor-based methods @xcite @xcite @xcite . the former maps the natural language utterances into logical forms and then executes them to derive the answers. for example, @xcite propose nerd. it includes a reader to encode the passage and question, and a programmer to generate a logical form for multi-step reasoning. intuitively, this method has an advantage in interpretability. however, semantic parsing over text is nontrivial and even a little perturbation will result in wrong answers, which hinders the mrc performance. to alleviate the heavy dependence on logical forms in the first category, the latter directly employs multiple predictors to derive different types of answers. for example, @xcite and @xcite divide instances of the drop dataset into several types and design a model with multi-predictors to deal with different answer types, i.e., question/passage span(s), count, and arithmetic expression. it is the capability of deriving different types of answers that improves the performance of models. however, such methods are lack of the necessary components to imitate discrete reasoning, which leads to inadequacy in reasoning ability and interpretability. to alleviate the shortcomings of the above methods and preserve their advantages, we attempt to summarize reasoning steps into a set of operations and adopt them as the pivot to connect the question and the answer, which makes it possible to perform discrete reasoning. for example, to answer the question in to verify the effectiveness of the proposed method, comprehensive experiments are conducted on both the drop and racenum datasets, where racenum used in this paper is a subset of the race dataset following @xcite . experimental results indicate that our method outperforms strong baselines and achieves the stateof-the-art on both datasets under the single model setting. we further analyze the interpretability of opera. overall, this paper primarily makes the following contributions: ## related work recently, machine reading comprehension (mrc) methods tend to deal with more practical problems @xcite @xcite , for example, answering complex questions that require discrete reasoning @xcite such as arithmetic computing, sorting, and counting. intuitively, semantic parsing-based methods, which are well explored to deal with discrete reasoning in question answering with structured knowledge graphs @xcite or tables, have potential to address the discrete reasoning mrc problem. therefore, semantic parsing-based methods for discrete reasoning over text are proposed to firstly convert the unstructured text into a table, and then answer questions over the structured table with a grammar-constrained semantic parser @xcite . nerd @xcite ) is a generative model that consists of a reader and a programmer, which are responsible for encoding the context into vector representation and generating grammar-constrained logical forms, respectively. nmns @xcite learned to parse compositional questions as executable logical forms. however, it only adapts to limited question types matched with a few pre-defined templates. multi-predictor-based methods employ multiple predictors to derive different types of answers. naqanet @xcite , a number-aware framework, employed multiple predictors to produce corresponding answer types, including a span, count, and arithmetic expression. based on naqanet, mtmsn @xcite added a negation predictor to solve the negative question and re-ranked arithmetic expression candidates. to aggregate the relative magnitude relation between two numbers, numnet @xcite and num-net+ leveraged a graph convolution network to perform multi-step reasoning over a number graph. qdgat @xcite proposed a questiondirected graph attention network for reasoning over a heterogeneous graph composed of entity and number nodes. evidr @xcite , an evidence- operations description examples addition/diff addition or subtraction how many more yards was kris browns's first field goal over his second? max/min select the maximum/minimum one from given numbers how many yards was the longest field goal in the game? argmax/argmin select key with highest/lowest value from key-value pairs which player had the longest touchdown reception? argmore/argless select key with higher/lower value from two key-value pairs who scored more field goals, david akers or john potter? count count the number of spans how many field goals did kris brown kick? key_value extract key-value pairs how many percent of forth worth households owned a car? span select spans from input sequence which team scored the final td of the game? ## conclusion we propose a novel framework opera for machine reading comprehension requiring discrete reasoning. lightweight and one-shot operations and heuristic rules to map questions to an operation set are systematically designed. opera can leverage the operations to enhance the model's reasoning capability and interpretability. experiments on drop and racenum demonstrate that opera achieves remarkable performance. further visualization and analysis verify its interpretability.