Teaching Machine Comprehension with Compositional Explanations (2020)

Abstract: Advances in machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated examples in the form of (question, paragraph, answer) triples. In contrast, humans are typically able to generalize with only a few examples, relying on deeper underlying world knowledge, linguistic sophistication, and/or simply superior deductive powers. In this paper, we focus on “teaching” machines reading comprehension, using a small number of semi-structured explanations that explicitly inform machines why answer spans are correct. We extract structured variables and rules from explanations and compose neural module teachers that annotate instances for training downstream MRC models. We use learnable neural modules and soft logic to handle linguistic variation and overcome sparse coverage; the modules are jointly optimized with the MRC model to improve final performance. On the SQuAD dataset, our proposed method achieves 70.14% F1 score with supervision from 26 explanations, comparable to plain supervised learning using 1,100 labeled instances, yielding a 12x speed up.

## introduction recent advances in neural sequence learning and pre-trained language models yield strong (humanlevel) performance on several reading comprehension datasets @xcite . however, state-of-the-art results mainly rely on large-scale annotated corpora, which are often timeconsuming and costly to collect @xcite . this often leads to a large gap between methods in the research settings and practical use cases, as large amounts of annotated data rarely exist for a new task or a low-resource domain @xcite . to reduce this dependency on annotation efforts, we seek to improve the efficiency in obtaining and applying human supervision. one strength of human cognition is the ability to generalize from relatively few examples; shown only a few instances of a problem and solution, humans often deduce patterns more readily than a machine, typically by bringing to bear a wealth of background information about what "really matters" in each example @xcite @xcite . this ability to quickly abstract "deduction rules" is the inspiration for this work, and we aim to gather these rules in the form of semi-structured explanations. in this paper, we focus on the extractive machine reading comprehension (mrc) task, where the system is given a query and is asked to identify an answer span from a particular paragraph. previous work soliciting explanations as part of the annotation process has been limited to classification tasks @xcite . however, mrc is more challenging, since it lacks explicit anchor words (e.g., subject and object in relation extraction), has no pre-defined set of labels, and there is sparser coverage for each explanation. q: what is the atomic number for zinc? c: … zinc is a chemical element with symbol zn and atomic number 30. … a: 30 explanation: x is atomic number. y is zinc. the question contains "number" so the answer should be a number. the answer is directly after x. "for" is directly before y and directly after x in the question. ## problem formulation our goal is to efficiently train an extractive mrc model f, which takes as input a tuple (q, c) of question q and context c, and extracts an answer span a within the context c. we assume a lowresource situation where a large set s of (q, c) pairs (without answer annotation) already exists, but we are limited in time to annotate only a small subset s o (< 200 instances) of s. ## neural module teacher a neural module teacher (nmteacher) acts as a program that tries to answer questions following an explanation. in this section, we introduce the basic modules used for rule execution (sec. 3.1), discuss how variables and rules are obtained from explanations (sec. 3.2), and present how a neural module teacher derives answers (sec. 3.3). fill module: (s ref , p ref , s) → p description: select the span p in a given sentence s that plays the same syntactic role of span p ref in sentence s ref . example: s @xmath0,3] (packet switching) @xmath1,2] (hunting) find module: (q ref , p ref , s) → p description: find the span p in a context sentence s that refers to the span p ref in the question q ref . example: q @xmath2,4] (a promoter sequence) @xmath3... → @xmath4,1] (promoter) compare module: (d 0 , d 1 ) → p description: softly evaluate the statement d 1 ≤ d 0 . description: perform soft logic and to two scalar probabilities. ## conclusion in this paper, we propose to teach extractive mrc with explanations, with a focus on annotation efficiency. we believe explanations stating "why" and justifying "deduction process" opens up a new way to communicate human's generalization abilities to mrc model training. we begin with a small set of semi-structured explanations and compose nmteachers to augment training data. nmteachers are modularized functions where each module has a strict and softened form, enabling broader coverage from each explanation. extensive experiments on different datasets and mrc models demonstrate the efficiency of our system. having achieved encouraging results for mrc, we look forward to extending this framework to tasks such as non-fact-based qa and multi-hop reasoning.