Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension (2021)

Abstract: How can we generate concise explanations for multi-hop Reading Comprehension (RC)? The current strategies of identifying supporting sentences can be seen as an extractive question-focused summarization of the input text. However, these extractive explanations are not necessarily concise i.e. not minimally sufficient for answering a question. Instead, we advocate for an abstractive approach, where we propose to generate a question-focused, abstractive summary of input paragraphs and then feed it to an RC system. Given a limited amount of human-annotated abstractive explanations, we train the abstractive explainer in a semi-supervised manner, where we start from the supervised model and then train it further through trial and error maximizing a conciseness-promoted reward function. Our experiments demonstrate that the proposed abstractive explainer can generate more compact explanations than an extractive explainer with limited supervision (only 2k instances) while maintaining sufficiency.

## introduction recent approaches to multi-hop reading comprehension (rc) have greatly improved its explainability, models ability to explain their own answers @xcite . some adopt a pipelined architecture, where they generate an explanation first and then use it to answer the question. this "faithful-by-construction" approach is aimed at ensuring that generated explanations are closer to the systems' internal reasoning (i.e. faithfulness). the explanation generation step is typically formulated as a sentence selection task over the input text -selecting a set of sentences which provide support for the answer output by the model @xcite . however, the main problem with these approaches is that the explanations obtained from the sentence selection tasks are not always minimal, sufficient, and comprehensible. the extractive explanations can include extraneous or superfluous texts which express information that is not necessary for answering questions. for example, as shown in fig. 1 (a) , the fragments such as 2007 british-american fantasy adventure and young tommy in "never let me go" are not needed to explain the answer northern lights. secondly, the extractive explanations may also not be sufficient: the interpretation of explanations may be dependent on its original paragraphs (e.g. pronouns). in fig. 1 (a), his film roles means charles rowe's film, but this is not included in the extractive explanation. these types of gaps can also limit comprehensibility of the explanations. ## related work explainable nlp three aspects of explainability have been explored @xcite ): (i) comprehensibility to humans @xcite , (ii) faithfulness, correlation with systems' internal decision @xcite , (iii) conciseness, namely minimality, comprehensibility and sufficiency for solving an end task @xcite . earlier approaches to explainable nlp focus on comprehensibility @xcite , and then the community moves towards ensuring faithfulness by a system's architecture (faithful by construction), ranging from natural language inference @xcite , fact verification @xcite to question answering @xcite @xcite . conciseness, in contrast, has been relatively unexplored. one exception is @xcite , who propose to learn to extract a minimal set of input sentences that are useful for solving downstream tasks by imposing information bottleneck on the nlp framework. although our work shares the similar spirit with their work, unlike our work, their explainer is extractive. our work is the first to incorporate abstractive explainers into rc systems. to date, more nlp datasets are being annotated with explanations @xcite , but most of them are based on extractive explanations @xcite . for abstractive explanations, there are a few resources: textual entailment dataset @xcite , and question answering dataset in non-rc settings (i.e. input paragraphs are not given) @xcite . as for rc, @xcite annotate hot-potqa @xcite with abstractive explanations, but only 2k of them (i.e. 3% of the whole dataset) are annotated. abstractive explainer a similar pipeline model has been proposed for textual entailment @xcite and commonsense qa @xcite , where the model first generates an explanation, and then the downstream classifier consumes it to predict a task label. although the architecture is the same as ours, the training process is different: they train the explainer in a fully supervised manner using input-explanation pairs, while our work additionally leverages a signal from the downstream qa model in rl. as demonstrated in §5.5, we show that this additional training is crucial when few annotated explanations are available. generating abstractive explanations is closely related to query-focused summarization (qfs), where a few datasets are publicly available @xcite @xcite . however, the task setting of qfs is radically different from our problem setting, which makes it difficult to leverage the datasets and models in a straightforward manner. the qfs task typically consists of non-question queries (e.g. keywords or complex sentences) or opinion-oriented questions (e.g. is x a good idea?), and gold summaries are not guaranteed to contain all information required for answering questions. we leave it the future work to explore how to effectively use their datasets and models in our task. definition 1. an explanation is concise if it is (i) minimal, (ii) comprehensible, and (iii) sufficient for answering the question. fig. 1 summarizes the overall architecture. to ensure the faithfulness of explanations, we use a pipeline architecture consisting of two main components: (i) an abstractive explainer (ax) and (ii) qa module (qam) ( §3.1). the ax takes a question and paragraph as inputs and is responsible for generating a question-focused, abstractive summary of input paragraphs. the qam then answers the question solely based on the generated summary. this summary is supposed to contain information necessary for answering questions and is the only factor that the qam relies on. thus, the generated summary can be interpreted as a faithful explanation of the model. ## conclusions we have proposed suqa, an rc system augmented with an abstractive explainer component. our experiments have demonstrated that the abstractive explainer can generate more concise explanations than an extractive explainer with limited supervison, while keeping explanations sufficient for qa. one limitation of our work is that the qa module is trained separately from the explainer. one can jointly optimize the ax and qam by extending our framework. finally, our abstractive explainer explains what facts were used for answering questions, but does not explain the inference process. it would be an interesting research direction to extend our work by explaining how these facts are combined to arrive at the answer.