Multi-choice Relational Reasoning for Machine Reading Comprehension (2020)

Abstract: This paper presents our study of cloze-style reading comprehension by imitating human reading comprehension, which normally involves tactical comparing and reasoning over candidates while choosing the best answer. We propose a multi-choice relational reasoning (McR ^2 ) model with an aim to enable relational reasoning on candidates based on fusion representations of document, query and candidates. For the fusion representations, we develop an efficient encoding architecture by integrating the schemes of bidirectional attention flow, self-attention and document-gated query reading. Then, comparing and inferring over candidates are executed by a novel relational reasoning network. We conduct extensive experiments on four datasets derived from two public corpora, Children’s Book Test and Who DiD What, to verify the validity and advantages of our model. The results show that it outperforms all baseline models significantly on the four benchmark datasets. The effectiveness of its key components is also validated by an ablation study.

## introduction machine reading comprehension (mrc) is a challenging task that requires much semantic understanding and reasoning using various clues from texts @xcite . its general form is to ask a computer to answer questions in natural language according to its understanding of a given article or a context. as a specific form of mrc, cloze-style reading comprehension has recently gained increasing attention. cloze-style mrc is a task to fill in a blank in the query with an appropriate word or phrase according to given context.several large-scale datasets @xcite @xcite for this task have been released, facilitating the development of various machine learning models @xcite @xcite . most of these learning systems are built upon multi-hop architectures and attention mechanisms @xcite , which have been shown to excel at distilling useful information and learning the "importance" distribution over inputs. there are also some works that mimic the cognitive process of human reasoning with complicated hypothesis testing frameworks @xcite . however, none of these models provide explicit reasoning among candidate answers with respect to a given context and query.as a practical skill for cloze test, humans often need to tactically compare two candidate answers while making decision, especially when more than one candidate appears to be competent. in this paper, we propose a multi-choice relational reasoning (mcr 2 ) model to imitate the above process. it first learns representations of document and query via a hierarchical multi-stage encoding architecture to explore the relations between document and query. the encoding architecture integrates the mechanisms of bidirectional attention flow @xcite , self-attention @xcite and document-gated query reading @xcite to learn dependencies between context and query and map them to representations rich in semantic information. the model then utilizes a multi-choice relational reasoning module to realize comparing and inferring over candidates. our reasoning module is inspired from the relational networks @xcite which have recently demonstrated a very success in several other relational reasoning tasks than mrc. to our knowledge, this work is the first attempt in cloze-style mrc to explicitly perform reasoning over candidate answers to facilitate answer deduction. we conducted extensive experiments on four datasets derived from two public corpora, children's book test and who did what, to verify the validity and advantages of our model. the experimental results confirm its outperformance over other state-of-the-art models. in addition, an ablation study also validates the effectiveness of its key components in contrast to other alternatives, and a case study with visualization further proves the effect of its relational reasoning module. finally, by time analysis we show the advantage of our multi-stage encoding architecture for training. ## overview the inputs of cloze-style reading comprehension can be represented as a tuple (d, q, c, a), where @xmath0, ..., w m ] is a context document of length m, @xmath1, .., q n ] represents a query of length n with a placeholder, and @xmath2, .., c k ] is a set of k candidates. as shown in figure 1 , the framework of our model can be divided into two parts, namely, a fusion representation module and a multi-choice relational reasoning module. while the first module aims to produce fusion representations for document and query by means of a hierarchical multi-stage architecture, the second module to perform comparing and reasoning over candidates based on the fusion representations. in the following sections, we first introduce the fusion representation, and then present the multi-choice relational reasoning module. ## fusion representation taking into consideration a number of previous works @xcite @xcite , this module is designed to consist of four layer, which will be described in detail in the following paragraphs. it should be noted that this module is replaceable in our model, because our model is modular and applicable to many other encoding options, as we will discuss in the experiments section. ## multi-choice relational reasoning this module mainly derives inspiration from the relational networks @xcite which have demonstrated its success in several relational reasoning tasks. the details of this module can be described in three steps: aggregating, comparing, and inferring. ## additional features in addition to the features introduced above, there are also some other useful features in previous works. among them, two are incorporated into our model. firstly, like @xcite , we also find the question evidence common word feature (qe-comm) helpful to further boost the performance of our model. a qe-comm is defined by a one-hot vector f @xmath3, 1} 2 for each document token w i , indicating whether it is also present in the query. the qe-comm embedding qe i for w i can be obtained by qe @xmath4, where @xmath5. specifically, we obtain g i by concatenating g i and qe i for token w i . accordingly, the original inputs @xmath6. secondly, character-level word embeddings have been widely used to alleviate the problem of modeling out-of-vocabulary (oov) tokens @xcite . let @xmath7, a 2 , ..., a l ] be a word in the document or query, where a i is a character of the word. to compute the character-level word embedding of token w, we first map each character a i into a continous representation by means of a similar process as word embedding, then pass these continous representations through a bi-gru, and finally take the final output in forward direction as character-level word embedding. ## experiments and results in this section, we first introduce the datasets used for extensive experiments and then report the results. ## related work existing approaches to cloze-style reading comprehension can be categorized into the following categories in terms of the methodologies used to generate answers. ## conclusion in this paper, we have proposed and tested a novel multi-choice relational reasoning (mcr 2 ) model for cloze-style reading comprehension. mcr 2 is built upon effective hierarchical multi-stage representations of document and query, a novel pointer-sum vector layer to aggregate representations of candidate answers, and relational reasoning on candidate answers to piece together evidence from all candidates. our experiments on several benchmark datasets show that this model compares favorably against state-of-theart models. an ablation study also confirms the effectiveness of its key components and a time analysis further provides evidence for its efficiency. in essence, mcr 2 adopts an extract-then-reason framework. although introduced for cloze-style reading comprehension task in this paper, it can be applied to many other tasks for which interaction among multiple candidates is needed to deal with a single query.