M a P : A Matrix-based Prediction Approach to Improve Span Extraction in Machine Reading Comprehension (2020)

Abstract: Span extraction is an essential problem in machine reading comprehension. Most of the existing algorithms predict the start and end positions of an answer span in the given corresponding context by generating two probability vectors. In this paper, we propose a novel approach that extends the probability vector to a probability matrix. Such a matrix can cover more start-end position pairs. Precisely, to each possible start index, the method always generates an end probability vector. Besides, we propose a sampling-based training strategy to address the computational cost and memory issue in the matrix training phase. We evaluate our method on SQuAD 1.1 and three other question answering benchmarks. Leveraging the most competitive models BERT and BiDAF as the backbone, our proposed approach can get consistent improvements in all datasets, demonstrating the effectiveness of the proposed method.

## introduction machine reading comprehension (mrc), which requires the machine to answer comprehension questions based on the given passage of text, has been studied extensively in the past decades @xcite . due to the increase of various large-scale datasets (e.g., squad @xcite and ms marco @xcite ), and the enhancement of pre-trained models (e.g., elmo @xcite , bert @xcite , and xlnet @xcite ), remarkable advancements have been made recently in this area. among various mrc tasks, span extraction is one of the essential tasks. given the context and question, the span extraction task is to extract a span of the most plausible text from the corresponding context as a in our setting, every start (or end) position has an end (or start) probability vector, which leads that the output probabilities is a matrix (best seen in color). the conditional approach empirically has an advantage over the independent approach. however, the output distributions of the previous conditional approaches are two probability vectors. it ignores some more possible start-end pairs. as an extension, every possible start (or end) position should have an end (or start) probability vector. thus, the output conditional probabilities is a matrix. we propose a matrix-based prediction approach (map) based on the above consideration in this paper. as figure 1 shown, the key point is to consider as many probabilities as possible in training and inference phases. specifically, we calculate a conditional probability matrix instead of a probability vector to expand the choices of start-end pairs. because of more values contained in a matrix than a vector, there is a big challenge in the training phase of the map. that is the high computational cost and memory issues if the input sequence is long. as an instance, the matrix contains 262, 144 probability values if the sequence length is 512. therefore, we propose a sampling-based training strategy to speed up the training and reduce the memory cost. the main contributions of our work are fourfold. ## methodology in this section, we first give the problem definition. then we introduce a typical vector-based conditional approach. next, we mainly introduce our matrix-based conditional approach and samplingbased training strategy. finally, an ensemble approach on both start-to-end and end-to-start directions of conditional probability is discussed. ## experiments in this section, we conduct experiments to evaluate the effectiveness of the proposed map. ## related work machine reading comprehension is an important topic in the nlp community. more and more neural network models are proposed to tackle this problem, including dcn @xcite , r-net @xcite , @xcite , match-lstm @xcite , s-net @xcite , sdnet @xcite , qanet @xcite , has-qa @xcite . among various mrc tasks, span extraction is a typical task that extracting a span of text from the corresponding passage as the answer of a given question. it can well overcome the weakness that words or entities are not sufficient to answer questions @xcite . previous models proposed for span extraction mostly focus on the design of architecture, especially on the representation of question and passage, and the interaction between them. there are few works devoted to the top-level design of span output, which refers to the probabilities generation from the representation. we divide the previous toplevel design into two categories, independent approach and conditional approach. the independent approach is to predict the start and end positions in the given passage independently @xcite . although the independent approach has a simple assumption, it works well when the input features are strong enough, e.g., combining with bert @xcite , xl-net @xcite , and spanbert @xcite . nevertheless, since there is a kind of dependency relationship between start and end positions, the conditional approach has advancements over the independent approach. a typical work on the conditional approach comes from @xcite . they proposed two different models based on the pointer network. one is the sequence model which produces a se-quence of answer tokens as the final output, and another is the boundary model which produces only the start token and the end token of the answer. the experimental results demonstrate that the boundary model (span extraction) is superior to the sequence model on both em and f1. the r-net @xcite , s-net @xcite , sdnet @xcite have the same output layer and inference phase with the boundary model in @xcite . @xcite presented an architecture that builds fixed length representations of all spans in the passage with a recurrent network to address the answer extraction task. the computation cost is decided by the max-length of the possible span and the sequence length. the experimental results show an improvement on em compared with the endpoints prediction that independently predicts the two endpoints of the answer span. however, previous works related to the conditional approach are always based on a probability vector. we investigate another possible matrixbased conditional approach in this paper. besides, a well-matched training strategy is proposed to our approach, and forward and backward conditional possibilities are also integrated to improve the performance. ## conclusion in this paper, we first investigate different approaches of span extraction in mrc. to improve the current vector-based conditional approach, we propose a matrix-based conditional approach. more careful consideration of the dependencies between the start and end positions of the answer span can predict their values better. we also propose a sampling-based training strategy to address the training process of the matrix-based conditional approach. the final experimental results on a wide of datasets demonstrate the effectiveness of our approach and training strategy.