F orce R eader: a BERT -based Interactive Machine Reading Comprehension Model with Attention Separation (2020)

Abstract: The release of BERT revolutionized the development of NLP. Various BERT-based reading comprehension models have been proposed, thus updating the performance ranking of reading comprehension tasks. However, the above BERT-based models inherently employ BERT’s combined input method, representing the input question and paragraph as a single packed sequence, without further modification for reading comprehension. This paper makes an in-depth analysis of this input method, proposes a problem of this approach. We call it attention deconcentration. Accordingly, this paper proposes ForceReader, a BERT-based interactive machine reading comprehension model. First, ForceReader proposes a novel solution called the Attention Separation Representation to respond to attention deconcentration. Moreover, starting from the logical nature of reading comprehension tasks, ForceReader adopts Multi-mode Reading and Interactive Reasoning strategy. For the calculation of attention, ForceReader employs Conditional Background Attention to solve the lack of the overall context semantic after the separation of attention. As an integral model, ForceReader shows a significant improvement in reading comprehension tasks compared to BERT. Moreover, this paper makes detailed visual analyses of the attention and propose strategies accordingly. This may be another argument to the explanations of the attention.

## introduction reading comprehension can be used as a comprehensive test task to evaluate machine understanding of human language @xcite . therefore, it has become a primary research topic in natural language processing. in the past few years, due to new datasets, algorithms, and computing capabilities, deep learning technology has evolved rapidly. driven by deep learning approaches, machine reading comprehension has made significant breakthroughs. hermann et al. @xcite of deepmind proposed cnn/daily mail, which known as one of the first largescale supervised training datasets for machine reading comprehension. they also proposed the attentive reader @xcite , an attention-based long short-term memory network @xcite model for reading comprehension. in order to overcome the shortage of cnn/daily mail dataset, researchers such as rajpurkar proposed the squad1.0 @xcite dataset in 2016, and squad2.0 @xcite dataset in 2019. these two datasets have rapidly driven a series of studies on deep learning-based machine reading comprehension, resulting in many efficient and ingenious models. @xcite proposed the stanford attentive reader. this end-to-end reading comprehension model combines multi granular language knowledge and attentional mechanisms, become an early representative of neural machine reading comprehension models. @xcite proposed a bidirectional attention flow reading model, bidaf. based on the bilstm backbone network that inherits from stanford attentive reader, bidaf novelly proposes a bi-directional attention interaction, which is not only from query to context but also from context to query. qanet proposed by @xcite in 2018, adopting a completely different approach. qanet is no longer base on bilstm but uses a multi-layer convolution and self-attention mechanism as the encoding blocks, which is similar to the transformer @xcite that appeared later. the introduction of bert in 2018 @xcite pushed the whole nlp society to a new height. the reading comprehension task also achieved a significant breakthrough, reaching the human-level performance on the squad and other datasets. bert is entirely based on the self-attention mechanism of the transformer stacking structure. however, when dealing with the reading comprehension task, it concat the question and paragraph into a single sequence. although very simple and effective, we argue that this approach may cause attention deconcentration. despite all the arguments about the interpretability of attention @xcite @xcite , we still start from the visual distribution of attention, propose an interactive reading comprehension model based on the idea of separate representation of the attention. specifically, our main contributions in this paper are as follows. according to the experiment results, our model obtains a significant improvement compared to the bert. ## attention deconcentration machine reading comprehension requires a machine to answer question q based on a given paragraph p . bert handles this task by encoding the q and p into a single sequence of words as the input. then, it performs the classification task only on the output fragment corresponding to the context. although bert shows excellent performance, we argue that there are problems with this approach. first, based on the common sense of reading comprehension, in order to effectively reason, we need to have an accurate understanding of the question and paragraph's semantics. however, bert's joint-input method may let the semantic of one section affected by the words of the other. these interfering words are often not helpful for understanding. second, it is difficult for the self-attentive mechanism to accurately distinguish the question and paragraph pair(< q, @xmath0, thereby establishing the appropriate bi-directional attention between words of question and paragraph. to better understand these problems, we give a brief example of question-paragraph-answer triple for further discussion as follows. q: i want to ask you a question, who is the founder of microsoft? p: i know from the book that bill gates founded microsoft. a: bill gates (1) in the example1, we can easily tell that when we try to understand the q, we only need to pay attention to few words such as who, founder, and microsoft. words like i, know, book, in p do not play big roles in this particular q&a scenario. therefore, a basic understanding is that when modeling this @xmath1, @xmath2, the reading comprehension model should pay more attention to the more important words, while less critical words do not need much attention. however, the self-attention mechanism, finetuned from the bert pre-trained model, will always assign attention to all words in p . for the second problem, we can see that when reasoning interactively with @xmath3, @xmath4, word like founder in q should pay more attention to the wordsbill, gates, founded and microsoft in p , than to the words you, question, is and the in it's own section. however, in the present bert based approaches, attention is always given to these words. we call this problem attention deconcentration. compare with the masked language modelling, and next sentence prediction task in bert pre-training, the supervised training of the machine reading comprehension task is extremely insufficient. hence, we think it is impossible to eliminate attention deconcentration through adequate training. ## forcereader model we give the overall structure of the forcereader model, which we proposed, in fig. 2 , including attention separate representation, multi-mode reading, conditional background attention, and interactive reasoning. ## conclusion in this paper, we analyzed the use of bert in machine reading comprehension. we proposed the attention deconcentration problem and conducted a detailed analysis of the impact on reading comprehension. in response to this problem, we have proposed an efficient and straightforward model that integrates attention separate representation, multi-mode reading, conditional background attention, and interactive reasoning. attention separate representation effectively solves the problem of attention deconcentration. multi-mode reading, conditional background attention, and interactive reasoning can make the model better adapt to the high interaction of reading comprehension. the experimental results proved the contribution of each improvement. besides, we visually analyzed the attention and intuitively demonstrated the learning ability and interpretability of our model. this work is a new approach for machine reading comprehension and provides a new argument for the interpretability of the transformer model.