R iki N et: Reading W ikipedia Pages for Natural Question Answering (2020)

Abstract: Reading long documents to answer open-domain questions remains challenging in natural language understanding. In this paper, we introduce a new model, called RikiNet, which reads Wikipedia pages for natural question answering. RikiNet contains a dynamic paragraph dual-attention reader and a multi-level cascaded answer predictor. The reader dynamically represents the document and question by utilizing a set of complementary attention mechanisms. The representations are then fed into the predictor to obtain the span of the short answer, the paragraph of the long answer, and the answer type in a cascaded manner. On the Natural Questions (NQ) dataset, a single RikiNet achieves 74.3 F1 and 57.9 F1 on long-answer and short-answer tasks. To our best knowledge, it is the first single model that outperforms the single human performance. Furthermore, an ensemble RikiNet obtains 76.1 F1 and 61.3 F1 on long-answer and short-answer tasks, achieving the best performance on the official NQ leaderboard.

## introduction machine reading comprehension (mrc) refers to the task of finding answers to given questions by reading and understanding some documents. it represents a challenging benchmark task in natural language understanding (nlu). with the progress of large-scale pre-trained language models @xcite , state-of-the-art mrc models @xcite @xcite @xcite have already surpassed human-level performance on certain commonly used mrc benchmark datasets, such as squad 1.1 @xcite , squad 2.0 @xcite , and coqa @xcite . recently, a new benchmark mrc dataset called natural questions foot_2 (nq) @xcite has presented a substantially greater challenge for the existing mrc models. specifically, there are two main challenges in nq compared to the previous mrc datasets like squad 2.0. firstly, instead of providing one relatively short paragraph for each question-answer (qa) pair, nq gives an entire wikipedia page which is significantly longer compared to other datasets. secondly, nq task not only requires the model to find an answer span (called short answer) to the question like previous mrc tasks but also asks the model to find a paragraph that contains the information required to answer the question (called long answer). in this paper, we focus on the nq task and propose a new mrc model called rikinet tailored to its associated challenges, which reads the wikipedia pages for natural question answering. for the first challenge of the nq task mentioned above, rikinet employs the proposed dynamic paragraph dual-attention (dpda) reader which contains multiple dpda blocks. in each dpda block, we iteratively perform dual-attention to represent documents and questions, and employ paragraph self-attention with dynamic attention mask to fuse key tokens in each paragraph. the resulting context-aware question representation, questionaware token-level, and paragraph-level representations are fed into the predictor to obtain the answer. the motivations of designing dpda reader are: (a) although the entire wikipedia page contains a large amount of text, one key observation is that most answers are only related to a few words in one paragraph; (b) the final paragraph representation can be used naturally for predicting long answers. we describe the details of dpda reader in § 3.1. for the second challenge, unlike prior works on nq dataset @xcite that only predict the short answer and directly select its paragraph as long answer, rikinet employs a multi-level cascaded answer predictor which jointly predict the short answer span, the long answer paragraph, and the answer type in a cascaded manner. another key intuition motivating our design is that even if the relevant documents are not given, humans can easily judge that some questions have no short answers @xcite . take this question as a motivating example:"what is the origin of the nobel prize?" the answer should be based on a long story, which cannot be easily expressed in a short span of entities. therefore we also feed the question representation into the predictor as an auxiliary prior to answer type prediction. the details will be given in § 3.2. on the nq test set, our single model obtains 74.3 f1 scores on the long-answer task (la) and 57.9 f1 scores on the short-answer task (sa) compared to the published best single model @xcite ) results of 66.8 f1 on la and 53.9 f1 on sa. to the best of our knowledge, rikinet is the first single model that outperforms the single human performance @xcite on both la and sa. besides, our ensemble model obtains 76.1 f1 on la and 61.3 f1 on sa, which achieves the best performance of both la and sa on the official nq leaderboard. ## preliminaries before we describe our model in detail, we first introduce the notations and problem formalization. our paper considers the following nq @xcite task: given a natural question q, a related wikipedia page p (in the top 5 search results returned by the google search engine), the model outputs a paragraph within the wikipedia page p as the long answer which contains enough information to infer the answer to the question, and an entity span within the long answer that answers the question as the short answer. also, the short answer of the 1% wikipedia page is "yes" or "no", instead of a short span. both long answers and short answers can be null (i.e., no such answer could be found). given a natural question q and its paired wikipedia page p, we tokenize them with the 30,522 wordpiece vocabulary as used in @xcite . following @xcite , we generate multiple document spans by splitting the wikipedia page with a sliding window. then, we obtain multiple 6-tuple training instances (q, d, c, s, e, t) for each nq data pair (q, p), where q and d are wordpiece ids of question with length n and document span with length m, @xmath0.e, all long answer candidates) within d, s, @xmath1, 1, ..., m -1} are inclusive indices pointing to the start and end of the short answer span, and @xmath2, 1, 2, 3, 4} represents the five answer types, corresponding to the labels "null" (no answer), "short" (has short answer), "long" (only has long answer), "yes", and "no". for each tuple (q, d, c, s, e, t) of the data pair (q, p), rikinet takes d and q as inputs, and jointly predicts c, s, e, t. finally we merge the prediction results of every tuple to obtain the final predicted long answer, short answer, and their confidence scores of the data pair (q, p) for evaluation. ## methodology we propose the rikinet which reads the wikipedia pages for natural question answering. as shown in fig. 1 , rikinet consists of two modules: (a) the dynamic paragraph dual-attention reader as described in §3.1, and (b) the multi-level cascaded answer predictor as described in §3.2. ## conclusion we propose the rikinet, which reads the wikipedia pages to answer the natural question. the rikinet consists of a dynamic paragraph dual-attention reader which learns the token-level, paragraphlevel and question representations, and a multilevel cascaded answer predictor which jointly predicts the long and short answers in a cascade manner. on the natural questions dataset, the rikinet is the first single model that outperforms the single human performance. furthermore, the rikinet ensemble achieves the new state-of-the-art results at 76.1 f1 on long-answer and 61.3 f1 on shortanswer tasks, which significantly outperforms all the other models on both criteria.