Towards building a Robust Industry-scale Question Answering System (2020)

Abstract: Industry-scale NLP systems necessitate two features. 1. Robustness: “zero-shot transfer learning” (ZSTL) performance has to be commendable and 2. Efficiency: systems have to train efficiently and respond instantaneously. In this paper, we introduce the development of a production model called GAAMA (Go Ahead Ask Me Anything) which possess the above two characteristics. For robustness, it trains on the recently introduced Natural Questions (NQ) dataset. NQ poses additional challenges over older datasets like SQuAD: (a) QA systems need to read and comprehend an entire Wikipedia article rather than a small passage, and (b) NQ does not suffer from observation bias during construction, resulting in less lexical overlap between the question and the article. GAAMA consists of Attention-over-Attention, diversity among attention heads, hierarchical transfer learning, and synthetic data augmentation while being computationally inexpensive. Building on top of the powerful BERTQA model, GAAMA provides a ∼2.0% absolute boost in F1 over the industry-scale state-of-the-art (SOTA) system on NQ. Further, we show that GAAMA transfers zero-shot to unseen real life and important domains as it yields respectable performance on two benchmarks: the BioASQ and the newly introduced CovidQA datasets.

## introduction a relatively new task in open domain question answering (qa) is machine reading comprehension (mrc), which aims to read and comprehend a given text and then answer questions based on it. recent work on transfer learning, from large pre-trained language models like bert @xcite and xlnet @xcite has practically solved squad @xcite , the most widely used mrc benchmark. this necessitates harder qa benchmarks for the field to advance. additionally, squad and other existing datasets like narrativeqa @xcite and hotpotqa @xcite suffer from observation bias: annotators had read the passages before creating their questions. in industry research, there is an urgent demand to build a usable mrc qa system that not only provides very good performance on academic benchmarks but also real life industry applications @xcite in a zstl environment. in this paper, to build such a system, we first focus on natural questions (nq) @xcite : a mrc benchmark dataset over wikipedia articles where questions (see figure 1 ) were sampled from google search logs. this key difference from past datasets eliminates annotator observation bias. also, nq requires systems to extract both a short (sa, one or more entities) and a long answer (la, typically a paragraph that contains the short answer when both exist). the dataset shows human upper bounds of 76% and 87% on the short and long answer selection tasks respectively (for a "super-annotator" composed of 5 human annotators). the authors show that systems designed for past datasets perform poorly on nq. we propose gaama that possesses several mrc technologies that are necessary to perform well on nq and achieve significant boosts over another industry setting competitor system @xcite pre-trained on a large language model (lm) and then over millions of synthetic examples. specifically, gaama builds on top of a large pre-trained lm and focusses on two broad dimensions: this need for improved alignment leads us to explore two additional attention mechanisms. ## related work most recent mrc systems either achieve sota by adding additional components on top of bert @xcite such as syntax @xcite or perform attention fusion @xcite without using bert. however, we argue that additional attention mechanisms should be explored on top of bert such as computing additional cross-attention between the question and the passage and maximizing the diversity among different attention heads in bert. our work is also generic enough to be applied on recently introduced transformer based language models such as albert @xcite and reformer @xcite . another common technique is da @xcite by artificially generating more questions to enhance the training data or in a mtl setup @xcite @xcite . @xcite combine models of question generation with answer extraction and filter results to ensure round-trip consistency to get the sota on nq. contrary to this, we explore several strategies for da that either involve diverse question generation from a dynamic nucleus @xcite of the probability distribution over question tokens or shuffling the existing dataset to produce adversarial examples. recently @xcite focus on "open" nq, a modified version of the full nq dataset for document retrieval qa that discards unanswerable questions. contrary to that, we specifically focus on the full nq dataset and believe there is room for improvement from a mrc research standpoint. ## model architecture in this section, we first describe bert qa , gaama's underlying qa model, and two additional attention layers on top of it. figure 2 shows our overall model architecture with details explained below. ## model training our models follow the now common approach of starting with the pre-trained bert language model and then finetune over the nq dataset with an additional qa sequence prediction layer as described in section 3.1. note that unless we specify otherwise, we are referring to the pre-trained "large" version of bert with whole word masking (bert w ). bert w has the same model structure as the original bert model, but masks whole words instead of word pieces for the masked language model pre-training task and we empirically find this to be a better starting point for the nq task. ## analysis of gaama's components the new collective bargaining agreement (cba) formulated in 2011 had an initial salary cap of @xmath034.6 million. both the cap and... the later sentence contains the correct answer, 1994, since the question is asking for when the salary cap was initially adopted. the sos augmented model correctly makes this prediction whereas our squad 2 augmented models predict 2011 from the earlier sentence. there are also cases where the correct answer span appears in the middle or later part of a paragraph and, though our squad 2 augmented models predict the spans correctly, they assign a lower score (relative to its optimal threshold) than the sos augmented model. the position bias, therefore, appears to hurt the performance of the system in certain situations where location of the answer span relative to the paragraph is not a useful signal of correctness. on average, of course, the bert w +squad2 pt + aoa + ad configuration performs the best and manual error analysis indicates some ability to better attend to supporting evidence when it is further out from the correct answer span. for example, the correct answer in example 1 from figure 1 is december 24, 1836 which the aoa + ad model correctly identifies the answer span despite the question's and context's lack of lexical and grammatical alignment. while the base bert w models fail at extracting the date (instead predicting a span more closely associated with the keywords in the query such as seven years earlier). ## conclusion although large pre-trained language models have shown super-human performance on benchmark datasets like squad, we show that there is plenty of room to make improvements on top of bert qa . specifically, we outline prior strategies that do not work on a real benchmark consisting of "natural 7 @xcite notes that human performance was measured on a random sample of nq dev.