ARES : A Reading Comprehension Ensembling Service (2020)

Abstract: We introduce ARES (A Reading Comprehension Ensembling Service): a novel Machine Reading Comprehension (MRC) demonstration system which utilizes an ensemble of models to increase F1 by 2.3 points. While many of the top leaderboard submissions in popular MRC benchmarks such as the Stanford Question Answering Dataset (SQuAD) and Natural Questions (NQ) use model ensembles, the accompanying papers do not publish their ensembling strategies. In this work, we detail and evaluate various ensembling strategies using the NQ dataset. ARES leverages the CFO (Chakravarti et al., 2019) and ReactJS distributed frameworks to provide a scalable interactive Question Answering experience that capitalizes on the agreement (or lack thereof) between models to improve the answer visualization experience.

## introduction machine reading comprension (mrc) involves computer systems that can take a question and some text and produce an answer to that question using the content in that text. this field has recently received considerable attention, yielding popular leaderboard challenges such as squad @xcite and nq @xcite . currently, the top submissions on both the squad and nq leaderboards combine multiple system outputs. these ensembled systems traditionally outperform single models by 1-4 fmeasure. unfortunately, many of the papers for these systems provide little to no information about the ensembling techniques they use. in this work, we use gaama, a prototype question-answering system using the mrc techniques of @xcite , as our starting point and explore how to ensemble multiple mrc models from gaama foot_0 . we evaluate these techniques on the nq short answer task. using our ensemble of models, for each example (question, passage pair), we take the top predictions per system, group by span (answer extracted from the passage), normalize and aggregate the scores, take the mean score across systems for each span, and then take the highest scoring short and long answer spans as our final prediction. these improved ensembling techniques are applied to our mrc systems to produce stronger answers. whereas other systems such as @xcite and allen nlp's foot_1 make use of a single model, we are able to use multiple models to produce a stronger result. we further take advantage of the fact that both the individual model predictions and the ensembed predictions are returned to help increase explainability for the user. for the graphical interface we use a heatmap to show the level of (dis)agreement between the underlying models along with the "best ensemble" answer. an example of this can be seen in figure 1 . ## system overview we describe the architecture of the system and additionally provide an overview of the client (gui) used in this demonstration. the system is composed of mrc and ensembling services which are orchestrated by cfo. the mrc services (in our case gaama) provide reading comprehension via a transformer model @xcite ; multiple services utilizing different model architectures are run to extract answers for a given question and passage. after the mrc services extract their answers, they are all passed to ares which ensembles the results. the ensembling algorithm used by ares is detailed in sections 4 and 5. note that the mrc service only extracts short answers, therefore only those portions of our ensembling approach are used. both the ensembled and original answers are then returned to the caller, allowing the clients to display the final ensembled answers and the original answers they were generated from to the end user. more completely, the system takes the following as input through a grpc @xcite interface: question, passage, minimum confidence score threshold δ, maximum number of answers n , maximum number of answers per model n, and number of models k. these inputs are sent from the client (we discuss our client below) and received by the cfo node which orchestrates the containers. the choice of k is bounded on how many gaama containers are deployed (e.g. if there are 3 then @xmath1, 2, 3}). by tweaking the parameter k, clients can opt for increased accuracy (higher k) or decreased latency (lower k) as when multiple models run on the same gpu the request latency increases. as depicted in figure 2 (where there are 3 mrc models running), each of the @xmath2, returning at most n answers to cfo. these answers, together with their confidence scores, are then sent to the ensembler by cfo which produces at most n ensembled answers (each with confidence score at least δ) and returns them to cfo. finally, both the answers of the k models and the ensembled answers predicted by ares are returned by cfo to the caller. the gui client for our system is based on a reac-tjs 4 web interface. a request is taken as input from the user and sent to the system where is is processed as described above. when an answer with sufficient confidence score is returned, it is displayed to the user as seen in figure 1 . both the ensembled answer and the individual answers are shown together with their respective confidence scores. these answers are also shown in the context of the original passage. the ensembled answer is boxed in. for the individual answers a character heatmap is created representing how many of the candidate answers each character appears in. this heatmap is used to highlight the passage different different colors corresponding to the heatmap (characters not used in any answers are not highlighted). both the boxing and highlighting of answers are done using markjs 5 . note that while these visualizations only show the top answer for each mrc model, n answers per model are ensembled together. if 4 https://reactjs.org/ 5 https://markjs.io/ an answer with sufficient confidence score is not returned, this is relayed to the user through the gui. ## methods we investigate a number of strategies for ensembling models on the nq dataset. we use the nq dataset as it is more realistic and challenging than squad, as its questions were created by google search users prior to seeing the answer documents (so they do not suffer from observational bias). in order to formally compare approaches we partition the nq dev set into "dev-train" and "dev-test" by taking the first three dev files for the "train" set and using the last two for the "test" set (the original dev set for nq is partitioned into 5 files for distribution). this yields "train" and "test" sets of 4,653 and 3,177 examples respectively. for each strategy considered we search for the best k-model ensemble over the "train" set and then evaluate on the "test" set. for these experiments we use @xmath3. we begin by outlining our core strategy that underlies the approaches we have investigated. using this strategy we investigate a baseline approach of ensembling multiple versions of the same model trained with different seeds. we then investigate search strategies for choosing the best models from candidates trained with different hyperparameters, in addition to different normalization and aggregation strategies that are used on a set of candidates. ## experiments and results we examine two types of ensembling experiments: (i) ensembling the same model trained with different seeds and (ii) ensembling different models. ensembling the same model trained on different seeds attempts to smooth the variance to produce a stronger result. on the other hand ensembling different models attempts to find models that may not be the strongest individually but harmonize well to produce strong results. throughout this section we will use sa f 1 and la f 1 to denote the short and long answer performance on "dev-test". similarly we will use n s to indicate the number of models searched for an experiment and types sa and la to indicate optimization for sa and la f1 respectively. ## ensemble candidate contributions when doing manual error analysis on the nq dev set, we do observe patterns suggesting that each of the ensemble components do bring different strengths over the single best model. for example, the wikipedia article for salary cap contains multiple sentences related to the query "when did the nfl adopt a salary cap": the new collective bargaining agreement (cba) formulated in 2011 had an initial salary cap of @xmath034.6 million. both the cap and... the later sentence contains the correct answer, 1994, since the question is asking for when the salary cap was initially adopted. one of our models a correctly makes this prediction whereas another one of our models b predicts 2011 from the earlier sentence. there are also cases where the correct answer span appears in the middle or later part of a paragraph and, though our model b predict the spans correctly, they assign a lower score (relative to its optimal threshold) than the model a. the position bias, therefore, appears to hurt the performance of the system in certain situations where location of the answer span relative to the paragraph is not a useful signal of correctness. finally, in our manual review we do see that the ensemble of these models performs better in the expected ways: (1) boosting scores when multiple models agree on an answer span even though no one model is extremely confident (2) reducing confidence when there is disagreement among models. ## conclusion we introduce a novel concept for a mrc service, ares, which uses an ensemble of models to respond to requests. this provides for multiple advantages over the traditional single model paradigm: improved f1, the ability to control the performance vs runtime tradeoff for each individual request, and improved explaiability of results by showing both candidate answers in addition to the final ensembled answer. we outline several ensembling approaches for question answering models investigated for use in ares and compare their performance on the nq challenge. our findings show that ensembling unique models outperforms ensembling the same model trained with different seeds and provide further analysis to show how ensembling diverse models improves performance. we also show that using unnormalized max aggregation for short answers and logistic regression normalized noisy-or aggregation for long answers yields an f1 improvement of 2 to 4 points over single model performance on the nq challenge.