Don’t Read Too Much Into It: Adaptive Computation for Open-Domain Question Answering (2020)

Abstract: Most approaches to Open-Domain Question Answering consist of a light-weight retriever that selects a set of candidate passages, and a computationally expensive reader that examines the passages to identify the correct answer. Previous works have shown that as the number of retrieved passages increases, so does the performance of the reader. However, they assume all retrieved passages are of equal importance and allocate the same amount of computation to them, leading to a substantial increase in computational cost. To reduce this cost, we propose the use of adaptive computation to control the computational budget allocated for the passages to be read. We first introduce a technique operating on individual passages in isolation which relies on anytime prediction and a per-layer estimation of early exit probability. We then introduce SKYLINEBUILDER, an approach for dynamically deciding on which passage to allocate computation at each step, based on a resource allocation policy trained via reinforcement learning. Our results on SQuAD-Open show that adaptive computation with global prioritisation improves over several strong static and adaptive methods, leading to a 4.3x reduction in computation while retaining 95% performance of the full model.

## introduction open-domain question answering (odqa) requires a system to answer questions using a large collection of documents as the information source. in contrast to context-based machine comprehension, where models are to extract answers from single paragraphs or documents, it poses a fundamental technical challenge in machine reading at scale @xcite . most odqa systems consist of two-stage pipelines, where 1) a context retriever such as bm25 @xcite or dpr @xcite first selects a small subset of passages that are likely to contain the answer to the question, and 2) a machine reader such as bert @xcite then examines the retrieved contexts to extract the answer. this two-stage process leads to a computational trade-off that is indicated in fig. 1 . we can run computationally expensive deep networks on a large number of passages to increase the probability that we find the right answer ("all layers, all passages"), or cut the number of passages and layers to reduce the computational footprint at the possible cost of missing an answer ("6 layers, top-2 passages"). we hypothesise that a better accuracy-efficiency trade-off can be found if the computational budget is not allocated statically, but based on the complexity of each passage, see "adaptive computation" in fig. 1 . if a passage is likely to contain the answer, allocate more computation. if it isn't, allocate less. the idea of conditioning neural network computation based on inputs has been pursued in previous work on adaptive computation @xcite @xcite , however how to apply this idea to odqa is still an open research question. in this work, we introduce two adaptive computation methods for odqa: towerbuilder and skylinebuilder. towerbuilder builds a tower, a composition of transformer layers on a single passage, until an early stopping condition is met-we find that this method already helps reducing the computational cost required for reading the retrieved passages. then, for coordinating the construction of multiple towers in parallel, we introduce a global method, skylinebuilder, that incrementally builds multiple towers one layer at a time and learns a policy to decide which tower to extend one more layer next. rather than building single transformer towers in isolation, it constructs a skyline of towers with different heights, based on which passages seem most promising to process further. our experiments on the squad-open dataset show that our methods are very effective at reducing the computational footprint of odqa models. in particular, we find that skylinebuilder retains 95% of the accuracy of a 24-layer model using only 5.6 layers on average. in comparison, an adaptation of the method proposed by @xcite requires 9 layers for achieving the same results. improvements are even more substantial for smaller number of layers-for example, with an average of 3 layers skylinebuilder reaches 89% of the full performance, whereas the approach of @xcite yields 57% and a model trained to use exactly 3 layers reaches 65%. finally, skylinebuilder retains nearly the same accuracy at full layer count. to summarise, we make the following contributions: 1) we are the first to explore adaptive computation for odqa by proposing two models: towerbuilder and skylinebuilder; 2) we experimentally show that both methods can be used for adaptively allocating computational resources so to retain the predictive accuracy with a significantly lower cost, and that coordinating the building of multiple towers via a learned policy yields more accurate results; 3) when compared to their non-adaptive counterparts, our proposed methods can reduce the amount of computation by as much as 4.3 times. ## background we first give an overview of odqa and the relevant work in adaptive computation. ## adaptive computation in odqa our goal is to incrementally build up towers of transformer layers for all passages in d q in a way that minimises unnecessary computation. our algorithms maintain a state, or skyline, @xmath0, a), consisting of current tower heights @xmath1, . . . , h n ), indicating how many layers have been processed for each of the n towers, and the last representations @xmath2, . . . , a n ) computed for each of the towers. we want to build up the skyline so that we reach an accurate solution fast and then stop processing. ## discussions and future works in this paper, we focus on reducing the number of layers and operations of odqa models, but the actual latency improvement also depends on the hardware specifications. on gpus we cannot expect a reduction in the number of operations to translate 1:1 to lower execution times, since they are highly optimised for parallelism. 3 we leave the parallelism enhancements of skylinebuilder for future work. we also notice that the distillation technique is complementary to the adaptive computation methods. it will be interesting to integrate these two approaches to achieve further computation reduction for odqa models. ## conclusions in this work we show that adaptive computation can lead to substantial efficiency improvements for odqa. in particular, we find that it is important to allocate budget dynamically across a large number of passages and prioritise different passages according to various features such as the probability that the passage has an answer. our best results emerge when we learn prioritisation policies using reinforcement learning that can switch between exploration and exploitation. on our benchmark, our method achieves 95% of the accuracy of a 24-layer model while only needing 5.6 layers on average.