The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020)

Abstract: Large transformer-based language models have been shown to be very effective in many classification tasks. However, their computational complexity prevents their use in applications requiring the classification of a large set of candidates. While previous works have investigated approaches to reduce model size, relatively little attention has been paid to techniques to improve batch throughput during inference. In this paper, we introduce the Cascade Transformer, a simple yet effective technique to adapt transformer-based models into a cascade of rankers. Each ranker is used to prune a subset of candidates in a batch, thus dramatically increasing throughput at inference time. Partial encodings from the transformer model are shared among rerankers, providing further speed-up. When compared to a state-of-the-art transformer model, our approach reduces computation by 37% with almost no impact on accuracy, as measured on two English Question Answering datasets.

## introduction recent research has shown that transformer-based neural networks can greatly advance the state of the art over many natural language processing tasks. efforts such as bert @xcite , roberta @xcite , xlnet @xcite , and others have led to major advancements in several nlp subfields. these models are able to approximate syntactic and semantic relations between words and their compounds by pre-training on copious amounts of unlabeled data @xcite . then, they can easily be applied to different tasks by just fine-tuning them on training data from the target domain/task @xcite . the impressive effectiveness of transformer-based neural networks can be partially attributed to their large number of parameters (ranging from 110 million for "base" models to over 8 billion @xcite ); however, this also makes them rather expensive in terms of computation time and resources. being aware of this problem, the research community has been developing techniques to prune unnecessary network parameters @xcite or optimize the transformer architecture @xcite . in this paper, we propose a completely different approach for increasing the efficiency of transformer models, which is orthogonal to previous work, and thus can be applied in addition to any of the methods described above. its main idea is that a large class of nlp problems requires choosing one correct candidate among many. for some applications, this often entails running the model over hundreds or thousands of instances. however, it is well-known that, in many cases, some candidates can be more easily excluded from the optimal solution @xcite , i.e., they may require less computation. in the case of hierarchical transformer models, this property can be exploited by using a subset of model layers to score a significant portion of candidates, i.e., those that can be more easily excluded from search. additionally, the hierarchical structure of transformer models intuitively enables the re-use of the computation of lower blocks to feed the upper blocks. following the intuition above, this work aims at studying how transformer models can be cascaded to efficiently find the max scoring elements among a large set of candidates. more specifically, the contributions of this paper are: first, we build a sequence of rerankers sr @xmath0, r 2 , ..., r n } of different complexity, which process the candidates in a pipeline. each reranker at position i takes the set of candidates selected by (i -1)-th reranker and provides top k i candidates to the reranker of position i + 1. by requiring that k @xmath1, . . . , n -1, this approach allows us to save computation time from the more expensive rerankers by progressively reducing the number of candidates at each step. we build r i using transformer networks of 4, 6, 8, 10, and 12 blocks from roberta pre-trained models. second, we introduce a further optimization on sr n to increase its efficiency based on the observation that models r i in sr n process their input independently. in contrast, we propose the cascade transformer (ct), a sequence of rerankers built on top of a single transformer model. rerankers r 1 , . . . , r n are obtained by adding small feedforward classification networks at different transformer block positions; therefore, the partial encodings of the transformer blocks are used as both input to reranker r i , as well as to subsequent transformer encoding blocks. this allows us to efficiently re-use partial results consumed by r i for rankers r i+1 , . . . , r n . to enable this approach, the parameters of all rerankers must be compatible. thus, we trained ct in a multi-task learning fashion, alternating the optimization for different i, i.e., the layers of r i are affected by the back-propagation of its loss as well as by the loss of r j , with @xmath2. finally, as a test case for ct, we target answer sentence selection (as2), a well-known task in the domain of question answering (qa). given a question and a set of sentence candidates (e.g., retrieved by a search engine), this task consists in selecting sentences that correctly answer the question. we tested our approach on two different datasets: (i) asnq, recently made available by @xcite ; and (ii) a benchmark dataset built from a set of anonymized questions asked to amazon alexa. our code, asnq split, and models trained on asnq are publicly available. 1 our experimental results show that: (i) the selection of different k i for sr n determines different trade-off points between efficiency and accuracy. for example, it is possible to reduce the overall computation by 10% with just 1.9% decrease in accuracy. (ii) most importantly, the ct approach largely improves over sr, reducing the cost by 37% with almost no loss in accuracy. (iii) the rerankers trained through our cascade approach achieve equivalent or better performance than transformer models trained independently. finally, (iv) our results suggest that ct can be used with other nlp tasks that require candidate ranking, e.g., parsing, summarization, and many other structured prediction tasks. ## related work in this section, we first summarize related work for sequential reranking of passages and documents, then we focus on the latest methods for as2, and finally, we discuss the latest techniques for reducing transformer complexity. reranking in qa and ir the approach introduced in this paper is inspired by our previous work @xcite ; there, we used a fast as2 neural model to select a subset of instances to be input to a transformer model. this reduced the computation time of the latter up to four times, preserving most accuracy. before our paper, the main work on sequential rankers originated from document retrieval research. for example, @xcite formulated and developed a cascade ranking model that improved both top-k ranked effectiveness and retrieval efficiency. @xcite proposed two stage approaches using a limited set of textual features and a final model trained using a larger set of query-and document-dependent features. @xcite focused on quickly identifying a set of good candidate documents that should be passed to the second and further cascades. @xcite presented a new general framework for learning an end-to-end cascade of rankers using back-propagation. @xcite studied effectiveness/efficiency trade-offs with three candidate generation approaches. while these methods are aligned with our approach, they target document retrieval, which is a very different setting. further, they only used linear models or simple neural models. @xcite focused on as2, but just applied linear models. ## preliminaries and task definition we first formalize the problem of selecting the most likely element in a set as a reranking problem; then, we define sequential reranking (sr); finally, we contextualize as2 task in such framework. ## sr with transformers in this section, we explain how to exploit the hierarchical architecture of a traditional transformer model to build an sr model. first, we briefly recap how traditional transformer models (we refer to them as "monolithic") are used for sequence classification, and how to derive a set of sequential rerankers from a pre-trained transformer model (section 4.1). then, we introduce our cascade transformer (ct) model, a sr model that efficiently uses partial encodings of its input to build a set of sequential rerankers r i (section 4.3). finally, we explain how such model is trained and used for inference in sections 4.3.1 and 4.3.2, respectively. ## experiments we present three sets of experiments designed to evaluate ct. in the first (section 5.3), we show that our proposed approach without any selection produces comparable or superior results with respect to the state of the art of as2, thanks to its stability properties; in the second (section 5.4), we compare our cascade transformer with a vanilla transformer, as well as a sequence of transformer models trained independently; finally, in the third (section 5.5), we explore the tuning of the drop ratio, α. ## conclusions and future work this work introduces ct, a variant of the traditional transformer model designed to improve inference each point corresponds to a configuration of drop ratios {α p1 , . . . , α p4 } with α p @xmath3.1, 0.2, . . . , 0.6}; values on the x-axis represent the relative computational cost per batch of a configuration compared to tanda base . the three runs reported in