FASTMATCH : Accelerating the Inference of BERT -based Text Matching (2020)

Abstract: Recently, pre-trained language models such as BERT have shown state-of-the-art accuracies in text matching. When being applied to IR (or QA), the BERT-based matching models need to online calculate the representations and interactions for all query-candidate pairs. The high inference cost has prohibited the deployments of BERT-based matching models in many practical applications. To address this issue, we propose a novel BERT-based text matching model, in which the representations and the interactions are decoupled. Then, the representations of the candidates can be calculated and stored offline, and directly retrieved during the online matching phase. To conduct the interactions and generate final matching scores, a lightweight attention network is designed. Experiments based on several large scale text matching datasets show that the proposed model, called FASTMATCH, can achieve up to 100X speed-up to BERT and RoBERTa at the online matching phase, while keeping more up to 98.7% of the performance.

## introduction many natural language processing tasks, such as answer selection @xcite in question answering, query-document matching @xcite in information retrieval, natural language inference @xcite and paraphrase identification @xcite can be formulated as text matching. typical text matching model takes a pair of sentences (texts) as input and outputs their similarity score or label. recently large pre-trained language models (plm) like bert @xcite , roberta @xcite have achieved state-of-the-art results in this field. however, these plm-based models usually have high computational cost in both training and inference phase due to their very deep architectures with up to billions of parameters. in particular, the high latency caused by the intensive computation during the online inference makes the deployment of plms prohibitive for many real-world applications. in particular, the high latency during online inference of bert becomes crucial when text matching is applied to ir and qa, where a given query sentence needs to be matched to n (> 10 2 ) retrieved candidates sentences (documents). the plm-based matching model has to repeatedly calculate the representations and interactions for all the n [query, candidate] pairs in an online manner. denoting the time cost for matching each pair as t , the total time cost for processing a single query is n × t , as shown in figure 1 (a) . for such candidate-rich text matching, the inference speed per query deteriorates by the increase of the number of candidates, n . thus, how to accelerate the inference becomes a key challenge in applying plm-based text matching models to ir/qa products. to address this challenge, we propose fastmatch, an inference acceleration method for bert-based text matching. its key idea is process decomposition, where matching is decomposed into intra-sentence encoding with bert and inter-sentence interaction with a lightweight core matching model. since the encoding for all the candidates can be conducted offline, the inference procedure can be accelerated dramatically. specifically, fastmatch works in a two-stage manner, as shown in figure 1 @xcite . in the first stage, though the representation of the query is still computed online by bert with the time cost of t , the representations of the n candidates can be pre-computed offline and cached with ignorable online retrieval time. in the second stage, the n pairs of [query, candidate] are fed to, instead of the original plm, the core matching model, which takes time t to predict the label for each pair, arriving at a time cost of n × t. to sum up, the total time cost for fastmatch is approximately t + n × t. thus, the speed-up ratio r is the ratio of the time cost of the plm to that of fastmatch, as shown in (1). with large n , the speed-up ratio is dominated by t /t, i.e. the inference time of the plm over that of the core matching model. while the query-independent and thus pre-computable encoding of candidates lays the foundation for efficient online inference, the real challenge lies in designing a substantially faster core matching model that can, at the same time, keep most of the plm's accuracy. a naïve approach would be choosing an existing, lightweight text matching model to server the purpose. unfortunately, using classic text matching architectures as the core matching model typically leads to dramatic accuracy drop compared with bert-based model (section 4.2). it turns out that it does not only require balancing speed and performance, but also calls for complementing bert so that the whole architecture can get the most out of powerful contextualized representations. in contrast with off-the-shelf models, the proposed core matching model (section 3.1) is designed to take advantage of contextualized nature of bert representations. recall that in bert, each token is recursively encoded by its context with a self-attention layer, using the representations of the most relevant words throughout the sentence from the previous layer. as a result, the word representations from a plm carry a rich set of linguistic features at various granuarities. with such representations, a single interaction layer can largely capture the interactions between the query and candidate at different levels, a proven recipe for accurate matching @xcite . based on such intuitions, we design an attention-based core matching model. in our model, bert-based representations of query and candidate are fed to a cross-attention layer to mimic multi-level, query-candidate interactions. then a self-attention layer is introduced to aggregate token-wise matching results into sentence-level matching result. finally, we combine the matching results using yet another attention layer, after injecting diversities to interaction signals with a pooling layer. in practice, fastmatch can achieve more than 100x speed-up, while achieving comparable accuracy as the original plm. the main contributions of this paper are three-fold: ## related work text matching many text-based applications, such as information retrieval (ir), question answering (qa), natural language inference (nli) and paraphrase identification (pi) can be formalized as text matching. recently, bert establishes itself as the new state-of-the-art in text matching and neural ranking @xcite , thanks to the pre-training, as well as the modeling of all-to-all interactions between tokens in both query and candidate. in this case, the representations of candidate are built dynamically by taking the query into account, where the drawback is that the offline pre-computation of representations becomes infeasible. the proposed method, fastmatch, differs from bert in that (1) its querycandidate interaction is conducted by a much faster core matching model rather than bert itself and (2) the candidate representation is built independently from the query, thus can enjoy the benefits of offline computation of candidate representations, similar to representation-based method. however, unlike representation-based method, fastmatch explicitly computes query-candidate interactions in a multi-granularity manner. at the same time, fastmatch also distinguishes itself from interactionbased methods, as the interactions in fastmatch is based on dynamic, contextualized representation via bert, instead of static word embeddings. ## the fastmatch method the overall structure of fastmatch is depicted in figure 1 (b ). it combines a generic plm with a taskspecific core matching model, leading to a two-stage matching procedure. in order to conduct efficient matching, we propose an attention-based @xcite core matching model, the architecture of which is depicted in figure 2 . in the rest of this section, we go through each layer. we only show the equations for query q, as the readers can infer that for candidate p. ## experiments this section presents fastmatch on a variety of text matching datasets. we first describe the setups and details before presenting main results in section 4.1 with two standard qa datasets. in section 4.2 the choice of core matching model is discussed. ablation study comes in section 4.3. and in section 4.4 auxiliary evaluation of glue is conducted, where fastmatch achieves competitive results compared with model compression techniques for inference acceleration. we use a batch size 32-64, with max sequence length of 64. max training iteration is 12 with early stopping based on the dev set. first 10% steps are used for warming up. adam (kingma and ba, 2014) optimizer is used with initial learning rate 1 × 10 -5 . bert-base and roberta-base are used as encoders for fastmatch. both bert-base/roberta-base have 12 layers, with 12 heads and hidden size @xmath0. all models were trained in a single-task manner, with no ensemble or multi-task learning employed. all experiments were conducted on intel xeon gold 6130 cpu@2.10ghz, tesla v100 gpu and cuda 10.1. ## conclusion and future work to deal with the prohibitive online computational cost of plm for real-world applications, we propose fastmatch, an inference acceleration method of bert for text matching tasks with a rich set of candidates, such as answer selection in qa and document ranking in ir. fastmatch decouples text matching into representation encoding and interaction computation, where the candidates encoding are calculated and stored offline with ignorable online retrieval time. for online interaction-based matching, we introduce an attention-based core matching model, which has significant lower time complexity than bert. experiments show that fastmatch can achieve up to 100x speed-up to bert and roberta in online matching, while keeping most of their accuracy. fastmatch offers an alternative to model compression for inference acceleration with plm. with superior speed-up ratio, it can be useful for other similar tasks. moreover, as fastmatch is orthogonal to model compression, combining them together can yield more promising results.