Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling (2023)

Abstract: Transformer-based models have achieved great success on sentence pair modeling tasks, such as answer selection and natural language inference (NLI). These models generally perform cross-attention over input pairs, leading to prohibitive computational cost. Recent studies propose dual-encoder and late interaction architectures for faster computation. However, the balance between the expressive of cross-attention and computation speedup still needs better coordinated. To this end, this paper introduces a novel paradigm TopicAns for efficient sentence pair modeling. TopicAns involves a lightweight cross-attention mechanism. It conducts query encoding only once while modeling the query-candidate interaction in parallel. Extensive experiments conducted on four tasks demonstrate that our TopicAnscan speed up sentence pairing by over 113x while achieving comparable performance as the more expensive cross-attention models.

## introduction sentence pair modeling, such as natural language inference, question answering, and information retrieval, is an essential task in natural language processing @xcite @xcite . these tasks can be depicted as a procedure of scoring the candidates given a query. recently, transformer-based models @xcite have shown promising performance on sentence pair modeling tasks due to the expressiveness of the pre-trained cross-encoder. as shown in figure 1 (a), the cross-encoder takes a pair of query and candidate as input and calculates the interaction between them at each layer by the input-wide self-attention mechanism. despite the effective text representation power, the cross-encoder leads to exhaustive computation costs, especially when the number of candidates is very large ( e.g., the interaction will be calculated n times if there are n candidates). this computation cost, therefore, restricts the use of these cross-encoder models in many real-world applications @xcite . to tackle this issue, we propose a lightweight cross-attention mechanism, called mixencoder, that speeds up the inference while maintaining the expressiveness of cross-attention. specifically, the proposed mixencoder accelerates the crossattention by performing attention only from candidates to the query, involving few tokens and only at a few layers. this lightweight cross-attention avoids repetitive query encoding, supporting the processing of multiple candidates in parallel and thus reducing computation costs. additionally, mixencoder allows to pre-compute the candidates into several dense context embeddings and to store them offline to accelerate the inference further. we evaluate mixencoder for sentence pair modeling on four benchmark datasets related to tasks of natural language inference, dialogue, and information retrieval. the results demonstrate that mix-encoder better balances the effectiveness and efficiency. for example, mixencoder achieves a substantial speedup of more than 113x over the crossencoder and provides competitive performance. ## background extensive studies, including dual-encoder @xcite and late interaction models @xcite @xcite , have been proposed to accelerate the transformer inference on sentence pair modeling tasks. as shown in figure 1 , dual-encoders process the query and candidates separately, allowing precomputing the candidates to accelerate online inference, resulting in fast inference speed. however, this speedup is built upon sacrificing the expresquery cls candidate cls (a) candidate pre-computation (b) online encoding & classification/ranking cache classifier interaction layer cache ln ln ln pooling att att gate att interaction layer candidates figure 2: overview of proposed mixencoder. ## method in this section, we introduce the details of the proposed mixencoder, which simplifies crossattention by enabling pre-computation, reducing the times of query encoding, and reducing the number of involved tokens and layers. ## experiments datasets. we evaluate mixencoder on three paired-input tasks over four datasets, including mnli @xcite for natural language inference, ms marco passage reranking @xcite for information retrieval, and dstc7 @xcite , ubuntu v2 @xcite for utterance selection for dialogue. baselines. training details. while training models on mnli, we use the labels provided in the dataset. while training models on the other three datasets, we use in-batch negatives @xcite . detailed settings are provided in a.1. ## results