VIRT : Improving Representation-based Text Matching via Virtual Interaction (2022)

Abstract: Text matching is a fundamental research problem in natural language understanding. Interaction-based approaches treat the text pair as a single sequence and encode it through cross encoders, while representation-based models encode the text pair independently with siamese or dual encoders. Interaction-based models require dense computations and thus are impractical in real-world applications. Representation-based models have become the mainstream paradigm for efficient text matching. However, these models suffer from severe performance degradation due to the lack of interactions between the pair of texts. To remedy this, we propose a Virtual InteRacTion mechanism (VIRT) for improving representation-based text matching while maintaining its efficiency. In particular, we introduce an interactive knowledge distillation module that is only applied during training. It enables deep interaction between texts by effectively transferring knowledge from the interaction-based model. A light interaction strategy is designed to fully leverage the learned interactive knowledge. Experimental results on six text matching benchmarks demonstrate the superior performance of our method over several state-of-the-art representation-based models. We further show that VIRT can be integrated into existing methods as plugins to lift their performances.

## introduction text matching aims to model the semantic correlation between a pair of texts, which is a fundamental problem in various natural language understanding applications. for instance, in community question answering (cqa) @xcite systems, a key component is to find similar questions from the database regarding a user question via question matching @xcite ., * equal contribution. † corresponding author. recently, the wide use of deep pre-trained transformers @xcite has made remarkable progress in text matching tasks @xcite @xcite .two paradigms based on fine-tuned transformer encoders are typically built: interaction-based models and representation-based models, as illustrated in figure 1 (a) & (b). interaction-based models (e.g., bert @xcite ) jointly encode the text pair, which allows the two text sequences to attend each other from the bottom layer to the top layer, resulting in effective matching signals. however, full interaction leads to high computational cost with large inference latency. in addition, text embedding can not be cached or pre-computed, which makes them impractical in many real-world scenarios. for example, in an e-commerce search system, it will cost dozens of days to score millions of query-product pairs with interaction-based models @xcite . representation-based models @xcite encode two texts independently with siamese or dual encoders @xcite , which enable the offline-computing of text embeddings and thus significantly reduce the online latency. unfortunately, independent encoding without any interaction fails to capture the correlation between the text pair, resulting in severe performance degradation. to balance efficiency and efficacy, several works attempt to equip the siamese structure with late interaction modules. these late interactions are essentially light-weight interaction layers that fuse the two text embeddings from the individual encoders. a variety of late interaction strategies have been proposed, including mlp layers @xcite , cross-attention layers @xcite and transformer layers @xcite , which obtain considerable improvements on different text matching tasks with reasonable costs. however, these interaction modules are added after siamese encoders, while interactions in the encoding process of siamese encoders are still ignored, leaving a large performance gap compared to the interaction-based models. in this work, we propose a virtual interaction (virt) mechanism with interactive knowledge distillation for improving representation-based text matching while keeping its efficiency. specifically, siamese encoders learn interactive information between the pair of texts by mimicking the full interaction, with transferred knowledge from the interaction-based models as guidance. we employ the knowledge transfer as an attention map distillation during training, which is removed during inference to keep the siamese property, and thus called "virtual interaction". moreover, we design a virt-adapted interaction strategy after siamese encoding to further leverage the learnt interactive knowledge. our proposed virt is illustrated in figure 1(c ). experimental results on six text matching benchmarks show the superior performance of virt over several state-of-the-art baselines. we summarize the main contributions of this work as follows: recently transformer-based models @xcite leverage self-attention to achieve promising performance on several text matching tasks @xcite @xcite . generally, these models can be classified into interaction-based models @xcite and representation-based models @xcite . as a typical interaction-based model, bert @xcite concatenates the text pair as the input and uses its [cls] token embedding to predict the matching @xcite . in contrast, representation-based models utilize dual encoders to encode the pair of texts individually, which achieve high inference efficiency by pre-computing and storing all text embeddings in the database. however, there is usually a large performance degradation compared to interaction-based models. more recently, late interactions with light attention layers @xcite @xcite have been introduced after dual encoders to balance efficiency and efficacy. however, rich interactive information between the text pair is still ignored during encoding. knowledge distillation knowledge distillation @xcite is to transfer knowledge from a teacher model with better quality to a less complex student model. various works @xcite @xcite have been proposed to compress bert to a tiny structure with fewer transformer layers and smaller hidden size through distilling predicted logits and hidden states. there are several recent distillation works that are closely related to our work. dipair @xcite performs extra interaction through a light transformer layer, and distills predicted logits from the interaction-based model. deformer @xcite adopts multiple transformer-based interaction layers and distills the representations as well as the predicted logits from the interaction-based model. however, these methods merely distill logits/representations from interaction-based models to the late interaction layer of representation-based models. in contrast, virt distills the attention map from the interactionbased model directly to the encoding process of siamese encoders, which transfers interactive knowledge more effectively. ## conclusion representation-based models are widely used in text matching tasks due to their high efficiency while under-performing the interaction-based ones caused by lacking interaction. previous works often introduce extra interaction layers while the interaction in siamese encoders is still missing. in this paper, we propose a virtual interaction (virt) mechanism , which could approximate the interactive modeling ability by distilling the attention map from interaction-based models to the siamese encoders of representation-based models, with no additional inference cost. the proposed virt, which employs knowledge distillation as well as adapted interaction strategy, achieves state-of-theart performance among existing representationbased models on several text matching tasks.