Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching (2022)

Abstract: Previous studies have proved that cross-lingual knowledge distillation can significantly improve the performance of pre-trained models for cross-lingual similarity matching tasks. However, the student model needs to be large in this operation. Otherwise, its performance will drop sharply, thus making it impractical to be deployed to memory-limited devices. To address this issue, we delve into cross-lingual knowledge distillation and propose a multi-stage distillation framework for constructing a small-size but high-performance cross-lingual model. In our framework, contrastive learning, bottleneck, and parameter recurrent strategies are delicately combined to prevent performance from being compromised during the compression process. The experimental results demonstrate that our method can compress the size of XLM-R and MiniLM by more than 50%, while the performance is only reduced by about 1%.

## introduction on the internet, it is widespread to store texts in dozens of languages in one system. cross-lingual similar text matching in multilingual systems is a great challenge for many scenarios, e.g., search engines, recommendation systems, question-answer robots, etc. @xcite @xcite . in the monolingual scenario, benefiting from the robust performance of the pre-trained language models (plms) (e.g., bert @xcite , roberta @xcite , t5 @xcite , etc.), significant success has been achieved in textsimilarity matching tasks. for example, @xcite proposed the sbert model trained with similar text pairs and achieved the state-of-the-art performance in the supervised similarity matching. in unsupervised scenarios, @xcite proposed the simcse model, which figure 1 : evaluation results of xlm-r with different number of encoder layers on the sts2017 monolingual task and the sts2017-extend cross-lingual task, using sbert-paraphrases for knowledge distillation. drawing on the success in the monolingual scenario, researchers began to introduce pre-training technology into cross-lingual scenarios and proposed a series of multilingual pre-trained models, e.g., mbert @xcite , xlm @xcite , xlm-r @xcite , etc. due to the vector collapse issue @xcite , the performances of these cross-lingual models on similarity matching tasks are still not satisfactory. @xcite injected the similarity matching ability of sbert into the cross-lingual model through knowledge distillation, which alleviated the collapse issue and improved the performance of cross-lingual matching tasks. although the cross-lingual matching tasks have achieved positive results, the existing cross-lingual models are huge and challenging to be deployed in devices with limited memory. we try to distill the sbert model into an xlm-r with fewer layers following @xcite . however, as shown in figure 1 , the performance will be significantly reduced as the number of layers decreases. this phenomenon indicates that crosslingual capabilities are highly dependent on the model size, and simply compressing the number of layers will bring a serious performance loss. in this work, we propose a multi-stage distillation compression framework to build a small-size but high-performance model for cross-lingual similarity matching tasks. in this framework, we design three strategies to avoid semantic loss during compression, i.e., multilingual contrastive learning, parameter recurrent, and embedding bottleneck. we further investigate the effectiveness of the three strategies through ablation studies. besides, we respectively explore the performance impact of reducing the embedding size and encoder size. experimental results demonstrate that our method effectively reduces the size of the multilingual model with minimal semantic loss. finally, our code is publicly available foot_0 . the main contributions of this paper can be summarized as follows: ## method in this section, we will introduce our method in detail. first, we exhibit the model architecture, and then introduce the multi-stage distillation strategy for the model training. an overview of our approach is shown in figure 2 . ## conclusion in this work, we realize that the cross-lingual similarity matching task requires a large model size. to obtain a small-size model with cross-lingual matching ability, we propose a multi-stage distillation framework. knowledge distillation and contrastive learning are combined in order to compress model with less semantic performance loss. our experiments demonstrate promising sts results with three monolingual and six cross-lingual pairs, covering eight languages. the empirical results show that our framework can shrink xlm-r or minilm by more than 50%. in contrast, the performance is only reduced by less than 0.6% on monolingual and 1.1% on cross-lingual tasks. if we slack the tolerated loss performance in 4%, the size of xlm-r can be reduced by 80%.