Dual-Alignment Pre-training for Cross-lingual Sentence Embedding (2023)

Abstract: Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding. However, our research indicates that token-level alignment is also crucial in multilingual scenarios, which has not been fully explored previously. Based on our findings, we propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding that incorporates both sentence-level and token-level alignment. To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. This reconstruction objective encourages the model to embed translation information into the token representation. Compared to other token-level alignment methods such as translation language modeling, RTL is more suitable for dual encoder architectures and is computationally efficient. Extensive experiments on three sentence-level cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding. Our code is available at https://github.com/ChillingDream/DAP .

## introduction cross-lingual sentence embedding encodes multilingual texts into a single unified vector space for a variety of natural language processing (nlp) tasks, including cross-lingual sentence retrieval @xcite and crosslingual natural language inference @xcite . the text sequences can be efficiently retrieved and compared using the inner product between their dense representations. the task of sentence embedding now heavily depends on pre-trained language models (devlin et al., 2019; @xcite . by fine-tuning the cls token of the pre-trained model, they encode the input text sequence into a single vector representation. recent research has shown that using the translation ranking task in combination with a dual pretrained encoder can result in superior sentence embeddings @xcite @xcite @xcite . the purpose of fine-tuning the cls token is to learn sentence-level alignment and to compress the entire sentence's information into the cls token. this method makes the cls tokens of semantically relevant sentences have larger inner products. however, token-level alignment in multilingual scenarios is also crucial, and the fine-grained alignment task in cross-lingual sentence embedding has not been fully explored. as shown in figure 1 , we visualize the token representation similarities between a pair of parallel corpora. training for an objective solely with regard to cls token causes the token representations to disperse across the embedding space. based on our observations, we propose an efficient dual-alignment pre-training (dap) framework for cross-lingual sentence embedding. the embedding model is trained towards both sentencelevel alignment and token-level alignment. previous cross-lingual pre-training studies @xcite employ translation language modeling (tlm) to achieve token alignment. in this paper, we introduce a novel representation translation learning (rtl) method that reconstructs the entire english input based on the token representations of parallel non-english sentences using a transformer model. by optimizing the rtl objective, the model learns to embed the information of english sentences into the representation of its non-english counterpart. unlike tlm, computing rtl only needs one-side self-contextualized representation and does not involve extra feedforward propagation. we train our model on public corpora and evaluate it on three cross-lingual tasks: bitext retrieval, bitext mining, and cross-lingual natural language inference. our results demonstrate dap can effectively improve cross-lingual sentence embedding. our contributions are summarized as follows: ## experiments in this section, we first describe the training setup. then we compare our method with previous works on three sentence-level cross-lingual tasks. ## analysis in this section, we conduct experiments to get a deeper understanding of dap. in each setting, we report the average accuracy over 36 languages and two retrieval directions on tatoeba, average f1 score on bucc test set and average accuracy on xnli. all variants are trained from mbert. ## conclusion in this paper, we find that token-level alignment is crucial for cross-lingual tasks. based on this observation, we present a dual-alignment pre-training framework for cross-lingual sentence embedding that enables both sentence-level and token-level alignment. the framework consists of a translation ranking task and a newly proposed representation translation learning task, which encourages the token representation to contain all information from its translation counterpart in an efficient way. we train our models on a moderate-size corpus. the model trained with dap significantly outperforms variants without token-level alignment or using tlm as the alignment task across three sentence-level cross-lingual tasks, and achieves performance comparable with those state-of-the-art pre-training work trained on 10 times more data with larger batch size and training steps. these results show our approach brings essential improvement for cross-lingual sentence embedding.