Multilingual Sentence Transformer as A Multilingual Word Aligner (2022)

Abstract: Multilingual pretrained language models (mPLMs) have shown their effectiveness in multilingual word alignment induction. However, these methods usually start from mBERT or XLM-R. In this paper, we investigate whether multilingual sentence Transformer LaBSE is a strong multilingual word aligner. This idea is non-trivial as LaBSE is trained to learn language-agnostic sentence-level embeddings, while the alignment extraction task requires the more fine-grained word-level embeddings to be language-agnostic. We demonstrate that the vanilla LaBSE outperforms other mPLMs currently used in the alignment task, and then propose to finetune LaBSE on parallel corpus for further improvement. Experiment results on seven language pairs show that our best aligner outperforms previous state-of-the-art models of all varieties. In addition, our aligner supports different language pairs in a single model, and even achieves new state-of-the-art on zero-shot language pairs that does not appear in the finetuning process.

## introduction word alignment aims to find the correspondence between words in parallel texts @xcite . it is useful in a variety of natural language processing (nlp) applications such as noisy parallel corpus filtering @xcite , bilingual lexicon induction @xcite , code-switching corpus building @xcite and incorporating lexical constraints into neural machine translation (nmt) models @xcite . recently, neural word alignment approaches have developed rapidly and outperformed statistical word aligners like giza++ @xcite and fast-align @xcite . some works @xcite @xcite , 2019, 2020; @xcite @xcite induce alignments from nmt model or its variants. however, these bilingual models only support the language pair involved in the training process. they also treat the source and target side differently, thus two models are required for bidirectional alignment extraction. another line of works @xcite build multilingual word aligners with contextualized embeddings from the multilingual pretrained language model @xcite . thanks to the language-agnostic representations learned with multilingual masked language modeling task, these methods are capable of inducing word alignments even for language pairs without any parallel corpus. different from previous methods, in this paper we present accalign, a more accurate multilingual word aligner with the multilingual sentence transformer labse @xcite , see figure 1 ). the labse is trained on large scale parallel corpus of various language pairs to learn language-agnostic sentence embeddings with contrastive learning. however, it is unclear whether labse has learned language-agnostic word-level embeddings, which is the key for the success of word alignment extraction. specifically, we first direct induce word alignments from labse and demonstrate that labse outperforms other mplms currently used in the alignment task. this indicates that labse has implicitly learned languageagnostic word-level embeddings at some intermediate layer. then we propose a simple and effective finetuning method to further improve performance. empirical results on seven language pairs show that our best aligner outperforms previous sota models of all varieties. in addition, our aligner supports different language pairs in a single model, and even achieves new sota on zero-shot language pairs that does not appear in finetuning process. foot_0 2 accalign 2.1 background: labse labse @xcite is the state-of-the-art model for the cross-lingual sentence retrieval task. given an input sentence, the model can retrieve the most similar sentence from candidates in a different language. labse is first pretrained on a combination of masked language modeling @xcite and translation language modeling (conneau and lample, 2019) tasks. after that, it is effectively finetuned with contrastive loss on 6b parallel sentences across 109 languages. we leave the training detail of labse in the appendix. however, as labse does not include any word-level training loss when finetuning with contrastive loss, it is unclear whether the model has learned high-quality language-agnostic word-level embeddings, which is the key for a multilingual word aligner. ## conclusion in this paper, we introduce accalign, a novel multilingual word aligner based on multilingual sentence transformer labse. the best proposed approach finetunes labse on a few thousands of annotated parallel sentences and achieves state-of-the-art performance even for zero-shot language pairs. ac-calign is believed to be a valuable alignment tool that can be used out-of-the-box for other nlp tasks.