Language-agnostic Representation from Multilingual Sentence Encoders for Cross-lingual Similarity Estimation (2021)

Abstract: We propose a method to distill a language-agnostic meaning embedding from a multilingual sentence encoder. By removing language-specific information from the original embedding, we retrieve an embedding that fully represents the sentence’s meaning. The proposed method relies only on parallel corpora without any human annotations. Our meaning embedding allows efficient cross-lingual sentence similarity estimation by simple cosine similarity calculation. Experimental results on both quality estimation of machine translation and cross-lingual semantic textual similarity tasks reveal that our method consistently outperforms the strong baselines using the original multilingual embedding. Our method consistently improves the performance of any pre-trained multilingual sentence encoder, even in low-resource language pairs where only tens of thousands of parallel sentence pairs are available.

## introduction pre-trained sentence encoders @xcite @xcite @xcite boost the performance of various natural language understanding (nlu) tasks @xcite . among them, the combination of selfattention networks (sans) @xcite and masked language modelling @xcite has proved to be remarkably successful. these techniques are generalised across languages @xcite and are even applied to cross-lingual and multilingual nlu tasks such as cross-lingual semantic textual similarity (sts) @xcite and quality estimation (qe) of machine translation @xcite . in the latest qe competitions at the conference on machine translation (wmt) @xcite , all top-ranked systems @xcite @xcite employed pre-trained multilingual sentence encoders, such as multilingual bert (mbert) @xcite and xlm-roberta (xlm-r) @xcite . these multilingual sentence encoders form a single self-attention network pre-trained on monolingual corpora in over 100 languages with the objective function of masked language modelling. fine-tuning with a human-annotated corpus is mandatory for these models to enable them to estimate the semantic similarity between sentences across languages. otherwise, these models are not sensitive to semantic similarity. a sentence encoder that can estimate semantic similarity across languages without fine-tuning for the target task is desirable because bilingual corpora with human annotations are unavailable in most language pairs. figure 1 plots embeddings of parallel sentences in three languages extracted from mbert without fine-tuning. this visualisation implies that the mbert embeddings form clusters by language rather than by meaning. we propose a method for distilling language-agnostic meaning embeddings by removing language-specific information from sentence embeddings generated by off-the-shelf multilingual sentence encoders. our embeddings allow efficient cross-lingual sentence similarity estimation using simple cosine similarity. our method does not require human annotations specific to the target task and is based solely on the bilingual corpora. experimental results on both the wmt20 qe task @xcite and the semeval-2017 cross-lingual sts task @xcite in unsupervised settings revealed that our method consistently outperformed the strong baselines using the existing pre-trained multilingual sentence encoders. ## proposed method although multilingual sentence encoders are useful for cross-lingual nlu, their embeddings are highly biased by language-specific information, which separates sentence embeddings into multiple languages, as shown in figure 1 . we distil language-agnostic meaning embedding from multilingual sentence embeddings to estimate crosslingual sentence similarity in an unsupervised manner. by training with bilingual corpora, we unite embeddings of semantically similar sentences from pre-trained multilingual sentence encoders. our model is an autoencoder comprising two multi-layer perceptrons (mlps), mlp m and mlp l , as shown in figure 3 . the former is responsible for extracting meaning, while the latter extracts language-specific information, and then, these outputs are summed to reconstruct the input sentence embedding. we train these mlps using multilingual and multitask learning using three loss functions: figure 2 presents an overview of our multitask training, for which we input a pair of bilingual sentences, (a) and (b), as well as randomly selected sentences of each language, (c) and (d). sentences (a) and (c) are from the same language, as are (b) and (d). the constraints in l m make the meaning embeddings derived from ( a closer, while the meaning embeddings derived from (a) and (c) (also (b) and ( d )) become distant. in contrast, the constraints of l l make the language embeddings derived from (a) and (c) as well as (b) and (d) to come closer, respectively. in addition, it further acts as a constraint on how language embeddings retain language-specific information using language identification. we perform multitask learning in a multilingual manner, that is, by mixing all languages to support the target task. all parameters of mlp m are shared (same for mlp l ). obviously, our model is trained using only parallel corpora without any human annotations, such as qe labels. ## evaluation we evaluated the effectiveness of the proposed method in two regression tasks: the wmt20 qe task @xcite and semeval-2017 crosslingual sts task @xcite . as shown in figure 4 , the meaning embeddings of each input sentence are extracted using our meaning encoder. in this experiment, we evaluated the correlation between the cosine similarity of meaning embeddings and human labels. following the official evaluation metrics, we used pearson correlation for both tasks implemented in the scipy foot_0 package. ## analysis we further analyse our method through an ablation study and visualisation of sentence embeddings. ## conclusion to achieve unsupervised language-agnostic sentence similarity estimation, we distilled the meaning embeddings using pre-trained multilingual sentence encoders. we trained the autoencoder consisting of two mlps, that is, meaning encoder and language encoder, in a multitask and multilingual manner. our method successfully distils languageagnostic (i.e., meaning embedding) information by removing language-specific (i.e., language embedding) information from the original sentence embedding. our method has following advantages: (1) it can be trained using only parallel corpora without any human annotations. (2) based on pre-trained multilingual sentence encoders, our single model can cover more than 100 languages. experimental results in both the qe and crosslingual sts tasks revealed that our method consistently improves the performance of original multilingual sentence encoders, such as mbert, xlm-r, and labse. substantial improvements were obtained even from tens of thousands of parallel sentence pairs, achieving the highest performance in qe for low-resource language pairs.