Reversible Disentanglement of Meaning and Language Representations from Multilingual Sentence Encoders (2025)

Abstract: We propose an unsupervised method to disentangle sentence embeddings from multilingual sentence encoders into language-specific and language-agnostic representations. Such language-agnostic representations distilled by our method can estimate cross-lingual semantic sentence similarity by cosine similarity. Previous studies have trained individual extractors to distill each language-specific and -agnostic representation. This approach suffers from missing information resulting in the original sentence embedding not being fully reconstructed from both language-specific and -agnostic representations; this leads to performance degradation in estimating cross-lingual sentence similarity. We only train the extractor for language-agnostic representations and treat language-specific representations as differences from the original sentence embedding; in this way, there is no missing information. Experimental results for both tasks, quality estimation of machine translation and cross-lingual sentence similarity estimation, show that our proposed method outperforms existing unsupervised methods.

## introduction estimating semantic textual similarity (sts) @xcite is one of the fundamental techniques in natural language processing (nlp). this technology has many potential applications, including information retrieval @xcite and automatic evaluation of nlp-generated sentences @xcite . in recent years, this task has commonly been based on transformer-based sentence encoders @xcite that are pre-trained in objectives such as masked language modeling @xcite and contrastive learning @xcite . these techniques are generalized across languages @xcite , and multilingual sentence encoders @xcite ; feng 1 https://github.com/ehimenlp/seed • ▲ ▲ meaning embeddings in english meaning embeddings in nepali ## proposed method our proposed method disentangles sentence embeddings @xmath0. note that d is the dimension of sentence embeddings. previous studies @xcite used two mlps (mlp m and mlp l ) to distill meaning embeddings êm = mlp m (e) and language embeddings êl = mlp l (e) independently. mlps are trained by adding these embeddings together to reconstruct the original sentence embeddings, however, complete reconstruction from independently extracted embeddings is difficult, and some information is lost. in contrast, the proposed method treats language embeddings as the difference between original sentence embeddings and meaning embeddings, allowing the addition of language and meaning embeddings to reconstruct the original sentence embeddings completely. we use only one mlp to extract meaning embeddings, as follows. as shown in figure 2 , our extractor for meaning embeddings is trained in a multi-task learning manner based on the following three loss functions. we train the mlp with bilingual parallel corpora. figure 2 shows how the mlp is trained by combining loss functions based on cosine similarity between embeddings. meaning embedding ŝm , language embedding ŝl , and others are disentangled from the sentence embeddings s and t from the multilingual sentence encoder for bilingual sentences consisting of a sentence in the source language s and a sentence in the target language t. as in previous study @xcite , sentences s ′ and t ′ , randomly selected from the source and target languages, respectively, are also used for training supplementally. ## conclusion we disentangled sentence embeddings from multilingual sentence encoders into language-specific and language-agnostic embeddings, and applied the latter to cross-lingual sentence similarity estimation. the model architecture of our method has the advantage that there is no missing information during disentangling embeddings. experimental results on qe and cross-lingual sts tasks in an unsupervised manner revealed the effectiveness of the proposed method for both state-of-the-art multilingual sentence encoders, labse and me5.