Linear Cross-Lingual Mapping of Sentence Embeddings (2024)

Abstract: Semantics of a sentence is defined with much less ambiguity than semantics of a single word, and we assume that it should be better preserved by translation to another language. If multilingual sentence embeddings intend to represent sentence semantics, then the similarity between embeddings of any two sentences must be invariant with respect to translation. Based on this suggestion, we consider a simple linear cross-lingual mapping as a possible improvement of the multilingual embeddings. We also consider deviation from orthogonality conditions as a measure of deficiency of the embeddings.

## introduction the approximately linear mapping between crosslingual word embeddings in different languages is based on assumption that the word semantic meaning is conserved in a translation @xcite . the linearity is only approximate because the corresponding words in different languages have different cultural background, different multiple meanings and different dependencies on context @xcite @xcite . there are multiple patterns of polysemy, and the corresponding counts of word senses are different in different languages @xcite . we expect, however, that a sentence has a less ambiguous meaning than a word, simply because the sentence context reduces ambiguity of each of its words. indeed, in @xcite it is demonstrated that additional context helps to reduce disambiguation errors. the idea that a sentence semantics should be better conserved in a translation was used in @xcite . in appendix a we provide simple examples illustrating the loss of word ambiguity in a sentence, and suggest that a good translation can preserve the residual ambiguity, if any. the examples show that if semantics of a sentence is somewhat changed in translation, then a better translation is possible. unlike a lone word, which often has different sets of meaning in different languages, a sentence is not only less ambiguous but also allow differently phrased translations, among which there is usually at least one that fully preserves the semantics. in order to explore the preservation of sentence semantics in translation, we consider here a linear mapping between multilingual embeddings in two languages. unlike the removal of a languagespecific bias in each language separately @xcite , this mapping depends on both languages of interest and, while computationally cheap, may provide a better correspondence between the embeddings. our contribution: ## cross-lingual linear mapping translation of a word can lose or add some of its meanings. but meaning of a sentence or of several contiguous sentences is better defined, and a good translation in most cases (except special idiomatic cases) should preserve the semantics (appendix a). embeddings of the translated sentences should be rigidly related to embeddings of the original sentences: the semantic similarities (or distances) between different embeddings should be preserved. in this section we assume that the 'sentence' is either a (not too short) sentence, or a larger segment of a text. suppose we have n sentences, translated from language l to language l ′ , and then embedded into a space of the same dimension m in each of these languages: the embeddings e 1 , ...e n in l and the embeddings e ′ 1 , ...e ′ n in l ′ . if the measure of semantic similarity in both spaces is cosine, then we should expect that the normalized embeddings e i and e ′ i are related by rotation (orthonormal transform t): if semantic similarity is measured by euclidean distance, and the embeddings are not normalized, then we should allow the orthogonal transform to be accompanied by dilation and shift: the above transformations should be observed if the translations preserved the semantics of the sentences, and if the embeddings represent the semantics correctly. in the following section we will allow any linear transformation (a, b) between the embeddings in l and l ′ : for our illustration here we created embeddings by one of sota aligned multilingual sentenceembedding model, on a set of translated sentences (section 3.2). we optimize the linear transformation on a set of embeddings, so that the mean squared distance between ẽ and e ′ is minimal. in the next section we consider the obtained linear transformation (a, b) from two points of view: ## conclusion we considered a simple and inexpensive method of improving the alignment between sentence embeddings in two languages: a linear transformation, tuned on embeddings of the paired sentences. in the examples we analyzed, a training on sentences also improves an alignment between titles and texts (lower-half texts) of the articles -the articles from our wikinews dataset. if embeddings were capable of perfectly encoding semantics even when not perfectly aligned, then the linear transformation would be an orthogonal transformation, accompanied by dilation and shift. measuring deviation from this condition allows us to judge the quality of the embeddings. for example, we observed lower quality for embeddings of berber and esperanto languages compared to other languages considered here, and also a lower quality of unpc-trained transformations compared to tatoeba-trained transformations. it would be interesting to consider deviation from orthogonality for individual samples, as the strong deviations could point either to bad translations or to the samples difficult to embed by the model.