Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment (2024)

Abstract: The field of cross-lingual sentence embeddings has recently experienced significant advancements, but research concerning low-resource languages has lagged due to the scarcity of parallel corpora. This paper shows that cross-lingual word representation in low-resource languages is notably under-aligned with that in high-resource languages in current models. To address this, we introduce a novel framework that explicitly aligns words between English and eight low-resource languages, utilizing off-the-shelf word alignment models. This framework incorporates three primary training objectives: aligned word prediction and word translation ranking, along with the widely used translation ranking. We evaluate our approach through experiments on the bitext retrieval task, which demonstrate substantial improvements on sentence embeddings in low-resource languages. In addition, the competitive performance of the proposed model across a broader range of tasks in high-resource languages underscores its practicality.

## introduction cross-lingual sentence embedding encodes multilingual texts into a shared semantic embedding space in which the texts are understandable across different languages. various applications including bitext retrieval @xcite and cross-lingual semantic textual similarity tasks @xcite rely on cross-lingual sentence embedding. current approaches to obtaining cross-lingual sentence embeddings primarily utilize multilingual pre-trained language models @xcite @xcite that employ masked language modeling and translation language modeling objectives to predict masked tokens within the context. such models implicitly align the contextual representations of semantically similar units of sentences in different w/o word-aligned training en es de jv ka w/ word-aligned training figure 1 : t-sne visualization of sampled word embeddings from both high-resource and low-resource languages. the red points represent the word embeddings from high-resource languages, and the blue points correspond to those from low-resource languages. this comparison highlights the differences of word representation in the models w/ and w/o the explict word-aligned training. left: words in low-resource languages are under-aligned with their translations in high-resource languages. right: the phenomenon of under-alignment is mitigated through the proposed explicit word-aligned training. the details of word sampling, word embeddings and word-aligned training are described in section 4.3. languages @xcite , thereby enabling the models to understand texts in various languages. while the field of cross-lingual sentence embedding has recently seen great advancements @xcite @xcite , research concerning low-resource languages has lagged due to the scarcity of parallel corpora. in figure 1 , we observe that word embeddings from low-resource languages, which are derived from current cross-lingual models trained solely with a sentence-level alignment objective, are under-aligned with those from high-resource languages. to address this under-alignment, we introduce a new framework featuring two word-level alignment objectives: aligned word prediction and word translation ranking. these objectives are designed to align the word-level signals of parallel sentences. additionally, a sentence-level alignment objective, known as translation ranking @xcite , is also used to ensure the basic sentence understanding. we name our proposed framework wacse (word aligned cross-lingual sentence embedding). the right sub-figure in figure 1 shows the distribution of word embeddings obtained from the model trained with the proposed aligned word prediction and word translation ranking. it demonstrates that the under-alignment phenomenon can be mitigated through the explicitly word-aligned objectives. the experiment results demonstrate that the proposed word-aligned training objectives can enhance cross-lingual sentence embedding, particularly for low-resource languages, as evidenced on the tatoeba dataset @xcite . this finding matches our observations on word representations in figure 1 . furthermore, our model retains competitive results across a broader range of tasks, including sts22 @xcite , bucc @xcite , and xnli @xcite , in which most languages are high-resource. this indicates the practicality and robustness of the proposed framework. ## method to enhance cross-lingual sentence embeddings of low-resource languages through explicit alignment of words, wacse incorporates three tasks, translation ranking (tr), aligned word prediction (awp) and word translation ranking (wtr) tasks. these tasks collectively aim to learn the cross-lingual sentence representations of parallel sentences. the framework is depicted in figure 2 . formally, we start with a parallel dataset (x, y) in two languages and the i-th parallel sentence pair is denoted as (x i , y i ). x i and y i can be represented as a sequence of words: x @xmath0, x 2 , . . . , x |x i | and y @xmath1, y 2 , . . . , y |y i | , respectively where | • | denotes the length of the given sentence. after inputting a sentence into the model, we obtain the hidden representations from the last layer as follows: note that h x i i could be a sequence of embeddings because a word could be tokenized into multiple tokens. this could affect some minor implementation in the practice. refer to section 4.3 for the detailed implementation regarding this issue. particularly, h x i cls is the hidden state of the cls token for representing the whole sentence. ## analysis in this section, we carry out experiments to gain a deeper understanding of the proposed framework, specifically investigating the role of language identification information and the three losses in wacse. our primary focus is on the tatoeba dataset as it is the only one that encompasses the low-resource language setting. all models discussed in this section are trained with a fixed seed (42) and the training step is 10k. ## conclusion in this paper, we observe an intriguing phenomenon: the distributions of word embeddings of low-resource languages are under-aligned with those of high-resource languages in current multilingual pre-trained language models. based on this observation, we propose a framework designed to align word-level semantically equivalent units in parallel sentences between high-resource languages and low-resource languages, thereby enhancing the cross-lingual sentence embeddings for low-resource languages. furthermore, we demonstrate that aligning word-level semantically units between two high-resource languages with our proposed method may detrimentally affect the language-specific features learned during the pretraining phase. our experimental results show the effectiveness of our method in improving crosslingual sentence embeddings for low-resource languages. additionally, wacse preserves the performance of the model on other tasks that involve high-resource languages.