Zero-shot Cross-lingual Alignment for Embedding Initialization (2024)

Abstract: For multilingual training, we present CrossInit, an initialization method that initializes embeddings into similar geometrical structures across languages in an unsupervised manner. CrossInit leverages a common cognitive linguistic mechanism, Zipf’s law, which indicates that similar concepts across languages have similar word ranks or frequencies in their monolingual corpora. Instead of considering point-to-point alignments based on ranks, CrossInit considers the same span of consecutive ranks in each language as the Positive pairs for alignment, while others out of the span are used as Negative pairs. CrossInit then employs Contrastive Learning to iteratively refine randomly initialized embeddings for similar geometrical structures across languages. Our experiments on Unsupervised NMT, XNLI, and MLQA showed significant gains in low-resource and dissimilar languages after applying CrossInit.

## introduction zipf's law suggests that words with similar meanings and senses in different languages may have a similar word rank or frequency in the individual language foot_1 . the starting point is how zipf's law reflects on the multilingual corpus we use. to observe it from an inspiring example, we counted words appearing in wikipedia dumps † foot_2 and ranked them in order of frequency, presenting the result in figure 1 . in a short conclusion, as supported by the literature, languages are motivated by common cognitive mechanisms to form similar structural patterns across languages, thus conforming to zipf's law @xcite @xcite . one idea we can derive from zipf's law is how to align words in different languages before multilingual training, e.g., aligning words in initialization. however, directly aligning words across languages based on word ranks is still challenging because it is impossible to use point-to-point alignments based on word ranks in practice. on multilingual corpus, although we use a shared vocabulary for all the languages, each language has a different local vocabulary, and words with similar concepts and meanings only have a similar rank (not identical) in different languages, as observed in figure 1 . to tackle these challenges, we can approximately separate irrelevant words to some extent. intuitively, we can consider positive pairs between the same spans of consecutive word ranks and negative pairs between different ones in different languages. in this way, the multilingual model is encouraged to understand possible and impossible alignments between words across languages. another motivation comes from self-inference multilingual models @xcite . existing works have shown that a pre-trained multilingual model can infer translations for input words, where translations and input words have similar word ranks or frequencies on their monolingual corpora. if the model is more likely to under-stand words with similar ranks across languages as cross-lingual transferable entries, we can align these words in the initialization phase to provide meta-learning supervision. in this work, we present crossinit, a method to iteratively initialize an embedding space for a multilingual model before formal training or pre-training on a multilingual corpus. in each initialization step, according to word ranks in each language, we randomly sample a span of consecutive ranks and use all words in this span across languages for positive pairs. in contrast, we create negative pairs between words inside and outside this span across languages. we show the idea in figure 2 . we experimented with contrastive learning to train these positive and negative pairs in each initialization step, but we believe there is a significant potential for the development of new alternatives. our experimental results demonstrated that crossinit can improve results in low-resource and dissimilar languages on unsupervised nmt, xnli, and mlqa. we summarize contributions and findings as follows: ## cross-lingual initialization crossinit aims to iteratively initialize random embeddings before any multilingual pre-training for downstream tasks. ## experiment our analysis, including fast xnli experiments, demonstrated the effectiveness of crossinit. in scaled experiments, we evaluated our method in larger-scale settings. model card xlm facebook/xlm-mlm-xnli15-1024 mbart facebook/mbart-large-en-ro wiki wikimedia/wikipedia (version: 20231101) cc cc100 wmt16 wmt/wmt16 xnli facebook/xnli mlqa facebook/mlqa flores facebook/flores ## conclusion we present crossinit, a method for initializing embeddings to create similar geometric structures across languages in an unsupervised man- given n e's extremely low resources, we use its similar language hi in our multilingual training (+hi ). ⋆ denotes models we reimplement with model cards. +cc25 stands for using cc25 corpora.