Identifying the Importance of Content Overlap for Better Cross-lingual Embedding Mappings (2021)

Abstract: In this work, we analyze the performance and properties of cross-lingual word embedding models created by mapping-based alignment methods. We use several measures of corpus and embedding similarity to predict BLI scores of cross-lingual embedding mappings over three types of corpora, three embedding methods and 55 language pairs. Our experimental results corroborate that instead of mere size, the amount of common content in the training corpora is essential. This phenomenon manifests in that i) despite of the smaller corpus sizes, using only the comparable parts of Wikipedia for training the monolingual embedding spaces to be mapped is often more efficient than relying on all the contents of Wikipedia, ii) the smaller, in return less diversified Spanish Wikipedia works almost always much better as a training corpus for bilingual mappings than the ubiquitously used English Wikipedia.

## introduction word embedding methods (e.g. @xcite @xcite have become an essential tool for representing words in most nlp tasks. these algorithms assign a low-dimensional vector to words based on the patterns of their contexts in a training corpus, and this way they locate the words in the vector space in a consistent way, so that words with similar meaning are assigned to similar vectors. therefore, it can be assumed that the layout of the word vectors are near equivalent in independently trained models, so word embedding models in different languages can be aligned into a common space. such alignments are a standard way of creating bi-or multilingual word embedding spaces, which are very useful for machine translation and a wide range of cross-lingual nlp tasks. although large pre-trained language models are superior to traditional word embeddings in many nlp tasks, one strength of these mapping-based methods is their extensive applicability, e.g. for low-resource languages or special domain (e.g. medical) data. additionally, probably due to significantly lower resource requirements @xcite and often competitive results @xcite , a large proportion of industrial nlp applications is still based on static word embeddings @xcite . on the other hand, the results of the mappings and the performance of the multilingual models was still shown to be extremely dependent on the mapping scenario @xcite @xcite . previous work attributes low performance and non-isomorphism of monolingual embedding models to typological differences between languages, domain differences in the training corpora, insufficient resources, and under-training @xcite @xcite . we rely on popular bilingual alignment methods, and conduct a thorough analysis of the connection between the evaluation scores of the mappings and language relatedness, isomorphism of the source embeddings measured by several metrics, and some newly identified, easy to calculate corpus properties that are highly predictive of the bilingual mapping performance: the token overlap ratio in the vocabularies, and the distance between word distributions of the corpora. these combined with corpus size surpass existing isomorphism measures as predictors of bilingual mapping score. our goal is to help researchers and developers use resources more efficiently, and find the most appropriate setting for creating bilingual word embedding models. 1 2 related work ## conclusion we examined the connection of embedding mapping scores to languages, corpus properties, and embedding isomorphism. we found that the spanish wikipedia is better for this purpose than the english wikipedia, often used by default. this is explained by our other experiments on the relationship of corpus properties and mapping quality, where it turned out that corpus similarity is at least as important as corpus size, therefore the hugeness and wide diversity of the english wikipedia can be harmful. moreover, we have seen that language similarity is really beneficial for very closely related languages only, e.g. between germanic or romance languages. mapping scores are well predictable even without any information about the languages, based on three properties of the corpora: corpus size, proportion of common tokens, and distance of the word distributions. these data also surpass existing embedding isomorphism measures as predictors. on the other hand, this paper focuses on bli scores only, which were shown to not correlate perfectly with bilingual performance on downstream tasks @xcite . we suppose that to some extent our findings hold in downstream situations as well, since downstream performance cannot be independent of bli scores, but this question should be part of further research. the main difference between downstream and bli evaluation scores is probably the importance of monolingual embedding quality: while two embedding matrices can be trained almost perfectly isomorphically on a relatively small parallel corpus, the monolingual performance of these embeddings probably lags behind embeddings trained on a big corpus, wikipedia for example. but at the same time, this also shows that embeddings can be mapped very well even if they are not of the highest quality, but their corpora are similar enough.