Are Language-Agnostic Sentence Representations Actually Language-Agnostic? (2021)

Abstract: With the emergence of pre-trained multilingual models, multilingual embeddings have been widely applied in various natural language processing tasks. Language-agnostic models provide a versatile way to convert linguistic units from different languages into a shared vector representation space. The relevant work on multilingual sentence embeddings has reportedly reached low error rate in cross-lingual similarity search tasks. In this paper, we apply the pre-trained embedding models and the cross-lingual similarity search task in diverse scenarios, and observed large discrepancy in results in comparison to the original paper. Our findings on cross-lingual similarity search with different newly constructed multilingual datasets show not only correlation with observable language similarities but also strong influence from factors such as translation paths, which limits the interpretation of the language-agnostic property of the LASER model. %

## introduction multilingual joint embeddings map language units from different languages into the same embedding space in order to make them comparable, which facilitates cross-lingual transfer. being essential for building nlp models of low resourced languages, such an integrated representation is also useful for cross-lingual tasks like machine translation, especially when multiple languages are involved or there is a lack of appropriate data. while word embeddings are widely used in nlp tasks, sentence representations become quite important for capturing underlying semantic relations in texts across different languages. hence, instead of simply pooling word representation together, various neural network methods have proposed to produce more coherent sentence representations. recent advances in multilingual sentence embedding modeling @xcite @xcite begin to show strong performance on many multilingual nlp tasks, but it does not always work equally well for all languages. we repeat the cross-lingual similarity search task with laser @xcite with more challenging corpora in order to identify what affects the actual performance. based on our findings, we propose directions for future development of such models. ## laser laser (language-agnostic sentence representations) @xcite is contextualized language model based on a bilstm encoder trained using a translation objective on parallel data from europarl @xcite , united nations @xcite , opensubtitles2018 @xcite , global voices @xcite , tanzil and tatoeba mostly available on the opus website @xcite . the laser model is able to handle 93 different languages. also @xcite has proposed a similarity-search-based framework in order to evaluate multilingual joint representations. with a collection of s parallel sentences for a given language pair, a multilingual similarity search is performed for the closest target sentence for each of the source sentences, and an error is counted if it is not the reference translation of that sentence in the target language. this approach requires calculating s 2 distance metrics. duplicate sentences need to be removed from the experiment, otherwise the error rates are senseless. in order to have a meaningful comparison across n languages, a similarity search must in addition be performed on an nway parallel sentence set. as the similarity search mainly evaluates the multilingual closeness prop-erty of multilingual joint sentence embeddings, the representations of the same sentence for different languages should be as similar as possible within the joint representation space. being able to process 93 languages within a unified framework, supposedly without bias, laser clearly has limitations. the evaluations have been executed with multilingual data that is either indomain or fairly close to domains of training data. the applications have focused on a small subset of relatively high resource languages. also the training data consist of more informational controlled translations, like official documents. it is unclear whether the agnostic property still holds for new domains, for different genres or for all language. after all, languages and translations in reality are much more diverse and robust than the available parallel corpora. therefore, we conduct a series of evaluations to examine the framework from different angles, to understand its disadvantages and to find paths for future development. ## similarity search on multilingual corpora we apply the laser toolkit to two multilingual corpora that are not part of the training data used to build the pre-trained embedding models: the ted 1 https://github.com/facebookresearch/laser/ corpus @xcite and the appropriate part of the russian national corpus (rnc) @xcite . ## similarity search on different translation paths as we have discussed in previous sections, the majority of multilingual parallel corpora are collections of translations of the same source documents into different target languages, between which cross-lingual similarity search appears to be more difficult. to investigate the underlying factors that affect similarity search by sentence embeddings, we construct new 4-way parallel data by adding translations from different paths to an existing 3 way parallel set in the following manner: we first select 8 ted talks from the recent online release so as to minimize the translator's prior knowledge of the talks. all the talks have been transcribed in english and translated into both german and chinese. there are 629 sentences for each of the 3 languages. the texts are sentence aligned across all 3 languages manually. we sent the sentence segmented german translations as source documents to 4 german-chinese profes the distance between the english original and the english-chinese translations is not far from that between the english and the german ones. the differences in error rate are within 1, around 6 sentences out of 629. the german-chinese translations did turn out to be much closer to the german texts in the sentence representations. we believe it is due to the strict requirements given to the translator. despite being translations into the same language, the two chinese texts lead to the highest search error rates. en: there was just one problem--de: es gab nur ein problem. ## conclusion neural embeddings have been widely applied in all fields of natural language processing. multilingual embeddings with shared representation space enable few-shot and zero-shot transfer from one language to another with minimum additional training or data requirement. recent developments on multilingual sentence representations such as laser have opened up new path towards competitive nlp performance across high-and low-resource languages. yet, our evaluation of laser reveals many constraints when applied in realistic and challenging scenarios. the performance of the framework is largely influenced by the similarity between languages in the multilingual application. not all languages work equally well currently. low performance on specific languages is attributed to the small training data size. these observations caution the interpretation of language-agnostic property of such cross-lingual sentence representations and their application in multi-lingual nlp applications. the newly constructed multilingual corpus in this paper can be used as a new evaluation benchmark for future cross-lingual representation learning research. we plan to release the data set to the research community.