Limitations and Challenges of Unsupervised Cross-lingual Pre-training (2022)

Abstract: Cross-lingual alignment methods for monolingual language representations have received notable attention in recent years. However, their use in machine translation pre-training remains scarce. This work tries to shed light on the effects of some of the factors that play a role in cross-lingual pre-training, both for cross-lingual mappings and their integration in supervised neural models. The results show that unsupervised cross-lingual methods are effective at inducing alignment even for distant languages and they benefit noticeably from subword information. However, we find that their effectiveness as pre-training models in machine translation is severely limited due to their cross-lingual signal being easily distorted by the principal network during training. Moreover, the learned bilingual projection is too restrictive to allow said network to learn properly when the embedding weights are frozen.

## introduction unsupervised cross-lingual embeddings (cle) concern a group of methods that exploit latent similarities between word embeddings in different languages to generate their own bilingual dictionary or parallel corpus. fully unsupervised cross-lingual mappings have received notable attention due to being solely reliant on the distributional similarities of language continuous vector representations. however, semi-supervised cross-lingual pre-training methods, which require very small amounts of parallel corpora -and in some cases only a simple bilingual dictionary for initialization -tend to outperform their unsupervised counterparts @xcite @xcite . it is only in situations where no bilingual data exists that unsupervised techniques are preferable. in the case of languages with extensive written records where resources are plentiful, there is little reason to use a fully unsupervised method rather than one that takes advantage of a small bilingual dataset @xcite . even for low-resource languages for which a reduced number of corpora exist, it is exceedingly probable that some sort of translation dictionary that associates them with a more widely studied language is available. this leads us to the question: are unsupervised cross-lingual models useful at all? while they might not be the best performing tools in a realistic use case, drawing a hard line between unsupervised and semi-supervised cross-lingual mappings does not make much sense, because they are extremely similar processes. semi-supervised cross-lingual methods are just forfeiting the generation of a seed bilingual dictionary in favor of one provided by the user or other strategies based on back-translation. but their remaining steps are analogous to that of unsupervised cross-lingual projection methods: they both use the seed translation dictionary to align and project the monolingual subspaces in a hypothetical cross-lingual space. in the case of many unsupervised models, seed generation is actually an adaptation of these previous steps, where some assumption on the structures of the matrices is made in order to obtain an initial set of translation pairs. given that unsupervised and semi-supervised crosslingual strategies share so many traits, most improvements over unsupervised methods can be transferred to semi-supervised ones. this work aims to uncover some of the limitations of pre-training strategies based on unsupervised cross-lingual embeddings. these methods are fully dependent on intrinsic language similarities to operate, and therefore constitute a great vehicle to explore how different continuous representations may capture distinct linguistic and structural features. previous research has already taken advantage of this property to analyze the behavior of language representation spaces @xcite . in this work, we consider three different approaches to unsupervised cross-lingual embedding projection, and explore their interaction with different linguistic characteristics and model features, such as subword encoding and vector space structure. the resulting evaluations are put into context to propose potential improvements to language representation strategies as a whole. ## related work modern mapping-based cross-lingual embeddings, which are sometimes also called projectionbased embeddings, have become very popular by outperforming earlier mapping methods and many supervised cross-lingual methods that are dependent on segment-level alignment. as many initial mapping-based approaches @xcite , they require monolingual embeddings, but often also a small parallel corpus or a bilingual seed translation dictionary to perform the initial alignment. however, some do not need any parallel signal at all, as they can also perform the original alignment relying only in estimations based on structural similarities between the monolingual vector spaces @xcite . another family of cross-lingual word embeddings are the so-called pseudo-bilingual word embeddings @xcite . these approaches use a concatenation of large monolingual corpora and integrate it with some amount explicit bilingual information -such as replacing translation pairs in the dataset @xcite or substituting tokens based on their semantic cluster @xcite -. both mapping-based and pseudo-bilingual word embeddings are especially useful for prototyping in extremely low-resource language models when no parallel data is available. however, cross-lingual embeddings have not been particularly effective in deep neural network pre-training, and often do not seem to represent a significant improvement over using off-the-shelf monolingual embeddings @xcite . in contrast, cross-lingual language models such as bert @xcite have fared better. some approaches try to create a universal neural language model for all the languages included in a final multilingual system @xcite . a very successful alternative to these strategies has also been proposed by @xcite , who create a cross-lingual neural language model by training a masked language model @xcite using a shared vocabulary between languages and subsampling frequent outputs as per @xcite . @xcite refine this masked language model (mlm) by introducing an explicit cross-language training objective, creating a cross-lingual masked language model (cmlm). recent research in @xcite has obtained top-of-the-line results by using a large-scale cmlm and training the final supervised model using a joint optimization objective @xcite that aims to maintain the original distribution of the cmlm while maximizing translation performance. ## unsupervised cross-lingual pre-training 3.1 unsupervised cross-lingual embeddings this work explores cross-lingual pre-training through three particular unsupervised crosslingual embedding methods: vecmap @xcite , muse @xcite and embeddings trained over multilingual corpora. both vecmap and muse are projection-based cross-lingual embeddings. projectionbased cle generate an alignment between monolingual word embeddings, subsequently projecting them into a common representation space that facilitates a direct mapping between the distribution of both embeddings. the initial alignment is commonly produced using a seed dictionary of translation pairs. however, in some cases these translation pairs are estimated by the cross-lingual method itself. this capacity to generate a common vector space of aligned embeddings with no bilingual signal defines fully unsupervised cross-lingual embeddings. the specifics behind alignment and projection are also dependent on the general topology of the embeddings that are to be mapped. in this work, all experiments are performed over word2vec embeddings, which offers two possible topologies: continuous bag-of-words (cbow) and skip-gram. the former trains a shallow network to predict a word given an input context, while the latter learns to predict a context window from an input word. in this work, only skip-gram is used for all experiments, particularly skip-gram with negative sampling (sgns) @xcite . for vecmap, @xcite assume that word translations have approximately identical vectors of monolingual similarity distribution. the proposed method operates on top of this idea, adding empirically motivated enhancements that make the procedure more robust. in contrast, muse @xcite uses adversarial training to create a generator network able to project word vectors from each monolingual embedding in a way such that it is very difficult to distinguish the space to which they originally belonged, thus achieving a common mapping between both embeddings. another approach to cross-lingual embeddings explored in this work are embeddings trained over multilingual corpora. the model proposed in this section is trained on corpora with no alignment or contextual proximity. the procedure is remarkably simple: two monolingual corpora in different languages are concatenated, and a word embedding is trained over the resulting multilingual text. just as in the previous two methods, the result is a bilingual vector space that tends to group translation candidates close to each other. therefore, it can also be used as bilingual pre-training for a translation model in the task studied in this work. this approach serves as a baseline to determine how much of the alignment achieved from crosslingual embeddings is innate to the distribution of both languages and can be extracted with no mapping procedures. ## conclusions this work makes use of fully unsupervised cross-lingual models to explore some of the factors that affect the performance of cross-lingual methods. the experimental results showcase agreement with prior publications regarding the usefulness of subword information in crosslinguality, and suggest that character-level encoding might be especially relevant for language pairs of low similarity. moreover, reveal that phylogenetic relatedness should not be directly taken to dictate cross-lingual performance, and that purely semantic similarity is not the only typological feature captured by projection-based mappings. cross-lingual transfer remains ineffective even if the structure of the pre-trained encoder and decoder is fixed during fine-tuning, which is indicative of the degeneration of cross-lingual projections in supervised training, and the limited scope of pre-trained embeddings. current pre-training approaches that rely on joint optimization appears to be the most promising approach going forward. future research may benefit from designing strategies that adapt cross-mapping methods to different language pairs according to their features, and that are able to capture typological information that is currently lost. model initialization seems to be a fundamental factor that is able to limit machine translation ceiling, especially for long distance pairs.