A Call for More Rigor in Unsupervised Cross-lingual Learning (2020)

Abstract: We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them. An existing rationale for such research is based on the lack of parallel data for many of the world’s languages. However, we argue that a scenario without any parallel data and abundant monolingual data is unrealistic in practice. We also discuss different training signals that have been used in previous work, which depart from the pure unsupervised setting. We then describe common methodological issues in tuning and evaluation of unsupervised cross-lingual models and present best practices. Finally, we provide a unified outlook for different types of research in this area (i.e., cross-lingual word embeddings, deep multilingual pretraining, and unsupervised machine translation) and argue for comparable evaluation of these models.

## introduction the study of the connection among human languages has contributed to major discoveries including the evolution of languages, the reconstruction of proto-languages, and an understanding of language universals @xcite . in natural language processing, the main promise of multilingual learning is to bridge the digital language divide, to enable access to information and technology for the world's 6,900 languages @xcite . for the purpose of this paper, we define "multilingual learning" as learning a common model for two or more languages from raw text, without any downstream task labels. common use cases include translation as well as pretraining multilingual representations. we will use the term interchangeably with "cross-lingual learning". ## background in this section, we briefly review existing work on ucl, covering cross-lingual word embeddings ( §2.1), deep multilingual pre-training ( §2.2), and unsupervised machine translation ( §2.3). ## motivating fully unsupervised learning in this section, we challenge the narrative of motivating ucl based on a lack of parallel resources. we argue that the strict unsupervised scenario cannot be motivated from an immediate practical perspective, and elucidate what we believe should be the true goals of this research direction. ## methodological issues in this section, we describe methodological issues that are commonly encountered when training and evaluating unsupervised cross-lingual models and propose measures to ameliorate them. ## bridging the gap between unsupervised cross-lingual learning flavors the three categories of ucl ( §2) have so far been treated as separate research topics by the community. in particular, cross-lingual word embeddings have a long history @xcite , while deep multilingual pretraining has emerged as a separate line of research with its own best practices and evaluation standards. at the same time, unsupervised machine translation has been considered a separate problem in its own right, where cross-lingual word embeddings and deep multilingual pretraining have just served as initialization techniques. while each of these families have their own defining features, we believe that they share a strong connection that should be considered from a more holistic perspective. in particular, both cross-lingual word embeddings and deep mul-tilingual pretraining share the goal of learning (sub)word representations, and essentially differ on whether such representations are static or contextdependent. similarly, in addition to being a downstream application of the former, unsupervised machine translation can also be useful to develop other multilingual applications or learn better crosslingual representations. this has previously been shown for supervised machine translation @xcite and recently for bilingual lexicon induction @xcite . in light of these connections, we call for a more holistic view of ucl, both from an experimental and theoretical perspective. evaluation. most work on cross-lingual word embeddings focuses on bilingual lexicon induction. in contrast, deep multilingual pretraining has not been tested on this task, and is instead typically evaluated on zero-shot cross-lingual transfer. we think it is important to evaluate both familiescross-lingual word embeddings and deep multilingual representations-in the same conditions to better understand their strengths and weaknesses. in that regard, @xcite recently showed that deep pretrained models are much stronger in some downstream tasks, while cross-lingual word embeddings are more efficient and sufficient for simpler tasks. however, this could partly be attributed to a particular integration strategy, and we advocate for using a common evaluation framework in future work to allow a direct comparison between the different families. theory. from a more theoretical perspective, it is still not well understood in what ways crosslingual word embeddings and deep multilingual pretraining differ. while one could expect the latter to be learning higher-level multilingual abstractions, recent work suggests that deep multilingual models might mostly be learning a lexical-level alignment @xcite . for that reason, we believe that further research is needed to understand the relation between both families of models. ## recommendations to summarize, we make the following practical recommendations for future cross-lingual research: ## conclusions in this position paper, we review the status quo of unsupervised cross-lingual learning-a relatively recent field. ucl is typically motivated by the lack of cross-lingual signal for many of the world's languages, but available resources indicate that a scenario with no parallel data and sufficient monolingual data is not realistic. instead, we advocate for the importance of ucl for scientific reasons. we also discuss different monolingual and crosslingual training signals that have been used in the past, and advocate for carefully reporting them to enable a meaningful comparison across different approaches. in addition, we describe methodological issues related to the unsupervised setting and propose measures to ameliorate them. finally, we discuss connections between cross-lingual word embeddings, deep multilingual pre-training, and unsupervised machine translation, calling for an evaluation on an equal footing. we hope that this position paper will serve to strengthen research in ucl, providing a more rigorous look at the motivation, definition, and methodology. in light of the unprecedented growth of our field in recent times, we believe that it is essential to establish a rigorous foundation connecting past and present research, and an evaluation protocol that carefully controls for the use of parallel data and assesses models in diverse, challenging settings.