On the Cross-lingual Transferability of Monolingual Representations (2020)

Abstract: State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.

## introduction multilingual pre-training methods such as multilingual bert (mbert, @xcite have been successfully used for zero-shot cross-lingual transfer @xcite . these methods work by jointly training a * work done as an intern at deepmind. transformer model @xcite to perform masked language modeling (mlm) in multiple languages, which is then fine-tuned on a downstream task using labeled data in a single languagetypically english. as a result of the multilingual pre-training, the model is able to generalize to other languages, even if it has never seen labeled data in those languages. such a cross-lingual generalization ability is surprising, as there is no explicit cross-lingual term in the underlying training objective. in relation to this, @xcite hypothesized that: in this paper, we empirically test this hypothesis by designing an alternative approach that violates all of these assumptions. as illustrated in figure 1 , our method starts with a monolingual transformer trained with mlm, which we transfer to a new language by learning a new embedding matrix through mlm in the new language while freezing parameters of all other layers. this approach only learns new lexical parameters and does not rely on shared vocabulary items nor joint learning. however, we show that it is competitive with joint multilingual pre-training across standard zero-shot cross-lingual transfer benchmarks (xnli, mldoc, and paws-x). we also experiment with a new cross-lingual question answering dataset (xquad), which consists of 240 paragraphs and 1190 question-answer pairs from squad v1.1 @xcite translated into ten languages by professional translators. question answering as a task is a classic probe for language understanding. it has also been found to be less susceptible to annotation artifacts commonly found in other benchmarks @xcite . we believe that xquad can serve as a more comprehensive cross-lingual benchmark and make it publicly available at https://github. com/deepmind/xquad . our results on xquad show that the monolingual transfer approach can be made competitive with mbert by learning second language-specific transformations via adapter modules @xcite . our contributions in this paper are as follows: (i) we propose a method to transfer monolingual representations to new languages in an unsupervised fashion ( §2) 1 ; (ii) we show that neither a shared subword vocabulary nor joint multilingual training is necessary for zero-shot transfer and find that the effective vocabulary size per language is an important factor for learning multilingual models ( §3 and §4); (iii) we show that monolingual models learn abstractions that generalize across languages ( §5); and (iv) we present a new cross-lingual question answering dataset ( §4). ## cross-lingual transfer of monolingual representations in this section, we propose an approach to transfer a pre-trained monolingual model in one language l 1 (for which both task supervision and a monolingual corpus are available) to a second language l 2 (for which only a monolingual corpus is available). the method serves as a counterpoint to existing joint multilingual models, as it works by aligning new lexical parameters to a monolingually trained deep model. as illustrated in figure 1 , our proposed method consists of four steps: language-specific position embeddings. the basic approach does not take into account different word orders commonly found in different languages, as it reuses the position embeddings in l 1 for l 2 . we relax this restriction by learning a separate set of position embeddings for l 2 in step 2 (along with l 2 token embeddings). 3 we treat the [cls] symbol as a special case. in the original implementation, bert treats [cls] as a regular word with its own position and segment embeddings, even if it always appears in the first position. however, this does not provide any extra capacity to the model, as the same position and segment embeddings are always added up to the [cls] embedding. following this observation, we do not use any position and segment embeddings for the [cls] symbol. noised fine-tuning. the transformer body in our proposed method is only trained with l 1 embeddings as its input layer, but is used with l 2 embeddings at test time. to make the model more robust to this mismatch, we add gaussian noises sampled from the standard normal distribution to the word, position, and segment embeddings during the fine- ## experiments our goal is to evaluate the performance of different multilingual models in the zero-shot cross-lingual setting to better understand the source of their generalization ability. we describe the models that we compare ( §3.1), the experimental setting ( §3.2), and the results on three classification datasets: xnli ( §3.3), mldoc ( §3.4) and paws-x ( §3.5). we discuss experiments on our new xquad dataset in ## xquad: cross-lingual question answering dataset our exclusively on english). one possible explanation for this behaviour is that existing cross-lingual benchmarks are flawed and solvable at the lexical level. for example, previous work has shown that models trained on multinli-from which xnli was derived-learn to exploit superficial cues in the data @xcite . to better understand the cross-lingual generalization ability of these models, we create a new crosslingual question answering dataset (xquad). question answering is a classic probe for natural language understanding @xcite and has been shown to be less susceptible to annotation artifacts than other popular tasks @xcite . in contrast to existing classification benchmarks, extractive question answering requires identifying relevant answer spans in longer context paragraphs, thus requiring some degree of structural transfer across languages. xquad consists of a subset of 240 paragraphs and 1190 question-answer pairs from the development set of squad v1.1 foot_7 together with their translations into ten languages: spanish, german, greek, russian, turkish, arabic, vietnamese, thai, chinese, and hindi. both the context paragraphs and the questions are translated by professional human translators from gengo foot_8 . in order to facilitate easy annotations of answer spans, we choose the most frequent answer for each question and mark its beginning and end in the context paragraph using special symbols, instructing translators to keep these symbols in the relevant positions in their translations. appendix b discusses the dataset in more details. we show f 1 scores on xquad in ## discussion joint multilingual training. we demonstrate that sharing subwords across languages is not necessary for mbert to work, contrary to a previous hypothesis by @xcite . we also do not observe clear improvements by scaling the joint training to a large number of languages. rather than having a joint vs. disjoint vocabulary or two vs. multiple languages, we find that an important factor is the effective vocabulary size per language. when using a joint vocabulary, only a subset of the tokens is effectively shared, while the as such, we believe that future work should treat the effective vocabulary size as an important factor. ## related work unsupervised lexical multilingual representations. a common approach to learn multilingual representations is based on cross-lingual word embedding mappings. these methods learn a set of monolingual word embeddings for each language and map them to a shared space through a linear transformation. recent approaches perform this mapping with an unsupervised initialization based on heuristics @xcite or adversarial training @xcite , which is further improved through self-learning @xcite . the same approach has also been adapted for contextual representations @xcite . ## conclusions we compared state-of-the-art multilingual representation learning models and a monolingual model that is transferred to new languages at the lexical level. we demonstrated that these models perform comparably on standard zero-shot crosslingual transfer benchmarks, indicating that neither a shared vocabulary nor joint pre-training are necessary in multilingual models. we also showed that a monolingual model trained on a particular language learns some semantic abstractions that are generalizable to other languages in a series of probing experiments. our results and analysis contradict previous theories and provide new insights into the basis of the generalization abilities of multilingual models. to provide a more comprehensive benchmark to evaluate cross-lingual models, we also released the cross-lingual question answering dataset (xquad).