From Zero to Hero: O n the Limitations of Zero-Shot Language Transfer with Multilingual T ransformers (2020)

Abstract: Massively multilingual transformers (MMTs) pretrained via language modeling (e.g., mBERT, XLM-R) have become a default paradigm for zero-shot language transfer in NLP, offering unmatched transfer performance. Current evaluations, however, verify their efficacy in transfers (a) to languages with sufficiently large pretraining corpora, and (b) between close languages. In this work, we analyze the limitations of downstream language transfer with MMTs, showing that, much like cross-lingual word embeddings, they are substantially less effective in resource-lean scenarios and for distant languages. Our experiments, encompassing three lower-level tasks (POS tagging, dependency parsing, NER) and two high-level tasks (NLI, QA), empirically correlate transfer performance with linguistic proximity between source and target languages, but also with the size of target language corpora used in MMT pretraining. Most importantly, we demonstrate that the inexpensive few-shot transfer (i.e., additional fine-tuning on a few target-language instances) is surprisingly effective across the board, warranting more research efforts reaching beyond the limiting zero-shot conditions.

## introduction and motivation labeled datasets of sufficient size support supervised learning in nlp. the notorious tediousness, subjectivity, and cost of linguistic annotation @xcite @xcite , coupled with plethora of structurally different nlp tasks, lead to existence of such datasets only for a handful of resource-rich languages @xcite @xcite . this data scarcity renders the need for effective cross-lingual transfer strategies: how can we exploit abundant labeled data from resource-rich languages to make the current mainstay of cross-lingual transfer in nlp are approaches based on continuous crosslingual representation spaces such as cross-lingual word embeddings (clwes) @xcite and, most recently, massively multilingual transformer networks (mmts), pretrained on multilingual corpora with language modeling (lm) objectives @xcite @xcite . the latter have de facto become the default language transfer paradigm, with multiple studies reporting their unparalleled transfer performance @xcite @xcite @xcite . ## background and related work for completeness, we provide a brief overview of 1) cross-lingual transfer approaches, with a focus on 2) massively multilingual transformer (mmt) models, and then 3) position our work w.r.t. other studies that examine different properties of mmts. ## zero-shot transfer: analyses we first address q1 and q2 (see §1): we conduct zero-shot language transfer experiments for five different tasks and analyze the factors behind the varying performance drops across target languages. ## conclusion research on zero-shot language transfer in nlp is motivated by inherent data scarcity: the fact that most languages have no annotated data for most nlp tasks. massively multilingual transformers (mmts) have recently been praised for their zero-shot transfer capabilities that mitigate the data scarcity issue. in this work, we have demonstrated that, similar to earlier language transfer paradigms, mmts perform poorly in zero-shot transfer to distant target languages, and to languages with smaller monolingual corpora available for exploitation in mmt pretraining. we have presented a detailed empirical analysis of factors affecting zero-shot transfer performance of mmts across diverse tasks and languages. our results have revealed that structural language similarity determines the transfer success for lower-level tasks like pos-tagging and dependency parsing; on the other hand, the pretraining corpora size of the target language is crucial for explaining transfer results for higher-level language understanding tasks, such as question answering and natural language inference. finally and most importantly, we have shown that the mmt potential on distant and low-resource target languages can be quickly unlocked if they are provided a handful of annotated instances in the target language. this finding provides a strong incentive for intensifying future research efforts that focus on cheap or naturally occurring supervision @xcite @xcite , quick and simple annotation procedure, and the more effective few-shot transfer learning setups.