Few-shot Learning with Multilingual Generative Language Models (2022)

Abstract: Large-scale generative language models such as GPT-3 are competitive few-shot learners. While these models are known to be able to jointly represent many different languages, their training data is dominated by English, potentially limiting their cross-lingual generalization. In this work, we train multilingual generative language models on a corpus covering a diverse set of languages, and study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters sets new state of the art in few-shot learning in more than 20 representative languages, outperforming GPT-3 of comparable size in multilingual commonsense reasoning (with +7.4% absolute accuracy improvement in 0-shot settings and +9.4% in 4-shot settings) and natural language inference (+5.4% in each of 0-shot and 4-shot settings). On the FLORES-101 machine translation benchmark, our model outperforms GPT-3 on 171 out of 182 directions with 32 training examples, while surpassing the official supervised baseline in 45 directions. We conduct an in-depth analysis of different multilingual prompting approaches, showing in particular that strong few-shot learning performance across languages can be achieved via cross-lingual transfer through both templates and demonstration examples.

## introduction large autoregressive language models such as gpt-3 can be adapted, via few-and zero-shot learning, to a wide range of tasks with significantly less cost than full fine-tuning @xcite . these models have been primarily developed for english. although the training data of gpt-3 contains a small percentage of non-english text (7%) allowing it to achieve some promising cross-lingual generalization, the model is almost exclusively deployed for use cases in english. multilingual masked and sequence-to-sequence language models have been studied, including mbert, xlm-r, mt5, and mbart @xcite @xcite @xcite . these models are typically fine-tuned on large amount of labeled data in downstream tasks. despite notable recent work at smaller scales @xcite and for domain-specific tasks @xcite , the multilingual few-shot learning capabilities of language models are less well understood. in this paper, we train four multilingual generative language models (up to 7.5 billion parameters), xglm's, and present a comprehensive study of multilingual zero-and in-context few-shot learning. we train the models using a large-scale corpus of 500b tokens that comprises 30 diverse languages, up-sampling the less-resourced languages to render a more balanced language representation. we evaluate the models on multiple multilingual natural language understanding (nlu) tasks, machine translation and a subset of english tasks demonstrated in @xcite . we found xglm demonstrate strong crosslingual capability where using english prompts together with non-english examples yields competitive zero-and few-shot learning performance. our largest model (xglm 7.5b ) achieves strong zeroand few-shot learning performance on language completion and inference tasks (e.g. xstorycloze: 65.4% 0-shot, 66.5% 4-shot; xnli: 46.3% 0-shot, 47.3% 4-shot). it also establishes a new state-ofthe-art on few-shot machine translation across a large number of language pairs in the flores-101 benchmark @xcite , significantly outperforming the gpt-3 model of comparable size (6.7 billion parameters). on the other hand, multilingual pre-training causes performance drop on english. on 8 english nlu tasks, xglm 7.5b underperforms gpt-3 6.7b by 10.9% on average in zero-shot learning. gpt-3 6.7b also surpasses xglm 7.5b in machine translation among several high-resource language pairs, including wmt-14 en↔fr, wmt-16 en↔de and wmt-19 en↔zh. we conduct an in-depth analysis of different multilingual prompting approaches by examining cross-lingual transfer through template and demonstration examples respectively. we show the non-english prompts sometimes yield unexpected low zero-and few-shot learning accuracy even if they are crafted by native speakers ( §4.3). both using the english template ( §4.4) and adding demonstration examples ( §4.5) provide effective remedy. however, using demonstration examples from another language often cannot further improve the zero-shot learning performance when a strong prompting language (e.g. engilsh) is used, which indicates room for improvement in cross-lingual pre-training and in-context transfer approaches. ## multilingual in-context learning we measure the performance of our multilingual language models on downstream tasks in different languages given the tasks and few-shot demonstrations specified via prompts without further parameter updates (appendix b). ## conclusion we introduce four multilingual generative language models (xglms) at different scales, and study their in-context few-and zero-shot learning capabilities. we show that the few-shot learning capability of xglm steadily improves as it scales. our largest model (7.5b parameters) sets a new state of the art for few-shot learning in more than 20 languages (including mid-and low-resource languages) on commonsense reasoning, nli and machine translation tasks. an in-depth analysis shows the models are highly cross-lingual, which leads to strong few-shot learning performance in non-english languages.