m GPT : Few-Shot Learners Go Multilingual (2024)

Abstract: This paper introduces mGPT, a multilingual variant of GPT-3, pretrained on 61 languages from 25 linguistically diverse language families using Wikipedia and the C4 Corpus. We detail the design and pretraining procedure. The models undergo an intrinsic and extrinsic evaluation: language modeling in all languages, downstream evaluation on cross-lingual NLU datasets and benchmarks in 33 languages, and world knowledge probing in 23 languages. The in-context learning abilities are on par with the contemporaneous language models while covering a larger number of languages, including underrepresented and low-resource languages of the Commonwealth of Independent States and the indigenous peoples in Russia. The source code and the language models are publicly available under the MIT license.

## introduction the advent of the transformer architecture @xcite has facilitated the development of various language models (lms; @xcite . although the well-established ''pretrain & finetune'' paradigm has led to rapid progress in nlp @xcite , it imposes several limitations. finetuning relies on an extensive amount of labeled data. collecting high-quality labeled data for new tasks and languages is expensive and resource-consuming @xcite . lms can learn spurious correlations from finetuning data @xcite and demonstrate inconsistent generalization, catastrophic forgetting, or brittleness to finetuning data order @xcite . last but not least, finetuning requires additional computational resources and, therefore, * work done while at salutedevices. † now at university of oslo. the latest approaches address these limitations with zero-shot and few-shot learning, performing a task with lm scoring or conditioning on a few demonstration examples without parameter updates @xcite . autoregressive lms adopted via these paradigms have been widely applied in many nlp tasks @xcite , notably in cross-lingual knowledge transfer @xcite and low-resource language scenarios @xcite . however, model development for underrepresented typologically distant and low-resource languages @xcite @xcite and cross-lingual generalization abilities of autoregressive lms @xcite have been left understudied. this paper presents mgpt, a multilingual version of gpt-3 @xcite available in 1.3b (mgpt 1.3b ) and 13b (mgpt 13b ) parameters. we aim (i) to develop a large-scale multilingual autoregressive lm that inherits the gpt-3's generalization benefits and (ii) to increase the linguistic diversity of multilingual lms, making the first attempt to address languages of the commonwealth of independent states (cis) and under-resourced languages of the indigenous peoples in russia. we pretrain mgpt in 61 languages from 25 language families on wikipedia and colossal clean crawled corpus (c4; @xcite . we analyze the mgpt's performance on various intrinsic and extrinsic tasks and compare it with the contemporaneous generative lms. ## discussion our key takeaways on pretraining and evaluating large-scale multilingual autoregressive lms are summarized below. ## conclusion we introduce the mgpt 1.3b and mgpt 13b models, which cover 61 languages from linguistically diverse 25 language families. our model is one of the first autoregressive lms for economically endangered and underrepresented cis and low-resource languages. the architecture design choices are based on the preliminary tokenization experiments and their perplexity-based evaluation. the model evaluation experiments include language modeling, standardized cross-lingual nlu datasets and benchmarks, world knowledge probing, and social bias tasks. we evaluate the in-context learning abilities in zero and few-shot settings with a negative log-likelihood probability. we present a detailed analysis of the model performance, limitations, and ethical considerations. despite the space for further quality growth and solving the highlighted limitations, the model shows significant potential and can become the basis for developing generative pipelines for languages other than english, especially the low-resource ones. this initiative has been developed for 23 diverse languages through collaboration with the nlp community. we hope to benefit cross-lingual knowledge transfer, annotation projection, and other potential applications for economically challenged and underrepresented languages and diversify the research field by shifting from the anglo-centric paradigm.