Language Models are Few-shot Multilingual Learners (2021)

Abstract: General-purpose language models have demonstrated impressive capabilities, performing on par with state-of-the-art approaches on a range of downstream natural language processing (NLP) tasks and benchmarks when inferring instructions from very few examples. Here, we evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages without any parameter updates. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones. Finally, we find the in-context few-shot cross-lingual prediction results of language models are significantly better than random prediction, and they are competitive compared to the existing state-of-the-art cross-lingual models and translation models.

## introduction the progress in language model (lm) pretraining @xcite @xcite @xcite @xcite @xcite has led to the possibility of conducting few-shot learning, that is, learning a new task using a small number of examples without any further training or gradient computation. few-shot learning alleviates the cost for extensive labeled data, which is beneficial since collecting high-quality labeled data is resource-intensive and expensive. it also reduces the cost for model fine-tuning, which requires tremendous gpu or tpu resources. fewshot learning can be seen as a one-for-all plugand-play computational model that can be applied to various natural language tasks, from sentiment analysis for text classification to story generation, provided only a small context @xcite ). ## few-shot multilingual learners first, we briefly define the notation of the input and output of the task, and then we introduce our method to design prompts for few-shot in-context learning. foot_0 ## baselines in this work, we compare the few-shot learning performance with other common approaches: zeroshot, zero-shot cross-task, and fine-tuning. ## conclusion this paper demonstrates the multilingual skills of pre-trained lms, gpt and t5, in conducting incontext learning without parameter updates. this work is our initial attempt to show the effectiveness of in-context learning in the multilingual and crosslingual setting. it covers four different languages and explores the possibility of conducting efficient inference on low-resource tasks. we find that lms can predict samples correctly, significantly better than the random prediction, in cross-lingual tasks with no training examples of the target languages. we would like to investigate further the applicability of this method to other tasks and languages in future work. figure 11 : the acc results on the cross-lingual setting, english-german (de) mtop dataset with gpt models. figure 12 : the f1 results on the cross-lingual setting, english-german (de) mtop dataset with gpt models. figure 13 : the acc results on the cross-lingual setting, english-spanish (es) mtop dataset with gpt models. figure 14 : the f1 results on the cross-lingual setting, english-spanish (es) mtop dataset with gpt models. figure 15 : the acc results on the cross-lingual setting, english-french (fr) mtop dataset with gpt models. figure 16 : the f1 results on the cross-lingual setting, english-french (fr) mtop dataset with gpt models. figure 17 : the acc results on the cross-lingual setting, english-spanish (es) multilingual nlu dataset with gpt models. figure 18 : the f1 results on the cross-lingual setting, english-spanish (es) multilingual nlu dataset with gpt models.