BUFFET : Benchmarking Large Language Models for Few-shot Cross-lingual Transfer (2024)

Abstract: Despite remarkable advancements in few-shot generalization in natural language processing, most models are developed and evaluated primarily in English. To establish a rigorous and equitable evaluation framework for few-shot cross-lingual transfer, we introduce a new benchmark, called BUFFET, which unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions. Using BUFFET, we perform thorough evaluations of ten state-of-the-art multilingual large language models with different transfer methods, namely in-context learning and fine-tuning. Our findings reveal significant room for improvement in few-shot in-context cross-lingual transfer. Strong multilingual pre-trained or instruction-tuned models such as BLOOM or ChatGPT often lag behind much smaller mT5-base models given the same number of few-shot samples, particularly in low-resource languages. Our analysis suggests avenues for future research in few-shot cross-lingual transfer.

## introduction recent advances in nlp primarily focus on english @xcite . as there is a shortage of adequate training data for most languages worldwide @xcite , zero-shot cross-lingual transfer @xcite is an active research area. this involves training models on high-resource languages like english, and then directly applying them to new languages without any training data in the target language. this approach often results in limited success when the target language is significantly different from the source language, motivating recent efforts to adapt models to a task in a new language using a limited number of training data in the target language. such few-shot transfer often boosts performance, especially in languages that are dissimilar to the source language @xcite . although there has been significant research on few-shot learning in english, employing techniques like in-context learning that do not necessitate parameter updates @xcite , few-shot cross-lingual transfer is still underexplored @xcite . while several recent work demonstrates the effectiveness of in-context learning in non-english languages on specific target tasks @xcite , it remains uncertain how well in-context learning performs in comparison to widely-employed finetuning-based transfer, particularly in a comparable setup involving diverse tasks and languages. to comprehensively assess the capabilities of language models (lms) for few-shot cross-lingual transfer, we introduce buffet: benchmark of unified format few-shot transfer evaluation (figure 1 ) to enable rigorous evaluations and advance research on few-shot cross-lingual transfer. similar to a rich buffet, buffet curates a diverse mix of tasks: 15 different tasks-including classification, structured prediction, and natural language generation-across 54 languages. following prior work @xcite , for each task, models can only use k few-shot demonstrations (e.g., @xmath0, while additional resources such as english training data or task instructions (in english or the target language) are also available. buffet has several unique characteristics that are not present in prior multi-task multilingual benchmarks: on this new benchmark, we extensively evaluate the current state-of-the-art multilingual large language models (llms), including mt5 @xcite , mt0 @xcite , bloom @xcite , bloomz @xcite , and chatgpt @xcite , using both fine-tuning and in-context learning approaches. we also evaluate recent englishcentric powerful open lms such as llama-2 @xcite and @xcite . in particular, buffet enables us to investigate the following research questions: (rq1) is in-context learning competitive with fine-tuning in few-shot cross-lingual transfer? notably, given the same small numbers of examples in the target languages, in-context learning on llms often under-performs much smaller specialized mt5-base models (figure 1 bottom left). (rq2) how well do different transfer methods perform across tasks and languages? the performance gap between in-context learning and fine-tuning baselines is more significant in underrepresented languages (figure 1 bottom center). however, these llms perform well on generative tasks where a smaller task-specific lm struggles, demonstrating their superiority in generating fluent text for across languages. meanwhile, although recent strong open lms such as llama2 or mistral demonstrate strong performance in high-resource languages, possibly benefiting from a small amount of multilingual pre-training data @xcite , they often show significant drops in performance on other languages less represented in english-centric pre-training corpora. ## background and related work while few-shot cross-lingual transfer methods such as fine-tuning and in-context learning have been investigated (section 2.1), limited research explores different methods under comparable conditions. we introduce buffet as a benchmark (section 2.2) to facilitate fair comparisons between models and learning methods. ## benchmark: buffet we introduce a new standardized few-shot cross-lingual evaluation benchmark: buffet (benchmark of unified format few-shot transfer evaluation). buffet unifies diverse nlp tasks and provides fixed sets of few-shot samples per task to facilitate fair comparisons ( ## conclusion and discussion we introduce buffet, a few-shot cross-lingual transfer benchmark that encompasses a diverse range of discriminative and generative tasks across many typologically diverse languages. while llms utilizing in-context learning excel in generation tasks, they are often surpassed by smaller finetuned models specifically trained for target tasks. our analysis sheds light on several important open questions for better multilingual instruction-tuning, and more balanced multilingual pre-training, and suggests the necessity of evaluating across lan-guages and tasks under comparable settings. 8 limitations selection of tasks. as the first step toward standardized evaluation for few-shot cross-lingual transfer, buffet focuses on popular discriminative tasks and some generative tasks, with wellstudied evaluation protocols and rich annotated resources. due to the lack of high-quality non-english annotated data, buffet does not include many datasets that require complex reasoning tasks. further exploration can expand these evaluations to more diverse and complex tasks, such as mtop @xcite or mgms8k @xcite , or knowledge-intensive tasks @xcite ). yet, it should be noted that high-quality generation or reasoning task data are often only available handful of resource-rich languages, which makes buffet-style comprehensive comparisons across languages difficult. we encourage the community to work towards diverse high-quality evaluation datasets in more world languages. hyper-parameter search or prompting. since our main focus is to benchmark different lms and learning methods in a comparable format, we do not explore sophisticated prompting methods or conduct task-or language-dependent hyperparameter searches. we anticipate that buffet will encourage the llm community to explore new methods to further improve in-context learning beyond english.