BLOOM +1: Adding Language Support to BLOOM for Zero-Shot Prompting (2023)

Abstract: The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. To extend the benefits of BLOOM to other languages without incurring prohibitively large costs, it is desirable to adapt BLOOM to new languages not seen during pretraining. In this work, we apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. We find language adaptation to be effective at improving zero-shot performance in new languages. Surprisingly, we find that adapter-based finetuning is more effective than continued pretraining for large models. In addition, we discover that prompting performance is not significantly affected by language specifics, such as the writing system. It is primarily determined by the size of the language adaptation data. We also add new languages to BLOOMZ, which is a multitask finetuned version of BLOOM capable of following task instructions zero-shot. We find including a new language in the multitask fine-tuning mixture to be the most effective method to teach BLOOMZ a new language. We conclude that with sufficient training data language adaptation can generalize well to diverse languages. Our code is available at https://github.com/bigscience-workshop/multilingual-modeling .

## introduction although access to transformer-based language models has expanded greatly over the past several years @xcite @xcite @xcite , these technologies are overwhelmingly concentrated in a few high resource * corresponding author: contact.yong@brown.edu languages @xcite . @xcite , the largest publicly available multilingual language model to date with 176b parameters, covers only 46 natural languages and even excludes high-resource languages such as korean and russian which has tens of millions of speakers. this limitation was driven by a number of factors, most notably only considering languages for which the community had enough expertise to manually validate the data quality @xcite , deduplicate and remove personally identifiable information @xcite and had sufficient access to licensed unlabeled text @xcite . all of these factors are contingent facts about the group that trained the model, and leave open the idea that other researchers could contribute more languages. as regularly retraining such a model is prohibitively expensive, the question of whether this model can be productively adapted to understand additional languages after training becomes pressing. we hypothesize that language adaptation scenario is especially interesting for low-resource languages that would benefit from knowledge transfer. therefore, we adapt bloom models to support eight new languages (german, russian, bulgarian, thai, turkish, greek, korean, and guarani) in the resource-constrained settings, where we only use a limited amount of samples (maximum 100k samples) for each language. we evaluate their zeroshot prompting on various nlu tasks after adaptation. the new languages cover both seen and unseen scripts in the pretraining data, and they differ in their language families and word orders. we benchmark existing language adaptation methods, such as continued pretraining and mad-x @xcite , as well as a state-of-the-art parameterefficient transfer learning method, (ia) 3 @xcite . current work on adapting large multilingual models has mostly explored continued pretraining @xcite @xcite ) of eleutherai's gpt-j-6b @xcite . moreover, @xcite showed that continued pretraining outperforms other strategies for adapting small/medium-sized language models (i.e., models with fewer than one billion parameters). however, our experiments demonstrate that, for large language models such as bloom with comparable sizes to gpt-j-6b, continued pretraining underperforms adapters under a resourceconstrained setting. in addition, our work focuses on studying the effects of language adaptation on prompting, which has been underexplored in previous language adaptation work @xcite @xcite . prompting can benefit many languages that lack large amounts of labeled data as it allows language models to generalize to a wide range of tasks with significantly less training cost and data than full finetuning @xcite . ## related work language adaptation language adaptation enables pretrained language models to support languages outside of their pretraining data. most works investigating language adaptation consider masked language models such as mbert @xcite and xlm-r @xcite ) that are pretrained on 100+ languages. language adaptation approaches can be broadly categorized into three categories: (1) continued pretraining of the model (restricted to the embedding layer training only in some cases) @xcite @xcite @xcite ; (2) training of language-specific adapters @xcite @xcite @xcite @xcite for the target language; and closest work to our benchmarking efforts is @xcite study on different approaches (i.e., continued pretraining, vocabulary expansion and adapter layers) to extend the xlm-r model to 30 new languages on token-level clas-sification tasks. they conclude that continued pretraining is the most promising direction. however, the cost of such pretraining will grow with the size of the pretrained model and can be prohibitive for many researchers working with low-resource languages. our results also show that continued pretraining does not necessarily bring a prompting performance gain for larger language models. multilingual prompting prompting reformulates nlp tasks into masked or generative language modeling problem, depending on the models' pretraining objective. @xcite and @xcite show that finetuning xlm-r on clozestyle prompts yield better performance than standard finetuning under a low-resource regime for xnli. on the other hand, @xcite find that standard finetuning of xlm-r outperforms prompt-based learning for sentiment prediction in low-resource indonesian dialects. some work shows that multitask prompt-based training on a variety of tasks and english or translated prompts improves zero-shot cross-lingual and cross-task performance @xcite . multilingual prompt-based learning can also be achieved without performing gradient updates for downstream tasks. for instance, @xcite demonstrate success in prompting gpt-like pretrained models with in-context learning for nlu tasks, using either english or translated prompt templates. @xcite find that when language models scale up, they can perform better multilingual chain-of-thought reasoning. ## conclusion we compare the compute-performance trade-off of different language adaptation strategies for extending bloom of various sizes to new languages. contrary to previous work, we find that adapterbased strategies best adapt larger bloom models for prompting under low-resource settings. we also investigate different language adaptation factors such as the size of language adaptation data and capacity of adapters. finally, we investigate the relationship between language adaptation and instruction tuning using the bloomz model, where we find including new languages during instruction tuning most effective.