Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM (2023)

Abstract: The NLP community recently saw the release of a new large open-access multilingual language model, BLOOM (BigScience et al., 2022) covering 46 languages. We focus on BLOOM’s multilingual ability by evaluating its machine translation performance across several datasets (WMT, Flores-101 and DiaBLa) and language pairs (high- and low-resourced). Our results show that 0-shot performance suffers from overgeneration and generating in the wrong language, but this is greatly improved in the few-shot setting, with very good results for a number of language pairs. We study several aspects including prompt design, model sizes, cross-lingual transfer and the use of discursive context.

## introduction large language models (llms) trained at scale with simple objectives have been found to achieve results that match dedicated systems on numerous nlp tasks @xcite , as long as tasks are formulated as text generation though "prompting" @xcite . llms' multi-task performance can even be improved with "instruction" fine-tuning @xcite , few-shot priming, and better strategies to select or learn prompts @xcite @xcite @xcite . in multilingual settings, their performance on machine translation (mt) tasks, as measured by automatic scores, is often close to state of the art, even when mostly trained on monolingual data @xcite . moreover, prompting-based mt offers the prospect of better control of outputs, e.g. in terms of quality, style and dialect @xcite . however, these abilities remain poorly understood, as llm analyses primarily focus on their multitask rather than multilingual ability (see however @xcite @xcite ), which we discuss in section 2). in this work, we focus on the mt performance of bloom @xcite , a (family of) open-access multilingual llm(s), designed and trained by the collaborative bigscience project. 1 our main aims are to (i) evaluate bloom's zeroand multi-shot behaviour, (ii) study the effect of prompt design, (iii) evaluate a diverse set of language pairs and (iv) assess its ability to use linguistic context. our main conclusions, which extend those in @xcite , are (i) 0-shot ability is blighted by overgeneration and generating in the wrong language, (ii) using few-shot improves both issues, with results much closer to state of the art across datasets and language pairs, (iii) there are clear transfer effects, with high scores for languages not officially seen in training, and successful transfer across language pairs via few-shot examples and (iv) although linguistic context does not lead to higher scores, there is evidence that bloom's translations are influenced by it. we release our code and translation outputs. 2 2 related work since the early attempts at using language models (lms) as multi-task learners @xcite , mt has been a task of choice to gauge lms' multilingual ability. results for the zero-and few-shot ability of lms were discussed for both gpt-2 and gpt-3 @xcite . these results have since been confirmed for other monolingual lms such as t5 @xcite and multilingual lms such as xglm @xcite , palm @xcite , and alex-atm @xcite . however, the focus of these studies has mainly been multi-task performance, with little analysis of mt results. moreover, results are often only for a few well-resourced language pairs (e.g. english-french and english-german) and the scores reported (mostly bleu) not always easy to compare. there are however a number of recent in-depth analyses of mt performance of llms, each focusing, like we do, on one specific lm. most discuss, as we do, the variation of performance with respect to prompt design and number of few-shots examples. this is the case for example of @xcite , who reanalyse palm's translations and @xcite , who focus on glm-130b, a bilingual (chinese and english) llm @xcite . consistent with our findings, these studies observe commandable zero-shot performance, with a great variation depending on prompt choices, which tends to diminish when more prompts are used. using more than 5-10 examples, however, seems to bring very little return. the choice of few-shot examples does make a difference, as also observed by @xcite in their evaluation of openai's gpt-3 @xcite . 3 the study considers a single prompt resembling our xglm-source+target prompt, but varies the strategy used to select examples, showing that prompting can effectively serve as a vehicle to perform local adaptation and to enforce terminological consistency. finally it is worth mentioning the preliminary evaluation of chatgpt in @xcite , and the more detailed one in @xcite , which confirms the strong translation abilities of this model, at least for "well-resourced" foot_3 language pairs. overall, all these studies contribute to a better understanding of the abilities of instruction-based mt, and provide complementary angles, with variation across tasks, domains, language pairs, settings (e.g. context-aware mt or translation-memory-based mt), as well as evaluation metrics (bleu, bleurt, comet) and protocols. in comparison, ours brings some additional observations related to mt performance across model sizes and for a large number of language pairs, as well as a new task (multilingual conversations). multilingual mt is also the subject of dedicated (monotask) architectures and training regimes. originally introduced in @xcite @xcite with limited language coverage, the latest versions of these approaches are able to handle hundreds of languages, including very low-resource language pairs @xcite @xcite . although we found that bloom is able to match this performance, given sufficient training data, we also see that it still lags behind for many languages pairs that are under-represented in its training data. ## bloom language model bloom is a large open-access multilingual model trained on 46 natural languages developed within the bigscience project @xcite . it is an auto-regressive language model designed to generate text to complete a user-entered text prefix, known as a prompt. it can be used for multiple tasks, including mt, question answering, etc. bloom was trained on 1.6tb of text (of which 30% english), from various sources, although 38% of the data, known as the roots corpus (laurenc ¸on et al., 2022), foot_4 is from oscar web data @xcite . the model is openly released on hug-gingface in multiple sizes, ranging from 560m to 176b parameters. 6 4 evaluating bloom on the mt task ## evaluation results our evaluation of bloom starts with a comparison across the three datasets and detection of major mt errors with a focus on wmt (section 5.1) and then we present more in-depth analyses of particular aspects: (i) using wmt, a comparative study of bloom model sizes (section 5.2) and prompts (section 5.3), (ii) using flores-101 an evaluation of more language pairs and cross-lingual few-shot transfer (section 5.4), and (ii) using diabla, a study of the use of linguistic context (section 5.5). ## conclusion we have evaluated bloom's mt performance across three datasets and multiple language pairs. while there remain problems of overgeneration and generating in the wrong language (particularly for 0-shot mt), mt quality is significantly improved in few-shot settings, closer to state-of-the-art results. low-resource mt remains challenging for some language pairs, despite the languages being in the training data, questioning what it means to be a bloom language. however, we see evidence for cross-lingual transfer for non-bloom languages and when using few-shot examples from other language pairs. finally, although using linguistic context does not give improvements with automatic metrics, there is evidence that discursive phenomena are taken into account.