Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in P a LM ’s Translation Capability (2023)

Abstract: Large, multilingual language models exhibit surprisingly good zero- or few-shot machine translation capabilities, despite having never seen the intentionally-included translation examples provided to typical neural translation systems. We investigate the role of incidental bilingualism—the unintentional consumption of bilingual signals, including translation examples—in explaining the translation capabilities of large language models, taking the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method approach to measure and understand incidental bilingualism at scale. We show that PaLM is exposed to over 30 million translation pairs across at least 44 languages. Furthermore, the amount of incidental bilingual content is highly correlated with the amount of monolingual in-language content for non-English languages. We relate incidental bilingual content to zero-shot prompts and show that it can be used to mine new prompts to improve PaLM’s out-of-English zero-shot translation quality. Finally, in a series of small-scale ablations, we show that its presence has a substantial impact on translation capabilities, although this impact diminishes with model scale.

## introduction recent work has shown that large language models (llms) exhibit impressive capabilities in performing various natural language generation tasks, even in the zero-shot paradigm. in particular, such models have shown interesting machine translation (mt) capabilities @xcite @xcite -especially when translating into english, despite never having been explicitly and intentionally exposed to translation data in the way their supervised counterparts are. this raises the question: where do these translation capabilities come from? we hypothesize that the translation capabilities of llms connect to incidental bilingualism: the unintentional consumption of bilingual text within a single training instance. to test this hypothesis, we take palm (chowdhery et al., 2022)-a 540billion parameter transformer language model-as a case study. we first conduct a large-scale analysis of its training data in order to characterize the nature and quantity of bilingual text, then perform experiments to assess the impact of this text on translation performance. to measure incidental bilingualism at scale, we develop a processing pipeline that alternates between quantitative and qualitative analysis ( §3): first detect bilingual versus monolingual text using a language tagger, then qualitatively analyze the nature of bilingual text, and finally measure the amount of translation data within bilingual instances. our analysis spans 44 languages, for which we study bilingualism paired with english. our findings are: after establishing that both bilingual and translation content are incidentally consumed during palm's training, we study how they connect to its mt capabilities ( §4). we run a series of training and prompting experiments and found that: translation capabilities of llms large-scale generative language models, such as gpt-3 @xcite , @xcite , and xglm @xcite have been shown to exhibit translation capabilities, despite not being explicitly trained to translate. these capabilities are surprisingly strong, particularly when translating into english with few-shot examples. one explanation for this behavior is that it results from incidental multitask learning @xcite . this hypothesis has not been explored for mt, where recent work has mostly focused on improving llm translation capabilities by optimizing few-shot prompting strategies @xcite . rather than trying to improve translation quality for llms, our goal is to understand where their translation abilities stem from by tracing them back to the properties of the pretraining data. ## measuring & understanding incidental bilingualism we introduce a mixed-method approach (creswell and clark, 2017; @xcite to measure and understand incidental bilingualism-the unintentional consumption of bilingual signalsat scale. since we expect bilingual signals to be rare, we explore the huge data space by alternating between quantitative and qualitative steps, with results from each step complementing and informing one another (figure 1 ). the quantitative steps play the role of inducing a smaller-scale focus space to study, while the qualitative steps provide insights into the nature of bilingual signals. preliminaries palm's pretraining dataset consists of 780 billion tokens from a mixture of multilingual sources (social media conversations (50%), filtered webpages (27%), and wikipedia (4%)), presumably english sources (books (13%) and news articles (1%)), and source code (5%). palm was trained on 2,048-subword-token examples formed by concatenating and truncating documents. as palm is a multi-source lm, a document may be a web page, a book, or a conversation, depending on the source. our primary units for data analysis are instances we created by splitting training examples along document boundaries. as such, each instance is either a complete document or a contiguous fragment of one, up to 2,048 tokens in length. a more detailed discussion of instances is given in appendix a. we study bilingualism between english and 44 other languages. we choose language pairs that: a) are supported by our language identification models, and b) have flores-101 @xcite evaluation data. we divide languages into high, medium, and low-resource groups according to their monolingual instance counts, as shown below: ## analyzing the impact of bilingualism we analyze the impact of bilingualism on the translation capabilities of palm with a series of mt experiments on the flores-101 @xcite evaluation set, which provides translations of a common set of english wikipedia sentences into all of our 44 languages. we report results on the 1,012 sentence devtest set. we use the 997 sentence dev set primarily as a source of randomlydrawn exemplars when reporting 5-shot results. we report bleu @xcite for into-english translation and chrf @xcite for out-of-english translation, both computed by sacrebleu @xcite with default settings. for llmbased translation, we follow the template from vilar et al. (2022) unless stated otherwise: ## conclusion we explore the role of incidental bilingualism-the unintentional consumption of bilingual signalsin palm's translation capabilities. we introduce a mixed-method approach that alternates between quantitative and qualitative analyses to measure and understand incidental bilingualism at scale by processing 780 billion tokens. our work shows that palm consumes a significant amount of bilingual text: 1.4% of training instances in natural language are bilingual. at the same time, it is naturally exposed to translation signals, having seen more than 30 million translation pairs in 44 languages paired with english. furthermore, we extrinsically evaluate the quality of these translations, showing that they can be used to train supervised models that roughly match the quality of equal amounts of wmt data. finally, we show that incidental bilingualism connects to the machine translation capabilities of palm. first, we show that data-driven prompts extracted from incidental translations can improve the zero-shot abilities of palm when translating out of english by 14 chrf on average. second, we provide empirical evidence that bilingual and translation signals can partially explain the translation capabilities of smaller-scale llms.