Prompting P a LM for Translation: Assessing Strategies and Performance (2023)

Abstract: Large language models (LLMs) that have been trained on multilingual but not parallel text exhibit a remarkable ability to translate between languages. We probe this ability in an in-depth study of the pathways language model (PaLM), which has demonstrated the strongest machine translation (MT) performance among similarly-trained LLMs to date. We investigate various strategies for choosing translation examples for few-shot prompting, concluding that example quality is the most important factor. Using optimized prompts, we revisit previous assessments of PaLM’s MT capabilities with more recent test sets, modern MT metrics, and human evaluation, and find that its performance, while impressive, still lags that of state-of-the-art supervised systems. We conclude by providing an analysis of PaLM’s MT output which reveals some interesting properties and prospects for future work.

## introduction large language models (llms) trained to predict the next token from a lengthy context have demonstrated impressive machine translation capabilities, despite being trained on corpora that are overwhelmingly english, with no intentionallyincluded parallel text. in this paper, we carry out an in-depth investigation into the translation capabilities of llms, testing different prompting strategies and carefully assessing the resulting performance. we study the recently-introduced palm model @xcite , a 540b-parameter decoder-only language model trained on a heavily english-centric, multilingual corpus. it has achieved the strongest mt results among llms trained on non-parallel multilingual corpora. to ensure a fair assessment of palm's mt capability, we begin with an exploration of example selection methods for use with fixed prompt templates. we vary both the pool from which examples are chosen and the method for choosing them, comparing standard random selection to knearest-neighbour (knn) selection that customizes prompts for specific inputs. figure 1 although @xcite report interesting results on low-resource and non-english language pairs, their most striking findings concern high-resource pairs. accordingly, we limit our investigation to french, german, and chinese translation to and from english. we evaluate sentencelevel translation quality using recommended practices for high-quality mt, specifically: (i) we use recent wmt test sets to guard against train/test data leakage, and to facilitate comparison with state-of-the-art (sota) mt systems; (ii) we use a sota automatic metric (bleurt) instead of bleu which has been demonstrated to be suboptimal for high-quality translations @xcite ; and (iii) we conduct an expert-based human evaluation with detailed categories to characterize the error patterns of the automatically generated translations. our contributions are as follows: ## related work inspired by the findings of @xcite ; @xcite , prompting strategies for llms have become a topic of intense interest, generating work across a broad spectrum of methods and applications @xcite . a basic distinction can be made between hard (explicit text) prompting such as we use, and soft prompting that seeks to learn embeddings @xcite , activations @xcite , or attention weights @xcite ) that condition the model to perform a desired task. the latter approach is more expressive and more efficient at inference time, but performance can be sensitive to initialization @xcite , and some techniques require modifications to the model. hard prompts have the advantage of being easy to interpret and modify. work in this area includes tools to facilitate development of handcrafted prompts @xcite ; algorithms to find optimal prompts through gradient-guided search @xcite or exhaustive search through labels @xcite or both labels and templates @xcite ; as well as studies on the effect of example order @xcite . hard prompts have also been used to analyze model capabilities @xcite , the role of data @xcite , and the nature of prompting itself @xcite . with few exceptions, e.g. @xcite @xcite , early approaches to hard prompting tended to condition on the task rather than the specific input. our knn approach for conditioning on the input was pioneered by @xcite , who used roberta embeddings to identify relevant gpt-3 prompts for sentiment, table-to-text, and qa tasks. they found that knn works better than a random-selection baseline, and that the advantage grows as the size of the (domain-controlled) example pool increases. work on prompting llms for mt began with the gpt-3 and palm papers @xcite , which adopted similar approaches, comparing 0, 1, and n-shot foot_0 random selection of independent sentence pairs from wmt training corpora, and testing on older french, german, and romanian wmt test sets traditionally used in ml, augmented in palm with @xmath0. for both models, performance increased with number of shots, and n-shot bleu scores were found to be competitive with previous unsupervised sota, and in some settings-particularly into english-supervised sota as well. in other early mt work, reynolds and mc-donell (2021) experimented with prompt templates for gpt-3, and found that 0-shot prompts with carefully-chosen templates can outperform n-shot prompts with sub-optimal templates. garcia and firat (2022) explored using prompts with mt5 @xcite to control output attributes such as formality, and also examine the effect of using promptlike natural-language tags during fine-tuning. @xcite proposed autoregressive prompting: concatenating only the first predicted word to a prompt and output prefix at each step. ## data we experiment with translation into and out of english for chinese, french and german. after english (78.0%), german (3.5%) and french (3.3%) are the two largest languages in palm's 780b token training corpus; chinese (0.4%) is the 15th largest, and it also represents an inherently more difficult translation task. to facilitate comparisons with recent sota systems, and to minimize the chance of overlap with palm's training corpus, we test on news data from the wmt 2021 evaluation campaign @xcite . since french was not included in wmt21, we use data from wmt14; apart from being older, these test sets are not purely source-original @xcite like the more recent ones. for prompt selection, we use three distinct pools: the full wmt training corpus for each language pair (wmt-full), the corresponding wmt development sets (wmt-dev), and a manually-curated "high-end" pool. sizes are shown in the high-end pool comes from websites containing bilingual articles that we judged to be professionally edited, with native or near-native quality in both languages. the articles are drawn from various domains (biography, business, commentary, culture, fashion, food, news, and obituary), with the news domain of the test sets comprising less than 50% for each language. we treat these articles as symmetrical, and use them as prompt sources in both translation directions. due to the non-literal nature of the translations, there is frequently no 1-1 correspondence between sentence pairs, so we extract aligned paragraphs for prompting. more detailed information about the high-end pool is provided in appendix b. ## experiments for compatibility with @xcite , we ran all experiments at the sentence level, translating each test sentence individually and in isolation from its context. this deprives palm of the ability to exploit the longer contexts it was exposed to during training, but it matches the operating mode of our baselines (including sota baselines), and facilitates evaluation. 6 we leave an exploration of potential gains from conditioning on longer histories to future work. in preliminary experiments, we varied the number of shots from 0 to 10, and found clear performance gains as we increased the number of shots, with diminishing returns after 5 sentence pairs (see appendix a). accordingly we report all results on the wmt pools in the 5-shot setting, where each shot is a single sentence pair, matching the configuration in @xcite . for the high-end pool, lacking 1-1 sentence alignments, we use 1-shot examples, where each shot is a single paragraph pair. this provides roughly the same quantity of text as 5-shot with sentences, although it creates a stylistic mismatch with our test setup, as we still translate on a sentence-by-sentene basis, as in the other conditions. when randomly selecting examples, we observed that there is little variability in automatic scores when selecting different samples 7 (see appendix c). for the results reported in this section, we let palm produce translations with 5 different seeds and we selected the run with the median bleurt score. translation time was some orders of magnitude longer than a dedicated translation system. following recent recommendations @xcite we favour neural metrics (bleurt in our case) over bleu, although we also report bleu scores for completeness. we use a cased version of bleurt @xcite ) that is based on @xcite . we use bleu as implemented in sacrebleu foot_7 @xcite , with zh tokenization for english-chinese, and 13a tokenization for all other languages. to perform human evaluation, we hired professional translators (7 for @xmath1, 5 for @xmath2, 4 for @xmath3, and 4 for @xmath4. this includes using the same error categories, severity levels and error weighting schema. as suggested in the study, we weight each major error with 5 and each minor error with 1, except for minor punctuation errors which get a score of 0.1. we depart from @xcite in using only a single annotator per segment, and in not imposing a limit of 5 errors per sentence. additionally, due to technical restrictions on the length of an evaluation session, we limited the mqm evaluation to the first 12 segments per document. ## analysis in this section we delve further into various aspects of palm's mt performance. ## conclusion we perform a careful assessment of the sentencelevel mt capabilities of palm, which we compare to sota and a current off the shelf (cots) mt system for three high-resource languages-german, chinese, and french-into and out of english, using the latest test sets from wmt. we chose to focus on a small set of high-resource language pairs in order to test the claims of the original palm paper, which are most striking for these pairs. the time and expense of performing high-quality human evaluations precluded a broader investigation. comparing knn and random strategies for selecting 5-shot translation examples to instantiate fixed prompt templates, we find that knn's potential advantage in identifying examples relevant to the source sentence is outweighed by its susceptibility to corpus noise. choosing examples randomly from small, high-quality pools works well, and performance appears to be independent of the domain and translation style of the pool, suggesting that example quality is the most important factor. using both the bleurt metric and mqm human evaluations, we show that palm's performance, while very impressive for a system never deliberately exposed to parallel text, still significantly lags that of competition-grade sota systems on recent wmt test sets, and to a lesser extent the performance of cots systems as well. this contrasts with some of the findings of @xcite . as in that work, we find that performance into english is somewhat better than the reverse direction. finally, we perform an extensive analysis of the characteristics of palm's mt output, notably finding that in all languages we tested it tends to be creative and fluent but prone to omissions and other accuracy errors; broadly speaking, it matches the fluency but lags the accuracy of conventional nmt. in future work we look forward to testing palm on document-level translation tasks, unleashing its formidable capacity for leveraging long contexts. we would also like to explore prompt tuning methods that are more sophisticated than the hard-prompt setting we adopted for this paper, particularly to see if these might offer a way to tighten up palm's mt accuracy without destroying its impressive ability to generate highly-fluent text.