Prefix Embeddings for In-context Machine Translation (2022)

Abstract: Very large language models have been shown to translate with few-shot in-context examples. However, they have not achieved state-of-art results for translating out of English. In this work, we investigate an extremely lightweight fixed-parameter method for conditioning a large language model to better translate into the target language. Our method introduces additional embeddings, known as prefix embeddings which do not interfere with the existing weights of the model. Using unsupervised and weakly semi-supervised methods that train only 0.0001% of the model parameters, the simple method improves ~0.2-1.3 BLEU points across 3 domains and 3 languages. We analyze the resulting embeddings’ training dynamics, and where they lie in the embedding space, and show that our trained embeddings can be used for both in-context translation, and diverse generation of the target sentence.

## introduction under the paradigm of in-context learning, foot_0 large language models have been shown to generate translations when provided with several priming examples, each of which consists of a source sentence and the translated target sentence. these examples, also known as "prompts", are prefixed to the test source sentence, which then conditions the model to generate the test target sentence. this prompt-and-translate phenomena, or in-context translation, presents itself as a new paradigm for machine translation applications. first, the ability to adapt to different task specifications using prompts suggest that the same model can be used in multiple settings and domains. while there have been several multilingual translation models @xcite @xcite , the ability to perform unrelated tasks such as question-answering in addition to translation is relatively new. this also presents an interesting shift from supervised neural machine translation (nmt) in terms of data requirements. these models are trained on massive amounts of web text which are not explicitly parallel. 2 in contrast, modern nmt models are trained with millions of lines of parallel text. unsuprisingly, the lack of supervision comes at a cost. translating out of english for in-context models still lags behind state-of-art possibly due to low data quality and/or disproportionate amounts of english. ## related work large language models which perform in-context translation following gpt3 @xcite which first reported the in-context translation phenomena, subsequent autoregressive transformer decoder only architectures such as xglm @xcite and mgpt @xcite have explicitly trained in-context models to be multilingual. however decoding out of english still performs more poorly than decoding into english. hence we focus on the first scenario of decoding out of english. prefix tuning unlike previous work which directly prefixes the task by prepending to the input @xcite @xcite , we substitute the trained prefixes for the delimiters throughout the prompts before the target language sequence. our proposed method prefixes the target sequence, not the task. this small but significant difference allows monolingual training for the target language without explicit translation task supervision. ## methods our approach is motivated by the knowledge that for very large language models trained on web corpora, there is a weaker target language (being translated into) because english is the dominant language on the web. this trend persists even for explicitly multilingual language models @xcite . our method therefore aims to condition the language model to decode the weaker target language, by learning a language-specific prefix. we first describe the in-context translation setup at test time (subsection 3.1), followed by unsupervised training (subsection 3.2) and weakly supervised training (subsection 3.3) of the target language prefix embedding. at inference time, the corresponding prefix will be used as the separator token between source and target language. figure 1 illustrates this process. ## experiments we organise our experiments investigating the effects of prefix embedding tuning 1) across three en-fr domains, medical, social media, and ted talks, 2) across three languages in ted talks. 6 in both sets of experiments, we explore three basic initializations (described in prefix embedding initialisations). we also use priming examples of various sizes to investigate if the effects persist across different prompt sizes. to account for prompt selection and ordering effects, all inference runs were repeated with 5 randomly sampled prompt sets from the training data, where each of the source sentences in the prompt examples are between 10 to 20 words long. scores are reported using sacrebleu @xcite . 7 model we use gptneo2.7b (32 layers, 20 heads) @xcite which has been pretrained on the pile @xcite . the pile contains europarl which has been fed into the model at a document level and not a sentence level. 8 to our knowledge, there has not been any reports of sentence level parallel corpora in the training dataset of this model. note that unlike most dedicated machine translation models which have an encoder-decoder architecture, this model is trained autoregressively and is decoder only. 9 data we adopt three datasets; multilingual ted talks @xcite , med @xcite , and mtnt @xcite . we use 30,000 monolingual sentences for unsupervised training of the prefix embeddings (mono in results table). for ted, med and mtnt, the monolingual sentences are obtained from their bitext training data. we use 100 bitext sentence pairs for the weakly supervised case (bitext in results table). these bitext sentence pairs served as a self-contained prompt set and training data instances as described in subsection 3.3. in both the unsupervised and weakly supervised scenarios, during testing, we sample sentence pairs for prompts examples from the training set. the sentence pairs in weakly supervised training, validation, and inference time prompt selection are all separate splits; there is no overlap between prompt sets seen across these phases. preprocessing we preprocess digits to as we find that this helps the prefix tuning to converge for the med and ted domains, without compromising on the ability to copy or generate digits. we run a langid check and restrict training sentence length to 3 to 25 words to avoid trivial sequences and out-of-memory errors. ## conclusion in this paper, we show that priming of in-context learning models can be improved using primarily unsupervised methods. to our knowledge, this is the first work which emphasises the target side language during decoding of a large language model for in-context translation. the gains are modest but so are the number of parameters trained. in our experiments we have shown that the simple method gives up to 5 bleu point gains for monolingual training in the 1-shot setting, and weakly supervised bitext training in the 20-shot setting gives up to around 2 bleu point gains across 3 domains and 3 languages. given that we leverage primarily on unsupervised (monolingual) target side training and carefully control for random prompt selection, this could be a generic approach for improving decoding into a weaker target distribution, which is complementary to the vast literature on prompt example selection and optimisation @xcite . limitations we have used one model, gptneo2.7b, in this set of experiments. although this accessible off-the-shelf model is considered a replication of gpt3 in terms of architecture and is highly used (88k downloads in the month of january 2022), other factors such as different training data or scale of the model (100b parameter vs 2b parameters) may affect generalisability of the results. there are no known ethical concerns.