Simple and Effective Input Reformulations for Translation (2023)

Abstract: Foundation language models learn from their finetuning input context in different ways. In this paper, we reformulate inputs during finetuning for challenging translation tasks, leveraging model strengths from pretraining in novel ways to improve downstream performance. These reformulations are simple data level modifications, require no additional collection of training data or modification of data at inference time. They can be applied either on single language pair translation tasks or massively multilingual translation tasks. Experiments with these techniques demonstrate significant performance improvements up to 3.5 chrF++ on the Flores200 translation benchmark . We hope our research accessibly improves finetuning data efficiency, enabling more effective training to scalably improve state-of-the-art performance. Our code is released here.

## introduction foundation language models (flms) are powerful and task-agnostic models. they are pretrained on language understanding objectives, enabling strong performance on downstream language tasks @xcite @xcite @xcite @xcite @xcite . flms are then either prompted or finetuned for downstream use. in this paper, we present three different data efficient techniques for improving translation performance, applied to the multilingual flm mt5 during finetuning @xcite . in our first approach, we train mt5 on a classical tibetan to english (tib2eng) translation task. mt5 struggles heavily in the initial training steps. thus, for the first 20% of finetuning, we apply the "partial output english scaffold" or pose reformulation, shown in figure 1 . tib2eng translation examples consist of a classical tibetan source and english target translation pair. pose simply appends a prefix of the target english output to the classical tibetan input. we see qualitative improvements in the variance of the training curves. when evaluated on the same test set with no reformulations, pose significantly increases overall translation performance compared to the direct finetuning baseline, up to 10.3% / 2.8 bleu. the pose setup had many adjustable hyperperameters relating to task difficulty, task curriculum, and substring selection for scaffolding. we find that input reformulation setups should consist of 20% less informative examples, and 80% harder and more informative examples. more ablation details can be found below. second, we approach the massively multilingual flores200 translation benchmark @xcite . mt5 does not struggle in the initial steps of finetuning on flores200 in the same way it did on tib2eng. even so, we begin by replicating the tib2eng pose setup on flores200 by appending a partial output of the target translation to the input translation. as expected, this setup matched but did not improve upon the baseline performance. the flores200 benchmark consists of parallel examples of the same sentence in different languages. in our second approach, we extend the tib2eng pose reformulation to create the "parallel scaffold in english" or parse reformulation, shown in figure 1 . parse appends the corresponding full parallel english translation (provided by flores200) to the input. following the tib2eng setup, we use a data mix of 20% baseline (less informative) and 80% parse (more informative) examples. parse significantly improves translation performance, up to 17.2% / 3.5 chrf++. we postulate that pose and parse improve translation performance in part because they enable mt5 to attend to an in-distribution pretrain language with strong monolingual performance. in our third approach, we explore the efficacy of parallel scaffolding that does not require strong monolingual performance using the "mixed-language parallel scaffold" or mips reformulation, shown in figure 1 . mips appends a different parallel translation to both the input and output for a total of 4 distinct languages per input. again, we use a data mix of 20% baseline and 80% mips examples. mips also improves translation performance, up to 9.1% / 1.6 chrf++. scaffolding with the strongest performing pretraining language (parse) outperforms scaffolding with a mix of other languages (mips). finally, we perform analysis on the languages in the translation set. using a balanced dataset like flores200 allows mt5 to partially overcome pretraining dataset size biases. naturally, translating into lower resource languages is more difficult than translating into higher resource languages, but we find that the parse and mips reformulations improve translation into all languages across the board, rather than disproportionately improving performance on high resource languages. in summary, we propose input reformulations on translation tasks. these reformulations require no additional data, have few hyperparameters, and are simple to implement. when finetuning on a single language pair translation task, if the target output language is in the model's pretraining dataset distribution, the pose reformulation can be applied. when translating between multiple language pairs, the parse reformulation can be applied to the strongest performing pretraining language. our work can be viewed as a data efficiency technique for translation. past works in translation have explored data augmentation @xcite , sample re-weighting @xcite @xcite , and curriculum learning @xcite @xcite @xcite . these approaches vary in effectiveness, are not generalizable, and introduce complexity into the training process. curriculum learning approaches in particular are typically complicated and unsuccessful, because they are designed using intuition on how humans treat inputs, which may differ from how models treat inputs. in contrast, our input reformulations are simple and can be directly applied to any sequence-to-sequence task. previous work has explored prompting a frozen language model using manually curated prompts @xcite @xcite . results are typically sensitive to the exact prompt used. this technique cannot be applied to larger corpora because it is limited by the number of examples that can feasibly fit into a single input context. other works have explored finetuning with a fixed prompt without leveraging the target output as a part of the input @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . following the success of fixed prompt techniques, other works proposed prompt tuning setups @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . these prompt tuning setups were typically used in the context of compute efficiency: training a smaller number of prompt-related parameters to input into a larger frozen language model. these setups are an orthogonal improvement to our proposed input reformulations. previous approaches also investigated dataset improvements for better downstream task performance. these approaches gathered additional data for model training to augment the model's input context @xcite @xcite @xcite @xcite @xcite @xcite . they require large, specific, and high quality datasets to be collected. on the other hand, our input reformulations require no additional data. overall, our approach differs from previously explored approaches by avoiding prompts and leveraging the target output as a part of the input reformulation. our input reformulations are a data-level change that can be easily applied to any training setup. ## conclusion we have explored how flms learn from their input contexts. we provide two separate techniques that can be applied to any translation use case. for the case of a single language pair translation task, we recommend pose. for the case of a multi-language pair translation task, we recommend parse and mips. for challenging translation tasks, our scaffolding reformulations produce bet-ter conditioned training curves and significantly better performance. these input reformulations are simple to understand and implement, robust over hyperparameters, general to translation tasks, and effective. we hope our technique is used to accessibly improve data efficiency on translation tasks.