Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation? (2022)

Abstract: What can pre-trained multilingual sequence-to-sequence models like mBART contribute to translating low-resource languages? We conduct a thorough empirical experiment in 10 languages to ascertain this, considering five factors: (1) the amount of fine-tuning data, (2) the noise in the fine-tuning data, (3) the amount of pre-training data in the model, (4) the impact of domain mismatch, and (5) language typology. In addition to yielding several heuristics, the experiments form a framework for evaluating the data sensitivities of machine translation systems. While mBART is robust to domain differences, its translations for unseen and typologically distant languages remain below 3.0 BLEU. In answer to our title’s question, mBART is not a low-resource panacea; we therefore encourage shifting the emphasis from new models to new data.

## introduction pre-trained multilingual sequence-to-sequence (pmss) models, such as mbart @xcite and mt5 @xcite , are pre-trained on large general data, then fine-tuned to deliver impressive results for natural language inference, question answering, and text simplification @xcite . their performance on machine translation shows promise for translating low-resource languages @xcite @xcite , which remains an open challenge @xcite @xcite . when can mbart and mt5 succeed in translating a low-resource language? despite their promise, the specific conditions for their practical application are not yet clear. understanding their sensitivities is crucial to guide data acquisition efforts and apply pmss models to new languages. we introduce a framework for assessing datadependency of performance of machine translation systems. we then apply it in a large-scale study of mbart's viability for low-resource machine translation on 10 typologically and geographically varied languages. eight languages are low-resource, and four are unseen by mbart during pre-training. through our results, we gauge the importance of five dimensions of the training data: the closest work to ours @xcite considers only the first two. for the seen languages, mbart reaches acceptable performance with either 10k high-quality, indomain sentence pairs or 100k noisy ones. however, mbart's bleu score for unseen languages is often below 3.0-far below usability. for these unseen, low-resource languages, the fact that even mbart-which has already seen billions of sentences-cannot succeed in virtually any of our conditions speaks to the need for appropriate indomain data. therefore, the analytical framework in our experimental design can help to target new data acquisition efforts. after model weights have been learned, the models can be fine-tuned on parallel text for translation. the ideal fine-tuning scenario would be vast, clean data matching the language and domain of interest. because this scenario is unlikely for low-resource languages, we test the relaxation of these assumptions for pmss models. in a preliminary experiment comparing mbart and mt5, mbart performed better than mt5 on 11 of the 18 translation directions, especially the @xmath0, corroborating @xcite . because mbart performed better both in number of translation directions and average bleu, we focus hereafter on it. ## experimental setting we fine-tune mbart models on each of the training corpora and sizes listed above, and we evaluate their performance using the development and test sets from the domain-specific corpora and flores. we additionally train a standard transformer baseline @xcite to compare pretraining versus training from scratch. we score translations with sacrebleu @xcite . details of training and evaluation are given in appendix b. ## results and analysis the results of our empirical study are given in ## conclusion we have assessed the value of pmss models like mbart for low-resource machine translation. we designed a reusable framework of experiments, capturing mbart's sensitivity to five facets of data. consistently, mbart fails in learning to translate new under-resourced languages-those unseen in the pre-trained model. for languages used in monolingual pre-training, we find four-to tenfold data efficiency over a from-scratch transformer, plus robustness to domain differences. for domain-specific datasets, mbart might outperform standard transformers by an efficiency of five to ten times; future work can pinpoint the saturation size. fine-tuned mbart is robust to domain differences, while the transformer flounders for out-domain datasets. however, the performance on unseen languages is generally not indicative of usable translation system. taken in tandem, these results point to the paramountcy of monolingual pre-training for the bilingual task of translation. the biggest open issue, though, is not how to tune pmss models on limited data; instead, greater data acquisition is the hope for truly low-resource machine translation.