Small Data, Big Impact: Leveraging Minimal Data for Effective Machine Translation (2023)

Abstract: For many languages, machine translation progress is hindered by the lack of reliable training data. Models are trained on whatever pre-existing datasets may be available and then augmented with synthetic data, because it is often not economical to pay for the creation of large-scale datasets. But for the case of low-resource languages, would the creation of a few thousand professionally translated sentence pairs give any benefit? In this paper, we show that it does. We describe a broad data collection effort involving around 6k professionally translated sentence pairs for each of 39 low-resource languages, which we make publicly available. We analyse the gains of models trained on this small but high-quality data, showing that it has significant impact even when larger but lower quality pre-existing corpora are used, or when data is augmented with millions of sentences through backtranslation.

## introduction state of the art machine translation models are able to cover hundreds of languages @xcite @xcite by relying on large amounts of annotated @xcite @xcite and unannotated web crawled data @xcite . translation for low-resource languages still faces significant challenges related to data availability, since many of these languages have neither large-scale parallel corpora nor a big presence on the web @xcite . techniques such as self-supervised learning @xcite and backtranslation @xcite @xcite ) can be effective tools to reduce the reliance on annotation for translation models. in some cases, these techniques can be combined or even be applied iteratively @xcite , leading to a feedback loop that can generate increasingly better translations. in order to be effective, however, such methods still require a certain amount of seed parallel data, which can be used to kickstart the process. as a result, researchers and communities looking to train translation systems for low-resource languages may find themselves wondering how much parallel data is required to achieve a given performance target level. in this paper, we describe a data collection effort for 39 low-resource languages, involving the creation of over 6k seed sentence pairs per language by professional translators, which we make publicly available with an open license. we analyse the behaviour of bilingual translation systems trained on varying amounts of this data, with and without the addition of pre-existing publicly available parallel datasets, and find that even comparatively small amounts of professionally produced parallel sentences can have an outsized impact. we find that gains coming from high quality data are further enhanced when training multilingual models of closely related high-and low-resource languages, and even more so when augmenting the dataset via backtranslation. overall, our results show that employing relatively small but high-quality, professionally translated datasets constitutes a promising and viable way towards achieving performant machine translation for low-resource languages, especially for those with high-resource relatives. this holds true even for languages for which some pre-existing data might already be publicly available, further highlighting the importance of high-quality training datasets. notably, parallel datasets of the scale discussed here are compact enough that coverage for a new language could plausibly be collected by a relatively small group of volunteers in a week, making these results relevant for the usage of machine translation technologies in crisis situations @xcite . our main contributions are: ## background low-resource language translation despite very successful recent advances in neural machine translation, most of the gains have only benefited a handful of so called high-resource languages, which have enough textual resources to satisfy the substantial data requirements of state-of-theart techniques. the vast majority of the world's languages are low-resource, and researchers have increasingly been focusing on evaluating performance in this challenging setting @xcite . benchmarks traditionally, one of the biggest challenges to the development of low-resource translation systems has been the lack of high quality evaluation data. several benchmarks focus on specific sets of languages, such as the madar dataset for arabic dialects @xcite , the autshumato benchmark covering 11 south african languages @xcite , or the tico-19 benchmark covering 35 languages for the domain of medical information related to the covid-19 pandemic @xcite . more recently, the flores-101 dataset @xcite and its expansion to over 200 languages @xcite has enabled multilingual evaluation across tens of thousands of directions, including many low-resource languages. its domain is composed of an even mixture of travel guides (wikitravel), children's literature (wikijunior), and news content (wikinews). training corpora much important work has gone towards the development of parallel corpora for low-resource languages, most of which focusing on individual language pairs @xcite @xcite . @xcite study the case of 15 low-resource african languages, most of which already have tens or hundreds of thousands of parallel sentences in the religious domain, and investigate how combining pre-trained models and a newly created corpus can lead to effective domain transfer. low-resource training amongst the techniques that can be used to decrease the reliance on manually annotated data, bitext mining @xcite enables finding pairs of translations among large collections of unannotated monolingual text. @xcite show its effectiveness for low-resource languages, but point out that it can be limited for the most data scarce languages. backtranslation @xcite can be used to create pseudoparallel data from monolingual data in a target language. it relies on an initial, potentially low-quality translation model -thereby having some requirements on annotated data -and can also be applied iteratively for improved performance @xcite . self-supervision @xcite is a method employing monolingual text denoising as a joint training objective, and its use has been suggested as a way of kick-starting an iterative backtranslation pipeline. finally, multilingual translation, which is often combined with one or more of the above techniques, has been shown to improve low-resource translation performance via cross-lingual transfer @xcite @xcite @xcite training without parallel data within the area of low-resource translation, @xcite describe the development of translation systems for low-resource languages without using any parallel data at all, relying instead on crawled monolingual data and language transfer. methods which don't require parallel data are likely complementary to the seed data approach proposed in this paper. however, the over-reliance on cross-lingual transfer from a high-resource language opens up the risk of a translation system flattening the differences between related languages, as observed by nllb team et al. (2022) for arabic dialects. this is a particularly thorny issue for communities of speakers of endangered languages which are at risk of being displaced by a related higher-resource language -as is the case for several of the languages covered in this paper. in such cases, we recommend the seed data approach, which opens the door for the communities to take ownership in preserving their languages, and aligns well with their desire to preserve the distinctiveness of their language in technological applications. crisis mt low-resource machine translation has been studied in the context of crisis events, and has been proposed as a component of a rapid response infrastructure @xcite . in particular, @xcite describe the creation of a system for haitian creole after the devastating 2010 earthquake, and @xcite built a dataset to facilitate access to information related to the covid-19 pandemic. ## data collection regardless of the many modelling improvements aimed at reducing the amount of required supervision, it is likely impossible for translation models to reach acceptable levels of quality without even small amounts of parallel data. this is especially true for approaches that explicitly rely on the preexistence of parallel corpora, such as backtranslation. as a result, low-resource languages with corpora that are too small to enable the use of these techniques are cut off from the improvements they bring. with this in mind, we set up a data collection effort for a number of low-resource languages which fit this criterion, resulting in a dataset of around six thousand english sentences translated into each of 39 low-resource languages. language selection in order to choose which languages to collect data for, we took several factors into account. first, we looked at the list of languages supported by wikipedia. the usergenerated encyclopedia is one of the most visited websites in the world, and constitutes an important means of knowledge dissemination for many low-resource language communities. crucially, wikipedia has an open process towards support-ing new languages, 2 which has led to the platform supporting over 300 languages in 2022. 3 this list of languages was cross-referenced with those currently supported by machine translation benchmarks, including the large flores-200 dataset. we then focussed our attention to those languages for which not enough high quality data was currently publicly available for large-scale training, looking in particular at those languages with fewer than 100,000 parallel training sentences and prioritising those with the least amount of high quality data (as determined by automatic metrics such as language identification). finally, we partnered with linguists and identified those languages for which professional translators would be available. ## results and analysis we report all results using automatic evaluation metrics against the flores-200 benchmark. we rely on the chrf++ score @xcite , which is based on character-level n-gram overlap, and is complemented by unigram and bigram features. this score overcomes the limitations inherent to the more commonly used bleu metric @xcite , which relies on the availability of tokenisation tools for all languages and fails to accurately account for highly agglutinative languages. ## analysis figure 2 brings together the average performance of all models trained on the italic and indo-aryan language clusters -bilingual, multilingual, and multilingual with backtranslation -both when trained only on pre-existing data alone (first set of bars), and when trained with the addition of high-quality seed data (hatched bars). language eng-xxx xxx-eng the same trends hold throughout our experiments: even with modelling improvements that aim to reduce the amount of required supervision, such as multilingual training and backtranslation, we observe that models trained on as little as 6k high-quality seed parallel sentences always come out ahead. this is true even for languages such as mag_deva and hne_deva, for which tens of thousands of pre-existing parallel sentences are publicly available. crucially, we see that the multilingual model with seed data ("multilingual, p+6k" in the graph) outperforms in all but one case the version without seed data but with backtranslation ("multilin-gual+bt, p"). in other words, even adding vast amounts of monolingual data (as much as 2m sentences for xxx-eng) cannot make up the difference that 6k high-quality parallel sentences make. ∆ nobt denotes the performance difference between each model and its base multilingual version trained without backtranslation; ∆ denotes the effect of adding seed data to multilingual models using backtranslation, measured as the difference between the p+6k and p backtranslation models. #bt denotes the size of the backtranslation corpus for each direction, measured in millions of sentences. ## conclusions in this paper, we have described a parallel data collection effort involving 6k seed parallel sentences for 39 languages, and investigated the effects of this relatively small but high-quality dataset on machine translation performance. by training hundreds of bilingual translation models, we have looked at the data scaling properties, and found that even when several thousand pre-existing sentences are already available, adding as little as a thousand high-quality parallel sentences can significantly boost performance. to answer the question of whether stronger models can compensate for the lack of high-quality data, we moved beyond simple bilingual models and introduced two modelling improvements: multilingual training of closely related low-and highresource languages, and backtranslation. we found that models trained with the additional high-quality data performed consistently better. even when augmenting the models with vast amounts of monolingual data via backtranslation, the beneficial effects of seed data were still present. overall, the results show that collecting highquality parallel data, produced by native speakers and manually aligned, is a fundamentally important investment for training machine translation models.