Data Cartography for Low-Resource Neural Machine Translation (2022)

Abstract: While collecting or generating more parallel data is necessary to improve machine translation (MT) in low-resource settings, we lack an understanding of how the limited amounts of existing data are actually used to help guide the collection of further resources. In this paper, we apply data cartography techniques (Swayamdipta et al., 2020) to characterize the contribution of training samples in two low-resource MT tasks (Swahili-English and Turkish-English) throughout the training of standard neural MT models. Our empirical study shows that, unlike in prior work for classification tasks, most samples contribute to model training in low-resource MT, albeit not uniformly throughout the training process. Furthermore, uni-dimensional characterizations of samples – e.g., based on dual cross-entropy or word frequency – do not suffice to characterize to what degree they are hard or easy to learn. Taken together, our results suggest that data augmentation strategies for low-resource MT would benefit from model-in-the-loop strategies to maximize improvements.

## introduction while neural sequence-to-sequence models have led to dramatic increases in translation quality for many machine translation (mt) tasks, the large amounts of data needed to train high quality systems is only available for a small number of the 7000 languages spoken in the world @xcite . efforts to improve this state of affairs have focused on data cleaning, recognizing that parallel samples are often noisy and of limited domain coverage in these settings @xcite , and on data collection and augmentation, with techniques ranging from large scale crawling @xcite to automatic data augmentation via backtranslation @xcite to generating novel data through lexical or phrase replacement @xcite . however, to make the most of limited data, we argue that a better understanding of the role that data samples play in training low-resource mt models is needed. this paper presents an empirical study of the role that training samples play in low-resource mt training so that future work on improving data quality and quantity can prioritize properties of data that matter most. our study focuses on mt in a very low-resource setting (swahili-english, 63k training samples) and a low resource setting (turkish-english, 358k samples). as @xcite note, the term "low resource" encompasses a wide disparity of situations. we work with languages from two distinct classes in their taxonomy of language resources. turkish falls into the "underdogs" category, which comprises languages with large amounts of unlabeled data, comparable to those available for high-resource languages, and that are primarily challenged by lesser amount of labeled data (parallel text in the case of mt). swahili falls into the "hopefuls" category, which refers to languages for which only small labeled datasets have been collected, often by a community which strives to keep them alive in the digital world. in this work we use data maps @xcite , a technique that uses the model's training dynamics, to identify and characterize regions in realistic low-resource translation data and their impact on translation quality. data maps places training examples on a coordinate plane that describes a model's confidence and variability of these examples across the training process. we show that the notion of difficulty of an example changes at different stages of training, that samples are not uniformly useful, and that existing heuristics for sample difficulty do not characterize the samples that are hard to learn based on training dynamics. we hope that these findings can help inform data augmentation strategies in future work. ## methods we describe the main methods used to analyze the role of different samples in low-resource mt training through training dynamics. on classification tasks, training on various subsets of the data revealed that samples in three of the map regions play a distinct role: we divide the mt data maps into these three regions by using the midpoint between the max and min values of confidence and variability scores. figure 1 previews data maps for our two lowresource mt systems (swahili-english on the left, turkish-english on the right). we use bleu, an automatic translation quality metric, as an analog to the accuracy of the converged model's translation output compared against the reference. we will discuss the specific settings used to generate it and the findings in sections 3 and 4 respectively. ## empirical study settings we focus on translation from swahili and turkish into english, since these are two low-resource tasks involving under-studied languages that raise interesting reordering and morphology challenges for mt. swahili is a bantu language spoken by an estimated 200 million people. it has an agglutinative morphology and uses a subject-verb-object order. turkish is a turkic languages, with 70 to 80 million speakers. it uses extensive agglutination and follows a subject-object-verb order. data the swahili-english data is a combination of data taken from the iarpa material program foot_0 , globalvoices and commoncrawl. the training data totals 63k sentence pairs and we have a validation set and two test sets from material of sizes 2000 and 3367. the training corpus for turkish-english is a combination of publicly available data from wit 3 (cettolo et al., 2012) and setimes (alperen et al., 2010) at around 350k sentence pairs. the validation and test sets come from the 2016 wmt news translation task (bojar et al., 2017); we use newsdev2016 as validation and new-stest2016 and newstest2017 as test sets; with sizes 3000 and 3007, respectively. all data is tokenized and true-cased using the moses toolkit. we use sentencepiece to generate subword tokens with a joint vocabulary and decide on the sizes of 4000 and 16000 for sw-en and tr-en, respectively through hyper-parameter tuning. ## data maps across training phases maps overview the data maps for each of the three training phases (figure 2 ) show that samples do not play a constant role throughout the training process. for both language pairs, samples are initially distributed across the diagonal during phase 1 -target language modeling. samples variability decreases in later stages focused on cross-lingual learning, leading to more easy samples and fewer to none ambiguous samples. the range of confidence scores increases throughout training for turkish-english, with a small peak in distribution in the easiest upper left right corner in the later phases, while the confidence remains under 0.5 for most samples in the very low swahili-english setting. for both language pairs, a large fraction of samples remain in the hard region throughout training. ## characterizing region properties ## related work it is well established that data quality can have a large impact on the performance of mt systems. @xcite shows how different types of training set noise impact the quality of neural mt models. dual cross-entropy (junczys-dowmunt, 2018) and other techniques are routinely used to clean crawled datasets that are known to contain many types of noise @xcite . training sample difficulty is often considered in the context of curriculum learning, which aims to present training examples to the model in an order that benefits model performance or convergence speed. some of the earlier works by @xcite ; @xcite show that ordering training samples by sentence length can decrease training speed or improve model performance. @xcite go on to develop curriculum learning strategies building on the use of linguistic features such as sentence length or word frequency; but their results are mixed. later works have looked at more model-focused techniques for deciding the difficulty of training examples such as the norm of source words @xcite , language model scores on monolingual text @xcite or the change in the decrease of mt model loss @xcite . these techniques have shown more consistent improvements in translation quality but still do not query information from the mt model itself; except in the case of the latter which still only looks at local changes in the model loss. in contrast data maps are based on a holistic view of the training process. prior work on low-resource settings has primarily focused on modeling strategies or supplement-ing the training data. @xcite determined appropriate hyper-parameters for transformer models that were initially developed in high resource settings. @xcite paired data augmentation with a multi-task modeling approach to strengthen the power of the model's encoder and decoder. others considered restricted sampling from mt decoding @xcite , leveraging a high resource pivot language @xcite , generating new contexts for infrequent words @xcite and learning to edit noisy training samples @xcite . ## conclusion this work used training dynamics in the form of data maps to understand the role of samples in two low-resource machine translation tasks (swahili-english and turkish-english). the data maps show that the role of samples changes across the training of neural mt models. filtering experiments show that all samples contribute to training and that hard samples do not hurt translation quality. further analysis shows that the role of samples cannot be explained by simple uni-dimensional heuristics, although classifiers can be trained to detect which region a sample belongs to with a potentially useful level of accuracy. this suggests future work on using data maps to guide data augmentation, possibly using automatic classifiers to target the easy or ambiguous samples that are more likely to benefit further training.