CHIA : CH oosing Instances to Annotate for Machine Translation (2022)

Abstract: Neural machine translation (MT) systems have been shown to perform poorly on low-resource language pairs, for which large-scale parallel data is unavailable. Making the data annotation process faster and cheaper is therefore important to ensure equitable access to MT systems. To make optimal use of a limited annotation budget, we present CHIA (choosing instances to annotate), a method for selecting instances to annotate for machine translation. Using an existing multi-way parallel dataset of high-resource languages, we first identify instances, based on model training dynamics, that are most informative for training MT models for high-resource languages. We find that there are cross-lingual commonalities in instances that are useful for MT model training, which we use to identify instances that will be useful to train models on a new target language. Evaluating on 20 languages from two corpora, we show that training on instances selected using our method provides an average performance improvement of 1.59 BLEU over training on randomly selected instances of the same size.

## introduction machine translation (mt) systems have been widely adopted into daily use and facilitate easy communication and access to information. advances in neural mt have enabled systems to approach human performance @xcite . however, such high-performing mt systems are only available for a small subset of the world's languages as they require large training corpora @xcite . while unsupervised methods @xcite that use limited or no parallel data are effective for many languages, they perform poorly on low-resource language pairs * equal contribution figure 1 : overview of our method, chia. given data in high-resource languages such as english and spanish, chia selects data to annotate for low-resource languages such as irish or burmese. @xcite , and are outperformed by supervised methods @xcite . to ensure that advances in mt benefit all communities and users equally, we need efficient ways to collect parallel data. for high-resource languages with a large number of speakers, parallel data sources exist @xcite @xcite , and crowdsourcing has proved cheap and effective @xcite . however, for low-resource languages, collecting sufficient parallel sentences is harder, as bilingual translators may be difficult to find or expensive. the amount of instances which can be manually translated for use during model development is thus limited either by time or monetary constraints. here, we explore how existing or easily obtainable parallel sentences in high-resource languages, e.g., english and spanish, can help us construct high-quality datasets for a target language with low or no resources, under a limited annotation budget. we present chia (choosing instances to annotate), which requires a multi-way parallel dataset and automatically identifies those instances which will result in the strongest mt system if translated to construct a training set for a new language. chia is based on cross-lingual information: first, we identify the most effective instances to train mt systems between the center language -the language from which we wish to translate into a low-resource language -and multiple highresource languages. for this, we utilize mt model training dynamics to identify examples that help a model learn, as proposed by @xcite . we extend their method, originally proposed for classification tasks, to sequence-tosequence tasks. second, we use the intersection between the sets of informative instances for different language pairs to determine which instances will be most beneficial for training an mt system for a new language pair, cf. figure 1 . we perform experiments on two multi-way parallel datasets, the europarl corpus @xcite and the jhu bible corpus @xcite . our language pairs consist of english -our center language -and 15 simulated low-resource languages as well as 5 truly low-resource languages. we show that, on average, training on examples selected by chia results in gains of 1.59 bleu as compared to training on randomly selected instances. we further examine the characteristics of the selected training examples, and find that chia does not rely on simple properties such as sentence length or number of unique words. chia is based on the two contributions we present in this paper: 1) a method to identify the most useful training instances for sequence-tosequence tasks, and 2) an empirical demonstration that this method can be used to identify examples, which will be beneficial for a new low-resource language, based on a set of high-resource languages. with e being the number of training epochs, θ e being the model parameters at the end of epoch e, y * i being the true label in the training set, and x i being the respective input. thus, c i corresponds to the average probability of the true label of an example over epochs, and v i is the confidence's standard deviation. based on confidence and variability, @xcite partition the training set into three regions: instances with high confidence and low variability are designated as easy-to-learn, instances with high variability are designated as ambiguous, and instances with low confidence and low variability are designated as hard-to-learn. they show that instances in the ambiguous region contribute the most to the model's ability to generalize out of the training distribution, i.e., ambiguous instances are the most effective training examples. ## data and languages we use two corpora from different domains: the europarl corpus @xcite and the jhu bible corpus @xcite . ## conclusion we propose chia, an algorithm for choosing informative instances to annotate when creating mt datasets, thereby maximizing a limited annotation budget. chia is based on our two contributions: 1) by extending prior work we propose a method to identify beneficial training instances for sequence-to-sequence tasks. 2) using this method, we show that we can leverage existing parallel data in high-resource languages to identify informative instances for new languages. we find that in comparison to randomly selected data, mt models trained on data selected using chia achieve average improvements in bleu score of 1.59 points. notably, chia is effective even when evaluating on low-resource languages, providing an efficient data annotation strategy. limitations and future work in our experiments we use english as a center language. in future work, we will investigate alternate center languages, as well as the effectiveness of our method without a center language. additionally, a limitation of chia is that a multi-way parallel dataset containing the center language is required, which might be difficult to find for specific domains. our case study further indicates that selecting data using a single high-resource language alone may not be adequate to achieve performance improvements. in future work, we will investigate the effect of different numbers and combinations of seen languages on our method. furthermore, while we apply chia to select useful data for machine translation, our method can potentially be applied for collecting data in a lowresource language for any nlp task, provided a multi-way parallel dataset, which may be investigated in future work.