COMET - QE and Active Learning for Low-Resource Machine Translation (2022)

Abstract: Active learning aims to deliver maximum benefit when resources are scarce. We use COMET-QE, a reference-free evaluation metric, to select sentences for low-resource neural machine translation. Using Swahili, Kinyarwanda and Spanish for our experiments, we show that COMET-QE significantly outperforms two variants of Round Trip Translation Likelihood (RTTL) and random sentence selection by up to 5 BLEU points for 20k sentences selected by Active Learning on a 30k baseline. This suggests that COMET-QE is a powerful tool for sentence selection in the very low-resource limit.

## introduction active learning (al) is a technique that involves querying an oracle for labels based on a selection strategy. for neural machine translation (nmt), active learning involves selecting sentences for translation based on estimated model uncertainty on the sentence or the diversity of the sentence. the aim of al is to choose sentences that would maximally improve the nmt model. this is particularly attractive for low-resource languages where budgets are small, which includes most of africa's languages which have historically been neglected @xcite . there have been several advances in active learning for nmt with round trip translation likelihood (rttl) being the current state of the art @xcite @xcite . these advancements have typically been tested by artificially restricting high-resource languages which mimic true low-resource languages. there have been few empirical evaluations of how these al techniques perform on a low-resource african language with less than 100k sentences. we tackle this problem while simultaneously proposing a new al sentence selection strategy based on comet-qe @xcite and show that it outperforms two versions of rttl, one of the most successful al selection algorithms, on swahili, spanish and kinyarwanda. interestingly the rttl variants perform worse than random sentence sampling in this context. ## active learning for nmt there are several al sentence selection frameworks including pool-based al, membership query synthesis and stream-based al @xcite . we focus on pool-based al where we assume their exist a large pool of monolingual sentences for the language. pool-based al in nmt is an iterative process that involves selecting batches of sentences after each given al cycle @xcite . given a set of parallel sentences l, a model m is trained. m is then used to translate monolingual data, u and a selection strategy scores and ranks the sentences in u. the top sentences are then selected for translation. after translation, the selected sentences are added to set l and used to retrain and improve the model @xcite ; see figure 1 . this process is repeated until a specified stopping criteria is fulfilled, e.g. total translation budget reached. the selection strategy aims to choose the n sentences that will improve the model the most. selection strategies are typically either model-driven or data-driven. in model-driven techniques a model scores sentences whereas data-driven methods, also known as model-agnostic, involve comparing the set of labelled and unlabelled vernacular sentences to score the sentences. the current state-ofthe-art for model-driven selection is rttl @xcite which we discuss now. tion strategy. rttl involves training 2 models: i) model m that translates from the source to target language and ii) model m rev that reverse translates back to the source language. given a source sentence x, m generates a translation ŷ. m rev translates ŷ back to the source language. however, instead of comparing of x and x, the probability of getting the actual sentence x can be used to select sentences for translation. in rttl, the key likelihood is defined in terms of x and ŷ as: ## new active learning algorithms in this section, we detail our proposed new sentence selection techniques.