Few-shot learning through contextual data augmentation (2021)

Abstract: Machine translation (MT) models used in industries with constantly changing topics, such as translation or news agencies, need to adapt to new data to maintain their performance over time. Our aim is to teach a pre-trained MT model to translate previously unseen words accurately, based on very few examples. We propose (i) an experimental setup allowing us to simulate novel vocabulary appearing in human-submitted translations, and (ii) corresponding evaluation metrics to compare our approaches. We extend a data augmentation approach using a pretrained language model to create training examples with similar contexts for novel words. We compare different fine-tuning and data augmentation approaches and show that adaptation on the scale of one to five examples is possible. Combining data augmentation with randomly selected training sentences leads to the highest BLEU score and accuracy improvements. Impressively, with only 1 to 5 examples, our model reports better accuracy scores than a reference system trained with on average 313 parallel examples.

## introduction news agencies dealing with articles in multiple languages often rely on machine translation (mt) to provide an initial translation, which a journalist reworks into a final article. this can involve changing the structure or phrasing, but also correcting (or post-editing) mistranslated words or expressions, which can frequently occur when new topics emerge, bringing new vocabulary that has been rarely or never seen in the data used for training. the willingness of a journalist to use mt technology is dependent on the general quality of the models, but also on whether they can learn from the journalist's corrections, to avoid them having to correct the same errors time and time again. ## related work the topic of few-shot learning from post-edits is relatively novel, and we were therefore left with few comparison points. a somewhat similar task that requires quick adaptation is low-resource mt, for which transfer learning @xcite and meta-learning @xcite approaches exist. these techniques generally apply for adaptation from hundreds of thousands of sentences, rather than a dozen available in our scenario; this is because we aim to learn individual new words rather than a whole language or domain. a widespread technique in the mt literature to adapt a model to new data is fine-tuning, often used for domain adaptation. @xcite and @xcite explore the use of a similarity search in the training corpus in order to fine-tune an mt model before translating a novel sentence or to gradually adapt a model to post-edits. this approach does not apply to our scenario, where new words appear and therefore a similarity search cannot yield sentences containing these new words. our baseline approach (which we call finetune) is present in these works, known as adaptation a posteriori in @xcite ) and also appears as single-sentence adaptation in @xcite . a challenge these works report with finetuning is overfitting, which we encounter systematically when evaluating our various techniques. works on fine-tuning also explore several regularization techniques @xcite when adapting to new data, which we choose to leave out of our comparison due to added hyper-parameter choices and complexity of implementation in our experiments -we do however believe that future 1 https://gitlab.com/farid-fari/ fewshot-learning work implementing these techniques could potentially outperform ours. @xcite ; @xcite ; @xcite and @xcite explore a similar contextual data augmentation technique, albeit in different scenarios and with different goals. this technique synthesizes new sentences by using sentences from the training set and substituting different words into them. in @xcite the goal is to enhance overall translation performance by focusing on words that appear rarely in the training data, but in our case we are training our system to learn new words which were not in the training set at all, based simply on a ground-truth translation of this new word by a human. moreover, our technique uses more recent tools and techniques such as the bert contextual language model @xcite . @xcite and @xcite also use bert for contextual data augmentation, but with a goal of improving language model tasks such as sentiment analysis. the constraints for this task are very different; rather than having to produce a translation for augmented data, these approaches have to maintain the sentiment label of the sentences. @xcite work in a similar context to us, but focus on overall translation performance rather than learning new words, and apply contextual data augmentation during the training step, thus removing the challenge of adapting to new data as it becomes available. similar to us, @xcite focus on improving the mt of words which are unseen in the training set. they use bilingual lexicons to hypothesize translations for their unseen terms. they find these translations in monolingual target side data and backtranslate them inserting the unseen term. they show that this improves translation performance in the medical domain. however they do not analyze the accuracy of translation of the novel terms, or explore how fast you can learn from very few examples. we are interested in developing approaches to quickly learn from journalistic post-edits in a way that maintains the general high translation quality of the model. in order to analyze and objectively evaluate our different approaches, we simulate this scenario in a reproducible way by using a publicly available corpus in which we select the rarest words and separate out all sentences containing them: these words will be our evaluation words. these words must appear at least a few times in the test set as well in order to enable proper evaluation of adaptation using our approaches. the sentences separated from the training set containing our evaluation words will be used to simulate reference sentences submitted by a journalist for our models to use for adaptation. finally, the training set with the evaluation words removed will be our filtered training set, used for initial training of our model. our test set is also made up of publicly available data sets whose sentences also contain the evaluation words we selected. this setup prohibits the use of a development set and a test set due to the dependency of the rare words on the test set: they must appear a minimum number of times in the test set to have a way of evaluating our approaches. in turn, the choice of rare words changes the training set and therefore the models we train. we are therefore in a transductive learning scenario, where our method is adapted to the task we aim to solve. we also use as a reference point a model which has been trained on the complete (unfiltered) training set, which is a very strong comparison point since it has seen many more occurrences of the rare words we selected than our few-shot models. ## our approaches we propose and evaluate four approaches: finetune (our baseline), randompad, augmented and half. the first two approaches only involve fine-tuning whereas the latter two include contextual data augmentation. each approach works from a given number of reference sentence pairs con-taining our evaluation words, acting as humanprovided references. in our experimental setup, these sentences are chosen randomly among the held-out sentences from the filtered training set. ## results figures 2 and 3 present the results for the reference model and all approaches for both slow and fast adaptation speeds. we immediately notice that at both adaptation speeds our models are capable of surpassing the all-data model foot_7 in terms of accuracy on the new words, despite having seen fewer than 20 (and as few as 1) reference sentences containing those words, whereas all-data has seen each evaluation word over 300 times on average. our baseline, the finetune model, is shown to be largely surpassed by all other approaches in accuracy, both on the slow and fast settings. with hyper-parameters being chosen to approximately match accuracy curves, the main comparison point is the bleu score. in figure 2 , we see that in the slow setting our half approach is the only one able to improve the bleu score while learning the new words, nearly matching all-data in both bleu and accuracy at 3 occurrences seen. the augmented approach offers a slightly better accuracy curve than all other models on this speed, but loses out on bleu score at higher occurrence numbers. it is very important to look at the first points on the curve: 1-5 occurrences of each evaluation word is the realistic range to imagine a journalist making corrections, since a journalist could become frustrated with a model requiring each new word to be corrected up to 20 times for it to be correctly translated. in this respect, figure 3 for the fast setting offers the best results, with accu0 1 2 3 5 10 15 0 1 2 3 5 10 15 20 0.2 0.4 0.6 seen reference sentences per test word accuracy all-data finetune randompad(2) augmented(20) half(20) randompad(20) 0 1 2 3 5 10 15 20 20 22 24 26 seen reference sentences per test word bleu score figure 3: bleu and accuracy results of all of our approaches for the fast speed setting the randompad approaches surprised us in two ways, especially given half's success: they generally perform worse than finetune despite being designed as improvements over it, and randompad(20) has poor accuracy scores compared to all other approaches. to the first point, one possible explanation is that overfitting might occur with both the random sentences and the reference sentences when few data is used for fine-tuning (low seen occurrence numbers). this would explain why in figure 3 the bleu curve for randompad ends up overtaking the finetune curve once enough padding data is available to allow generalization. the second point can be explained by a form of 'dilution' of the reference sentences containing the evaluation words: the model over-adapts to the sentences provided to it but does not particularly improve on the evaluation words. we also tried various hyper-parameters for randompad(20) but were unable to find a compromise similar to other models, resulting in this approach standing out from the others. this might also be partially explained through variations inherent to the randomness in the selection of the padding sentences. ## conclusion we explored different techniques based on finetuning in order to adapt a base model to post-edits containing novel vocabulary. we proposed a data augmentation technique never applied to this task, allowing us to expand the number of occurrences of the new words available to our model to learn from. in our experiments, all proposed adaptation techniques offer better performance on the novel words than our reference model, which had seen the words hundreds of times each. our data augmentation approaches yield faster adaptation than our baseline, but with a greatly improved bleu score, especially when combined with generalization using padding with random training set data. these various techniques could all be applied to lifelong adaptation of an mt system often confronted with new vocabulary or expressions. our work can be generalized in several directions which we chose to leave for future work: word translations can automatically be retrieved with alignments, rare multi-word expressions (n-grams) can be used rather than rare single-token words, the language model generalization can be used in the source language or both languages and several word substitutions can be made in a single sentence. several approaches can be deployed to further improve the bleu score when fine-tuning, such as regularization techniques as explored by @xcite , keeping only the best half of synthetic sentences rather than a random half, and a more careful choice of the ratio of augmented and random data relative to reference sentences.