A dv A ug: Robust Adversarial Augmentation for Neural Machine Translation (2020)

Abstract: In this paper, we propose a new adversarial augmentation method for Neural Machine Translation (NMT). The main idea is to minimize the vicinal risk over virtual sentences sampled from two vicinity distributions, in which the crucial one is a novel vicinity distribution for adversarial sentences that describes a smooth interpolated embedding space centered around observed training sentence pairs. We then discuss our approach, AdvAug, to train NMT models using the embeddings of virtual sentences in sequence-to-sequence learning. Experiments on Chinese-English, English-French, and English-German translation benchmarks show that AdvAug achieves significant improvements over theTransformer (up to 4.9 BLEU points), and substantially outperforms other data augmentation techniques (e.g.back-translation) without using extra corpora.

## introduction recent work in neural machine translation @xcite @xcite has led to dramatic improvements in both research and commercial systems @xcite . however, a key weakness of contemporary systems is that performance can drop dramatically when they are exposed to input perturbations @xcite , even when these perturbations are not strong enough to alter the meaning of the input sentence. consider a chinese sentence, "zhejia feiji meiyou zhuangshang zhujia huo yiyuan, shizai shi qiji". if we change the word "huo (或)" to its synonym"ji (及)", the transformer model will generate contradictory results of "it was indeed a miracle that the plane did not touch down at home or hospital." versus "it was a miracle that the plane landed at home and hospital." such perturbations can readily be found in many public benchmarks and real-world applications. this lack of stability not only lowers translation quality but also inhibits applications in more sensitive scenarios. at the root of this problem are two interrelated issues: first, machine translation training sets are insufficiently diverse, and second, nmt architectures are powerful enough to overfit -and, in extreme cases, memorize -the observed training examples, without learning to generalize to unseen perturbed examples. one potential solution is data augmentation which introduces noise to make the nmt model training more robust. in general, two types of noise can be distinguished: (1) continuous noise which is modeled as a realvalued vector applied to word embeddings @xcite @xcite , and (2) discrete noise which adds, deletes, and/or replaces characters or words in the observed sentences @xcite @xcite @xcite . in both cases, the challenge is to ensure that the noisy examples are still semantically valid translation pairs. in the case of continuous noise, it only ensures that the noise vector lies within an l 2 -norm ball but does not guarantee to maintain semantics. while constructing semantics-preserving continuous noise in a high-dimensional space proves to be non-trivial, state-of-the-art nmt models are currently based on adversarial examples of discrete noise. for instance, @xcite generate adversarial sentences using discrete word replacements in both the source and target, guided by the nmt loss. this approach achieves significant improvements over the transformer on several standard nmt benchmarks. despite this promising result, we find that the generated adversarial sentences are unnatural, and, as we will show, suboptimal for learning robust nmt models. in this paper, we propose advaug, a new adversarial augmentation technique for sequence-tosequence learning. we introduce a novel vicinity distribution to describe the space of adversarial examples centered around each training example. unlike prior work @xcite , we first generate adversarial sentences in the discrete data space and then sample virtual adversarial sentences from the vicinity distribution according to their interpolated embeddings. our intuition is that the introduced vicinity distribution may increase the sample diversity for adversarial sentences. our idea is partially inspired by mixup @xcite , a technique for data augmentation in computer vision, and we also use a similar vicinity distribution as in mixup to augment the authentic training data. our advaug approach finally trains on the embeddings sampled from the above two vicinity distributions. as a result, we augment the training using virtual sentences in the feature space as opposed to in the data space. the novelty of our paper is the new vicinity distribution for adversarial examples and the augmentation algorithm for sequence-to-sequence learning. extensive experimental results on three translation benchmarks (nist chinese-english, iwslt english-french, and wmt english-german) show that our approach achieves significant improvements of up to 4.9 bleu points over the transformer @xcite , outperforming the former state-of-the-art in adversarial learning @xcite by up to 3.3 bleu points. when compared with widely-used data augmentation methods @xcite , we find that our approach yields better performance even without using extra corpora. we conduct ablation studies to gain further insights into which parts of our approach matter most. in summary, our contributions are as follows: ## approach: advaug in our approach advaug, the goal is to reinforce the model over virtual data points surrounding the observed examples in the training set. we approximate the density of p (x, y) in the vicinities of the generated adversarial examples and observed training examples. to be specific, we design two vicinity distributions @xcite to estimate the joint distribution of p (x, y): p adv for the (dynamically generated) adversarial examples and p aut for the (observed) authentic examples in s. given the training set s, we have: we train the nmt model on two vicinity distributions p adv and p aut . figure 1 illustrates examples sampled from them. as shown, a solid circle stands for an observed training example (i.e. a sentencepair) in s and a solid triangle denotes an adversarial example in a (x,y) . for p adv , we construct virtual adversarial examples (dashed triangles) to amend the sample diversity by interpolating the word embeddings of solid triangles. likewise, we interpolate the word embeddings of solid circles to model p aut for the (observed) authentic examples. this results in the dashed circles in figure 1 . unlike prior works on vicinal risk minimization @xcite , we do not directly observe the virtual sentences in p adv or p aut . this also distinguishes us from @xcite , who generate actual adversarial sentences in the discrete word space. in the remainder of this section, we will discuss the definition of p adv and p aut and how to optimize the translation loss over virtual sentences via mini-batch training. ## conclusion we have presented an approach to augment the training data of nmt models by introducing a new vicinity distribution defined over the interpolated embeddings of adversarial examples. to further improve the translation quality, we also incorporate an existing vicinity distribution, similar to mixup for observed examples in the training set. we design an augmentation algorithm over the virtual sentences sampled from both of the vicinity distributions in sequence-to-sequence nmt model training. experimental results on chinese-english, english-french and english-german translation tasks demonstrate the capability of our approach to improving both translation performance and robustness.