Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation (2022)

Abstract: Generating adversarial examples for Neural Machine Translation (NMT) with single Round-Trip Translation (RTT) has achieved promising results by releasing the meaning-preserving restriction. However, a potential pitfall for this approach is that we cannot decide whether the generated examples are adversarial to the target NMT model or the auxiliary backward one, as the reconstruction error through the RTT can be related to either. To remedy this problem, we propose a new definition for NMT adversarial examples based on the Doubly Round-Trip Translation (DRTT). Specifically, apart from the source-target-source RTT, we also consider the target-source-target one, which is utilized to pick out the authentic adversarial examples for the target NMT model. Additionally, to enhance the robustness of the NMT model, we introduce the masked language models to construct bilingual adversarial pairs based on DRTT, which are used to train the NMT model directly. Extensive experiments on both the clean and noisy test sets (including the artificial and natural noise) show that our approach substantially improves the robustness of NMT models.

## introduction in recent years, neural machine translation (nmt) @xcite @xcite has achieved rapid advancement in the translation performance @xcite . however, the nmt model is not always stable enough, as its performance can drop significantly when small perturbations are added into the input sentences @xcite . such perturbed inputs are often referred to as adversarial examples in the literature, and how to effectively generate and utilize adversarial examples for nmt is still an open question. conventional approaches @xcite for generating nmt adversarial examples always follow the meaning-preserving assumption, i.e., an nmt adversarial example should preserve the meaning of the source sentence but destroy the translation performance drastically @xcite . with the meaning-preserving restriction, the researchers try to add perturbations on the source inputs as small as possible to ensure the meaning of the source sentence is unchanged, which severely limits the search space of the adversarial examples. additionally, it is much problematic to craft a minor perturbation on discrete text data, since some random transformations (e.g., swap, deletion and replacement) may change, or even reverse semantics of the text data, breaking the aforementioned meaning-preserving assumption. to break this limitation, @xcite introduce a new criterion for nmt adversarial examples: an effective nmt adversarial example imposes minor shifting on the source and degrades the translation dramatically, would naturally lead to a semantic-destroyed round-trip translation result. take the case in figure 1 as an example: x δ reverses the semantics of input x by replacing "巨大 (huge)" to "轻便 (light)". since the semantics of x and x δ are com-pletely different, it is unreasonable to use the original target sentence of x to evaluate the attacks directly. therefore, @xcite propose to evaluate the bleu score between x δ and its reconstructed sentence xδ from the source-target-source round-trip translation (rtt), as well as the bleu score between the original sentence x and its reconstructed sentence x. they take the decrease between the two bleu scores mentioned above as the adversarial effect. specifically, if the bleu decrease exceeds a predefined threshold, x δ is concluded to be an adversarial example for the target nmt model. while achieving promising results by breaking the meaning-preserving constraint, there are two potential pitfalls in the work of @xcite : to address the issues mentioned above, we first propose a new criterion for nmt adversarial examples based on doubly round-trip translation (drtt), which can ensure the examples that meet our criterion are the authentic adversarial examples for the target nmt model. specifically, apart from the source-target-source rtt @xcite , we additionally consider a target-sourcetarget rtt on the target side. the main intuition is that an effective adversarial example for the target nmt model shall cause a large bleu decrease on the source-target-source rtt while maintaining a small bleu decrease on target-source-target rtt. based on this criterion, we craft the candidate adversarial examples with the source-target-source rtt as @xcite , and then pick out the authentic adversaries with the target-source-target rtt. furthermore, to solve the second problem, we introduce the masked language models (mlms) to construct the bilingual adversarial pairs by performing phrasal replacement on the generated monolingual adversarial examples and the original target sentences synchronously, which are then utilized to train the nmt model directly. experiments on both clean and noisy test sets (including five types of artificial and nature noise) show that the proposed approach not only generates effective adversarial examples, but also improves the robustness of the nmt model over all kinds of noises. to conclude, our main contributions are summarized as follows: ## method in this section, we first describe our proposed criterion for nmt adversarial examples, and then present the way of constructing the bilingual adversarial pairs. ## experimental settings we evaluate our model under artificial noise in @xmath0, and under natural noise in @xmath1. the details of the experiments are elaborated in this section. ## case study and limitations in although the proposed approach achieves promising results, it still has limitations. a small number of authentic adversarial examples may be filtered out when the large d tgt (y, y δ ) is caused by f (x δ ), we will ameliorate this problem in the further. ## conclusion and future work we propose a new criterion for nmt adversarial examples based on doubly round-trip translation, which can ensure the examples that meet our criterion are the authentic adversarial examples. additionally, based on this criterion, we introduce the masked language models to generate bilingual adversarial pairs, which can be used to improve the robustness of the nmt model substantially. extensive experiments on both the clean and noisy test sets show that our approach not only improves the robustness of the nmt model but also performs well on the clean test sets. in future work, we will refine the limitations of this work and then explore to improve the robustness of forward and backward models simultaneously. we hope our work will provide a new perspective for future researches on adversarial examples.