An Empirical Study on Adversarial Attack on NMT : Languages and Positions Matter (2021)

Abstract: In this paper, we empirically investigate adversarial attack on NMT from two aspects: languages (the source vs. the target language) and positions (front vs. rear). For autoregressive NMT models that generate target words from left to right, we observe that adversarial attack on the source language is more effective than on the target language, and that attacking front positions of target sentences or positions of source sentences aligned to the front positions of corresponding target sentences is more effective than attacking other positions. We further exploit the attention distribution of the victim model to attack source sentences at positions that have a strong association with front target words. Experiment results demonstrate that our attention-based adversarial attack is more effective than adversarial attacks by sampling positions randomly or according to gradients.

## introduction despite remarkable progress in recent years, neural machine translation (nmt) models are vulnerable to small perturbations @xcite . adversarial training, which allows nmt models to learn from adversarial samples with perturbations, as a general approach, is widely used to improve the robustness of nmt @xcite @xcite @xcite . generally, nmt models yield target translations in an autoregressive way 1 , which makes previous incorrectly predicted target tokens have a negative impact on future tokens to be generated. however, most approaches to generating nmt adversarial examples inject perturbations only into source sentences. hence, are nmt models more vulnerable to adversarial attack on the source side? what roles do injecting perturbations into source sentences or into target translations play in improving the robustness of nmt? the key interest of this paper is to attempt to answer these questions by an empirical and comparative study on different adversarial attacks on nmt models. first, we investigate adversarial attacks on the source side versus those on the target side. this study is to know which attack is more effective for nmt by measuring performance drop of the attacked models. second, we empirically study the impact of attacking different positions on either source sentences or target translations to find whether nmt robustness is sensitive to positions. third, based on the findings of the study, we propose a new adversarial attack generation method based on attention distribution. our contributions can be summarized as follows: ## related work robustness is a well-known problem for neural networks @xcite . recent years have witnessed that many adversarial training approaches have been proposed to improve the robustness of nmt models. cheng et al. (2018) generate adversarial samples at the lexical and feature level, and apply the adversarial learning to make adversarial samples natural. zhao et al. (2018) utilize generative adversarial networks to generate adversarial examples that lie on the data manifold by searching in the semantic space of dense and continuous data representations. ebrahimi et al. (2018) propose an attack framework for character-level nmt, which uses gradient to rank adversarial manipulations and to search for adversarial examples via either greedy search or beam search methods. belinkov and bisk (2018) attack character-level nmt by randomizing the order of letters or randomly replacing letters with their adjacent letters on the keyboard. vaibhav et al. (2019) use back translation to generate adversarial samples that emulate natural noises. cheng et al. (2020a) exploit the projected gradient method combined with gradient regularization to generate adversarial samples. zou et al. (2019) employ reinforcement learning to decide which positions to attack. tan et al. (2020) present a method to change inflectional morphology of words to craft plausible and semantically similar adversarial examples. emelin et al. (2020) propose to generate adversarial examples by eliciting disambiguation errors. all these approaches attack the source side of nmt in different ways. however distortions exist in not only the source language, but also the target language. this inspires us to compare the effectiveness of adversarial attack on the source and target side to nmt models. we have found that the nmt models are vulnerable to both the source and target attack. however, to our best knowledge, only @xcite and @xcite take noises in target sentences into account. they generate adversarial samples for both source and target sentences. their target-side adversarial examples are generated according to the attacked positions in corresponding source sentences, while their sourceside adversarial samples are generated by randomly sampling positions to attack. we improve their method by attacking the source side according to the attention distribution. experiments validate the effectiveness of our method. ## data and setup we conducted experiments on two translation tasks: english-chinese and english-japanese. data for english-chinese translation are from the united nations english-chinese corpus @xcite . we built the training/validate/test set for this task by randomly sampling 3m/2k/2k sentence pairs from the whole corpus. for the english-japanese translation task, we aggregated the training set of kftt @xcite , jesc @xcite and ted talks @xcite as our training set, which consists of 3.9m sentence pairs. we evaluated our models on the validation set and test set of kftt @xcite . we split words into sub-word units with subword regularization @xcite and built a shared vocabulary of 32k subwords for both english-chinese and english-japanese. we used the base transformer model @xcite with 512 hidden units as the victim model. the hyper-parameters of the base transformer follows the default setting in @xcite . we implemented the adversarial attack and training methods of @xcite and followed their hyper-parameter setting in our experiments. the details of our implementation is shown in section 4. we injected perturbations into either source sentences or target sentences to generate adversarial examples which were used to evaluate nmt models. since we could not inject perturbations into the target inputs of nmt models at the test time, we evaluated nmt models with target-side adversarial samples at training time on the validation dataset. except where otherwise specified, the performance of the victim model was measured by word accuracy on the validation data. if we evaluated the victim model on the test set, detokenized bleu @xcite and bertscore @xcite were reported. although the target-side inputs of nmt models could not be attacked at test time, there still exists noise or errors in them due to error propagation in the autoregressive decoding. evaluating nmt with perturbated target sentences at training time enables us to analyze the vulnerability of nmt to the noise in target-side inputs, and inspires us to improve the robustness of nmt models to such noise. ## implementation details the adversarial attack and training framework used in this paper is based on @xcite . they inject perturbations into the source/target sentences by replacing a word in a sentence with the words that are semantically similar to the words being replaced. words to be replaced in a source sentence are sampled according to the uniform distribution, while those in a target sentence are sampled according to the attention distribution. we tried three different ways to sample words to inject perturbations into source sentence in section 8. given a word to be replaced, @xcite use a bi-directional language model to choose candidate words from vocabulary which share similar semantics to it, and then use gradients to search a word from candidate words to replace it. @xcite combine a left-to-right and right-to-left language model to rank candidate words, while we combine the two uni-directional language models by multiplying their likelihood for simplicity. @xcite train their nmt models with both clean data and adversarial samples from scratch. to save training time, we pretrain our nmt models with clean data before adversarial training. ## adversarial attack on source vs. target in this section, we compare the effect of the source and target attack according to the performance of the victim model. we adversarially inject perturbations into source sentences and keep target translations unchanged (clean) for the source attack while the target attack works the other way around. our adversarial examples for both the source and target attack are generated by the method of @xcite . to make the comparison between the source and target attack fair, we randomly sample positions to attack for both of them. results are shown in ## adversarial attack at different positions in this section, we investigate the impact of attacked positions in the source and target sentences on nmt. we start with adversarial attack on the target side. adversarial attacks at the front of a target sentence are supposed to be more effective than those at the end of the target sentence, since noises in the front of the target sentence will negatively affect future target tokens, while noises at the end of the target sentence could not affect already generated tokens for a left-to-right decoder. given a sentence of length l, we uniformly select 10 anchor positions from the sentence: results of adversarial attack on different anchor positions on the target side are shown in figure 1 . on all translation tasks, the word translation accuracy of the victim model goes up as attacked positions move from the starting position to the end of target sentences, which confirms that attacking at the front of a target sentence is more effective than attacking at the end. we also perform adversarial attack on source sentences at different anchor positions. results are displayed in figure 2 . we measure the performance of the victim model for the source attack at different positions on the test set with the metric of bleu. on both english-chinese and english-japanese tasks, bleu scores go up as the attacked positions move from the start to the end of source sentences, which indicates that attacking the front of a source sentence is also more effective than attacking the end for both english-chinese and english-japanese translation. we suppose that the reason for this is that words at front positions of source sentences usually align to words at front positions of target sentences for the two language pairs. experiment results in section 7 empirically validate this hypothesis. ## attention weights at different positions in section 6, we suppose that words at front positions of source sentences usually align to words at front positions of target sentences for both english-chinese and english-japanese. we empirically validate this by comparing the attention weights from source tokens at different positions to the first target token. following the sampling technique in section 6, we uniformly select 10 anchor positions from a source sentence and then sample positions surrounding these anchor positions according to the distribution formulated in eq (2). for every anchor position, we report the sum of attention weights from the sampled source tokens around the anchor position to the first target token. the results are shown in figure 3 . as expected, the attention weights from the sampled source tokens to the first target token go down as the their corresponding positions move from the start to the end of the source sentence in both english-chinese and english-japanese translation, which confirms that words at front positions of source sentences have a stronger association with words at front positions of target sentences than other positions for the two language pairs. ## adversarial attack based on attention distribution in section 6, we have found that generating perturbations at front positions on the target side is more effective than attacking other positions. as attention weights in nmt models can be seen as the strength of association between the source and target tokens @xcite . hence we sample positions of a source sentence to inject perturbations according to the attention distribution. particularly, the query used to produce the attention distribution is the representation of the first target token and the key is the set of representations of source tokens. bleu scores and bertscores of the three adversarially trained models on the test set are shown in ## conclusion in this paper, we have empirically investigated adversarial attack on nmt models. we compare adversarial attack on the source vs. target side, and find that the former is more effective than the latter. we also study adversarial attack at different positions in either source or target sentences, and observe that attacking front positions in either source or target sentences for english-chinese and english-japanese translation is more effective than attacking back positions. we further exploit attention distribution to attack words of a source sentence at positions that have a high association with words at front positions of the corresponding target sentence. experiments validate the effectiveness of our proposed attention-based attack.