Adversarial Attack and Defense of Structured Prediction Models (2020)

Abstract: Building an effective adversarial attacker and elaborating on countermeasures for adversarial attacks for natural language processing (NLP) have attracted a lot of research in recent years. However, most of the existing approaches focus on classification problems. In this paper, we investigate attacks and defenses for structured prediction tasks in NLP. Besides the difficulty of perturbing discrete words and the sentence fluency problem faced by attackers in any NLP tasks, there is a specific challenge to attackers of structured prediction models: the structured output of structured prediction models is sensitive to small perturbations in the input. To address these problems, we propose a novel and unified framework that learns to attack a structured prediction model using a sequence-to-sequence model with feedbacks from multiple reference models of the same structured prediction task. Based on the proposed attack, we further reinforce the victim model with adversarial training, making its prediction more robust and accurate. We evaluate the proposed framework in dependency parsing and part-of-speech tagging. Automatic and human evaluations show that our proposed framework succeeds in both attacking state-of-the-art structured prediction models and boosting them with adversarial training.

## introduction adversarial examples, which contain perturbations to the input of a model that elicit large changes in the output, have been shown to be an effective way of assessing the robustness of models in natural language processing (nlp) @xcite @xcite @xcite a small perturbation to the sentence x changes one word from "am" to "fires". this change makes the perturbed example i fires a writer ungrammatical. even if the perturbed example is "i fire a writer" that meets the rules of grammar, the true output structure is still different from the input sentence "i am a writer". more importantly, this true parse is unknown to the attacker, which hinders the next update step. there are multiple challenges that have to be addressed in building an efficient and effective attacker for structured prediction models in nlp. @xcite pointed out two major prob-lems encountered by attackers of nlp tasks. first, since words are discrete, making small disturbances to words in the gradient direction is difficult. secondly, there is no guarantee that the generated adversarial examples are fluent. in addition to these two problems, there is a unique challenge faced by attackers of structured prediction tasks. while small perturbations to images or even texts typically do not change their classification labels, small perturbations to sentences in structured prediction may very likely change the true output structures. in other words, many structured prediction tasks are very sensitive to small perturbations in the input sentence. consequently, almost all the existing attacking methods are not directly applicable to structured prediction. to illustrate this challenge, we take adversarial attack of dependency parsing as an example (figure 1 ). we use the fast gradient sign method (fgsm) @xcite as the attack method, which is a classic gradient-based attacker that perturbs the input by minimizing the likelihood of the true output. when applied to nlp tasks @xcite , fgsm perturbs the embeddings of the words in the input sentence and then replaces individual words based on the new embeddings. however, there is no guarantee that the new sentence has the same parse tree as the original sentence. once the true output parse tree becomes unknown, subsequent updates become impossible in fgsm, resulting in perturbation that might be insufficiently adversarial. in figure 1 , after just one step of perturbation, the sentence indeed has a different parse tree that is unknown to the attacker. to address the aforementioned problems, we propose to attack structured prediction models with sequence-to-sequence (seq2seq) sentence generators. before attack, the seq2seq generator is trained by reinforcement learning based on a novelty designed reward function that evaluates the output of the victim structured prediction model against an ensemble of multiple reference models of the same structured prediction task. during attack, the seq2seq generator is simply applied to input sentences to produce adversarial examples. our framework has the following features. we evaluate our framework on the dependency parsing task and the pos tagging task. both automatic and human evaluations show that our method outperforms previous approaches in attacking state-of-the-art structured prediction models as well as boosting these models with adversarial training for better accuracy and robustness. the code and the trained model can be found at https: //github.com/winniehan/structure_adv. ## sentence-level adversarial attack and defense we aim to mislead a structured prediction model by generating adversarial examples x from the original examples x using a seq2seq generator. we train the generator using reinforcement learning following @xcite . the reward function for reinforcement learning evaluates whether the generated sentence could induce an incorrect output from the victim model, and the evaluation is facilitated by two reference models. in addition, the reward function also evaluates the quality of the generated sentence. figure 2 illustrates the overall architecture of our proposed model, which mainly consists of three parts: a generator, an evaluation module, and the victim model. we use dependency parsing as our example structured prediction task and name the victim parser as parser a and the reference parsers as parsers b and c. in contrast to the word-level attackers described in section 2.3, our proposed method aims to do sentence-level attacks that are capable of generating a sentence of a different length and structure instead of merely making local word changes. ## experiments on dependency parsing we first perform experiments on dependency parsing, a well-known structured prediction task. ## conclusion building an effective adversarial attacker for structured prediction models is challenging. the biggest challenge is the sensitivity of the output to small perturbations in the input in structured prediction. in this paper, we propose a novel framework to attack structured prediction models in nlp. our framework consists of a structured-output evaluation criterion based on reference models and a seq2seq sentence generator. we propose to utilize reinforcement learning to train the sentence generator based on the evaluation criterion. our attack experiments on dependency parsing and pos tagging show that our proposed framework can produce high-quality sentences that can effectively attack current state-of-the-art models. our defense experiments show that adversarial training using the adversarial samples generated by our model can be used to improve the original model. we believe that our framework is general and can be applied to many other structured prediction tasks in nlp, such as neural machine translation, semantic parsing and so on.