Towards Stronger Adversarial Baselines Through Human- AI Collaboration (2022)

Abstract: Natural language processing (NLP) systems are often used for adversarial tasks such as detecting spam, abuse, hate speech, and fake news. Properly evaluating such systems requires dynamic evaluation that searches for weaknesses in the model, rather than a static test set. Prior work has evaluated such models on both manually and automatically generated examples, but both approaches have limitations: manually constructed examples are time-consuming to create and are limited by the imagination and intuition of the creators, while automatically constructed examples are often ungrammatical or labeled inconsistently. We propose to combine human and AI expertise in generating adversarial examples, benefiting from humans’ expertise in language and automated attacks’ ability to probe the target system more quickly and thoroughly. We present a system that facilitates attack construction, combining human judgment with automated attacks to create better attacks more efficiently. Preliminary results from our own experimentation suggest that human-AI hybrid attacks are more effective than either human-only or AI-only attacks. A complete user study to validate these hypotheses is still pending.

## introduction humans have used language to deceive each other for millennia. with the advent of nlp systems, humans now work to deceive models and algorithms, from evading email spam filters in the early 2000s to defeating classifiers for social network spam, abusive language, misinformation, and more. more recently, humans have developed automated adversarial attacks that minimally modify text while changing the output of a classifier or other nlp systems @xcite . these automated attacks have the potential to be much more efficient than humans, helping attackers to find weaknesses in models and helping defenders find and patch those same weaknesses @xcite . the number of automated attacks continues to grow but their effectiveness remains low @xcite found that 90% of automated adversarial attacks changed the semantics of the original input or confused human annotators. we have observed similar behavior, as shown in doubtless, humans can be more effective than these attacks, given their effectiveness against realworld spam and abuse filters. we believe that the next step for adversarial attacks and robust nlp is human-ai collaboration, in which humans work with automated adversarial algorithms to pro-duce effective attacks efficiently. furthermore, realworld attackers are already doing this. spammers already use many different technologies to accomplish their tasks, including text spinners to rewrite text, html tricks to conceal suspicious text, botnets to scale up and avoid ip bans, and more. a typical spammer does not craft every message individually, but uses semi-automated techniques to generate many different messages 1 . in response, a growing amount of nlp research is now using human expertise through human-in-the-loop (hitl) methods to create new benchmarking datasets for evaluating and improving the robustness of nlp systems to adversarial inputs. thus far, human expertise in adversarial nlp tasks has been limited. there is a growing body of work in which humans are asked to craft inputs where a given model will perform poorly, but they receive little support in doing so -sometimes word saliences @xcite , sometimes model predictions @xcite , and sometimes even less. overall, the effort between humans and machines is still largely separate; that is, humans generate adversarial examples alone based on model interpretations, without directly interacting with any attack algorithms. in this paper, we study the potential of direct human-ai interaction for generating higher-quality adversarial examples for nlp tasks. we work with the state-of-the-art word-level attacks on benchmark datasets for sentiment analysis and abuse detection. we choose word-level attacks as they can be more subtle than character-level attacks, which have obvious misspellings. we design an interactive user interface that enables four types of attacks, including two human-ai collaboration methods. instead of a pure black-box environment, our interface explains the algorithm's search space and allows humans to modify and improve the perturbations while giving humans immediate feedback from the target nlp model. along with generated attacks, we collect data for user experience and user preference with regard to different attack approaches. we then further study the collected data and analyze the impact of proposed human-ai collaboration methods and the degree of improvement on the adversarial examples. at present, we have pilot data from using the system ourselves; a full user study is pending irb approval. 1 for an example of a spammer script that does this, see https://alexking.org/blog/2013/12/ 22/spam-comment-generator-script . we summarize our contributions as follows: the rest of the paper is structured as follows: section 2 discusses work related to our research. section 3 introduces our framework, the human-ai collaboration methods and the evaluation metrics. section 4 gives preliminary results and some brief analysis for our findings. section 5 explains the stages of experiments for generating and collecting quality data. finally, we conclude and discuss future work in section 6. ## related work we review prior work on automated adversarial attacks for nlp, and hitl in adversarial learning. automated adversarial attacks for nlp: with the growth of research that studies adversarial learning in nlp, a variety of attack methods have been developed on multiple levels. from characterlevel modifications such as hotflip @xcite , deepwordbug @xcite , and viper @xcite , to word-level perturbations such as bae @xcite , pso @xcite , pwws @xcite , and textfooler @xcite . many of them have been aggregated and organized by toolchains like textattack @xcite and openattack @xcite for easy access to researchers. for character-level attacks, although they show their effectiveness in many ways, they mainly fall in the following two categories: some of the character-level modifications can be seen as typos if an algorithm simply influences the embedding space by replacing/inserting/deleting one or a few characters in a word, such as deepwordbug @xcite , then they may be easily detected by a grammar checker tool, like grammarly foot_0 ; the others can introduce some unique encoding/decoding methods and transform letters to another form, such as viper @xcite that adds accent signs on top of each letter, and these modification may be easily identified by human. overall, character-level perturbations tend to be more obvious. on the other hand, the study of word-level attacks is more popular, as a substitute for a word may significantly impact the semantics of the text. many attack methodologies have been investigated for searching for the optimal synonym substitutions, including bert-based contextual prediction @xcite , gradient-based word swap @xcite , particle swarm optimization @xcite , and greedy word search with saliency scores @xcite . we summarize three attacks that are included in our framework. bae: bert-based adversarial examples (bae), a black-box contextual perturbation algorithm based on a bert masked language model (mlm). bae masks some part of the text, then replaces and inserts tokens into the text, using the bert-mlm to generate adversarial examples. pwws: probability weighted word saliency (pwws), a black-box greedy algorithm that ranks the importance of words based on the saliency score and calculates the classification probability that are used to determine the synonym substitution. textfooler: textfooler, a black-box greedy algorithm identifies the important words and replaces them with the words that are most semantically similar and grammatically correct with a higher priority until the prediction is altered. these automated word-level attacks mostly rely on the knowledge of existing target models and algorithms' intensive search to locate the best synonym substitutions. however, recent work @xcite shows that the quality of generated adversarial examples is actually far from satisfactory, with respect to the low attack success rate across domains, incorrect grammar, and distorted meaning. hitl in adversarial learning: as the capacity of automated algorithms may be limited, many researchers propose incorporating crowd-sourcing into generating and annotating adversarial exam-ples. the dynabench framework asks humans to manually construct examples where an nlp system would perform poorly @xcite . a hitl qa system that asks humans to write adversarial questions that break a qa system while remaining answerable by humans @xcite . the adversarial nli project asks humans to annotate mislabeled data and uses humans as adversaries to create a benchmark natural language inference (nli) dataset for a more robust nlp model @xcite . the most related work compares the performance of human-and machinegenerated word-level adversarial examples for nlp classification tasks @xcite . however, existing work falls short of direct collaboration between humans and ai. the advantages of human crowd-sourcing and that of automated algorithms are still quite distinct. ## framework in our framework, we study the potential of direct human-ai collaboration for generating higherquality adversarial examples. at the time of submission, we have completed the design of the framework, confirmed the details for human-ai collaboration, and implemented the interactive user interface. ## preliminary results our hypotheses are that with minimal human collaboration, compared to automated attacks alone, the attacks would yield more promising results that are meaningful while holding correct grammar and semantics. in our preliminary work, we already see promise for this direction. or. txt auto txt hitl txt 4 friends , 2 couples , 2000 miles , and all the pabst blue ribbon beer they can drink -it 's the ultimate road-trip . (pos. 62%) 4 friends , 2 couples , 2000 miles , and all the pabst disconsolate ribbon beer they can drinkit 's the ultimate road-trip . (neg. 84%) 4 friends , 2 couples , 2000 miles , and all the pabst cheap beer they can drinkit 's the ultimate road-trip . (neg. 83%) as a pilot experiment, to test the viability of the framework before recruiting participants, the authors used the framework on themselves to collect 532 unique adversarial examples generated from the sst-2 dataset. by studying these examples, we have seen the following patterns (which we hypothesize will extend to the full experiments): success rate: figure 2 shows the attack success rate across all attack methods. though an automated attack may have a higher attack success rate due to the advantage of intensive search and the nlp model-oriented design, humans can achieve comparable attack success rate if provided with better human-ai interaction. additionally, manually crafted attacks without any assist cannot compete with the those generated through other methods. grammar and plausibility: figure 3 presents the average scores for grammar and plausibility, where the error bars denote the standard errors of the scores. the scores are aggregated and averaged per the attack method from the self-evaluation results over the 532 adversarial examples. it is obvious that human-generated adversarial examples on average have higher scores considering the grammatical properties and plausibility. manual attack and hitl methods seem to produce higherquality adversarial examples with the assistance of automated algorithms, as compared to automated auto manual select saliency 1 2 3 4 5 evaluation scores grammar plausibility figure 3: grammar & plausibility 0 100 200 queries query auto manual select saliency 1 2 3 4 5 effort scores effort figure 4: queries & effort attacks, these methods loosen the constraints on various degrees and grant humans more freedom to make more modifications if needed. therefore humans have more flexibility crafting grammatically correct and plausible adversarial examples. queries and human effort: the top of figure 4 displays the number of queries it takes for an automated algorithm or a human to choose their word substitutions. the bottom of the figure gives the average effort scores for each attack method. the error bars denote the standard errors of the scores. the results illustrate that humans are able to perturb an nlp model with more effort but fewer queries, and the gray-box setting, which includes additional information for the participants, is easier to attack than the black-box settings. the extra information provides some insight and explanation about how an automate algorithm understands the nlp model and how an nlp model decides the predictions. ## planned experiments we plan to hire approximately 54 adult native english speakers, of whom we expect a subset to be experts in nlp or linguistics, from our local university to generate adversarial examples, and additional adult native english speaker amt workers for peer-evaluation. unlike the recent work of @xcite , which relies entirely on online crowd-sourcing on amt, we carry on in-person experiments for attack generation, where we provide a few examples and detailed instructions to the participants to show how our interface operates, and what the standards/baselines are for evaluating the adversarial examples. we expect to obtain higher-quality data by bringing participants into a more controlled environment where it's easier to provide instruction, answer questions, and receive feedback. to motivate participants through the process, we have designed an incentive payment plan. details are included in a.3 of the appendix. ## conclusion & future work humans have excellent intuition about language, but weak intuition about deep networks; automated attacks are often the opposite. given the weak performance of manual attacks and automated attacks against nlp systems, some type of human-ai collaboration is essential to truly evaluate their robustness, and to be prepared for the inevitable attacks from real-world adversaries. in the future, we will carry out the experiments as designed, and further include the imdb movie review dataset curated by @xcite . as the texts in the imdb dataset are often longer, this dataset may provide participants greater flexibility in modifying the examples. we believe that further study into collaboration methods will lead to a better understanding of adversarial attacks and more robust nlp models. we hope to provide a new benchmark for hitl adversarial learning while we continue exploring other effective human-ai collaboration methods. we hope that our framework will help researchers and practitioners better evaluate the robustness of nlp models to the best attacks that humans and algorithms can construct, and then improve their models by training on these adversarial examples.