Contrasting Human- and Machine-Generated Word-Level Adversarial Examples for Text Classification (2021)

Abstract: Research shows that natural language processing models are generally considered to be vulnerable to adversarial attacks; but recent work has drawn attention to the issue of validating these adversarial inputs against certain criteria (e.g., the preservation of semantics and grammaticality). Enforcing constraints to uphold such criteria may render attacks unsuccessful, raising the question of whether valid attacks are actually feasible. In this work, we investigate this through the lens of human language ability. We report on crowdsourcing studies in which we task humans with iteratively modifying words in an input text, while receiving immediate model feedback, with the aim of causing a sentiment classification model to misclassify the example. Our findings suggest that humans are capable of generating a substantial amount of adversarial examples using semantics-preserving word substitutions. We analyze how human-generated adversarial examples compare to the recently proposed TextFooler, Genetic, BAE and SememePSO attack algorithms on the dimensions naturalness, preservation of sentiment, grammaticality and substitution rate. Our findings suggest that human-generated adversarial examples are not more able than the best algorithms to generate natural-reading, sentiment-preserving examples, though they do so by being much more computationally efficient.

## introduction the vulnerability of natural language processing (nlp) models to adversarial examples has received widespread attention @xcite @xcite . text processing models have been shown to be susceptible to adversarial input perturbations across tasks, including question answering and text classification @xcite . the concept of adversarial examples originated in computer vision @xcite , and in that domain defines perturbations of input data to neural networks that are barely perceptible to the human viewer. due to the discrete nature of text, however, that definition is less applicable in an nlp context, since every perturbation to the input tokens is unavoidably perceptible. consequently, recent work aims to perturb textual inputs while preserving the sequence's naturalness and semantics (i.e., rendering changes imperceptible on these dimensions). however, as shown by @xcite , achieving these desiderata is challenging because even small perturbations can render a text meaningless, grammatically incorrect or unnatural, and furthermore several proposed adversarial attacks fail routinely to achieve them. if the algorithms are modified to ensure that they do achieve the desiderata then their rate of generating successful examples greatly diminishes, suggesting that the reported success rates of recently proposed attacks might represent an overestimation of their true capabilities. this, in turn, raises the question of whether valid word-level adversarial examples are routinely possible against trained nlp models. in this work, we aim to address this question by incorporating human judgments into the adversarial example generation process. specifically, we report on a series of data collection efforts in which we task humans to generate adversarial examples from existing movie reviews, while instructed to strictly adhere to a set of validity constraints. in contrast to previous work (e.g., @xcite , and in an attempt to replicate a word-level attack's mode of operation, human participants were only able to substitute individual words, and were not allowed to delete or insert new words into the sequence. this represents a blackbox attack scenario, since human participants do not have access to information about the model's parameters or gradients. participants worked in a web interface (figure 1 ) that allowed them to conduct word-level substitutions while receiving immediate feedback from a trained model. after collecting the human-generated adversarial examples, we compare them to a set of automated adversarial examples for the same sequences using four recently proposed attacks: textfooler @xcite , genetic @xcite , bae @xcite , and se-memepso @xcite . using human judgments from an independent set of crowdworkers, we assess for each generated adversarial example (human and automated) whether the perturbations changed the sequence's overall sentiment and whether they remained natural. we find that humans are capable of generating label-flipping word-level adversarial examples (i.e., the classifier misclassifies the sequence after human perturbation) in approximately 50% of the cases. however, when comparing the ground truth labels of perturbed sequences to the sentiment labels provided by the independent set of human annotators, we find that only 58% of the labelflipping human adversarial examples preserve their target sentiment after perturbation. this is considerably lower than for the best automated attacks, which exhibit a label consistency of up to 93% (textfooler) after perturbation. in terms of naturalness, we find no statistically significant differences between the human and machine attacks for the majority of comparisons. we furthermore observe that the human-generated sequences introduce fewer grammatical errors than most attacks. these findings show that under similar constraints, machine-generated, word-level adversarial examples are comparable to human-generated ones with respect to their naturalness and grammaticality. importantly, however, humans require, on average, only 10.9 queries to run against the model to generate label-flipping adversarial examples, while some attacks require thousands. we believe that our findings could further push the development of reliable word-level adversarial attacks in nlp, and our method and data might aid researchers in identifying human-inspired, more efficient ways of conducting adversarial word substitutions against neural text classification models. the remainder of this paper is structured as follows. section 2 discusses previous work related to our research. section 3 describes both phases of our data collection approach, i.e., the human generation of word-level adversarial examples and the subsequent validation of human-and machine-generated sequences with respect to their preservation of semantics and naturalness. this is followed by the analysis reported in section 4, and a discussion of our findings and future work in section 5. finally, we conclude our paper in section 6. ## related work adversarial attacks for nlp. adversarial attacks have been increasingly applied to nlp, with a diverse set of attack types being investigated, ranging from character-level edits @xcite , word-level replacements @xcite , adding text to the input @xcite , paraphrase-level modifications @xcite , to creating adversarial examples from scratch @xcite . this work focuses on word-level attacks. word-level attacks. our work builds on existing efforts on word-level adversarial attacks. attacks of this type can be further distinguished by whether the adversary has access to the model parameters (i.e., white-box) or is restricted to accessing only the predicted labels or confidence scores (i.e., black-box) @xcite . word-level attacks have been explored for nlp tasks such as question answering @xcite , natural language inference @xcite , and text classification @xcite . a range of methodologies has been explored for finding optimal synonym substitutions, including population-based gradient-free optimization via genetic algorithms @xcite , word saliency probability weighting @xcite , similarity and consistency filtering @xcite , sememe-based word substitution and particle swarm optimization-based search @xcite , and contextual perturbations from masked language models @xcite . word-level perturbations have also been used as part of data augmentation strategies to certifiably improve model robustness @xcite . existing efforts to detect and defend against word-level adversarial examples resort to adversarial data augmentation (e.g., @xcite as well as rule-based @xcite and learning-based @xcite approaches to identify adversarially perturbed inputs. evaluating word-level attacks. of particular importance for this paper is how adversarial at-tacks can be evaluated against various dimensions. adversarial attack performance has been shown to vary across evaluation dimensions including adversarial success rates, readability and content preservation @xcite , as well as linguistic constraints such as semantics, grammaticality, the edit distance between original and perturbed text, and non-suspicion @xcite . however, to the best of our knowledge, such evaluation efforts are limited to automated attacks and how humans perform at creating word-level adversarial examples across these dimensions remains unexplored. ## method our data collection process has two stages: first, we ask human annotators to perform a word-level adversarial attack for given input sequences. to this end, we prepared an online interface that lets participants perturb input sequences on a word-level whilst receiving immediate feedback as to how their changes affected classifier confidence. second, we ask an independent set of crowdworkers to evaluate the generated adversarial examples. ## analysis after collecting the human judgments we analyze both the human and machine attacks' performance on generating adversarial examples. the primary objective is to investigate the feasibility of wordlevel adversarial examples that adhere to validity criteria as suggested in previous work @xcite . we use the attack success rate (asr) as the initial metric to evaluate the performance of either attack mode (human and algorithmic). the attack success rate is defined as the percentage of successful adversarial examples (i.e., those that are misclassified after perturbation) to all perturbed sequences. we observe that overall, workers were generally able to generate successful movie reviews for task 1 (for 90% of the submitted sequences the model predicted the desired sentiment) and led the model into misclassification in task 2 for the majority of the cases (asr 80%). for task 3, workers also crucially, when we introduced constraints in task 4, the asr drops to 49%, suggesting an increased difficulty of generating word-level adversarial examples when attempting to preserve the sentiment and naturalness of the text. it is worth mentioning that we conducted additional experiments with expert annotators (i.e., academic researchers with experience in nlp) and found that the asr for task 4 was even lower compared to the crowdworkers. as a comparison, we report the asr of all word-level attacks in figure 2 depicts the distribution of times needed for the human participants to generated the wordlevel adversarial examples in task 4. we observe that participants need on average 111.29 minutes (standard deviation: 119.77) to complete the task. ## discussion despite some reported successes, recent work questions the validity of machine-generated word-level adversarial examples. central to that critical view are evaluation criteria on which the adversarial ex-amples fall short @xcite . the argument is that with these criteria as constraints, most (if not all) word-level adversarial examples are deemed invalid. in this work, we investigated how feasible such adversarial examples can be generated by humans when explicitly asked to respect a set of validity constraints. the underlying reasoning was that human performance might have been able to improve the quality standard of word-level adversarial examples. our findings suggest that with respect to the success rate as well as the preservation of semantics and naturalness, humans do not outperform stateof-the-art attack algorithms in generating wordlevel adversarial substitutions. but they also do not differ much. this finding speaks to the difficulty of the task. however, our findings suggest that while humans do not outperform machines with respect to the aforementioned criteria, they are able to generate adversarial examples of similar quality using a fraction of the attack iterations required by the algorithms. humans are able to generate label-flipping examples with only a handful of queries, while the algorithmic attacks might need thousands of inference steps to find successful word substitutions. further, humans do this without introducing more grammatical errors than the algorithmic attacks. in sum, this work suggests that humans produce adversarial examples comparable to state-of-the-art attacks but at a fraction of the computational costs. with a better understanding of how humans achieve this, future work could try to close that gap and develop more computationally efficient algorithmic adversarial attacks inspired by human language reasoning. ## conclusion this paper compared human and machine performance on generating word-level adversarial examples against a text classification model for sentiment analysis. we observe that human-generated adversarial examples do not preserve a sequence's sentiment as well as machine-generated ones do, but are similar in terms of their naturalness after label-flipping perturbation. while these findings do not suggest that humans outperform algorithms for this task, we find that they achieve similar performance in a much more efficient manner. we therefore believe that our work can build the foundation for future research aiming to further optimize algorithmic word-level attacks by potentially adapting human-inspired strategies for this task.