R einforce B ug: A Framework to Generate Adversarial Textual Examples (2021)

Abstract: Adversarial Examples (AEs) generated by perturbingining examples are useful in improving the robustness of Deep Learning (DL) based models. Most prior works generate AEs that are either unconscionable due to lexical errors or semantically and functionally deviant from original examples. In this paper, we present ReinforceBug, a reinforcement learning framework, that learns a policy that is transferable on unseen datasets and generates utility-preserving and transferable (on other models) AEs. Our experiments show that ReinforceBug is on average 10% more successful as compared to the state-of the-art attack TextFooler. Moreover, the target models have on average 73.64% confidence in wrong prediction, the generated AEs preserve the functional equivalence and semantic similarity (83.38%) to their original counterparts, and are transferable on other models with an average success rate of 46%

## introduction machine learning (ml) models have attained remarkable success in several tasks such as classification and decision analytics. however, ml models specifically deep learning (dl) based models, are often sensitive to adversarial examples (aes). aes consist of modified training data samples that preserve the intrinsic utilities of the ml solutions, but influence target classifier's predictions between original and modified inputs @xcite . recent works @xcite @xcite have demonstrated that (i) including aes as a part of training data can enhance the robustness and generalization of the ml models, and (ii) these examples can be utilized to test the robustness of ml models and help understand their security vulnerabilities and limitations. previous works on generating aes have attained success in image @xcite and on a few conventional text classification tasks such as sentiment, text entailment and movie reviews @xcite @xcite . nevertheless, generating aes for discrete textual data is still a challenge @xcite . textual aes generated by the prior works have the following limitations: utility preservation textual aes require to satisfy task-specific constraints such as lexical rules (spellings or grammar), semantic similarity and functional equivalence to original examples. yet, most of the current state-of-the-art methods do not satisfy these constraints, thereby generating imperceivable aes for end-users @xcite . a few recent works @xcite @xcite have considered semantic similarity constraint, but other constraints have been barely explored @xcite . knowledge transferability most prior works @xcite ; @xcite ; @xcite ) generate one to one example-specific aes, i.e., for a given example x, they create an example x . each instance x is considered independent in a corpus c, and no relationship between different instances in the corpus is assumed. therefore, the knowledge gained by transforming an example to ae is limited to a single example and is not reused on other examples. this process is both time-consuming and may not generalize the identified vulnerabilities of the target model. word replacement strategy most prior works use a single word replacement strategy such as synonym substitution @xcite @xcite or character perturbation @xcite to generate aes. this strategy has two main disadvantages: (i) the aes generated by using a character-based replacement strategy, contains many spelling errors that result in unnatural text; and (ii) multiple word transformations with its synonym in a synonym-based replacement strategy affect the language's fluency, making it sound unnatural. for instance, @xcite generated an ae "jimmy wales is a big, fucking idiot liar friggin nincompoop deception". this ae sounds unnatural and is grammatically incorrect. handling noisy datasets most prior works generated aes for datasets such as yelp (yelp) and fake news @xcite which do not contain many spelling mistakes or out-of-the-vocabulary words. however, human generated natural text is prone to lexical errors. for example, tweets usually contain informal language, misspellings and unknown words. prior works, such as @xcite @xcite ) that considered synonym substitution strategy, cannot deal with such noisy dataset. ## related works adversarial attacks are extensively studied in computer vision domain @xcite . early works in adversarial text attacks were inspired by generative adversarial networks (gans) @xcite . wong 2017 showed that gan-based reinforcement learning algorithms become unstable after a few perturbations. later, heuristic-based methods such as word removal @xcite , out-of-vocabulary (oov) words @xcite and synonym replacement @xcite @xcite have been proposed. among these studies, deepwordbug @xcite generates aes by randomly transforming a word by oov word in an example. this approach is practical in producing aes efficiently; however, it generates aes that can be detected by the end-user due to large proportion of lexical errors. an attack framework named textbugger @xcite was proposed to generate adversarial samples using the multi-level approach. it identified important words for each example and replaced them with optimal bugs. the approach considered both character-level and word-level transformations. another recent attack, textfooler @xcite generates utility-preserving aes by replacing an important word in an example with its grammatically equivalent synonym. the study also evaluates the generated aes against semantic similarity constraint. in this study, we compare our method with textbugger and textfooler. ## experimental setup this section presents our experiment details. ## results results of our experiments are presented here. ## analysis and discussion this section analyse the results. ## conclusion overall, this study proposes reinforcebug, a reinforcement learning-based framework to generate utility-preserving aes against the state-of-the-art text classifiers under black-box settings. extensive experiments demonstrate that it effectively generates utility-preserving aes that are transferable to other models, and the learned policy is transferable to the unseen datasets. it identifies semantic concatenation and synonym substitution attacks as a significant threat to text-classifiers and suggests defences against these attacks should be explored in the future to improve their robustness.