Adversarial Sample Generation for Aspect based Sentiment Classification (2022)

Abstract: Deep learning models have been proven vulnerable towards small imperceptible perturbed input, known as adversarial samples, which are indiscernible by humans. Initial attacks in Natural Language Processing perturb characters or words in sentences using heuristics and synonyms-based strategies, resulting in grammatical incorrect or out-of-context sentences. Recent works attempt to generate contextual adversarial samples using a masked language model, capturing word relevance using leave-one-out (LOO). However, they lack the design to maintain the semantic coherency for aspect based sentiment analysis (ABSA) tasks. Moreover, they focused on resource-rich languages like English. We present an attack algorithm for the ABSA task by exploiting model explainability techniques to address these limitations. It does not require access to the training data, raw access to the model, or calibrating a new model. Our proposed method generates adversarial samples for a given aspect, maintaining more semantic coherency. In addition, it can be generalized to low-resource languages, which are at high risk due to resource scarcity. We show the effectiveness of the proposed attack using automatic and human evaluation. Our method outperforms the state-of-art methods in perturbation ratio, success rate, and semantic coherence.

## introduction sentiment analysis is a well-established area in natural language processing (nlp), and finds its applications in recommendation systems, national security-sensitive applications, curating online trends, etc. @xcite @xcite @xcite . considering sentiment alone can only provide high-level insights, not sufficing to analyze reviews containing multiple attributes, known as aspects. aspect level sentiment analysis (absa) provides more fine-grained information by classi-fying the sentiment towards a specific aspect of the product @xcite . recently, deep learning and transformer-based approaches have obtained state-of-the-art results in numerous classification applications such as emotion, sarcasm detection, etc. including absa @xcite @xcite @xcite . however, these classification algorithms can be easily fooled by maliciously crafted (adversarial) examples @xcite . adversarial examples expose the system vulnerabilities and also help to improve the robustness of the model. the adversarial sample generation has been extensively explored to assess the resilience of the neural model, in the field of computer vision @xcite @xcite . it has shown improvement in the robustness and generalized capability of the model via adversarial training @xcite . the generation of such out-of-distribution samples against nlp models is more challenging than computer vision due to the discrete nature of text. in addition, semantic consistency and grammatical accuracy of generated adversarial samples should also be preserved. initial attempts to attack nlp models have shown to adapt the fast gradient sign methods (fgsm) @xcite and generative adversarial networks (gans) based methods from computer vision, to apply perturbations on the embedding space of the text @xcite @xcite . there is, however, a difficulty in mapping perturbed continuous embedding space to discrete token space in these methods. there are prior works which explored character-level and word-level perturbation algorithms using synonym replacement and language model based approaches @xcite @xcite . recent studies have revealed the vul-nerability of bert-based (bidirectional encoder representations from transformers) text classification models in a black box setting using synonymsbased @xcite and masked-language model (bert-mlm) based approaches @xcite @xcite . most existing attack methods are primarily focused on text classification, including document level sentiment classification and other question answering tasks. however, in the context of absa, these algorithms lack the design to maintain semantic coherency with the actual example, which is the foremost requirement of adversarial examples. for example, consider the example from semeval laptop dataset, thanks for great service and shipping! adversarial example generated by sota method for aspect service is thanks for continued concern and shipping! it is clear from the example that the overall semantics and aspect term have been changed. to maintain the semantic coherency with the actual example, aspect term should not be changed. additionally, the presence of multi-word aspects in the sentence presents another challenge to preserve the semantics. for example, quick and has built in virus control. here, the sentiment towards aspect built in virus control is positive. the adversarial example generated by sota method is quick and has flaws in virus control, which fails to preserve the semantic coherency and aspect term. a recent attempt was made to attack the absa classifier by adding misspellings and punctuation to the actual sentences in the black-box setting @xcite . these perturbations, however, can fool the classifier, but lack semantic and grammatical correctness. moreover, these modifications may also be corrected by grammar or spelling checkers, thus increasing the likelihood of an attack failure. these approaches measure the word saliency by removing it from the sentence and calculating the drop in probability of correct class prediction (loo). almost all the efforts have been directed towards high-resource languages such as english. there has been no work on exposing vulnerabilities in low-resource nlp models, which are at high risk due to resource scarcity in low-resource languages. the existing absa attack is only applicable to the english language, as it uses language-specific rules and dictionaries. for example, the letter e can be replaced by 3 (homoglyphs) or s with 5. to address these limitations, we propose an ad-versarial example generation algorithm designed for absa for a given aspect. our proposed method is not dependent on the language-specific rules; hence, it can be generalized to low-resource languages with some optimizations. our proposed algorithm applies perturbations at the word level by exploiting the model explainability technique, shap (shapley additive explanations) (lundberg and lee, 2017). we extend shap to bert based absa task by incorporating aspect information so that it can generate word saliency scores according to the given aspect. shap considers various combination of words to determine the word importance, unlike in earlier attempts where importance is dependent on only one word. moreover, our proposed algorithm can generate adversarial samples for single word and multi-word aspect terms. we summarize the contributions of our work as follows: generating the adversarial examples against neural models to assess resilience of the model has been explored extensively in the field of computer vision @xcite @xcite . however, attempts to expose the vulnerabilities of nlp models are relatively few @xcite . initial attempts to attack nlp models adapt the fgsm @xcite ) from computer vision. the key idea is to apply small perturbations to the embedding space of text @xcite in the direction of the gradient. gans @xcite based method are also explored by applying perturbations in the latent space. however, these approaches often lack in semantic correctness @xcite . subsequently, several methods focused on character level and word level perturbations in white box @xcite or black box setting @xcite . the generated adversarial samples are easily identifiable by human and also lacks the grammatical correctness and semantic coherency with the seed sentences. to maintain grammatical correctness and semantic consistency, @xcite proposed to perturb important words with the top k words obtained from the glove embedding vectors. authors also explored synonyms @xcite and language model @xcite based approaches for perturbations. morris et al. (2020) introduced textattack to implement adversarial attacks in python. it is composed of four basic components: a goal function, a set of constraints, a transformation, and a search method. textattack implements a wide range of adversarial attacks and supports a variety of datasets and models, such as bert and transformer-based models. ## threat model our target model is a bert based absa classifier. the adversary aims to generate adversarial samples against the target model due to their huge success in many nlp tasks, including absa @xcite @xcite . adversary's knowledge: the adversary has the black-box access of the target model. it queries the target model to get the prediction vector. the adversary does not have access to the data used to train the target model, rather it owns some test samples of similar distribution, which are used for the adversarial sample generation against the target model. adversary's goal: given an input sentence s, consisting of n tokens w 1 , w 2 , w 3 , . . . , w n with m aspects @xmath0, . . . , asp m , where asp @xmath1, . . . , w s i +l i (contiguous subsequence of words from s), with ground truth sentiment label y asp i towards aspect asp i , and a target model m (s, asp i ) = y asp i . here l i is the number of words in the asp i (m and @xmath2. the goal of the adversary is to perform an un-targeted attack, i.e., find adversarial sample s adv for aspect asp i , causes m to perform misclassification, i.e., m (s adv , asp i )! = y asp i . at the same time, s adv should satisfy the following properties: i). s adv should be semantically similar to s. this is achieved by sim(s, s adv ) > ϵ, where sim is cosine similarity and ϵ is the threshold value. ii). s adv should be grammatically correct. iii). humanp red(s adv , asp @xmath3, where humanp red is classification by human. iv). s adv should preserve the aspect asp i for which the opinion is expressed. 1 ## methodology we present model interpretable attack algorithm to generate high quality adversarial examples to assess the robustness of absa model by applying perturbations at the word-level. the detailed architecture of our proposed approach is depicted in figure 1 . there are 3 main components, viz. word saliency generator (1a), adversarial generator (1b) and the target model (1c). first, the saliency generator uses the model interpretation technique (shap) @xcite to score the relevance of tokens in the input sentence for a given aspect. relevance scores are used by the adversarial generator to generate the adversarial samples by applying perturbations at the word level. it runs iteratively until the generated adversarial sample is able to fool the target model. ## experimental setup we use bert-base and bert-base-multilingual as target models for english and hindi, respectively. for adversarial example generation, we set the value of top candidates j to 50 and the threshold value ϵ to 0.8. to identify the pos tags, we use stanfordnlp library foot_0 for english and hindi. datasets: to evaluate our proposed attack, we use the following absa datasets: the datasets are annotated with four classes, viz., positive, negative, neutral, and conflict. both the datasets contain fewer instances of conflict class. so, we focus on 3 classes by excluding the conflict class. (more details are present in a.1) baselines: we define the following baselines: baseline 1, 2, and 3 are originally proposed for text classification tasks. we extend them to an absa task by passing a pair of input containing sentence and aspect separated by [sep ] token. we implement baseline 2 and 3 using textattack framework. ## experimental results and analysis experimental results for both the languages for all the metrics are summarized in ## conclusion in this paper, we have presented an effective algorithm to generate adversarial examples for assessing the resilience of the bert based aspect based sentiment classification model. to generate adversarial examples, we exploit the model's explainability to identify the word saliency. we propose replace operation for contextual words and insert operation for aspect term to generate more semantically similar sentences. we have evaluated our proposed algorithm on two benchmark datasets, english and hindi. extensive experiments and human evaluation show that our proposed algorithm outperforms the state-of-art attack methods in success rate, perturbation ratio, and semantic preservation. in our current work, we have evaluated the robustness of the sentiment classification task only. in future, we would extend this work to evaluate the robustness of both aspect term extraction and sentiment classification.