Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis (2020)

Abstract: Aspect-based sentiment analysis (ABSA) aims to predict the sentiment towards a specific aspect in the text. However, existing ABSA test sets cannot be used to probe whether a model can distinguish the sentiment of the target aspect from the non-target aspects. To solve this problem, we develop a simple but effective approach to enrich ABSA test sets. Specifically, we generate new examples to disentangle the confounding sentiments of the non-target aspects from the target aspect’s sentiment. Based on the SemEval 2014 dataset, we construct the Aspect Robustness Test Set (ARTS) as a comprehensive probe of the aspect robustness of ABSA models. Over 92% data of ARTS show high fluency and desired sentiment on all aspects by human evaluation. Using ARTS, we analyze the robustness of nine ABSA models, and observe, surprisingly, that their accuracy drops by up to 69.73%. We explore several ways to improve aspect robustness, and find that adversarial training can improve models’ performance on ARTS by up to 32.85%. Our code and new test set are available at https://github.com/zhijing-jin/ARTS_TestSet

## introduction aspect-based sentiment analysis (absa) is an advanced sentiment analysis task that aims to classify the sentiment towards a specific aspect (e.g., burgers or fries in the review "tasty burgers, and crispy fries."). the key to a strong absa model is it is sensitive to only the sentiment words of the target aspect, and therefore not be interfered by the sentiment of any non-target aspect. although stateof-the-art models have shown high accuracy on existing test sets, we still question their robustness. specifically, given the prerequisite that a model outputs correct sentiment polarity for the test example, we have the following questions: a robust absa model should both meet the prerequisite and have affirmative answers to all the questions above. for example, if a model makes the correct sentiment classification (i.e., positive) for burgers in the original sentence "tasty burgers, and crispy fries", it should flip its prediction (to negative) when seeing the new context "terrible burgers, but crispy fries". hence, these questions together form a probe to verify if an absa model has high aspect robustness. unfortunately, existing absa datasets have very limited capability to probe the aspect robustness. for example, the twitter dataset @xcite has only one aspect per sentence, so the model does not need to discriminate against non-target aspects. in the most widely used semeval 2014 laptop and restaurant datasets @xcite , for 83.9% and 79.6% instances in the test sets, the sentiments of the target aspect, and all non-target aspects are all the same. hence, we cannot decide whether models that make correct classifications attend only to the target aspect, because they may also wrongly look at the non-target aspects, which are confounding factors. only a small portion of the test set can be used to answer our target questions proposed in the beginning. moreover, when we test on the subset of the test set (59 instances in laptop, and 122 instances in restaurant) where the target aspect sentiment differs from all nontarget aspect sentiments (so that the confounding subq. generation strategy example prereq. source: the original sample from the test set tasty burgers, and crispy fries. (tgt: burgers) q1 revtgt: reverse the sentiment of the target aspect terrible burgers, but crispy fries. q2 revnon: reverse the sentiment of the non-target aspects with originally the same sentiment as target tasty burgers, but soggy fries. ## data generation as shown in ## evaluating absa models we use our enriched test set as a comprehensive test on the aspect robustness of absa models. ## error analysis for data generation we analyze the error types in the subset of arts that was fixed by human judges. two most significant error types are wrong antonyms (∼2%), such as "the weight of the laptop is @xmath0, and negation which causes grammatical errors (∼1.1%). in future work, we can fix the latter by applying a grammatical error correction system on top of our generation. also, revtgt and revnon cannot be applied to 1.4∼6.6% instances with complicated sentiment expressions which rely on commonsense. for example, "a 2-hour wait" is negative bust too difficult to alter in our current generation framework. it needs more advanced models such as text style transfer @xcite . ## related work robustness in nlp robustness in nlp has attracted extensive attention in recent works @xcite . as a popular method to probe the robustness of models, adversarial text generation becomes an emerging research field in nlp. techniques include adding extraneous text to the input @xcite , character-level noise @xcite , and word replacement @xcite . using the adversarial generation techniques, new adversarial test sets are proposed for several tasks such as paraphrasing @xcite and entailment @xcite . aspect-based sentiment analysis absa has emerged as an active research area recently. early works hand-craft sentiment lexicons and syntactic features for rule-based classifiers @xcite . recent neural network-based models use architectures such as lstm @xcite , cnn @xcite , attention mechanisms @xcite , capsule network @xcite , and the pretrained model bert @xcite . similar to the motivation in our paper, some work shows preliminary speculation that the current absa datasets might be downgraded to sentence-level sentiment classification @xcite . ## conclusion in this paper, we proposed a simple but effective mechanism to generate test instances to probe the aspect robustness of the models. we enhanced the original semeval 2014 test sets by 294% and 315% in laptop and restaurant domains. using our new test set, we probed the aspect robustness of nine absa models, and discussed model designs and better training that can improve the robustness. in natural language processing and the 9th international joint conference on natural language processing, emnlp-ijcnlp 2019, hong kong, china, november 3-7, 2019, pages 4567-4577. association for computational linguistics. yuan zhang, jason baldridge, and luheng he. 2019b. paws: paraphrase adversaries from word scrambling. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, naacl-hlt 2019, minneapolis, mn, usa, june 2-7, 2019, volume 1 (long and short papers), pages 1298-1308. association for computational linguistics.