Open Aspect Target Sentiment Classification with Natural Language Prompts (2021)

Abstract: For many business applications, we often seek to analyze sentiments associated with any arbitrary aspects of commercial products, despite having a very limited amount of labels or even without any labels at all. However, existing aspect target sentiment classification (ATSC) models are not trainable if annotated datasets are not available. Even with labeled data, they fall short of reaching satisfactory performance. To address this, we propose simple approaches that better solve ATSC with natural language prompts, enabling the task under zero-shot cases and enhancing supervised settings, especially for few-shot cases. Under the few-shot setting for SemEval 2014 Task 4 laptop domain, our method of reformulating ATSC as an NLI task outperforms supervised SOTA approaches by up to 24.13 accuracy points and 33.14 macro F1 points. Moreover, we demonstrate that our prompts could handle implicitly stated aspects as well: our models reach about 77% accuracy on detecting sentiments for aspect categories (e.g., food), which do not necessarily appear within the text, even though we trained the models only with explicitly mentioned aspect terms (e.g., fajitas) from just 16 reviews - while the accuracy of the no-prompt baseline is only around 65%.

## introduction measuring targeted sentiments from text toward certain aspects or subtopics has immediate commercial value. for example, a hotel chain might want to base their business decisions on the proportion of customer reviews being positive toward their room cleanliness and front desk services. manually reading through thousands of reviews would be prohibitively expensive, calling for automated solutions. adopting existing supervised models for aspect target sentiment classification (atsc) @xcite figure 1 : an overview of our prompt-based models for atsc. (a) bert / gpt-2 lm is pretrained on unlabeled review texts and we convert atsc into a language modeling task during training and testing. (b) bert nli is pretrained on nli datasets and we convert atsc into two entailment tasks. @xcite appears to be an obvious choice at first glance. however, the accuracy of these models in practice are often unsatisfactory due to the lack of labeled data from domains of interest. we could direct practitioners to annotate few domainspecific review texts, but atsc models would need to generalize from those limited examples and return accurate predictions upon seeing any arbitrary aspects and sentiments. even if we could get hold of more data, it would be difficult to collect enough labels to handle all aspect-sentiment cases that customers could ever query. the way people would write customer reviews inspires our solutions to this label scarcity issue. for example, consider two plausible sentences from hotel reviews: mike and the rest of the staff were very polite and responded quickly to our requests. the front desk service is excellent. supposing we only observe the first one, if an atsc model could tell that the first entails the second, or the second naturally follows the first, we could determine that the customer feels positive about the front desk service. based on this intuition, we propose two atsc models based on natural language prompts: our first method appends cloze question prompts (e.g., the service is ) to the review text and predicts how likely it is to observe good, bad, and ok as the next word. the second method treats the review as a premise and the prompt sentence with the sentimental next word as a hypothesis, and predicts whether the review entails the prompt. with pretraining, these prompts give us natural ways of leveraging more abundant unlabeled reviews or large natural language inference (nli) datasets to overcome the scarcity of labeled data. an overview of our methods is provided in figure 1 . most of previous atsc efforts focus on fully supervised training, assuming enough in-domain labels on relevant aspects. going beyond this typical setting, we also test our model on more "open" situations where reviews and aspects for testing are less similar to the training examples. experimental results show that our methods are not only capable of producing reasonable predictions without any labeled data (zero-shot) but also constantly outperform sota no-prompt baselines under supervised settings. moreover, our promptbased models are robust to domain shifts: after training with aspect-based sentiments in one domain, we observe that the models can accurately predict sentiments associated with aspect terms appearing in reviews from a completely unseen domain. also, our models that are trained on explicitly mentioned aspect terms could generalize well to implied categorical aspects. foot_0 2 related work 2.1 aspect-based sentiment analysis @xcite is one of the first academic work to discuss the task of analyzing opinions targeted towards different aspects or topics within the text. ## methods to maximally leverage the large data resources of unlabeled review texts and nli datasets, we propose two ways of reformulating atsc: the first converts atsc into next/masked word prediction; the second transforms the task into nli entailment predictions. ## experiments in the experiments, we test the generalization capability of our methods on more real world-like conditions where there are far fewer training examples similar to the testing examples. using atsc datasets from semeval 2014 task 4 subtask 2 @xcite , we evaluate our models on the full spectrum of in-domain training data sizes covering the zero-shot and full-shot (i.e., fully supervised) cases. similar to the settings of scao and rush (2021), we train our models with randomly re-sampled training sets of sizes {zero, 16, 64, 256, 1024, full}. furthermore, we conduct cross-domain evaluation where we train the models on restaurant reviews and test on laptop reviews, and vice versa. finally, we train the models on atsc and test them on aspect category sentiment classification (acsc), another absa variant, to evaluate the robustness to an unseen querying aspect distribution. unlike aspect terms, these categories such as food, service, price, and ambience usually do not explicitly appear within the text, but implicitly stated through aspect terms or overall context. foot_3 ## conclusion and future work in this paper, we examined our prompt-based atsc models leveraging lm and nli under zero-shot, few-shot, and full supervised settings. we observe a significant amount of improvements over the noprompt baselines in nearly all configurations we have tested. in particular, we find that our nli model performs well with lower amounts of training data, while the bert lm model does better when more labels are available. in addition, we have seen that it could effectively utilize cross-domain labels and recognize implicit aspects, suggesting that it would potentially be more applicable in real-life scenarios. for future work, one direction is to adapt our aspect-dependent prompts to the models that jointly perform aspect term extraction and sentiment classification, such as @xcite . secondly, we could explore potential ways of combining atsc, masked/next word prediction, and nli into a unified task in order to take the full advantage of both our unlabeled text and nli pretraining. lastly, it would be an interesting analysis to determine whether there are any strong linguistic patterns among correct or incorrect predictions that each of our models make -such findings could allow us to have more detailed insights into the potential behaviors of our prompt-based models.