A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis (2022)

Abstract: Sentiment analysis is an important task in natural language processing. In recent works, pre-trained language models are often used to achieve state-of-the-art results, especially when training data is scarce. It is common to fine-tune on the downstream task, usually by adding task-specific layers on top of the model. In this paper, we focus on aspect-based sentiment analysis, which involves extracting aspect term, category, and predicting their corresponding polarities. In particular, we are interested in few-shot settings. We propose to reformulate the extraction and prediction tasks into the sequence generation task, using a generative language model with unidirectional attention (GPT2 is used unless stated otherwise). This way, the model learns to accomplish the tasks via language generation without the need of training task-specific layers. Our evaluation results on the single-task polarity prediction show that our approach outperforms the previous state-of-the-art (based on BERT) on average performance by a large margins in few-shot and full-shot settings. More importantly, our generative approach significantly reduces the model variance caused by low-resource data. We further demonstrate that the proposed generative language model can handle joint and multi-task settings, unlike previous work. We observe that the proposed sequence generation method achieves further improved performances on polarity prediction when the model is trained via joint and multi-task settings. Further evaluation on similar sentiment analysis datasets, SST-2, SST-5 and OOS intent detection validates the superiority and noise robustness of generative language model in few-shot settings.

## introduction sentiment analysis @xcite @xcite aims at detecting the overall polarity of a user generated text, which describes the user opinion for an entity. however, user may express opinions about an entity at different granularity. for example, a user may give an overall rate about a restaurant service, and then explains fine-grained review about specific aspects, such as food quality, waiting time, waitress service, environment, etc. aspect-based sentiment analysis task @xcite aims at addressing this problem, where user sentiment is annotated at coarse and fine-grained levels. moreover, user can express conflicting opinions for different aspects of an entity. traditionally, neural-based models are employed as a single-task model for aspect-based sentiment analysis (absa) task, similar to machine reading comprehension task (mrc) @xcite . for example, a pre-trained bert language model is fine-tuned for absa term polarity prediction (single-task) as a classifier. in this approach, a task-specific layer is fine-tuned for each downstream task, such as a layer for aspect term polarity classification, and a different layer for aspect term span extraction @xcite . recently, generative language models with unidirectional self-attention, which are pre-trained by causal language modeling loss (predicting next word given the history), have shown promising performance when fine-tuned on the downstream tasks (gpt2) @xcite . using this approach, the language model learns the downstream task as language generation, where the task is represented as a serialized text. @xcite proposed gpt3, a large-scale generative language model with few-shot ability. gpt3 learns to solve the downstream task by conditioning on few examples in the prompt, without any parameter update (in-context learning). motivated by the ability of the pre-trained generative language model (gpt2) for solving the downstream tasks in a generative manner, we propose a generative language model for absa task. the evaluation results indicate that the proposed approach achieves better performance with significantly lower variance compared to the previous state-of-the-art models (which are based on bert pre-trained model) on few-shot and full-shot settings, for single-task polarity prediction of aspect term and aspect category. for example, using 1% (20 examples) of training data on restaurant domain for aspect term polarity prediction task, our proposed gpt2 model outperforms bert-pt @xcite by 9 points on average accuracy and reduced standard deviation by 6.2 points, as shown in figure 1 @xcite . moreover, when fine-tuned on multiple tasks, such as aspect term extraction, term polarity, aspect category detection, and category polarity, the proposed model improved single-task performance, such as aspect term extraction (measured by f1 score). 1 the contributions of our proposed generative language model are, in the next sections, we discuss the proposed model and presents the evaluation results. in section 2, the previous state-of-the-arts are described. section 3 explains the task of aspect-based sentiment analysis (absa) (section 3.1) followed by reformulating absa task as language generation (section 3.2). in section 4, the evaluation results for single, joint and multi-task settings are presented for semeval14 @xcite and semeval16 @xcite and sst-2, sst-5 and oos intent detection datasets. ## related works sentiment analysis is characterized by three categorizes, i.e. document, sentence, and aspect level @xcite @xcite . in this section, we review the previous models developed for aspect-based sentiment analysis (absa) @xcite . earlier works on absa task focused on developing feature engineered models @xcite . @xcite proposed a model based on using convolutional neural network (cnn) for aspect term extraction task only. the approach uses two types of pre-trained embeddings, a generalpurpose embedding and a domain-specific one. then, a softmax classification layer is used to classify each word to identify aspect term start and end positions, or non-related words. @xcite proposed multi-granularity alignment network (mgan), a coarse-to-fine approach for single-task aspect term polarity prediction using recurrent neural network (rnn) @xcite . they defined aspect category as coarse-level and aspect term as fine-level sentiments, and further leveraged high-resource out-of-domain data for pre-training. this way, the knowledge is transferred from coarsegrain domains (single-opinion prediction) to multigrain domains (absa task). with the advent of bert @xcite as a pre-trained bidirectional language model, which presents a powerful contextualized word representation for the language understanding downstream tasks, several models are proposed for absa task using bert as feature extraction. @xcite defined absa task as question answering @xcite , named review reading comprehension (rrc), and used bert as the base model, with separate heads for aspect term extraction (as span extraction) and term polarity prediction. to enhance rrc performance, they introduced a post-training algorithm, which additionally pre-train the model on out-of-domain data from amazon and yelp review datasets, and additionally on mrc question answering dataset @xcite . these result in additional training set of 1, 151, 863 for laptop domain, 2, 677, 025 more examples for restaurant domain, and 87, 599 training examples from mrc dataset. @xcite proposed an approach based on conditional random field (crf) @xcite , combined with bert for aspect term extraction and term polarity prediction tasks. two modules are employed for improving aspect term extraction and term polarity prediction of bert model. first, a parallel approach is used which combines predictions for aspect term and polarity from last four layers of bert in parallel. moreover, a hierarchical aggregation module is also examined, where predictions of previous layers of bert are fed into the next layer. @xcite combines glove pre-trained embedding @xcite with deep contextualized representation of bert to enhance the representation of word vectors for predicting aspect term polarity. the proposed bert-il model predicts aspect term polarity by learning a similarity between glove vector of aspect term and its contextualized representation extracted from bert. first, the aspect term representations are extracted from multiple layers of bert, and fed into a self-attention layer. finally, it is further fine-tuned on absa task for performance improvement. @xcite proposed a model based on bart @xcite for aspect category detection. they rank all aspect categories with different polarities and select the pair with highest score. @xcite proposed an nli approach based on bert for single task of polarity prediction only, using extra pretraining on review datasets. in section 4, evaluation of our proposed generative language model are compared with the recent bert-based models. ## model this section describes aspect-based sentiment analysis task (absa), the proposed generative language model approach, details of the datasets, model training, and evaluation metrics. ## experiments the proposed generative language model is evaluated on five tasks. single-task setting includes aspect term polarity and aspect category polarity prediction, eq. ( 5 )(6), for restaurant and laptop domains. joint-task includes a) aspect term extraction and polarity eq. ( 7 ) and b) aspect category detection and polarity eq. ( 8 ). finally, multi-task setting comprises all sub-tasks, i.e. aspect term extraction (sb1), aspect category detection (sb3), and their polarity predictions (sb2 and sb4), eq. ( 9 ). the evaluation of our proposed generative language model is compared with recent bert-pt @xcite model. we have reproduced results of bert-pt on full-shot settings, since we include examples with conflict polarity. other bert-based models such as bert-il @xcite has not open-sourced code, and therefore they are not included in few-shot evaluation. ## conclusion in this paper, we proposed a generative language model for aspect based sentiment analysis (absa). by reformulating the task as language generation, the model learns to predict aspects and their polarities via language generation. evaluation results on single-task polarity prediction on few and full shot setting indicate that the proposed approach outperforms prior arts, which are based on discriminative classification using bert as encoder, with higher average performance and lower variance. on jointask and multi-task settings, the proposed model shows better performance on single-task polarity prediction metrics. additionally, evaluation results on coarse-grained (sst2), fine-grained (sst5) sentiment analysis datasets, and oos intent detection dataset indicate the better and more robust few-shot performance of generative language model. furthermore, qualitative analysis indicates that using multi-task setting improves model prediction via supervision across aspect term and category.