Understanding Pre-trained BERT for Aspect-based Sentiment Analysis (2020)

Abstract: This paper analyzes the pre-trained hidden representations learned from reviews on BERT for tasks in aspect-based sentiment analysis (ABSA). Our work is motivated by the recent progress in BERT-based language models for ABSA. However, it is not clear how the general proxy task of (masked) language model trained on unlabeled corpus without annotations of aspects or opinions can provide important features for downstream tasks in ABSA. By leveraging the annotated datasets in ABSA, we investigate both the attentions and the learned representations of BERT pre-trained on reviews. We found that BERT uses very few self-attention heads to encode context words (such as prepositions or pronouns that indicating an aspect) and opinion words for an aspect. Most features in the representation of an aspect are dedicated to the fine-grained semantics of the domain (or product category) and the aspect itself, instead of carrying summarized opinions from its context. We hope this investigation can help future research in improving self-supervised learning, unsupervised learning and fine-tuning for ABSA. The pre-trained model and code can be found at https://github.com/howardhsu/BERT-for-RRC-ABSA .

## introduction as a form of self-supervised learning in nlp, pre-trained language models (lms) like the masked lm in bert @xcite @xcite yield significant performance gains when later fine-tuned on downstream nlp tasks. recent studies also showed impressive results on tasks in aspect-based sentiment analysis (absa) @xcite @xcite @xcite , which aims to discover aspects and their associated opinions @xcite . although there are existing studies of the hidden representations and attentions of lms about tasks such as parsing and co-reference resolution @xcite @xcite , it is unclear how lms capture aspects and sentiment/opinion from large-scale unlabeled texts. this paper attempts to investigate and understand the inner workings of the pretext task of the masked language model (mlm) in transformer and their connections with tasks in absa. this may benefit the following problems: (1) improving fine-tuning of absa if we have a better understanding about the gap between pretext tasks and fine-tuning tasks; (2) more importantly, self-supervised (or unsupervised) absa without fine-tuning to save the expensive efforts on annotating absa datasets (for a new domain). we are particularly interested in fine-grained token-level features that are typically required by absa and how mlm as a general task can cover them during pre-training. typical tasks of absa are: aspect extraction (ae), aspect sentiment classification (asc) @xcite @xcite @xcite @xcite @xcite @xcite @xcite and end-to-end absa (e2e-absa) @xcite . ae aims to extract aspects (e.g., "battery" in the laptop domain), asc identifies the polarity for a given aspect (e.g., positive about battery) and e2e-absa is a combination of ae and asc that detects aspects and their associated polarities simultaneously. existing studies show that tasks in absa require the understanding of the interactions of aspects (e.g., "screen" in the laptop domain) and its contexts, including sentiment (e.g., "clear") @xcite @xcite . as such, we believe how the hidden representation of an aspect encodes features about being an aspect and summarizing opinions of that aspect are crucial for absa. this paper represents a new addition to the existing analysis of bert in @xcite , which focuses on studying the behavior of bert's hidden representation and self-attentions for general purposes. we focus on how the self-supervised training of bert is prepared for fine-grained features that are important for absa. we leverage the annotated data of absa in our analysis to draw the relevance between pre-trained features and labels for absa. note that we do not use any of such annotations for fine-tuning or training. also, we do not study lms that carry extra human supervision for sentiment analysis (such as using opinion lexicons as in @xcite )) because we are interested in to what degree a general self-supervised task such as mlm can cover specific tasks in absa. unlike @xcite @xcite , we focus on masked language model (mlm) for fine-grained token-level features instead of next sentence prediction (nsp) since the latter is not widely adopted for pre-trained models @xcite . our main finding is that bert (pre-trained on reviews) encodes rich semantic knowledge about the domain and aspect itself into the hidden representations of an aspect but uses almost no dedicated features for opinions. inside bert, very few self-attention heads are learned for encoding salient context words for finding an aspect or summarizing opinion words for an aspect. this suggests the pros and cons of mlm. for example, predicting a masked aspect word to learn features for an aspect is a weak selfsupervised learning task for absa. this leads to future directions on alternative self-supervised learning tasks for absa. ## pre-trained lms and datasets we expect to simplify our analysis on the same latent space for multiple domains of absa. as such, we pre-train bert on reviews with large coverage of domains (product categories). the training corpus is a combination of amazon reviews @xcite and yelp review datasets foot_1 , which give us a review corpus of 20+ gb in size. we start from fine-tuning bert base on such a corpus for 4 epochs. we train it on a single titan rtx gpu for 10 days. to understand the hidden representation of an aspect and its formulation from self-attention, we leverage the popular semeval 2014 task 4 and semeval-2016 task 5 in absa with annotations about aspects and their associated opinions. these benchmark datasets cover the domains of laptop and restaurant. we sample 150 examples from each domain as the validation data for analysis. note that we do not use any annotated data for fine-tuning or training because we are interested in how relevant the features from pre-trained bert to absa. ## analysis to see the inner workings of masked language modeling (mlm) on reviews, we first review mlm and transformers. then we perform two types of evaluations: self-attention of aspects and hidden representations on aspects. transformer @xcite is a neural architecture that purely uses multi-head self-attentions to learn hidden representations of input texts. unlike the classic lstm or cnn, the connections in selfattention can be viewed as a fully-connected graph on nodes of tokens without strong inductive bias on contexts or prior knowledge on relative positions. so transformer uses positional embeddings to encode tokens at different positions. further, multi-head attentions can be viewed as typed relations to model various kinds of relations among tokens. mlms aim to recover texts corrupted with masked tokens. for example, from "the [mask] is clear." in the laptop domain, one can easily guess that [mask] is probably "screen". as a result, the transformer model must use self-attentions to infer the hidden representation of "screen" from "the", "is" and "clear". note that the first embedding layer of bert and the mlm prediction heads are both context-independent because they just contain word embeddings, whereas the other internal layers inbetween are context-dependent. as such, how bert encodes such contexts into the representation of an aspect is important for absa. ## summary and open problems our analysis showed that mlm tends to learn very fine-grained features and dedicate most of the aspects' features to domains and semantics of aspects themselves rather than opinions. we believe using pretrained bert is good for ae or the extraction part of e2e-absa but poor for summarizing opinions in asc or polarity detection in e2e-absa. end tasks may still require a good amount of examples to explore the large feature space of bert. we believe this analysis leads to the following open problems: alternative self-supervised learning tasks besides mlm. we believe mlm is far from perfect as a pretext task for absa and both learned features for ae and asc can be improved. we believe the main weakness of mlm is that when learning representations for an aspect word, it does not need to know the sentiment in most cases and being an aspect or not is not a strong feature to learn mlm. how to design a pretext task to learn disentangle aspect features and opinion features from fine-grained features of other semantics is still an open problem. for aspect features, one may group reviews for the same item (e.g., a product) to encourage the signal for aspects against other items. for sentiment features, one may use rating as a weakly supervised signal to strengthen aspect words with sentiment.