S enti BERT : A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics (2020)

Abstract: We propose SentiBERT, a variant of BERT that effectively captures compositional sentiment semantics. The model incorporates contextualized representation with binary constituency parse tree to capture semantic composition. Comprehensive experiments demonstrate that SentiBERT achieves competitive performance on phrase-level sentiment classification. We further demonstrate that the sentiment composition learned from the phrase-level annotations on SST can be transferred to other sentiment analysis tasks as well as related tasks, such as emotion classification tasks. Moreover, we conduct ablation studies and design visualization methods to understand SentiBERT. We show that SentiBERT is better than baseline approaches in capturing negation and the contrastive relation and model the compositional sentiment semantics.

## introduction sentiment analysis is an important language processing task @xcite @xcite . one of the key challenges in sentiment analysis is to model compositional sentiment semantics. take the sentence "frenetic but not really funny." in figure 1 as an example. the two parts of the sentence are connected by "but", which reveals the change of sentiment. besides, the word "not" changes the sentiment of "really funny". these types of negation and contrast are often difficult to handle when the sentences are complex @xcite @xcite . in general, the sentiment of an expression is determined by the meaning of tokens and phrases and the way how they are syntactically combined. prior studies consider explicitly modeling compositional sentiment semantics over constituency structure with recursive neural networks @xcite figure 1 : illustration of the challenges of learning sentiment semantic compositionality. the blue nodes represent token nodes. the colors of phrase nodes in the binary constituency tree represent the sentiment of phrases. the red boxes show that the sentiment changes from the child node to the parent node due to negation and contrast. 2013). however, these models that generate representation of a parent node by aggregating the local information from child nodes, overlook the rich association in context. in this paper, we propose sentibert to incorporate recently developed contextualized representation models @xcite with the recursive constituency tree structure to better capture compositional sentiment semantics. specifically, we build a simple yet effective attention network for composing sentiment semantics on top of bert @xcite . during training, we follow bert to capture contextual information by masked language modeling. in addition, we instruct the model to learn composition of meaning by predicting sentiment labels of the phrase nodes. results on phrase-level sentiment classification on stanford sentiment treebank (sst) @xcite indicate that sentibert improves significantly over recursive networks and the baseline bert model. as phrase-level sentiment labels are expensive to obtain, we further explore if the compositional sentiment semantics learned from one task can be transferred to others. in particular, we find that sentibert trained on sst can be transferred well to other related tasks such as twitter sentiment analysis @xcite and emotion intensity classification @xcite and contextual emotion detection @xcite . furthermore, we conduct comprehensive quantitative and qualitative analyses to evaluate the effectiveness of sentibert under various situations and to demonstrate the semantic compositionality captured by the model. the source code is available at https://github.com/ wadeyin9712/sentibert. semantic compositionality semantic composition (pelletier, 1994) has been widely studied in nlp literature. for example, @xcite introduce operations such as addition or element-wise product to model compositional semantics. the idea of modeling semantic composition is applied to various areas such as sentiment analysis @xcite , semantic relatedness @xcite and capturing sememe knowledge @xcite . in this paper, we demonstrate that the syntactic structure can be combined with contextualized representation such that the semantic compositionality can be better captured. our approach resembles to a few recent attempts @xcite to integrate tree structures into self-attention. however, our design is specific for the semantic composition in sentiment analysis. ## model we introduce sentibert, a model that captures compositional sentiment semantics based on constituency structures of sentences. sentibert consists of three modules: 1) bert; 2) a semantic composition module based on an attention network; 3) phrase and sentence sentiment predictors. the three modules are illustrated in figure 2 and we provide an overview in below. bert we incorporate bert @xcite as the backbone to generate contextualized representation of input sentence. ## experiments we evaluate sentibert on the sst dataset. we then evaluate sentibert in a transfer learning setting and demonstrate that the compositional sentiment semantics learned on sst can be transferred to other related tasks. ## analysis we conduct experiments on sst-phrase using bert-base model as backbone to demonstrate the effectiveness and interpretability of the sentibert architecture in terms of semantic compositionality. we also explore potential of the model when lacking phrase-level sentiment information. in order to simplify the analysis of the change of sentiment polarity, we convert the 5-class labels to to 3-class: the classes 'very negative' and 'negative' are converted to be 'negative'; the classes 'very positive' and 'positive' are converted to be 'positive'; the class 'neutral' remains the same. the details of statistical distribution in this part is shown in appendix a.3. we consider the following baselines to evaluate the effectiveness of each component in sentibert. first we design bert w/ mean pooling as a base model, to demonstrate the necessity of incorporating syntactic guidance and implementing aggregation on it. then we compare sentibert with alternative aggregation approaches, tree-lstm, gcn and w/o attention to children. ## conclusion we proposed sentibert, an architecture designed for capturing better compositional sentiment semantics. sentibert considers the necessity of contextual information and explicit syntactic guidelines for modeling semantic composition. experiments show the effectiveness and transferability :kloh wkh jodvv volsshu grhv q w txlwh ilw 3xpsnlq lv ghilqlwho\ d xqltxh prghuq idlu\wdoh uhdoo\ ixqq\ qrw exw )uhqhwlf 1hjdwlyh 1hxwudo 3rvlwlyh figure 7: cases for interpretability of compositional sentiment semantics. the three color blocks between parents and children are the attention weights distributed to left child, the phrase itself and right child.