If you’ve got it, flaunt it: Making the most of fine-grained sentiment annotations (2021)

Abstract: Fine-grained sentiment analysis attempts to extract sentiment holders, targets and polar expressions and resolve the relationship between them, but progress has been hampered by the difficulty of annotation. Targeted sentiment analysis, on the other hand, is a more narrow task, focusing on extracting sentiment targets and classifying their polarity. In this paper, we explore whether incorporating holder and expression information can improve target extraction and classification and perform experiments on eight English datasets. We conclude that jointly predicting target and polarity BIO labels improves target extraction, and that augmenting the input text with gold expressions generally improves targeted polarity classification. This highlights the potential importance of annotating expressions for fine-grained sentiment datasets. At the same time, our results show that performance of current models for predicting polar expressions is poor, hampering the benefit of this information in practice.

## introduction sentiment analysis comes in many flavors, arguably the most complete of which is what is often called fine-grained sentiment analysis @xcite . this approach models the sentiment task as minimally extracting all opinion holders, targets, and expressions in a text and resolving the relationships between them. this complex task is further complicated by interactions between these elements, strong domain effects, and the subjective nature of sentiment. take the annotated sentence in figure 1 as an example. knowing that the target "umuc" is modified by the expression "5 stars" and not "don't believe" is important to correctly classifying the polarity. additionally, the fact that this is a belief held by "some others" as apposed to the author of the sentence can help us determine the overall polarity expressed in the sentence. compared to document-or sentence-level sentiment analysis, where distant labelling schemes can be used to obtain annotated data, fine-grained annotation of sentiment does not occur naturally, which means that current machine learning models are often hampered by the small size of datasets. furthermore, fine-grained annotation is demanding, leads to relatively small datasets, and has low inter-annotator agreement @xcite . this begs the question: is it worth it to annotate full fine-grained sentiment? targeted sentiment @xcite is a reduction of the fine-grained sentiment task which concentrates on extracting sentiment targets and classifying their polarity, effectively ignoring sentiment holders and expressions. the benefit of this setup is that it is faster to annotate and simpler to model. but would targeted sentiment models benefit from knowing the sentiment holders and expressions? in this work, we attempt to determine whether holder and expression information is useful for extracting and then classifying sentiment targets. specifically, we ask the following research questions: rq1: given the time and difficulty required to annotate opinion holders, expressions, and polarity, is this information useful to extract sentiment targets? (a) does augmenting the input text with holders and expressions improve target extraction? (b) do target extraction models benefit from predicting holders and expressions? have seen some others giving umuc { 5 stars } -{ don't believe } them. we conduct a series of experiments on eight english sentiment datasets (three with full finegrained sentiment and five targeted) with state-ofthe-art models based on fine-tuned bert models. we show that (1) it is possible to improve target extraction by also trying to predict the polarity, and that (2) classification models benefit from having access to information about sentiment expressions. we also (3) release the code foot_0 to reproduce the experiments, as well as the scripts to download, preprocess, and collect the datasets into a compatible json format, with the hope that this allows future research on the same data. ## related work fine-grained approaches to sentiment analysis attempt to discover opinions from text, where each opinion is a tuple of (opinion holder, opinion target, opinion expression, polarity, intensity). annotation of datasets for this granularity requires creating in-depth annotation guidelines, training annotators, and generally leads to lower interannotator scores than other sentiment tasks, e.g., document-or sentence-level classification, as deciding on the spans for multiple elements and their relationships is undeniably harder than choosing a single label for a full text. targeted sentiment, on the other hand, generally concentrates only on target extraction and polarity classification. this has the benefit of allowing non-experts and crowdsourcing to perform annotation, making it easier to collect larger datasets for machine learning. this simplified annotation can be crowd-sourced, leading to larger datasets for machine learning. ## data one of the difficulties of working with finegrained sentiment analysis is that there are only a few datasets (even in english) and they come in incompatible, competing data formats, e.g., brat or various flavors of xml. with the goal of creating a simple unified format to work on fine-grained sentiment tasks, we take the eight datasets mentioned in section 2 -mpqa @xcite , darmstadt services and universities @xcite , tdparse @xcite , semeval restaurant and laptop @xcite , open domain targeted sentiment @xcite , and the challenge dataset from @xcite -and convert them to a standard json format. the datasets are sentence and word tokenized using nltk @xcite , except for mpqa, ds. service and ds. uni, which already contain sentence and token spans. all polarity annotations are mapped to positive, negative, neutral, and conflict foot_1 . as such, each sentence contains a sentence id, the tokenized text, and a possibly empty set of opinions which contain a holder, target, expression, polarity, and intensity. we allow for empty holders and expressions in order generalize to the targeted corpora. finally, we use 10 percent of the training data as development and another 10 percent for test for the corpora that do not contain a suggested train/dev/test split. for training and testing models, however, we convert the datasets to conll format. ## experimental setup we split the task of targeted sentiment analysis into the extraction of sentiment targets and subsequent polarity classification of extracted targets, given their context. figure 2 shows the two tasks and the eight models used in the experiments. as a base model, we take the tar2020 ) and instead use larger amounts of unlabeled data -1,710,553 and 2,000,000 sentences for semeval laptop and restaurant respectively -taken from amazon laptop reviews @xcite and the yelp dataset challenge. 4 we further deviate from @xcite by not pretraining the models on the squad question answering dataset and in-domain sentiment questions which they create, as this data is not publicly available. finally, a linear prediction is added after the bert model and the full model is updated on the sentiment task. for target extraction, we use the contextualized bert embeddings as input to a softmax layer 4 https://www.yelp.com/dataset/ challenge and predict the sequence of tags. we compare three prediction strategies: for polarity classification, we take as a baseline the classification architecture from @xcite , which makes use of the two-sentence training procedure for bert, by prepending the target before the sentence separation token, and then adding the full sentence after. we compare five strategies for producing the input to the softmax layer for predicting the sentiment of the target: bert [cls] i like the pizza here [sep] the pizza [sep] mean max max mean min first [cls] 1) 2) 3) 4) 5) [cls] i like the pizza here [sep] bert extraction classification [cls] i like the pizza here [sep] [cls] i like the pizza here [sep] the pizza [sep] targ. o o o b i o o 1) pred. o b-hol b-e b-t i-t o o 2) +pol. o b-hol b-e+ b-e+ i-t+ o o 3) figure 2: our bert-based target extraction and classification models, with the three strategies for extraction ((1) predict only targets, (2) predict holders, targets and expressions, and (3) predict the polarity of the targets and expressions as well) and five strategies for sentiment classification (passing to the softmax layer the contextualized embedding from (1) the [cls] embedding, (2) the first token in the target (3) averaging all embeddings in the target phrase, (4) taking the max of the target embeddings, (5) concatenating the max, mean, and min). the targ. and [cls] models correspond to the models used in @xcite and serve as baselines. the extraction and classification models are fine-tuned for 50 epochs using adam with an initial learning rate of 3e-5, with a linear warmup of 0.1 and all other hyperparameters are left at default bert settings (further details in appendix b). the best model on the development set is used for testing. combined with the four input manipulations ( ## results in this section we describe the main results from the extraction and two classification experiments described in section 4. ## conclusion in this work we have explored the benefit of augmenting targeted sentiment models with holder and sentiment expressions. the experiments have shown that although augmenting text with holder and expression tags (rq1 a) or simultaneously predicting them (rq1 b) have no benefit for target extraction, predicting collapsed bio + polarity tags consistently improves target extraction (rq1 c). furthermore, augmenting the input text with gold expressions generally improves targeted polarity classification (rq2 a), although it is not clear which target representation strategy is best (rq2 b). furthermore, we have found benefits of including lexicon-based expressions for the more complex targeted datasets (rq2 c). the rather poor performance of the learned expression models and the difference between augmenting with gold or predicted expressions reveals the need to improve expression prediction approaches, both by creating larger corpora annotated with sentiment expressions, as well as performing further research on the modeling aspect. any future work interested in modelling more complex sentiment phenomena should therefore be aware that we may first require more highquality annotated data if we wish to do so with current state-of-the-art machine learning approaches. furthermore, we introduce a common format for eight standard english datasets in fine-grained sentiment analysis and release the scripts to download and preprocess them easily. we plan to include further datasets in our script in the future, as well as extending our work to other languages with available fine-grained corpora.