Entity-Level Sentiment Analysis ( ELSA ): An Exploratory Task Survey (2022)

Abstract: This paper explores the task of identifying the overall sentiment expressed towards volitional entities (persons and organizations) in a document - what we refer to as Entity-Level Sentiment Analysis (ELSA). While identifying sentiment conveyed towards an entity is well researched for shorter texts like tweets, we find little to no research on this specific task for longer texts with multiple mentions and opinions towards the same entity. This lack of research would be understandable if ELSA can be derived from existing tasks and models. To assess this, we annotate a set of professional reviews for their overall sentiment towards each volitional entity in the text. We sample from data already annotated for document-level, sentence-level, and target-level sentiment in a multi-domain review corpus, and our results indicate that there is no single proxy task that provides this overall sentiment we seek for the entities at a satisfactory level of performance. We present a suite of experiments aiming to assess the contribution towards ELSA provided by document-, sentence-, and target-level sentiment analysis, and provide a discussion of their shortcomings. We show that sentiment in our dataset is expressed not only with an entity mention as target, but also towards targets with a sentiment-relevant relation to a volitional entity. In our data, these relations extend beyond anaphoric coreference resolution, and our findings call for further research of the topic. Finally, we also present a survey of previous relevant work.

## introduction over the course of the last two decades, the field of nlp has generated a vast body of research on sentiment analysis (sa), i.e. the task of identifying opinions expressed in text. prior work has focused on a range of different levels of analysis; from document-or sentence-level polarity classification to more fine-grained prediction of various components of opinions, like source/holder expressions, polar expressions, target expressions and aspectbased sentiment classification. however, we observe that a more aggregated level of analysis, what we here dub "entity-level sentiment analysis" (or elsa for short), remains under-explored. for our purposes, we will define elsa as the task of determining the overall (i.e. document-level) polarity (positive/negative) expressed towards an entity in a text. moreover, for the current paper we will restrict the discussion to volitional entities, like persons (per) or organizations @xcite . a given text might make reference to multiple distinct entities, each of which might be mentioned multiple times, both directly and indirectly, and also have multiple opinions directed towards them. hence, solving the task of elsa in its full complexity may potentially involve several different sub-tasks, including (but not necessarily limited to) named entity recognition (ner), resolution of entity mentions, coreference-and anaphora-resolution, identification of sentiment targets and/or aspects and their polarities, and finally what we here refer to as "target-entity resolution", i.e. the task of identifying the particular entity with which a given target expression is associated. the main goal of this paper is to shed more light on this task of entity-level sentiment analysis. to better understand the complexity of the task, we quantify how far we can potentially get toward the goal of elsa by simply building on existing tools for sa at lower levels. we do this through an exploratory analysis of an existing sa dataset that comprises annotations for several levels of granularity, for which we experiment with different strategies for aggregating these gold annotations to infer entity-level sentiment. importantly, we also discuss which pieces appear to currently be missing in order to fully solve elsa. we start, however, by surveying relevant prior work, and also discuss the often diverging terminology used in the field. the paper is structured as follows. in section 2 we first present background literature on the task of analyzing sentiment towards individual entity representations, before we discuss the limited work we found on resolving sentiment-relevant elements to the document-level. we found no previous work describing the elsa task as we defined it, and very little in general on resolving sentiment from the sub-sentence level to the document-level. section 3 presents the datasets we sample from; norwegian data annotated for document-level, sentence-level and target-level sentiment classification. we also present our new, exploratory dataset, annotated directly for document-level sentiment classification for each volitional entity in the text. in section 4 we present our results from attempting to derive elsa from document-level and sentence-level sentiment analysis -what we dub target-independent approaches. we find that merely locating an entity mention inside a positive document is a very weak indicator of a positive sentiment towards that entity. section 5 presents our findings from deriving elsa classification from targeted sentiment analysis (tsa). besides discrepancies from annotator disagreement, we find that in order to fully solve the elsa task, future work should aim to add another level of analysis, corresponding to the relations between sentiment targets and their affiliated entities. in section 6 we report on a baseline model for elsa through tsa as a proxy task. ## literature review this section first presents prior work related to elsa. we then survey the terminology used in the literature, comparing this to our suggested definition of entity-level sa. the work of @xcite is highly related to our task in that they for each text identify the sentiment polarity towards each volitional entity mentioned in the text. however, while @xcite work on twitter data, hence by their nature very short texts, our goal in this work is to extend the task to longer texts where the sentiment towards one volitional entity may be expressed through multiple mentions and opinion expressions. on a similar note, @xcite also analyze sentiment towards each target in tweets. the goal of their work is, for a corpus of tweets and a query term, to return tweets classified as positive or negative towards the entity in the query term. the work goes beyond situations where the named entity is the sentiment target, into what is described as "extended targets". this work is a step in the direction of locating an extended set of segments of the text that are relevant for the sentiment towards each entity. ## the sentiment dataset in order to investigate how the elsa task relates to pre-existing sentiment analysis tasks, we wanted to build on an existing full-text document collection that is annotated for sentiment information at several levels of analysis. the suite of annotated datasets that are based on the norwegian review corpus -norec @xcite -fits this bill. while section 3.2 describes how we build on norec to create an exploratory dataset for elsa, we first describe the different levels of existing annotations in norec below. ## analysis of target-independent approaches to elsa if all volitional entities mentioned in a positive text are the target of positive sentiment, the task would be limited to an overall sentiment classification for the text. this naïve approach serves as a baseline for further studies. in the following, we compare our manually labeled entity-level polarities with those of the documents they appear in, as well as those of each sentence with a corresponding entity mention. ## a target-dependent approach to elsa an intuitively more promising approach is to derive elsa sentiment labels from targeted sentiment analysis. our tsa dataset presented in section 3.1 is annotated for sentiment towards each target with a scale from -3 (strong negative) to 3 (strong positive). we aggregate these labels from the targetlevel to the entity-level for each entity by including only the sentiment targets that overlap with an entity mention, and summing the sentiment values for these targets. this approach leaves 7 entities with an unresolved classification due to equally strong positive and negative sentiment towards its entity mentions. these entities are placed in the "mix" category. as shown in ## modeling the above data analysis was performed using gold standard data for the various sentiment-levels (document, sentence and target). in order to gauge the performance attainable in a more realistic setting, we present results using automatically derived tsa information in the following. since we have no directly annotated elsa training data, we create a baseline model using the proxy task of tsa. the exploratory elsa dataset was taken from the training split of norec fine , and we therefore join the remainder of the training split with the development split to create our baseline training data, and we perform the final evaluation on the 50 documents in our elsa dataset. the model setup is adapted from the huggingface example configuration for ner. 4 we use default hyperparameters and perform no hyperparameter tuning. we therefore chose to not set aside data for a dev set, and not touch the original norec fine test split, in order to allow for further research on this data split. we preprocess our training data the same way as we did with the elsa data in section 5, finding per and org labels in the data through ner, and deriving sentiment towards these through the pre-existing tsa target annotations. volitional entities that are sentiment targets, receive the sentiment label "positive" or "negative", while the volitional entities that are not sentiment targets are labeled "neutral". with this setup, volitional entities and their sentiment polarity is predicted for the 50 documents in the exploratory dataset. the predicted entity mentions were resolved documentwise through substring matching, before summing the predicted polarities. for evaluation, these predictions are compared with our manually resolved and annotated entities. 5 our evaluation ignores whether the model assigned the correct ner category per or org. our evaluation shows that this baseline model has a f 1 -score of 70.1%, as shown in ## conclusion in this paper we have explored the task of entitylevel sentiment analysis (elsa) -the task of determining the aggregated or overall (i.e. documentlevel) polarity expressed towards an entity in a text. in particular for longer text, which might comprise several distinct entities, entity mentions and opinions, this is a potentially complex task. the paper has surveyed existing literature and terminology in adjacent and related previous work, in addition to presenting an exploratory data analysis to shed more light on what is required to solve the task. relating to the latter point, in order to assess the relevance to elsa of existing approaches to sentiment analysis at different granularities -i.e. document-, sentence-, and target-level sa -we perform a task analysis on the basis of a norwegian multi-domain review dataset containing all these layers of sentiment annotation. when utilizing the fine-grained sentiment annotations of the norec fine dataset, we found that the overlap between a volitional entity and tsa annotations, gave us the correct sentiment category for 82% of our 279 manually evaluated elsa entities. we further found that, using automated tsa, we obtained an f 1 -score of 70.1%. tsa therefore appears to be highly relevant, though not sufficient, for the elsa task. for the elsa classifications that could not be derived from ner and tsa annotations, we found surprisingly few cases that would be correctly resolved through coreference resolution. this is in line with the findings of de clercq and hoste (2020). our dataset is likely too small, however, to draw any definitive conclusions about the importance of coreference resolution for elsa. on the other hand, we did observe a need for more generally resolving relations that tie target expressions to their corresponding entities. in our exploratory data analysis, we have identified at least four categories of information that could potentially benefit the classification of a document's overall sentiment towards entities: (i) named entity recognition, (ii) targeted sentiment analysis, (iii) coreference resolution, and (iv) what we dub target-entity resolution. the latter being a concept derived from working with the examples in our dataset, and refers to the task of identifying which entity a given target expression relates to. this appears to be a missing link in a pipeline for elsa based on tsa, and to explore methods for filling this gap is a suggestion for further work.