It’s absolutely divine! Can fine-grained sentiment analysis benefit from coreference resolution? (2020)

Abstract: While it has been claimed that anaphora or coreference resolution plays an important role in opinion mining, it is not clear to what extent coreference resolution actually boosts performance, if at all. In this paper, we investigate the potential added value of coreference resolution for the aspect-based sentiment analysis of restaurant reviews in two languages, English and Dutch. We focus on the task of aspect category classification and investigate whether including coreference information prior to classification to resolve implicit aspect mentions is beneficial. Because coreference resolution is not a solved task in NLP, we rely on both automatically-derived and gold-standard coreference relations, allowing us to investigate the true upper bound. By training a classifier on a combination of lexical and semantic features, we show that resolving the coreferential relations prior to classification is beneficial in a joint optimization setup. However, this is only the case when relying on gold-standard relations and the result is more outspoken for English than for Dutch. When validating the optimal models, however, we found that only the Dutch pipeline is able to achieve a satisfying performance on a held-out test set and does so regardless of whether coreference information was included.

## introduction in the last two decades, the field of sentiment analysis (sa) has yielded a lot of attention in both academia and commerce (see @xcite , @xcite or @xcite for overviews). the attention in sa research has shifted from the coarse-grained detection of the polarity of a given piece of text to the more fine-grained detection of not only polarity, but also the target of the expressed sentiment, as exemplified by the semeval shared tasks on aspect-based sentiment analysis @xcite @xcite ). in reviews, many references to different aspects of a given product, experience, etc. are made and in a large number of cases, these references are even implicit. regarding these implicit references, there are two options: either the referent is truly implicit meaning that the aspect can only be inferred from the implied meaning of the sentence, or the referent is an anaphor referring back to an antecedent that was or was not previously mentioned in the review. while it has been claimed that anaphora or coreference resolution plays an important role in opinion mining to resolve the relationship between the mentioned entities in a given text and across texts @xcite , it is not clear to what extent coreference resolution actually boosts sa performance, if at all. in this paper, we investigate the potential added value of coreference resolution in the aspect-based sa of restaurant reviews for two languages: english and dutch. by also manually annotating coreferential links in our data, we measure the incidence of referential links in our review corpus and investigate the upper bound of coreference resolution on sa performance. we reveal that although a certain number of coreferential instances are available in both languages, this does not alter the performance. on the contrary, when relying on automatic coreference resolution systems in both languages, we find that this hampers overall performance. the remainder of this paper is organised as follows. in the next section, we discuss related work. in section 3, we explain the datasets that were used for this research and how these have been annotated for both aspect-based sentiment analysis and coreference resolution. next, in section 4, we discuss the supervised machine learning classifier that was built and focus on how adding coreference resolution to the pipeline could alter accuracy. we present the results in section 5, after which we conclude our work in section 6 and offer prospects for future research. ## related work the large volume of existing work on sentiment analysis from its early days until now can roughly be divided into lexicon-based and machine learning approaches. lexicon-based methods determine the semantic orientation of a text based on scanning the words occurring in that text while relying on lexicons. until recently, machine learning approaches were feature-based and applied supervised machine learning algorithms such as support vector machines. with the advent of deep learning end-to-end approaches have also proven to perform well @xcite . both types of approaches have been applied at various levels of a text: the document @xcite , sentence @xcite , phrase @xcite and word @xcite level. for each of these levels, coarse-grained as well as fine-grained sentiment analysis can be performed. the latter means that the focus is not only on determining the polarity of a given utterance, but also on the identification of, for example, the source and target of the expressed sentiment @xcite . in the last decade, a substantial amount of research has been dedicated to target detection for aspectbased sentiment analysis @xcite . this task focuses on the detection of all sentiment expressions within a given document and the concepts and aspects (or features) to which they refer. following the semeval task description, aspect-based sentiment analysis can be decomposed into three subtasks: aspect term extraction, aspect term aggregation or classification and aspect term polarity estimation. the focus of the research presented here is on the second one. the idea is to predict several predefined and domain-specific categories, i.e. a multiclass classification task. the two systems achieving the best results on this individual subtask in semeval 2015 task 12 both used classification to this purpose, respectively individual binary classifiers trained on each possible category which are afterwards entered in a sigmoidal feedforward network @xcite and a single maximum entropy classifier @xcite . when it comes to feature engineering, especially lexical features in the form of bag-of-words such as word unigrams and bigrams @xcite or word and lemma unigrams @xcite and lexical-semantic features in the form of clusters learned from a large corpus of reference data @xcite were used. since then, these benchmark semeval datasets have been used many times to train and test neural models yielding state-of-the-art results on both this second subtask and end-to-end aspect-based sentiment analysis @xcite however, many methodologies start from he assumption that the target of a given polarity is explicitly lexicalized, which is certainly not always the case as people often use shorter or alternative linguistic structures, such as anaphors to refer to previously mentioned elements. many survey studies on sentiment analysis have claimed that the recognition of coreference is crucial for successful (aspect-based) sentiment analysis @xcite . @xcite were the first to use coreference resolution features to determine which mentions of opinion holders refer to the same entity. early research in incorporating basic coreference resolution in sentiment analysis was conducted by @xcite , who investigated how to perform sentiment analysis on parts of the document around topic terms. they demonstrated that using a proximity-based sentiment analysis algorithm can be improved by about 10%, depending on the topic, when using coreference to augment the focus area of the algorithm. the work by @xcite , though its main focus is on finding which sentiment expressions are semantically related, provided some valuable insights in the necessity of coreference as they found that 14% of the targets expressions that had been manually labeled in their corpus were expressed in the form of pronouns. @xcite introduced the problem of entity and aspect coreference resolution and aimed to determine which mentions of entities and/or aspects a certain pronoun refers to, taking a supervised machine learning approach. their system learns a function to predict whether a pair of nouns is coreferent, building coreference chains based on feature vectors that model a variety of contextual information about the nouns. they also added two opinion-related features, which implies that they used sentiment analysis for the purpose of better coreference resolution rather than the other way around. a similar coreference resolution methodology was used by @xcite to link target aspects to target objects. however, to our knowledge not much qualitative research has been performed investigating to what extent the availability of coreference information can actually help aspect-based sentiment analysis. this is the exact aim of this paper, we zoom in on the task of aspect category classification and investigate whether including coreference information prior to classification is useful. ## datasets and annotation for our experiments, we rely on datasets comprising restaurant reviews in two languages, namely english and dutch. both datasets were released in the framework of semeval: the english data (350 reviews) was released for the 2015 competition @xcite and the dutch data (400 reviews) for a rerun of this competition in 2016 @xcite . ## experimental setup the focus of our experiments is on the task of aspect category classification. this is a fine-grained classification task requiring a system to grasp subtle differences between various main-attribute categories (e.g. food-general versus food-prices versus food-quality versus food-style&options). moreover, as previously explained, reviewers refer to the various aspects of a restaurant in both an explicit and implicit manner. especially those implicit targets are challenging. this is why we will investigate whether including coreference information prior to classification is useful. we envisage two experimental settings: a setting where coreferential anaphor-antecedent pairs were not derived beforehand and one where they were. in the latter setting, both gold-standard and automatically-derived coreference relations were used in order to investigate the true upper bound of incorporating this type of information. we relied on gold-standard explicit and implicit targets for all experiments. as experimental data we employed the same train and test splits of the semeval shared tasks on absa @xcite , see ## results in setting a, coreference resolution is not performed prior to classificiation, so only the explicit aspect terms are targeted. in setting b, coreference resolution is included as an additional processing step prior to classification. having coreference information available should allow us to derive additional semantic information for those 'null' targets constituting an anaphor-antecedent pair. we differentiate between a setup where we incorporate this information assuming we have a perfect coreference resolution system (gold), i.e. using gold-standard coreferential links, and a setup where coreference relations have been resolved automatically (auto). coreference resolution as an additional processing step prior to classification. the results, expressed in accuracy, are presented in from the above-mentioned results, it can also be concluded that the added value of including coreference information is not outspoken. when relying on coreference resolution systems, the performance mostly deteriorated mainly because wrong antecedents have been linked to anaphors, causing erroneous lexical-semantic features. however, our results also revealed that incorporating gold-standard anaphorantecedent relations leads to the best overall scores in both languages after jointly optimizing libsvm's hyperparameters and performing individual feature selection. if we compare these scores to the best individual scores achieved in setting a, we observe that the difference is more outspoken for english, which confirms our hypothesis. in the next section we will analyse whether incorporating coreference information also meant that the lexical-semantic features were considered more important. ## conclusion the objective of this research was to investigate to what extent coreference resolution can boost sentiment analysis performance. our focus was on aspect-based sentiment analysis of english and dutch restaurant reviews and more specifically the task of classifying aspect terms into predefined aspect categories. we worked with two datasets that were released and annotated in the framework of semeval. working with these datasets, we found that people often refer to aspect terms implicitly in both languages (24.9% in english versus 31.6% in dutch). this is why we investigated whether including coreference information prior to classification would be useful for pinpointing those implicit aspect terms constituting a referential relation with an antecedent. we manually annotated coreferential relations in both datasets and observed a different tendency in both languages. in english, an anaphor is frequently preceded by an antecedent within one review, whereas for dutch the anaphors more frequently refer to extra-linguistic entities which are not explicitly mentioned in the review. when exploiting coreferential information in an aspect-based sentiment analysis pipeline, we therefore hypothesized that this would be more successful for english than for dutch. to investigate this, experiments were conducted in two different settings: a first setting where coreferential anaphor-antecedent pairs were not derived beforehand and a second setting where they were. in the latter setting, both gold-standard and automatically-derived coreference relations were used in order to investigate the true upper bound of incorporating this type of information. our classifier relied on a combination of lexical (bag-of-words) and lexical-semantic information in the form of wordnet @xcite and dbpedia @xcite features. besides exploring the added value of coreference information, we also used a wrapper-based genetic algorithm optimization approach to optimize our classifiers and get more insights into which features are most important. the results reveal that resolving coreferential relations prior to classification is beneficial in both settings in a setup where both the hyperparameters and individual features are jointly optimized. however, this is only the case when relying on gold-standard coreferential information and the result is more outspoken for english (from 67.23% to 68.20%) than for dutch (from 63.16% to 63.59%). regarding the selected features in the optimal models we could conclude that lexical bag-of-words are necessary to include and that including semantic information in the form of lexical-semantic features is also important. comparing the optimal setting with and without performing coreference resolution prior to classification, we observe that especially for english more dbpedia features are being turned on. in a final set of experiments we envisaged to validate these findings by testing the optimal models on a held-out test set in both languages. for english this led to poor results whereas for dutch we were able to achieve a satisfying performance of 66.42%. in both languages, however, it was no added value to have gold-standard coreference information available before classification. though the results now seem to indicate that coreference information is not necessary to include in a fine-grained sentiment analysis pipeline, it will be interesting to corroborate these findings on larger datasets and on data coming from different domains. now the focus was on resolving anaphor-antecedent pairs within one review, but in reality coreference also appears across texts which offers other interesting