A Comparison of Sense-level Sentiment Scores (2019)

Abstract: In this paper, we compare a variety of sense-tagged sentiment resources, including SentiWordNet, ML-Senticon, plWordNet emo and the NTU Multilingual Corpus. The goal is to investigate the quality of the resources and see how well the sentiment polarity annotation maps across languages.

## introduction there are several semantic resources with senses annotated by sentiment polarity, e.g. sentiword-net 3.0 @xcite and even emotions, e.g. wordnet-affect @xcite . however, most of them were built on the basis of automated expansion of a small subset of senses described manually. in addition the majority of them were built for a single language, namely english, with ml-senticon @xcite a notable exception. this paper present the results of comparing two very different sense-level sentiment resources: a very large semantic lexicon annotated manually for polish, i.e. plwordnet @xcite expanded with manual emotive annotations @xcite ; the annotation of two english short stories (the adventure of the speckled band and the adventure of the dancing men @xcite ) and their chinese and japanese translations @xcite . as the stories have been annotated on the basis of senses, not words -i.e. all words were assigned princeton wordnet synsets -this opens an unique possibility of cross-lingual comparison of manual sentiment annotation at the level of word senses. these are then compared with sentiwordnet and ml-senticon and finally they are all compared to a small gold standard sample micro-wnop corpus @xcite . our technical goal is to analyse the feasibility and technical means of correlation between independently created resources as the first step towards cross-lingual applications. taking a more fundamental perspective, we want to investigate the level and distribution of correlation between sentiment polarity expression on the sense level between languages. in addition this is also an exercise in utilisation of the interlingual manual mapping between plwordnet and princeton wordnet that has been built independently. ## resources in this section we describe the resources we used. ## comparisons we are going to compare four languages and two types of resources: a corpus and a lexicon from the perspective of sentiment polarity annotation. in order to make the comparison feasible, we focus on word senses -that can be represented by concepts -and their mappings across languages, as links between the different resources. there are both manually annotated and automatically built (to a very large extent) resources among the compared ones. finally two types of the sentiment polarity annotations that are represented by the compared resources use similar but slightly different models: the semi-continuous scale, e.g. ntu-mc and the discrete scale, e.g. the five-grades scale of plword-net emo. concept freq score english score chinese score japanese score i40833 24 +50 marriage 39 婚事 34 結婚 58 wedding 34 i11080 5 +40 rich 33 有钱 34 裕福 66 i72643 4 +33 smile 32 微笑 34 笑み i23529 40 -68 die -80 去世 -60 亡くなる -63 死亡 -64 死ぬ -62 i36562 5 -83 murder -95 谋杀 -95 殺し -64 殺害 -63 ## the combined sentiment lexicon one clear results of this comparison is that comparing the lexicons with each other improves them. places where there was a difference in polarity or in zero vs non-zero sentiment were almost all errors. once discovered there are easy to fix, and we have shared the results with the resource creators. because the scores are different (a continuous score for ntu-mc and a 5 point scale for plwordnet emo) we can combine in two ways: binning ntu-mc or setting values for weak and strong for plwordnet emo (we used 0.4 and 0.8). they can then be combined over all synsets, to give a single resource that should be somewhat more accurate then either alone. to combine the lexicons we decided to use binning strategy on ntu-mc and micro-wnop followed by a simple selection procedure. to represent matched concepts within the same category set we used thresholding function with thresholds being a result of score distribution analysis. in case of ntu-mc the following bins were proposed: |s| ≤ 0.18 for neutral category, 0.18 < |s| ≤ 0.54 for weak polarity and |s| > 0.54 for strong polarity. first we selected a subset of paired synsets annotated both in ntu-mc and plwordnet emo which were compatible in terms of their polarity categories. to reduce the discrepancy between the annotations we also decided to remove all of paired synsets having different polarity categories. in the last step we introduce a group of unmatched synsets with their annotations to extend the coverage of joint lexicon. the final lexicon was evaluated again on micro-wnop ( ## conclusion and future work in this paper we presented a comparison of wordnet-based sense-level sentiment lexicons. we showed that the two manually annotated resources were more accurate than the semi-automatically created resources. we also showed that linking across languages preserved most of the valence (ρ = 0.65 -0.77 for equivalent synsets). this means that the resources can be used for other languages, linked either directly or through an interlingual index. finally we showed how they could be improved further by cross-checking and resolving inconsistencies, or by combining them. in future work, we will: (i) correct the errors in the two resources and recalculate their correlation (as it is sensitive to outliers). (ii) create further sense-annotated sentiment tagged text • another sherlock holmes story (the red-headed league)