Towards Emotive Annotation in pl W ord N et 4.0 (2018)

Abstract: The paper presents an approach to building a very large emotive lexicon for Polish based on plWordNet. An expanded annotation model is discussed, in which lexical units (word senses) are annotated with basic emotions, fundamental human values and sentiment polarisation. The annotation process is performed manually in the 2+1 scheme by pairs of linguists and psychologies. Guidelines referring to the usage in corpora, substitution tests as well linguistic properties of lexical units (e.g. derivational associations) are discussed. Application of the model in a substantial extension of the emotive annotation of plWordNet is presented. The achieved high inter-annotator agreement shows that with relatively small workload a promising emotive resource can be created.

## introduction since plwordnet @xcite achieved good coverage with the version 2.0 the number of its users and applications has been quickly growing. many users declared sentiment analysis, as their intended use of plwordnet, contrary to the lack of sentiment information in it. in response, a pilot project on emotive annotation of a selected subset of senses in plwordnet was conducted @xcite which next resulted with plwordnet 2.3 emo including emotive annotation for more than 31,000 lexical units (word senses). this prototype emotive annotation showed its usefulness in lexicon-based sentiment analysis, but its coverage was limited and selective (i.e. around 10% of noun senses and 25% of adjective senses of plwordnet 3.0 emo). our goal is to develop an improved and expanded model of emotive annotation for a word-net, and also an expanded version of the manual annotation procedures. in addition we will also present application of the model in a substantial extension of the emotive annotation of plwordnet. ## emotions in wordnets several sentiment lexicons are available for english, but hardly any for most other languages. chen and @xcite found 12 publicly available sentiment lexicons for 5 languages; there were none for polish. some sentiment lexicons were built upon princeton wordnet (pwn), a natural starting point because of its comprehensive coverage and its numerous applications. the lexicons not based on pwn consider lemmas rather than lexical meanings or concepts. wordnet-affect is a selection of synsets very likely to represent "affective concepts" @xcite . a small core of 1903 lemmas was selected and described manually with "affective labels". next, a set of rules based on wordnet relation semantics drove the transfer of the sentiment description onto the synsets connected to the core by wordnet relations. this produced 2874 synsets and 4787 lemmas. sentiwordnet @xcite annotates a synset with three values from the interval 0, 1 . they describe "how objective, positive, and negative the terms contained in the synset are". about 10% of the adjectives were manually annotated, each by 3-5 annotators @xcite . in sentiwordnet 3.0, the automated annotation process starts with all the synsets which include 7 "paradigmatically positive" and 7 "paradigmatically negative" lemmas. 1 in the end, sentiwordnet 3.0 added automatic sentiment annotation to all of pwn 3.0. sentisense @xcite is also a concept-based affective lexicon, with emotion categories assigned to pwn synsets. the initial list of 20 categories, a sum of several sets including wordnet-affect, was reduced to 14 after some work with annotators. the authors write: "the manual labelling techniques generate resources with very low coverage but very high precision", but note that such precision can be only achieved for specific domains. the construction of sentisense began with a manual annotation of only 1200 synsets with 14 emotions. annotation was transferred onto other synsets using wordnet relations. the authors' visualisation and editing tools, designed to allow relatively easy expansion and adaptation, did not add much to the resource, so every user must enlarge it further to make it really applicable. to sum up, a wordnet may be a good starting point for the construction of a sentiment lexicon: annotation can be done at the level of lexical meanings (concepts) or lemmas. pwn appears to be a good choice due to its sense-based model and large coverage. all large wordnetbased sentiment lexicons have been built by giving very limited manual annotation to algorithms for automated expansion onto other synsets. this, however, seems to have to result in lower precision, as noted, e.g., by @xcite : "currently available lexical resources for opinion polarity and affect recognition such as sentiwordnet @xcite or wordnet-affect are known to be rather noisy and limited." no large wordnets are available for most languages other than english. many sentiment lexicons were created by translating sentimentannotated pwn, e.g., bengali wordnet-affect @xcite , japanese wordnet-affect @xcite and chinese emotion lexicon @xcite . it is not clear how well annotations of that kind can be transferred across the language barrier. moreover, as we discuss it in section 3, plwordnet's model differs slightly from that of pwn. crowdsourcing has also been used to develop sentiment lexicons (mohammad and @xcite . it can outdo automated annotation (or automatic expansion of a manually annotated part), but the consistency of the result is low compared to manual description by trained annotators. unlike most of the existing methods, our aim is a manual annotation of a substantial part of plwordnet by a team of linguists and psychologists. the manually annotated part -several times larger than other known manually created sentiment lexicons -can be an important resource on its own. it can also be a solid basis for the development of automated sentiment annotation methods for more lexical material in a wordnet. we have adopted a rich annotation model in which sentiment polarity description is combined with emotion categories. ## annotation model for the sake of compatibility with plword-net 2.3 emo, the main assumptions and annotation scheme have been preserved without significant changes, see sec. 3.1,3.2. however, we plan to encompass by emotive annotation all pos in plword-net (i.e. nouns, adjectives, verbs and adverbs) and expand it very substantially by 100,000 annotated lexical units. it goes beyond typical sentiment polarity encoding and includes: sentiment polarity, basic emotions and fundamental values. on the basis of the analysis of the results of @xcite we modified the annotation guidelines for nouns and adjectives to improve annotation quality, see sec. 4. ## annotation procedure the annotation is performed foot_2 by: linguists and psychologists, where each lus is annotated by a mixed pair: one psychologist and one linguist. the annotators must follow guidelines that consist of a core common to all poss and detailed guidelines dedicated to each pos. the work of annotators is coordinated and verified by a supervisor, who can access all annotations and solve disagreements foot_3 by adding the final annotation. the common core is similar to the procedure in @xcite : step 3 recognition of the lu polarity direction: negative or positive, but also ambiguous, if the collected use examples show both behaviours; step 5 illustration of the assigned annotation by sentences representing use examples: at least one sentence in the case of positive and negative lus, and at least two example sentences for ambiguous lus. each step is associated with several linguistic tests, including substitution tests and requires consulting corpus data. the detailed specification of the subsequent steps is dependant on a particular pos. in the case of nouns see @xcite , the specification for adjectives proposed by us is presented in sec. 4.2. annotators can returned from the later steps to the previous ones. we could observe that, e.g., assignment of fundamental human values or basic emotions can be helpful in verifying the polarity of the given lu. for the annotation process, we use wordnet-loom -a wordnet editing system @xcite -which has been extended by additional windows and database tables (to eliminate errors in the annotation representation), as well as a mechanism that separates work of individual annotators. they do not see annotation decisions of other annotators and they do not know who is the second annotator of the given lu. moreover, annotators are rotated in the pairs in order to minimise a potential bias. this strict separation of annotators is a significant difference in relation to @xcite , where the second annotator was told not to take a look into the decision of the first annotator before having made his own one, but he could see it and could change his own one later. the second could report a possible error of the first one in the pilot project, but we decided to resign from this possibility and to separate them strictly. the inter-annotator agreement is on a high level, but inevitably lower than reported in @xcite , see sec.5. however, we sometimes observed a tendency to too prompt classification of a lu as a neutral one. if such a decision is taken without a detailed analysis, then the annotation process is actually discontinued after the first step and any change of mind of the given annotator later along the process is impossible. to amend this potential problem we paid more attention to the detailed guidelines for step 1, as well as to the training of annotators and verification of their work. ## intermediate results during the pilot project more than 31,000 lus (19,625 noun lus and 11,573 adjective lus) were described in plwordnet 3.0 emo by emotive annotation @xcite . from that point we started the annotation process aiming at its expansion by complete emotive annotations (2+1) for around 100k more lus. annotations done in the pilot project including decisions of only one annotator had to be completed. we started adding emotive annotation from noun lus with focus on hypernymic branches that are likely to include lus with polarised sentiment. in addition we try to distribute manual annotations across the network of synsets in such a way that it will be possible to apply an algorithm for automated spreading annotations to the rest of lus. the statistics describing the current state of the work are presented in tab. 1. only lus annotated by two annotators are counted as completed. this number includes also completed annotations for lus processed during the pilot project. as annotators are mixed in pairs and subsets of lus are assigned to them in diversified ways, a large number of lus have received so far only one annotation. as it was also the case in the pilot project, more than half of the noun lus are annotated as neutral. however, only ≈30% of adjective lus are neutral contrary to almost 60% in plword-net 3.0 emo. this difference can be caused by a much broader coverage of noun lus, while adjective lus were selected by @xcite in a slightly accidental way (there was an ongoing plwordnet expansion work on that time). negative sentiment polarity values dominate in annotation: 33.84% vs 15.38% in tab. 2. this correlates with the dominance of the negative basic emotions that can be observed in the statistics presented in tab. 3, i.e. 76.48% emotions associated with noun lus and 70.13% with adjective lus are negative. a similar dominance of words marked negatively could be also observed in the dictionary of the colloquial polish language @xcite . for instance, if we compare two thematic fields of this dictionary, namely: acting towards somebody's harm -enforcing some particular behaviours (id:2.3.2) and acting towards somebody's profit (id.: 2.3.3), we can notice that the former includes 324 entries while the latter only 20. in plwordnet emo it is also characteristic that almost all emotions except fear are approximately frequent while joy is a single dominating positive emotion. this bias can be a result of assigning joy not only as a simple emotions, but also as a basic component of the complex emotions. contrary to the basic emotions, the fundamental human values are evenly distributed between the positive and negative ones, see tab. 2: 55.38% negative values assigned to nouns and 50.57% to adjectives. there are no single fundamental human values that are substantially more frequent across the annotations, but only some of them, e.g. prawda 'truth' are significantly less frequent. language users mostly perform evaluations of an emotional or utility (advantageous vs nonadvantageous) character. they relatively infrequently assess phenomena from the rational perspective. emotively marked lus are more frequent in colloquial or informal communication where emotions and advantages are more important than rational thinking. we checked also combinations of sentiment polarity values inside synsets. almost all synsets are consistent with respect to the sentiment polarity, i.e. only ≈ 20 synsets from many thousands analysed included lus of both positive and negative polarity, and most of them result from errors in plwordnet, e.g. too broad synsets. synsets including marked lus and neutral or ambiguous ones are more frequent, but perfectly compatible with the annotation guidelines. lu linked by hypernymy (via synset hypernymy) are in the vast majority of cases in the same polarity. we found only less than 700 hundred lus linked by hypernymy per more than 70,000 analysed pairs in which both lus were in the different polarity, among which we found only 32 -s, +s pairs. ## conclusions and further works a large emotive lexicon can be an indispensable language resources for sentiment analysis and opinion mining, if it is of good coverage and quality, especially if the lexicon-based method is expanded with domain adaptation on the basis of machine learning. at least the use of the lexicon can help to improve the domain independent aspect of the method. the pilot project @xcite showed that with relatively small workload a promising emotive resource was be created. we presented an annotation process following this project and aiming at building a very large emotive lexicon of polish of more than 130k manually annotated lexical units from plwordnet, i.e. on a scale incomparable to the majority of existing resources. the intended size is meant to suit the envisaged applications. a slightly modified general model and annotation guidelines were presented, together with improved specific guidelines for adjectives. both the lexicon as well guidelines utilise the rich relation structure of plwordnet. the observed high values of the inter-annotator agreement (measured on a large sample of data according to an objective procedure) is very promising for the future applications and is a strong argument in favour of the assumed model and annotation procedure. the presented first results for nouns and adjectives, but for quite large sample, allows for collecting interesting observation that are in line with qualitative analysis in literature. we plan to complete annotation (>130k lexical units in total) of all parts of speech in plwordnetby the july 2018. the results will be completely open. the annotation will be extended to the rest of plword-net by automated method (e.g. based on activation propagation or machine learning.) we plan also to compare our annotation with annotation built for english using the mapping of plwordnet onto princeton wordnet.