Getting Hold of Villains and other Rogues (2021)

Abstract: In this paper, we introduce the first corpus specifying negative entities within sentences. We discuss indicators for their presence, namely particular verbs, but also the linguistic conditions when their prediction should be suppressed. We further show that a fine-tuned Bert-based baseline model outperforms an over-generating rule-based approach which is not aware of these further restrictions. If a perfect filter were applied, both would be on par.

## introduction in online media including social media, the world is often conceptualized as being divided into beneficiaries and benefactors, victims and villains. for quite some time, the most interesting questions seem to have been: who is to blame and who benefits most. in this work, we strive to create a dataset and a first model to answer the first of these questions, i.e. to identify the villains in texts. but what is a villain, anyway? are we compelled to reveal our moral convictions in order to answer this question? a murderer, a cheater, a liar seem to be clear cases. but what about white lies and the cheating in a card game? we could introduce a severeness score in order to quantify the villainousness grade. in this paper, we describe our annotation efforts to create a corpus of sentences that comprises at least one entity that realises a negative (semantic) role. the filler of a negative semantic role might be a person, organization etc. but it also might be an event or even a non-animate physical object. especially in metonymic constructions, non-animate fillers are to be expected. although we also have started to annotate the strength of negativity, in this short paper we focus on the language usage that gives rise to the assignment of negative roles per se. in the second part of the paper we discuss two models: a rule-based and a bert-based one. the goal of our annotation is to identify those entities of a sentence that occupy a negative semantic role. a number of constructions can be used to take a negative perspective on some entity. one can do it explicitly by a noun phrase (the lies of the president), a predicative construction (he is a liar) or by using a verb who implies a negative actor (he vilifies the people). in this paper, we focus on verbs. it turns out, though, that not every usage of such a verb assigns a negative role. only if the situation at hand is factual, then a negative role actually indicates a villain @xcite . also ambiguity has to be taken into account. we, thus, are talking about a probability distribution, depending on various grammatical parameters. before we have a more detailed look at this grammatical means, please note that quite a couple of verbs do have negative roles especially at the actor position. we have identified about 400 for the german language @xcite . among them are verbs that indicate a crime (e.g. to murder, to kill, to injure, to torture ...), but also verbs like to vilify, to rebuff, to lie, to cheat, to mock, to demoralize, to prejudge and so on. most of the time, the subject of the verbs bears the negative role. as we said, metonymic reference has to be taken into account. besides classical cases of metonymy like producer for product (e.g. pynchon is hard to read), we also consider all references to be metonymic when humans are involved, e.g. this agreement destroys our hope. there are a number of grammatical means (see fig. 1 ) that indicate non-factuality and thus block the assignment of negative roles. in reported speech (1), the actor of the reported event (china) is blocked. subjunctive mood (2) inhibits the inference (agreement) since nothing has happened, which is also true for future tense (3). for verbs that have a theme dependent nega- in traditional machine learning we would use the items from fig. 1 as features. a rule-based system could try to use them as filters. in a deep learning scenario, e.g. a bert-based model, we could hope that the fine-tuning process will be sufficient to learn the regularities. ## annotated corpus as source for sentences that might have a negative role, we selected 1300 sentences from two corpora foot_0 . the first one is a german newspaper corpus called tuebadz @xcite comprising more than 100,000 sentences (publically available) and the second one are facebook posts of a german right-wing party (afd) with more than 300,000 sentences foot_1 . the afd texts also contain offensive language. the tuebadz data, on the other hand, comes from a left-oriented newspaper. we deliberately have chosen two different world views in order to have a broader range of examples. we generated the candidate sentences by the following procedure: we parsed the sentences with the german parzu parser @xcite and then for those sentences that had a verb from our lexicon, we extracted the predicate argument structures (as a preprocessing step of our rule-based system, see @xcite ) from the dependency parse trees. finally, we identified the agent position (arg0) and suggested it to be a negative actor. given unser land wird von den medien zerstört which translates to our country is being destroyed by the media, the predicate argument structure (as a formula in predicate logic) is destroy(media,country). from this, media was extracted to denote a negative actor negative role filler(media). the two annotators were presented with the full sentence and had to determine whether the suggested negative actor actually is one. moreover, the strength of negativity had to be determined on the basis of a scale from 1 (low) up to 3 (high). a zero means false positive. in the course of the annotation, we removed a couple of sentences, because no actor was found by the predication extractor. we ended up with 1260 sentences. 460 cases are false positives, i.e. the found actor was not a negative actor, exactly 800 were true positives. we had a closer look at the reasons for the false positives, i.e. how the criteria from fig. 1 are distributed. only 4 cases would need coreference resolution, 18 are errors based on negation, 19 cases were future tense, 19 reported speech, 38 subjunctive mood, 48 reflexive usage, 59 conditional forms and 162 were wrong readings. we also had a number of parsing errors, namely 93 (wrong candidate). from 460 cases of false positives, thus, 183 cases (39.78 %) can be blocked by a perfect filter, coreference, negation, parsing errors and cases of wrong readings due to ambiguity are out of reach. our inter-annotator agreement is a kappa score of 0.78: i.e. whether the annotators agreed that a noun candidate was really a negative actor or a false positive. ## related work our task, detecting negative actors, is somehow related to the task of opinion role identification (see e.g. @xcite ), where the goal is to identify the source (our case) and the target of an opinion event expressed by a sentence. however, our task is more specific and more general at the same time. we are interested in opinion sources that also are conceptualized as negative actors as in he vilifies the people. but we not only are looking at opinion sources but are also interested in any event source (or actor) that is negatively connotated through the sentence (e.g. he deliberately injured others). there is a superficial similarity with the work of @xcite where also a rulebased approach was used. our rule-based system is described in @xcite . among others, it also produces predictions of negative actors. the system uses a large verb lexicon where each verb is specified according to its various syntactic frames and where the frame elements are further specified with respect to their polar roles (e.g. negative actor) and the pro or con relation among each other. the shortcoming of the system clearly is that it is overgenerating, it only partially is able to identify nonfactuality and has no means to distinguish relevant from irrelevant readings of a verb. we have also tried to find related work in the fields of stance detection and even argumentation mining. but we are not aware of any approaches that directly focuses as we do on that task, neither for german nor for any other language. the field of emotion classification is relevant, as negative actors might evoke strong emotions. inspired by (oberländer et al., 2020)'s paper title, we want to investigate "which semantic roles enable machine learning to infer" the negative sentiments towards agent entities. based on cognitive appraisal theories, the corpus of thousand sentences described in @xcite explicit the link between emotions caused by events and the appraisal dimensions. our ongoing attempt to quantify the severeness of negativity involved might benefit from a closer look at the emotional side. @xcite annotated emotions in english news headlines via crowdsourcing, together with semantic roles and the reader's perception. to gain more insights into the severeness dimension, crowdsourcing could be a good way in order to come to a more representative since larger corpus. ## conclusion we have introduced a dataset of 1260 actor-verb pairs (including their sentences) where each pair either identifies a negative actor of a factual situation described by the verb or the actor is not a negative actor mostly because the verb denotationgiven the sentence -is non-factual. factuality could in principle be determined on the basis of certain grammatical indicators, but other inference blockers are harder to identify (verb ambiguity). if the rule-based system had a basic (and perfect) filter, both approaches, dl and rb, are on par. we have shown that the neural models (dl) are able to learn the needed distinctions without relying on manual feature engineering or manual filter specifications. the identification of negative actors might be useful for a system that detects offensive language or hate speech, where targets (e.g. migrants) quite often are being conceptualized as villains. future work will focus on the determination of the strength of negativity. manually quantifying negativity of actors is an error prone task. moreover, the actual strength value is not so crucialit is the right ranking (is actor a more negative than actor b) that counts. we have started to experiment with a lexicon-based quantification metric (see @xcite for the lexicon) that takes into account different types of sentiment specifications, e.g. appraisal categories @xcite (judgement, emotion, apprehension words) and the classification of emotion words according to the base emotions they express @xcite .