Modality and Negation in Event Extraction (2021)

Abstract: Language provides speakers with a rich system of modality for expressing thoughts about events, without being committed to their actual occurrence. Modality is commonly used in the political news domain, where both actual and possible courses of events are discussed. NLP systems struggle with these semantic phenomena, often incorrectly extracting events which did not happen, which can lead to issues in downstream applications. We present an open-domain, lexicon-based event extraction system that captures various types of modality. This information is valuable for Question Answering, Knowledge Graph construction and Fact-checking tasks, and our evaluation shows that the system is sufficiently strong to be used in downstream applications.

## introduction linguistic modality is frequently used in natural language to express uncertainty with respect to events and states. downstream nlp tasks that depend on knowing whether an event actually occurred, such as knowledge graph construction, fact-checking, question answering and entailment graph construction, can benefit from understanding modality. such information is crucial in the medical domain, for instance, where it facilitates more accurate information extraction and search for radiology reports @xcite . similarly, if we pose a question in the socio-political domain, such as did the protesters attack the police?, our answer will be different depending on the evidence that the system has observed: protesters attacked the police @xcite or protesters are unlikely to have attacked the police [uncertain] 1 . these challenges are exacerbated by the prevalence of the phenomenon. in a multi-domain uncertainty corpus @xcite , sentences containing uncertainty cues are significantly more common in newswire text (18%) compared to encyclopedic text (13%). modality is also frequently observed in editorials @xcite . we show that within the news genre, modality is common in the politics and sports domains, where experts often make predictions and state their opinions on the possible outcomes of events such as elections or sports matches, and analyse alternative outcomes where situations unfold differently. we present montee foot_0 , an open-domain system for modality and negation tagging in event extraction. tagging these phenomena allows us to distinguish between events that took place (e.g. protesters attacked the police), those that did not take place (had protesters attacked the police...), or are uncertain at the time that a document is written (protesters may have attacked the police). the extracted relations include a predicate and one or two arguments, for example: protestersattack-police (from the sentence protesters attacked the police). the predicates are analysed according to the following semantic phenomena: negation, lexical negation, modal operators, conditionality, counterfactuality and propositional attitude. see we contribute a lexicon of words and phrases that trigger modality, a parser that extracts and tags open-domain event relations for modality (along with an intrinsic evaluation), and a corpus study focusing on the politics domain of a large corpus of news text. category example ∅ protesters attacked the police negation protesters did not attack the police lexical negation protesters refrained from attacking the police modal operator protesters may have attacked the police conditional if protesters attack the police... counterfactual had protesters attacked the police... propositional journalists said that attitude protesters attacked the police modality: in this work, the focus is on any kind of modality indicating uncertainty, including modal verbs, conditionals, propositional attitudes, and negation. we see modality primarily as a signal for determining whether or not the event in question actually occurred, so that downstream applications can take this into account. we begin by discussing the typical, more specific category of modal operators. linguistic modality communicates a speaker's attitude towards the propositional content of their utterance. formally, modality has been defined in terms of quantification over possible worlds @xcite . other definitions focus on categorising the speaker's attitude, such as epistemic necessity (that must be john.), epistemic possibility (it might rain tomorrow.), deontic necessity (you must go.), and deontic possibility (you may enter.) @xcite . sometimes a lexical trigger of modality is ambiguous between categories; english may, for example, is ambiguous between an epistemic possibility reading (it may rain tomorrow.) and a deontic possibility reading (you may enter.) these definitions have brought about a variety of annotation schemes in practice. @xcite propose five classes of modality: ability, effort, intention, success, and want, and train a classifier on crowd-sourced annotated data. @xcite extend the number of modality classes to include requirement, permission, and belief, and combine these with negation. @xcite take a coarser, epistemic approach, asking whether events are asserted, negated, or speculated, and @xcite enrich the timeml specification language with yet other categories (e.g. evidentiality, conditionality). in english, modality can be expressed in a va-riety of ways. the modal auxiliaries (e.g. might, should, can) are commonly used, but modality can be lexicalised in many other trigger words. nouns (e.g. possibility), adjectives (e.g. obligatory), adverbs (e.g. probably) and verbs (e.g. presume that) can all indicate modality. in the long tail, speakers have access to vastly productive phrases that indicate their attitude. the following examples occurred naturally in the news domain @xcite ): that's how close they were to ..., i cannot come up with a scenario that has..., that's based on the world wide assumption that.... ## event extraction system overview whilst many event extraction systems have been developed, none capture the wide range of modality phenomena introduced in section 2.1. for example, neither openie nor ollie extract unary relations. they also fail to adequately handle all of the phenomena we are interested in, in particular counterfactuals and lexical negation. (see sec- johnson doubts that labour will win the election n (s[dcl]\np)/s[em] s[em]/s[dcl] n (s[dcl]\np)/(s[b]\np) (s[b]\np)/np np/n n tc @xmath0: ccg dependency graph for johnson doubts that labour will win the election; marked paths from doubts (blue, dotted) and will (orange, solid) to win. our system takes as input a text document, and for each sentence outputs a set of event relations. an event relation tuple consists of a predicate and either one, or two arguments (e.g. (the) protest-ended, angela merkel-addressed-npd protesters). we use a pipeline approach similar to that described by @xcite , which allows us to extract open-domain relations. each sentence in the document is parsed using the rotatingccg parser (stanojević and steedman, 2019) over which we construct a ccg dependency graph using a method similar to the one proposed by @xcite . (see figure 2 for an example of a dependency graph and figure 1 for the ccg parse tree from which it was extracted.) ccg dependency graphs are more expressive than standard dependency trees because they can encode long-range dependencies, coordination and reentrancies. we traverse the dependency graph, starting from verb and preposition nodes, until an argument node is reached. the traversed nodes, which are used to form the predicate strings, may include (non-auxiliary) verbs, verb particles, adjectives, and prepositions. the ccg argument slot position, corresponding to the grammatical case of the argument (e.g. 1 for nominative, 2 for accusative), is appended to the predicate. our focus is on the extraction of binary and unary relations. binary relations may be extracted from dependency paths between two entities. extraction of unary relations, which have only one such endpoint, poses a harder challenge (szpektor and dagan, 2008) -we must decide whether they are truly a unary relation, or form part of a binary relation. therefore linguistic knowledge must be carefully applied to extract meaningful unary relations. we extract unary relations for the following cases: verbs with a single argument including intransitives (bombs exploded) and passivised transitives (protests were held), and copular constructions (greta thunberg is a climate activist). in addition to binary and unary relations we also extract n-ary relations which combine two binary relations via prepositional attachment. these are of the form: arg1-predicate-arg2-preposition-arg3, and are constructed by combining the two binary relations arg1-predicate-arg2 and arg2-preposition-arg3. for example protestersmarched on-parliament square and parliament square-in-london combine to form the new relation protesters-marched on parliament square in-london (from the sentence: protesters marched on parliament square in london). passive predicates are mapped to active ones. modifiers such as managed to as in the example boris johnson managed to secure a brexit deal are also included in the predicate. as these may be rather sparse, we provide the option to also extract the relation without the modifier. arguments are classified as either a named entity (extracted by the corenlp @xcite named entity recogniser), or a general entity (all other nouns and noun phrases). arguments are mapped to types by linking to their freebase @xcite ids using aida-light @xcite , and subsequently mapping these ids to their fine-grained figer types @xcite . for example, angela merkel would be mapped to person/politician and npd (nationaldemokratische partei deutschland) to government/political party. the type system may be leveraged to identify events belonging to specific domains, for example, to identify and track political events such as elections, debates, protests etc. and the entities involved. ## lexicon since many of the phenomena we capture involve lexical trigger items, we opt for a lexicon-based approach. triggers identified using the lexicon can then be linked to event nodes in the ccg dependency graph. entries in the lexicon cover modality, lexical negation, propositional attitude, and conditionality, with counterfactuality handled separately. each entry contains the lemma, the categories that it covers, the pos-tag and an estimate of the epistemic strength that the word would normally indicate. a few examples are included in our lexicon is constructed by pooling together various lexical resources. the majority of the entries derive from the modality lexicon presented by @xcite , who use it for a similar rule-based tagging approach. their lexicon contains just under a thousand instances, but includes multiple forms for each verb inflection. using only infinitival forms, we add approximately 200 of the modal entries to our own lexicon. for modelling propositional attitude, we include a list of reporting verbs found in @xcite . this added roughly another 120 phrases to the resource. algorithm 1 tagging modal events 1: procedure tagmodalevents(sentence s, events e, lexicon l) 2: g, event nodes ← ccg dep parse(s, e) 3: trigger nodes ← [ ] 4: the new entries were separated by attitudes expressed through speech (tag att say, e.g. say, state) and attitudes of thought (tag att think, e.g. suspect, assume). more phrases expressing uncertainty are found in a data set of news domain sentences describing conflicting events, such as a win and a loss @xcite . such sentences often contained descriptions of events that didn't actually happen. yet more related words were found by generating each entry's wordnet synonyms and antonyms @xcite . we filtered and annotated these manually to obtain just under another 200 phrases, and added these to the lexicon. we also took inspiration from @xcite , especially for conditionals. in aggregate, this work resulted in a resource of 530 phrases. we also annotated each phrase with a modal category. our lexicon contains the categories deontic, intention and desire, and for the remaining phrases lists a indication of epistemic strength, with values 4 (definitely), 3 (probably), 2 (possibly), 1 (probably not) and 0 (definitely not). the latter correspond to lexical negation. the epistemic strength values were manually annotated by the authors, and are proposed as a means to collect subsets of events, such as all events marked as probable or higher. this phenomenon deserves more attention in future research however, as it is highly contextualised. for example, could win the lottery should deserve a different annotation to could have breakfast. ## modality parser we use the ccg-based event extraction system (section 3) and the expanded modality lexicon (section 4) in tandem to assign modal categories to events. the procedure is described in algorithm 1. the focus of the tagger is to identify the bulk of uncertain events: we prioritise recall over precision, so that we can expect events without a tag to have actually happened. the event extractor produces a ccg dependency graph g that contains a node n for each word in the sentence (line 2 of the algorithm). we then decide which of these nodes is a trigger (lines 4-7). for modality, negation, lexical negation, propositional attitude and conditionals, we tag these nodes if the node's lemma is present in the lexicon (check lexicon function, line 5). the loop in the algorithm covers the simple case of single token modal triggers (such as possible), and can be extended to multi token triggers (e.g. shoot for) 3 . counterfactual nodes are identified separately. the check cf function (line 5) finds instances of the token "had" that are assigned one of two indicative ccg supertags: for example in the sentence the protesters would have been arrested, had they attacked the police, the token "had" would be assigned the ccg supertag (((s\np)\(s\np))/(s[pt]\np))/np and is therefore recognised as an instance of counterfactual had. additionally, any instance of "if" that governs an instance of "had", is labelled as counterfactual. upon realising that even this common counterfactual pattern was rare in the corpus, we decided not to engineer further counterfactual patterns. we can then decide whether an event node should be tagged, by checking whether there is a path in the dependency graph from the trigger nodes to the event node (lines 9-12). figure 2 illustrates the intuition behind walking the dependency graph. the graph shows a path from both doubt and will to win. this works because the existence of a path between a trigger node and an event node corresponds to the trigger node taking syntactic scope over the event node. the semantic phenomena we handle all rely heavily on this syntactic process (for example negation, see @xcite ). 3 we implement this as a recursive loop over a trie data structure. a single event node may be connected to multiple triggers, so we choose the final tag on line 15. since our primary concern is whether the event happened, we do not combine tags and instead assign a single tag based on the following order of precedence: mod, att say, att think, cond, count, lneg, neg. the negation categories need to be ordered last because an event that is negated and modal is still uncertain (e.g. might not play shouldn't result in neg play), but the ordering is otherwise arbitrary. ## comparison with existing event extraction systems we highlight the capabilities of our system on five example sentences, comparing with two existing event extraction systems: openie @xcite and ollie @xcite . note that this is not intended as a conclusive evaluation of systems, but rather as a high-level overview of the phenomena captured by each of the systems. see ollie @xcite handles the phenomena in more detail. it identifies conditionals by detecting markers such as "if" and "when", and labels the enabling condition for extracted relations that are governed by a conditional foot_1 . it typically includes modals and negation as part of the predicate, montee openie ollie the guerrillas are ready to talk with the soviets, if moscow is willing. mod (guerrillas; talk; soviets) (guerrillas; are; ready) (moscow; is; willing) cond (moscow; be willing) (guerrillas; talk with; soviets) (guerrillas; talk; if moscow is willing) (guerrillas; talk; willing) (moscow; is; if moscow is willing) (moscow; is; willing) had trump won the election, cummings would still be in downing street. count (trump; win; election) (trump; had trump won; election) (trump; had won; the election) mod (cummings; be in; d.st.) (cummings; would; would still be in d.st.) (cummings; would still be in; d.st.) protesters did not attack the police. neg (protesters; attack; police) ∅ (protesters; did not attack; the police) parliament failed to investigate the kremlin. (parliament; failed to investigate; kremlin) (parliament; investigate; kremlin) (parliament; failed to investigate; the kremlin) lneg (parliament.; investigate; kremlin) (parliament; to investigate; the kremlin) ed miliband says the government betrayed yorkshire. att say (government; betray; yorkshire) ∅ (the government; betrayed; yorkshire) (ed-miliband; say) [@xmath1 and captures propositional attitude in its handling of attribution (e.g. ed miliband says...). like ope-nie, ollie is not designed to handle counterfactuals. in terms of lexical negations, ollie extracts the predicate both with and without the negation cue, which is undesirable if the downstream nlp application needs to be able to distinguish between events that took place and those that did not. ## evaluating system performance in the absence of a pre-existing open-domain evaluation dataset that closely matches the task we are interested in, we conduct an intrinsic evaluation of our modality-aware event extraction system. we measure performance on a set of 100 extracted event relations with manually annotated labels denoting the degree of certainty (happened, didn't happen, uncertain). an event relation consists of a predicate plus argument pair (e.g. (protesters; attack; police)). note that we exclude both ollie and openie from this evaluation as neither system is designed to handle the complete set of modality or negation phenomena we are interested in (c.f. section 6). we filtered the articles in the newsspike corpus @xcite to obtain those where at least 20% of the event relations are tagged (to guarantee a reasonably dense distribution of modality). we then randomly selected five articles and processed them using our system to extract event relations. from these articles we selected 100 event relations foot_2 . at the sentence-level we ensured that we include only one event relation for each predicate node in the dependency graph, since all event relations with the same predicate node will be assigned the same modality. the set of 100 event relations was manually annotated by two of the authors of this paper, one native english speaker and one fluent speaker. for each event relation, we asked the annotators to answer the question does the text entail that the event definitely happens? using the following labels: the event happened (2), is uncertain (1), didn't happen (0). inter-annotator agreement over the set of 100 event relations was measured using cohen's kappa @xcite . the agreement score was 0.77, indicating substantial agreement, and the annotations differed for only 16 examples. following the initial annotation task, the two annotators resolved the disagreements, which resulted in the gold standard test set. to evaluate our system, we mapped systemassigned modal and negation tags to the set of certainty labels, with lneg and neg tags mapped to 0 (didn't happen), empty tags mapped to 2 (happened), and all other tags mapped to 1 (uncertain). in we performed an error analysis of the 17 errors made by our system on the test set of 100 event relations. parsing was a common issue, with five errors attributed to general parsing mistakes, and five errors due to missing dependency links between reporting verbs and events in quoted text (e.g. "police were attacked", they said). two mistakes were due to human error, as the annotators also missed these reporting verbs in longer sentences. then, three errors arose from issues with the lexicon. two of these stemmed from lack of coverage: our lexicon does not handle temporal displacement, as in we won't act until the white house gives more information. the other was caused by incorrect application of a lexical entry, which would need to be disambiguated using context. finally, two errors could also have been avoided by handling linguistic aspect, as in they began the process to.... future research could thus focus on expanding the lexicon by these final categories of displacement, and taking context into account when linking a word to the lexicon. ## corpus analysis we conducted a corpus analysis of extracted relations over the newsspike corpus @xcite . newsspike contains approximately 540k multi-source news articles (approximately 20m sentences) collected over a period of six weeks. we report on the distributions of tagged phenomena over the set of binary relations 6 extracted from news articles in the complete corpus (general domain), and for the subsets of articles related to the politics and sports domains. the newsspike corpus does not include topic or domain information in the article-level metadata. therefore to identify articles belonging to the politics and sports domains we leveraged the named 6 the corpus study of unary relations is left for future work the distribution of relation tags over the general, politics, and sports domains is shown in the top ten most frequent trigger words found in the general domain are: the propositional attitude trigger say, the modal triggers will, would, can, could, may, should, want and have to, and the conditional trigger if. the same top ten are also observed for the politics domain (with different frequencies), and for the sports domain the propositional attitude trigger think replaces want. the similarity of these lists is perhaps not surprising as all three domains belong to the news genre. ## future work an obvious limitation of our approach is that it does not take into account the context in which events and trigger words occur. modality is a contextdependent phenomenon, so using the sentential context would improve accuracy. for example, the word unbelievable is ambiguous between an unlikely and an amazing, and happened reading. relatedly, our concept of epistemic strength is highly context-sensitive, and requires further development. a promising avenue is to develop a pre-training procedure for a modality-aware contextualised language model @xcite . we plan to use our modal lexicon to identify sentences with modality triggers. we will then gather human annotations of the certainty that each event happened, and use this annotated data to train a modality-aware language model able to classify event uncertainty. such a system might eventually even tackle the long-tail of modal examples mentioned in section 2.1. we will also investigate the application of zero shot and few shot learning to the problem of detecting modality and negation. this could provide a way to leverage a large pre-trained language model together with a small annotated corpus. our system was developed for english, but work is already underway to develop event extraction systems for other languages including german and chinese. extending to other languages would allow us to apply our methods to multilingual and crosslingual nlp tasks. finally, most ccg parsers, including the one used in this work, are trained on english ccgbank @xcite . this makes them perform well on news text, but accuracy suffers on out-of-domain sentences, primarily those involving questions. the results could be improved by retraining the parser on the ccg annotated questions dataset @xcite , allowing us to apply our system to the task of open-domain question answering in an extrinsic evaluation. ## conclusion we have presented montee, a modality-aware event extraction system that can distinguish between events that took place, did not take place, and for which there is a degree of uncertainty. being able to make such distinctions is crucial for many downstream nlp applications, including knowledge graph construction and question answering. our parser performs strongly on an intrinsic evaluation of examples from the politics domain and our corpus analysis supports our claim that modality is an important phenomenon to handle in this domain.