Is Killed More Significant than Fled? A Contextual Model for Salient Event Detection (2020)

Abstract: Identifying the key events in a document is critical to holistically understanding its important information. Although measuring the salience of events is highly contextual, most previous work has used a limited representation of events that omits essential information. In this work, we propose a highly contextual model of event salience that uses a rich representation of events, incorporates document-level information and allows for interactions between latent event encodings. Our experimental results on an event salience dataset demonstrate that our model improves over previous work by an absolute 2-4% on standard metrics, establishing a new state-of-the-art performance for the task. We also propose a new evaluation metric that addresses flaws in previous evaluation methodologies. Finally, we discuss the importance of salient event detection for the downstream task of summarization.

## introduction identifying the salient information in a given piece of text is a ubiquitous and important problem in natural language understanding. while important parts of the text have been identified by attending to entities @xcite , elementary discourse units @xcite , or whole sentences @xcite , in this work, we choose to model extracting important events @xcite . events are the core parts of most sentences -they center around a predicate and include its key arguments -yet they are compact semantic units, and a salient event in a sentence could carry the sentence's meaning efficiently. extracting important events has been shown to be central to many downstream tasks, such as summarization @xcite , storyline creation @xcite and question answering @xcite . to model the importance of an event, it is critical to understand its context: who is involved, where did it happen, what other events is it related to, and more. for instance, in figure 1 , the difference in salience of the "fled" events in each document is significantly influenced by their arguments ("2,100 colombians" versus "shopkeepers"). previous work that identifies salient events has a limited event representation that is unable to capture these important contextual signals @xcite . in contrast, the model which we propose in this work ( §3) directly models the context of an event in three different ways as follows. first, instead of representing an event by only its predicate mention, our representation includes the subject, object, time, and location of the event ( §3.1). second, we directly incorporate global features into the model that capture hierarchical relations between events, abstract event frames, position information, and more ( §3.2). third, we use a neural network architecture that includes an inter-event interaction layer, which allows information to be passed between latent event encodings so that other events may increase or decrease another's importance ( §3.3). our experimental results on a standard event salience dataset @xcite demonstrate that these contextual signals significantly increase performance ( §4.4). we find that our model performs 2-4% better than previous work, setting a new state-of-the-art performance for the task. ## related work important information identification has been a topic of interest in nlp community since the 1980s and through these years, researchers have defined salience in multiple ways. @xcite divides the text into nuclei and satellite with an idea that a satellite is incomprehensible without the nucleus but, after removing some satellites, the text can still be understood. @xcite discusses identifying events that would have triggered the author to write that article. @xcite proposed the idea that the central events have a large number of coreferential event mentions and those mentions are spread throughout the document. however, we believe that in realistic documents, redundancy is commonly used for various rhetorical or other reasons, it is not necessarily the case that frequently mentioned events convey the main point of the article. we follow proposals from entity salience @xcite and event salience work @xcite that suggest that these are difficult to explicitly define, but can be learned from observing human summaries: events that appear in the summary are salient. event representation has also evolved over time. @xcite represented narrative events as pairs of verb and the grammatical dependency relation between the verb and the entity. @xcite included nominal predicates, using the nominal form of verbs and lexical items under the event frame in framenet @xcite . further work by @xcite , @xcite , and others incorporated arguments such as propositional objects. however, most of the work on event salience identification has represented events as verbs/nominal event mentions. to address this, we use a more holistic event representation which includes nominal/verbal event mentions, entities in the subject and object of the predicate as well as the time and location. a similar representation was used by @xcite to build an event detection and co-reference system. from modeling perspective, most earlier work @xcite on event salience has built rule based systems (e.g. presence of the event in the main clause, its voice, etc.). more recent work has focused on capturing coreference relation between events @xcite and on automatically capturing salient specific interactions between the discourse units @xcite . however, as mentioned earlier, we believe that events are highly contextual, so we use a more expressive model to obtain contextualized embeddings for events. it additionally helps us in capturing local interevent interactions and, to capture the document level interactions, we design a number of global document level features. ## salient event extraction we choose to model salient event extraction as a binary classification task. for every verbal and nominal event mention e 1 , . . . , e n in a document, the goal is to predict whether or not each event is salient. in our experimental study we assume that the event mentions are provided. in the following sections, we describe our model's event representation ( §3.1), the global features which are incorporated into the representation ( §3.2), and the network architecture and inter-event interaction mechanisms ( §3.3). ## case study: extractive summarization in this section, we discuss the potential benefits of modeling event salience for the downstream task of extractive summarization. ideally, a good summary captures information about the key events in the original document. however, most common models for extractive summarization operate at the more coarse-grained sentence level. we discuss below why we believe that modeling event importance rather than sentence importance has the potential to benefit summarization. an improved supervision signal first, the extractive summarization training oracles often miss important signals in the data. extractive summarization systems are trained on 0/1 labels assigned to each sentence in the document to indicate whether or not that sentence should be selected to the summary. the most common procedure for obtaining these labels is to greedily select document sentences while the rouge @xcite score between the selected sentences and a reference summary still increases @xcite . we compared this labeling procedure to an alternative method based on events. the event-based oracle selects the sentence which contains the first occurrence of an event in the document for each event in the summary. for our analysis, the sentences selected by the rouge-based and event-based methods are divided into three sets: b, sentences chosen by both; r, sentences chosen by rouge only; and e, sentences chosen by the event method only. the relative sentence coverage for both sets of selected sentences is calculated as follows. we observe that, e @xmath0.8% which means that most of the sentences in the rouge-based oracle are covered by event based summaries. however, r @xmath1.4%, which means that the standard supervision signal misses a significant number of events present in the reference summary (all salient events in the remaining 51.6% sentences; see figure 4 (middle) ). the lower value of r cov is evidence that rouge-based oracles are missing a valuable supervision signal. finer-grained selection since the most popular extractive summarization models are forced to select full sentences to be in the summary, it likely that the model selects a lot of unimportant information. for instance, if a document sentence is quite long and covers a lot of information, it is likely that some of the predicate mentions in it are not salient and may not need to be in the summary. consequently, including only semantic unit that consist of important events and their arguments may be a better method of identifying text that should belong in the summary. to quantify this, we calculate the number of salient events in the sentences selected by rouge (set {b ∪ r}) and observed that 27.6% of the events in these selected sentences are not salient (see figure 4 (right) ). together, these two points demonstrate that the current method of creating oracle for training summarization systems misses important events which are present in the gold summaries while including a significant number of non salient events. this underscores that a good summarization system can benefit from our event salience model to select better events for generating summaries. ## conclusion in this work, we proposed a contextual model for salient event identification. we demonstrated that the three different components of our model (the event representation, global features, and inter-event interaction) combine to produce state-of-the-art results on the new york times annotated corpus. further, we identified issues with previous evaluation metrics and proposed new intuitive evaluation methods. finally, we discussed how event salience can be helpful for other downstream applications through a case study of extraction summarization.