Distinguishing Between Foreground and Background Events in News (2020)

Abstract: Determining whether an event in a news article is a foreground or background event would be useful in many natural language processing tasks, for example, temporal relation extraction, summarization, or storyline generation. We introduce the task of distinguishing between foreground and background events in news articles as well as identifying the general temporal position of background events relative to the foreground period (past, present, future, and their combinations). We achieve good performance (0.73 F1 for background vs. foreground and temporal position, and 0.79 F1 for background vs. foreground only) on a dataset of news articles by leveraging discourse information in a featurized model. We release our implementation and annotated data for other researchers

## prior work both @xcite and @xcite demonstrated approaches for identifying the central event in news articles. @xcite proposed a rule-based system to identify the central event in a human-generated document summary. they evaluated their system on a human generated summaries from the new york times corpus @xcite where the central event had been identified. @xcite used several rule-based systems and statistical classifiers to identify the most important event in a news article. they trained and evaluated their systems on 30 news articles from the red corpus @xcite and 74 news articles from the kbp 2015 corpus (o' @xcite . both were focused only on identifying a single central event, whereas we seek to label all events in a document as either foreground, background, or other. @xcite demonstrated an approach to placing events in news articles into three coarse temporal categories: past events that have already occurred; on-going events that are currently happening; and future events that may happen. in that work, the temporal category was relative to the document creation time (dct) and did not distinguish between foreground and background events. in contrast, our work seeks to mark the general temporal position of all background events relative to the foreground period. ## corpus we annotated 99 news articles from the intelligence community (ic) corpus @xcite . the ic corpus contains 100 news article but one article was merely a list of events rather than being a narrative. we used the gold event mentions that had been annotated on the corpus. the definition of event in hovy et al. follows that of timeml @xcite , which has been well studied and shown to be reliably annotatable: we mean both events and states when we say 'event'. a state refers to a fixed, or regularly changing, configuration of entities in the world, such as 'it is hot' or 'he is running'. an event occurs when there is a change of state in the world, such as 'he stops running' or 'the plane took off'. @xcite the first and second authors labeled each event in the ic corpus with one of eight categories: foreground, other, or six varieties of background (listed in ## model our model is a straightforward featurized logistic regression classifier. the features can be divided into five categories: lexical, syntactic, semantic, discourse and time. lexical and syntactic temporal signals (e.g., after or before) often occur before background events. we used the temporal signals list collected by @xcite . this feature is encoded as bag of signals capturing whether a temporal signal is present in the text between the target event and the immediate preceding event. for syntactic features, we use the major part of speech (pos), tense, and aspect, all encoded as one-hot vector. we used spacy @xcite to compute both lexical and syntactic features. semantic to capture semantics we computed an event contextualized representation using @xcite 's implementation of bert model bert-base-uncased @xcite . the vector for an event is defined as the weighted sum of all subword embeddings extracted from bert's last layer. we also captured the semantic frame of the event using semafor @xcite , encoded as one-hot vector. fine-grained coarse-grained discourse we employed two discourse features: rst discourse relation and the position of the event's sentence in the text. rhetorical structure theory (rst) @xcite is useful for many nlp tasks including sentiment analysis @xcite , information extraction @xcite , and subevent detection @xcite . we used feng-hirst discourse parser (feng and hirst, 2014) to build a discourse tree of each text, and post-processed the output to build a graph @xcite . for events after the first, we extracted the rhetorical relation between the target event mention and the immediately preceding event. this feature is encoded as a one-hot vector covering all 16 main relation classes. we also captured the position of the event's sentence in the discourse. this was encoded as real number, normalized to a value between 0 and 1 by the number of sentences in the article. time we compute the difference, in days, between the date of the event mention and the date in the first sentence. if there is no date in the first sentence, we use the document creation time. the date of the event mention is taken to be any date used as an argument to the event, or otherwise the nearest date that appears in the sentence; if the event has neither, we assume the difference is zero. we normalized both dates to a calendar value using heideltime utility @xcite . the difference is then encoded as one-hot vector feature with three possible values: negative, zero, or positive. classifier we used a logistic regression classifier from scikit-learn package @xcite for classification over the gold annotated event mentions. the classifier handles multi-class classification using a one-vs-rest scheme. most of the parameters were left at their default settings foot_1 . we addressed data imbalance (seen in ## results we evaluated model performance using both macro and micro f 1 . we conducted two experiments: in the first (the fine-grained condition) we use all classes from in the second experiment (the coarse-grained condition) we collapsed all background classes into one. ## discussion as shown in finally, the lowest performing class is the bpresent class, which is to be expected because of the low number of examples. ## error analysis upon detailed inspection we were able to discern several error classes aside from the usual noise introduced by the various sub-components. we observe that the model wrongly classifies foreground events as background if the event appears towards the end of the article. in our analysis, we also observe that this mislabeling occurs when the event is referred to in conjunction with some sort of temporal reference. for example, in an article regarding the capture of two people, in the sentence, "the captured bomb-maker, sami muhammad ali said al-jaaf , was seized in baghdad on jan. 15", the word seized is labeled as a background event even though it is directly tied to the foreground. as shown in within the fine-grained labeling of background events, we see that errors occur mainly between the distinction between events that are bpast and bpastpresent. of the fine-grained error, the mislabeling of bpast as bpastpresent was 22% of the error (see israeli security forces have been on high alert to guard against possible terror attacks by hamas, which has suspended operations against israel since its spiritual leader and founder sheikh ahmed yassin was released from israeli jail last october. figure 3 : example text showing an event which is subject to the common mislabeling of bpastpresent for bpast. with regard to general temporal position, changes of tense related to the document creation time (i.e., an event is in the past relative to the dct, but in the future relative to the foreground period), caused difficulties in distinguishing between bfut and bpast. though this error did not occur frequently, the failure to distinguish between the two classes comprises 19% of the overall fine-grained background error, as shown in ## applications to validate the importance of capturing background and foreground events as well as the temporal position of background events to the foreground events, we experimented with incorporating this feature into three different nlp tasks, namely: subevent detection, event coreference resolution and temporal relation extraction. the goal of this experiment was to measure the performance with and without including foreground/background fine-grained classes as features. even though some of the experiments we developed along the way outperform the state of the art, the emphasis here is on the contribution of these features in these tasks. all experiments were performed under the fine-grained condition and all experiments' implementation are released. in another fatality 1 , a spanish military adviser, gonzalo perez garcia, who fell 2 into a coma after being seriously wounded 3 in a shootout last month died wednesday, the spanish defense ministry said. figure 4 : example of a text where the relationship between one event and two others events is misclassified without the fine-grained foreground/background feature. subevent detection task the subevent detection is the process of identifying when one event is a subevent of another. the subevent relationship is defined in terms of (e i ,e j ), where e i and e j are events: event e j is a subevent of event e i if e j is spatiotemporally contained by e i @xcite . the task is to classify a pair of events into one of the three classes: parent-child, child-parent or no relation (norel), corresponding to the direction in the discourse flow. for subevent and event coreference experiments, we used the ic coprus, annotated with both event coreference and subevent relation, and compare our experiment to the state of the art models reported on the corpus (i.e. we used the gold annotation of background and foreground events). in the subevent experiment, we completely re-implemented the state of the art in this task @xcite and used the same evaluation metric, we refer interested readers to the paper for more details. for each event of a pair of event, we include the foreground/background fine-grained features as one-hot vector. as shown in event coreference task event coreference is the task of determining whether two events refer to the same event in the real world. for this experiment, we adapted some of the features mentioned in ( §4) for event pairs coreference classification. that is, we used the major pos, tense, aspect, semantic frames, discourse relation between pairs and semantic similarity between pair's bert embedding. also we include an argument feature to determine whether the pair's arguments corefer. we extracted and resolved the arguments of events using allennlp's semantic role labeling @xcite and coreference resolution @xcite . we train a pairwise logistic regression classifier from scikit-learn over the features using parameters shown in temporal relation extraction task last but not least, extracting temporal information from text is a challenging but important task in nlp. in this experiment, we target the extraction of temporal relation between events which is one of the fundamental tasks in temporal processing as identified in the series tempeval (te) workshops @xcite @xcite . we used the recently published dataset tddiscourse @xcite , an augmented dataset of timebank-dense @xcite focused on discourse-level temporal ordering and used the same set of temporal relations as timebank-dense (i.e., after, before, simultaneous , includes and is-included). the annotation of the corpus consists of two sets: manual annotation (tdd-man) and automatic inference (tdd-auto), we experiment on both. for this task, we designed a simple and effective approach by concatenating pair's bert embedding 5 , pos, tense and aspect as one vector. we trained a logistic regression classifier over these features using hyper-paramters shown in ## contributions we have presented a novel task: distinguishing between foreground and background events, as well marking the general temporal position of background events relative to the foreground period. we provided an annotated dataset, demonstrated a simple, featurized logistic regression model that performs well on this task which relies heavily on discourse understanding and show the utility of this task for three different nlp tasks. our error analysis shows that while our model's performance is reasonable, there is still room for improvement by the introduction of commonsense or world knowledge to aid in reasoning.