Automatic Data Acquisition for Event Coreference Resolution (2021)

Abstract: We propose to leverage lexical paraphrases and high precision rules informed by news discourse structure to automatically collect coreferential and non-coreferential event pairs from unlabeled English news articles. We perform both manual validation and empirical evaluation on multiple evaluation datasets with different event domains and text genres to assess the quality of our acquired event pairs. We found that a model trained on our acquired event pairs performs comparably as the supervised model when applied to new data out of the training data domains. Further, augmenting human-annotated data with the acquired event pairs provides empirical performance gains on both in-domain and out-of-domain evaluation datasets.

## introduction event coreference resolution aims to determine and cluster event mentions that refer to the same realworld event. it is a relatively less studied nlp task despite being crucial for various nlp applications such as topic detection and tracking, question answering, and summarization. a typical event coreference resolution system relies on scoring similarity between two event mentions in a document followed by clustering. however, event coreference chains are very sparsely distributed and only certain key events are repeated in a document, which makes manually labeling many event coreference relations very time-consuming. furthermore, event mentions tend to appear in extremely diverse contexts and few are accompanied by a full set of their arguments. the two challenges, the absence of abundant human-annotated event coreference data and the high diversity of contexts containing coreferential event mentions, make it hard to build effective event coreference resolution systems. we aim to improve the effectiveness of event coreference resolution systems by automatically acquiring coreferential event pairs from many documents requiring minimal supervision. specifically, coreferential event mentions are associated with discourse function of sentences in a news document @xcite foot_0 . we propose to use them to identify sentence pairs that are likely to contain coreferential event mentions as well as sentence pairs that are likely to contain noncoreferential event pairs. consider the two example sentence pairs below, each pair having an event pair with synonymous trigger words. in example (1), the first sentence describes a historical event about the reduction in poverty during the last decade, while the second sentence projects the challenges of further reducing poverty in the coming years. here, the two reduce events are non-overlapping in the temporal space and are noncoreferential. on the contrary, in example (2), both mentions for the event resettle refer to the same real-world event and can be so ascertained by knowing that both sentences describe the same main event in a news article. in general, we can recognize pairs of sentences in news articles that are likely to contain coreferential or non-coreferential event mention pairs by knowing the sentence's discourse function following van dijk's theory. to ascertain our hypothesis, we first use the discourse profiling system and dataset introduced by @xcite to identify the discourse role for each sentence in a news article. then, we use multiple rules to capture the distributional correlation between event coreference chains and discourse roles of sentences and collect a diverse set of 9,210 coreferential and 232,135 non-coreferential event pairs foot_1 . to assess the reliability of the proposed data augmentation strategy, we perform manual validation on subsets of both coreferential and non-coreferential event pairs. then, we train event coreference resolution systems using the acquired data alone or using the acquired data to augment a human-annotated training dataset. we evaluate trained systems on two datasets, the news portion foot_2 of the widely used benchmark evaluation corpus kbp 2017 as well as the news portion foot_3 of the richer event description (red) corpus @xcite . unlike the kbp corpora that only consider eight event types for event coreference annotations, the red corpus comprehensively annotates all the event types that appear in a document, and is arguably the only comprehensively annotated corpus of event coreference relations. assuming the automatically acquired event coreference data is not available, we also train a supervised event coreference resolution system using the kbp 2015 corpus foot_4 . on the kbp 2017 corpus, the event coreference resolution system trained on the acquired data performs slightly worse than the system trained using the kbp 2015 corpus, the human-annotated in-domain training data. but, on the red corpus, both the systems trained on either the annotated kbp 2015 corpus or the acquired data obtain roughly the same evaluation results. further, the system trained on combined annotated kbp 2015 and automatically acquired data yields the best results on both the kbp 2017 dataset and the red dataset. lastly, we evaluate all the trained systems on a different text genre, discussion forum articles from the kbp 2017 corpus, and found that all the systems obtain comparable results. overall, the performance gain of all the trained systems on discussion forum documents is marginal compared to a simple trigger word match baseline. thus, increasing training data size does not improve the performance of an event coreference resolution system on a new text genre. we suspect that, for generalization across different text genres, we may require specialized learning algorithms, e.g., text style adaptation, which is not in the scope of this work. ## related work the existing literature on supervised event coreference resolution primarily focuses on designing pairwise classifier based on the surface linguistic features such as lexical features comprising of lemma and part-of-speech tag similarity of event words @xcite @xcite @xcite , argument overlap @xcite @xcite @xcite @xcite , semantic similarity based on lexical resources such as wordnet @xcite @xcite and word embeddings @xcite @xcite @xcite @xcite , and discourse features such as token and sentence distance @xcite . the resulting classifier is used to cluster event mentions. the commonly used strategies include agglomerative clustering that selects the antecedent closest in mention distance that is classified as coreferent or the antecedent with the highest coreference likelihood @xcite , hierarchical bayesian @xcite or spectral clustering algorithms @xcite . in this work, we use the pre-trained bert model to extract both event and context features and use agglomerative clustering to form event coreference chains. supervised models suffer from a lack of human-annotated event coreference data. to address the annotation scarcity problem, @xcite proposed to learn structured event representations on large amounts of text and use the similarity score between two event representations to form event coreference chains. their model uses a small human-annotated event coreference dataset to find the appropriate similarity score threshold for linking two events. unsupervised models based on probabilistic generative modeling have also been successfully used for event coreference resolution @xcite . however, both semi-supervised and unsupervised approaches have been found empirically lagging behind the supervised models @xcite . the closest to our work are weakly-supervised and self-training methods that have been shown useful for many information extraction and classification tasks @xcite @xcite . but, to the best of our knowledge, we are the first to explore discourse-aware strategies to automatically label event coreference relations and use them exclusively or use them to augment existing human-annotated data for training event coreference resolution systems. ## event coreference data acquisition to acquire coreferential event-pairs without direct supervision, we first collect event trigger words along with their potential set of coreferential event mentions using the paraphrase database (ppdb 2.0) @xcite foot_5 . then, we use high precision rules informed by the functional news discourse structures @xcite to identify seed coreferential and non-coreferential event pairs followed by a single bootstrapping iteration to collect additional non-coreferential event pairs. ## event coreference resolution system we design a neural network-based mention-pair classifier for event coreference resolution. we represent each event pair using 50 context words to the left and right of the first and second event trigger words respectively, and with the maximum of 200 words in between the two event words 10 . given the event context (w 1 , ., e 1 , ., e 2 , ., w n ), we first transform the context words sequence to word embeddings sequence (b w1 , ., b e1 , ., b e2 , ., b wn ) using the pre-trained bert-large-uncased model @xcite . then, we model the semantic associations between two event mentions by measuring similarity between their event embeddings (b e1 , b e2 ) through element-wise product and difference. further, we obtain context embedding (c) through maxpool operation over the word embeddings sequence to model contextual cues. while the context provides important cues for identifying coreferential event mentions, it may not always be relevant for resolving coreference links. for instance, many event trigger word pairs such as ("injuries", "recommended") are extremely unlikely to exhibit coreferential relations irrespective of their context. therefore, we use the similarity between event embeddings to control the context input and use them only in the scenarios where event trigger words are likely to possess coreferential link. to achieve so, we apply linear neural layer over element-wise product and differences of two event mention embeddings followed by the sigmoid activation, and multiply them with context embedding c. finally, we concatenate the resulting set of embeddings and then use a three-layer feed-forward neural network classifier to score the coreference likelihood. the exact formulation of the coreference classifier is described in eq. 1. we train the model using binary cross-entropy 10 we take 100 context words to the right and left of the first and second event trigger words respectively when the number of context words in between them exceeds 200. loss. during inference, we use the best-first clustering approach, where we select the antecedent having the highest pairwise coreference score based on the coreference classifier, to build event chains. ## conclusions and future work we presented an automatic data acquisition strategy for event coreference resolution by mining the func-tional news discourse structure. we performed both qualitative and empirical studies to determine the effectiveness of our proposed strategy. we found that the model trained on automatically acquired event pairs performs similarly to the model trained on human-annotated corpus when evaluated on the test set covering general event domains. further, augmenting acquired event pairs to existing humanannotated data improves the performance of the model on both training-domain and broader domain test sets. for future work, we intend to develop new training algorithms to improve the generalization capability of models on a new text genre. further, we plan to evaluate a similar event coreference data acquisition strategy for new languages.