Conundrums in Event Coreference Resolution: Making Sense of the State of the Art (2021)

Abstract: Despite recent promising results on the application of span-based models for event reference interpretation, there is a lack of understanding of what has been improved. We present an empirical analysis of a state-of-the-art span-based event reference systems with the goal of providing the general NLP audience with a better understanding of the state of the art and reference researchers with directions for future research.

## introduction recent years have seen the successful application of span-based neural models to key entity-based information extraction (ie) tasks such as entity coreference resolution (cr) @xcite and relation extraction @xcite . unlike many non-span-based neural models, which typically learn task-specific contextualized word representations @xcite , span-based models are designed to learn task-specific representations of text spans. this potentially allows span-based models to create better representations of the entity mentions involved in entity-based ie tasks than their non-span-based counterparts as many entity mentions are multi-word expressions. can the successes of span-based models be extended to event cr? the vast majority of existing event coreference resolvers were developed in the pre-neural nlp era, focusing primarily on feature engineering @xcite @xcite @xcite @xcite @xcite @xcite @xcite and adapting the models originally developed for entity coreference to event coreference @xcite @xcite @xcite . neural event coreference models @xcite @xcite are few and far between, let alone span-based neural event coreference models. recently, @xcite designed the first spanbased model that has achieved state-of-the-art results on a standard event coreference dataset. despite these promising results, the use of span-based models in event-based ie tasks such as event coreference is still in its infancy. in particular, there is little understanding of what has been improved. in light of the above discussion, we present an empirical analysis of our state-of-the-art spanbased event coreference resolver @xcite with the goal of gaining insights into its behavior. we believe that our analysis will not only provide the general nlp audience with a better understanding of the strengths and weaknesses of span-based event coreference models, but also provide coreference researchers with directions for future work. 1 2 tasks and definitions in this section, we define the six tasks to be learned by our span-based event coreference model. the event coreference task involves identifying the event mentions in a document that refer to the same real-world event. in the example in the trigger detection (td) task aims to (1) extract from a document the event triggers, each of which is a word/phrase that expresses the occurrence of an event, and (2) assign an event subtype to each trigger that is chosen from a corpus-specific subtype inventory. in our example, ev1, ev2, and ev3 are triggered by "hire", "start", and "hired" respectively with subtype person-nel_startposition. two mentions cannot be coreferent unless they have the same subtype. the argument extraction task aims to ( 1 ) extract the arguments of an event mention and (2) determine the role played by each argument in the event. in our example, en3 and en4 are the arguments of ev1 with roles entity and person respectively, for instance. note that event arguments are entity mentions. two event mentions cannot be coreferent if there is a role for which their arguments refer to different entities. for instance, ev1 and ev3 are not coreferent because the person role of these two event mentions are filled by different entity mentions ("angela ahrendts" and "paul deneve"). the attribute prediction task aims to predict an event mention's attributes, each of which denotes a linguistic property of the event mention. for instance, one of the attributes defined in the ace 2005 event coreference annotation guidelines is tense, which denotes the tense associated with an event and has four possible values, past, present, future, and unspecified. the anaphoricity prediction task determines whether an event mention is coreferent with any preceding mentions. for example, ev1 is nonanaphoric while ev2 is anaphoric. entity cr involves identifying the entity mentions in a document that refer to the same realworld entity. for instance, en4 and en6 are coreferent because they refer to ahrendts. ## results of model variants we first evaluate the variants of our model on ace 2005 and kbp 2017. these variants differ along three dimensions, each of which has two choices: joint. this yields eight possible variants, whose results are shown in to study the impact of the encoder on event coreference performance, we compare the performances of the spanbert-b and spanbert-l resolvers. event coreference results, which are expressed in avg-f, are shown in the avgf columns of to better understand whether the spanbert-l resolvers offer better event coreference performance than their spanbert-b counterparts because of better td, better identification of coreference links, better identification of singleton clusters, or a combination of these three factors, we report in pipeline vs. joint. while there have been claims that joint models of event coreference outperform their pipeline counterparts because of their ability to address error propagation, to our knowledge there has never been a head-to-head comparison of pipeline and joint models in a controlled setting that differ only w.r.t. whether the tasks involved are learned jointly or independently. the results in to understand why trigger detection is easier on ace than on kbp, for each candidate event mention in the ace/kbp test set, we computed the entropy of the event subtype distribution of its underlying trigger, and found that the entropy (averaged over the candidate mentions in the test set) is lower for ace (0.433) than for kbp (0.517). this means that the subtype distribution of a candidate mention that appears in the ace test set is on average more skewed than that in the kbp test set. at the same time, we found that our model's trigger detector was not good at exploiting context for subtype prediction, as most of the time the model simply assigned an event mention the subtype that co-occurred most frequently with the underlying trigger in the training set. specifically, the average entropies of the predicted subtype distribution of a candidate mention in ace and kbp are 0.330 and 0.366 respectively, which are lower than the average entropies of the gold subtype distribution mentioned above. consequently, trigger detection results on kbp are worse than those on ace. the better event cr results obtained on ace could in part be attributed to better td, but the question is: is event cr easier on ace than on kbp? to answer this question, we feed our model with gold event mentions (i.e., gold mention boundaries and gold subtypes) during testing. this means that any error observed in the output of the model must be due to incorrect resolution. the results are as follows. the percentage of gold event mentions in the test set that are resolved incorrectly is 11.3% for ace and 19.2% for kbp. in other words, resolution performance on ace is indeed better than that on kbp. these results suggest that the better coreference results on ace can be attributed to not only better td but also better resolution. the next question is: what makes event cr easier on ace than on kbp? one reason is anaphoricity determination: the anaphoricity determination accuracies on the ace and kbp test sets are 66.5% and 57.2% respectively. we believe that anaphoricity determination is easier on ace because the event mentions are less ambiguous w.r.t. anaphoricity than those on kbp, as seen in the entropy values of anaphoricity distribution in ## results on resolution classes to gain additional insights into our best resolver (joint knowledge-rich spanbert-l), we analyze its performance on different classes of event mentions by partitioning the gold event mentions in the test set into 11 resolution classes. the first three classes contain event mentions that can be resolved via lemma matching or rote learning. (1) lemma match and seen pair (lm&sp). an event mention e is assigned to this class if it has an antecedent (i.e., an event mention preceding e that is coreferent with e) such that the two have the same lemma and have appeared as a coreferent pair in the training data at least once. we partition the remaining anaphoric mentions (i.e., those not covered by the first three classes) based on two factors. the first factor is the sentence distance (sd) between the gold mention e and its closest antecedent. the possible values @xmath0. the second factor depends on the number of coreferent arguments (ca) shared by e and an antecedent. the possible values @xmath1.r.t. each role, there are at least two roles containing coreferent arguments), = 1 (if e fails @xmath2, but has an antecedent such that the two have exactly one role in which their arguments are coreferent) @xmath3. we use these two factors to define six classes: (4) @xmath4, @xmath5, (5) @xmath6, @xmath7, (6) @xmath8, @xmath9, (7) @xmath10, @xmath11, (8) @xmath12, @xmath13, and (9) @xmath14, @xmath15. finally, we partition the non-anaphoric event mentions into two classes: (10) seen nonanaphoric (sna) mentions. a non-anaphoric mention is assigned to this class if it is seen in the training set. (11) unseen non-anaphoric (una) mentions. a non-anaphoric mention is assigned to this class if it does not appear in the training set. results of our resolver on these 11 resolution classes are shown in among the resolution classes involving anaphoric mentions, the ones with the highest ras are lm&sp, lm, and sp. this should not be surprising: lemma matching and memorization of pairs seen in the training set are by far among the most reliable indicators of event coreference. td f-scores are lower for lm than for the other two classes. this should not be surprising either, as td performance is strongly correlated with whether the mentions are seen in the training set or not. the six classes defined by sd and ca contain anaphors that are harder to resolve than the above three owing to the absence of reliable indicators of event coreference. two points deserve mention. first, somewhat contrary to expectation, sentence distance does not seem to affect resolution difficulty, as the ras of the @xmath16. this may have to do with the fact that these ras are only computed over difficult resolution classes. second, on kbp, we see that the ras of the @xmath17. these results seem to be consistent with our intuition that an event mention is easier to resolve if it shares common arguments with its antecedent. interestingly, the opposite seems to be true on ace. this may have to do with the fact that the number of gold event mentions in the ace test set is much smaller than that in the kbp test (403 vs. 4174). in particular, the number of anaphors covered by the six classes involving sd and ca in ace is probably too small to draw reliable conclusions. the highest ras are achieved by the two nonanaphoric resolution classes, sna and una. for non-anaphoric event mentions, a high ra implies that the corresponding resolver has successfully determined that these mentions should not be resolved. in addition, the td f-scores associated with these two classes again confirm that td performance correlates with the percentage of mentions that are seen in the training set. we divide the different kinds of perturbations into two broad categories, mention-internal perturbations and mention-external perturbations. ## using oracles how can the performance of our best resolver (joint knowledge-rich spanbert-l) be improved? to answer this question, we perform oracle experiments in which we feed the resolver with different types of perfect information and examine how much performance gains can be obtained. ## conclusion while space limitations preclude a reiteration of all the observations we made in our empirical analysis of our resolver, we believe the key conclusions are: (1) a knowledge-rich joint event cr model trained using spanbert-l achieves better performance than other model variants; (2) resolving anaphoric event mentions that cannot be resolved using lemma matching and memorization of mention pairs seen in the training set remains a challenge in event cr; (3) while our state-of-the-art span-based resolver is sensitive to all kinds of perturbations we considered, there is not one perturbation it is particularly sensitive to; and (4) improving mention boundary detection, anaphoricity detection, and subtype detection will likely lead to the largest improvement in event cr performance. veselin stoyanov, nathan gilbert, claire cardie, and ellen riloff. 2009. conundrums in noun phrase coreference resolution: making sense of the stateof-the-art. in proceedings of the joint conference of the 47th annual meeting of the acl and the 4th international joint conference on natural language processing of the afnlp, pages 656-664, suntec, singapore. association for computational linguistics. marc vilain, john burger, john aberdeen, dennis connolly, and lynette hirschman. 1995. a modeltheoretic coreference scoring scheme. in sixth message understanding conference (muc-6): proceedings of a conference held in columbia, maryland, november 6-8, 1995. david wadden, ulme wennberg, yi luan, and hannaneh hajishirzi. 2019. entity, relation, and event extraction with contextualized span representations. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 5784-5789, hong kong, china. association for computational linguistics. bishan yang, claire cardie, and peter frazier. 2015. a hierarchical distance-dependent bayesian model for event coreference resolution. transactions of the association for computational linguistics, 3:517-528.