Okay, Let’s Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation (2024)

Abstract: In NLP, Event Coreference Resolution (ECR) is the task of connecting event clusters that refer to the same underlying real-life event, usually via neural systems. In this work, we investigate using abductive free-text rationales (FTRs) generated by modern autoregressive LLMs as distant supervision of smaller student models for cross-document coreference (CDCR) of events. We implement novel rationale-oriented event clustering and knowledge distillation methods for event coreference scoring that leverage enriched information from the FTRs for improved CDCR without additional annotation or expensive document clustering. Our model using coreference-specific knowledge distillation achieves SOTA B^3 F_1 on the ECB+ and GVC corpora and we establish a new baseline on the AIDA Phase 1 corpus. Our code can be found at https://github.com/csu-signal/llama_cdcr.

## introduction event coreference resolution (ecr) is the task of connecting mentions that refer to the same underlying real-life event. since descriptions of similar events often use similar words in similar context, systems can achieve strong baseline performance simply by comparing the lemmas of the event triggers @xcite . however, most ecr datasets contain many event pairs that might be coreferent despite different lemmas, or non-coreferent despite similar lemmas or tokens. consider this example: "video of brooklyn woman's fatal shooting is played at trial of two men charged in rooftop gunplay." the lexical similarity in the event triggers is misleading since they are actually not coreferent. typical heuristic-based systems fail in such cases if the decision is made at the event pair level. intuitively, a human might solve a challenging coreference problem by engaging in a stepby-step "inner monologue" that reasons about the context, participants, actions, locations, etc., in both events @xcite . recent works like instruct-gpt @xcite have shown that generative large language models (llms) can engage in chain-of-thought reasoning, a kind of step-by-step reasoning that appears human-like. such models have also demonstrated abductive reasoning capabilities about relations between events @xcite and zero-shot resolution abilities in various coreference benchmarks like conll-2012 and ecb+ @xcite . in this paper, we seek to model such an innermonologue or a step-by-step reasoning process in an event coreference system. we augment existing coreference corpora using a novel instruction-based zero-shot prompting framework @xcite that guides a generative llm to produce outputs displaying abductive reasoning about the coreference samples therein. these intermediate reasoning steps, consolidated into free-text rationales (ftrs) for the coreference labels of the mention pairs, are then used to guide a two-stage modeling procedure. we first perform "rationale-oriented event clustering (roec)" by directly optimizing a "student" model to encode cluster-level information in the coreference graph. since the generated rationales comes from a disparate model distribution, we simultaneously align event pairs with their corresponding ftrs in the student model's latent space. we then use the optimized student distribution as our backbone encoder, which learns coreference probabilities of event pairs by jointly optimizing the task-supervision component with additional supervision from the generative llm "teacher" distribution using the rationales as soft labels. our novel contributions are: rationales provide an additional soft supervisory signal through which we train the student model with additional information about contextual cues for event coreference, but are not required at inference, increasing our approach's generalizability. we evaluate our method on three event cdcr corpora: event coreference bank plus (ecb+), the gun violence corpus (gvc), and aida phase 1. our method achieves state-of-theart b 3 score on ecb+ and gvc, and sets a novel performance baseline on aida phase 1, without a document clustering step as used in many other methods. we perform detailed ablations of individual components of our method and evaluate how each one contributes to the final performance. our code, weights, and ftr sets are available at https: //github.com/csu-signal/llama_cdcr . ## related work event cdcr most previous works in cdcr address the challenge of pairing across documents with a document clustering step that reduces the search space for potential candidates and ensures tractability of pairwise computations @xcite @xcite @xcite . however, preprocessing with document clustering misses a non-trivial amount of coreferring pairs between clusters @xcite . @xcite also demonstrate that this tends to overfit cdcr systems to corpora like ecb+ @xcite , that have unrealistic lexical distinction between topics, thus reducing systems' generalizability to corpora with more referential ambiguity (e.g., gvc; vossen et al. (2018) while @xcite also suggest using knowledge distillation techniques @xcite to enhance pairwise computations even further, using a heuristic allows us to approach knowledge distillation using a single encoder to model both cluster-level and pairwise signals without using separate encoders for candidate retrieval and candidate scoring. in line with this, we use ahmed et al. (2023)'s heuristic-based approach for candidate generation during training and inference. abductive reasoning and free-text rationales previous work on abductive reasoning in coreference resolution has used rule-based and entityspecific approaches, with causal relations between events only appearing as supporting evidence for the coreference decision @xcite @xcite . zhao et al. (2023b) and bhagavatula et al. (2019) research in cognitive psychology @xcite suggests that a welldeveloped "inner monologue" is crucial for different aspects of problem-solving, often as a "working memory" or a cognitive rehearsal tool @xcite . @xcite use this strategy for temporal reasoning in gpt-3-generated ftrs to improve coreference computations. however, due to the black-box nature of llms like gpt-3, their method assumes the ftrs themselves as the complete knowledge, with no access to the llm's internal distribution. in contrast, our research uses an open-weight llm both to generate step-by-step ftrs with reasoning about coreference, and so that we can access the underlying model distribution. ## method our method for event coreference resolution consists of three parts: 1) using curated zero-shot instructions specifying the ecr task, we gener- teacher (llama 2) student (longformer) l task l attention l pooled inner-monologue (im) -rationales ... two men charged in @xmath2. ... shooting @xmath3.... rationale-oriented event clustering (roec) event mention representations rationale representations ...the first sentence mentions the fatal shooting video being played at trial, while the second sentence mentions the playing action itself ... c b a d f e g h i r r r 0.9 a b 0.8 d e 0.3 h c 0.4 g e coreference linking probabilities l kd ... upset for everybody involved in @xmath4... ... what actually @xmath5, said assistant police chief. knowledge-enhanced student l cluster l rationale inference inner-monologue based knowledge distillation figure 1: schematic system overview: step-by-step ftrs resembling an "inner-monologue" are generated using an llm (teacher model) conditioned on the gold coreference label. ftrs are then clustered along with event pairs to optimize the student's latent space (roec). the optimized student learns further coreference-specific contextual cues in the rationales from the teacher's latent space. arrows show the gradient flow during training from the teacher (blue) and student (green) during the roec (dashed) and knowledge distillation (dotted) phases, respectively. solid black line indicates inference samples, which include no rationale text or signal from the teacher model. letters a-i in the roec block represent distinct event mentions, and the colors represent an event cluster (such that all the blue circles cluster together). "r" represents a set of rationales that justify the linking of different mentions in a single cluster. ## results to demonstrate that a generative llm alone does not perform equivalently, we compare to zeroshot results from llama 2-7b-chat (our teacher model) and gpt-3.5-turbo. these models were prompted to provide a single-word (yes/no) answer to whether or not the events given in the prompt are coreferent. prompt format is a slight variant of that given for rationale generation in fig. 2 . see appendix d for more details. our best model, using both roec and knowledge distillation, outperforms previous baselines on both ecb+ (+1.1 b 3 f 1 ) and gvc (+0.6 b 3 f 1 ), as well as zero-shot llama 2-7b-chat and gpt-3.5-turbo. aida phase 1 is a new, challenging dataset, where we establish a new baseline. long +roec,+kd learns from the teacher distribution, but we find it significantly outperforms both the zero-shot teacher model (llama 2-7b-chat) and the much larger gpt-3.5-turbo. without taskspecific finetuning, general purpose models tend to under-perform in reasoning-based tasks compared to smaller fine-tuned variants @xcite , but our results suggest that the teacher-generated rationales contain useful information for coreference decisions, such that when they are conditioned on the gold label, the distilled knowledge optimizes the task-based encoder for better performance, despite such larger models performing poorly in a zero-shot setting @xcite . ## discussion we ran ahmed et al. (2023)'s pipeline over the same data and compared where their method and ours make different decisions on ecb+ and gvc samples. since we use the same preprocessing here, this compares the performance of the two pairwise discriminators used: plain longformer vs. our knowledge-enhanced version. using knowledge distilled from ftrs, we are able to make more correct coreference decisions than plain longformer, even without using the rationale in the input at inference time. this holds true across both ecb+ and gvc test sets. our method also establishes a new baseline for the aida phase 1 corpus. qualitative analysis this shows that in areas like exposing alignment or divergence between named entities, temporal contexts, or syntax across mention pairs, ftrs are providing useful information by making explicit what may be implicit in the raw text. our knowledge distillation procedure is able to take this information from the attention heads and hidden states of the teacher model and project it into the student model for finer-grained coreference decisions. ## conclusion and future work in this paper, we presented a novel event cdcr technique that used free-text rationales from a generative llm to provide additional supervisory signals for coreference. we accomplished this through a combination of clustering rationale representations with the corresponding event mentions in a student model's space, and by distilling information from the generative teacher model into the smaller student. we achieved sota b 3 scores on the ecb+ and gvc benchmarks, and established novel benchmark performance on the challenging aida phase 1 dataset. we also ablated the contributions of different components of our model and examined the kinds of information our model is likely leveraging toward its performance. it would be too strong a claim to say that the generated ftrs consistently display coherent reasoning in and of themselves (for example, sample (b) in our results demonstrate that although imperfect, automatically-generated rationales for event coreference contain useful information toward the decision. using ai-generated rationales as soft-labels might prove useful as a way to decrease annotator workload in cognitively-heavy tasks like annotating coreference resolution corpora @xcite . this opens the way to future work to improve the utility of ftrs. for example, filtering methods may be used to exclude ftrs with lower-quality coreference knowledge. techniques such as @xcite that employ a separate but smaller critic model can be trained on a small sub-sample of high-quality rationales written by trained coreference annotators, to further enhance coreferencespecific knowledge distillation with lower annotation expenses. ftrs generated using a more powerful model like gpt-4 could also be beneficial in extending extant cdcr corpora with more explicit soft-labels and can likely enhance systems that need to detect cross-subtopic coreference in corpora such as fcc @xcite , albeit at the cost of accessibility to the source model. ftrs with validated gold cluster-level information could be leveraged especially in preclustering to reduce cross-computations, making such systems more generalizable.