A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution (2024)

Abstract: Based on Pre-trained Language Models (PLMs), event coreference resolution (ECR) systems have demonstrated outstanding performance in clustering coreferential events across documents. However, the state-of-the-art system exhibits an excessive reliance on the ‘triggers lexical matching’ spurious pattern in the input mention pair text. We formalize the decision-making process of the baseline ECR system using a Structural Causal Model (SCM), aiming to identify spurious and causal associations (i.e., rationales) within the ECR task. Leveraging the debiasing capability of counterfactual data augmentation, we develop a rationale-centric counterfactual data augmentation method with LLM-in-the-loop. This method is specialized for pairwise input in the ECR system, where we conduct direct interventions on triggers and context to mitigate the spurious association while emphasizing the causation. Our approach achieves state-of-the-art performance on three popular cross-document ECR benchmarks and demonstrates robustness in out-of-domain scenarios.

## introduction the goal of cross-document event coreference resolution (ecr) is to group event mentions referring to the same real-world event together across documents. it is an essential task in nlp and has provided valuable prior event-related knowledge for many downstream tasks, e.g., topic detection and tracking @xcite , multi-hop question answering @xcite and information extraction @xcite . in real life, event coreference systems commonly assist decision-makers in important fields such as intelligence analysis and security event warnings @xcite . to resolve the task, existing state-of-the-art ecr systems perform binary classification to pairwise compare event mentions @xcite @xcite . in their pipelines, a fraction of coreferential and non-coreferential mention pairs are retrieved from the corpus to fine-tune a cross-encoder, which is used as a coreference scorer to gauge the likelihood of pairwise events being coreferential. finally, coreferential mentions are merged into clusters based on the predicted coreference score. however, most coreference scorers are troubled by the curse of 'triggers lexical matching', which is also discussed in previous works @xcite . the histogram in figure 1 demonstrates that when constructing eventmention pairs, it is natural that coreferential men-tions frequently share lexically similar foot_0 event triggers, whereas non-coreferential mentions typically have lexically divergent ones. this skewed feature distribution brings a bias where lexically similar trigger words often correspond to coreference, especially for trigger-centric ecr systems @xcite using the event representation in appx sec a. 2.3. however, what truly determines the ecr outcome of event mentions is the coreference of eventrelevant arguments, which include (non-)human participants, times, locations, and actions (i.e., event triggers) @xcite . in other words, these deeper semantic features constitute the rationales of the ecr task, as they demonstrate the task's corresponding causal associations. unfortunately, the example in figure 1 implies that the state-of-the-art system @xcite relies too much on the surface feature of trigger term similarity but ignores contextual semantics. to resolve this issue, we think about adjusting the distribution of key features in the training data through data governance. counterfactual data augmentation (da) is a promising way for debiasing the classification system @xcite , which enhances the robust causal thinking ability of models with a human-like logic: 'what the output label would be if certain phrases within the input text were altered?'. in practice, we can intervene with rationales in the original example input text to ensure minimal editing to flip the output label, thus generating counterfactual augmented data (cad). the minimal editing constraint is to prevent the introduction of unnecessary noise into the augmented data, which allows the model trained with cad to focus directly on the causal associations from rationales, rather than on other parts of the input text @xcite . given this, we propose llm-rcda, a rationalecentric counterfactual da method with a large language model (llm) in the loop, aiming to enhance the model to think causally and understand deeper semantics in the pairwise context. as shown in figure 2 , our method focuses on intervening triggers and rationales in the event-mention sentence. in the phase of trigger intervention, lexically divergent synonyms of the original trigger are generated to force the system to capture the coreferential meaning between triggers, while in the phase of context intervention, we use the llm to merely change the rationales of the target event-mention based on prompts, and keep the discourse of the event-mention unchanged. to evaluate the efficacy of our method, we evaluate our method on three popular crossdocument ecr benchmarks: ecb+ @xcite , football coreference corpus (fcc) @xcite and gun violence corpus (gvc) @xcite . our enhanced system achieves state-of-the-art performance on all of them, with improvements varying from 1.8 to 2.3 conll f1 over baselines. on ecb+, our approach significantly surpasses the performance of directly employing llms, showcasing its superiority to the current llm-qa paradigm in the task. additionally, the cross-corpus experiment on the out-of-the-domain data shows a robustness improvement of our method, with a 7.2 conll f1 gain over the baseline. to the best of our knowledge, we are the first to evaluate and analyze the performance of popular llms on the cross-document ecr benchmark, and the first to formalize the decision process of the mainstream ecr model from a causal view. moreover, we are the first to utilize rationale-centric cad generated by the llm to causally enhance the ecr system foot_1 . ## related work event coreference resolution currently, the pretrained language models @xcite have significantly enhanced the contextual semantics of text data. in recent ecr systems @xcite @xcite , the pairwise representation of events (known as crossencoding) becomes mainstream. such representation combines contextual embeddings with pairwise token-level trigger embeddings to represent the event-mention pair and expects the embeddings to encode the event-relevant argument information implicitly. some other works enhanced the pairwise events representation by explicitly fusing the encoding of event-relevant arguments which are extracted by semantic role labeling (srl) systems @xcite @xcite , achieving success in performance improvements. in our approach, we also emphasize the crucial features that influence event coreference, such as argument features. however, we do not alter the model structure or existing representation methods. instead, we induce the model to learn these key features through rationale-centric counterfactual data augmentation, thereby enhancing the causal reasoning capability of the ecr system. our work is the first one specifically designed for the ecr task, which also involves the llm-inthe-loop to automatically and efficiently construct the required cad that meets the task's causal requirements. ## method we analyze the decision process of the ecr system on event coreference with structural causal model (scm) @xcite . formally, the event coreference process of the baseline ecr system into the following equation: ## experimental results llms methods conll f1 lemma matching (barhom et al., 2019) 76.5 e2e neural system (cattan et al., 2020) 81.0 pipeline system (held et al., 2021) 85.7 causally enhanced system (ours) 86.0 claude-2 56.9↓ gpt-4 70↓ in the case of ecb+, we observe a significant improvement in recall for our enhanced system compared to the baseline system, as measured by muc, b 3 , and lea, with an average improvement of 3.5 points. this improvement can be attributed to the trigger intervention in algo. 1. the introduction of diverse trigger expressions enhances the model's comprehension of event semantics, thereby rectifying false negatives caused solely by literal differences in trigger terms. the case study is presented in appx. sec a.6.2. fcc and gvc represent single-domain datasets focused on football and gun violence news. they include a substantial volume of coreferential event mentions across various topics, resulting in a con- methods muc b 3 ceafe lea conll r p f1 r p f1 r p f1 r p f1 f1 ecb+ barhom et al. (2019) 77.6 84.5 80.9 76.1 85.1 80.3 81.0 73.8 77.3 ---79.5 cattan et al. (2020) 85.1 81.9 83.5 82.1 82.7 82.4 75.2 78.9 77.0 ---81.0 bugert et al. (2021) 76.0 76 .1 76.1 71.8 81.2 76.2 72.2 72.1 72.2 55.1 67.9 60.8 74.8 caciularu et al. (2021) 87.1 89.2 88.1 84.9 87.9 86.4 83.3 81.2 82.2 76.7 77.2 76 we first discuss the necessity of ti. the absence of ti in ori&tia leads to approximately 92% of coreferential training data involving lexically similar trigger pairs (22% higher than that of ori), which inevitably exacerbates the 'triggers lexical matching' bias. consequently, ori&tia experiences a performance decline of 0.3 conll f1 compared to ori, despite increasing the training data volume by threefold through the addition of plausible counterfactuals. however, with the help of ti, ori&cad shows greater lexical variation in the literal surface forms of trigger terms, with only 19.9% (v.s., 91.4% for ori&tia and 70.0% for ori) of the coreferential data containing lexically similar trigger pairs. this data combination leads to a substantial performance improvement in the ecr system, surpassing ori by 1.8 conll f1 and ori&tia by 2.1 conll f1, which indicates that furthermore, the performance comparison between ori&cia and ori&cad demonstrates the necessity of utilizing ci. due to the relaxation of the minimum edit distance constraint, cia becomes the implausible counterfactual data, with a moverscore of 0.6928 which is lower than 0.7339 of cad. the performance of ori&cia falls behind that of ori&cad by 1.0 conll f1. this implies that ci ablation weakens the model's ability to capture and understand rationales, resulting in only sub-optimal performance. the ablation study highlights the importance of both ti and ci and validates our analysis of spurious and causal associations. therefore, the most effective way to enhance cross-document ecr performance is by fully utilizing our llm-rcda algorithm. comparison with temporal commonsense da ravi et al. enriched the event context by introducing possible preceding or succeeding scenarios related to event mentions based on the temporal commonsense event coreference data augmentation (tcda) method, thereby increasing the distinction between events. they also designed an inference-enhanced pairwise scorer specifically to capture such temporal information. we leverage their prompt to generate the temporal commonsense augmented data (tad) and then incorporate them into the original dataset (ori) to train our baseline ecr system @xcite (appx. algo. 4). results in from figure 4 , we observe that 50% of fps fall under the category of 'ignore argument counterparts', which represents simple cases for the ecr system. in these samples, the coreference of eventrelevant arguments (i.e., rationale) is clearly present within the sentences where the event mention oc-curs. for example: among fns, 'ignore argument counterparts' remains an important component, accounting for 24%. unlike the same error type in fps, the baseline system fails to capture coreferential evidence from argument counterparts in the pairwise context. besides, 'require contextual understanding' takes 32% of resolved fns. such errors pose greater challenges, as their pairwise context exhibits significant syntactic structural differences. while contexts provide ample information about event-relevant arguments, accurately identifying and understanding the relationships between these arguments is not easy for the baseline system. for example: to give a correct answer, the ecr system needs to infer, based on the context, that the antecedent 'who' in the first text snippet refers to 'belgium', and 'the red devils' in the second text snippet refers to 'belgium' as well. then, it needs to link the coreferential relationship between counterpart arguments to conclude that these two mentions corefer to the event 'belgium lost to france'. examples and analyses of other error types can be found in appx. sec a.6.3. the error analysis reveals a deficiency in the baseline system's comprehension of complete semantics. in contrast, our enhanced ecr system effectively addresses these errors. this is attributed to llm-rcda, which enables a rationale-centric decision in solving ecr by enhancing the system's ability to identify and understand the evidence of event-relevant arguments within the pairwise context, which is the key to prediction. ## conclusion we proposed a novel rationale-centric counterfactual data augmentation method specialized for the pairwise text input of cross-document ecr system, which leverages interventions from llm to enhance the system's event coreference decision causally. experimental results verify the significant performance and robustness improvement of the enhanced ecr system with our method.