Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning (2024)

Abstract: Event temporal reasoning aims at identifying the temporal relations between two or more events from narratives. However, knowledge conflicts arise when there is a mismatch between the actual temporal relations of events in the context and the prior knowledge or biases learned by the model. In this paper, we propose to detect knowledge-conflict examples in event temporal reasoning using bias indicators, which include event relation prior bias, tense bias, narrative bias, and dependency bias. We define conflict examples as those where event relations are opposite to biased or prior relations. To mitigate event-related knowledge conflicts, we introduce a Counterfactual Data Augmentation (CDA) based method that can be applied to both Pre-trained Language Models (PLMs) and Large Language Models (LLMs) either as additional training data or demonstrations for In- Context Learning. Experiments suggest both PLMs and LLMs suffer from knowledge conflicts in event temporal reasoning, and CDA has the potential for reducing hallucination and improving model performance.

## introduction an important goal of event understanding is to identify the temporal relations (temprels) among events described in natural language text @xcite . this task aligns with human's cognitive ability @xcite , which often involves routinely reasoning about how events happening around us are temporally sequenced, planned, and lead to consequences and decisions @xcite . from the intelligent system perspective, it also benefits many nlp applications for narrative understanding @xcite , schema induction @xcite , and question answering @xcite ). 1 code and data are available at https://github.com/ tqfang/event-temporal-knowledge-conflict corpus statistics: count(see, before, sick): 3 count(see, after, sick): 8 see the doctor happens [mask] sick. before: 5.8×10 !" after: 7.0×10 !" [mask] the actual temprel in the context differs from the biased or prior temprel in the corpus and the language model, leading to the emergence of knowledge conflicts. in event temporal reasoning, the input includes two parts, the event mentions and the context. the temprel a model seeks to infer should be based on the context, rather than only revealed by the event mentions themselves. for example, in fig. 1 , without a context, the event mention see (the doctor) and sick have certain temporal prior where see the doctor statistically happen more often after sick, either by corpus statistics or probing a masked plm. however, under the context of "i went to see the doctor, however, i was more seriously sick," we can infer that see happens before sick instead of after due to the presence of the connective however. this is known as the phenomenon of knowledge conflicts @xcite , where the contextual information contradicts the knowledge memorized by the language model. hence, the essential requirement for accountable temporal reasoning is context-faithfulness @xcite , where models are expected to perform reasoning based on the context instead of guessing using only the prior knowledge about the events encoded in their parameters. however, most current language models, including both pre-trained language models (plms) and large language models (llms) foot_0 , rely on shortcuts from the mentions without being faithful to the context @xcite to varying degrees, leading to hallucination. this issue is particularly severe in contexts where event or entity mentions have a different relation prior than what is presented in the context. though entityrelated knowledge conflicts @xcite @xcite have recently attracted much attention, questions about eventrelated knowledge conflicts remained intact. first, it is necessary to understand the conflicts regarding relations of events, which is more complicated than that of a single event. second, the substitution-based paradigm defined in entity knowledge conflicts or spurious correlation detection @xcite cannot be directly applied to events. entity mentions can often be replaced randomly with other entities with the same typing to study the faithfulness towards the context other than the entity mention, which remains unchanged after the replacement. for example, in open-domain qa, a possible question can be "who is the ceo of twitter?" based on the context "yaccarino succeeded elon musk as the ceo of twitter". to check whether models faithfully rely on the context instead of hallucinating, yaccarino in the context can be changed to a random name to see if the model can still output the "correct" ceo instead of yaccarino as they have learned in pretraining. however, events are usually denoted by predicates in the context @xcite , and directly substituting the predicate (e.g., from see in fig. 1 to another random verb such as play) will alter the semantic meaning of the whole context, including both the predicate and its dependency with the arguments, making it infeasible to analyze the faithfulness towards the original context. thus, instead of resorting to a substitution, in this paper, we study the effect of knowledge conflicts in event temporal reasoning by selecting conflict examples from the original dataset based on corpus statistics, and evaluate models on the conflict subsets. we outline the contributions of this paper as follows. first, we define four types of bias that can lead to knowledge conflicts, including eventrelation bias, narrative bias, tense bias, and dependency bias. the data instances where the actual temprel contradicts with the prior temprel are referred to as knowledge-conflict instances ( §3), as they conflict with the prior knowledge provided to language models. second, to mitigate the effect of knowledge conflicts, we propose a counterfactual data augmentation (cda) technique that explicitly generates contexts with knowledge-conflict elements, thereby reducing the overall bias in the data distribution. cda can be applied to both fine-tuned plms and llms with (test-time) incontext learning ( §3.3). third, we study the effect of various kinds of knowledge conflicts and our proposed bias mitigation method on two popular event temporal reasoning benchmarks, torque @xcite and matres @xcite . we show that models suffer from performance drop on knowledge-conflict subsets, and our bias-aware data augmentation method outperforms baselines by a remarkable margin on both bias mitigation and overall performance ( §4). ## event knowledge conflict we introduce the problem definition ( §3.1) and formally define four types of bias and how to select knowledge-conflict data ( §3.2). we then introduce our proposed counterfactual data augmentation for mitigating knowledge conflict ( §3.3). ## experiments in this section, we introduce the datasets ( §4.1), the settings of knowledge conflict diagnosis ( §4.2), and conflict mitigation ( §4.3), the primary experimental results and analysis ( §4.4). ## conclusion in this paper, we investigate knowledge conflicts in event temporal reasoning by formally defining four types of biases to identify a knowledge conflict diagnoses evaluation set. we observe that both plms and llms are susceptible to knowledge conflicts in this task, resulting in decreased performance on knowledge-conflict datasets. to address this issue, we propose a cda method that is suitable for both plms through pre-training and llms through in-context learning. our experiments demonstrate the effectiveness of our proposed method in mitigating knowledge conflicts.