Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction (2023)

Abstract: In this paper, we seek to improve the faithfulness of TempRel extraction models from two perspectives. The first perspective is to extract genuinely based on contextual description. To achieve this, we propose to conduct counterfactual analysis to attenuate the effects of two significant types of training biases: the event trigger bias and the frequent label bias. We also add tense information into event representations to explicitly place an emphasis on the contextual description. The second perspective is to provide proper uncertainty estimation and abstain from extraction when no relation is described in the text. By parameterization of Dirichlet Prior over the model-predicted categorical distribution, we improve the model estimates of the correctness likelihood and make TempRel predictions more selective. We also employ temperature scaling to recalibrate the model confidence measure after bias mitigation. Through experimental analysis on MATRES, MATRES-DS, and TDDiscourse, we demonstrate that our model extracts TempRel and timelines more faithfully compared to SOTA methods, especially under distribution shifts.

## introduction event temporal relation @xcite extraction is an essential step towards understanding narrative text, such as stories, novels, news, and guideline articles. with a robust temporal relation extractor, one can easily construct a storyline from text and capture the trend of temporally connected event mentions. temprel extraction is also broadly beneficial to various downstream tasks including clinical narrative processing @xcite , question answering @xcite @xcite , and schema induction @xcite @xcite . most existing temprel extraction models are developed with data-driven machine learning approaches, for which recent studies also incorporate a) i went to e 1 :see the doctor. however, i was more seriously e 2 :sick. =⇒ e 1 after e 2 b) microsoft said it has e 3 :identified three companies for the china program to run through june. the company also e 4 :gives each participating startup in the seattle program $20,000 to create software. =⇒ e 3 before e 4 figure 1 : examples of unfaithful extractions. before and after that follow the arrows denote the extracted temprel's from the sentences by @xcite . though these recent approaches have achieved promising evaluation results on benchmarks, whether they provide faithful extraction is an unexplored problem. the faithfulness of a relation extraction system is not simply about how much accuracy a system can offer. instead, a faithful extractor should concern the validity and reliability of its extraction process. specifically, when there is a temprel to extract, a faithful extractor should genuinely obtain what is described in the context but not give trivial guesses from surface names of events or most frequent labels. besides, when there is no relation described in the context, the system should selectively abstain from prediction. we observe that in recent models, biases from prior knowledge in plms and statistically skewed training data often lead to unfaithful extractions 541 (see fig. 1 ). example a thereof exhibits a case where the model adheres to the prior knowledge where people usually see the doctor after getting sick, but in this context getting sick is obviously a consequent of seeing the doctor. in example b, before is extracted due to statistical biases learned from training data that before is not only the most frequent temprel between identify and give, but is also the most frequent temprel between the first and second event in narrative order @xcite . however, with a closer inspection, it can be noticed that the two events in example b are involved in different programs, one in the china program, the other in the seattle program. therefore, the system should abstain from prediction and give vague as output. in this paper, we seek to improve the faithfulness of temprel extraction models from two perspectives. the first perspective is to guide the model to genuinely extract the described temprel based on a relation-mentioning context. to achieve this goal, we conduct counterfactual analysis @xcite to capture and attenuate the effects of two typical types of training biases: event bias caused by treating event trigger names as shortcuts for temprel prediction, and label bias that causes the model prediction to lean towards more frequent training labels. we also propose to affix tense information to event mentions to explicitly place an emphasis on the contextual description. the second perspective is to teach the model to abstain from extraction when no relation is described in the text. to know when to abstain, the models need to have a good estimate of the correctness likelihood. by incorporating dirichlet prior @xcite in the training phase of current temprel extraction models, we improve the predictive uncertainty estimation of the models and make the temprel predictions more selective. furthermore, since the counterfactual analysis component (from the first perspective) may shift the model-predicted categorical distribution, we also employ temperature scaling @xcite in inference to allow for recalibrated confidence measure of the model. the technical contributions of our work are twofolds. first, to the best of our knowledge, this is the first study on the faithfulness issue of eventcentric information extraction. evidently, the development of a faithful temprel extraction system contributes to more robust and reliable machine comprehension of events and narratives. second, we propose training and inference techniques that can be easily plugged into existing neural tem-prel extractors and effectively improve model faithfulness by mitigating prediction shortcuts and enhancing the capability of selective prediction. our contributions are verified with temprel extraction experiments conducted on matres @xcite , tddiscourse @xcite and distribution-shifted version of matres (matres-ds). particularly, we evaluate on how precise and selective our temprel extraction method is on in-distribution data, and how well it generalizes under distribution shift. experimental results demonstrate that the techniques explored within the two aforementioned perspectives bring about promising results in improving faithfulness of current models. in addition, we also apply our method to the task of timeline construction @xcite , showing that faithful temprel extraction greatly benefits the accurate construction of timelines. event temprel extraction. recent event tem-prel extraction approaches are mainly built on plms to obtain representations of event mentions and are improved with various learning and inference methodologies. to improve the quality of event representations, mathur et al. (2021) embrace rhetorical discourse features and temporal arguments; trong et al. (2022) select optimal context sentences via reinforcement learning to achieve sota performances; while liu et al. (2021b); mathur et al. (2021); zhang et al. (2022) employ graph neural networks to avoid complex feature engineering. from the learning perspective, ning et al. (2018a), ballesteros et al. (2020), and wang et al. (2020) enrich the models with auxiliary training tasks to provide complementary supervision signals, while ning et al. (2018b), zhao et al. (2021) and zhou et al. (2021) bring into play distant supervision from heuristic cues and patterns. nevertheless, recent data-driven models risk amplifying bias by exacerbating biases present in the pretraining and task training data when making predictions @xcite . to rectify the models' biases towards prior knowledge in plms and shortcuts learned from biased training examples, our work proposes several training and inference techniques, seeking to improve the faithfulness of neural temprel extractors as described in §1. bias mitigation in nlp. methods for mitigating prediction biases can be categorized as retraining and inference @xcite . retraining methods address the bias in early stages or at its source. for instance, @xcite masks the entities with special tokens to prevent relation extraction models from learning shortcuts from entity names, whereas several works conduct data augmentation @xcite @xcite or sample reweighting techniques @xcite to reduce biases in training. however, masking would result in the loss of semantic information and performance degradation, and it is costly to manipulate data or find proper unbiased data in temporal reasoning. directly debiasing the training process may also hinder the model generalization on out-of-distribution (ood) data @xcite . therefore, inspired by several recent studies on debiasing text classification or entity-centric information extraction @xcite , our work adopts counterfactual inference to measure and control prediction biases based on automatically generated counterfactual examples. ## methods in this section, we first present how we obtain event representations and categorical distribution y in a local classifier for temprel ( §4.1). then we introduce proposed learning and inference techniques to improve model faithfulness from the perspectives of selective prediction ( §4.2) and prediction bias mitigation ( §4.3), before we combine these two techniques with temperature scaling and introduce the ood detection method in §4.4. ## experiments in this section, we describe the experiments foot_4 on two tasks: temprel extraction and timeline construction. we first introduce the datasets that we adopt or create for evaluation ( §5.1), followed by the evaluation protocols ( §5.2). evaluation results are discussed in §5.3 before we provide a detailed ablation study and case study in §5.4 and §5.5. ## conclusion in this paper, we investigate on improving faithfulness for event temprel extraction from two perspectives. to enhance the selectiveness of model predictions, we parameterize a dirichlet prior over the model-predicted categorical distribution to regularize the model to behave differently with id and ood data. to mitigate two types of biases from plms and training data, we add tense information to obtain robust event representations and conduct a counterfactual analysis to reduce the risk of carrying prediction shortcuts into inference. we also employ temperature scaling to combine the two faithful perspectives, which recalibrates the confidence measure of the model after bias mitigation. through experimental analysis on matres, matres-ds, and tddiscourse, we demonstrate that our model faithfully extracts event temporal relations and timelines from text, so as to generalize well under distribution shifts.