Consistent Document-level Relation Extraction via Counterfactuals (2024)

Abstract: Many datasets have been developed to train and evaluate document-level relation extraction (RE) models. Most of these are constructed using real-world data. It has been shown that RE models trained on real-world data suffer from factual biases. To evaluate and address this issue, we present CovEReD, a counterfactual data generation approach for document-level relation extraction datasets using entity replacement. We first demonstrate that models trained on factual data exhibit inconsistent behavior: while they accurately extract triples from factual data, they fail to extract the same triples after counterfactual modification. This inconsistency suggests that models trained on factual data rely on spurious signals such as specific entities and external knowledge – rather than on context – to extract triples. We show that by generating document-level counterfactual data with CovEReD and training models on them, consistency is maintained with minimal impact on RE performance. We release our CovEReD pipeline as well as Re-DocRED-CF, a dataset of counterfactual RE documents, to assist in evaluating and addressing inconsistency in document-level RE.

## introduction relation extraction (re) extracts triples, semantic relations between two entities, from text. in document-level re, triples can span multiple sentences @xcite @xcite . re datasets such as docred @xcite and re-docred @xcite consist of a factual corpus (wikipedia) annotated with triples. most recent docre models are based on pretrained language models (plms) @xcite @xcite trained on these datasets. while plms perform strongly, they are susceptible to factual biases and other spurious correlations. to generate triples, instead of @xcite and counterfactual version generated with entity replacement. a model trained on factual data extracts the original triple, but fails on its counterfactual (cf) counterpart. thus, the model is relying on spurious patterns such as entity biases. we address this by generating cf data and training re models on them. ## counterfactual pipeline and dataset to evaluate and address robustness against factuality bias, we need to generate documents from which such biases have been removed. hence, in this section we describe covered, our mechanism for generating counterfactual (cf) documents from a document-level re dataset. our seed dataset is re-docred @xcite . for each re-docred document d, we have a set of entities e and a set of relation triples r. for each entity node e @xmath0, the dataset provides the positioning of each mention of e i and its type (org, time . . . ). in a triple @xmath1, we have the indices (i) of head and tail entities, the relation r t and -if the triple comes from the original docred (yao et al., 2019) -the ids of the sentences that are the evidence for r. to generate counterfactuals from re-docred's documents, our pipeline covered proceeds with the following three steps ( §2.1- §2.3). ## experiments we generate re-docred-cf, our counterfactual dataset, from re-docred using covered. we run covered five times on re-docred train to produce re-docred-cf train (so it consists of five different counterfactual datasets). we run covered on re-docred test once to generate re-docred-cf test. we set τ e[max] = .8, τ e[min] = .2, τ @xmath2.4, m @xmath3, in terms of embedding similarity would be selected. ## conclusion in this work, we present a method for generating counterfactual examples for document-level relation extraction. our approach searches for suitable entity replacements over a document and applies them to a point where most of the relations are affected by these replacements. by generating a counterfactual test set, we demonstrate the high level of inconsistency docre model have when trained only with factual data. adding counterfactuals to the training sets improves consistency by a large margin while keeping performance high. we make our pipeline covered and dataset re-docred-cf publicly available we hope our findings and resources will raise awareness and support future efforts in addressing entity and factual biases in document relation extraction.