Improving Distantly Supervised Document-Level Relation Extraction Through Natural Language Inference (2022)

Abstract: The distant supervision (DS) paradigm has been widely used for relation extraction (RE) to alleviate the need for expensive annotations. However, it suffers from noisy labels, which leads to worse performance than models trained on human-annotated data, even when trained using hundreds of times more data. We present a systematic study on the use of natural language inference (NLI) to improve distantly supervised document-level RE. We apply NLI in three scenarios: (i) as a filter for denoising DS labels, (ii) as a filter for model prediction, and (iii) as a standalone RE model. Our results show that NLI filtering consistently improves performance, reducing the performance gap with a model trained on human-annotated data by 2.3 F1.

## introduction relation extraction (re) is the task of identifying relations between two entities in natural language text. it has an important role in many nlp applications, such as knowledge base population and question answering. existing work on re has been focused mostly on extraction within a sentence @xcite @xcite . however, sentence-level re has one major limitation: it is not designed to extract relational facts expressed in multiple sentences. 1 to address this, recent work has explored models which use document-level context to extract both intra-and inter-sentence relations from text @xcite @xcite currently, high-performance re models require large-scale human-annotated data, which is expensive and does not scale to a large number of relations or new domains. to reduce the reliance on human-annotated data, @xcite introduce the distant supervision (ds) approach, which assumes that if two entities are connected through a relation in a knowledge base, sentences that mention the two entities express that relation. while this assumption allows the creation of large-scale training data without expensive human annotation, it also produces many noisy labels @xcite . 2 as a result, the performance of models trained on ds datasets is considerably lower (∼5%) than models trained on human-annotated datasets. this paper aims to reduce the performance gap between models trained on ds versus annotated data through natural language inference (nli). nli, also known as textual entailment, is the task of determining whether a premise entails a hypothesis. recently, @xcite used an nli model as a standalone re model and demonstrated its effectiveness for zero-shot and few-shot sentence-level re. in line with their work, we investigate if nli can also benefit document-level re in this paper. specifically, we apply nli for document-level re in three scenarios: (i) as a filter for denoising ds labels, (ii) as a filter for model prediction, and (iii) as a standalone re model. we experiment with docred @xcite , the largest document-level re dataset to date. it consists of both ds and human-annotated datasets, which is ideal for our study. across all scenarios, we find that nli is especially effective when it is used as a filter; we observe improvement up to 2.3 f1, reducing the gap with a model trained on annotated data from 5.3 to 3.0 f1. however, the gains are less significant when the model has access to human-annotated data. finally, we highlight the importance of having high-quality entity type information when using nli as a standalone re model. ## nli for re we first describe the approach by @xcite , which uses an nli model as a standalone model for sentence-level re. let p be an input text containing two entity mentions m 1 and m 2 . we take p as the premise and generate the hypothesis h by verbalizing each relation r using a template t, m 1 , and m 2 . for example, the relation "capital of " can be verbalized using the template "{m 1 } is the capital of {m 2 }". one relation can be verbalized using multiple templates, leading to multiple hypotheses. to avoid mismatch between the entity types and the relation, a set of allowed types for the first and the second entities is created for each relation, e.g., the relation "date of birth" should involve a person and a date entities. we use a function f r to determine whether a relation @xmath0, e 1 and e 2 : adapting to document-level re for our experiments with document-level re, we adapt the same setup as @xcite by treating the whole document context as the premise. we apply nli in three scenarios: (i) as a filter to for denoising ds labels (pre-filter), (ii) as a filter for model predictions (post-filter), and (iii) as a standalone re model. in the pre-filtering scenario, we verbalize the labels (relations) identified using the ds assumption and remove all labels that do not surpass the threshold t from the ds dataset. similarly, in the post-filtering scenario, we verbalize the relations predicted by an re model and remove those which do not surpass t . in both scenarios, we do not need to generate candidate relations (eq. 1) since they are provided by the ds labels or the re model predictions. unlike @xcite which chooses one relation label that maximizes the probability of the hypothesis (eq. 3), we use all relation labels that have entailment probability above t . 3 in our experiments, we set @xmath1.5, i.e., taking all relations that the nli model predicts as entailment. additionally, since the ds dataset is known to be noisy, for the pre-filtering scenario, we also experiment with higher thresholds to study the effect of using more strict filters on the re performance. we experiment with two types of nli models: a model that is not trained specifically for re (zeroshot nli) and a model that is fine-tuned using a small number of human-annotated re examples (few-shot nli). the zero-shot nli model simulates a case when we do not have any annotations, while the few-shot nli model simulates a case when we have a small budget for annotations. we fine-tune the nli model for a binary entailment task (entail or not entail). since docred annotations do not contain negative examples (no-relation label), we generate the non-entail examples for nli as follows. first, we train a model using the ds dataset and generate predictions for the human-annotated training data. we then use the model's incorrect predictions as the non-entail examples. we use a maximum @xmath2, 100} examples per relation. for the relation classifier module, we use the multiinstance version, which predicts relation on the mention-level. jerex is originally designed for end-to-end re without the need for entity information. however, since our main focus is on the re side, we use its standard re pipeline, which assumes that entity clusters are given. ## results and analysis zero-shot nli we find that nli improves re performance in both pre-filter and post-filter scenarios. postfiltering with nli achieves the best performance with 56.2 f1, reducing the gap with the supervised model by 2.3 f1. looking into the other metrics, it is evident that nli filtering yields re models with higher precision but lower recall. we observe that our most aggressive pre-filtering (high) outperforms the precision of the supervised model. this result suggests that pre-filtering is especially useful for applications where having high precision is preferable to recall. we also experiment with the double-filter scenario, where we apply both our best pre-filter (low) and post-filter. we find it has minimal effect on the model performance. few-shot nli this scenario assumes that a small human-annotated dataset is available, so in the next set of experiments, all re models are trained using the ds dataset and then fine-tuned using the small annotated dataset. 7 unlike nli fine-tuning, where we limit the maximum number of examples per relation when fine-tuning the re models, we use all annotations in the document since we want the model to learn all and not just the subset of correct triples. we fine-tune the re models using 427 and 1,761 annotated documents for the 10-shot and the 100-shot nli settings, respectively. as shown in ## conclusion in this paper, we presented a systematic study on the use of nli for distantly supervised documentlevel re, focusing on the case when humanannotated data is not available. our results demonstrate that nli is most effective when used as a prefilter to denoise ds labels. in the absence of human annotations, we show that nli filtering reduces the gap with a model trained on human-annotated data by 2.3 f1. we also show that nli filtering still benefits the re model (+1.1 f1) when we have small human-annotated data. our experiment on using nli as a standalone model for document-level re leads to worse performance than using it as a prefilter, suggesting that using nli directly as an re model for document-level is more challenging than sentence-level re. for future work, we plan to explore other strategies to better leverage the entity type information for re with nli and investigate if document-level nli is also more challenging than sentence-level nli. another potential direction is to experiment with other ds techniques, such as integrating a denoising module to the re model @xcite or using ds-trained models as a ds filter @xcite .