Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models (2023)

Abstract: Document-level Relation Extraction (DocRE), which aims to extract relations from a long context, is a critical challenge in achieving fine-grained structural comprehension and generating interpretable document representations. Inspired by recent advances in in-context learning capabilities emergent from large language models (LLMs), such as ChatGPT, we aim to design an automated annotation method for DocRE with minimum human effort. Unfortunately, vanilla in-context learning is infeasible for DocRE due to the plenty of predefined fine-grained relation types and the uncontrolled generations of LLMs. To tackle this issue, we propose a method integrating an LLM and a natural language inference (NLI) module to generate relation triples, thereby augmenting document-level relation datasets. We demonstrate the effectiveness of our approach by introducing an enhanced dataset known as DocGNRE, which excels in re-annotating numerous long-tail relation types. We are confident that our method holds the potential for broader applications in domain-specific relation type definitions and offers tangible benefits in advancing generalized language semantic comprehension.

## introduction document-level relation extraction (docre) is a task that focuses on extracting fine-grained relations between entity pairs within a lengthy context @xcite @xcite @xcite . the abundance of entity pairs in a document, coupled with a vast array of fine-grained relation types, makes docre inherently more challenging than sentence-level re. the challenge is observed not only in model learning but also in human annotations. the original document-level re dataset do-cred @xcite has been recognized for its false negative issue and subsequently re-annotated to address this concern by supplementing a significant number of relation triples. notably, two representative works, @xcite and @xcite , have contributed to this re-annotation process. @xcite undertook manual annotation from scratch, employing two expert annotators to annotate 96 documents. on the other hand, @xcite utilized pre-trained re models in conjunction with manual revision to construct re-docred. despite their contributions to supplementing relations for docred, both methods have certain limitations. first, achieving complete manual annotation is challenging: each document within this dataset contains an average of 19.5 entities, requiring consideration of approximately 37,000 candidate triples (including 97 relation types, including null). second, the supplementary annotations are derived from the existing data distribution: @xcite first pre-trained a re model with distantly supervised data from docred, then utilized this model to predict triple candidates. such a process may introduce model bias, potentially resulting in the exclusion of sparse relations that exist beyond the scope of the existing data distribution. figure 1 illustrates the counts of some relation types across various datasets. it is evident that the supplementary annotations in re-docred exhibit a distribution similar to that of the docred test set. in light of the limitations inherent in prior annotation techniques and the imperative to enhance the completeness of docre datasets, we propose a novel approach aimed at augmenting the re-docred dataset through the utilization of the powerful generalization capabilities of large language models (llms). by leveraging our method, as shown in figure 1 , our revised test set, named docgnre, exhibits the advantage of re-annotating a greater number of long-tail types, such as p276 and p551. recent studies have utilized gpt @xcite for various structural prediction tasks, such as named entity prediction and relation extraction @xcite @xcite @xcite , as well as text classification labeling @xcite . notably, researchers such as @xcite ; @xcite have demonstrated the effectiveness of in-context learning by incorporating prompts containing suitable example demonstrations for re tasks. however, it is worth noting that without explicit instructions, gpt may generate uncontrolled relations that do not align with predefined types. therefore, recent methods only work in sentencelevel re and especially highlight one distinguished challenge for llm in-context learning: unable to fitting detailed instructions for long-context documents @xcite . to align gpt-generated relations and predefined relation types, we first combine natural language inference (nli) models (maccartney, 2009) with gpt to solve zero-shot docre. the results show that although gpt generations only hit partial ground truth of re-docred, it detects substantial external valid relation triples (details in sec. 3.1). therefore, we design a pipeline framework to further complement the test set of re-docred and automatically generate distant training annotations by combining gpt and nli modules. to verify that we supplement many relation triples beyond the scope of the original data distribution, we test previous models in our docgnre test set. additionally, we train the state-of-the-art (sota) model using our distant training dataset. our contributions can be summarized as follows: £ we conduct a quantitative analysis to evaluate the performance of gpt in zero-shot documentlevel re. £ we propose a novel framework that integrates an nli module and an llm to automatically generate distant relation triples, supplementing existing docre datasets. £ we create an enhanced version of the re-docred test set, named docgnre, with minimal human intervention, ensuring high quality. additionally, we augment the re-docred training set by supplementing it with distant relation triples automatically generated by our framework foot_0 , referring to tabel 1. ## llm enhanced automatic data generation our approach consists of two main procedures: constructing prompts for llm to generate relations triples as proposals and employing natural language inference (nli) models to align generated relations with predefined relation types. fig. 2 shows the whole framework. in the first procedure, we observed that even though we imposed restrictions on the entity and relation lists in the prompts, llms (both gpt-3.5 and gpt-4) still generated triples that fell outside of our intended constraints. furthermore, we found that the generated relations expressed by contextual words in the document were more accurate than those with the restricted relation types. based on these insights and to fully leverage the potential of llms, hereby generating more accurate relation triples, we removed the restrictions of specific relation types for llms. instead, we subsequently utilize an nli module to map the generated relations to the predefined relation types in the re-docred dataset. ## conclusion llms face challenges in extracting fine-grained relations within lengthy contexts. to address this limitation, we present a novel framework that integrates an nli module in this work. with our framework, we improve the performance of gpt in zero-shot document-level re. above all, our framework enhances the automatic data generation capability with minimum human effort. we supplement the existing docre dataset, providing a complete test set docgnre and a distant training set. given the inherent presence of false negative instances in numerous re datasets, particularly those constructed through a recommend-revise scheme or distant supervision, we believe our framework possesses a broad utility that extends to a wider array of datasets.