P rompt RE : Weakly-Supervised Document-Level Relation Extraction via Prompting-Based Data Programming (2024)

Abstract: Relation extraction aims to classify the relationships between two entities into pre-defined categories. While previous research has mainly focused on sentence-level relation extraction, recent studies have expanded the scope to document-level relation extraction. Traditional relation extraction methods heavily rely on human-annotated training data, which is time-consuming and labor-intensive. To mitigate the need for manual annotation, recent weakly-supervised approaches have been developed for sentence-level relation extraction while limited work has been done on document-level relation extraction. Weakly-supervised document-level relation extraction faces significant challenges due to an imbalanced number “no relation” instances and the failure of directly probing pretrained large language models for document relation extraction. To address these challenges, we propose PromptRE, a novel weakly-supervised document-level relation extraction method that combines prompting-based techniques with data programming. Furthermore, PromptRE incorporates the label distribution and entity types as prior knowledge to improve the performance. By leveraging the strengths of both prompting and data programming, PromptRE achieves improved performance in relation classification and effectively handles the “no relation” problem. Experimental results on ReDocRED, a benchmark dataset for document-level relation extraction, demonstrate the superiority of PromptRE over baseline approaches.

## introduction relation extraction is a crucial task in natural language processing that aims to classify the relationships between two entities (e.g., pacific fair and queensland) into pre-defined categories (e.g., located in). it has various downstream applications such as question answering @xcite 2017) and knowledge graph construction @xcite . while previous research has mainly focused on relation extraction within a single sentence, recent studies have expanded the scope to document-level relation extraction @xcite . traditional relation extraction methods @xcite heavily rely on human annotation for training data, which is time-consuming and laborintensive. to mitigate the need for manual annotation, recent weakly-supervised approaches @xcite have been developed for relation extraction with minimal or no manual annotation. for example, @xcite extracted textual patterns from seed examples and used those patterns as weak supervisions for re-lation extraction. @xcite represented each relation class using a label verbalizer and then solving the relation extraction task by a textual entailment model. @xcite analyzed an "extremely unlabeled" scenario where each relation type had only one instance, reducing the training set to about five thousand labeled relation triplets. however, these methods were primarily designed and evaluated for sentence-level relation extraction, which limits their generalizability to document-level relation extraction datasets like re-docred @xcite , where the presence of a substantial number of "no relation" or na classes poses additional challenges. to address this limitation, we study the problem of weakly-supervised document-level relation extraction. recent large language models (llms) have achieved great success in a wide range of natural language processing tasks @xcite . we investigate the ability of the pretrained large language models on the document-level relation extraction task. we focus on three pretrained large language models: unifiedqa @xcite , llama, llama2 @xcite , and chatgpt @xcite . unifiedqa is a t5 model @xcite ) pretrained on four different question-answering settings: extractive, abstractive, multiple-choice, and yes/no questions. unifiedqa performs comparably to specialized state-of-the-art models on most relation extraction datasets. chatgpt, developed by openai, is a powerful generative large language model known for its impressive generalization capabilities. however, the closed-source nature of the chatgpt model limits its accessibility for downstream applications. we utilize the text output of chatgpt without accessing its internal embedding space or doing any model fine-tuning. llama is a collection of foundation language models ranging from 7b to 65b parameters trained on only publicly available datasets. after fine-tuning on an instruction-following dataset @xcite , llama and llama2 are able to produce reasonable responses to the input instructions. we use llama-7b and llama2-7b, which has a good balance between model performance and efficiency. we propose promptre, a novel weaklysupervised document-level relation extraction method that combines prompting-based techniques with data programming (figure 1 ). given a known type-relation distribution, we first investigate var-ious ways of prompting the pretrained large language models for relation classification. we then investigate different ways to select the most confident outputs using data programming, a technique that combines multiple sources of weak supervision. by leveraging the strengths of both prompting and data programming, we achieve improved performance in relation classification and effectively handle the "no relation" problem. furthermore, we leverage chatgpt as a summarizer to extract relevant information about the entities of interest. this plays a crucial role especially when dealing with lengthy documents which contain unnecessary extra information. to the best of our knowledge, we are the first to propose weakly-supervised document-level relation extraction. our contributions are summarized as follows: 1. we propose the first weakly-supervised method, promptre, for the document-level relation extraction task. 2. promptre is a novel method that combines various types of prompting outputs with data programming. promptre further incorporates the label distribution and entity types as prior knowledge to improve performance. 3. extensive experiments on the redocred dataset demonstrate the capability of promptre over baseline methods. ablations provide a comprehensive study on weakly-supervised inference ability. multiple case studies show the incompleteness of existing document relation extraction datasets. ## methodology we propose promptre, a weakly supervised document-level relation extraction method that combines large language model prompting with data programming. an illustration of the overall framework of promptre is shown in figure 3 . ## conclusion in this paper, we investigate several methods to integrate prompting and data programming for relation classification and evaluate our model on re-docred. results show that our best results yield around 10.2 f1 on the development set, a promising result for almost no supervision. since this is a novel application, further research is required to investigate strategies for improvement. some ideas include the following. the na issue: the large number of "no relations" continues to be an issue for less than supervised methods for document relation extraction on existing datasets. further work should focus on more efficient and accurate ways to mine distant labels to address this issue. one major roadblock that coincides with na is the lack of complete labels in the dataset as shown in the case studies. future work could improve on existing document relation extraction datasets accordingly. extending to the few-shot case: it is usually possible to query human experts for a few examples of the required classification task. researching ways to take maximum advantage of a small set of labels would also be highly practical, and would not require much extra effort on the annotators. this could also tune a model to better address the "no relation" issue. final thoughts: we find that weakly supervised document-level relation extraction is a uniquely difficult problem due to the incomplete labels in popular datasets, and we propose promptre to attempt to solve it by combining prompting and data programming. we show the effect of tuning different experimental setups, including model size, entity-oriented summarization, and the effect of our relation-type distribution assumption. case studies support the finding that existing document-level relation extraction datasets may be severely lacking in label completeness. although the results are a considerable margin from contemporary supervised methods, we hope that this work can serve as a stepping stone in this novel area of less-thansupervised document relation extraction.