A Unified Positive-Unlabeled Learning Framework for Document-Level Relation Extraction with Different Levels of Labeling (2022)

Abstract: Document-level relation extraction (RE) aims to identify relations between entities across multiple sentences. Most previous methods focused on document-level RE under full supervision. However, in real-world scenario, it is expensive and difficult to completely label all relations in a document because the number of entity pairs in document-level RE grows quadratically with the number of entities. To solve the common incomplete labeling problem, we propose a unified positive-unlabeled learning framework - shift and squared ranking loss positive-unlabeled (SSR-PU) learning. We use positive-unlabeled (PU) learning on document-level RE for the first time. Considering that labeled data of a dataset may lead to prior shift of unlabeled data, we introduce a PU learning under prior shift of training data. Also, using none-class score as an adaptive threshold, we propose squared ranking loss and prove its Bayesian consistency with multi-label ranking metrics. Extensive experiments demonstrate that our method achieves an improvement of about 14 F1 points relative to the previous baseline with incomplete labeling. In addition, it outperforms previous state-of-the-art results under both fully supervised and extremely unlabeled settings as well.

## introduction relation extraction (re) aims to identify the relations between two entities in a given text. it has rich applications in knowledge graph construction, question answering, and biomedical text understanding. most of the previous work was to extract relations between entities in a single sentence @xcite . recently, document-level re aiming to identify the relations among various entity pairs expressed in multiple sentences has received increasing research @xcite . previous document-level re methods mainly deal with fully supervised scenarios. however, in real-world scenarios, incomplete labeling is a common problem in document-level re because the number of entity pairs grows quadratically with the number of entities. docred @xcite is a popular dataset for document-level re. recent studies @xcite found that docred, which annotates data with a recommend-revise scheme, contains a large number of false negative samples, i.e., many positive samples being unlabeled. as shown in figure 1 , document alecu russo contains a large number of unlabeled positive relations. consequently, the models trained on this dataset tend to overfit in real scenarios and get lower recall. as a result, document-level re with incomplete labeling has become an emergency need. to solve this problem, we propose a unified positive-unlabeled learning frameworkshift and squared ranking loss positive-unlabeled (ssr-pu) learning, which can be adapted to labeling under different levels. we use positive-unlabeled (pu) learning for the first time on the document-level re task. since document-level re is a multi-label classification task, we apply a binary pu learning method for each class (one-vs-all), converting it to multi-label pu learning. in addition, according to our observations, a considerable portion of the relations in docred, a dataset annotated by recommend-revise scheme, have already been annotated. this leads to the deviation between the prior distribution of the unlabeled data and the overall prior distribution. to address this problem, we introduce an adaptive pu learning under prior shift of training data that adjusts the model based on the estimated overall prior distribution and the labeled positive sample distribution to be similar to ordinary pn learning or ordinary pu learning. here positive-negative (pn) learning means treating all unlabeled samples as negative samples. also, to distinguish between none-class and predefined classes, we propose a squared ranking loss for none-class ranking such that positive predefined labels are ranked higher than none-class label and negative pre-defined labels are ranked lower. this is an ideal multi-label surrogate loss metric, and we theoretically prove its bayesian consistency with the multi-label ranking metric proposed by @xcite . this loss function can be well adapted to pu learning. we conduct extensive experiments on two multilabel document-level re datasets with incomplete labeling, docred @xcite and chemdis-gene @xcite , a newly proposed multilabeled biomedical document-level re dataset. experimental results show that our method ssr-pu outperforms previous baseline that did not consider the labeling incompleteness phenomenon by about 14 f1 points. in addition, we perform fully super-vised experiments, as well as experiments on an extremely unlabeled data that is newly constructed, in which the number of each relation type labeled in each document is limited to 1. experiments under two complementary settings demonstrate the effectiveness of our method with different levels of labeling. the contributions of this paper are summarized as follows: ## methodology in this section, we introduce the details of our method shift and squared ranking loss positiveunlabeled (ssr-pu) learning for document-level re with incomplete labeling. firstly, we introduce the definition of positive-unlabeled learning for document-level re. next, we present the pu learning under prior shift of training data. finally, squared ranking loss using the none-class score as an adaptive threshold is proposed. ## conclusion and future work in this paper, we propose a unified positiveunlabeled learning framework, ssr-pu, which can effectively solve the incomplete labeling of document-level re. we use pu learning on document-level re for the first time and introduce a pu learning under prior shift of training data to adapt to different levels of labeling. also, we propose squared ranking loss, using none-class score as an adaptive threshold. experiments demonstrate that our method achieves state-of-the-art results with different levels of labeling and provides a robust new baseline for incompletely labeled document-level re. in the future, we will consider methods that do not require estimation of priors, allowing generalization to unknown distributions more accurately, as well as addressing the problem of biased distributions with incomplete labeled positive samples and further improving the extraction performance of long-tail relations.