Uncertainty Guided Label Denoising for Document-level Distant Relation Extraction (2023)

Abstract: Document-level relation extraction (DocRE) aims to infer complex semantic relations among entities in a document. Distant supervision (DS) is able to generate massive auto-labeled data, which can improve DocRE performance. Recent works leverage pseudo labels generated by the pre-denoising model to reduce noise in DS data. However, unreliable pseudo labels bring new noise, e.g., adding false pseudo labels and losing correct DS labels. Therefore, how to select effective pseudo labels to denoise DS data is still a challenge in document-level distant relation extraction. To tackle this issue, we introduce uncertainty estimation technology to determine whether pseudo labels can be trusted. In this work, we propose a Document-level distant Relation Extraction framework with Uncertainty Guided label denoising, UGDRE. Specifically, we propose a novel instance-level uncertainty estimation method, which measures the reliability of the pseudo labels with overlapping relations. By further considering the long-tail problem, we design dynamic uncertainty thresholds for different types of relations to filter high-uncertainty pseudo labels. We conduct experiments on two public datasets. Our framework outperforms strong baselines by 1.91 F1 and 2.28 Ign F1 on the RE-DocRED dataset.

## introduction document-level relation extraction (docre) aims to extract relations among entities in a document. in contrast to the conventional re task that mainly focuses on sentence-level @xcite @xcite , docre is more challenging due to the complex semantic scenarios, discourse structure of the document, and long-distant interactions between entities. to understand complex inter-sentence entity relations, most existing methods employ transformer- ... @xcite one of his last film was the last days of @xcite . @xcite he had to leave production early due to an illness, so the film had to be completed by sergio leone ... another is losing the correct ds label as shown by the red dotted line. we also show the proposed instancelevel uncertainty estimation (ue) scores of pseudo labels. we present partly entities that are marked with different colors. despite a great quantity of training data autolabeled by distant supervision can enhance the performance of the model, noisy labels in ds data are non-negligible. @xcite show that there are 61.8% noisy inter-sentence instances in their provided document-level distant relation extraction dataset. current efforts @xcite to alleviate the noise problem mainly employ a pre-denoising model. they train a docre model on human-annotated data first and then re-label ds data by the trained model. however, the above methods still persist the risk of noise induction in the ds data due to false positive re-labeling. besides, false negative pseudo labels also lead to the loss of effective labels in ds data. as shown in figure 1 , we obtain an extra false instance (the last days of pompeii, mario bonnard, composer) and lose the correct ds instance (mario bonnard , rome, place of birth), when merely relying on pseudo labels. thus, how to mitigate noise caused by pseudo labels and take full advantage of ds data is still a challenge in document-level distant re. in this work, we propose a document-level distant relation extraction framework with uncertainty guided label denoising, ugdre. we first train a pre-denoising docre model with both ds and human-annotated data to generate pseudo labels. since false pseudo labels predicted by the pre-denoising model are inevitable, we introduce uncertainty estimation (ue) to determine whether model predictions can be trusted or not. as shown in figure 1 , we can remove the false positive pseudo instance (the last days of pompeii, mario bonnard, composer) according to its high uncertainty score. in this way, we can abstain from unreliable decisions of the pre-denoising model, which can mitigate the risk of false pseudo labels. considering there might be multiple relations between an entity pair, we propose an instance-level ue method to capture uncertainty scores for overlapping relations. moreover, we design a re-labeling strategy with dynamic class uncertainty thresholds by taking the docre long-tail problem into account to obtain high-quality ds data. with the proposed uncertainty guided label denoising mechanism, we design a multi-phase training strategy to further boost the performance of our final docre model. the main contributions of our work are summarized as follows: recent works observe that utilizing large-scale auto-labeled data generated by distant supervision @xcite to pretrain the docre model can attain great performance improvements @xcite @xcite . most of them directly utilize the ds data and ignore the accuracy of ds labels. to obtain high-quality ds data, several methods introduce re-label strategies based on the predenoising re model trained on human-annotated data @xcite . however, these methods ignore the noise caused by pseudo labels. in this work, we introduce uncertainty estimation to determine the reliability of pseudo labels, which can reduce the noisy pseudo labels to further improve the quality of ds data. ## methodology in this section, we introduce our proposed framework in detail. as shown in figure ## analysis and discussion in this section, we conduct extensive experiments to further analyze the effectiveness of our proposed denoising strategy and instance-level ue method. we also conduct the ablation study to discuss the contribution of each component of the framework. ## conclusion in this paper, we propose a document-level distant relation extraction framework with uncertainty guided label denoising, ugdre. specifically, we propose instance-level uncertainty estimation to measure the reliability of pseudo labels. considering the long-tail problem, we design dynamic class uncertainty thresholds to filter high-uncertainty pseudo labels. our proposed uncertainty guided denoising strategy can improve the quality of ds data. experimental results demonstrate that our ugdre outperforms competitive baselines. moreover, extensive experiments verify the effectiveness of our label denoising. there are various challenges in docre worth exploring, one is to research the lowresource relation extraction.