L ogic ST : A Logical Self-Training Framework for Document-Level Relation Extraction with Incomplete Annotations (2024)

Abstract: Document-level relation extraction (DocRE) aims to identify relationships between entities within a document. Due to the vast number of entity pairs, fully annotating all fact triplets is challenging, resulting in datasets with numerous false negative samples. Recently, self-training-based methods have been introduced to address this issue. However, these methods are purely black-box and sub-symbolic, making them difficult to interpret and prone to overlooking symbolic interdependencies between relations.To remedy this deficiency, our insight is that symbolic knowledge, such as logical rules, can be used as diagnostic tools to identify conflicts between pseudo-labels. By resolving these conflicts through logical diagnoses, we can correct erroneous pseudo-labels, thus enhancing the training of neural models.To achieve this, we propose **LogicST**, a neural-logic self-training framework that iteratively resolves conflicts and constructs the minimal diagnostic set for updating models. Extensive experiments demonstrate that LogicST significantly improves performance and outperforms previous state-of-the-art methods. For instance, LogicST achieves an increase of **7.94%** in F1 score compared to CAST (Tan et al., 2023a) on the DocRED benchmark (Yao et al., 2019). Additionally, LogicST is more time-efficient than its self-training counterparts, requiring only **10%** of the training time of CAST.

## introduction document-level relation extraction (docre) aims to extract relational facts between entities within a document, playing a critical role in knowledge graph construction @xcite and question answering @xcite . unlike sentence-level relation extraction, which focuses on individual entity pairs @xcite , docre is challenged by the vast number of potential entity pairs. this number increases quadratically with the number of entities, making it nearly impossible for annotators to meticulously verify the validity of each triplet. although semiautomatic strategies, such as the recommend-revise annotation method @xcite , can alleviate annotators' workload, they still fail to provide goldquality datasets. consequently, these datasets are prone to contain numerous false negative samples. for example, over 60% triplets are not annotated in docred @xcite . therefore, training models from incompletely annotated datasets is crucial and practical for docre. there has been extensive research aimed at alleviating the impact of false negative samples @xcite @xcite . one of the most advanced strategies is self-training @xcite , wherein the model reassigns labels to annotated negative triplets based on its predictions. these adjusted labels are then used iteratively to refine the model's training process. however, selftraining is highly vulnerable to confirmation bias @xcite . specifically, inaccurately predicted pseudo-labels may impair the model's subsequent training. previous works have attempted to mitigate this issue by sampling the pseudo-labels based on class frequencies @xcite or scores calculated on the development set @xcite . however, these pseudo-labeling methods are still far from satisfactory. first, they are purely sub-symbolic approaches. they benefit from the powerful representations provided by language models @xcite , but struggle with symbolic reasoning among entity pairs. therefore, they are prone to making mistakes when logical reasoning is required, leading to conflicts with the inherent interdependencies among relations. second, the purely data-driven nature of these pseudo- @xcite "have you ever been in love" is a song recorded by canadian recording artist celine dion, included first on her seventh english studio album "a new day has come" @xcite and later, on her eighth english studio album "one heart" @xcite . @xcite the song is a power ballad, written by anders bagg, and laila bagge, while production was handled by bagge. labeling methods makes them hard to interpret. third, to achieve optimal performance, they require multiple rounds of training across various folds, significantly increasing the time consumption and limiting their practical application. transcending these limitations calls for a fundamental paradigm shift: i) moving away from independent classification of each triplet to structured prediction; and ii) moving away from pure representation learning towards neural-symbolic computing. our key insight is that symbolic knowledge, such as logical rules, can be utilized as diagnostic tools to identify conflicts between pseudo-labels. for example, in figure 1 , we can identify conflicts such as asserting that have you ever been in love is part of one heart while simultaneously claiming that one heart does not have have you ever been in love as a part, which conflicts with the logical rule (h, part of, t) ⇔ (t, has part, h). by correctly flipping certain pseudo-labels to resolve this conflict, we can enhance the quality of pseudolabels and mitigate the pervasive issue of confirmation bias. building upon this insight, we propose logicst, a novel self-training framework that uses logical rules to diagnose pseudo-labels. logicst is implemented within a teacher-student framework @xcite , where the teacher model is first pre-trained to establish a robust initial state. then, the diagnosed pseudo-labels from the teacher iteratively update the student, whose parameters are in turn used to gradually update the teacher. given the multitude of potential candidates and the high time complexity of computing diagnoses, logicst employs a sequential diagnosis approach. specifically, logicst defines a scoring function that dynamically evaluates the probabilities and rewards of each diagnosis. it eliminates those with lower scores at each updating step and ultimately uses the highest-scoring diagnosis to update the student model. by doing so, our logicst framework i) introduces symbolic reasoning into representation learning, ii) achieves better performance and interpretability, and iii) reduces the need for multiple rounds of training and pseudo-labeling, thus significantly improving time efficiency. our main contributions are listed as follows: ## methodology this paper introduces the logicst framework, which integrates symbolic knowledge into the selftraining process and constructs minimal diagnostic sets to refine pseudo-labels. the overall architecture is described in section 3.1. the method for developing minimal diagnostic sets is detailed in section 3.2. to streamline the diagnostic procedure and identify the optimal diagnosis, the sequential diagnosis approach and the scoring function employed are elaborated in section 3.3. figure 2 visually outlines the workflow of logicst. ## conclusion in this work, we introduce the logicst framework for docre with incomplete annotations. logicst utilizes logical rules to identify conflicts among pseudo-labels and develop minimal diagnoses to correct potential errors. experiments on various benchmarks demonstrate that logicst achieves state-of-the-art results. the effectiveness and efficiency of logicst highlight the potential of the neural-logic paradigm in incompletely labeled information extraction. we believe this paper opens a new avenue for future exploration.