End-to-end Learning of Logical Rules for Enhancing Document-level Relation Extraction (2024)

Abstract: Document-level relation extraction (DocRE) aims to extract relations between entities in a whole document. One of the pivotal challenges of DocRE is to capture the intricate interdependencies between relations of entity pairs. Previous methods have shown that logical rules can explicitly help capture such interdependencies. These methods either learn logical rules to refine the output of a trained DocRE model, or first learn logical rules from annotated data and then inject the learnt rules into a DocRE model using an auxiliary training objective. However, these learning pipelines may suffer from the issue of error propagation. To mitigate this issue, we propose Joint Modeling Relation extraction and Logical rules or JMRL for short, a novel rule-based framework that jointly learns both a DocRE model and logical rules in an end-to-end fashion. Specifically, we parameterize a rule reasoning module in JMRL to simulate the inference of logical rules, thereby explicitly modeling the reasoning process. We also introduce an auxiliary loss and a residual connection mechanism in JMRL to better reconcile the DocRE model and the rule reasoning module. Experimental results on four benchmark datasets demonstrate that our proposed JMRL framework is consistently superior to existing rule-based frameworks, improving five baseline models for DocRE by a significant margin.

## introduction relation extraction (re) plays a vital role in information extraction (ie). it aims at identifying relations between two entities in a given text. early efforts focus mainly on sentence-level re. in recent years, document-level relation extraction (docre) has received increasing attention. it aims at identifying relations of all entity pairs in a document. nowadays docre has been widely applied in * corresponding authors document title: parvathy jayaram downstream applications such as question answering (qa) @xcite , knowledge graph construction @xcite , etc. compared to sentence-level re, docre imposes a greater challenge for modeling longer contexts and capturing the more complex interdependencies between entity pairs. most previous methods for docre focus on capturing interdependencies between entity pairs by learning powerful representations through neural models, such as pre-trained language models @xcite and graph neural networks @xcite @xcite . however, these methods tend to be insufficient in inferring potential facts. figure 1 illustrates such an example, where sub-figure (a) in figure 1 shows an example of a document in the docred dataset, and sub-figure (b) shows the corresponding predictions yielded by atlop, a state-of-the-art (sota) method for docre. we can observe that atlop @xcite only extracts apparent facts such as "(parvathy, hasspouse, jayaram)" and "(parvathy, haschild, kalidasjayaram)", but fails to identify potential facts such as "(kalidasjayaram, hasfather, jayaram)" and "(jayaram, haschild, malavikajayaram)" which are not explicitly mentioned in the context of the given document. it is well known that logical rules can be used to improve the performance for docre by inferring missing facts from existing ones. sub-figure (c) in figure 1 illustrates three logical rules, and sub-figure (d) shows their ability in inferring missing facts. to enhance existing docre models with logical rules, two rule-based frameworks have been proposed, namely logicre @xcite and milr @xcite . in more detail, logicre first learns logical rules based on the output logits of a trained neural model and then refines its predicted relations by reasoning with the learnt rules, whereas milr first learns logical rules from annotated data and then trains a neural model penalized by an auxiliary loss for reflecting the violation of learnt rules. although both logicre and milr have shown promising results in enhancing docre, they still suffer from the error propagation issue due to their pipeline natures. to mitigate the error propagation issue, we target jointly learning a neural module for docre and a neural module for approximating logical rules in an end-to-end fashion. in this paper we propose a novel framework named joint modeling relation extraction and logical rules or jmrl for short, as illustrated in figure 2 . the intuition of jmrl is to reduce the rule learning problem in discrete space to a parameter learning problem in continuous space, yielding a neural module, called rule reasoning module, for approximating logical rules, and then to integrate it into an existing docre model. the parameters of the rule reasoning module is tuned along with the parameters of the backbone docre model so that the whole model can be trained in an end-to-end fashion. furthermore, we introduce an auxiliary loss and a residual connection mechanism in jmrl to better incorporate the backbone docre model and the rule reasoning module, so as to further improve the performance. we exploit jmrl to enhance five baseline models for docre, including lstm @xcite , bi-lstm @xcite , gain @xcite , atlop @xcite and dreeam @xcite . experimental re-sults on four benchmark datasets dwie @xcite , docred @xcite , re-docred @xcite and docgnre @xcite demonstrate that the proposed jmrl framework is superior to all sota rule-based framework for docre, improving the baseline models by a significant margin on all datasets. besides, our analysis and case study further clarify why jmrl is able to improve the performance. the main contributions of this work include: (1) we propose a novel framework named jmrl to integrate a neural module for approximating logical rules into a baseline docre model, so that the enhanced docre model can be trained in an end-to-end fashion. as far as we know, this is the first end-to-end approach for imposing logical rules upon docre models. (2) we theoretically analyze the faithfulness between the rule reasoning module and logical rules. ## preliminaries problem formulation for docre. given a document d involving a set of named entities e @xmath0, the task of docre aims at predicting the relations among all entity pairs {(e h , e t ) | e h , e @xmath1, e h ̸ = e t }. the set of predictable relations is defined as r + = r ∪ {⊥}, where r is a pre-defined relation set and ⊥ the "no relation". ## the jmrl framework to impose logical rules upon a docre model, we propose a novel rule-based framework named joint modeling relation extraction and logical rules or jmrl for short, as illustrated in figure 2 . by and large, jmrl first employs a docre model to calculate output logits for all potential facts in a document, and then feeds them into a rule reasoning module to produce the rule-enhanced logits. the ultimately predicted logits are calculated by the residual connection of the original docre logits and the rule-enhanced logits. then the entire model is trained by minimizing a weighted sum of classification losses calculated from the original docre logits and the ultimately predicted logits. document title: parvathy jayaram [1] ashwathy kurup, better known by her stage name parvathy, is an indian film actress and classical dancer … [2] parvathy married film actor jayaram who was her co-star in many films on 7th september 1992 at town hall, ernakulam. ## conclusion and future work in this paper we have proposed an end-to-end learning framework named jmrl to empower existing docre models with stronger reasoning abilities. notably, we have proposed a novel rule reasoning module in jmrl to simulate the inference of logical rules, thereby enhancing the reasoning ability. furthermore, we have shown theoretically that the parameterization of this module is faithful to the formalization of logical rules. experimental results on four benchmark datasets verify the effectiveness of jmrl. future work will extend jmrl to jointly learn named entity recognition (ner), docre and more expressive rules in an end-to-end fashion.