Learning Logic Rules for Document-Level Relation Extraction (2021)

Abstract: Document-level relation extraction aims to identify relations between entities in a whole document. Prior efforts to capture long-range dependencies have relied heavily on implicitly powerful representations learned through (graph) neural networks, which makes the model less transparent. To tackle this challenge, in this paper, we propose LogiRE, a novel probabilistic model for document-level relation extraction by learning logic rules. LogiRE treats logic rules as latent variables and consists of two modules: a rule generator and a relation extractor. The rule generator is to generate logic rules potentially contributing to final predictions, and the relation extractor outputs final predictions based on the generated logic rules. Those two modules can be efficiently optimized with the expectation-maximization (EM) algorithm. By introducing logic rules into neural networks, LogiRE can explicitly capture long-range dependencies as well as enjoy better interpretation. Empirical results show that significantly outperforms several strong baselines in terms of relation performance and logical consistency. Our code is available at https://github.com/rudongyu/LogiRE .

## introduction extracting relations from a document has attracted significant research attention in information extraction (ie). recently, instead of focusing on sentencelevel @xcite @xcite @xcite , researchers have turned to modeling directly at the document level @xcite @xcite , which provides longer context and requires more complex reasoning. early efforts focus mainly on learning a powerful relation (i.e., entity pair) representation, which the three labeled sentences describe the relations royalty_of (harry,uk), sib-ling_of (william,harry), and spouse_of(kate,william), respectively. the identification of the relation roy-alty_of (kate,uk) requires the synthesis of information in three sentences. it can be easily derived from the demonstrated rule and the other three relations. the sequence-based model first leverages different sequence encoder (e.g., bert @xcite , roberta @xcite ) to obtain token representations, and then computes relation representations by various pooling operations, e.g., average pooling @xcite , attentive pooling @xcite . to further capture long-range dependencies, graph-based models are proposed. by constructing a graph, words or entities that are far away can become neighbor nodes. on top of the sequence encoder, the graph encoder (e.g., gnn) can aggregate information from all neighbors, thus capturing longer dependencies. various forms of graphs are proposed, including dependency tree @xcite , co-reference graph @xcite , mention-entity graph @xcite , entity-relation bipartite graph @xcite and so on. despite their great success, there is still no comprehensive understanding of the internal representations, which are often criticized as mysterious "black boxes". learning logic rules can discover and represent knowledge in explicit symbolic structures that can be understood and examined by humans. at the same time, logic rules provide another way to explicitly capture interactions between entities and output relations in a document. for example in fig. 1 , the identification of royalty_of(kate,uk) requires information in all three sentences. the demonstrated logic rule can be applied to directly obtain this relation from the three relations locally extracted in each sentence. reasoning over rules bypasses the difficulty of capturing long-range dependencies and interprets the result with intrinsic correlations. if the model could automatically learn rules and use them to make predictions, then we would get better relation extraction performance and enjoy more interpretation. in this paper, we propose logire, a novel probabilistic model modeling intrinsic interactions among relations by logic rules. inspired by rnn-logic @xcite , we treat logic rules as latent variables. specifically, logire consists of a rule generator and a relation extractor, which are simultaneously trained to enhance each other. the rule generator provides logic rules that are used by the relation extractor for prediction, and the relation extractor provides some supervision signals to guide the optimization of the rule generator, which significantly reduces the search space. in addition, the proposed relation extractor is model agnostic, so it can be used as a plug-and-play technique for any existing relation extractors. those two modules can be efficiently optimized with the em algorithm. by introducing logic rules into neural networks, logire can explicitly capture long-range dependencies between entities and output relations in a document and enjoy better interpretation. our main contributions are listed below: ## related work for document-level relation extraction, prior efforts on capturing long-range dependencies mainly focused on two directions: pursuing stronger sequence representation @xcite @xcite or including prior for interactions among entities as graphs @xcite . for more powerful representations, they introduced pre-trained language models @xcite , leveraged attentions for context pooling @xcite , or integrated the scattered information according to a hierarchical level @xcite . aiming to model the intrinsic interactions among entities and relations, they utilized implicit reasoning structures by carefully designing graphs connecting: mentions to entities, mentions in the same sentence @xcite , mentions of the same entities @xcite , etc. @xcite ; @xcite directly integrated similar structural dependencies to attention mechanisms in the encoder. these approaches contributed to obtaining powerful representations for distinguishing various relations but lacked interpretability on the implicit reasoning. another approach that can capture dependencies between relations is the global normalized model @xcite . in this work, we focus on how to learn and use logic rules to capture long-range dependencies between relations. another category of related work is logical reasoning. many studies were conducted on learning or applying logic rules for reasoning. most of them @xcite concentrated on reasoning over knowledge graphs, aiming to deduct new knowledge from existing triples. neural symbolic systems @xcite combined logic rules and neural networks to benefit from regularization on deep learning approaches. these efforts demonstrated the effectiveness of integrating neural networks with logical reasoning. despite doc-re providing a suitable scenario for logical reasoning (with relations serving as predicates and entities as variables), no existing work attempted to learn and utilize rules in this field. using hand-crafted rules, wang and pan (2020); @xcite achieved great success on sent-level information extraction tasks. however, the rules were predefined and limited to lowlevel operations, restricting their applications. ## method in this section, we describe the proposed method logire that learns logic rules for document-level relation extraction. we first define the task of document-level relation extraction and logic rules. ## experiments we conduct experiments on multi-relational document-level relation extraction datasets: do-cred @xcite and dwie @xcite . the statistics of the two datasets are listed in evaluation besides the commonly used f1 metric for relation extraction, we also include other two metrics for comprehensive evaluation of the models: ign f1, logic. ign f1 was proposed in @xcite for evaluation with triples appearing in the training set excluded. it avoids information leakage from the training set. we propose logic for evaluation of logical consistency among the prediction results. specifically, we use the 41 pre-defined rules on the dwie dataset to evaluate whether the predictions satisfy these gold rules. the rules have a similar form to logic rules defined in sec. 3. we name the precision of these rules on predictions as logic score. note that these rules are independent of the rule learning and utilization in sec. 3 but only used for logic evaluation. experimental settings the rule generator in our experimental settings is implemented as a transformer with a two-layer encoder and a two-layer decoder, hidden size set to 256. we empirically find the tiny structure is enough for modeling the required rule set. we set the size of the latent rule set 𝑁 to 50. we limit the maximum length of logic rules to 3 in our setting. ## conclusion in this paper, we proposed a probabilistic model lo-gire, which utilizes rules and conducts reasoning over the rules for document-level relation extraction. the logic rules are treated as latent variables. we utilize the em algorithm to efficiently maximize the overall likelihood. by injecting rules to the relation extraction framework, our logire explicitly models the long-range dependencies in docre as interactions among relations and entities, thus enjoying better interpretability. empirical results and analysis show that logire outperforms strong baselines on overall performance, logical consistency, and capability for capturing long-range dependencies. the approximation is obtained by the following second-order taylor expansion: by such approximation, we can decompose the posterior to each rule in the latent rule set. we first define the score for each rule: then, it's easy to obtain that the approximated posterior 𝑞(𝒛) and the prior 𝑝 𝜃 are conjugate distributions.