Not Just Plain Text! Fuel Document-Level Relation Extraction with Explicit Syntax Refinement and Subsentence Modeling (2022)

Abstract: Document-level relation extraction (DocRE) aims to identify semantic labels among entities within a single document. One major challenge of DocRE is to dig decisive details regarding a specific entity pair from long text. However, in many cases, only a fraction of text carries required information, even in the manually labeled supporting evidence. To better capture and exploit instructive information, we propose a novel expLicit syntAx Refinement and Subsentence mOdeliNg based framework (LARSON). By introducing extra syntactic information, LARSON can model subsentences of arbitrary granularity and efficiently screen instructive ones. Moreover, we incorporate refined syntax into text representations which further improves the performance of LARSON. Experimental results on three benchmark datasets (DocRED, CDR, and GDA) demonstrate that LARSON significantly outperforms existing methods.

## introduction relation extraction (re) is an essential task in information extraction. it aims to model relational patterns between entities in unstructured text. one strikingly significant variant of re, document-level relation extraction (docre) is designed to identify relations among entity pairs distributed throughout the document. compared to traditional sentencelevel re @xcite @xcite , where entities are located in the same sentence, docre further fits the need in real scenes and has received increasing attention lately @xcite @xcite @xcite . in an ordinary document, interactions between entities are complex. since pre-trained language models (plms) have shown their great potential emancipation proclamation of 1863 in …, united states. [2] washington place was built by william and annie washington in north romney between 1863 and 1874 … [3] william washington later acquired other properties on the hills north of romney along west virginia route 28 and became the first african-american land developer in the state of west virginia … subject: william (per) object: united states (loc) golden & predicted label: country of citizenship subject: mic relation: lan [1] during he in northwes national scho [3] her abilit to pursue a c michelle ferr chan in the 1 [6] jackie c while being i subject: mic relation: lan [1] during he in northwes national scho [3] her abilit to pursue a c michelle ferr chan in the 1 [6] jackie c while being i subject: relation: m lan subject: relation: ho pl [1] during her youth, michelle lived with her grandparents in northwestern france before enrolling into an international school in kobe, japan. [2] ferre later enrolled in… [3] her ability to speak both english and japanese led her to pursue a career in journalism. [4] she lived in … [5] michelle ferre 's first acting experience was opposite jackie chan in the 1998 hong kong martial arts film who am i ?. [6] jackie chan became intrigued by ferre's personality while being interviewed on the set of who am i ? … subject: object: relation: evidence: michelle language spoken subject: object: relation: evidence: hong kong place of origin english [1, 3] [ who am i in many downstream tasks @xcite , some works implicitly capture such interactions through pre-trained language models (plms) @xcite @xcite . some other works model this information explicitly. they first construct document graphs that consist of different nodes (e.g., mentions, entities, sentences, or the document) @xcite @xcite . then, graph convolutional networks (gcns) @xcite ) are adopted to encode precise node representations and infer final relations. according to their experiments, gcns can better capture complex interconnections between nodes which is the foundation of our method. one major challenge around docre concerns inferring relations of multiple entity pairs across long text, which may contain irrelevant or even noisy information. take figure 1 as an example. it includes an inter-sentence and an intra-sentence relation instance. to infer the inter-sentence relation between michelle and english, human can quickly locate the subsentence "her ability ..." and deduce that her is referring to michelle here. furthermore, given (hong kong, who am i), human can identify their intra-sentence relation just by a prepositional phrase "in the 1998 hong kong martial arts film who am i? ". unfortunately, such fine-grained annotations are not available in current docre datasets. with massive irrelevant information, docre models sometimes struggle with complicated relation instances, indicating that implicitly learning instructive context is not sufficient. another factor that hinders the further development of docre algorithms is syntax information. as pointed out in @xcite plms are trained with massive realworld text data, there is still a big gap between implicitly learned syntax and the golden syntax. moreover, syntax information is widely incorporated in many sentence-level re models @xcite @xcite and is not yet developed sufficiently under docre scenario @xcite . in order to better solve docre, we propose a novel explicit syntax refinement and subsentence modeling based framework (lar-son). we mainly exploit dependency and constituency trees to incorporate extra syntax information and model subsentences. the two trees depict complete yet different aspects of syntax information. for example, the evidence sentence as in the previous intra-sentence relation instance can be parsed into corresponding dependency tree and consituency tree 1 . we can see from figure 2a that a dependency tree describes the dependencies between words within a single sentence that strongly complement the original plain text. moreover, it facilities coupling syntactically associated words. the constituency tree shown in figure 2b can organize different words of a single sentence hierarchically and reasonably, eliminating the process of enumerating different word combinations while maintaining hierarchical syntax information. with the aid of dedicated attention module @xcite , we can even gather all subsentences with proper weights to resolve inter-sentence dependencies and produce final relations. to the best of our knowledge, larson is the first docre model that considers subsentence modeling. through extensive experiments on three public docre benchmarks, docred @xcite , cdr @xcite , and gda @xcite , we demonstrate that our model larson outperforms existing methods by a large margin. our key contributions of this work can be summarized as follows: ## problem formulation given a document d made up of sentences {sen i } i @xmath0, the goal of docre is to correctly infer all the existing relations between each entity pair (e s , e o ) s,@xmath1...n ;s̸ =o where e s is the subject entity and e o is the object entity. among which, entity e i usually consists of multiple mentions and the j th mention of e i is denoted as m (i,j) . predicted relations are either a subset of predefined relations r or {na} (no relation). ## methodology the overall architecture of larson is illustrated in figure 3 . we first extract two kinds of trees and encode plain text using plms. with dependency trees and graph attention networks (gats) @xcite , syntax complemented text representations are derived. on this basis, we extract embeddings of subject entity, object entity, and entity pair-related context (section 3.1). then, on the strength of tree-lstm @xcite , we lever-age constituency trees to model subsentences of arbitrary granularity (section 3.2). final relations are inferred based on the dynamic fusion of embeddings obtained in section 3.1 and subsentence embeddings obtained in section 3.2. during the process, extra sentence embeddings acquired from both trees are utilized for auxiliary evidence extraction to bring into full play of the syntax information (section 3.3). ## related work relation extraction (re) is an indispensable part of many real-world applications like question answering @xcite , biomedical text analysis @xcite , etc. many studies have been done to solve sentence-level re where entities are within the same sentence, such as sequencebased methods @xcite , graph-based methods @xcite , and plms based methods @xcite . nguyen 1] washington place (william washington house) is one of the first homes built by freed slaves after the emancipation proclamation of 1863 in hampshire county, west virginia, united states. [2] washington place was built by william and annie washington in north romney … [3] william washington later acquired other properties on the hills north of romney along west virginia route 28 and became the first african-american land developer in the state of west virginia. [4] one of his subdivisions is the " blacks hill " neighborhood of romney , adjacent to the washington place homestead. [5] washington place was bought and restored by ralph w. haines , a local attorney and historic preservationist . subject: object: william united states golden and predicted relation: country of citizenship [1] washington place (william washington house) is one of the first homes built by freed slaves after the emancipation proclamation of 1863 in hampshire county, west virginia, united states. [2] washington place was built by william and annie washington in north romney … [3] william washington later acquired other properties on the hills north of romney along west virginia route 28 and became the first african-american land developer in the state of west virginia. [4] one of his subdivisions is the "blacks hill" neighborhood of romney, adjacent to the washington place homestead. [5] washington place was bought and restored by ralph w. haines, a local attorney and historic preservationist. and moschitti, 2011 even successfully utilized both dependency and constituency trees to better solve sentence-level re task. these methods have shown tremendous performance in sentence-level re task. nevertheless, according to yao et al., 2019, a considerable proportion of relation instances can only be inferred from multiple sentences, and existing methods are still inadequate for real-world re problems. docre, due to its more realistic setting, has gained more and more attention lately. with the rapid development of plms, they have been widely adopted as the first step in docre to encode plain text. according to the rest steps, existing high performance methods can be divided into two categories. one category inherits implicitly learned attention distribution inside plms or designs extra attention modules to capture long range dependencies. @xcite plms to encode semantic features of document and weights different sentences based on different entities via attention mechanism. in order to combine sentence-level and document-level features together, gating mechanism is developed. instead, @xcite inherits implicitly learned attention distribution inside plms to determine entity-related context. in addition, they also adopt adaptive threshold to better distinguish positive labels from negative labels. the other category exploits gcns to capture complex interactions between different components (e.g., entities, or sentences) and conduct logical reasoning. @xcite proposes a heterogeneous mention-level graph to capture complex interactions among different mentions. they further aggregate different mention representations to construct an entity-level graph. based on the graphs, they develop a novel path reasoning mechanism for final relation extraction. @xcite encourages the model to reconstruct reasoning paths while identifying correct relations. unlike previous works, larson tries to better solve docre task by integrating explicitly refined syntax and subsentence modeling. to the best of our knowledge, larson is the first work investigating the effects of subsentences under the docre scenario. 6 conclusion in this work, we propose a novel larson model for document-level relation extraction task. lar-son mostly exploits two kinds of extra syntax information, namely dependency syntax and constituency syntax. graph attention networks and tree-lstm are adopted to encode the two kinds of information. furthermore, through leveraging dedicated attention module, we can dynamically weight different subsentences to assist larson in capturing instructive information regardless of the granularity. experiments on three public docre datasets demonstrate that our larson model outperforms existing methods by a large margin.