Relation-Specific Attentions over Entity Mentions for Enhanced Document-Level Relation Extraction (2022)

Abstract: Compared with traditional sentence-level relation extraction, document-level relation extraction is a more challenging task where an entity in a document may be mentioned multiple times and associated with multiple relations. However, most methods of document-level relation extraction do not distinguish between mention-level features and entity-level features, and just apply simple pooling operation for aggregating mention-level features into entity-level features. As a result, the distinct semantics between the different mentions of an entity are overlooked. To address this problem, we propose RSMAN in this paper which performs selective attentions over different entity mentions with respect to candidate relations. In this manner, the flexible and relation-specific representations of entities are obtained which indeed benefit relation classification. Our extensive experiments upon two benchmark datasets show that our RSMAN can bring significant improvements for some backbone models to achieve state-of-the-art performance, especially when an entity have multiple mentions in the document.

## introduction relation extraction (re) is one important task of information extraction, aiming to detect the relations among entities in plain texts. recently, many scholars have paid more attention to document-level re @xcite which aims to identify the relations of all entity pairs in a document, since it is more in demand than sentencelevel re in various real scenarios. in general, one document contains multiple entities and an entity may have multiple mentions across different sentences. furthermore, one entity may be involved by multiple valid relations and different relations are points of the same color and marker are different mentions' embeddings of an entity, which are encoded by bert @xcite . a key step of existing document-level re methods is to aggregate the information of different mentions of an entity (mention-level features) to obtain the entity's representation (entity-level feature) at first, since relation classification is generally achieved on entity level. to this end, previous re models simply apply average pooling @xcite , max pooling @xcite , or logsumexp pooling @xcite . finally, a fixed representation is obtained for the given entity, which is then fed into the classifier for relation classification. however, different mentions of an entity in a document may hold distinct semantics. a simple pooling operation of generating a fixed entity representation may confound the semantics of different mentions, and thus degrades the performance of relation classification when the entity is involved by multiple valid relations. we call such situation as multi-mention problem in this paper. in fig. 1 , we display the t-sne (van der maaten and hinton, 2008) visualization of a toy example's mention embedding space to validate this problem. as the figure shows, different mentions' embeddings of an entity (marked by the same color) in a document are scattered over the whole embedding space, indicating that different mentions of an entity are not semantically adjacent. we further illustrate it by the toy example in fig. 2, the first mention samuel herbert cohen of the person entity is more important for the classifier to identify the relation country of citizenship between him and australian. but for extracting the relation place of birth, the second mention he should be considered more. it implies that different mentions should play different roles when extracting the different relations involving the same entity. in other words, different mentions function differently in different relation recognitions. inspired by this intuition, we propose a novel relation-specific mention attention network (rsman) to improve the model performance of document-level re. in rsman, each relation's essential semantics is first encoded into a prototype representation. then, the relevance weight (attention) between the prototype of a specific candidate relation and each mention's representation of the given entity is calculated. based on these attentions, we get an attentive (weighted) sum of all mentions' representations as the entity's synthetic representation. in this manner, rsman enables the model to attend to the information of multiple mentions from different representation space when representing an entity, indicating that the entity's representation is flexible and relation-specific with respect to different candidate relations. our contributions in this paper can be summarized as follows: 1. to the best of our knowledge, this is the first to consider different mentions' significance with respect to candidate relations on representing an entity to achieve document-level re. 2. we propose a novel rsman which can be used as a plug-in of a backbone re model, to learn a relation-specific representation for a given entity which enhances the model's performance further. 3. our empirical results show that rsman can significantly promote some backbone models to achieve state-of-the-art (sota) re performance, especially when an entity have multiple mentions in the document. the rest of this paper is organized as follows. in section 2, we briefly introduce some works related to our work. then we introduce the proposed method in section 3 and the experiment results in section 4. at last, we conclude our work in section 5. ## related work prior efforts on document-level re mainly focused on representation learning and reasoning mechanism. @xcite employed four different sentence-level representation models to achieve document-level re, including cnn, lstm, bil-stm, and context-aware. for more powerful representations, later work introduced pre-trained language models into their neural architectures @xcite @xcite . in particular, @xcite added a novel mention reference prediction task during pre-training and presented corefbert to capture the coreferential relations in contexts. @xcite proposed atlop to learn an adjustable threshold and thus enhanced the entity pair's representation with localized context pooling. @xcite defined various mention dependencies in a document and proposed ssan to model entity structure for document-level re. in addition, other work built various kinds of document graphs to model reasoning mechanism explicitly @xcite @xcite . for example, @xcite induced the latent document-level graph and performed multi-hop reasoning on the induced latent structure. @xcite constructed a global heterogeneous graph and used a stacked r-gcn @xcite to encode the document information. @xcite proposed gain to leverage both mention-level graph and entity-level graph to infer relations between entities. however they all ignore the multimention problem described in sec. 1. ## methodology at first, we formalize the task of document-level re addressed in this paper as follows. suppose a document d mentions p entities, denoted as @xmath0, and the i-th entity e i has q i mentions in d, denoted as {m i j } q i @xmath1, the task of document-level re is to extract a set of relational triples {(e s , r, e o )|e s , e @xmath2, @xmath3. ## conclusion in this paper, we focus on the multi-mention problem in document-level re and propose rsman to address it. our experiment results demonstrate rsman's effectiveness especially on the scenario of multi-mention situation. in the future, we plan to adapt rsman to more document-level re models.