An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning (2021)

Abstract: We present a joint model for entity-level relation extraction from documents. In contrast to other approaches - which focus on local intra-sentence mention pairs and thus require annotations on mention level - our model operates on entity level. To do so, a multi-task approach is followed that builds upon coreference resolution and gathers relevant signals via multi-instance learning with multi-level representations combining global entity and local mention information. We achieve state-of-the-art relation extraction results on the DocRED dataset and report the first entity-level end-to-end relation extraction results for future reference. Finally, our experimental results suggest that a joint approach is on par with task-specific learning, though more efficient due to shared parameters and training steps.

## introduction information extraction addresses the inference of formal knowledge (typically, entities and relations) from text. the field has recently experienced a significant boost due to the development of neural approaches @xcite @xcite . this has led to two shifts in research: first, while earlier work has focused on sentence level relation extraction @xcite @xcite , more recent models extract facts from longer text passages @xcite . this enables the detection of inter-sentence relations that may only be implicitly expressed and require reasoning across sentence boundaries. current models in this area do not rely on mention-level annotations and aggregate signals from multiple mentions of the same entity. the second shift has been towards multi-task learning: while earlier approaches tackle entity mention detection and relation extraction with separate models, recent joint models address these tasks the portland golf club is a private golf club in the northwest united states, in suburban portland, oregon. the pgc is located in the unincorporated raleigh hills area of eastern washington county, southwest of downtown portland and east of beaverton. pgc was established in the winter of 1914, when a group of nine businessmen assembled to form a new club after leaving their respective clubs. the golf club hosted the ryder cup matches of 1947, the first renewal in a decade, due to world war ii. the u.s. team defeated great britain 11 to 1 in wet conditions in early november. figure 1: our goal is to perform end-to-end entity-level relation extraction on whole documents. we extract entity mentions ("pgc"), entity clusters ({portland golf club, pgc, golf club}), their types (org) and relations to other entities in the document, such as ({portland golf club, pgc, golf club} org , inception, {1914} t im e ), with a single, joint model. note that document-level relation extraction requires the aggregation of relevant information from multiple sentences, such as in ({raleigh hills} loc , country, {united states, u.s.}) loc ). other entities in the example document are omitted for clarity. at once @xcite @xcite . this does not only improve simplicity and efficiency, but is also commonly motivated by the fact that tasks can benefit from each other: for example, knowledge of two entities' types (such as person+organization) can boost certain relations between them (such as ceo of). we follow this line of research, and present jerex 1 ("joint entity-level relation extractor"), a novel approach for joint information extraction. jerex is to our knowledge the first approach that combines a multi-task model with entity-level relation extraction: in contrast to previous work, our model jointly learns relations and entities without annotations on mention level, but extracts document-level entity clusters and predicts relations between those clusters using a multi-instance learning (mil) @xcite @xcite approach. the model is trained jointly on mention detection, coreference resolution, entity classification and relation extraction (figure 1 ). while we follow best practices for the first three tasks, we propose a novel representation for relation extraction, which combines global entity-level representations with localized mention-level ones. we present experiments on the docred @xcite dataset for entity-level relation extraction. though it is arguably simpler compared to recent graph propagation models @xcite or special pre-training @xcite , our approach achieves state-of-the-art results. we also report the first results for end-to-end relation extraction on docred as a reference for future work. in ablation studies we show that (1) combining a global and local representations is beneficial, and (2) that joint training appears to be on par with separate per-task models. ## related work relation extraction is one of the most studied natural language processing (nlp) problems to date. most approaches focus on classifying the relation between a given entity mention pair. here various neural network based models, such as rnns @xcite , cnns @xcite , recursive neural networks @xcite or transformer-type architectures @xcite have been investigated. however, these approaches are usually limited to local, intrasentence, relations and are not suited for documentlevel, inter-sentence, classification. since complex relations require the aggregation of information distributed over multiple sentences, document-level relation extraction has recently drawn attention (e.g. @xcite @xcite . still, these models rely on specific entity mentions to be given. while progress in the joint detection of entity mentions and intra-sentence relations has been made @xcite @xcite , the combination of coreference resolution with relation extraction for entity-level reasoning in a single, jointly-trained, model is widely unexplored. ## approach jerex processes documents containing multiple sentences and extracts entity mentions, clusters them to entities, and outputs types and relations on entity level. jerex consists of four task-specific components, which are based on the same encoder and mention representations, and are trained in a joint manner. an input document is first tokenized, yielding a sequence of n byte-pair encoded (bpe) @xcite tokens. we then use the pretrained transformer-type network bert @xcite to obtain a contextualized embedding sequence (e 1 , e 2 , ...e n ) of the document. since our goal is to perform end-to-end relation extraction, neither entities nor their corresponding mentions in the document are known in inference. ## experiments we evaluate jerex on the docred dataset @xcite . docred ist the most diverse relation extraction dataset to date (6 entity and 96 relation types). it includes over 5,000 documents, each consisting of multiple sentences. according to yao et al. (2019), docred requires multiple types of reasoning, such as logical or common-sense reasoning, to infer relations. note that previous work only uses docred for relation extraction (which equals our relation classifier component) and assumes entities to be given (e.g. wang et al. 2019; nan et al. 2020). on the other hand, docred is exhaustively annotated with mentions, entities and entity-level relations, making it suitable for end-to-end systems. therefore, we evaluate jerex both as a relation classifier (to compare it with the state-of-the-art) and as a joint model (as reference for future work on joint entity-level relation extraction). while prior joint models focus on mention-level relations (e.g. @xcite @xcite , we extend the strict evaluation setting to entity level: a mention is counted as correct if its span matches a ground truth mention span. an entity cluster is considered correct if it matches the ground truth cluster exactly and the corresponding mention spans are correct. likewise, an entity is considered correct if the cluster as well as the entity type matches a ground truth entity. lastly, we count a relation as correct if its argument entities as well as the relation type are correct. we measure precision, recall and micro-f1 for each sub-task and report micro-averaged scores. ## conclusions we have introduced jerex, a novel multi-task model for end-to-end relation extraction. in contrast to prior systems, jerex combines entity mention localization with coreference resolution to extract entity types and relations on an entity level. we report first results for entity-level, end-to-end, relation extraction as a reference for future work. furthermore, we achieve state-of-the-art results on the docred relation extraction task by enhancing multi-instance reasoning with global entity representations and a localized context, outperforming several more complex solutions. we showed that training a single model jointly on all subtasks instead of using a pipeline approach performs roughly on par, eliminating the need of training separate models and accelerating inference. one of the remaining shortcomings lies in the detection of false positive relations, which may be expressed according to the entities' types but are actually not expressed in the document. exploring options to reduce these false positive predictions seems to be an interesting challenge for future work.