Explicitly Capturing Relations between Entity Mentions via Graph Neural Networks for Domain-specific Named Entity Recognition (2021)

Abstract: Named entity recognition (NER) is well studied for the general domain, and recent systems have achieved human-level performance for identifying common entity types. However, the NER performance is still moderate for specialized domains that tend to feature complicated contexts and jargonistic entity types. To address these challenges, we propose explicitly connecting entity mentions based on both global coreference relations and local dependency relations for building better entity mention representations. In our experiments, we incorporate entity mention relations by Graph Neural Networks and show that our system noticeably improves the NER performance on two datasets from different domains. We further show that the proposed lightweight system can effectively elevate the NER performance to a higher level even when only a tiny amount of labeled data is available, which is desirable for domain-specific NER.

## introduction named entity recognition (ner) has been well studied for the general domain, and recent systems have achieved close to human-level performance for identifying a small number of common ner types, such as person and organization, mainly benefiting from the use of neural network models @xcite and pretrained language models (lms) @xcite . however, the performance is still moderate for specialized domains that tend to feature diverse and complicated contexts as well as a richer set of semantically related entity types (e.g., cell, tissue, organ etc. for the biomedical domain). with these challenges in view, we hypothesize that being aware of the re-occurrences of the same entity as well as semantically related entities will lead to better ner performance for specific domains. therefore, we propose to explicitly connect entity mentions in a document that are coreferential or in a tight semantic relation to better learn entity mention representations. precisely, as shown in figure 1 , we first connect repeated mentions of the same entity even if they are sentences away. for example, the named entity "tumor vasculature" appears both in the title and sentence s6 but in quite different contexts. connecting the repeated mentions in a document enables the integration of contextual cues as well as enables consistent predictions of their entity types. second, we also connect entity mentions based on sentence-level dependency relations to effectively identify semantically related entities. for example, the two entities in sentence s3, "bone marrow" of the type multi-tissue structure and "endothelial progenitors" of the type cell, are the subject and object of the predicate "contains" respectively in the dependency tree. if the system can reliably predict the type of one entity, we can infer the type of the other entity more easily, knowing that they are closely related on the dependency tree. we incorporate both relations by using graph neural networks (gnns), specifically, we use the graph attention networks (gats) @xcite that have been shown effective for a range of tasks @xcite . empirical results show that our lightweight method can learn better word representations for sequence tagging models and further improve the ner performance over strong lms-based baselines on two datasets, the anatem @xcite dataset from the biomedical domain and the mars @xcite dataset from the planetary science domain. in addition, considering the lack of annotations challenge for domain-specific ner, we plot learning curves and show that leveraging relations between entity mentions can effectively and consistently improve the ner performance when limited annotations are available. ## related work ner research has a long history and recent approaches @xcite @xcite using neural network models like bilstm-cnn-crf (ma and hovy, 2016) and contextual embeddings such as bert @xcite and flair @xcite have improved the ner performance in the general domain to the human-level. however, the ner performance for specific domains is still moderate due to the challenges of limited annotations and dealing with complicated domain-specific contexts. we aim to further improve ner performance by considering coreference relations and semantic relations between entity mentions. this is in contrast to the usual way of thinking about ner as an up-stream task conducted before coreference resolution or entity relation extraction. the idea aligns with recent works that conduct joint inferences among multiple information extraction tasks @xcite @xcite @xcite , including ner, coreference resolution and relation extraction, by mining dependencies among the extractions. however, joint inference approaches require annotations for all the target tasks and aim to improve performance for all the tasks as well, while our lightweight approach aims to improve the performance of the basic ner task requiring no additional annotations (usually unavailable for specific domains). our approach is also related to several recent neural approaches for ner that encourage label dependencies among entity mentions. the pooled flair model @xcite proposed a global pooling mechanism to learn word representations. @xcite used a coreference layer with a regularizer to harmonize word representations. closely related to our work, @xcite used graph neural nets to capture repetitions of the same word as well, but in a denser graph that includes edges between adjacent words and is meant to completely overlay the lower encoding layers. memory networks @xcite were also used to store and refine predictions of a base model by considering repetitions or co-occurrences of words. in addition, dependency relations have been commonly used to connect entities for relation extraction @xcite , but we aim to better infer the type of an entity by associating it with other closely related entities in a sentence. ## model architecture our system with entity relation graphs (enrel-g) mainly contains 5 layers as in figure 2 : an embedding layer, an encoding layer, a gnns layer, a fusion layer, and a decoding layer. ## experiments 4 we test our model on two domain-specific datasets: the anatem (pyysalo and ananiadou, 2014) from the biomedical domain and the mars @xcite ) from the planetary science domain. the anatem has annotated 12 types of entities in 1,212 documents with 13,701 entity mentions; the mars has 117 longer documents with 4,458 entity mentions containing 3 types. ## conclusion in this work, we explicitly capture the global coreference and local dependency relations between entity mentions, and use graph neural nets to incorporate the relations to improve domain-specific ner tasks. experimental results on two datasets show the effectiveness of this lightweight approach. we also find that the selection of entity relations is important to the system performance. future work may consider about using gnns to incorporate external knowledge for performance improvement. for the flair and pooled flair baselines, we use the pubmed version (pretrained on the biomedical corpus) for the anatem dataset and the general english version (pretrained on the english news articles) for the mars dataset. particularly, for the pooled flair model, we set the mean pooling mechanism to calculate the average of embeddings for multiple occurrences of a word, and then use it as the representation for the word. for the tuning bert baselines, we use biobert-base v1.1 for the anatem dataset and scibert-scivocab-uncased for the mars dataset. for our enrel-g system, we keep the embeddings layer the same as the tuning bert baselines. as for the gnns layer, we use one layer of the graph attention mechanism with 4 heads, and each head has a hidden dimension of 128. for the optimization related parameters, as in the we train all the systems on a single nvidia geforce gtx 2080ti gpu. we set the maximum epoch as 100 and use the best-performed model on the development set to evaluate the test data.