Enhancing Ontology Knowledge for Domain-Specific Joint Entity and Relation Extraction (2023)

Abstract: “Pre-trained language models (PLMs) have been widely used in entity and relation extractionmethods in recent years. However, due to the semantic gap between general-domain text usedfor pre-training and domain-specific text, these methods encounter semantic redundancy anddomain semantics insufficiency when it comes to domain-specific tasks. To mitigate this issue,we propose a low-cost and effective knowledge-enhanced method to facilitate domain-specificsemantics modeling in joint entity and relation extraction. Precisely, we use ontology and entitytype descriptions as domain knowledge sources, which are encoded and incorporated into thedownstream entity and relation extraction model to improve its understanding of domain-specificinformation. We construct a dataset called SSUIE-RE for Chinese entity and relation extractionin space science and utilization domain of China Manned Space Engineering, which contains awealth of domain-specific knowledge. The experimental results on SSUIE-RE demonstrate theeffectiveness of our method, achieving a 1.4% absolute improvement in relation F1 score overprevious best approach. Introduction”

## introduction extracting relational triples from plain text is a fundamental task in information extraction and it's an essential step in knowledge graph (kg) construction @xcite . traditional methods perform named entity recognition (ner) and relation extraction (re) in a pipelined manner, that is, first extract entities, and then perform relation classification on entity pairs @xcite @xcite . however, since the entity model and relation model are modeled separately, pipelined methods suffer from the problem of error propagation. to address this issue, some joint methods have been proposed @xcite @xcite @xcite . the task of joint entity and relation extraction aims to simultaneously conduct entity recognition and relation classification in an end-to-end manner. in recent years, with the development of pre-trained language models (plms) such as bert @xcite and gpt @xcite , many entity and relation extraction methods have adopted the paradigm of pre-training and fine-tuning. they utilize plms to encode the contextual representations of input text and design various downstream models for task-specific fine-tuning. however, when employed for domain-specific entity and relation extraction, this paradigm suffers from problems of semantic redundancy and insufficiency of domain-specific semantics, particularly in highly specialized domains. on the one hand, plms are usually trained on general-domain corpora, which results in a significant amount of redundant semantic information that may not be relevant to specific domains and a lack of sufficient domain-specific semantic information. on the other hand, modeling domain-specific information in this paradigm depends primarily on the role of downstream model and domain-specific labeled data in the fine-tuning stage. however, due to the significantly smaller parameter size of downstream model compared to plms and the limited availability of domain-specific labeled data, the effectiveness of domain-specific semantic information modeling is constrained. consequently, some methods attempt to incorporate domain knowledge into entity and relation extraction models to enhance the their comprehension of domain-specific information . these methods can be broadly categorized into two groups according to how knowledge is introduced: pre-training domain-specific language models and integrating domain-specific knowledge graph information into models. methods of domain-specific pre-training utilize large-scale domain corpora to facilitate continuous pre-training of existing general-domain language models @xcite @xcite or, alternatively, to perform domain-specific pre-training from scratch @xcite . however, in certain specialized domains, there may be a dearth of enough domain-specific corpora to support domain-specific pre-training. another category of methods involve integrating domainspecific knowledge graph information into models, where entity mentions in input text are linked to the corresponding entities in knowledge graph, and then the relevant information of the linked entities in the knowledge graph is incorporated into models @xcite @xcite . some of these knowledge graph integration methods are designed simply for the task of relation extraction (re) where the entities in the sentence are pre-specified, rather than the task of joint entity and relation extraction. in addition, a prerequisite for this kind of approaches is the availability of a well-constructed domain-specific knowledge graph, which is scarce and expensive for some highly specialized domains. in this study, we explore how to incorporate domain knowledge for the task of joint entity and relation extraction in space science and utilization domain of china manned space engineering. due to the lack of sufficient domain-specific corpora to support the pre-training of large-scale language models and the absence of well-constructed domain-specific knowledge graphs, the aforementioned approaches cannot be directly used for domain knowledge enhancement. we propose an ontology-enhanced joint entity and relation extraction method (ontore) for space science and utilization domain. the predefined domainspecific ontology involves many highly specialized entity types that interconnected by different semantic relations, which frames the knowledge scope and defines the knowledge structure in this domain, so it is an appropriate source of domain knowledge. the ontology can be formalized as a graph structure containing nodes and edges, where nodes represent entity types and edges represent relation types. furthermore, drawing inspiration from the manner in which humans comprehend specialized terminology, we add descriptions for each entity type in the ontology to enhance the semantic information of entity types. we serialize the ontology graph and then adopt an ontology encoder to learn the embeddings of ontology knowledge. the encoded ontology features are fused with input sentence features, and then the entity and relation extraction is carried out under the guidance of ontology knowledge. to evaluate our model, we construct a dataset called ssuie (space science and utilization information extraction), which contains rich knowledge about space science and utilization in the aerospace field. this work exclusively pertains to the problem of entity and relation extraction, therefore our model was evaluated on the subset of ssuie specifically designed for entity and relation extraction, namely ssuie-re. the main contributions of this work are summarized below: 1. a dataset named ssuie-re is proposed for chinese entity and relation extraction in space science and utilization domain of china manned space engineering. the dataset is enriched with domainspecific knowledge, which contains 19 entity types and 36 relation types. 2. an ontology-enhanced method for domain-specific joint entity and relation extraction is proposed, which substantially enhances domain knowledge without the need of domain knowledge graphs or large-scale domain corpora. experimental results show that our model outperforms previous state-of-the-art works in terms of relation f1 score. 3. the effect of domain ontology knowledge enhancement is carefully examined. our supplementary experiments show that the ontology knowledge can improve the extraction of relations with varying degrees of domain specificity. notably, the benefit of ontology knowledge augmentation is more evident for relations with higher domain specificity. ## method in this section, we introduce the architecture of ontore. as shown in figure 1 , the model mainly includes four parts: knowledge source, knowledge serialization, knowledge encoding and knowledge fusion. in the following subsections, we provide a detailed description of each component.