Learning Knowledge-Enhanced Contextual Language Representations for Domain Natural Language Understanding (2023)

Abstract: Knowledge-Enhanced Pre-trained Language Models (KEPLMs) improve the performance of various downstream NLP tasks by injecting knowledge facts from large-scale Knowledge Graphs (KGs). However, existing methods for pre-training KEPLMs with relational triples are difficult to be adapted to close domains due to the lack of sufficient domain graph semantics. In this paper, we propose a Knowledge-enhanced language representation learning framework for various closed domains (KANGAROO) via capturing the implicit graph structure among the entities. Specifically, since the entity coverage rates of closed-domain KGs can be relatively low and may exhibit the global sparsity phenomenon for knowledge injection, we consider not only the shallow relational representations of triples but also the hyperbolic embeddings of deep hierarchical entity-class structures for effective knowledge fusion. Moreover, as two closed-domain entities under the same entity-class often havel locally dense neighbor subgraphs counted by max point biconnected component, we further propose a data augmentation strategy based on contrastive learning over subgraphs to construct hard negative samples of higher quality. It makes the underlying KELPMs better distinguish the semantics of these neighboring entities to further complement the global semantic sparsity. In the experiments, we evaluate KANGAROO over various knowledge-aware and general NLP tasks in both full and few-shot learning settings, outperforming various KEPLM training paradigms performance in closed-domains significantly.

## introduction the performance of downstream tasks @xcite ) can be further improved by keplms @xcite @xcite @xcite which leverage rich knowledge triples from kgs to enhance language representations. in the literature, most knowledge injection approaches for keplms can be roughly categorized into two types: knowledge embedding and joint learning. (1) knowledge embedding-based approaches aggregate representations of knowledge triples learned by kg embedding models with plms' contextual representations @xcite @xcite . (2) joint learning-based methods convert knowledge triples into pre-training sentences without introducing other parameters for knowledge encoders @xcite . these works mainly focus on building keplms for the open domain based on large-scale kgs @xcite @xcite . despite the success, these approaches for building open-domain keplms can hardly be migrated directly to closed domains because they lack the in-depth modeling of the characteristics of closeddomain kgs @xcite @xcite . as in figure 1 , the coverage ratio of kg entities w.r.t. plain texts is significantly lower in closed domains than in open domains, showing that there exists a global sparsity phenomenon for domain knowledge injection. this means injecting the retrieved few relevant triples directly to plms may not be sufficient for closed domains. we further notice that, in closed-domain kgs, the ratios of maximum-point biconnected components are much higher, which means that entities under the same entity-class in these kgs are more densely interconnected and exhibit a local density property. hence, the semantics of these entities are highly similar, making the underlying keplms difficult to capture the differences. yet a few approaches employ continual pre-training over domain-specific corpora @xcite @xcite , or devise pre-training objectives over in-domain kgs to capture the unique domain semantics (which requires rich domain expertise) @xcite . therefore, there is a lack of a simple but effective unified framework for learning keplms for various closed domains. to overcome the above-mentioned issues, we devise the following two components in a unified framework named kangaroo. it aggregates the above implicit structural characteristics of closeddomain kgs into keplm pre-training: due to the semantic deficiency caused by the global sparsity phenomenon, we utilize the poincaré ball model @xcite to obtain the hyperbolic embeddings of entities based on the entity-class hierarchies in closed-domain kgs to supplement the semantic information of target entities recognized from the pre-training corpus. it not only captures richer semantic connections among triples but also implicit graph structural information of closed-domain kgs to alleviate the sparsity of global semantics. in the experiments, we compare kangaroo against various mainstream knowledge injection paradigms for pre-training keplms over two closed domains (i.e., medical and finance). the results show that we gain consistent improvement in both full and few-shot learning settings for various knowledge-intensive and general nlp tasks. ## analysis of closed-domain kgs in this section, we analyze the data distributions of open and closed-domain kgs in detail. specifically, we employ openkg foot_0 as the data source to construct a medical kg, denoted as medkg. in addition, a financial kg (denoted as finkg) is constructed from the structured data source from an authoritative financial company in china foot_1 . as for the open domain, cn-dbpedia foot_2 is employed for further data analysis, which is the largest open source chinese kg constructed from various chinese encyclopedia sources such as wikipedia. to illustrate the difference between open and closed-domain kgs, we give five basic indicators @xcite , which are described in detail in appendix a due to the space limitation. from the statistics in ## kangaroo framework in this section, we introduce the various modules of the model in detail and the notations are described in appendix b.5 due to the space limitation. the whole model architecture is shown in figure 3 . ## experiments in this section, we conduct extensive experiments to evaluate the effectiveness of the proposed framework. due to the space limitation, the details of datasets and model settings are shown in appendix b and the baselines are described in appendix c. ## related work open-domain keplms. we summarize previous keplms grouped into four types: (1) knowledge enhancement by entity embeddings @xcite @xcite . closed-domain keplms. due to the lack of in-domain data and the unique distributions of domain-specific kgs @xcite , previous works of closed-domain ke-plms focus on three domain-specific pre-training paradigms. (1) pre-training from scratch. for example, pubmedbert @xcite derives the domain vocabulary and conducts pre-training using solely in-domain texts, alleviating the problem of out-of-vocabulary and perplexing domain terms. ## conclusion in this paper, we propose a unified closeddomain framework named kangaroo to learn knowledge-aware representations via implicit kgs structure. we utilize entity enrichment with hyperbolic embeddings aggregator to supplement the semantic information of target entities and tackle the semantic deficiency caused by global sparsity. additionally, we construct high-quality negative samples of knowledge triples by data augmentation via local dense graph connections to better capture the subtle differences among similar triples.