BERT - MK : Integrating Graph Contextualized Knowledge into Pre-trained Language Models (2020)

Abstract: Complex node interactions are common in knowledge graphs (KGs), and these interactions can be considered as contextualized knowledge exists in the topological structure of KGs. Traditional knowledge representation learning (KRL) methods usually treat a single triple as a training unit, neglecting the usage of graph contextualized knowledge. To utilize these unexploited graph-level knowledge, we propose an approach to model subgraphs in a medical KG. Then, the learned knowledge is integrated with a pre-trained language model to do the knowledge generalization. Experimental results demonstrate that our model achieves the state-of-the-art performance on several medical NLP tasks, and the improvement above MedERNIE indicates that graph contextualized knowledge is beneficial.

## methodology our model consists of two modules: the knowledge learning module and the language model pretraining module. the first module is utilized to learn graph contextualized knowledge existing in kgs, and the second one integrates the learned knowledge into the language model for knowledge generalization. the details will be described in the following subsections. ## related work pre-trained language models represented by @xcite , gpt @xcite and bert @xcite have attracted great attention, and a large number of variant models have been proposed. among these studies, some researchers devote their efforts to introducing knowledge into language models @xcite @xcite . ernie-baidu @xcite introduces new masking units such as phrases and entities to learn knowledge information in these masking units. as a reward, syntactic and semantic information from phrases and entities is implicitly integrated into the language model. furthermore, a different knowledge information is explored in ernie-tsinghua @xcite , which incorporates knowledge graph into bert to learn lexical, syntactic and knowledge information simultaneously. @xcite introduce entity replacement checking task into the pre-trained language model, and improve several entity-related downstream tasks, such as question answering and entity typing. @xcite propose a plug-in way to infuse knowledge into language models, and their method keeps different kinds of knowledge in different adapters. the knowledge information introduced by these methods does not pay much attention to the graph contextualized knowledge in the kg. recently, several krl methods have attempted to introduce more contextualized information into knowledge representations. relational graph convolutional networks (r-gcns) @xcite is proposed to learn entity embeddings from their incoming neighbors, which greatly enhances the information interaction between related triples. @xcite further extend the information flow from 1-hop in-entities to n-hop during the learning process of entity representations, and achieves the sota performance on multiple relation prediction datasets, especially for the ones containing higher in-degree nodes. we believe that the information contained in knowledge graphs is far from being sufficiently exploited. in this study, we develop an approach to integrate more graph contextualized information, which models subgraphs as training samples. this module has the ability to model any information in the kg. in addition, this learned knowledge is integrated into the language model to obtain an enhanced version of the medical pre-trained language model. ## conclusion and future work we propose a novel approach to learn more comprehensive knowledge, focusing on modeling subgraphs in the knowledge graph by a knowledge learning module. additionally, the learned medical knowledge is integrated into the pre-trained language model, which outperforms bert-base and another two domain-specific pre-trained language models on several medical nlp tasks. our work validates the intuition that medical knowledge is beneficial to some medical nlp tasks and provides a preliminary exploration for the application of medical knowledge. in the follow-up work, some knowledge-guided tasks will be used to validate the effectiveness of the knowledge learning module gcke. moreover, we will explore some other knowledge injection ways to combine medical knowledge with language models, such as multi-task learning. more subgraph sampling strategies need to be explored, such as r-ego subgraph @xcite and degreedependent subgraph.