KLM o: Knowledge Graph Enhanced Pretrained Language Model with Fine-Grained Relationships (2021)

Abstract: Interactions between entities in knowledge graph (KG) provide rich knowledge for language representation learning. However, existing knowledge-enhanced pretrained language models (PLMs) only focus on entity information and ignore the fine-grained relationships between entities. In this work, we propose to incorporate KG (including both entities and relations) into the language learning process to obtain KG-enhanced pretrained Language Model, namely KLMo. Specifically, a novel knowledge aggregator is designed to explicitly model the interaction between entity spans in text and all entities and relations in a contextual KG. An relation prediction objective is utilized to incorporate relation information by distant supervision. An entity linking objective is further utilized to link entity spans in text to entities in KG. In this way, the structured knowledge can be effectively integrated into language representations. Experimental results demonstrate that KLMo achieves great improvements on several knowledge-driven tasks, such as entity typing and relation classification, comparing with the state-of-the-art knowledge-enhanced PLMs.

## introduction knowledge graph (kg) with entities and relations provides rich knowledge for language learning @xcite . recently, researchers have explored to incorporate kg information into plms @xcite to enhance language representations, such as ernie-thu @xcite , wklm @xcite , kepler @xcite , @xcite , bert-mk @xcite and kalm @xcite , . however, they only utilize entity information and ignore the finegrained relationships between entities. the finegrained semantic information of relations between entities is also critical to language representation learning. taking figure 1 as example, for entity typing, without explicitly knowing the fine-grained relation guest between lang lang and trio of happiness, which is different from the relation performer between lang lang and bbc proms, it's impossible to correctly predict the type of trio of happiness as tv show, since the input sentence literally implies that trio of happiness belongs to the same type as bbc proms. the fine-grained relations between entities in kg provide specific constraint on entities, thus can play an important role in language learning for knowledge-driven tasks. to explicitly incorporate entities and finegrained relations in kg into plms, one main challenge we are faced with is the text-knowledge alignment (tka) problem: it's difficult to make token-relation and token-entity alignments for the fusion of text and knowledge. to handle this problem, the kg-enhanced pretrained language model (klmo) is proposed to integrate kg (i.e. both entities and fine-grained relations) into the language representation learning. the main component of klmo is a knowledge aggregator, which is responsible for text and knowledge information fusion from two individual embedding spaces, i.e. token embedding space and kg embedding space. the knowledge aggregator models the interaction between entity spans in text and all entities and relations in a contextual kg via an entity span-level cross-kg attention to make tokens attend to highly related entities and relations in kg. based on the kg-enhanced token representations, a relation prediction objective is utilized to predict the relation of each pair of entities in text based on the distant supervision of kg. furthermore, an entity linking objective is utilized to predict entities in kg based on the corresponding entity spans in text. the relation prediction and entity linking objectives are the key to the integration of kg information into text representations. we conduct experiments on two chinese knowledge-driven nlp tasks, i.e. entity typing and relation classification. the experimental results demonstrate that klmo obtains large improvements over bert and existing knowledgeenhanced plms, by taking full advantage of a structured kg including both entities and fine-grained relations. we also will publish a chinese entity typing dataset for the evaluation of chinese plms. ## model description as shown in figure 2 , klmo is designed as a multilayer transformer-based @xcite model, which accepts a token sequence and the entities and relations in its contextual kg as input. the token sequence is firstly encoded by a multi-layer transformer-based text encoder. the output of the text encoder is further used as input for the knowledge aggregator that fuses the knowledge embeddings of entities and relations into the token sequence to obtain kg-enhanced token representations. based on the kg-enhanced representations, novel relation prediction and entity linking objectives are jointly optimized as the pre-training objectives, which help incorporate high-related entity and relation information in the kg into the text representations. ## experiments although the dataset are very difficult, we can still observe that: (1) all knowledge-enhanced plms largely improve the precision and micro-f1 scores over bert baseline, which shows that both entity information and kg information can enhance language representations and accordingly prompt the performance of relation classification. ## conclusion in this paper, we propose a novel kg-enhanced pretrained language model klmo to explicitly integrate kg entities and fine-grained relations into the language representation learning. accordingly, the novel knowledge aggregator is designed to handle the heterogeneous information fusion and textknowledge alignment problems. further, the relation prediction and entity linking objectives are jointly optimized to encourage the knowledge information integration. the experiment results show that klmo outperforms the other state-of-the-art knowledge-enhanced plms, which validates the intuition that fine-grained relationships in kg can enhance the language representation learning and benefit some knowledge-driven nlp tasks.