Enhancing Contextual Word Representations Using Embedding of Neighboring Entities in Knowledge Graphs (2022)

Abstract: Pre-trained language models (PLMs) such as BERT and RoBERTa have dramatically improved the performance of various natural language processing tasks. Although these models are trained on large amounts of raw text, they have no explicit grounding in real-world entities. Knowledge graphs (KGs) are manually annotated with factual knowledge and store the relations between nodes corresponding to entities as labeled edges. This paper proposes a mechanism called KG-attention, which integrates the structure of a KG into recent PLM architectures. Unlike the existing PLM+KG integration methods, KG-attention generalizes the embeddings of neighboring entities using the relation embeddings; accordingly, it can handle relations between unconnected entities in the KG. Experimental results demonstrated that our method achieved significant improvements in a relation classification task, an entity typing task, and several language comprehension tasks.

## introduction pre-trained language models (plms) have significantly improved the performance of various natural language processing (nlp) tasks @xcite . although these models are trained on large amounts of raw text, they have no explicit grounding in real-world entities. figure 1 shows a causal reasoning task on which plm fails. the performance can be improved by representing factual knowledge as various relations between entities, e.g., "engine is part of a car", in the plm. knowledge graphs (kgs) are manually annotated with factual knowledge and store the relations between the nodes corresponding to entities as labeled edges (e.g., hasa, isa, and partof ) @xcite @xcite . although the various relations recorded in kgs can potentially improve the performance of plms, kgs are graphs and structurally unsuitable for direct incorporation into plms. some of the more suitable forms of factual knowledge are pre-trained kg embeddings @xcite @xcite . kg embeddings embed two entities registered in a kg such that the corresponding embedding vectors are at a specific relative position depending on the relation between the entities. the embedding vectors corresponding to the entities are called kg entity embeddings and the specific relative positions are called relation embeddings. in addition to avoiding the direct use of structural encoding of kgs, kg embeddings can be generalized; that is, entities having a specific relation to a query entity can be estimated even when their relation is not actually recorded in the kgs @xcite . for example, if there is no partof connection between car and engine in the kg, the embedding of engine can be estimated from the embedding of car and the relation embedding of partof. one remaining problem is the incorporation of kg entity embeddings and relation embeddings into plms. recently, several plm and kg integration methods have been proposed @xcite @xcite @xcite . for example, @xcite concatenated embedding vectors obtained from plm word embeddings and kg entity embeddings for downstream tasks. @xcite jointly trained a masked language model (mlm) and a kg embedding model to align factual knowledge and language representation in the same semantic space. although kg entity embeddings have been utilized, how the informative relation embeddings should be incorporated into plms is not obvious. actually, the relation embeddings have been ignored, while kg entity embeddings have been well utilized. herein, we propose integrating plms and kgs through attention mechanisms (ilka), which handles both the kg entity embeddings and the relation embeddings through our attention mechanism (kg-attention). a key feature of kg-attention is that it fully utilizes the generalization ability of pre-trained kg embeddings, meaning that kgattention can handle relations between entities not connected in the original kg. our contributions are summarized below. knowledge graph embedding kgs such as wordnet @xcite and conceptnet (speer approximate formula transe @xcite vr(t) ≈ vr(h) + v(r) rotate @xcite v(t) ≈ v(h) • v(r) pairre @xcite several learning methods have been proposed for embedding kgs into a continuous vector space while preserving the relational structure @xcite @xcite @xcite . in most kg embedding models, the triplet (h, r, t) is embedded to satisfy the following: one benefit of kg embedding is that entities with a specific relation to the query entity can be retrieved even when the relation between the entities is not connected in the kg @xcite . for example, in figure 2 , entities e connecting the "isa" relation with car can be estimated by extracting the entities satisfying v kg (e) ≈ ϕ h (v kg (car), v r (isa)). v kg (e) encodes not only words having the "isa" relation with car in the kg (i.e., automobile, vehicle), but also words not having the "isa" relation with car but having similar meanings to automobile and vehicle (i.e., motorcar in figure 2 ). in this paper, the vector ϕ h (v kg (w), v r (r)) is called the kg-neighbor embeddings of entity w and the entities corresponding to the kg-neighbor embeddings (i.e., e) are called kg-neighbors. note that kg-neighbors may be unconnected to entity w in the original kg. for some kg embedding methods such as transe, we can also consider kg-neighbor embeddings as ϕ t (v kg (w), v r (r)), where w corresponds to a tail entity. in figure 2 , the kg-neighbor embeddings in the "partof" relation with car (i.e., seats, ignition, engine) are represented by ϕ in the following, ϕ refers to either ϕ h or ϕ t . integrating plms and kgs several recent studies have integrated kgs with plm. in some studies, the kgs and plm were combined to perform specific downstream tasks @xcite @xcite . @xcite proposed an end-to-end question answering (qa) model that leverages plm and kg by scoring the relevance of kg nodes conditional on a given qa context. these downstream task-focused methods are effective and robust for the given task, but they do not enhance plm for general purposes. in another approach, plm and kg are integrated at the model level @xcite @xcite @xcite @xcite . @xcite concatenated plm word embeddings and kg entity embeddings and @xcite trained an mlm by concatenating the entity sequences obtained from a kg and input word sequences. however, these plm+kg methods are limited in that they do not effectively employ relation embeddings. a relation embedding encodes the semantic direction of the predefined relations in kgs and plays an important role in representing neighboring entities in kg and generalizing relations between entities not included in the kg. the existing methods either do not consider relation embeddings @xcite or cannot handle the relations between entities not connected in the kg @xcite . ## conclusion we proposed the kg-attention mechanism and developed the ilka model to integrate kgs into plms in a consistent manner. unlike the existing plm+kg integration methods, kg-attention generalizes the embeddings of neighboring entities using the relation embeddings, and selects a proportion of these embeddings through the attention mechanism. accordingly, it can handle relations between unconnected entities in the kg. in the relation classification and entity typing experiments, ilka yielded higher recall and f1 scores than conventional plm+kg methods. in the glue task, factual knowledge produced no negative effect on language comprehension by ilka. in addition, the higher scores for copa than for the plm alone suggest that our method adequately processes complex language comprehension tasks. our proposed method has one limitation: when the number of relations is large, the dimension of the attention enlarges accordingly. to avoid this size explosion, we could employ the weighted sums of relation embeddings as keys and queries instead of arranging all relation embeddings in the direction of attention.