H itt ER : Hierarchical Transformers for Knowledge Graph Embeddings (2021)

Abstract: This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical Transformer model to jointly learn Entity-relation composition and Relational contextualization based on a source entity’s neighborhood. Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block. We further design a masked entity prediction task to balance information from the relational context and the source entity itself. Experimental results show that HittER achieves new state-of-the-art results on multiple link prediction datasets. We additionally propose a simple approach to integrate HittER into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets.

## introduction knowledge graphs (kg) are a major form of knowledge bases where knowledge is stored as graphstructured data. because of their broad applications in various intelligent systems including natural language understanding @xcite @xcite and reasoning @xcite @xcite , learning representations of knowledge graphs has been studied in a large body of literature. to learn high quality representations of knowledge graphs, many researchers adopt the idea of mapping the entities and relations in a knowledge graph to points in a vector space. these knowledge graph embedding (kge) methods usually leverage geometric properties in the vector space, such as translation @xcite , bilinear transformations @xcite , or rotation @xcite . multi-layer convolutional networks are also used for kge @xcite . such kge methods are conceptually simple and can be applied to tasks like factoid question answering @xcite and language modeling @xcite . however, it is rather challenging to encode all of the information about an entity into a single vector. for example, to infer the missing object in the incomplete @xmath0, county, ?> (figure 1 ), traditional kge methods rely on the geographic information stored in the embedding of sunnyvale. while we can read such information from its graph context, e.g., from a neighbor node that represents the state it belongs to (i.e., california). in this way, we allow the model to store and utilize information about an entity via its relational context. to implement this process, previous work uses graph neural networks (gnn) or attention-based approaches to learn representations based on both entities and their graph context @xcite @xcite . however, these methods are usually restricted in expressiveness because of the shallow network architecture they use. 1 in this paper, we present hitter, a deep hierarchical transformer model to learn representations of entities and relations in a knowledge graph jointly by aggregating information from graph neighborhoods. although prior work shows transformers can learn relational knowledge from large amounts of unstructured textual data @xcite , hitter explicitly operates over structured inputs using a hierarchical architecture. essentially, hitter consists of two levels of transformer blocks. as shown in figure 2 , the bottom block provides relation-dependent entity embeddings for the neighborhood around an entity and the top block aggregates information from its graph context. to ensure hitter work across graphs of different properties, we further design a masked entity prediction task to balance the contextual relational information and information from the training entity itself. we evaluate the proposed method using the link prediction task, which is one of the canonical tasks in statistical relational learning (srl). link prediction (essentially kg completion) serves as a good proxy to evaluate the effectiveness of learned graph representations, by measuring the ability of a model to generalize relational knowledge stored in training graphs to unseen facts. meanwhile, it has an important application to knowledge graph completion given the fact that most of the knowledge graphs are still highly incomplete @xcite . our approach achieves new state-of-the-art results on two standard benchmark datasets: fb15k-237 @xcite and wn18rr @xcite . unlike the previous shallow kge methods that cannot be trivially utilized by widely used transformer-based models for language tasks (peters et al., 2019), our approach benefits from the unified transformer architecture and its extensibility. as a case study, we show how to integrate the learned representations of hitter into pre-trained language models like bert @xcite . our experiments demonstrate that hitter significantly improves bert on two freebase factoid question answering (qa) datasets: freebaseqa @xcite and webques-1 gnn methods' depth is tied to their receptive fields and thus constrained by over-smoothing issues @xcite . @xcite . our experimental code as well as multiple pretrained models are publicly available. 2 ## hitter we introduce our proposed hierarchical transformer model (figure 2 ) in this section. in section 2.1, we provide the background about how link prediction can be done with a simple transformer scoring function. we then describe the detailed architecture of our proposed model in section 2.2. finally, we discuss our strategies to learn balanced contextual representations of an entity in section 2.3. ## factoid qa experiments in addition to hitter's superior intrinsic evaluation results, in this section, we conduct a case study on the factoid question answering (qa) task to demonstrate hitter's potential to enhance popular pre-trained transformer-based language models' performance on knowledge-intensive tasks. as a transformer-based model, hitter enables us to integrate its multilayer knowledge representation into other transformer models (bert in our case) using the multi-head attention mechanism. in each bert layer, after the original self-attention module we add a cross-attention module where the queries come from the previous bert layer while the keys and values come from the output of a corresponding hitter layer @xcite , so that hitter's knowledge information can flow into bert (figure 4 ). we perform experiments on two factoid qa datasets: freebaseqa @xcite and @xcite , both pertaining to facts on freebase. each question in the two datasets is labeled with a context entity and an inferred relation between the context entity and the answer entity, which we use for preparing the entity and relation inputs for hitter. to better exploit the knowledge in bert, we follow its pretraining task to create a word-based qa setting, where factoid questions are converted to cloze questions by appending the special [mask] tokens to the end. both models are trained to recover these [mask] tokens to the original words. 9 we use the bert-freebaseqa webquestionsp full filtered full filtered our experimental results in ## conclusion and future work in this work, we proposed hitter, a novel transformer-based model with effective training strategies for learning knowledge graph embeddings in complex multi-relational graphs. we show that with contextual information from a local neighborhood, our proposed method outperforms all previous approaches in long-standing link prediction tasks, achieving new sota results on fb15k-237 and wn18rr. moreover, we show that the knowledge representation learned by hitter can be effectively utilized by a transformer-based language model bert to answer factoid questions. it is worth mentioning that our proposed balanced contextualization is also applicable to other context-aware kge methods such as gnn-based approaches. future work can also apply hitter to other graph representation learning tasks besides link prediction. currently, our proposed hitter model performs well while only aggregating contextual information from a local graph neighborhood. it would be interesting to extend it with a broader graph context to obtain potential improvements. today, the transformer has become the de facto modeling architecture in natural language processing. as experimental results on factoid question answering tasks showcasing hitter's great potential to be integrated into common transformerbased model and generate substantial gains in performance, we intend to explore training hitter on large-scale knowledge graphs, so that more nlp models would benefit from hitter in various knowledge-intensive tasks.