HAHE : Hierarchical Attention for Hyper-Relational Knowledge Graphs in Global and Local Level (2023)

Abstract: Link Prediction on Hyper-relational Knowledge Graphs (HKG) is a worthwhile endeavor. HKG consists of hyper-relational facts (H-Facts), composed of a main triple and several auxiliary attribute-value qualifiers, which can effectively represent factually comprehensive information. The internal structure of HKG can be represented as a hypergraph-based representation globally and a semantic sequence-based representation locally. However, existing research seldom simultaneously models the graphical and sequential structure of HKGs, limiting HKGs’ representation. To overcome this limitation, we propose a novel Hierarchical Attention model for HKG Embedding (HAHE), including global-level and local-level attention. The global-level attention can model the graphical structure of HKG using hypergraph dual-attention layers, while the local-level attention can learn the sequential structure inside H-Facts via heterogeneous self-attention layers. Experiment results indicate that HAHE achieves state-of-the-art performance in link prediction tasks on HKG standard datasets. In addition, HAHE addresses the issue of HKG multi-position prediction for the first time, increasing the applicability of the HKG link prediction task. Our code is publicly available.

## introduction knowledge graphs (kgs) are semantic networks that define entity relationships. early kg research @xcite @xcite use binary relationships, often expressed as a triple-based fact (subject, relation, object). yet, n-ary relational facts (containing more than two entities) are abundant in real-world kgs like freebase @xcite and wikidata @xcite . @xcite represent an n-ary relational fact as a hyper-relational fact (h-fact) consisting of a main triple(s,r,o) and several auxiliary attribute-value qualifiers {(a i :v i )}, and kgs composed of h-facts are called hyper-relational knowledge graphs (hkgs). as shown in figure 1 , an h-fact can describe a real-world fact. unlike traditional triple-based facts, h-facts do not just raise the number of entities in facts from two to n. it structurally and effectively represents the n-ary relational facts prevalent in reality. globally, it extends ordinary graph structure to hypergraph @xcite structure. locally, it defines five heterogeneous roles of s,r,o,a,v within facts to capture the semantic information of the fact'barack obama held position as us president', as illustrated in figure 2 . recent research has demonstrated various embedding strategies for hyper-relational representations. however, current approaches only consider global hypergraph structures or local semantic sequence structures. for instance, stare @xcite employs the information transfer function of graph neural networks (gnn) to unidirectionally pass auxiliary key-value pair information into the main triples' relations, thereby capturing the graph structure but insufficiently between multiple entities and relations within the h-facts. in barack obama president of united states 20 january 2009 george w. bush 44 20 january 2017 donald trump 45 20 january 2021 joe biden 46 hyperedge_1 hyperedge_2 hyperedge_3 barack obama position held president of united states start time 20 january 2009 end time 20 january 2017 replaces george w. bush series ordinal 44 replaced by donald trump donald trump position held president of united states start time 20 january 2017 end time 20 january 2021 replaces barack obama series ordinal 45 replaced by joe biden joe biden position held president of united states start time 20 january 2021 [pad] [pad] replaces donald trump series ordinal 46 [pad] [pad] heterogeneous semantic sequences_1: heterogeneous semantic sequences_2: heterogeneous semantic sequences_3: local-level sequence-based representation global-level hypergraph-based representation hyper-relational facts barack obama position held president of united states start time 20 january 2009 end time 20 january 2017 replaces george w. bush series ordinal 44 replaced by donald trump main triple qualifiers donald trump position held president of united states start time 20 january 2017 end time 20 january 2021 replaces barack obama series ordinal 45 replaced by joe biden main triple qualifiers joe biden position held president of united states start time 20 january 2021 replaces donald trump series ordinal 46 main triple qualifiers barack obama george w. bush donald trump joe biden president of united states 20 january 2009 20 january 2017 20 january 2021 44 45 46 hypergraph incidence matrix contrast, gran @xcite initially incorporates the transformer encoder @xcite into the hkg embedding, capturing the fully connected semantic information locally inside hfacts, while disregarding the global structure. consequently, representing the global and local structure of hkg simultaneously with hierarchical attention becomes a promising research direction, but an inadequate representation of hkg structure constrains hkg embeddings. to overcome this limitation, we propose a novel hierarchical attention model for hkg embedding (hahe) that incorporates global-level and locallevel attention. we update the global node embeddings using the hkg hypergraph structure. however, by complete connectivity, the previous hypergraph attention network @xcite just converts all hypergraph nodes into a regular graph and then utilizes the gat @xcite layer for node embedding updates, rendering it unable to distinguish which nodes comprise a hyperedge. consequently, we design hypergraph dual-attention layers to aggregate node embedding information into hyper-edge embedding through the attention mechanism. after obtaining the hyper-edge embedding, we update the node embedding by feeding it back to the node through the attention mechanism. in this way, nodes are allowed to learn more distant information from the whole hkg. this hypergraph dual-attention method significantly enhances learning capacity. it then transfers the updated node information to the local level's attention. inspired by gran's heterogeneous attention @xcite , we define five types of nodes and fourteen types of edges in a single h-fact and develop heterogeneous self-attention layers with both nodebias and edge-bias attention to learn the semantic content of h-facts. the last step is to output the link prediction findings using an mlp-based decoding process for one-position or multi-position link prediciton tasks on hkgs. experiments on link prediction were performed on three hkg standard datasets, jf17k @xcite , wikipeople @xcite , and wd50k @xcite . the state-of-theart results indicate that hahe is effective in the link prediction task. in addition, adequate ablation experiments were designed to highlight the importance of global and local focus, and hahe is also used for the hkg multi-position prediction task, i.e., predicting two or more entities or relations simultaneously in a single h-fact, hence increasing the applicability of the hkg link prediction task. ultimately, we make our code publicly available and discuss the limitations and future work of hkg embedding representation. ## related work early approaches consider hyper-relational fact(h-fact) mainly as graph structure, focusing more on the topological relations of entities. for example, m-transh @xcite projects the entities onto the relation hyperplane. rae, nalp, neuinfer, and n-tucker @xcite @xcite optimize the method of the hyper-relational knowledge graph (hkg) embedding based on m-transh. however, none of them adopts the hyper-relational structure. hinge @xcite firstly proposes the attribute-value qualifiers for embedding hyperrelational representation using cnn, and hy-per2 @xcite initializes the relation and entity in the poincaré ball vectors to improve the model accuracy. yet, neither of these methods considers graphical structure or semantic sequences of hkgs. stare @xcite employs gnn as the message-passing mechanism to encode entities and relationships, while transformer is the decoder to get the result. hytransformer @xcite applies layer normalization and dropout methods to replace stare's encoder. mseahkg @xcite proposed that a message-passing function significantly impacts the model performance, so it replaced the static message-passing function in stare with a dynamic one. these models consider the graph structure within the hyper-relational facts, but disregard the semantic sequences. gran @xcite is an improvement for transformer @xcite . it replaces transformer's self-attention with edge-biased fullyconnected attention and accurately collects semantic information. despite this, it ignores the graph structure. unlike earlier models, hahe considers graph structure and semantic sequences simultaneously and employs hierarchical attention for link prediction on hkgs. moreover, it is the first to improve previous methods by modeling the structure of hkg via global-level embedding and can perform multi-position prediction. ## preliminaries this section presents important concepts and techniques in hyper-relational knowledge graph (hkg), including definitions of hkgs, hypergraph learning, global and local structure of hkgs, and multi-position prediction on hkgs. ## methodology this section introduces our hyper-relational knowledge graph (hkg) embedding model hahe, including global and local representation, two hierarchical attention layers, and mlp decoder. ## experiments this section introduces the experimental settings, results and analysis. we answer the following research questions (rqs). rq1: can hahe outperform other hyper-relational knowledge graphs (hkg) embedding models on hkg datasets? rq2: how does the hierarchical attention mechanism contribute to hahe? rq3: how do hypergraph dual-attention mechanisms contribute to hahe in global level? rq4: how do heterogeneity of nodes and edges in hyper-relation fact contribute to hahe in local level? rq5: how hahe performs in multi-position prediction tasks on hkgs? ## conclusion in this paper, we present hahe, a model with hierarchical attention in global and local level. hahe outperforms other baselines link prediction tasks on hyper-relational knowledge graphs (hkgs). the experimental results demonstrate that our hypergraph dual-attention layers and heterogeneous selfattention layers are effective in learning the global and local structure of hkgs. we also use hahe to solve the hkg multi-location prediction task and analyze the results for the first time.