A Relational Memory-based Embedding Model for Triple Classification and Search Personalization (2020)

Abstract: Knowledge graph embedding methods often suffer from a limitation of memorizing valid triples to predict new ones for triple classification and search personalization problems. To this end, we introduce a novel embedding model, named R-MeN, that explores a relational memory network to encode potential dependencies in relationship triples. R-MeN considers each triple as a sequence of 3 input vectors that recurrently interact with a memory using a transformer self-attention mechanism. Thus R-MeN encodes new information from interactions between the memory and each input vector to return a corresponding vector. Consequently, R-MeN feeds these 3 returned vectors to a convolutional neural network-based decoder to produce a scalar score for the triple. Experimental results show that our proposed R-MeN obtains state-of-the-art results on SEARCH17 for the search personalization task, and on WN11 and FB13 for the triple classification task.

## introduction knowledge graphs (kgs) -representing the genuine relationships among entities in the form of triples (subject, relation, object) denoted as (s, r, o) -are often insufficient for knowledge presentation due to the lack of many valid triples @xcite . therefore, research work has been focusing on inferring whether a new triple missed in kgs is likely valid or not @xcite @xcite . as summarized in @xcite , kg embedding models aim to compute a score for each triple, such that valid triples have higher scores than invalid ones. early embedding models such as transe @xcite , transh @xcite , transr @xcite , transd @xcite , dist-mult @xcite and complex @xcite often employ simple linear oper-ators such as addition, subtraction and multiplication. recent embedding models such as conve @xcite and @xcite successfully apply deep neural networks to score the triples. existing embedding models are showing promising performances mainly for knowledge graph completion, where the goal is to infer a missing entity given a relation and another entity. but in real applications, less mentioned, such as triple classification @xcite that aims to predict whether a given triple is valid, and search personalization @xcite that aims to re-rank the relevant documents returned by a user-oriented search system given a query, these models do not effectively capture potential dependencies among entities and relations from existing triples to predict new triples. to this end, we leverage the relational memory network @xcite to propose r-men to infer a valid fact of new triples. in particular, r-men transforms each triple along with adding positional embeddings into a sequence of 3 input vectors. r-men then uses a transformer self-attention mechanism @xcite to guide the memory to interact with each input vector to produce an encoded vector. as a result, r-men feeds these 3 encoded vectors to a convolutional neural network (cnn)-based decoder to return a score for the triple. in summary, our main contributions are as follows: figure 1 : processes in our proposed r-men for an illustration purpose. "m" denotes a memory. "mlp" denotes a multi-layer perceptron. "g" denotes a memory gating. "cnn" denotes a convolutional neural networkbased decoder. let g be a kg database of valid triples in the form of (subject, relation, object) denoted as (s, r, o). kg embedding models aim to compute a score for each triple, such that valid triples obtain higher scores than invalid triples. we denote v s , v r and v @xmath0, r and o, respectively. besides, we hypothesize that relative positions among s, r and o are useful to reason instinct relationships; hence we add to each position a positional embedding. given a triple (s, r, o), we obtain a sequence of 3 vectors {x 1 , x 2 , x 3 } as: we assume we have a memory m consisting of n rows wherein each row is a memory slot. we use m (t) to denote the memory at timestep t, and m (t) i,: ∈ r k to denote the i-th memory slot at timestep t. we follow @xcite to take x t to update m (t) i,: using the multi-head selfattention mechanism @xcite as: to a multi-layer perceptron followed by a memory gating to produce an encoded vector y @xmath1,: as a result, we obtain a sequence of 3 encoded vectors {y 1 , y 2 , y 3 } for the triple (s, r, o). we then use a cnn-based decoder to compute a score for the triple as: we illustrate our proposed r-men as shown in figure 1 . in addition, we employ the adam optimizer @xcite to train r-men by minimizing the following loss function @xcite : ## conclusion we propose a new kg embedding model, named r-men, where we integrate transformer self-attention mechanism-based memory interactions with a cnn decoder to capture the potential dependencies in the kg triples effectively. experimental results show that our proposed r-men obtains the new state-of-the-art performances for both the triple classification and search personalization tasks. in future work, we plan to extend r-men for multihop knowledge graph reasoning. our code is available at: https://github.com/daiquocnguyen/ r-men .