Real-world entities (e.g., people and places) are often connected via relations, forming multirelational data. Modeling multi-relational data is important in many research areas, from natural language processing to biological data mining [6] . Prior work on multi-relational learning can be categorized into three categories: (1) statistical relational learning (SRL) [10] , such as Markovlogic networks [21] , which directly encode multi-relational graphs using probabilistic models; (2) path ranking methods [16, 7] , which explicitly explore the large relational feature space of relations with random walk; and (3) embedding-based models, which embed multi-relational knowledge into low-dimensional representations of entities and relations via tensor/matrix factorization [24, 18, 19] , Bayesian clustering framework [15, 26] , and neural networks [20, 1, 2, 25] . Our work focuses on the study of neural-embedding models, where the representations are learned in a neural network architecture. They have shown to be powerful tools for multi-relational learning and inference due to their high scalability and strong generalization abilities. A number of techniques have been recently proposed to learn entity and relation representations using neural networks [1, 2, 25] . They all represent entities as low-dimensional vectors and represent relations as operators that combine the representations of two entities. The main difference among these techniques lies in the parametrization of the relation operators. For instance, given two entity vectors, the model of Neural Tensor Network (NTN) [25] represents each relation as a bilinear tensor operator and a linear matrix operator. The model of TransE [2], on the other hand, represents each relation as a single vector that linearly interacts with the entity vectors. Both models report promising performance on predicting unseen relationships in knowledge bases. However, they have not been directly compared in terms of the different choices of relation operators and of the resulting effectiveness. Neither has the design of entity representations in these recent studies been carefully explored. For example, NTN [25] first shows the benefits of representing entities as an average of word vectors and initializing word vectors with pre-trained vectors from large text corpora. This idea is promising as pre-trained vectors tend to capture syntactic and semantic information from natural language and can assist in better generalization of entity embeddings. However, many real-world entities are expressed as non-compositional phrases (e.g. person names, movie names, etc.), of which meaning cannot be composed by their constituent words. Therefore, averaging word vectors may not provide an appropriate representation for such entities. In this paper, we examine and compare different types of relation operators and entity vector representations under a general framework for multi-relational learning. Specifically, we derive several recently proposed embedding models, including TransE [2] and NTN [25] , and their variants under the same framework. We empirically evaluate their performance on a knowledge base completion task using various real-world datasets in a controlled experimental setting and present several interesting findings. First, the models with fewer parameters tend to be better than more complex models in terms of both performance and scalability. Second, the bilinear operator plays an important role in capturing entity interactions. Third, with the same model complexity, multiplicative operations are superior to additive operations in modeling relations. Finally, initializing entity vectors with pre-trained phrase vectors can significantly boost performance, whereas representing entity vectors as an average of word vectors that are initialized with pre-trained vectors may hurt performance.
REF find that multiplicative operations are superior to additive operations in modeling relations.
Learning Multi-Relational Semantics Using Neural-Embedding Models
ArXiv
ArXiv
Computer Science Mathematics