Original: R OTAT E: K NOWLEDGE G RAPH E MBEDDING BY R ELATIONAL R OTATION IN C OMPLEX S PACE
Revised: R OTAT E: K NOWLEDGE G RAPH E MBEDDING BY R ELATIONAL R OTATION IN C OMPLEX S PACE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the problem of learning representations of entities and relations in knowledge graphs for predicting missing links.
Revised: We study the problem of learning representations of entities and relations in knowledge graphs for predicting missing links.

Original: The success of such a task heavily relies on the ability of modeling and inferring the patterns of (or between) the relations.
Revised: The success of such a task heavily relies on the ability of modeling and inferring the patterns of (or between) the relations.

Original: In this paper, we present a new approach for knowledge graph embedding called RotatE, which is able to model and infer various relation patterns including: symmetry/antisymmetry, inversion, and composition.
Revised: In this paper, we present a new approach for knowledge graph embedding called RotatE, which is able to model and infer various relation patterns including: symmetry/antisymmetry, inversion, and composition.

Original: Speciﬁcally, the RotatE model deﬁnes each relation as a rotation from the source entity to the target entity in the complex vector space.
Revised: Speciﬁcally, the RotatE model deﬁnes each relation as a rotation from the source entity to the target entity in the complex vector space.

Original: In addition, we propose a novel self-adversarial negative sampling technique for efﬁciently and effectively training the RotatE model.
Revised: In addition, we propose a novel self-adversarial negative sampling technique for efﬁciently and effectively training the RotatE model.

Original: Experimental results on multiple benchmark knowledge graphs show that the proposed RotatE model is not only scalable, but also able to infer and model various relation patterns and signiﬁcantly outperform existing state-of-the-art models for link prediction.
Revised: Experimental results on multiple benchmark knowledge graphs show that the proposed RotatE model is not only scalable, but also able to infer and model various relation patterns and signiﬁcantly outperform existing state-of-the-art models for link prediction.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Knowledge graphs are collections of factual triplets, where each triplet ( h , r , t ) represents a relation r between a head entity h and a tail entity t .
Revised: Knowledge graphs are collections of factual triplets, where each triplet ( h , r , t ) represents a relation r between a head entity h and a tail entity t .

Original: Examples of real-world knowledge graphs include
Revised: Examples of real-world knowledge graphs include

Original: Freebase (Bollacker et al., 2008), Yago (Suchanek et al., 2007), and WordNet (Miller, 1995).
Revised: Freebase (Bollacker et al., 2008), Yago (Suchanek et al., 2007), and WordNet (Miller, 1995).

Original: Knowledge graphs are potentially useful to a variety of applications such as question-answering (Hao et al., 2017), information retrieval (Xiong et al., 2017), recommender systems (Zhang et al., 2016), and natural language processing (Yang & Mitchell, 2017).
Revised: Knowledge graphs are potentially useful to a variety of applications such as question-answering (Hao et al., 2017), information retrieval (Xiong et al., 2017), recommender systems (Zhang et al., 2016), and natural language processing (Yang & Mitchell, 2017).

Original: Research on knowledge graphs is attracting growing interests in both academia and industry communities.
Revised: Research on knowledge graphs is attracting growing interests in both academia and industry communities.

Original: Since knowledge graphs are usually incomplete, a fundamental problem for knowledge graph is predicting the missing links.
Revised: Since knowledge graphs are usually incomplete, a fundamental problem for knowledge graph is predicting the missing links.

Original: Recently, extensive studies have been done on learning low-dimensional representations of entities and relations for missing link prediction (a.k.a., knowledge graph embedding)
Revised: Recently, extensive studies have been done on learning low-dimensional representations of entities and relations for missing link prediction (a.k.a., knowledge graph embedding)

Original: (Bordes et al., 2013; Trouillon et al., 2016; Dettmers et al., 2017).
Revised: (Bordes et al., 2013; Trouillon et al., 2016; Dettmers et al., 2017).

Original: These methods have been shown to be scalable and effective.
Revised: These methods have been shown to be scalable and effective.

Original: The general intuition of these methods is to model and infer the connectivity patterns in knowledge graphs according to the observed knowledge facts.
Revised: The general intuition of these methods is to model and infer the connectivity patterns in knowledge graphs according to the observed knowledge facts.

Original: For example, some relations are symmetric (e.g., marriage) while others are antisymmetric (e.g., ﬁliation); some relations are the inverse of other relations (e.g., hypernym and hyponym); and some relations may be composed by others (e.g., my mother’s husband is my father).
Revised: For example, some relations are symmetric (e.g., marriage) while others are antisymmetric (e.g., ﬁliation); some relations are the inverse of other relations (e.g., hypernym and hyponym); and some relations may be composed by others (e.g., my mother’s husband is my father).

Original: It is critical to ﬁnd ways to model and infer these patterns, i.e., symmetry/antisymmetry , inversion , and composition , from the observed facts in order to predict missing links.
Revised: It is critical to ﬁnd ways to model and infer these patterns, i.e., symmetry/antisymmetry , inversion , and composition , from the observed facts in order to predict missing links.

Original: Indeed, many existing approaches have been trying to either implicitly or explicitly model one or a few of the above relation patterns (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015b; 
Revised: Indeed, many existing approaches have been trying to either implicitly or explicitly model one or a few of the above relation patterns (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015b; [Table]

Original: Yang et al., 2014; Trouillon et al., 2016).
Revised: Yang et al., 2014; Trouillon et al., 2016).

Original: For example, the TransE model (Bordes et al., 2011), which represents relations as translations, aims to model the inversion and composition patterns;
Revised: For example, the TransE model (Bordes et al., 2011), which represents relations as translations, aims to model the inversion and composition patterns;

Original: the
Revised: the

Original: DisMult model (Yang et al., 2014), which models the three-way interactions between head entities, relations, and tail entities, aims to model the symmetry pattern.
Revised: DisMult model (Yang et al., 2014), which models the three-way interactions between head entities, relations, and tail entities, aims to model the symmetry pattern.

Original: However, none of existing models is capable of modeling and inferring all the above patterns.
Revised: However, none of existing models is capable of modeling and inferring all the above patterns.

Original: Therefore, we are looking for an approach that is able to model and infer all the three types of relation patterns.
Revised: Therefore, we are looking for an approach that is able to model and infer all the three types of relation patterns.

Original: In this paper, we propose such an approach called RotatE for knowledge graph embedding.
Revised: In this paper, we propose such an approach called RotatE for knowledge graph embedding.

Original: Our motivation is from Euler’s identity e iθ = cos θ + i sin θ , which indicates that a unitary complex number can be regarded as a rotation in the complex plane.
Revised: Our motivation is from Euler’s identity e iθ = cos θ + i sin θ , which indicates that a unitary complex number can be regarded as a rotation in the complex plane.

Original: Speciﬁcally, the RotatE model maps[Table] the entities and relations to the complex vector space and deﬁnes each relation as a rotation from the source entity to the target entity.
Revised: Speciﬁcally, the RotatE model maps the entities and relations to the complex vector space and deﬁnes each relation as a rotation from the source entity to the target entity.

Original: Given a triplet ( h , r , t ) , we expect that t = h ◦ r , where h , r , t ∈ C k are the embeddings, the modulus | r i | = 1 and ◦ denotes the Hadamard (element-wise) product.
Revised: Given a triplet ( h , r , t ) , we expect that t = h ◦ r , where h , r , t ∈ C k are the embeddings, the modulus | r i | = 1 and ◦ denotes the Hadamard (element-wise) product.

Original: Speciﬁcally, for each dimension in the complex space, we expect that:t i = h
Revised: Speciﬁcally, for each dimension in the complex space, we expect that:

Original: i r i , where h i , r i , t i ∈ C and | r i | = 1 .
Revised: [Equation]

Original: It turns out that such a simple operation can effectively model all the three relation patterns: symmetric/antisymmetric, inversion, and composition.
Revised: It turns out that such a simple operation can effectively model all the three relation patterns: symmetric/antisymmetric, inversion, and composition.

Original: For example, a relation r is symmetric if and only if each element of its embedding r , i.e. r i , satisﬁes r i =
Revised: For example, a relation r is symmetric if and only if each element of its embedding r , i.e. r i , satisﬁes r i =

Original: e 0 /iπ = ± 1 ; two relations r 1 and rare inverse if and only if their embeddings are conjugates: r 2 = ¯ r 1 ; a relation r 3 =
Revised: e 0 /iπ = ± 1 ; two relations r 1 and rare inverse if and only if their embeddings are conjugates: r 2 = ¯ r 1 ; a relation r 3 =

Original: e i θ 3 is a combination of other two relations r 

Revised: e i θ 3 is a combination of other two relations r

Original: =
Revised: =

Original: e i θ 1 and r 

Revised: e i θ 1 and r

Original: =
Revised: =

Original: e i θ 2 if and only if r
Revised: e i θ 2 if and only if r

Original: 3 =
Revised: 3 =

Original: r 1 ◦ r 2 (i.e. θ 3 = θ 1 + θ 2 ).
Revised: r 1 ◦ r 2 (i.e. θ 3 = θ 1 + θ 2 ).

Original: Moreover, the RotatE model is scalable to large knowledge graphs as it remains linear in both time and memory.
Revised: Moreover, the RotatE model is scalable to large knowledge graphs as it remains linear in both time and memory.

Original: To effectively optimizing the RotatE, we further propose a novel self-adversarial negative sampling technique, which generates negative samples according to the current entity and relation embeddings.
Revised: To effectively optimizing the RotatE, we further propose a novel self-adversarial negative sampling technique, which generates negative samples according to the current entity and relation embeddings.

Original: The proposed technique is very general and can be applied to many existing knowledge graph embedding models.
Revised: The proposed technique is very general and can be applied to many existing knowledge graph embedding models.

Original: We evaluate the RotatE on four large knowledge graph benchmark datasets including FB15k (Bordes et al., 2013), WN18 (Bordes et al., 2013), FB15k-237 (Toutanova & Chen, 2015) and WN18RR (Dettmers et al., 2017).
Revised: We evaluate the RotatE on four large knowledge graph benchmark datasets including FB15k (Bordes et al., 2013), WN18 (Bordes et al., 2013), FB15k-237 (Toutanova & Chen, 2015) and WN18RR (Dettmers et al., 2017).

Original: Experimental results show that the RotatE model signiﬁcantly outperforms existing state-of-the-art approaches.
Revised: Experimental results show that the RotatE model signiﬁcantly outperforms existing state-of-the-art approaches.

Original: In addition, RotatE also outperforms state-of-the-art models on Countries (Bouchard et al., 2015), a benchmark explicitly designed for composition pattern inference and modeling.
Revised: In addition, RotatE also outperforms state-of-the-art models on Countries (Bouchard et al., 2015), a benchmark explicitly designed for composition pattern inference and modeling.

Original: To the best of our knowledge, RotatE is the ﬁrst model that achieves state-of-the-art performance on all the benchmarks.
Revised: To the best of our knowledge, RotatE is the ﬁrst model that achieves state-of-the-art performance on all the benchmarks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original:  Predicting missing links with knowledge graph embedding (KGE) methods has been extensively investigated in recent years.
Revised: [Table] Predicting missing links with knowledge graph embedding (KGE) methods has been extensively investigated in recent years.

Original: The general methodology is to deﬁne a score function for the triplets.
Revised: The general methodology is to deﬁne a score function for the triplets.

Original: Formally, let E denote the set of entities and R denote the set of relations, then a knowledge graph is a collection of factual triplets ( h , r , t ) , where h , t ∈ E and r ∈ R .
Revised: Formally, let E denote the set of entities and R denote the set of relations, then a knowledge graph is a collection of factual triplets ( h , r , t ) , where h , t ∈ E and r ∈ R .

Original: Since entity embeddings are usually represented as vectors, the score function usually takes the form f r ( h , t ) , where h and t are head and tail entity embeddings.
Revised: Since entity embeddings are usually represented as vectors, the score function usually takes the form f r ( h , t ) , where h and t are head and tail entity embeddings.

Original: The score function f r ( h , t ) measures the salience of a candidate triplet ( h , r , t ) .
Revised: The score function f r ( h , t ) measures the salience of a candidate triplet ( h , r , t ) .

Original: The goal of the optimization is usually to score true triplet ( h , r , t ) higher than the corrupted false triplets ( h (cid:48) , r , t ) or ( h , r , t (cid:48) ) .
Revised: The goal of the optimization is usually to score true triplet ( h , r , t ) higher than the corrupted false triplets ( h (cid:48) , r , t ) or ( h , r , t (cid:48) ) .

Original: Table 1 summarizes different score functions f r ( h , t ) in previous state-of-the-art methods as well as the model proposed in this paper.
Revised: Table 1 summarizes different score functions f r ( h , t ) in previous state-of-the-art methods as well as the model proposed in this paper.

Original: These models generally capture only a portion of the relation patterns.
Revised: These models generally capture only a portion of the relation patterns.

Original: For example, TransE represents each relation[Table] as a bijection between source entities and target entities, and thus implicitly models inversion and composition of relations, but it cannot model symmetric relations; ComplEx extends DistMult by introducing complex embeddings so as to better model asymmetric relations, but it cannot infer the composition pattern.
Revised: For example, TransE represents each relation as a bijection between source entities and target entities, and thus implicitly models inversion and composition of relations, but it cannot model symmetric relations; ComplEx extends DistMult by introducing complex embeddings so as to better model asymmetric relations, but it cannot infer the composition pattern.

Original: The proposed RotatE model leverages the advantages of both.
Revised: The proposed RotatE model leverages the advantages of both.

Original: A relevant and concurrent work to our work is the TorusE (Ebisu & Ichise, 2018) model, which deﬁnes knowledge graph embedding as translations on a compact Lie group.
Revised: A relevant and concurrent work to our work is the TorusE (Ebisu & Ichise, 2018) model, which deﬁnes knowledge graph embedding as translations on a compact Lie group.

Original: The TorusE model can be regarded as a special case of RotatE, where the modulus of embeddings are set ﬁxed;
Revised: The TorusE model can be regarded as a special case of RotatE, where the modulus of embeddings are set ﬁxed;

Original: our
Revised: our

Original: RotatE is deﬁned on the entire complex space, which has much more representation capacity.
Revised: RotatE is deﬁned on the entire complex space, which has much more representation capacity.

Original: Our experiments show that this is very critical for modeling and inferring the composition patterns.
Revised: Our experiments show that this is very critical for modeling and inferring the composition patterns.

Original: Moreover, TorusE focuses on the problem of regularization in TransE while this paper focuses on modeling and inferring multiple types of relation patterns.
Revised: Moreover, TorusE focuses on the problem of regularization in TransE while this paper focuses on modeling and inferring multiple types of relation patterns.

Original: There are also a large body of relational approaches for modeling the relational patterns on knowledge graphs (Lao et al., 2011; Neelakantan et al., 2015; Das et al., 2016;
Revised: There are also a large body of relational approaches for modeling the relational patterns on knowledge graphs (Lao et al., 2011; Neelakantan et al., 2015; Das et al., 2016;

Original: Rockt ¨ aschel & Riedel, 2017; Yang et al., 2017).
Revised: Rockt ¨ aschel & Riedel, 2017; Yang et al., 2017).

Original: However, these approaches mainly focus on explicitly modeling the relational paths while our proposed RotatE model implicitly learns the relation patterns, which is not only much more scalable but also provides meaningful embeddings for both entities and relations.
Revised: However, these approaches mainly focus on explicitly modeling the relational paths while our proposed RotatE model implicitly learns the relation patterns, which is not only much more scalable but also provides meaningful embeddings for both entities and relations.

Original: Another related problem is how to effectively draw negative samples for training knowledge graph embeddings.
Revised: Another related problem is how to effectively draw negative samples for training knowledge graph embeddings.

Original: This problem has been explicitly studied by Cai & Wang (2017), which proposed a generative adversarial learning framework to draw negative samples.
Revised: This problem has been explicitly studied by Cai & Wang (2017), which proposed a generative adversarial learning framework to draw negative samples.

Original: However, such a framework requires simultaneously training the embedding model and a discrete negative sample generator, which are difﬁcult to optimize and also computationally expensive.
Revised: However, such a framework requires simultaneously training the embedding model and a discrete negative sample generator, which are difﬁcult to optimize and also computationally expensive.

Original: We propose a self-adversarial sampling scheme which only relies on the current model.
Revised: We propose a self-adversarial sampling scheme which only relies on the current model.

Original: It does require any additional optimization component, which make it much more efﬁcient.
Revised: It does require any additional optimization component, which make it much more efﬁcient.

Original:  R OTAT E: R ELATIONAL R OTATION IN C OMPLEX V ECTOR S PACE
Revised:  R OTAT E: R ELATIONAL R OTATION IN C OMPLEX V ECTOR S PACE

Original: In this section, we introduce our proposed RotatE model.
Revised: In this section, we introduce our proposed RotatE model.

Original: We ﬁrst introduce three important relation patterns that are widely studied in the literature of link prediction on knowledge graphs.
Revised: We ﬁrst introduce three important relation patterns that are widely studied in the literature of link prediction on knowledge graphs.

Original: Afterwards, we introduce our proposed RotatE model, which deﬁnes relations as rotations in complex vector space.
Revised: Afterwards, we introduce our proposed RotatE model, which deﬁnes relations as rotations in complex vector space.

Original: We also show that the RotatE model is able to model and infer all three relation patterns
Revised: We also show that the RotatE model is able to model and infer all three relation patterns.

Original: . 3. M ODELING AND I NFERRING R ELATION P ATTERNS
Revised: 3. M ODELING AND I NFERRING R ELATION P ATTERNS

Original: The key of link prediction in knowledge graph is to infer the connection patterns, e.g., relation patterns, with observed facts.
Revised: The key of link prediction in knowledge graph is to infer the connection patterns, e.g., relation patterns, with observed facts.

Original: According to the existing literature (Trouillon et al., 2016; Toutanova & Chen, 2015; Guu et al., 2015; Lin et al., 2015a), three types of relation patterns are very important  and widely spread in knowledge graphs: symmetry, inversion and composition.
Revised: According to the existing literature (Trouillon et al., 2016; Toutanova & Chen, 2015; Guu et al., 2015; Lin et al., 2015a), three types of relation patterns are very important [Figure] and widely spread in knowledge graphs: symmetry, inversion and composition.

Original: We give their formal deﬁnition here:
Revised: We give their formal deﬁnition here:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A relation r is symmetric ( antisymmetric )if ∀ x , y
Revised: A relation r is symmetric ( antisymmetric )

Original: [Equation]
Revised: [Equation]

Original: A clause with such form is a symmetry (antisymmetry) pattern.
Revised: A clause with such form is a symmetry (antisymmetry) pattern.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Relation r 1 is inverse to relation r 2 if ∀ x , y
Revised: Relation r 1 is inverse to relation r 2 if ∀ x , y

Original: [Equation]
Revised: [Equation]

Original: A clause with such form is a inversion pattern.
Revised: A clause with such form is a inversion pattern.

Original: [Figure]

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Relation r 1 is composed of relation r 2 and relation r 3{+
+} if ∀ x, y , z r 2 ( x , y )
Revised: Relation r 1 is composed of relation r 2 and relation r {+
+} if ∀ x

Original: ∧ r 3 ( y, z ) ⇒ r
Revised: [Equation] y

Original: 1 ( x , z )
Revised: [Equation]

Original: A clause with such form is a composition pattern.
Revised: A clause with such form is a composition pattern.

Original: According to the deﬁnition of the above three types of relation patterns, we provide an analysis of existing models on their abilities in inferring and modeling these patterns.
Revised: According to the deﬁnition of the above three types of relation patterns, we provide an analysis of existing models on their abilities in inferring and modeling these patterns.

Original: Speciﬁcally, we provide an analysis on TransE, TransX, DistMult, and ComplEx.
Revised: Speciﬁcally, we provide an analysis on TransE, TransX, DistMult, and ComplEx.

Original:  We did not include the analysis on HolE and ConvE since HolE is equivalent to ComplEx (Hayashi & Shimbo, 2017), and ConvE is a black box that involves two-layer neural networks and convolution operations, which are hard to analyze.
Revised: 3 We did not include the analysis on HolE and ConvE since HolE is equivalent to ComplEx (Hayashi & Shimbo, 2017), and ConvE is a black box that involves two-layer neural networks and convolution operations, which are hard to analyze.

Original: The results are summarized into Table 2.
Revised: The results are summarized into Table 2.

Original: We can see that no existing approaches are capable of modeling all the three relation patterns.
Revised: We can see that no existing approaches are capable of modeling all the three relation patterns.

Original: 3. M ODELING R ELATIONS AS R OTATIONS IN C OMPLEX V ECTOR S PACE
Revised: 3. M ODELING R ELATIONS AS R OTATIONS IN C OMPLEX V ECTOR S PACE

Original: In this part, we introduce our proposed model that is able to model and infer all the three types of relation patterns.
Revised: In this part, we introduce our proposed model that is able to model and infer all the three types of relation patterns.

Original: Inspired by Euler’s identity, we map the head and tail entities h , t to the complex embeddings, i.e., h , t ∈ C k ; then we deﬁne the functional mapping induced by each relation r as an element-wise rotation from the head entity h to the tail entity t .
Revised: Inspired by Euler’s identity, we map the head and tail entities h , t to the complex embeddings, i.e., h , t ∈ C k ; then we deﬁne the functional mapping induced by each relation r as an element-wise rotation from the head entity h to the tail entity t .

Original: In other words, given a triple ( h , r , t ) , we expect that: 
Revised: In other words, given a triple ( h , r , t ) , we expect that: [Equation] |

Original: [Equation]
Revised: [Equation]

Original: and ◦ is the Hadmard (or element-wise) product.
Revised: and ◦ is the Hadmard (or element-wise) product.

Original: Speciﬁcally, for each element in the embeddings, we have t i
Revised: Speciﬁcally, for each element in the embeddings, we have t i

Original: = h
Revised: = h

Original: i r i .
Revised: i r i .

Original: Here, we constrain the modulus of each element of r ∈ C k , i.e., r i ∈ C , to be | r i | = 1 .
Revised: Here, we constrain the modulus of each element of r ∈ C k , i.e., r i ∈ C , to be | r i | = 1 .

Original: By doing this, r i is of the form e iθ r,i , which corresponds to a counterclockwise rotation by θ r,i radians about the origin of the complex plane, and only affects the phases of the entity embeddings in the complex vector space.
Revised: By doing this, r i is of the form e iθ r,i , which corresponds to a counterclockwise rotation by θ r,i radians about the origin of the complex plane, and only affects the phases of the entity embeddings in the complex vector space.

Original: We refer to the proposed model as RotatE due to its rotational nature.
Revised: We refer to the proposed model as RotatE due to its rotational nature.

Original: According to the above deﬁnition, for each triple ( h , r , t ) , we deﬁne the distance function of RotatE as:
Revised: According to the above deﬁnition, for each triple ( h , r , t ) , we deﬁne the distance function of RotatE as:

Original: [Equation]
Revised: [Equation]

Original: By deﬁning each relation as a rotation in the complex vector spaces, RotatE can model and infer all the three types of relation patterns introduced above.
Revised: By deﬁning each relation as a rotation in the complex vector spaces, RotatE can model and infer all the three types of relation patterns introduced above.

Original: Formally, we have following results 3 :
Revised: Formally, we have following results 4 :

Revised: We relegate all proofs to the appendix.

Revised: [Table]

Original: Lemma 1.
Revised: Lemma 1.

Original: RotatE can infer the symmetry/antisymmetry pattern.
Revised: RotatE can infer the symmetry/antisymmetry pattern.

Original: (See proof in Appendix B)
Revised: (See proof in Appendix B)

Original: Lemma 2.
Revised: Lemma 2.

Original: RotatE can infer the inversion pattern.
Revised: RotatE can infer the inversion pattern.

Original: (See proof in Appendix C)
Revised: (See proof in Appendix C)

Original: Lemma 3.
Revised: Lemma 3.

Original: RotatE can infer the composition pattern.
Revised: RotatE can infer the composition pattern.

Original: (See proof in Appendix D)
Revised: (See proof in Appendix D)

Original: These results are also summarized into Table 2.
Revised: These results are also summarized into Table 2.

Original: We can see that the RotatE model is the only model that can model and infer all the three types of relation patterns.
Revised: We can see that the RotatE model is the only model that can model and infer all the three types of relation patterns.

Original: [Table]

Original: Connection to TransE. From Table 2, we can see that TransE is able to infer and model all the other relation patterns except the symmetry pattern.
Revised: Connection to TransE. From Table 2, we can see that TransE is able to infer and model all the other relation patterns except the symmetry pattern.

Original: The reason is that in TransE, any symmetric relation will be represented by a 0 translation vector.
Revised: The reason is that in TransE, any symmetric relation will be represented by a 0 translation vector.

Original: As a result, this will push the entities with symmetric relations to be close to each other in the embedding space.
Revised: As a result, this will push the entities with symmetric relations to be close to each other in the embedding space.

Original: RotatE solves this problem and is a able to model and infer the symmetry pattern.
Revised: RotatE solves this problem and is a able to model and infer the symmetry pattern.

Original: An arbitrary vector r that satisﬁes r i = ±can be used for representing a symmetric relation in RotatE, and thus the entities having symmetric relations can be distinguished.
Revised: An arbitrary vector r that satisﬁes r i = ±can be used for representing a symmetric relation in RotatE, and thus the entities having symmetric relations can be distinguished.

Original: Different symmetric relations can be also represented with different embedding vectors.
Revised: Different symmetric relations can be also represented with different embedding vectors.

Original: Figure 1 provides illustrations of TransE and RotatE with only 1-dimensional embedding and shows how RotatE models a symmetric relation.
Revised: Figure 1 provides illustrations of TransE and RotatE with only 1-dimensional embedding and shows how RotatE models a symmetric relation.

Original: 3. O PTIMIZATION
Revised: 3. O PTIMIZATION

Original: Negative sampling has been proved quite effective for both learning knowledge graph embedding (Trouillon et al., 2016) and word embedding (Mikolov et al., 2013).
Revised: Negative sampling has been proved quite effective for both learning knowledge graph embedding (Trouillon et al., 2016) and word embedding (Mikolov et al., 2013).

Original: Here we use a loss function similar to the negative sampling loss (Mikolov et al., 2013) for effectively optimizing distance-based models:
Revised: Here we use a loss function similar to the negative sampling loss (Mikolov et al., 2013) for effectively optimizing distance-based models:

Original: [Equation]
Revised: [Equation]

Original: where γ is a ﬁxed margin, σ is the sigmoid function, and ( h (cid:48)
Revised: where γ is a ﬁxed margin, σ is the sigmoid function, and ( h (cid:48)

Original: i , r , t (cid:48) i ) is the i -th negative triplet.
Revised: i , r , t (cid:48) i ) is the i -th negative triplet.

Original: We also propose a new approach for drawing negative samples.
Revised: We also propose a new approach for drawing negative samples.

Original: The negative sampling loss samples the negative triplets in a uniform way.
Revised: The negative sampling loss samples the negative triplets in a uniform way.

Original: Such a uniform negative sampling suffers the problem of inefﬁciency since many samples are obviously false as training goes on, which does not provide any meaningful information.
Revised: Such a uniform negative sampling suffers the problem of inefﬁciency since many samples are obviously false as training goes on, which does not provide any meaningful information.

Original: Therefore, we propose an approach called self-adversarial negative sampling, which samples negative triples according to the current embedding model.
Revised: Therefore, we propose an approach called self-adversarial negative sampling, which samples negative triples according to the current embedding model.

Original: Speciﬁcally, we sample negative triples from the following distribution:
Revised: Speciﬁcally, we sample negative triples from the following distribution:

Original: [Equation]
Revised: [Equation]

Original: where α is the temperature of sampling.
Revised: where α is the temperature of sampling.

Original: Moreover, since the sampling procedure may be costly, we treat the above probability as the weight of the negative sample.
Revised: Moreover, since the sampling procedure may be costly, we treat the above probability as the weight of the negative sample.

Original: Therefore, the ﬁnal negative sampling loss with self-adversarial training takes the following form:
Revised: Therefore, the ﬁnal negative sampling loss with self-adversarial training takes the following form:

Original: [Equation]
Revised: [Equation]

Original: In the experiments, we will compare different approaches for negative sampling.
Revised: In the experiments, we will compare different approaches for negative sampling.

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETTING
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETTING

Original: We evaluate our proposed model on four widely used knowledge graphs.
Revised: We evaluate our proposed model on four widely used knowledge graphs.

Original: The statistics of these knowledge graphs are summarized into Table 3.
Revised: The statistics of these knowledge graphs are summarized into Table 3.

Original: • FB15k
Revised: • FB15k

Original: (Bordes et al., 2013) is a subset of Freebase (Bollacker et al., 2008), a large-scale knowledge graph containing general knowledge facts.
Revised: (Bordes et al., 2013) is a subset of Freebase (Bollacker et al., 2008), a large-scale knowledge graph containing general knowledge facts.

Original: Toutanova & Chen (2015) showed that almost 81% of the test triplets ( x , r , y ) can be inferred via a directly linked triplet ( x , r (cid:48) , y ) or ( y , r (cid:48) , x ) .
Revised: Toutanova & Chen (2015) showed that almost 81% of the test triplets ( x , r , y ) can be inferred via a directly linked triplet ( x , r (cid:48) , y ) or ( y , r (cid:48) , x ) .

Original: Therefore, the key of link prediction on FB15k is to model and infer the symmetry/antisymmetry and inversion patterns.
Revised: Therefore, the key of link prediction on FB15k is to model and infer the symmetry/antisymmetry and inversion patterns.

Original: [Table] • WN18 (Bordes et al., 2013) is a subset of WordNet (Miller, 1995), a database featuring lexical relations between words.
Revised: • WN18 (Bordes et al., 2013) is a subset of WordNet (Miller, 1995), a database featuring lexical relations between words.

Original: This dataset also has many inverse relations.
Revised: This dataset also has many inverse relations.

Original: So the main relation patterns in WN18 are also symmetry/antisymmetry and inversion .
Revised: So the main relation patterns in WN18 are also symmetry/antisymmetry and inversion .

Original: • FB15k-

Revised: • FB15k-

Original: (Toutanova & Chen, 2015) is a subset of FB15k, where inverse relations are deleted.
Revised: (Toutanova & Chen, 2015) is a subset of FB15k, where inverse relations are deleted.

Original: Therefore, the key of link prediction on FB15k-237 boils down to model and infer the symmetry/antisymmetry and composition patterns.
Revised: Therefore, the key of link prediction on FB15k-237 boils down to model and infer the symmetry/antisymmetry and composition patterns.

Original: • WN18RR
Revised: • WN18RR

Original: (Dettmers et al., 2017) is a subset of WN18.
Revised: (Dettmers et al., 2017) is a subset of WN18.

Original: The inverse relations are deleted, and the main relation patterns are symmetry/antisymmetry and composition .
Revised: The inverse relations are deleted, and the main relation patterns are symmetry/antisymmetry and composition .

Original: Hyperparameter Settings.
Revised: Hyperparameter Settings.

Original: We use Adam (Kingma & Ba, 2014) as the optimizer and ﬁne-tune the hyperparameters on the validation dataset.
Revised: We use Adam (Kingma & Ba, 2014) as the optimizer and ﬁne-tune the hyperparameters on the validation dataset.

Original: The ranges of the hyperparameters for the grid search are set as follows: embedding dimension 4 k ∈ { 125 , 250 , 500 , 1000 } , batch size b ∈ { 512 , 1024 , 2048 } , self-adversarial sampling temperature α ∈ { 0 . 5 , 1 . 0 } , and ﬁxed margin γ ∈ { 3 , 6 , 9 , 12 , 18 , 24 , 30 } .
Revised: The ranges of the hyperparameters for the grid search are set as follows: embedding dimension 5 k ∈ { 125 , 250 , 500 , 1000 } , batch size b ∈ { 512 , 1024 , 2048 } , self-adversarial sampling temperature α ∈ { 0 . 5 , 1 . 0 } , and ﬁxed margin γ ∈ { 3 , 6 , 9 , 12 , 18 , 24 , 30 } .

Original: Both the real and imaginary parts of the entity embeddings are uniformly initialized, and the phases of the relation embeddings are uniformly initialized between 0 and 2 π .
Revised: Both the real and imaginary parts of the entity embeddings are uniformly initialized, and the phases of the relation embeddings are uniformly initialized between 0 and 2 π .

Original: No regularization is used since we ﬁnd that the ﬁxed margin γ could prevent our model from over-ﬁtting.
Revised: No regularization is used since we ﬁnd that the ﬁxed margin γ could prevent our model from over-ﬁtting.

Original: Evaluation Settings.
Revised: Evaluation Settings.

Original: We evaluate the performance of link prediction in the ﬁltered setting: we rank test triples against all other candidate triples not appearing in the training, validation, or test set, where candidates are generated by corrupting subjects or objects: ( h (cid:48) , r , t ) or ( h , r , t (cid:48) ) .
Revised: We evaluate the performance of link prediction in the ﬁltered setting: we rank test triples against all other candidate triples not appearing in the training, validation, or test set, where candidates are generated by corrupting subjects or objects: ( h (cid:48) , r , t ) or ( h , r , t (cid:48) ) .

Original: Mean
Revised: Mean

Original: Rank (MR), Mean Reciprocal Rank (MRR) and Hits at N (H@N) are standard evaluation measures for these datasets and are evaluated in our experiments.
Revised: Rank (MR), Mean Reciprocal Rank (MRR) and Hits at N (H@N) are standard evaluation measures for these datasets and are evaluated in our experiments.

Original: Baseline.
Revised: Baseline.

Original: Apart from RotatE, we propose a variant of RotatE as baseline, where the modulus of the entity embeddings are also constrained: | h i | =
Revised: Apart from RotatE, we propose a variant of RotatE as baseline, where the modulus of the entity embeddings are also constrained: | h i | =

Original: | t i | =
Revised: | t i | =

Original: C , and the distance function is thus 2 C (cid:13) (cid:13) sin θ h
Revised: C , and the distance function is thus 2 C (cid:13)(cid:13) sin θ h

Original: + θ r − θ t(cid:13) (cid:13) (See Equation 17 at Appendix F for a detailed derivation).
Revised: + θ r − θ t(cid:13)(cid:13) (See Equation 17 at Appendix F for a detailed derivation).

Original: In this way, we can investigate how RotatE works without modulus information and with only phase information.
Revised: In this way, we can investigate how RotatE works without modulus information and with only phase information.

Original: We refer to the baseline as pRotatE. It is obvious to see that pRotatE can also model and infer all the three relation patterns.
Revised: We refer to the baseline as pRotatE. It is obvious to see that pRotatE can also model and infer all the three relation patterns.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: We compare RotatE to several state-of-the-art models, including TransE (Bordes et al., 2013), DistMult (Yang et al., 2014), ComplEx (Trouillon et al., 2016), HolE (Nickel et al., 2016), and ConvE (Dettmers et al., 2017), as well as our baseline model pRotatE, to empirically show the importance of modeling and inferring the relation patterns for the task of predicting missing links.
Revised: We compare RotatE to several state-of-the-art models, including TransE (Bordes et al., 2013), DistMult (Yang et al., 2014), ComplEx (Trouillon et al., 2016), HolE (Nickel et al., 2016), and ConvE (Dettmers et al., 2017), as well as our baseline model pRotatE, to empirically show the importance of modeling and inferring the relation patterns for the task of predicting missing links.

Original: Table 4 summarizes our results on FB15k and WN18.
Revised: Table 4 summarizes our results on FB15k and WN18.

Original: We can see that RotatE outperforms all the state-of-the-art models.
Revised: We can see that RotatE outperforms all the state-of-the-art models.

Original: The performance of pRotatE and RotatE are similar on these two datasets.
Revised: The performance of pRotatE and RotatE are similar on these two datasets.

Original: [Table] [Table] Table 5 summarizes our results on FB15k-237 and WN18RR, where the improvement is much more signiﬁcant.
Revised: Table 5 summarizes our results on FB15k-237 and WN18RR, where the improvement is much more signiﬁcant.

Original: The difference between RotatE and pRotatE is much larger on FB15k-237 and 
Revised: The difference between RotatE and pRotatE is much larger on FB15k-237 and [Table] [Table]

Original: WN18RR, where there are a lot of composition patterns.
Revised: WN18RR, where there are a lot of composition patterns.

Original: This indicates that modulus is very important for modeling and inferring the composition pattern.
Revised: This indicates that modulus is very important for modeling and inferring the composition pattern.

Original: Moreover, the performance of these models on different datasets is consistent with our analysis on the three relation patterns (Table 2):
Revised: Moreover, the performance of these models on different datasets is consistent with our analysis on the three relation patterns (Table 2):

Original: • On FB15K, the main relation patterns are symmetry/antisymmetry and inversion.
Revised: • On FB15K, the main relation patterns are symmetry/antisymmetry and inversion.

Original: We can see that ComplEx performs well while TransE does not perform well since ComplEx can infer both symmetry/antisymmetry and inversion patterns while TransE cannot infer symmetry pattern.
Revised: We can see that ComplEx performs well while TransE does not perform well since ComplEx can infer both symmetry/antisymmetry and inversion patterns while TransE cannot infer symmetry pattern.

Original: Surprisingly, DistMult achieves good performance on this dataset although it cannot model the antisymmetry and inversion patterns.
Revised: Surprisingly, DistMult achieves good performance on this dataset although it cannot model the antisymmetry and inversion patterns.

Original: The reason is that for most of the relations in FB15K, the types of head entities and tail entities are different.
Revised: The reason is that for most of the relations in FB15K, the types of head entities and tail entities are different.

Original: Although DistMult gives the same score to a true triplet ( h , r , t ) and its opposition triplet ( t , r , h ) , ( t , r , h ) is usually impossible to be valid since the entity type of t does not match the head entity type of h .
Revised: Although DistMult gives the same score to a true triplet ( h , r , t ) and its opposition triplet ( t , r , h ) , ( t , r , h ) is usually impossible to be valid since the entity type of t does not match the head entity type of h .

Original: For example, DistMult assigns the same score to ( Obama , nationality , USA ) and ( USA , nationality , Obama ) .
Revised: For example, DistMult assigns the same score to ( Obama , nationality , USA ) and ( USA , nationality , Obama ) .

Original: But ( USA , nationality , Obama ) can be simply predicted as false since USA cannot be the head entity of the relation nationality .
Revised: But ( USA , nationality , Obama ) can be simply predicted as false since USA cannot be the head entity of the relation nationality .

Original: • On WN18, the main relation patterns are also symmetry/antisymmetry and inversion.
Revised: • On WN18, the main relation patterns are also symmetry/antisymmetry and inversion.

Original: As expected, ComplEx still performs very well on this dataset.
Revised: As expected, ComplEx still performs very well on this dataset.

Original: However, different from the results on FB15K, the performance of DistMult signiﬁcantly decreases on WN18.
Revised: However, different from the results on FB15K, the performance of DistMult signiﬁcantly decreases on WN18.

Original: The reason is that DistMult cannot model antisymmetry and inversion patterns, and almost all the entities in WN18 are words and belong to the same entity type, which do not have the same problem as FB15K. • On FB15k-237, the main relation pattern is composition.
Revised: The reason is that DistMult cannot model antisymmetry and inversion patterns, and almost all the entities in WN18 are words and belong to the same entity type, which do not have the same problem as FB15K. • On FB15k-237, the main relation pattern is composition.

Original: We can see that TransE performs really well while ComplEx does not perform well.
Revised: We can see that TransE performs really well while ComplEx does not perform well.

Original: The reason is that, as discussed before, TransE is able to infer the composition pattern while ComplEx cannot infer the composition pattern.
Revised: The reason is that, as discussed before, TransE is able to infer the composition pattern while ComplEx cannot infer the composition pattern.

Original: • On WN18RR, one of the main relation patterns is the symmetry pattern since almost each word has a symmetric relation in WN18RR, e.g., also see and similar to .
Revised: • On WN18RR, one of the main relation patterns is the symmetry pattern since almost each word has a symmetric relation in WN18RR, e.g., also see and similar to .

Original: TransE does not well on this dataset since it is not able to model the symmetric relations. 
Revised: TransE does not well on this dataset since it is not able to model the symmetric relations. [Figure] [Table] [Figure]

Revised: [Table]

Original: 4. I NFERRING R ELATION P ATTERNS ON C OUNTRIES D ATA S ET
Revised: [Figure] [Table] [Figure] [Figure]

Revised: The symmetry, inversion and composition pattern is represented in Figure 2a, 2c and 2g, respectively.

Revised: 4. I NFERRING R ELATION P ATTERNS ON C OUNTRIES D ATA S ET

Original: We also evaluate our model on the Countries dataset (Bouchard et al., 2015; Nickel et al., 2016), which is carefully designed to explicitly test the capabilities of the link prediction models for composition pattern modeling and inferring.
Revised: We also evaluate our model on the Countries dataset (Bouchard et al., 2015; Nickel et al., 2016), which is carefully designed to explicitly test the capabilities of the link prediction models for composition pattern modeling and inferring.

Original: It contains 2 relations and 272 entities (244 countries,regions and 23 subregions).
Revised: It contains 2 relations and 272 entities (244 countries,regions and 23 subregions).

Original: Unlike link prediction on general knowledge graphs, the queries in
Revised: Unlike link prediction on general knowledge graphs, the queries in

Original: [Figure]

Original: Countries are of the form locatedIn ( c , ?) , and the answer is one of the ﬁve regions.
Revised: Countries are of the form locatedIn ( c , ?) , and the answer is one of the ﬁve regions.

Original: The Countries dataset has 3 tasks, each requiring inferring a composition pattern with increasing length and difﬁculty.
Revised: The Countries dataset has 3 tasks, each requiring inferring a composition pattern with increasing length and difﬁculty.

Original: For example, task S2 requires inferring a relatively simpler composition pattern: neighborOf
Revised: For example, task S2 requires inferring a relatively simpler composition pattern: neighborOf

Original: [Equation]
Revised: [Equation]

Revised: c 1 , c 2 )

Original: [Equation]
Revised: [Equation]

Revised: c 2 , r ) ⇒ locatedIn ( c 1 , r ) , while task S3 requires inferring the most complex composition pattern: neighborOf

Original: [Equation]
Revised: [Equation]

Original: locatedIn
Revised: s ,

Original: [Equation]
Revised: [Equation]

Original: c 2 , r ) ⇒ locatedIn ( c 1 , r ) , while task S3 requires inferring the most complex composition pattern: neighborOf ( c 1 , c 2 ) ∧ locatedIn ( c 2 , s ) ∧ locatedIn ( s , r ) ⇒ locatedIn ( c 1 , r ) .
Revised: ⇒ locatedIn ( c 1 , r ) .

Original: In Table 6, we report the results with respect to the AUC-PR metric, which is commonly used in the literature.
Revised: In Table 6, we report the results with respect to the AUC-PR metric, which is commonly used in the literature.

Original: We can see that RotatE outperforms all the previous models.
Revised: We can see that RotatE outperforms all the previous models.

Original: The performance of RotatE is signiﬁcantly better than other methods on S3, which is the most difﬁcult task.
Revised: The performance of RotatE is signiﬁcantly better than other methods on S3, which is the most difﬁcult task.

Original: 4. I MPLICIT R ELATION P ATTERN I NFERENCE
Revised: 4. I MPLICIT R ELATION P ATTERN I NFERENCE

Original: In this section, we verify whether the relation patterns are implicitly represented by RotatE relation embeddings.
Revised: In this section, we verify whether the relation patterns are implicitly represented by RotatE relation embeddings.

Original: We ignore the speciﬁc positions in the relation embedding θ r and plot the histogram of the phase of each element in the relation embedding, i.e., { θ r,i } .
Revised: We ignore the speciﬁc positions in the relation embedding θ r and plot the histogram of the phase of each element in the relation embedding, i.e., { θ r,i } .

Original: Symmetry pattern requires the symmetric relations to have property r ◦ r = 1 , and the solution is r i = ± 1 .
Revised: Symmetry pattern requires the symmetric relations to have property r ◦ r = 1 , and the solution is r i = ± 1 .

Original: We investigate the relation embeddings from a 500 -dimensional RotatE trained on WN18.
Revised: We investigate the relation embeddings from a 500 -dimensional RotatE trained on WN18.

Original: Figure 2a gives the histogram of the embedding phases of a symmetric relation similar to .
Revised: Figure 2a gives the histogram of the embedding phases of a symmetric relation similar to .

Original: We can ﬁnd that the embedding phases are either π ( r i = − 1 ) or 0 , 2 π ( r i = 1 ).
Revised: We can ﬁnd that the embedding phases are either π ( r i = − 1 ) or 0 , 2 π ( r i = 1 ).

Original: It indicates that the RotatE  model does infer and model the symmetry pattern.
Revised: It indicates that the RotatE [Table] [Table] model does infer and model the symmetry pattern.

Original: Figure 2b is the histogram of relation hypernym , which shows that the embedding of a general relation does not have such a ± 1 pattern.
Revised: Figure 2b is the histogram of relation hypernym , which shows that the embedding of a general relation does not have such a ± 1 pattern.

Original: Inversion pattern requires the embeddings of a pair of inverse relations to be conjugate.
Revised: Inversion pattern requires the embeddings of a pair of inverse relations to be conjugate.

Original: We use the same RotatE model trained on WN18 for an analysis.
Revised: We use the same RotatE model trained on WN18 for an analysis.

Original: Figure 2c illustrates the element-wise addition of the embedding phases from relation r 1 = hypernym and its inversed relation r 2 = hyponym .
Revised: Figure 2c illustrates the element-wise addition of the embedding phases from relation r 1 = hypernym and its inversed relation r 2 = hyponym .

Original: All the additive embedding phases are 0 or 2 π , which represents that r 1 = r − 1 2 .
Revised: All the additive embedding phases are 0 or 2 π , which represents that r 1 = r − 1 2 .

Original: This case shows that the inversion pattern is also inferred and modeled in the RotatE model.
Revised: This case shows that the inversion pattern is also inferred and modeled in the RotatE model.

Original: Composition pattern requires the embedding phases of the composed relation to be the addition of the other two relations.
Revised: Composition pattern requires the embedding phases of the composed relation to be the addition of the other two relations.

Original: Since there is no signiﬁcant composition pattern in WN18, we study the inference of the composition patterns on FB15k-237, where a 1000 -dimensional RotatE is trained.
Revised: Since there is no signiﬁcant composition pattern in WN18, we study the inference of the composition patterns on FB15k-237, where a 1000 -dimensional RotatE is trained.

Original: Figure 2d - 2g illustrate such a r 1 = r 2 ◦ r 3 case, where θ 2 ,i + θ 3 ,i
Revised: Figure 2d - 2g illustrate such a r 1 = r 2 ◦ r 3 case, where θ 2 ,i + θ 3 ,i

Original: =
Revised: =

Original: θ
Revised: θ

Original: 1 ,i or θ 2 ,i + θ 3 ,i
Revised: 1 ,i or θ 2 ,i + θ 3 ,i

Original: =
Revised: =

Original: θ
Revised: θ

Original: 1 ,i + 2 π .
Revised: 1 ,i + 2 π .

Original: More results of implicitly inferring basic patterns are presented in the appendix.
Revised: More results of implicitly inferring basic patterns are presented in the appendix.

Original: [Table]

Original: [Table]

Original: 4. C OMPARING DIFFERENT NEGATIVE SAMPLING TECHNIQUES
Revised: 4. C OMPARING DIFFERENT NEGATIVE SAMPLING TECHNIQUES

Original: In this part, we compare different negative sampling techniques including uniform sampling, our proposed self-adversarial technique, and the KBGAN model (Cai & Wang, 2017), which aims to optimize a generative adversarial network to generate the negative samples.
Revised: In this part, we compare different negative sampling techniques including uniform sampling, our proposed self-adversarial technique, and the KBGAN model (Cai & Wang, 2017), which aims to optimize a generative adversarial network to generate the negative samples.

Original: We re-implement a 50dimension TransE model with the margin-based ranking criterion that was used in (Cai & Wang, 2017) [Table] and evaluate its performance on FB15k-237, WN18RR and WN18 with self-adversarial negative sampling.
Revised: We re-implement a 50dimension TransE model with the margin-based ranking criterion that was used in (Cai & Wang, 2017), and evaluate its performance on FB15k-237, WN18RR and WN18 with self-adversarial negative sampling.

Original: Table 7 summarizes our results.
Revised: Table 7 summarizes our results.

Original: We can see that self-adversarial sampling is the most effective negative sampling technique.
Revised: We can see that self-adversarial sampling is the most effective negative sampling technique.

Original: 4. F URTHER E XPERIMENTS ON T RANS E AND C OMPL E X
Revised: 4. F URTHER E XPERIMENTS ON T RANS E AND C OMPL E X

Original: One may argue that the contribution of RotatE comes from the self-adversarial negative sampling technique.
Revised: One may argue that the contribution of RotatE comes from the self-adversarial negative sampling technique.

Original: In this part, we conduct further experiments on TransE and ComplEx in the same setting as RotatE to make a fair comparison among the three models.
Revised: In this part, we conduct further experiments on TransE and ComplEx in the same setting as RotatE to make a fair comparison among the three models.

Original: Table 8 shows the results of TransE and ComplEx trained with the self-adversarial negative sampling technique on FB15k and FB15k237 datasets, where a large number of relations are available.
Revised: Table 8 shows the results of TransE and ComplEx trained with the self-adversarial negative sampling technique on FB15k and FB15k237 datasets, where a large number of relations are available.

Original: In addition, we evaluate these three models on the Countries dataset, which explicitly requires inferring the composition pattern.
Revised: In addition, we evaluate these three models on the Countries dataset, which explicitly requires inferring the composition pattern.

Original: We also provide a detailed ablation study on TransE and RotatE in the appendix.
Revised: We also provide a detailed ablation study on TransE and RotatE in the appendix.

Original: From Table 8, we can see that similar results are observed as Table 4 and 5.
Revised: From Table 8, we can see that similar results are observed as Table 4 and 5.

Original: The RotatE model achieves the best performance on both FB15k and FB15k-237, as it is able to model all the three relation patterns.
Revised: The RotatE model achieves the best performance on both FB15k and FB15k-237, as it is able to model all the three relation patterns.

Original: The TransE model does not work well on the FB15k datasets, which requires modeling the symmetric pattern; the ComplEx model does not work well on FB15k-237, which requires modeling the composition pattern.
Revised: The TransE model does not work well on the FB15k datasets, which requires modeling the symmetric pattern; the ComplEx model does not work well on FB15k-237, which requires modeling the composition pattern.

Original: The results on the Countries dataset are a little bit different, where the TransE model slightly outperforms RoateE on the S3 task.
Revised: The results on the Countries dataset are a little bit different, where the TransE model slightly outperforms RoateE on the S3 task.

Original: The reason is that  the Countries datasets do not have the symmetric relation between different regions, and all the three tasks in the Countries datasets only require inferring the region for a given city.
Revised: The reason is that [Table] the Countries datasets do not have the symmetric relation between different regions, and all the three tasks in the Countries datasets only require inferring the region for a given city.

Original: Therefore, the
Revised: Therefore, the

Original: TransE model does not suffer from its inability of modeling symmetric relations.
Revised: TransE model does not suffer from its inability of modeling symmetric relations.

Original: For ComplEx, we can see that it does not perform well on Countries since it cannot infer the composition pattern.
Revised: For ComplEx, we can see that it does not perform well on Countries since it cannot infer the composition pattern.

Original: C ONCLUSION
Revised: 4. E XPERIMENTAL RESULTS ON FB15 K BY RELATION CATEGORY

Revised: We also did some further investigation on the performance of RotatE on different relation categories: one-to-many, many-to-one, and many-to-many relations 6 . The results of RotatE on different relation categories on the data set FB15k are summarized into Table 9. We also compare an additional approach KG2E KL

Revised: (He et al., 2015), which is a probabilistic framework for knowledge graph embedding methods and aims to model the uncertainties of the entities and relations in knowledge graphs with the TransE model.

Revised: We also summarize the statistics of different relation categories into

Revised: Table 10 in the appendix.

Revised: We can see that besides the one-to-one relation, the RotatE model also performs quite well on the non-injective relations, especially on many-to-many relations.

Revised: We also notice that the probabilistic framework KG2E KL(bern)

Revised: (He et al., 2015) is quite powerful, which consistently outperforms its corresponding knowledge graph embedding model, showing the importance of modeling the uncertainties in knowledge graphs.

Revised: We leave the work of modeling the uncertainties in knowledge graphs with RotatE as our future work.

Revised:  C ONCLUSION

Original: We have proposed a new knowledge graph embedding method called RotatE, which represents entities as complex vectors and relations as rotations in complex vector space.
Revised: We have proposed a new knowledge graph embedding method called RotatE, which represents entities as complex vectors and relations as rotations in complex vector space.

Original: In addition, we propose a novel self-adversarial negative sampling technique for efﬁciently and effectively training the RotatE model.
Revised: In addition, we propose a novel self-adversarial negative sampling technique for efﬁciently and effectively training the RotatE model.

Original: Our experimental results show that the RotatE model outperforms all existing state-of-theart models on four large-scale benchmarks.
Revised: Our experimental results show that the RotatE model outperforms all existing state-of-theart models on four large-scale benchmarks.

Original: Moreover, RotatE also achieves state-of-the-art results on a benchmark that is explicitly designed for composition pattern inference and modeling.
Revised: Moreover, RotatE also achieves state-of-the-art results on a benchmark that is explicitly designed for composition pattern inference and modeling.

Original: A deep investigation into RotatE relation embeddings shows that the three relation patterns are implicitly represented in the relation embeddings.
Revised: A deep investigation into RotatE relation embeddings shows that the three relation patterns are implicitly represented in the relation embeddings.

Original: In the future, we plan to evaluate the RotatE model on more datasets and release our source codes.
Revised: In the future, we plan to evaluate the RotatE model on more datasets and leverage a probabilistic framework to model the uncertainties of entities and relations.

Original: A PPENDIX A D ISCUSSION ON THE A BILITY OF P ATTERN M ODELING AND I NFERENCE
Revised: A PPENDIX A D ISCUSSION ON THE A BILITY OF P ATTERN M ODELING AND I NFERENCE

Original: No existing models are capable of modeling all the three relation patterns.
Revised: No existing models are capable of modeling all the three relation patterns.

Original: For example, TransE cannot model the symmetry pattern because it would yield r = 0 for symmetric relations; TransX can infer and model the symmetry/antisymmetry pattern when g r, 1 = g r, 2 , e.g. in TransH (Wang et al., 2014), but cannot infer inversion and composition as g r, 1 and g r, 2 are invertible matrix multiplications; due to its symmetric nature, DistMult is difﬁcult to model the asymmetric and inversion pattern; ComplEx addresses the problem of DisMult and is able to infer both the symmetry and asymmetric patterns with complex embeddings.
Revised: For example, TransE cannot model the symmetry pattern because it would yield r = 0 for symmetric relations; TransX can infer and model the symmetry/antisymmetry pattern when g r, 1 = g r, 2 , e.g. in TransH (Wang et al., 2014), but cannot infer inversion and composition as g r, 1 and g r, 2 are invertible matrix multiplications; due to its symmetric nature, DistMult is difﬁcult to model the asymmetric and inversion pattern; ComplEx addresses the problem of DisMult and is able to infer both the symmetry and asymmetric patterns with complex embeddings.

Original: Moreover, it can infer inversion rules because the complex conjugate of the solution to arg max r Re( (cid:104) x , r , y (cid:105) ) is exactly the solution to arg max r Re( (cid:104)
Revised: Moreover, it can infer inversion rules because the complex conjugate of the solution to arg max r Re( (cid:104) x , r , y (cid:105) ) is exactly the solution to arg max r Re( (cid:104)

Original: y , r , x (cid:105) ) .
Revised: y , r , x (cid:105) ) .

Original: However, ComplEx cannot infer composition rules, since it does not model a bijection mapping from h to t via relation r .
Revised: However, ComplEx cannot infer composition rules, since it does not model a bijection mapping from h to t via relation r .

Original: These concerns are summarized in Table 2.
Revised: These concerns are summarized in Table 2.

Original: B P ROOF OF L EMMA
Revised: B P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: if r ( x , y ) and r ( y , x ) hold, we have
Revised: if r ( x , y ) and r ( y , x ) hold, we have

Original: [Equation]
Revised: [Equation]

Original: Otherwise, if r ( x , y ) and ¬ r ( y , x ) hold, we have y 
Revised: Otherwise, if r ( x , y ) and ¬ r ( y , x ) hold, we have y =

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: =

Original: C P ROOF OF L EMMA
Revised: C P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: if r 1 ( x , y ) and r 2 ( y , x ) hold, we have y =
Revised: if r 1 ( x , y ) and r 2 ( y , x ) hold, we have y =

Original: r 1 ◦ x ∧ x = r 2 ◦ y ⇒ r
Revised: r 1 ◦ x ∧ x = r 2 ◦ y ⇒ r

Original: 1 = r − 

Revised: 1 = r −

Original: D P ROOF OF L EMMA
Revised: D P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: if r 1 ( x , z ) , r 2 ( x , y ) and r 3 ( y , z ) hold, we have 
Revised: if r 1 ( x , z ) , r 2 ( x , y ) and r 3 ( y , z ) hold, we have z

Original: [Equation]
Revised: [Equation]

Original: E P ROPERTIES OF R OTAT E
Revised: E P ROPERTIES OF R OTAT E

Original: A useful property for RotatE is that the inverse of a relation can be easily acquired by complex conjugate.
Revised: A useful property for RotatE is that the inverse of a relation can be easily acquired by complex conjugate.

Original: In this way, the RotatE model treats head and tail entities in a uniform way, which is potentially useful for efﬁcient 1-N scoring (Dettmers et al., 2017):
Revised: In this way, the RotatE model treats head and tail entities in a uniform way, which is potentially useful for efﬁcient 1-N scoring (Dettmers et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: Moreover, considering the embeddings in the polar form, i.e., h i =
Revised: Moreover, considering the embeddings in the polar form, i.e., h i =

Original: m
Revised: m

Original: h,i e iθ h,i , r i =
Revised: h,i e iθ h,i , r i =

Original: e iθ r,i , t i
Revised: e iθ r,i , t i

Original: = m t,i e iθ t,i , we can rewrite the RotatE distance function as:
Revised: = m t,i e iθ t,i , we can rewrite the RotatE distance function as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This equation provides two interesting views of the model:
Revised: This equation provides two interesting views of the model:

Original: (1) When we constrain the modulus m h,i
Revised: (1) When we constrain the modulus m h,i

Original: = m t,i =
Revised: = m t,i =

Original: C , the distance function is reduced to 2 C (cid:13) (cid:13) sin θ h + θ r − θ t(cid:13) (cid:13) .
Revised: C , the distance function is reduced to 2 C (cid:13) (cid:13) sin θ h + θ r − θ t(cid:13) (cid:13) .

Original: We can see that this is very similar to the distance function of TransE: (cid:107) h + r − t (cid:107) .
Revised: We can see that this is very similar to the distance function of TransE: (cid:107) h + r − t (cid:107) .

Original: Based on this intuition, we can show that:
Revised: Based on this intuition we can show that:

Original: Theorem 4.
Revised: Theorem 4.

Original: RotatE can degenerate
Revised: RotatE can degenerate

Original: into TransE (See proof at Appendix
Revised: into TransE. (See proof at Appendix

Original: F
Revised: F

Original: ) which indicates that RotatE is able to simulate TransE. (2)
Revised: ) which indicates that RotatE is able to simulate TransE. (2)

Original: The modulus provides the lower bound of the distance function, which is (cid:107) m h − m t (cid:107) .
Revised: The modulus provides the lower bound of the distance function, which is (cid:107) m h − m t (cid:107) .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: F P ROOF OF T HEOREM
Revised: F P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: By further restricting | h i | =
Revised: By further restricting | h i | =

Original: | t i | =
Revised: | t i | =

Original: C

Original: [Table]

Original:  we can rewrite h [Table] r , t by
Revised: C , we can rewrite h , r , t by

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: ) (cid:107) (14) =

Original:  [Equation]
Revised: C [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: If the embedding of ( h , r , t ) in TransE is h (cid:48) , r (cid:48) , t (cid:48) , let θ h
Revised: If the embedding of ( h , r , t ) in TransE is h (cid:48) , r (cid:48) , t (cid:48) , let θ h

Original: =
Revised: =

Original: c h (cid:48) , θ r = c r (cid:48) , θ t =
Revised: c h (cid:48) , θ r = c r (cid:48) , θ t =

Original: c t (cid:48) and C = 1 /c , we have
Revised: c t (cid:48) and C = 1 /c , we have

Original: [Equation]
Revised: [Equation]

Original: G L INK P REDICTION ON YAGO3- 
Revised: G L INK P REDICTION ON YAGO3-

Original: YAGO3-10 is a subset of YAGO3 (Mahdisoltani et al., 2013), which consists of entities that have a minimum of 10 relations each.
Revised: YAGO3-10 is a subset of YAGO3 (Mahdisoltani et al., 2013), which consists of entities that have a minimum of 10 relations each.

Original: It has 123,182 entities and 37 relations.
Revised: It has 123,182 entities and 37 relations.

Original: Most of the triples deal with descriptive attributes of people, such as citizenship, gender, profession and marital status.
Revised: Most of the triples deal with descriptive attributes of people, such as citizenship, gender, profession and marital status.

Original: Table 9 shows that the RotatE [Table] [Table]
Revised: Table 11 shows that the RotatE model also outperforms state-of-the-art models on YAGO3-10.

Original: H H YPERPARAMETERS
Revised: H H YPERPARAMETERS

Original: We list the best hyperparameter setting of RotatE w.r.t the validation dataset on several benchmarks in Table 10.
Revised: We list the best hyperparameter setting of RotatE w.r.t the validation dataset on several benchmarks in Table 12. [Table] [Table]

Original: I A BLATION S TUDY
Revised: I A BLATION S TUDY

Original: Table 11 shows our ablation study of self-adversarial sampling and negative sampling loss on
Revised: Table 13 shows our ablation study of self-adversarial sampling and negative sampling loss on

Original: FB15k-237.
Revised: FB15k-237.

Original: We also re-implement a 1000-dimension TransE and do ablation study on it.
Revised: We also re-implement a 1000-dimension TransE and do ablation study on it.

Original: From the table, We can ﬁnd that self-adversarial sampling boosts the performance for both models, while negative sampling loss is only effective on RotatE; in addition, our re-implementation of TransE also outperforms all the state-of-the-art models on FB15k-237.
Revised: From the table, We can ﬁnd that self-adversarial sampling boosts the performance for both models, while negative sampling loss is only effective on RotatE; in addition, our re-implementation of TransE also outperforms all the state-of-the-art models on FB15k-237.

Original: J V ARIANCE OF THE R ESULTS
Revised: J V ARIANCE OF THE R ESULTS

Original: In Table [Table] We provide the average and variance of the MRR results on FB15k, WN18, FB15k237 and WN18RR.
Revised: In Table 14, We provide the average and variance of the MRR results on FB15k, WN18, FB15k237 and WN18RR.

Original: Both the average and the variance is calculated by three runs of RotatE with difference random seeds.
Revised: Both the average and the variance is calculated by three runs of RotatE with difference random seeds.

Original: We can ﬁnd that the performance of RotatE is quite stable for different random initialization.
Revised: We can ﬁnd that the performance of RotatE is quite stable for different random initialization.

Original: K M ORE RESULTS OF IMPLICIT BASIC PATTERN INFERENCE
Revised: K M ORE RESULTS OF IMPLICIT BASIC PATTERN INFERENCE

Original: We provide more histograms of embedding phases in Figure 3 - 5.
Revised: We provide more histograms of embedding phases in Figure 3 - 5.

Original: [Figure] [Figure]
Revised: [Table] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]