Original: G ENERALIZED G RAPH E MBEDDING M ODELS
Revised: G ENERALIZED G RAPH E MBEDDING M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many types of relations in physical, biological, social and information systems can be modeled as homogeneous or heterogeneous concept graphs.
Revised: Many types of relations in physical, biological, social and information systems can be modeled as homogeneous or heterogeneous concept graphs.

Original: Hence, learning from and with graph embeddings has drawn a great deal of research interest recently, but only ad hoc solutions have been obtained this far.
Revised: Hence, learning from and with graph embeddings has drawn a great deal of research interest recently, but only ad hoc solutions have been obtained this far.

Original: In this paper, we conjecture that the one-shot supervised learning mechanism is a bottleneck in improving the performance of the graph embedding learning algorithms, and propose to extend this by introducing a multi-shot unsupervised learning framework.
Revised: In this paper, we conjecture that the one-shot supervised learning mechanism is a bottleneck in improving the performance of the graph embedding learning algorithms, and propose to extend this by introducing a multi-shot unsupervised learning framework.

Original: Empirical results on several real-world data set show that the proposed model consistently and signiﬁcantly outperforms existing state-of-the-art approaches on knowledge base completion and graph based multi-label classiﬁcation tasks.
Revised: Empirical results on several real-world data set show that the proposed model consistently and signiﬁcantly outperforms existing state-of-the-art approaches on knowledge base completion and graph based multi-label classiﬁcation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent studies have highlighted the importance of learning distributed representations for symbolic data in a wide variety of artiﬁcial intelligence tasks (Bengio et al., 2013).
Revised: Recent studies have highlighted the importance of learning distributed representations for symbolic data in a wide variety of artiﬁcial intelligence tasks (Bengio et al., 2013).

Original: Research on word embeddings (Mikolov et al., 2013) has led to breakthroughs in many related areas, such as machine translation (Bahdanau et al., 2015), question answering (Xiong et al., 2016), and visual-semantic alignments (Karpathy & Fei-Fei, 2017).
Revised: Research on word embeddings (Mikolov et al., 2013) has led to breakthroughs in many related areas, such as machine translation (Bahdanau et al., 2015), question answering (Xiong et al., 2016), and visual-semantic alignments (Karpathy & Fei-Fei, 2017).

Original: However, learning to predict for large-scale knowledge graphs (KGs) is still a challenging problem left, this is largely due to the diversity of the ontologies, and the semantic richness of the concepts, which makes it really hard to generate proper and universally applicable graph embeddings, simply based on word-level embeddings (Cai et al., 2017).
Revised: However, learning to predict for large-scale knowledge graphs (KGs) is still a challenging problem left, this is largely due to the diversity of the ontologies, and the semantic richness of the concepts, which makes it really hard to generate proper and universally applicable graph embeddings, simply based on word-level embeddings (Cai et al., 2017).

Original: Being able to generate reasonable and accurate distributed representations for large-scale knowledge graphs would be particularly valuable, in that it may help predict unobserved facts from limited concepts, uncover gaps in our knowledge, suggest new downstream applications, which clearly reﬂects the central concerns of the artiﬁcial intelligence (Nickel et al., 2016a; Henaff et al., 2017).
Revised: Being able to generate reasonable and accurate distributed representations for large-scale knowledge graphs would be particularly valuable, in that it may help predict unobserved facts from limited concepts, uncover gaps in our knowledge, suggest new downstream applications, which clearly reﬂects the central concerns of the artiﬁcial intelligence (Nickel et al., 2016a; Henaff et al., 2017).

Original: Therefore, massive attention has been devoted to the potential of embedding entities and relationships of multi-relational data in low-dimensional vector spaces in recent years (Wang et al., 2017).
Revised: Therefore, massive attention has been devoted to the potential of embedding entities and relationships of multi-relational data in low-dimensional vector spaces in recent years (Wang et al., 2017).

Original: In this paper, we consider the problem of developing simple and efﬁcient model for learning neural representation of generalized knowledge graphs , including the multi-relational heterogeneous graphs, and more speciﬁcally deﬁned homogeneous graphs (such as social and biological networks).
Revised: In this paper, we consider the problem of developing simple and efﬁcient model for learning neural representation of generalized knowledge graphs , including the multi-relational heterogeneous graphs, and more speciﬁcally deﬁned homogeneous graphs (such as social and biological networks).

Original: Following the pioneer work of Nickel et al.
Revised: Following the pioneer work of Nickel et al.

Original: (2011) and Bordes et al.
Revised: (2011) and Bordes et al.

Original: (2013), almost all of the stateof-the-art approaches try to model the graph embedding learning problem as supervised binary classiﬁcation problems, their objective functions are usually one-shot (single purpose) .
Revised: (2013), almost all of the stateof-the-art approaches try to model the graph embedding learning problem as supervised binary classiﬁcation problems, their objective functions are usually one-shot (single purpose) .

Original: We argue that prior research in this area might have been affected and biased by “ established priors”, which prevents the formulation of a methodology that is objective enough to cope with the highly sparse knowledge graphs.
Revised: We argue that prior research in this area might have been affected and biased by “ established priors”, which prevents the formulation of a methodology that is objective enough to cope with the highly sparse knowledge graphs.

Original: We propose to handle the embedded learning problem of knowledge graphs with an unsupervised neural network model, called the Graph Embedding Network ( GEN ).
Revised: We propose to handle the embedded learning problem of knowledge graphs with an unsupervised neural network model, called the Graph Embedding Network ( GEN ).

Original: The proposed model consists of three simple multi-layer perceptron (MLP) cells, each cell operates in response to a different “query” with regard to the input fact, which will be trained sequentially.
Revised: The proposed model consists of three simple multi-layer perceptron (MLP) cells, each cell operates in response to a different “query” with regard to the input fact, which will be trained sequentially.

Original: The formulation of the model is inspired by the neural sequence-to-sequence (seq2seq) model (Sutskever et al., 2014), except that we attempt to use the MLP cells to mimic the sequence learning capability of the recurrent neural network (RNN), to model the semantic structure of the knowledge graphs.
Revised: The formulation of the model is inspired by the neural sequence-to-sequence (seq2seq) model (Sutskever et al., 2014), except that we attempt to use the MLP cells to mimic the sequence learning capability of the recurrent neural network (RNN), to model the semantic structure of the knowledge graphs.

Original: The major contribution of this paper is that: (1) we propose GEN, a novel and efﬁcient multishot framework for embedding learning in generalized knowledge graphs.
Revised: The major contribution of this paper is that: (1) we propose GEN, a novel and efﬁcient multishot framework for embedding learning in generalized knowledge graphs.

Original: (2) We show how GEN is in accordance with established principles in cognitive science, providing ﬂexibility in learning representations that works on graphs conforming to different domains.
Revised: (2) We show how GEN is in accordance with established principles in cognitive science, providing ﬂexibility in learning representations that works on graphs conforming to different domains.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: During the last few years, an increasing amount of research attention has been devoted to the challenge of representation learning on knowledge graphs, especially focused on the potential beneﬁts for the knowledge base completion (KBC) tasks, including the link prediction problem and the relation prediction problem.
Revised: During the last few years, an increasing amount of research attention has been devoted to the challenge of representation learning on knowledge graphs, especially focused on the potential beneﬁts for the knowledge base completion (KBC) tasks, including the link prediction problem and the relation prediction problem.

Original: Among which, the relation translating model TransE (Bordes et al., 2013), the tensor factorization based semantic matching model RESCAL (Nickel et al., 2011), and the neural network based semantic matching model ER-MLP (Dong et al., 2014; Nickel et al., 2016b), are probably the most heavily studied from the methodology perspective.
Revised: Among which, the relation translating model TransE (Bordes et al., 2013), the tensor factorization based semantic matching model RESCAL (Nickel et al., 2011), and the neural network based semantic matching model ER-MLP (Dong et al., 2014; Nickel et al., 2016b), are probably the most heavily studied from the methodology perspective.

Original: For good surveys on such embedding learning algorithms, see Nickel et al.
Revised: For good surveys on such embedding learning algorithms, see Nickel et al.

Original: (2016a), Wang et al.
Revised: (2016a), Wang et al.

Original: (2017), and Cai et al.
Revised: (2017), and Cai et al.

Original: Broadly speaking, related works can be divided into two categories: linear and non-linear, according to whether the output embedding has a reasonable linear interpretation.
Revised: Broadly speaking, related works can be divided into two categories: linear and non-linear, according to whether the output embedding has a reasonable linear interpretation.

Original: State-of-the-art linear models include the TransE, RESCAL, TranH (Wang et al., 2014), DistMult (Yang et al.), and
Revised: State-of-the-art linear models include the TransE, RESCAL, TranH (Wang et al., 2014), DistMult (Yang et al.), and

Original: ANALOGY (Liu et al., 2017), while the popular non-linear models include the ER-MLP, ComplEX 

Revised: ANALOGY (Liu et al., 2017), while the popular non-linear models include the ER-MLP, ComplEX

Original: (Trouillon et al., 2016), HoIE (Nickel et al., 2016b), ProjE (Shi & Weninger, 2017) and
Revised: (Trouillon et al., 2016), HoIE (Nickel et al., 2016b), ProjE (Shi & Weninger, 2017) and

Original: ConvE (Dettmers et al., 2017).
Revised: ConvE (Dettmers et al., 2017).

Original: The proposed GEN model is also a non-linear model.
Revised: The proposed GEN model is also a non-linear model.

Original: The graph embedding learning models that is most closely related to this work is probably the ProjE model, which makes use of an embedding projection function deﬁned as:
Revised: The graph embedding learning models that is most closely related to this work is probably the ProjE model, which makes use of an embedding projection function deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where h ; r ; t denote the embedding vectors, f ( (cid:1) ) and g ( (cid:1) ) are non-linear activation functions, w 0 , w r 1 and w t 1 are learnable weight matrices, b 0 and b 1 are bias vectors.
Revised: where h ; r ; t denote the embedding vectors, f ( (cid:1) ) and g ( (cid:1) ) are non-linear activation functions, w 0 , w r 1 and w t 1 are learnable weight matrices, b 0 and b 1 are bias vectors.

Original: The output ranking scores of entity h with regard to the given query ( ? ; r; t ) can be obtained through a softmax function:
Revised: The output ranking scores of entity h with regard to the given query ( ? ; r; t ) can be obtained through a softmax function:

Original: [Equation]
Revised: [Equation]

Original: However, as one could see from above functions, the ProjE model is built upon the query ( ? ; r; t ), hence is a one-shot solution, which is distinctly different from our GEN model.
Revised: However, as one could see from above functions, the ProjE model is built upon the query ( ? ; r; t ), hence is a one-shot solution, which is distinctly different from our GEN model.

Original: Still another difference lies in the deﬁnition of the objective loss function, the ProjE model choose to use the (selective) cross-entropy loss based on the open world assumption, while our model uses a simpliﬁed cross-entropy loss based on the close world assumption.
Revised: Still another difference lies in the deﬁnition of the objective loss function, the ProjE model choose to use the (selective) cross-entropy loss based on the open world assumption, while our model uses a simpliﬁed cross-entropy loss based on the close world assumption.

Original: In order to save the computation cost, the
Revised: In order to save the computation cost, the

Original: ProjE model introduced a negative sampling process, this could cause potential risks for introducing additional bias.
Revised: ProjE model introduced a negative sampling process, this could cause potential risks for introducing additional bias.

Original: Besides, its candidate sampling process is time consuming and hard to be paralleled.
Revised: Besides, its candidate sampling process is time consuming and hard to be paralleled.

Original: Another model that is closely related to the GEN model is the ER-MLP model, which can be interpreted as creating representation for each element of triples and deriving their existence from this representation (Nickel et al., 2016a).
Revised: Another model that is closely related to the GEN model is the ER-MLP model, which can be interpreted as creating representation for each element of triples and deriving their existence from this representation (Nickel et al., 2016a).

Original: The ER-MLP model can be deﬁned as:
Revised: The ER-MLP model can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where symbol (cid:8) denotes the vector concatenation operator, vector w and matrix C are global weight vectors shared by all the entities and relations, g ( (cid:1) ) is an element-wise non-linear activation function.
Revised: where symbol (cid:8) denotes the vector concatenation operator, vector w and matrix C are global weight vectors shared by all the entities and relations, g ( (cid:1) ) is an element-wise non-linear activation function.

Original: This model is built upon the fourth query as we deﬁned in Section 3, it is a supervised solution, which is quite different from ours.
Revised: This model is built upon the fourth query as we deﬁned in Section 3, it is a supervised solution, which is quite different from ours.

Original: One well-known disadvantage of the ER-MLP is that, even properly regularized, it is still easily prone to over-ﬁtting on knowledge graph datasets (Nickel et al., 2016b), therefore we do not compare with it in this work, but instead with the ProjE model.
Revised: One well-known disadvantage of the ER-MLP is that, even properly regularized, it is still easily prone to over-ﬁtting on knowledge graph datasets (Nickel et al., 2016b), therefore we do not compare with it in this work, but instead with the ProjE model.

Original: As mentioned before, the primary motivation of this study is to develop a graph embedding model that is universally applicable to a wide variety of situations.
Revised: As mentioned before, the primary motivation of this study is to develop a graph embedding model that is universally applicable to a wide variety of situations.

Original: In order to verify the validity of our solution on heterogeneous networks, we further test it on multi-label network classiﬁcation tasks for social networks (BlogCatalog) and biological networks (Protein-Protein Interaction), and compare our results with two state-of-the-art techniques, namely, DeepWalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016).
Revised: In order to verify the validity of our solution on heterogeneous networks, we further test it on multi-label network classiﬁcation tasks for social networks (BlogCatalog) and biological networks (Protein-Protein Interaction), and compare our results with two state-of-the-art techniques, namely, DeepWalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016).

Original: Both of them are derived directly from the word2vec model (Mikolov et al., 2013), which creating node embeddings of the graphs based on the skip-gram framework, and train the model with corpus generated through random walking on that graph.
Revised: Both of them are derived directly from the word2vec model (Mikolov et al., 2013), which creating node embeddings of the graphs based on the skip-gram framework, and train the model with corpus generated through random walking on that graph.

Original: However, it is shown that the random walk sampling can be insufﬁcient for supervised learning tasks in the sparse network environment (Liu et al., 2016).
Revised: However, it is shown that the random walk sampling can be insufﬁcient for supervised learning tasks in the sparse network environment (Liu et al., 2016).

Original: Our results support this conjecture, the experimental results on benchmark tests provide strong evidence that our model performs much better.
Revised: Our results support this conjecture, the experimental results on benchmark tests provide strong evidence that our model performs much better.

Original: Embedding Dictionary (h,r,t) (h,r,?)
Revised: Embedding Dictionary (h,r,t) (h,r,?)

Original: (?,r,t) (h,?,t)
Revised: (?,r,t) (h,?,t)

Original: Ne
Revised: Ne

Original: E_CELL E_CELL
Revised: E_CELL E_CELL

Original: R_CELL
Revised: R_CELL

Original: Nr concatenation concatenation
Revised: Nr concatenation concatenation

Original: EntitySoftmaxVector
Revised: EntitySoftmaxVector

Original: RelationSoftmaxVector
Revised: RelationSoftmaxVector

Original: Sharing parameters k 2d ´
Revised: Sharing parameters k 2d ´

Original: Ne k ´
Revised: Ne k ´

Original: N r k ´ k 2d ´ 1 r r W ,b 0 r r W , b 1 e e W , b 0 e e W , b 1 e e W , b 0 e e W , b 1 1 e e W , b 0 

Revised: N r k ´ k 2d ´ 1 r r W ,b 0 r r W , b 1 e e W , b 0 e e W , b 1 e e W , b 0 e e W , b 1 1 e e W , b 0

Original: e e W , b 1 1 r r W ,b 0 0 r r W , b
Revised: e e W , b 1 1 r r W ,b 0 0 r r W , b

Original:  A PPROACH AND M ODEL A RCHITECTURE
Revised:  A PPROACH AND M ODEL A RCHITECTURE

Original: Most of the prevalent semantic knowledge databases are built upon the Resource Description Framework (RDF) , in which the facts are represented and stored in the form of SPO (Subject, Predicate,
Revised: Most of the prevalent semantic knowledge databases are built upon the Resource Description Framework (RDF) , in which the facts are represented and stored in the form of SPO (Subject, Predicate,

Original: Object) triples .
Revised: Object) triples .

Original: Following the convention, we will use the symbol ( h; r; t ) to represent a unit of facts , in which h; r and t denote the head entity, the relation , and the tail entity, respectively.
Revised: Following the convention, we will use the symbol ( h; r; t ) to represent a unit of facts , in which h; r and t denote the head entity, the relation , and the tail entity, respectively.

Original: The primary motivation of this paper is to develop a representation learning method that is suitable and ﬂexible enough for modeling different types of knowledge graphs from a universal perspective.
Revised: The primary motivation of this paper is to develop a representation learning method that is suitable and ﬂexible enough for modeling different types of knowledge graphs from a universal perspective.

Original: To achieve this objective, the most important problems to be faced are associated with: how to deﬁne the optimization problem and how to solve it .
Revised: To achieve this objective, the most important problems to be faced are associated with: how to deﬁne the optimization problem and how to solve it .

Original: As mentioned above, previous works only consider a one-shot mapping from the embedding space to the criterion space, which we conjecture, would be vulnerable to loss considerable amount of the structured semantic information.
Revised: As mentioned above, previous works only consider a one-shot mapping from the embedding space to the criterion space, which we conjecture, would be vulnerable to loss considerable amount of the structured semantic information.

Original: For instance, if given a fact ( Elvis Presley, profession, singer ), one could immediately learn the following queries:
Revised: For instance, if given a fact ( Elvis Presley, profession, singer ), one could immediately learn the following queries:

Original: (cid:15) Q1:
Revised: (cid:15) Q1:

Original: What is the profession of Elvis Presley ?
Revised: What is the profession of Elvis Presley ?

Original: A1: singer .
Revised: A1: singer .

Original: (cid:15) Q2:
Revised: (cid:15) Q2:

Original: Can you name a person whose profession is singer ?
Revised: Can you name a person whose profession is singer ?

Original: A2: Elvis Presley .
Revised: A2: Elvis Presley .

Original: (cid:15) Q3:
Revised: (cid:15) Q3:

Original: What is the possible relationship in between Elvis Presley and singer ?
Revised: What is the possible relationship in between Elvis Presley and singer ?

Original: A3: profession .
Revised: A3: profession .

Original: (cid:15) Q4:
Revised: (cid:15) Q4:

Original: Is it true that Elvis Presley ’s profession is singer ?
Revised: Is it true that Elvis Presley ’s profession is singer ?

Original: A4:
Revised: A4:

Original: Yes.
Revised: Yes.

Original: In fact, this is the actual way we humans learn the meaning of concepts expressed by a statement.
Revised: In fact, this is the actual way we humans learn the meaning of concepts expressed by a statement.

Original: These self-labeled queries reﬂect the following modeling philosophy:
Revised: These self-labeled queries reﬂect the following modeling philosophy:

Original: (1) ( h; r ) ) t ; (2) ( t; r ) ) h ; (3) ( h; t ) )
Revised: (1) ( h; r ) ) t ; (2) ( t; r ) ) h ; (3) ( h; t ) )

Original: r ; (4) ( h; r; t ) )
Revised: r ; (4) ( h; r; t ) )

Original: T/F; respectively.
Revised: T/F; respectively.

Original: This has been exclusively adopted by the previous research.
Revised: This has been exclusively adopted by the previous research.

Original: However, none of them have systematically investigated the effect of combination all of such information.
Revised: However, none of them have systematically investigated the effect of combination all of such information.

Original: In this section, we propose a novel multi-shot model to solve this problem.
Revised: In this section, we propose a novel multi-shot model to solve this problem.

Original: For a more detailed discussion of the motivation and intuition behind this model, see Appendix A.
Revised: For a more detailed discussion of the motivation and intuition behind this model, see Appendix A.

Original: 3. O VERVIEW OF THE MULTI - SHOT L EARNING F RAMEWORK
Revised: 3. O VERVIEW OF THE MULTI - SHOT L EARNING F RAMEWORK

Original: The proposed model (GEN) is designed to process data in sequential form.
Revised: The proposed model (GEN) is designed to process data in sequential form.

Original: As shown in Fig.1, GEN consists of three components (cells), each corresponding to an individual query with regard to the given input triple.
Revised: As shown in Fig.1, GEN consists of three components (cells), each corresponding to an individual query with regard to the given input triple.

Original: In this study, we propose to use a 2-layer MLP network to deal with the parameter estimation problem for each query individually, although it can be substituted by any other one-shot models, we only report the test results on MLP cells for simplicity.
Revised: In this study, we propose to use a 2-layer MLP network to deal with the parameter estimation problem for each query individually, although it can be substituted by any other one-shot models, we only report the test results on MLP cells for simplicity.

Original: In training mode, the training set is fed into the system sequentially, each of the triple is decomposed into three self-labeled queries: ( h; r; ?) )
Revised: In training mode, the training set is fed into the system sequentially, each of the triple is decomposed into three self-labeled queries: ( h; r; ?) )

Original: t , (? ; r; t ) ) h , and ( h; ?
Revised: t , (? ; r; t ) ) h , and ( h; ?

Original: ; t ) ) r .
Revised: ; t ) ) r .

Original: Each of the queries is fetched into the corresponding cell in order to update the parameters.
Revised: Each of the queries is fetched into the corresponding cell in order to update the parameters.

Original: Since for any given triple, our model would read it from three different perspective, we call it “multi-shot model” to distinguish it from other related works.
Revised: Since for any given triple, our model would read it from three different perspective, we call it “multi-shot model” to distinguish it from other related works.

Original: h r t h t
Revised: h r t h t

Original: R_CELL r h r
Revised: R_CELL r h r

Original: E_CELL t t r
Revised: E_CELL t t r

Original: E_CELL h
Revised: E_CELL h

Original: S TInput Sequence:
Revised: S TInput Sequence:

Original: S T+1 S T
Revised: S T+1 S T

Original: GEN 1 e e W , b 0 e e W , b 1 r r W , b 0 r r W , b 1 r r W , b 0 r r W , b 1 e e W , b 0 e e W , b
Revised: GEN 1 e e W , b 0 e e W , b 1 r r W , b 0 r r W , b 1 r r W , b 0 r r W , b 1 e e W , b 0 e e W , b

Original: Parameters of the model can be logically divided into two parts.
Revised: Parameters of the model can be logically divided into two parts.

Original: Firstly, the distribution representation of the entities and the relations are deﬁned in the same d -dimensional space, which, as shown in Fig.1, are organized together as a learnable dictionary of embeddings.
Revised: Firstly, the distribution representation of the entities and the relations are deﬁned in the same d -dimensional space, which, as shown in Fig.1, are organized together as a learnable dictionary of embeddings.

Original: Secondly, there exist two types of MLP cells in the model, one deals with the entity prediction tasks, the other is responsible for the relation prediction tasks, which are marked as “E CELL” and “R CELL” respectively.
Revised: Secondly, there exist two types of MLP cells in the model, one deals with the entity prediction tasks, the other is responsible for the relation prediction tasks, which are marked as “E CELL” and “R CELL” respectively.

Original: Each individual cell has its own parameter set f W 0 ; b 0 ; W 1 ; b 1 g representing certain network structures.
Revised: Each individual cell has its own parameter set f W 0 ; b 0 ; W 1 ; b 1 g representing certain network structures.

Original: Please note that two E CELLs are introduced to learn from the labeled entities, based on query ( h; r; ?) and (? ; r; t ) .
Revised: Please note that two E CELLs are introduced to learn from the labeled entities, based on query ( h; r; ?) and (? ; r; t ) .

Original: According to our modeling hypothesis, which claims that all of the relations should be treated conceptually instead of syntactically, we propose to share parameters between the
Revised: According to our modeling hypothesis, which claims that all of the relations should be treated conceptually instead of syntactically, we propose to share parameters between the

Original: E CELLs, the intuition behind is to let them share their memory of each known facts from both side of the relation , so that after training with enough knowledge, the E CELLs will eventually able to learn how to correctly distinguish between valid and invalid entities for the given queries.
Revised: E CELLs, the intuition behind is to let them share their memory of each known facts from both side of the relation , so that after training with enough knowledge, the E CELLs will eventually able to learn how to correctly distinguish between valid and invalid entities for the given queries.

Original: Another theoretical explanation of the GEN model is given below.
Revised: Another theoretical explanation of the GEN model is given below.

Original: We consider the proposed model as a variant of the RNN model, or more precisely, a neural seq2seq model, as illustrated in Fig.2.
Revised: We consider the proposed model as a variant of the RNN model, or more precisely, a neural seq2seq model, as illustrated in Fig.2.

Original: When training with the graph (a “document” of triples), the GEN model is organized as a stacked
Revised: When training with the graph (a “document” of triples), the GEN model is organized as a stacked

Original: RNN, which consists of two chains: the E CELL chains and the R CELL chains.
Revised: RNN, which consists of two chains: the E CELL chains and the R CELL chains.

Original: For any given input ( h; r; t ) , each of the cells works as an individual seq2seq model according to its responsive query.
Revised: For any given input ( h; r; t ) , each of the cells works as an individual seq2seq model according to its responsive query.

Original: For instance, the R CELL is responsible to query ( h; ?
Revised: For instance, the R CELL is responsible to query ( h; ?

Original: ; t ) )
Revised: ; t ) )

Original: r , it will take the embedding of h and r as input, and take r as its target label, and the parameters (memory) of the R CELL will be updated through back-propagation according to the discrepancy between the prediction results (in this case the softmax vector) and the desired label r .
Revised: r , it will take the embedding of h and r as input, and take r as its target label, and the parameters (memory) of the R CELL will be updated through back-propagation according to the discrepancy between the prediction results (in this case the softmax vector) and the desired label r .

Original: Therefore, the proposed model is completely unsupervised, which is distinctly different from previous works.
Revised: Therefore, the proposed model is completely unsupervised, which is distinctly different from previous works.

Original: Also please note that due to the lack of semantic connections between the adjacent triples in the input sequence, we did not consider the “long term memory” in this model, as usually did in real RNN models.
Revised: Also please note that due to the lack of semantic connections between the adjacent triples in the input sequence, we did not consider the “long term memory” in this model, as usually did in real RNN models.

Original: Therefore, there only exists one “global memory” in this model — the parameter of the two types of cells, which is responsible for “learning to remember” the rules of how the knowledge graph is constructed.
Revised: Therefore, there only exists one “global memory” in this model — the parameter of the two types of cells, which is responsible for “learning to remember” the rules of how the knowledge graph is constructed.

Original: 3. D EFINITION OF THE GEN CELLS
Revised: 3. D EFINITION OF THE GEN CELLS

Original: The network structure of the E CELLs and the R CELLs are quite similar, the only difference is that they have different number of neurons in the hidden layer and the output layer, which are deﬁned as hyper-parameters as shown in Fig.1.
Revised: The network structure of the E CELLs and the R CELLs are quite similar, the only difference is that they have different number of neurons in the hidden layer and the output layer, which are deﬁned as hyper-parameters as shown in Fig.1.

Original: For simplicity, we only present the implementation details of the E CELLs here.
Revised: For simplicity, we only present the implementation details of the E CELLs here.

Original: In order to answer query ( h; ? ; t ) ) r , the hidden layer of the E CELL takes input from the embedding dictionary according to label h and r , the hidden layer is deﬁned as:
Revised: In order to answer query ( h; ? ; t ) ) r , the hidden layer of the E CELL takes input from the embedding dictionary according to label h and r , the hidden layer is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where x 0 =
Revised: where x 0 =

Original: [ h (cid:8) r ] , denotes the concatenation of the embedding vectors, hence the x 0 is a 2 d (cid:2)real-value vector.
Revised: [ h (cid:8) r ] , denotes the concatenation of the embedding vectors, hence the x 0 is a 2 d (cid:2)real-value vector.

Original: W eo is a k (cid:2) 

Revised: W eo is a k (cid:2)

Original: d weights matrix, b 0 is a k (cid:2) 1 bias vector, k denotes the number of neurons in the hidden layer, and f ( (cid:1) ) is a non-linear activation function, in this work, we use the rectiﬁed linear unit (ReLU) function for all the experiments (Nair & Hinton, 2010).
Revised: d weights matrix, b 0 is a k (cid:2) 1 bias vector, k denotes the number of neurons in the hidden layer, and f ( (cid:1) ) is a non-linear activation function, in this work, we use the rectiﬁed linear unit (ReLU) function for all the experiments (Nair & Hinton, 2010).

Original: The output layer takes the hidden state vector x 1 as input, mapping it to the target label space:
Revised: The output layer takes the hidden state vector x 1 as input, mapping it to the target label space:

Original: [Equation]
Revised: [Equation]

Original: where W e 1 is a N e (cid:2) k weights matrix, b 1 is a N e (cid:2) 1 bias vector, N e denotes the number of entities in the dictionary, g ( (cid:1) ) denotes the softmax function.
Revised: where W e 1 is a N e (cid:2) k weights matrix, b 1 is a N e (cid:2) 1 bias vector, N e denotes the number of entities in the dictionary, g ( (cid:1) ) denotes the softmax function.

Original: Hence, ^ y is a N e (cid:2) 1 probability vector, which means that, when training the model with a given fact ( h; r; t ) to answer the query ( h; r; ? ), the predictive results output by the model is a probabilistic distribution over all of the possible candidate entities.
Revised: Hence, ^ y is a N e (cid:2) 1 probability vector, which means that, when training the model with a given fact ( h; r; t ) to answer the query ( h; r; ? ), the predictive results output by the model is a probabilistic distribution over all of the possible candidate entities.

Original: The cross-entropy loss with regard to prediction results is then deﬁned as:
Revised: The cross-entropy loss with regard to prediction results is then deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where y denotes the ground truth, which is a one-hot vector exclusively activated by t .
Revised: where y denotes the ground truth, which is a one-hot vector exclusively activated by t .

Original: To speed-up the stochastic convex optimization process, we use a mini-batch setting, and rewrite the averaged cross-entropy loss over a batch of multiple samples of size N as following simpliﬁed form:
Revised: To speed-up the stochastic convex optimization process, we use a mini-batch setting, and rewrite the averaged cross-entropy loss over a batch of multiple samples of size N as following simpliﬁed form:

Original: [Equation]
Revised: [Equation]

Original: where the subscript i denotes the i -th sample of the batch, t i represent the index of label t in the ground truth vector of that sample.
Revised: where the subscript i denotes the i -th sample of the batch, t i represent the index of label t in the ground truth vector of that sample.

Original: Eq.4 is computationally efﬁcient, however, it tend to ignores the existing knowledge for query ( h; r; ?)
Revised: Eq.4 is computationally efﬁcient, however, it tend to ignores the existing knowledge for query ( h; r; ?)

Original: other than the current fact ( h; r; t ) , which has been proven to be useful for improving performance (Shi & Weninger, 2017).
Revised: other than the current fact ( h; r; t ) , which has been proven to be useful for improving performance (Shi & Weninger, 2017).

Original: But, our experimental results show that the impact of such a problem can be controlled by means of collaborative correction with related facts under our model framework, which further demonstrate the validity of our modelling assumptions.
Revised: But, our experimental results show that the impact of such a problem can be controlled by means of collaborative correction with related facts under our model framework, which further demonstrate the validity of our modelling assumptions.

Original: Hopefully, the lessons learned for designing reasonable and computationally efﬁcient cost functions in this study can serve as exemplars for future work.
Revised: Hopefully, the lessons learned for designing reasonable and computationally efﬁcient cost functions in this study can serve as exemplars for future work.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We evaluate the proposed model on two distinctly different types of graph embedding learning tasks.
Revised: We evaluate the proposed model on two distinctly different types of graph embedding learning tasks.

Original: Firstly, we evaluate our model on knowledge base completion tasks with the conventional datasets
Revised: Firstly, we evaluate our model on knowledge base completion tasks with the conventional datasets

Original: FB15K and WN18 2 , and their upgrade version FB15k-237 and WN18RR 3 .
Revised: FB15K and WN18 2 , and their upgrade version FB15k-237 and WN18RR 3 .

Original: Secondly, we evaluate our model on graph based multi-label classiﬁcation tasks with two benchmark datasets from the complex network research area: BlogCatalog and Protein-Protein Interaction (PPI) 4 .
Revised: Secondly, we evaluate our model on graph based multi-label classiﬁcation tasks with two benchmark datasets from the complex network research area: BlogCatalog and Protein-Protein Interaction (PPI) 4 .

Original: Background information of the datasets and the implementation details of our model are given in Appendix B.
Revised: Background information of the datasets and the implementation details of our model are given in Appendix B.

Original: 4. E VALUATION ON K NOWLEDGE B ASE C OMPLETION T ASKS
Revised: 4. E VALUATION ON K NOWLEDGE B ASE C OMPLETION T ASKS

Original: The aim of the ﬁrst evaluation was to assess the performance of the proposed model in link prediction tasks, by comparing it with other state-of-the-art approaches.
Revised: The aim of the ﬁrst evaluation was to assess the performance of the proposed model in link prediction tasks, by comparing it with other state-of-the-art approaches.

Original: We report the ﬁltered P @
Revised: We report the ﬁltered P @

Original: N scores following the protocols proposed by Bordes et al.
Revised: N scores following the protocols proposed by Bordes et al.

Original: (2013), which means that all of the known facts would be screened out from the ranking list before calculating the statistics of the hits .
Revised: (2013), which means that all of the known facts would be screened out from the ranking list before calculating the statistics of the hits .

Original: The numerical results are presented in Table 1, where the highest scores in each column are presented in bold.
Revised: The numerical results are presented in Table 1, where the highest scores in each column are presented in bold.

Original: We reproduced all of the results of the existing studies (mostly with the released code), whereas some of which are below the reported record.
Revised: We reproduced all of the results of the existing studies (mostly with the released code), whereas some of which are below the reported record.

Original: For a fair comparison of the models, we cite those numbers from the original publications (marked with ⋆ symbols).
Revised: For a fair comparison of the models, we cite those numbers from the original publications (marked with ⋆ symbols).

Original: Also, it seems that results reported by Dettmers et al.
Revised: Also, it seems that results reported by Dettmers et al.

Original: (2017) only consider the tail entity prediction scenario (without averaging with the head entity prediction results), hence we report two version of the test results of our model, the averaged version is named as GEN(avg.), while the tail entity prediction results are reported with
Revised: (2017) only consider the tail entity prediction scenario (without averaging with the head entity prediction results), hence we report two version of the test results of our model, the averaged version is named as GEN(avg.), while the tail entity prediction results are reported with

Original: [Table]
Revised: [Table]

Original: model named GEN(tail).
Revised: model named GEN(tail).

Original: Besides, we found that our model tends to remember the reverse facts with regard to the triples that has been processed during the training phase.
Revised: Besides, we found that our model tends to remember the reverse facts with regard to the triples that has been processed during the training phase.

Original: We argue that this is an inherent characteristic of our modeling methodology, since it would treat such reverse facts as conceptually correct .
Revised: We argue that this is an inherent characteristic of our modeling methodology, since it would treat such reverse facts as conceptually correct .

Original: Therefore, we also report P@N scores after screening out such reverse facts, this model is named as GEN(opt).
Revised: Therefore, we also report P@N scores after screening out such reverse facts, this model is named as GEN(opt).

Original: We consider that under certain practical circumstances, it is reasonable to care about such results, because the reverse facts are direct reﬂections of the known facts
Revised: We consider that under certain practical circumstances, it is reasonable to care about such results, because the reverse facts are direct reﬂections of the known facts

Original: [Table]
Revised: [Table]

Original: and in many scenarios, they themselves are useful and effective facts.
Revised: and in many scenarios, they themselves are useful and effective facts.

Original: From Table 1 one could see that the performance of ComplEX seems much more competitive than other models on both of the WordNet subset, however, according to our tests, TransE and HoIE perform (generalized) more stable than others for all of the subtasks.
Revised: From Table 1 one could see that the performance of ComplEX seems much more competitive than other models on both of the WordNet subset, however, according to our tests, TransE and HoIE perform (generalized) more stable than others for all of the subtasks.

Original: Also please note that, after ﬁltering out the reverse facts from the ranking list, we recorded a signiﬁcant increase in P@1 score on WN18, which was not observed in other models.
Revised: Also please note that, after ﬁltering out the reverse facts from the ranking list, we recorded a signiﬁcant increase in P@1 score on WN18, which was not observed in other models.

Original: Since most of the semantic relations deﬁned in WordNet are reﬂexive (Miller, 1995), we believe that these results help verify the efﬁcacy of our model framework.
Revised: Since most of the semantic relations deﬁned in WordNet are reﬂexive (Miller, 1995), we believe that these results help verify the efﬁcacy of our model framework.

Original: Further evidence can be found by looking at evaluation results on FB15K and
Revised: Further evidence can be found by looking at evaluation results on FB15K and

Original: FB15K-237, in which our model consistently and signiﬁcantly outperforms others for all settings.
Revised: FB15K-237, in which our model consistently and signiﬁcantly outperforms others for all settings.

Original: The goal of the second evaluation was three-folded.
Revised: The goal of the second evaluation was three-folded.

Original: (1) To assess the relation prediction performance of our model.
Revised: (1) To assess the relation prediction performance of our model.

Original: (2) To verify the validity of the multi-shot learning framework.
Revised: (2) To verify the validity of the multi-shot learning framework.

Original: (3) To evaluate the quality (representability) of different embedding schemes.
Revised: (3) To evaluate the quality (representability) of different embedding schemes.

Original: To achieve this goal, we carried out a group of experiments depicted in Table 2, where the model name shown in the parentheses indicate that the test is based on the embeddings generated by that model, but being re-trained with our model for fair comparison.
Revised: To achieve this goal, we carried out a group of experiments depicted in Table 2, where the model name shown in the parentheses indicate that the test is based on the embeddings generated by that model, but being re-trained with our model for fair comparison.

Original: For example, before testing the GEN(TransE) model, we need to train a GEN model with TransE embeddings, the only difference is that the pre-trained embeddings will not be updated during the training process, such that the quality of the different embedding schemes can be assessed more objectively.
Revised: For example, before testing the GEN(TransE) model, we need to train a GEN model with TransE embeddings, the only difference is that the pre-trained embeddings will not be updated during the training process, such that the quality of the different embedding schemes can be assessed more objectively.

Original: The results of GEN(HoIE) were obtained similarly from the pre-trained
Revised: The results of GEN(HoIE) were obtained similarly from the pre-trained

Original: HoIE embeddings.
Revised: HoIE embeddings.

Original: The pre-trained word2vec embedding 5 and GloVe embedding 6 are obtained from the publicly available dictionaries released respectively by Google and Stanford NLP Group for research purpose, which are also heavily studied by recent researches.
Revised: The pre-trained word2vec embedding 5 and GloVe embedding 6 are obtained from the publicly available dictionaries released respectively by Google and Stanford NLP Group for research purpose, which are also heavily studied by recent researches.

Original: For entities and relations consisting of many words, we use the weighted sum of the word embeddings as their distributed representation for the test.
Revised: For entities and relations consisting of many words, we use the weighted sum of the word embeddings as their distributed representation for the test.

Original: The three models listed in the bottom of Table 2 demonstrate the oneshot learning capability of GEN, for instance, the results of GEN( h; r ) t ) were obtained by only considering the query ( h; r; ?)
Revised: The three models listed in the bottom of Table 2 demonstrate the oneshot learning capability of GEN, for instance, the results of GEN( h; r ) t ) were obtained by only considering the query ( h; r; ?)

Original: during the training stage.
Revised: during the training stage.

Original: From the studies, the following conclusions could be obtained.
Revised: From the studies, the following conclusions could be obtained.

Original: (1) The performance of GEN on relation prediction tasks has been demonstrated.
Revised: (1) The performance of GEN on relation prediction tasks has been demonstrated.

Original: However, it seems that such strong performance mainly comes from our GEN framework, under which the predictive capability of a variety of em-
Revised: However, it seems that such strong performance mainly comes from our GEN framework, under which the predictive capability of a variety of em-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: beddings can be enhanced.
Revised: beddings can be enhanced.

Original: In considering the ratio of the number of facts to relations involved, this problem seems much easier than the link prediction problem.
Revised: In considering the ratio of the number of facts to relations involved, this problem seems much easier than the link prediction problem.

Original: The validity of the multi-shot framework has been veriﬁed, since each of the one-shot GEN model performs signiﬁcantly worse than the multi-shot model for almost all the tests, except that in relation prediction tasks, GEN( h; t ) r ) performs comparable to GEN, this is probably because that it was exclusively trained for that task, which is prone to overﬁt the data.
Revised: The validity of the multi-shot framework has been veriﬁed, since each of the one-shot GEN model performs signiﬁcantly worse than the multi-shot model for almost all the tests, except that in relation prediction tasks, GEN( h; t ) r ) performs comparable to GEN, this is probably because that it was exclusively trained for that task, which is prone to overﬁt the data.

Original: (3) Comparing with their performance on link prediction tasks, we argue that the embeddings generated by GEN are probably more representative and informative than other embedding schemes, which we will provide more empirical (visual) evidence in Appendix C.
Revised: (3) Comparing with their performance on link prediction tasks, we argue that the embeddings generated by GEN are probably more representative and informative than other embedding schemes, which we will provide more empirical (visual) evidence in Appendix C.

Original: 4. E VALUATION ON G RAPH BASED M ULTI - LABEL C LASSIFICATION T ASKS
Revised: 4. E VALUATION ON G RAPH BASED M ULTI - LABEL C LASSIFICATION T ASKS

Original: In previous section, the term “knowledge graph” was used to refer to a multi-relational database, in which the entities were engaged in one or more heterogeneous relations, which means the relations related with a entity may range over different domains.
Revised: In previous section, the term “knowledge graph” was used to refer to a multi-relational database, in which the entities were engaged in one or more heterogeneous relations, which means the relations related with a entity may range over different domains.

Original: In this section, we consider the problem of embedding learning on another type of graph — the homogeneous graphs (networks), in which the entities were engaged in a speciﬁc relationship, which is a natural structure people use to model the physical world, such as the various social network and the biological information systems.
Revised: In this section, we consider the problem of embedding learning on another type of graph — the homogeneous graphs (networks), in which the entities were engaged in a speciﬁc relationship, which is a natural structure people use to model the physical world, such as the various social network and the biological information systems.

Original: In this study, we consider it as a generalized form of the knowledge graphs, and attempt to come up with a general-purpose framework that could be used for embedding learning on different graphs.
Revised: In this study, we consider it as a generalized form of the knowledge graphs, and attempt to come up with a general-purpose framework that could be used for embedding learning on different graphs.

Original: To verify the validity of the proposed model, we evaluate GEN by comparing its performance on some benchmark multi-label classiﬁcation tasks with the state-of-the-art DeepWalk and Node2vec models.
Revised: To verify the validity of the proposed model, we evaluate GEN by comparing its performance on some benchmark multi-label classiﬁcation tasks with the state-of-the-art DeepWalk and Node2vec models.

Original: Besides, we also report results on TransE and HoIE embeddings for comparison purpose, the supervised model used for multi-label classiﬁcation are identical to each other (but differ from the embeddings).
Revised: Besides, we also report results on TransE and HoIE embeddings for comparison purpose, the supervised model used for multi-label classiﬁcation are identical to each other (but differ from the embeddings).

Original: For fair comparison, all of the results with regard to the DeepWalk (Perozzi et al., 2014) and Node2vec (Grover & Leskovec, 2016) are cited from their original sources.
Revised: For fair comparison, all of the results with regard to the DeepWalk (Perozzi et al., 2014) and Node2vec (Grover & Leskovec, 2016) are cited from their original sources.

Original: Following the convention of previous authors, we randomly sample a portion of the labeled nodes as training set (and the rest are used for test), we repeat this process 9 times (with the training ratio increased from 10% to 90%), and report two of the averaged measures (w.r.t.
Revised: Following the convention of previous authors, we randomly sample a portion of the labeled nodes as training set (and the rest are used for test), we repeat this process 9 times (with the training ratio increased from 10% to 90%), and report two of the averaged measures (w.r.t.

Original: recall, precision, and
Revised: recall, precision, and

Original: F1-measure) on each of the test, namely, macro-average and micro-average.
Revised: F1-measure) on each of the test, namely, macro-average and micro-average.

Original: The Macro-F1 weights equally all the categories regardless of how many labels belong to it, while the Micro-F1 weights equally all the labels , thus favouring the performance on common categories .
Revised: The Macro-F1 weights equally all the categories regardless of how many labels belong to it, while the Micro-F1 weights equally all the labels , thus favouring the performance on common categories .

Original: Numerical results are presented in Table 3 and 4 respectively, the highest scores in each column are presented in bold face.
Revised: Numerical results are presented in Table 3 and 4 respectively, the highest scores in each column are presented in bold face.

Original: From Table 3 one could see that the performance of DeepWalk proves much more competitive than other models when labeled data is sparse, but GEN still consistently outperforms when given 50% of the data, which demonstrates the validity of the proposed embedding learning framework for modeling author connections on social networks.
Revised: From Table 3 one could see that the performance of DeepWalk proves much more competitive than other models when labeled data is sparse, but GEN still consistently outperforms when given 50% of the data, which demonstrates the validity of the proposed embedding learning framework for modeling author connections on social networks.

Original: Next, we investigate the performance of our model on even more sparse graphs, i.e. the Protein-Protein Interactions network.
Revised: Next, we investigate the performance of our model on even more sparse graphs, i.e. the Protein-Protein Interactions network.

Original: Table 4 shows that GEN performs consistently and signiﬁcantly better than other baselines.
Revised: Table 4 shows that GEN performs consistently and signiﬁcantly better than other baselines.

Original: In fact, when trained with only 20% of the labeled proteins, GEN performs signiﬁcantly better than other approaches when they are given 90% of the data.
Revised: In fact, when trained with only 20% of the labeled proteins, GEN performs signiﬁcantly better than other approaches when they are given 90% of the data.

Original: We argue that this strong performance not only indicates that our model is ﬂexible enough to the biological networks, but also provides new insights into their underlying biological mechanisms.
Revised: We argue that this strong performance not only indicates that our model is ﬂexible enough to the biological networks, but also provides new insights into their underlying biological mechanisms.

Original: Also please note that Macro-F1 scores in Table
Revised: Also please note that Macro-F1 scores in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: -F
Revised: -F

Original: DeepWalk 15.
Revised: DeepWalk 15.

Original: Node2vec 16.
Revised: Node2vec 16.

Original: TransE 12.
Revised: TransE 12.

Original: HoIE 14.
Revised: HoIE 14.

Original: GEN 16.
Revised: GEN 16.

Original: Macro-F
Revised: Macro-F

Original: DeepWalk 12.
Revised: DeepWalk 12.

Original: Node2vec 13.
Revised: Node2vec 13.

Original: TransE 8.
Revised: TransE 8.

Original: HoIE 9.
Revised: HoIE 9.

Original: GEN 14.
Revised: GEN 14.

Original: and 4 demonstrate that, comparing with other embedding schemes, GEN performs more stable (and better) in both common and rare categories, which indicates that the embeddings generated by GEN are probably more representative and informative than other solutions, thus the supervised model built on top of it is less vulnerable to global under-ﬁtting and local over-ﬁtting.
Revised: and 4 demonstrate that, comparing with other embedding schemes, GEN performs more stable (and better) in both common and rare categories, which indicates that the embeddings generated by GEN are probably more representative and informative than other solutions, thus the supervised model built on top of it is less vulnerable to global under-ﬁtting and local over-ﬁtting.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: Representation learning of knowledge graphs is a key concern for artiﬁcial intelligence and cognitive science.
Revised: Representation learning of knowledge graphs is a key concern for artiﬁcial intelligence and cognitive science.

Original: Many types of relations in physical, biological, social and information systems can be modeled with concept (knowledge) graphs.
Revised: Many types of relations in physical, biological, social and information systems can be modeled with concept (knowledge) graphs.

Original: In this paper, we present an efﬁcient scalable framework for learning conceptual embeddings of entities and relations in generalized knowledge graphs, including the homogeneous and heterogeneous graphs.
Revised: In this paper, we present an efﬁcient scalable framework for learning conceptual embeddings of entities and relations in generalized knowledge graphs, including the homogeneous and heterogeneous graphs.

Original: We give evidence that the proposed model learns good representations of all these graphs for knowledge inference and supervised learning.
Revised: We give evidence that the proposed model learns good representations of all these graphs for knowledge inference and supervised learning.

Original: For future work, we plan to investigate more thoroughly the efﬁcacy of the proposed modeling framework, with respect to the decomposition of the semantic information conveyed by the linked concepts into elementary information, i.e. the four Q&A pairs.
Revised: For future work, we plan to investigate more thoroughly the efﬁcacy of the proposed modeling framework, with respect to the decomposition of the semantic information conveyed by the linked concepts into elementary information, i.e. the four Q&A pairs.

Original: Also, we seek to enhance the quality of scientiﬁc investigations and theoretical conceptualizations on graph embedding learning in the context of semantic interoperability , for there is usually no possibility to interpret the embedded information meaningfully and accurately in order to produce useful results as deﬁned by existing algorithms.
Revised: Also, we seek to enhance the quality of scientiﬁc investigations and theoretical conceptualizations on graph embedding learning in the context of semantic interoperability , for there is usually no possibility to interpret the embedded information meaningfully and accurately in order to produce useful results as deﬁned by existing algorithms.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We are grateful to the anonymous reviewers for taking time read and provide helpful comments.
Revised: We are grateful to the anonymous reviewers for taking time read and provide helpful comments.

Original: A PPENDIX A: M OTIVATION AND I NTUITION
Revised: A PPENDIX A: M OTIVATION AND I NTUITION

Original: To get an intuitive understanding of the problem, consider the following examples taken from three typical KGs that have been heavily studied by the academic and industrial communities:
Revised: To get an intuitive understanding of the problem, consider the following examples taken from three typical KGs that have been heavily studied by the academic and industrial communities:

Original: (cid:15) ( Elvis Presley, instance of, rock star ) : taken from the WordNet 7 , one of the largest online lexical database of English, in which each distinct concept (called synset ) are interlinked by means of rigidly deﬁned (hence limited) conceptual-semantic or lexical relations.
Revised: (cid:15) ( Elvis Presley, instance of, rock star ) : taken from the WordNet 7 , one of the largest online lexical database of English, in which each distinct concept (called synset ) are interlinked by means of rigidly deﬁned (hence limited) conceptual-semantic or lexical relations.

Original: (cid:15) ( Elvis Presley, /people/person/profession, singer ) : taken from the Freebase 8 , which was once to be the largest collaboratively edited knowledge base (deprecated at this time and absorbed by the Wikidata project).
Revised: (cid:15) ( Elvis Presley, /people/person/profession, singer ) : taken from the Freebase 8 , which was once to be the largest collaboratively edited knowledge base (deprecated at this time and absorbed by the Wikidata project).

Original: In which each named entities are interlinked by means of ﬁne-grained relation types deﬁned in the meta-schema.
Revised: In which each named entities are interlinked by means of ﬁne-grained relation types deﬁned in the meta-schema.

Original: Due to the loosely-deﬁned nature of the relation types, redundancy or alternate facts are allowed to exist simultaneously, such as, ( Elvis Presley, profession, musician ) and ( Elvis Presley, profession, actor ).
Revised: Due to the loosely-deﬁned nature of the relation types, redundancy or alternate facts are allowed to exist simultaneously, such as, ( Elvis Presley, profession, musician ) and ( Elvis Presley, profession, actor ).

Original: (cid:15) ( Elvis Presley, rdf:type, American rock singers ) : taken from the YAGO 9 , one of the largest and most active semantic knowledge base developed at the Max Planck Institute for Computer Science in Saarbr¨ucken , which combines the clean taxonomy (relation types) of WordNet with the richness of the Wikipedia category system (classes of entities ).
Revised: (cid:15) ( Elvis Presley, rdf:type, American rock singers ) : taken from the YAGO 9 , one of the largest and most active semantic knowledge base developed at the Max Planck Institute for Computer Science in Saarbr¨ucken , which combines the clean taxonomy (relation types) of WordNet with the richness of the Wikipedia category system (classes of entities ).

Original: As can be perceived from above examples, the use of different ontologies can lead to different (and incoherent) relations between the same pair of concepts, similarly, applying different ontologies can lead to diverse kinds of conceptualizations.
Revised: As can be perceived from above examples, the use of different ontologies can lead to different (and incoherent) relations between the same pair of concepts, similarly, applying different ontologies can lead to diverse kinds of conceptualizations.

Original: Therefore, it is (arguably) impractical to rely on using the word-level embeddings to precisely represent the knowledge graphs under the diverse conditions, and it is necessary to develop a universal solution that is applicable to all of the ontology infrastructures, for phrase-level embedding learning of the different concept representations.
Revised: Therefore, it is (arguably) impractical to rely on using the word-level embeddings to precisely represent the knowledge graphs under the diverse conditions, and it is necessary to develop a universal solution that is applicable to all of the ontology infrastructures, for phrase-level embedding learning of the different concept representations.

Original: As mentioned in Section 3, in order to develop a representation learning method that is ﬂexible enough for modeling different types of knowledge graphs, the most important problems to be faced are associated with how to deﬁne the optimization problem and how to solve it.
Revised: As mentioned in Section 3, in order to develop a representation learning method that is ﬂexible enough for modeling different types of knowledge graphs, the most important problems to be faced are associated with how to deﬁne the optimization problem and how to solve it.

Original: According to our survey, most state-of-the-art models, including the translating models derived from the TransE (Bordes et al., 2013; Lin et al., 2015), the latent semantic models derived from the RESCAL (Nickel et al., 2011; 2016b), and the neural network models derived from the
Revised: According to our survey, most state-of-the-art models, including the translating models derived from the TransE (Bordes et al., 2013; Lin et al., 2015), the latent semantic models derived from the RESCAL (Nickel et al., 2011; 2016b), and the neural network models derived from the

Original: NTN (Socher et al., 2013), were all trying to deﬁne the graph embedding learning problem as a supervised binary classiﬁcation problem, in which the optimization objectives are deﬁned in the form of a relation-speciﬁc cost function of the entity and/or relation embeddings, and then to solve it with a stochastic gradient decent (SGD) process.
Revised: NTN (Socher et al., 2013), were all trying to deﬁne the graph embedding learning problem as a supervised binary classiﬁcation problem, in which the optimization objectives are deﬁned in the form of a relation-speciﬁc cost function of the entity and/or relation embeddings, and then to solve it with a stochastic gradient decent (SGD) process.

Original: Typical criteria used to evaluate the cost functions include the logistic loss and the pairwise margin-based criterion , and the negative samples used for training the model are usually sampled from the complement of the knowledge graph based on the open world assumption (Drumond et al., 2012).
Revised: Typical criteria used to evaluate the cost functions include the logistic loss and the pairwise margin-based criterion , and the negative samples used for training the model are usually sampled from the complement of the knowledge graph based on the open world assumption (Drumond et al., 2012).

Original: However, we doubt that there are many situations where such modeling strategies would have theoretical and practical disadvantages.
Revised: However, we doubt that there are many situations where such modeling strategies would have theoretical and practical disadvantages.

Original: Firstly, we speculate that the reason why most previous studies did not consider the ﬁrst and second queries simultaneously (see Section 3), is probably due to the difﬁcult in modeling the inverse semantic relatedness of the entities from the given fact.
Revised: Firstly, we speculate that the reason why most previous studies did not consider the ﬁrst and second queries simultaneously (see Section 3), is probably due to the difﬁcult in modeling the inverse semantic relatedness of the entities from the given fact.

Original: In other words, shall we use the embedding of r to represent its reverse r ′ ?
Revised: In other words, shall we use the embedding of r to represent its reverse r ′ ?

Original: If we do so, it seems that it will inevitably lead to semantic paradox like:
Revised: If we do so, it seems that it will inevitably lead to semantic paradox like:

Original: Presley ’s profession is Presley , since from the model’s perspective, there is no difference between the entity Presley and other entities that may appear on both side of the relation profession .
Revised: Presley ’s profession is Presley , since from the model’s perspective, there is no difference between the entity Presley and other entities that may appear on both side of the relation profession .

Original: Considering the sparsity of the knowledge graph, models trained with limited facts would very likely tend to give higher scores to the entities that have been “seen in the right place”.
Revised: Considering the sparsity of the knowledge graph, models trained with limited facts would very likely tend to give higher scores to the entities that have been “seen in the right place”.

Original: In order to solve this problem, we propose to model the facts conceptually instead of concretely (or literally, syntactically), which means that we will focus on the semantic meanings of the embeddings (of the entities and relations), rather than their syntactic features.
Revised: In order to solve this problem, we propose to model the facts conceptually instead of concretely (or literally, syntactically), which means that we will focus on the semantic meanings of the embeddings (of the entities and relations), rather than their syntactic features.

Original: Such a conceptual embedding scheme allow us to unify the representation of a relation (r) and its reverse counterpart (r’), and to accommodate the lexical variety in use by various knowledge bases.
Revised: Such a conceptual embedding scheme allow us to unify the representation of a relation (r) and its reverse counterpart (r’), and to accommodate the lexical variety in use by various knowledge bases.

Original: [Table]
Revised: [Table]

Original: The intuition behind is, for any given fact ( h; r; t ) , one would instantly recognize the bidirectional semantic connection between h and t , without need of translating it into ( t; r ′ ; h ) explicitly in his/her mind.
Revised: The intuition behind is, for any given fact ( h; r; t ) , one would instantly recognize the bidirectional semantic connection between h and t , without need of translating it into ( t; r ′ ; h ) explicitly in his/her mind.

Original: We believe this is crucial for efﬁcient utilization of the structure information of the KGs for representation learning, empirical evidence is provided in Section 4 and Appendix C, respectively.
Revised: We believe this is crucial for efﬁcient utilization of the structure information of the KGs for representation learning, empirical evidence is provided in Section 4 and Appendix C, respectively.

Original: Secondly, we propose to use unsupervised learning techniques for graph embedding learning tasks, because: (1) Presently, almost all of the large-scale knowledge graphs are extremely sparse, which would unavoidably degrade the quality and reliability of the supervised learning algorithms.
Revised: Secondly, we propose to use unsupervised learning techniques for graph embedding learning tasks, because: (1) Presently, almost all of the large-scale knowledge graphs are extremely sparse, which would unavoidably degrade the quality and reliability of the supervised learning algorithms.

Original: Further, considering the relation-speciﬁc solution that dominates the current research, the situation might get even worse.
Revised: Further, considering the relation-speciﬁc solution that dominates the current research, the situation might get even worse.

Original: (2) Selecting negative examples for pair-wise training would be tricky and expensive, since in practice, it is very hard to generate a “proper and informative” negative sample responsive to each of the positive examples.
Revised: (2) Selecting negative examples for pair-wise training would be tricky and expensive, since in practice, it is very hard to generate a “proper and informative” negative sample responsive to each of the positive examples.

Original: For example, when learning from the fact ( Einstein, employer, IAS ), the false fact ( Einstein, employer, Stanford ) would seem to be more reasonable and informative than ( Einstein, employer, singer ) — if the objective is to further improve the predictive capability of the model to discriminate between similar objects.
Revised: For example, when learning from the fact ( Einstein, employer, IAS ), the false fact ( Einstein, employer, Stanford ) would seem to be more reasonable and informative than ( Einstein, employer, singer ) — if the objective is to further improve the predictive capability of the model to discriminate between similar objects.

Original: To solve the data sparsity problem, we propose to model each of the facts as a short sentence, the entire KG can be regarded as a huge document, so that it can be processed by unsupervised encoderdecoder neural models, which has demonstrate to be efﬁcient and useful in concept learning from the large-scale and feature sparse data (Sutskever et al., 2014).
Revised: To solve the data sparsity problem, we propose to model each of the facts as a short sentence, the entire KG can be regarded as a huge document, so that it can be processed by unsupervised encoderdecoder neural models, which has demonstrate to be efﬁcient and useful in concept learning from the large-scale and feature sparse data (Sutskever et al., 2014).

Original: In order to avoid the sampling bias due to the selection of uninformative entities, we propose to use the softmax cross-entropy loss as a measure of the predictive discrepancy for model training, because its probability interpretation is more objective than those squared or logistic errors conventionally used in this area, and, it has been proven to be convex for the MLP we used in this paper (Bengio et al., 2005).
Revised: In order to avoid the sampling bias due to the selection of uninformative entities, we propose to use the softmax cross-entropy loss as a measure of the predictive discrepancy for model training, because its probability interpretation is more objective than those squared or logistic errors conventionally used in this area, and, it has been proven to be convex for the MLP we used in this paper (Bengio et al., 2005).

Original: A PPENDIX B: B ACKGROUND INFORMATION AND IMPLEMENTATION DETAILS B.1 D ATASETS
Revised: A PPENDIX B: B ACKGROUND INFORMATION AND IMPLEMENTATION DETAILS B.1 D ATASETS

Original: WN18 is a subset of WordNet, which contains mostly the conceptual-semantic and lexical relations, and the entities are organized in a strictly hierarchical manner.
Revised: WN18 is a subset of WordNet, which contains mostly the conceptual-semantic and lexical relations, and the entities are organized in a strictly hierarchical manner.

Original: FB15k is a subset of Freebase, which contains facts gathered from Wikipedia, mostly focused on the topic of movies and sports.
Revised: FB15k is a subset of Freebase, which contains facts gathered from Wikipedia, mostly focused on the topic of movies and sports.

Original: These datasets have been used as a de facto benchmarks for comparative evaluation, however, recent research (Toutanova & Chen, 2015; Dettmers et al., 2017) show that the test sets of WN18 and
Revised: These datasets have been used as a de facto benchmarks for comparative evaluation, however, recent research (Toutanova & Chen, 2015; Dettmers et al., 2017) show that the test sets of WN18 and

Original: FB15k contain a lot of reversed triples that have been presented in the training set, i.e., ( h; r; t ) versus ( t; r; h ) .
Revised: FB15k contain a lot of reversed triples that have been presented in the training set, i.e., ( h; r; t ) versus ( t; r; h ) .

Original: Which, we consider would favor our model over those one-shot alternatives.
Revised: Which, we consider would favor our model over those one-shot alternatives.

Original: Therefore, we provide results on FB15k-237, which is introduced by Toutanova & Chen (2015), it is a subset of FB15K where reversing relations are removed.
Revised: Therefore, we provide results on FB15k-237, which is introduced by Toutanova & Chen (2015), it is a subset of FB15K where reversing relations are removed.

Original: And, we also test on WN18RR provided by Dettmers et al.
Revised: And, we also test on WN18RR provided by Dettmers et al.

Original: (2017), which is a reverse duplication free new sample of WordNet.
Revised: (2017), which is a reverse duplication free new sample of WordNet.

Original: The multi-relational data sampled from WordNet and Freebase can be seen as typical of the heterogeneous graphs, in order to verify the generality of the developed model, we also perform evaluation in the multi-label classiﬁcation setting on some typical homogeneous graphs.
Revised: The multi-relational data sampled from WordNet and Freebase can be seen as typical of the heterogeneous graphs, in order to verify the generality of the developed model, we also perform evaluation in the multi-label classiﬁcation setting on some typical homogeneous graphs.

Original: BlogCatalog is a social network sampled from the BlogCatalog website, which contains only one relationship: the social connection between the blog authors, while the labels represent the interested topic categories provided by the bloggers.
Revised: BlogCatalog is a social network sampled from the BlogCatalog website, which contains only one relationship: the social connection between the blog authors, while the labels represent the interested topic categories provided by the bloggers.

Original: Protein-Protein Interactions is a biological network sampled from the PPI network for Homo Sapiens, which also contains only one relationship: the existence of interactions between the proteins, while the labels represent the biological states of the proteins.
Revised: Protein-Protein Interactions is a biological network sampled from the PPI network for Homo Sapiens, which also contains only one relationship: the existence of interactions between the proteins, while the labels represent the biological states of the proteins.

Original: In the training set of these graph corpus, every entity (node) is assigned one or more labels from a ﬁnite set, the task is to predict the labels for the nodes in the test set.
Revised: In the training set of these graph corpus, every entity (node) is assigned one or more labels from a ﬁnite set, the task is to predict the labels for the nodes in the test set.

Original: The statistics of these data sets are summarized in Table 5.
Revised: The statistics of these data sets are summarized in Table 5.

Original: B.2 E XPERIMENTAL S ETUP
Revised: B.2 E XPERIMENTAL S ETUP

Original: We optimized the hyper-parameters of all the datasets via extensive grid search and selected the model with the best ﬁltered P@10 score on the validation set.
Revised: We optimized the hyper-parameters of all the datasets via extensive grid search and selected the model with the best ﬁltered P@10 score on the validation set.

Original: Hyper-parameter ranges for the grid search were the following: embedding dimension d in f 50 ; 100 ; 200 ; 300 g , hidden layer dimension k in f 256 ; 512 ; 1024 ; 2048 g , MLP dropout rate p in f 0 : 0 ; 0 : 1 ; 0 : 2 ; 0 : 3 g , learning rate (cid:17) in f 0 : 001 ; 0 : 01 ; 0 : 1 ; 1 ; 5 ; 10 g , learning rate decay (cid:21) in f 0 : 7 ; 0 : 75 ; 0 : 8 ; 0 : 85 ; 0 : 9 ; 0 : 95 g .
Revised: Hyper-parameter ranges for the grid search were the following: embedding dimension d in f 50 ; 100 ; 200 ; 300 g , hidden layer dimension k in f 256 ; 512 ; 1024 ; 2048 g , MLP dropout rate p in f 0 : 0 ; 0 : 1 ; 0 : 2 ; 0 : 3 g , learning rate (cid:17) in f 0 : 001 ; 0 : 01 ; 0 : 1 ; 1 ; 5 ; 10 g , learning rate decay (cid:21) in f 0 : 7 ; 0 : 75 ; 0 : 8 ; 0 : 85 ; 0 : 9 ; 0 : 95 g .

Original: In this study, we use the following combination of parameters for all of the graph embedding learning tasks :
Revised: In this study, we use the following combination of parameters for all of the graph embedding learning tasks :

Original: (cid:15) E CELLS: f d : 200 ; k : 2048 ; p : 0 : 2 ; (cid:17) :
Revised: (cid:15) E CELLS: f d : 200 ; k : 2048 ; p : 0 : 2 ; (cid:17) :

Original: ; (cid:21)
Revised: ; (cid:21)

Original: (cid:15) R CELLS:
Revised: (cid:15) R CELLS:

Original: f d : 200 ; k : 512 ; p : 0 : 2 ; (cid:17) : 5 ; (cid:21) : 0 : 9 g .
Revised: f d : 200 ; k : 512 ; p : 0 : 2 ; (cid:17) : 5 ; (cid:21) : 0 : 9 g .

Original: (cid:15) Mini-batch Settings: f batch size : 512 ; epoch : 50 g
Revised: (cid:15) Mini-batch Settings: f batch size : 512 ; epoch : 50 g

Original: For multi-label classiﬁcation tasks, we implement a single layer perceptron model for multi-task learning with: f k : 128 ; (cid:17) : 0 : 1 ; (cid:21) : 0 : 9 g , which is selected through grid search with the best averaged Macro-F1 score on randomly sampled validation set from the labeled nodes.
Revised: For multi-label classiﬁcation tasks, we implement a single layer perceptron model for multi-task learning with: f k : 128 ; (cid:17) : 0 : 1 ; (cid:21) : 0 : 9 g , which is selected through grid search with the best averaged Macro-F1 score on randomly sampled validation set from the labeled nodes.

Original: A PPENDIX C: I NVESTIGATING AND V ISUALIZING THE E MBEDDING S CHEMES
Revised: A PPENDIX C: I NVESTIGATING AND V ISUALIZING THE E MBEDDING S CHEMES

Original: In this section, we provide qualitative analysis on four typical embedding schemes (GEN, HoIE,
Revised: In this section, we provide qualitative analysis on four typical embedding schemes (GEN, HoIE,

Original: TransE and word2vec), with the intention of better understanding the connection between the existing graph embedding schemes, and highlighting areas that remain poorly understood for further investigation.
Revised: TransE and word2vec), with the intention of better understanding the connection between the existing graph embedding schemes, and highlighting areas that remain poorly understood for further investigation.

Original: The reason we choose these models (except the word2vec) is because, according to our tests, they have demonstrated to be efﬁcient and scalable to large-scale problems, and are also exhibiting good generalization ability on real data sets.
Revised: The reason we choose these models (except the word2vec) is because, according to our tests, they have demonstrated to be efﬁcient and scalable to large-scale problems, and are also exhibiting good generalization ability on real data sets.

Original: We also consider the word2vec embeddings because we found that with the help of our multi-shot learning model, it achieves state-of-the-art performance on most of the knowledge base completion tasks (see Section 4), which is interesting, and worth some consideration (probably indicates a promising potential for transfer learning).
Revised: We also consider the word2vec embeddings because we found that with the help of our multi-shot learning model, it achieves state-of-the-art performance on most of the knowledge base completion tasks (see Section 4), which is interesting, and worth some consideration (probably indicates a promising potential for transfer learning).

Original: The argument in the ﬁrst case is to claim that the graph embeddings generated by GEN are different from other solutions.
Revised: The argument in the ﬁrst case is to claim that the graph embeddings generated by GEN are different from other solutions.

Original: We calculate the cosine similarities for each pair of the 1,345 relations in
Revised: We calculate the cosine similarities for each pair of the 1,345 relations in

Original: FB15K w.r.t.
Revised: FB15K w.r.t.

Original: four embedding schemes respectively, and compare their top-N ranking list (which is a set of relation pairs for each embedding scheme) through Venn diagrams, as illustrated in Fig.3.
Revised: four embedding schemes respectively, and compare their top-N ranking list (which is a set of relation pairs for each embedding scheme) through Venn diagrams, as illustrated in Fig.3.

Original: Figure 3 reveals very different clustering pattern between GEN and other alternatives in their corresponding embedding space.
Revised: Figure 3 reveals very different clustering pattern between GEN and other alternatives in their corresponding embedding space.

Original: What’s particularly interesting is that, although the TranE model is inspired by and heavily reliant on the spatial distribution of the wrod2vec embeddings (Bordes et al., 2013), but they are, in fact, not similar at all.
Revised: What’s particularly interesting is that, although the TranE model is inspired by and heavily reliant on the spatial distribution of the wrod2vec embeddings (Bordes et al., 2013), but they are, in fact, not similar at all.

Original: On the contrary, the results of TransE and HoIE share a lot of similarities, it appears that in their top-300 and top-500 lists, almost half of the relation pairs were contained in their intersection.
Revised: On the contrary, the results of TransE and HoIE share a lot of similarities, it appears that in their top-300 and top-500 lists, almost half of the relation pairs were contained in their intersection.

Original: Which probably indicates that the translating embedding hypothesis (Bordes et al., 2013) is theoretically similar in nature to the holographic embedding hypothesis (Nickel et al., 2016b), when used for graph modeling.
Revised: Which probably indicates that the translating embedding hypothesis (Bordes et al., 2013) is theoretically similar in nature to the holographic embedding hypothesis (Nickel et al., 2016b), when used for graph modeling.

Original: This is not an easily testable hypothesis, we consider it to be an open question, which we hope to further explore in the future.
Revised: This is not an easily testable hypothesis, we consider it to be an open question, which we hope to further explore in the future.

Original: The goal of the second experiment is to verify the claim that the embeddings generated by GEN are more representative and informative than other embedding schemes.
Revised: The goal of the second experiment is to verify the claim that the embeddings generated by GEN are more representative and informative than other embedding schemes.

Original: Here we provide a case study on a randomly selected relation from FB15K, namely “/location/location/time zones”.
Revised: Here we provide a case study on a randomly selected relation from FB15K, namely “/location/location/time zones”.

Original: There are 137 triples related to this relation (#425) in the test set, all of the head entities are names of the countries or regions, and the tail entities are the corresponding time zones.
Revised: There are 137 triples related to this relation (#425) in the test set, all of the head entities are names of the countries or regions, and the tail entities are the corresponding time zones.

Original: The heads are uniquely different from each other, while there are only 10 different time zones existed in the tails.
Revised: The heads are uniquely different from each other, while there are only 10 different time zones existed in the tails.

Original: We plot all of the 137 triples in Fig.4, in which (Fig.4a and Fig4b)
Revised: We plot all of the 137 triples in Fig.4, in which (Fig.4a and Fig4b)

Original: the input multi-dimensional vectors are projected to a 2-dimensional subspace spanned by x and y , by using of the principal component analysis (PCA), then we choose the ﬁrst two principal components as the principal axes.
Revised: the input multi-dimensional vectors are projected to a 2-dimensional subspace spanned by x and y , by using of the principal component analysis (PCA), then we choose the ﬁrst two principal components as the principal axes.

Original: In Fig.4a, the input is the concatenation of the head and tail entity of each triple, i.e. ( h (cid:8) t ) , with the intention of investigating the patterns of such feature vectors for relation prediction tasks.
Revised: In Fig.4a, the input is the concatenation of the head and tail entity of each triple, i.e. ( h (cid:8) t ) , with the intention of investigating the patterns of such feature vectors for relation prediction tasks.

Original: Hence, we choose the name of the tails as legend labels.
Revised: Hence, we choose the name of the tails as legend labels.

Original: As can be seen from Fig.4a, the feature vectors of the 137 triples shows clear clustering tendencies with regard to the categories in their tail entities.
Revised: As can be seen from Fig.4a, the feature vectors of the 137 triples shows clear clustering tendencies with regard to the categories in their tail entities.

Original: Based on this observation, we further plot the hidden layer of the R CELL (which is a 512-dimensional vector in this case) located before the output layer in our GEN model, as depicted in Fig.4b.
Revised: Based on this observation, we further plot the hidden layer of the R CELL (which is a 512-dimensional vector in this case) located before the output layer in our GEN model, as depicted in Fig.4b.

Original: From Fig.4b one could see that the distance between the data points is ampliﬁed, and the distinction becomes more prominent.
Revised: From Fig.4b one could see that the distance between the data points is ampliﬁed, and the distinction becomes more prominent.

Original: We plot the cumulative softmax in Fig.4c, in which the X-axis represents the 1,345 type of relations in FB15K, Y-axis denotes the cumulative softmax values.
Revised: We plot the cumulative softmax in Fig.4c, in which the X-axis represents the 1,345 type of relations in FB15K, Y-axis denotes the cumulative softmax values.

Original: The curve is obtained by adding all of the softmax vectors output by GEN with regard to the 137 triples.
Revised: The curve is obtained by adding all of the softmax vectors output by GEN with regard to the 137 triples.

Original: Obviously, the only peak observed in Fig.4c clearly exhibit that GEN can make good use of these (concatenated) features to identify the corresponding relations correctly.
Revised: Obviously, the only peak observed in Fig.4c clearly exhibit that GEN can make good use of these (concatenated) features to identify the corresponding relations correctly.

Original: For comparison purpose, we also visualize the other three embedding schemes with the same protocol, as illustrated in Fig.5.
Revised: For comparison purpose, we also visualize the other three embedding schemes with the same protocol, as illustrated in Fig.5.

Original: Since the corresponding models do not use MLP for relation prediction, we can not plot their “hidden state” and “accumulate softmax” for the second and the third subplots, hence we choose to visualize their predictive criterion vectors and output ranking list instead.
Revised: Since the corresponding models do not use MLP for relation prediction, we can not plot their “hidden state” and “accumulate softmax” for the second and the third subplots, hence we choose to visualize their predictive criterion vectors and output ranking list instead.

Revised: The processing practice is consistent with the protocol of the original literature.

Original: Speciﬁcally, for TransE, we plot ( t (cid:0) h ) as the hidden state for relation prediction, and use the softmax of ( t (cid:0) h )for calculating of the accumulate softmax .
Revised: Speciﬁcally, for TransE, we plot ( t (cid:0) h ) as the hidden state for relation prediction, and calculate the ℓ 1 -norm distance j r i (cid:0) ( t (cid:0) h )

Revised: j 1 w.r.t each of the relation r i in FB15K, then we process the distance vector with the softmax function for calculation of the accumulate softmax .

Original: While for HoIE, we plot the circular correlation vector ( h ⋆ t ) as the hidden state , and calculate the cosine similarity of ( h ⋆ t )  w.r.t all of the relations in FB15K. The processing practice is consistent with the protocol of the original literature.
Revised: While for HoIE, we plot the circular correlation vector ( h⋆t ) as the hidden state , and calculate the cosine similarity of ( h⋆t ) (cid:1) r i w.r.t each of the relation r i in FB15K, then we use the obtained (cosine) vector to calculate the accumulate softmax .

Original: For word2vec embeddings, we use the same protocol as dealing with TransE.
Revised: For word2vec embeddings, we use the same protocol as dealing with TransE.

Original: From Fig.5 one could see that, the concatenated embedding vectors of TranE and HoIE shows similar clustering pattern as the GEN case, which help explaining the reason that under our multi-shot learning framework, the embeddings generated by these models perform similar in relation prediction tasks (see Table 2).
Revised: From Fig.5 one could see that, the concatenated embedding vectors of TranE and HoIE shows similar clustering pattern as the GEN case, which help explaining the reason that under our multi-shot learning framework, the embeddings generated by these models perform similar in relation prediction tasks (see Table 2).

Original: It also provides evidence for our conjecture that these two embedding schemes could be inherently similar to each other.
Revised: It also provides evidence for our conjecture that these two embedding schemes could be inherently similar to each other.

Original: Form their criterion vectors (the second subplot for each models), one could see that their clustering pattern is not as clear as the case of GEN, which help explain their performance on relation prediction tasks (as shown in the third subplot) 10 .
Revised: Form their criterion vectors (the second subplot for each models), one could see that their clustering pattern is not as clear as the case of GEN, which help explain their performance on relation prediction tasks (as shown in the third subplot) 10 .

Original: We consider this as solid support for the validity of the proposed multi-shot learning framework.
Revised: We consider this as solid support for the validity of the proposed multi-shot learning framework.

Original: More evidence could be found with our source code release, which will be made publicly available on GitHub to encourage reproducible research, after anonymous review.
Revised: More evidence could be found with our source code release, which will be made publicly available on GitHub to encourage reproducible research, after anonymous review.