N EURAL M ETHODS FOR L OGICAL R EASONING OVER K NOWLEDGE G RAPHS
A BSTRACT
Reasoning is a fundamental problem for computers and deeply studied in Artiﬁcial Intelligence.
In this paper, we speciﬁcally focus on answering multi-hop logical queries on Knowledge Graphs (KGs).
This is a complicated task because, in real-world scenarios, the graphs tend to be large and incomplete.
Most previous works have been unable to create models that accept full First-Order Logical (FOL) queries, which include negative queries, and have only been able to process a limited set of query structures.
Additionally, most methods present logic operators that can only perform the logical operation they are made for.
We introduce a set of models that use Neural Networks to create one-point vector embeddings to answer the queries.
The versatility of neural networks allows the framework to handle FOL queries with Conjunction ( ∧ ), Disjunction ( ∨ ) and Negation ( ¬ ) operators.
We demonstrate experimentally the performance of our model through extensive experimentation on well-known benchmarking datasets.
Besides having more versatile operators, the models achieve a 10% relative increase over the best performing state of the art and more than 30% over the original method based on single-point vector embeddings.
I NTRODUCTION
Knowledge graphs (KGs) are a type of data structure that can capture many kinds of relationships between entities (e g.: Moscow cityIn −−−→ Russia ) and have been popularized since the creation of the semantic web or its introduction into Google’s search engine.
They can contain many kinds of different information, and they can be widely used in question-answering systems, search engines, and recommender systems (Palumbo et al., 2017; Xiong et al., 2017a).
Reasoning is a fundamental skill of human brains.
For example, we can infer new knowledge based on known facts and logic rules, and discern patterns/relationships to make sense of seemingly unrelated information.
It is a multidisciplinary topic and is being studied in psychology, neuroscience, and artiﬁcial intelligence (Fagin et al., 2003).
The ability to reason about the relations between objects is central to generally intelligent behavior.
We can deﬁne reasoning as the process of inferring new knowledge based on known facts and logic rules.
Knowledge graphs are a structure used for storing many kinds of information, therefore the ability to answer complex queries and extract answers that are not directly encoded in the graph are of high interest to the AI community.
To answer complex queries, the model receives a query divided in logical statements.
A full FirstOrder Logic (FOL) is necessary to process a wider range of queries, which includes negative queries.
FOL includes the following logical operators: Existential ( ∃ ), Conjunction ( ∧ ), Disjunction ( ∨ ), and
Negation ( ¬ ).
The power of representation of our logical framework is the key to process complex queries.
However, most frameworks have only been able to process Existential Positive First-Order
Logic (EPFO), which means that negative queries cannot be processed.
For example, One could ask a knowledge graph containing drugs and side effects the following question: “What drug can be used to treat pneumonia and does not cause drowsiness?” .
The ﬁrst step to answer such a query is to translate it into logical statements: q = V ?
· ∃ V : Treat(Pneumonia, V ? )
∧ ¬ Cause(Drowsiness, V ? ) .
Once the query is divided into logical statements, we obtain the computation graph, a directed acyclic graph (DAG) which deﬁnes the order of operations.
Afterwards, we can start traversing the graph.
However, many real-world graphs are incomplete and therefore traversing them becomes very hard and even computationally impossible.
There are many possible answer entities, and it requires modeling sets of entities.
As such, embedding methods become a good solution to answer these queries.
Previous works (Hamilton et al., 2018;
Ren et al., 2020; Ren & Leskovec, 2020) have created methods for embedding the query and the graph into a vector space.
The idea of graph embeddings reduces the problem to simply using nearest-neighbor search to ﬁnd the answers, without paying attention to the intermediate results.
The embedding approach solves many of the problems of query-answering in knowledge graphs.
In theory, we could answer the queries just by traversing the graph.
In practice, graphs are large and incomplete, and answering arbitrary logical queries becomes a complicated task.
The graph incompleteness means that traversing its edges would not provide the correct answers.
This work aims to create some models that allow complex queries and extract the correct answers from large incomplete knowledge graphs.
To this end, we present a set of models based on Neural
Networks that embed the query and the entities into a one-point vector space.
Then, it computes the distance between the query and the entities to rank the answers according to the likelihood to answer the query.
We use the versatility of Neural Networks to create the operators needed to process FOL queries.
We conduct experiments using well-known datasets for KG Reasoning: FB15k, FB15-237, and
NELL.
The experiments show that our models can effectively answer FOL and provide a noticeable improvement when compared with the state-of-the-art baselines.
Our models provide a relative improvement of 5% to 10% to the latest state-of-art method and about 30% to 40% when compared with the method that uses the same idea of one-point vector space embeddings (Hamilton et al., 2018).
The main contributions of this work are summarized as: (1).
New embedding-based methods for logical reasoning over knowledge graphs : two new models, plus variants, for KG Reasoning.
These methods embed the query and the entities in the same vector space with single-point vectors.
Implementing the logical operators with neural networks provides versatility to create any operator with virtually the same architecture.
Improved performance over the current state of the art .
Experimental results show that the models presented in this paper outperform the selected baselines: Graph Query Embeddings (GQE) (Hamilton et al., 2018), Query2Box (Q2B)
(Ren et al., 2020), and BetaE (Ren & Leskovec, 2020).
Handling of negative queries .
Modelling queries with negation has been an open question in KG Reasoning until recently.
BetaE (Ren & Leskovec, 2020) introduced the ﬁrst method able to do so.
This work takes advantages of the good relationship inference capabilities of Neural Networks and uses them to create the negation operator.
R ELATED W ORK
Traditional tasks on graphs include Link Prediction (Liben-Nowell & Kleinberg, 2007), Knowledge
Base Completion (Wang et al., 2015), or basic Query-Answering (one-hop).
They are all different versions of the same problem: Is link (h,r,t) in the KG? or Is t an answer to query (h,r,)?
, where only a variable is missing.
However, we face a more complicated problem, known as Knowledge
Graph Reasoning, that may involve several unobserved edges or nodes over massive and incomplete
KGs.
In this case, queries can be path queries, conjunctive queries, disjunctive queries or or a combination of them.
A formal deﬁnition of KG Reasoning can be found in Chen et al.
(2020), as stated in Deﬁnition 2.1.
Deﬁnition 2.1 (Reasoning over knowledge graphs) .
Deﬁning a knowledge graph as: G =
(cid:104)E , R , T (cid:105) , where E , T represent the set of entities, R the set of relations, and the edges in R link two nodes to form a triple as ( h, r, t ) ∈ T .
Then, reasoning over a KG is deﬁned as creating a triplet that does not exist in the original KG , G (cid:48)
= { ( h, r, t ) | h ∈ E , r ∈ R , t ∈ T , ( h, r, t ) (cid:54)∈ G}
Most related to our work are embedding approaches for multi-hop queries over KGs: (Hamilton et al., 2018), (Ren et al., 2020), (Ren & Leskovec, 2020) and (Das et al., 2016), as well as models for question answering (Yasunaga et al., 2021), (Feng et al., 2020).
The main differences with these methods rely on the ability to handle full First-Order Logical Queries and using Neural Networks to deﬁne all logical operators, including the projection.
We also deliver a more extensive range of network implementations.
On a broader outlook, we identify a series of works that aim to solve Knowledge Graph Reasoning with several different techniques, such as Attention Mechanisms (Wang et al., 2018), Reinforcement
Learning like DeepPath (Xiong et al., 2017b) or DIVA (Chen et al., 2018), or
Neural Logic Networks (Shi et al., 2020), (Qu & Tang, 2019).
M ODELS
Both models presented here follow the idea behind Graph Query Embedding – GQE (Hamilton et al., 2018): Learning to embed the queries into a low dimensional space.
Our models differ from it in the point that logical query operations are represented by geometric operators.
In our case, we do not follow the direct geometric sense and these operators are all represented by Neural Networks, instead of just the Intersection operator in GQE.
Similarly, however, the operators are jointly optimized with the node embeddings to ﬁnd the optimal representation.
In order to answer a query, the system receives a query q , represented as a DAG, where the nodes are the entities and the edges the relationships.
Starting with the embeddings e v 1 , ..., e v n of its anchor nodes and apply the logical operations represented by the edges to ﬁnally obtain an embedding q of the query (Guu et al., 2015).
3. F ORMAL P ROBLEM D EFINITION
A Knowledge Graph ( G ) is a heterogeneous graph with a set of entities – nodes – ( V ) and a set of relations – edges – ( R ).
In heterogeneous graphs, there can be different kinds of relations, which are deﬁned as binary functions r : V × V → { True, False } that connect two entities with a directed edge.
The goal is to answer First-Order Logical (FOL) Queries.
We can deﬁne them as follows:
Deﬁnition 3.1 (First-Order Logical Queries) .
A ﬁrst-order logical query q is formed by an anchor entity set V a ⊆ V , an unknown target variable V ?
and a series of existentially quantiﬁed variables
V 1 , ..., V k .
In its disjunctive normal form (DNF), it is written as a disjunction of conjunctions:
[Equation]
where c i represents a conjunctive query of one or several literals: c i
=
e i 1 ∧ e i 2 ∧ ...
∧ e im .
And the literals represent a relation or its negation: e ij =
r
( v i , v j ) or ¬ v ( v i , v j ) where v i , v j are entities and r ∈ R .
The entity embeddings are initialized to zero and later learned as part of the training process, along with the operators’ weights.
[Figure]
Computation Graph .
The Computation Graph can be deﬁned as the Direct Acyclic Graph (DAG) where the nodes correspond to embeddings and the edges represent the logical operations.
The computation graph can be derived from a query by representing the relations as projections, intersections as merges and negation as complement.
This graph shows the order of operations to answer the queries.
Each branch can be computed independently and then merged until the sink node is reached.
Each node represents a point in the embedding space and each edge represents a logical operation, computed via a Neural Network in our case.
The representation of a FOL as a computation graph can be seen as a heterogeneous tree where each leaf node corresponds to the anchor entities and the root is the ﬁnal target variable, which is a set of entities.
The logical operations corresponding to the edges are deﬁned below:
• Projection .
Given an entity v i ∈ V and a relation type r ∈ R .
It aims to return the set of adjacent entities with that relation.
Being P
r
i ( v i , r ) the set of adjacent entities through r , we deﬁne the projection as: P r i ( v i , r ) = { v (cid:48) ∈ V : ( v, v (cid:48) )
= True } .
• Intersection .
The intersection can be deﬁned as: I ( v i ) =
∩ ni =1 v i .
• Negation .
It calculates the complement of a set of entities T ⊆ V : N ( T ) =
T =
V \ T , where the set can either be the embedding corresponding to an entity or another embedding in between which represents a set of them.
A Union operation is unnecessary, as it will be later discussed in Sections 3.5.
Query2Box (Ren et al., 2020) shows that a union operator becomes intractable in distance-based metrics.
[Equation]
L AYER
[Equation]
ERCEPTRON
[Equation]
(MLP)
Based on the good results of Neural Tensor Networks (NTN) (Socher et al., 2013) for knowledge base completion, we have extended a similar approach to multi-hop reasoning.
We introduce three logical operators to compute the queries.
Each of them is represented by a simple neural network: a multilayer perceptron.
Each perceptron contains a feed-forward network: a linear layer plus a ReLu rectiﬁer.
The number of layers remains as a hyper-parameter.
Figures 2a and 2b show what the model looks like.
Neural operators.
We deﬁne the operators with a multi-layer perceptron.
The model will take as an input the embedding representation of the input entities and will return an approximate embedding of the answer.
Deﬁning the operators with a Neural Network has the advantage of generalization.
Thus, we distinguish between 2-input operators, Projection and Intersection and 1-input operator,
Negation.
- 2-input operator (Figure 2a): Projection P , and Intersection I .
The operator is composed by a multi-layer perceptron that takes 2 inputs and returns 1 as embedding as the output.
The training process will make the networks learn the weights to represent each operation.
Equation 2 expresses it formally:
[Equation]
where s i ∈ S is an embedding in the vector space S , r j ∈ R is a relation and NN k is a multi-layer perceptron with k layers.
The intersection can take more than two entities as an input, for instance the 3i query structure.
In this case we do a recursive operation, we use the result of the previous intersection to compute the next one.
- 1-input operator (Figure 2b): Negation N .
The goal of this operator is to represent the negation of a set of entities.
Following the same neural network approach, we can represent it as in the equation below (Equation 3).
[Equation]
where s i ∈ S is a vector in the embedding space, it can be an entity or the result of a previous operation.
NN k is a multi-layer perceptron with k layers and the same number of inputs as outputs.
3. M ULTI -L AYER P ERCEPTRON M IXER M ODEL (MLP-M IXER )
The MLP-Mixer (Tolstikhin et al., 2021) is a Neural Architecture originally built for computer vision applications, which achieves competitive results when compared to Convolutional Neural Networks (CNNs) and Attention-based networks.
The MLP-Mixer is a model based exclusively on multilayer perceptrons (MLPs).
It contains two types of layers: (1) one with MLPs applied independently to patches and (2) another one with MLPs applied across patches.
Figure 3 presents a diagram of the architecture.
Mixer operators.
We use the same procedure as in the MLP model.
We use a full MLP-Mixer block to train each of the 2 operators with 2 inputs: projection and intersection.
Since negation only has 1 input, the architecture cannot be accommodated for this use so far.
- 2-input operator (Figure 2a).
Represents Projection P or Intersection I with MLP-Mixer architecture.
[Equation]
where MLP mix represent the mixer architecture, s i an embedding in the entity vector space S ; and r j ∈ R a relation.
3. T RAINING OBJECTIVE , D ISTANCE AND I NFERENCE
Training Objective .
The goal is to jointly train the logical operators and the node embeddings, which are learning parameters that are initialized randomly.
Our training objective is to minimize the distance between the query and the query vector, while maximizing the distance from the query to incorrect random entities, which can be done via negative samples.
Equation 5 expresses this training objective in mathematical terms.
[Equation]
where q is the query, v ∈ [ q ] is an answer of query (the positive sample); v (cid:48)
j (cid:54)∈ [ q ] represents a random negative sample and γ refers to the margin.
Both, the margin γ and the number of negative samples k remain as hyperparameters of the model.
Distance measure .
When deﬁning the training objective, we still need to specify the distance measure to compare the entity vectors.
Unlike in previous works, we do not need a measure that compares between boxes or distributions.
The Euclidean distance is enough for this purpose, as it calculates the distance between two points in a Euclidean space: Dist ( v, q )
= | v − q | .
Inference .
Each operator provides an embedding.
Following the query’s computation graph, we obtain the ﬁnal answer embedding (or query representation).
Then, all entities are ranked according to the distance value of this embedding to all entity embeddings via near-neighbor search in constant time using Locality Sensitivity Hashing (Indyk & Motwani, 1998).
3. D ISCUSSION ON ANSWERING FOL QUERIES
The aim of this work is to answer a wide set of logical queries, speciﬁcally, be able to answer ﬁrst-order logical queries (FOL), which includes: conjunctive ∧ , disjunctive ∨ , existential ∃ and ¬ negation operations.
Notice that we will not consider universal quantiﬁcation ( ∀ ) as it does not apply to real-world knowledge graphs since no entity will ever be connected to all other entities in the graph.
Theorem 1 in Query2Box shows that any embedding-based method that retrieves entities using a distance-based method is not able to handle arbitrary disjunctive queries.
To overcome this problem, they transformed the queries into its Disjunctive Normal Form (DNF).
By doing so, the disjunction is placed at the end of the computational graph and can be easily aggregated.
The transformed computational graphs are equivalent to answering N conjunctive queries.
N is meant to be small in practice, and all the N computations can be parallelized.
As expressed in (Davey & Priestley, 2002), all First-Order Logical Queries can be transformed into its DNF form.
We refer readers to (Ren et al., 2020) to understand the transformation process.
E XPERIMENTS AND R ESULTS 4. D ATASETS
We perform experiments on three standard datasets in KG benchmarks.
These are the same datasets used in Query2Box (Ren et al., 2020) and BetaE (Ren & Leskovec, 2020): FB15k (Bordes et al., 2013), FB15k-
(Toutanova et al., 2015) and NELL995 (Xiong et al., 2017b).
In the experiments, we use the standard evaluation scheme for Knowledge Graphs, where edges are split into training, test and validation sets.
After augmenting the KG to also include inverse relations and double the number of edges in the graph, we effectively create 3 graphs: G train for training;
G valid , which contains G train plus the validation edges; and G test which contains G valid and the test edges.
Some statistics about the datasets can be found in Appendix A.
4. Q UERY /A NSWER GENERATION
To obtain the queries from the datasets and its ground truth answers, we consider the 9 query basic structures from Query2box (Ren et al., 2020), shown in Figure 4.
The training, validation, and test graphs were created as: G train ⊆
G val ⊆
G test , therefore the generated queries are also: (cid:74) q (cid:75)
[Table]
⊆
(cid:75) test \ (cid:74)
(cid:75)
the dataset
For the experiments we have used the train/valid/test set of queries-answers used for training and evaluating BetaE (Ren & Leskovec, 2020).
This query-generation system differs from the original in the fact that it limits the number of possible answers to a speciﬁc threshold since some queries in
Query2box had above 5000 answers, which is unrealistic.
More information about how queries are created in BetaE can be found in Appendix A.
To include FOL queries, BetaE (Ren & Leskovec, 2020) created some transformations to includeadditional query structures with negation, shown in the right panel of Figure 5.
4. E VALUATION
Given the rank of answer entities v i ∈ V to a non-trivial test query q we compute the evaluation metrics deﬁned according to Equation 6 below.
Then, we average all queries with the same query format.
[Equation]
where v i is the set of answers V ⊂ (cid:74) q (cid:75) test \ (cid:74) q (cid:75) val , f metric the speciﬁc metric function and rank ( v i ) the rank of answer entities returned by the model.
Mean Reciprocal Rank (MRR) :
It is a statistic measure used in Information Retrieval to evaluate the systems that returns a list of possible responses ranked by their probability to be correct.
Given an answer, this measure is deﬁned as the inverse of the rank for the ﬁrst correct answer, averaged over the sample of queries Q .
Equation 7 express this measure formally.
[Equation]
(7) where rank i refers to the rank position of the ﬁrst correct/relevant answer for the i-th query.
Hits rate at K (H@K) : This measure considers how many correct answers are ranked above K. It directly provides an idea of how the system is performing.
Equation 8 deﬁnes this metric mathematically.
H @
[Equation]
4. B ASELINES AND MODEL VARIANTS
To compare our results, we have selected the state-of-the-art baselines from some previous papers on
KG Logical Reasoning: Graph Query Embeddings (Hamilton et al., 2018), Query2Box (Ren et al., 2020) and BetaE (Ren & Leskovec, 2020).
Only the last method, BetaE, accepts negative queries.
We have modiﬁed the simpler model, MLP, to potentially improve the initial results.
Below, we describe the 3 modiﬁcations:
Heterogeneous Hyper-Graph Embeddings (Sun et al., 2021).
The Heterogeneous Hypergraph
Embeddings creates graph embeddings by projecting the graph into a series of snapshots and taking the Wavelet basis to perform localized convolutions.
In essence, it is an embedding transformation that aims to capture the information of related nodes.
We have added this transformation to our
MLP model right before computing the distance, and then we calculate the distance measure deﬁned speciﬁcally for this new hyper space.
Attention mechanism (Vaswani et al., 2017).
The goal of attention layers is to enhance the “important” parts of the input data and fade out the rest.
This allows the modelling of dependence without regard to the length of the input.
This method has proven a performance increase in many Deep
Learning Applications.
We have implemented an attention mechanism for the intersection operator in the MLP model.
2-Vector Average Approach .
In our models, we create an embedding of the query and the graph entities to a point in a hyperdimensional space.
Since we are using Neural Networks to create these embeddings, the resulting embeddings will depend on the training process and the optimization of the weights.
We have no assurance the embedding is correct.
To add more robustness to the computation, we have decided to calculate the embedding twice with two separate networks and average the results between the two networks.
R ESULTS
Table 1 shows the results of MRR for our models – MLP and MLP-Mixer – when compared to the baselines – GQE, Q2B, BetaE –.
As listed in the table, our methods yield promising results that improve those from the state of the art across all query forms and datasets.
Additionally, Tableshows the results for the model variants previously described in Section 4.4.
We observe the Hyper
Embedding space does not provide an improvement when compared to the basic version of the model.
On the other side, the other two variants do show a signiﬁcance improvement.
Nonetheless, it is worth noting the 2-vector approach may improve the result at the cost of more computer power, as it mainly computes each embedding twice.
Finally, in Table 3 we also show results for experiments on full FOL queries, which include negation.
Additional results for H@1 can be found in Appendix
C.
5. A NALYSIS OF R ESULTS
In general terms, we observe an increased performance over the selected baselines.
In this section we discuss some of the implications:
- Implementation of Logical Operators .
One of the main differences with other approaches such as GQE (Hamilton et al., 2018) or Q2B (Ren et al., 2020) is the use of Neural Networks to represent all logical operations: Projection, Intersection and Negation.
In comparison to GQE that only uses it on the Intersection, or Q2B that creates a set of geometrical operations to represent the logical operations.
In light of these results, it seems that the geometrical implications of the operations can be restraining some of the possible solutions. -
Correctly learning the logical operations .
It is hard to clearly ﬁnd out if the operators are learning the logical operations correctly.
At least, we can assure the Neural Networks do a better job at learning the approximate solution.
Neural Networks that implement directly a logical operation are already currently under research (Shi et al., 2019).
A too constrained implementation of a Neural Logic Network can be found in Appendix D.
- Performance of model variants .
We observe that both Attention Mechanism and the 2-Vector
Approach manage to improve the results from the original MLP model.
This indicates that HyperGraph Embeddings are not correctly transforming the embedding space in our case.
Additionally, the improvement original from the 2-vector approach seems to indicate the optimal solution was not yet reached and there is still room for improvement on learning the correct logical operations.
[Table]
[Table] [Table]
C ONCLUSIONS
In this work, we have introduced a competitive embedding framework for Logical Reasoning over
Knowledge Graphs.
It presents a ﬂexible approach to build logical operators through Neural Networks.
This method accepts queries in its First-Order Logical form, and it is one of the ﬁrst models to accept negative queries.
Extensive experimental results show a signiﬁcance performance improvement when compared to other state-of-the-art methods built for this purpose.
A Q UERY G ENERATION AND S TATISTICS
Statistics about the datasets can be found in Table
However, it is useful to understand how they have been created.
Given the 3 datasets, 3 graphs are created: G train , G val , G test for training, testing and validation, respectively.
As explained in Section 4.2, G train contains the training edges, while
G val contains edges from training + valid and G test from valid + test.
Then they apply pre-order traversal to the target nodes until all anchor entities are instantiated, and use post-order traversal to ﬁnd the answers.
The difference with the query generation process used in Query2Box is a limit on the number of queries, since the original dataset had queries with over 5000 answers, nearly 1/3 of some datasets.
The average number of queries and answers are shown in Tables 5 and
[Table]
[Table] [Table]
Average number of answers divided by query structure.
[Table]
B E XPERIMENTAL DETAILS
Our code is implemented using PyTorch.
For the baselines, we have used the implementation of the baselines and the testing framework from BetaE (Ren & Leskovec, 2020), this also includes the code for Query2Box (Ren et al., 2020) and GQE (Hamilton et al., 2018).
For a fair comparison with the models in the paper, we have selected the same hyperparameters listed in the paper.
This also includes the consideration of the single-point vector methods – GQE, MLP, MLPMixer – having embedding dimensions 2 n .
Since Query2Box needs to model the offset and the center of the box and BetaE has 2 parameters per distribution: α and β , and the embedding dimensions is set to n .
All our models and GQE use the following parameters: Embed dim = 800, learning rate = 0.0001, negative sample size = 128, batch size = 512, margin = 24, num. iterations = 300,000/450,000.
Q2B and BetaE differ from the previous conﬁguration in Embed.
dim = 400 and margin = 30/60.
All experiments have been computed
C A DDITIONAL RESULTS
Hits at K (H@K) is another metric of evaluation that captures the sense of precision of our model to retrieve the correct entities.
It is described in the Section 4.3.
In Table 7 we show the results of
H@1 for all models (MLP-Mixer, MLP) and baselines (BetaE, Q2B, GQE).
In Table 8 we show the
H@1 results for our model variants: HyperE, Attention Layers and 2-vector average.
Finally, Table 9 shows the results of BetaE and MLP on negative queries.
[Table] [Table]
HyperE 64.
FB15k2-vector 43.
Attetion 42.
HyperE 41.
NELL 2-vector 55.
Attention 55.
HyperE 54.
Dataset
Model
[Table]
pin
[Table]
avg
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
FB
[Table]
MLP 2.
BetaE 1.
[Table]
NELL
[Table]
D A PPENDIX : A DDITIONAL M ODELS
[Table]
During this research, we have additionally explored other models, like Convolutional Neural Networks and Neural Logic Networks.
We show the preliminary results of these models in the appendix for informative reasons, available in Table 10.
They have been tested on FB15k-237.
Convolutional Neural Networks (CNNs) CNNs manage to capture the intrinsic dependencies that happen between close items.
This is the reason why they are well suited for images, where closed pixels tend to have similar colors.
This is not the case we ﬁnd in Knowledge Graph embeddings, and we can intuitively its poor performance.
We have adapted a simple model of a CNN to compute the logic operators in our method.
As previously, we have created two different CNNs, one with 2 inputs (Intersection and Projection) and another one with 1 input (Negation).
- 1-input operator .
Represented by CNN with 2 convolutional layers (1st: in channels = 1, out channels = 10, kernel size = 6 , 2nd: in channels = 10, out channels = 10, kernel size = 6 ) + maxPool ( kernel size = 6 ), followed by 3 fully connected layers + ReLu, except for the last one which does not have ReLu.
Input and output sizes are the same.
- 2-input operator .
It uses the same architecture as the 1-input operator, but applies the layers
Conv+MaxPool to the 2 input entities separately and concatenates their outputs to feed the 3 fully connected layers.
Neural Logic Networks (NLN)
Neural Logic Networks (Shi et al., 2019) are a kind of network architecture created to conduct logical inference.
They use vectors to represent the variables, and each logic operation is learned as a neural model with some predeﬁned logic regularizers.
The logic regularizers constraint the neural module to complete the tasks they are conceived for.
They have deﬁned the required regularizers for the most common operations, like NOT, AND, and OR.
In this model, we have implemented the Intersection operator and used the same Projection operator from GQE (Hamilton et al., 2018): P ( q, r )
=
R r q , where R d × d r is a trainable parameter for edge type r and q an entity.
- Intersection operator The intersection operator is implemented as the AND logical operation.
Below, we show the formal deﬁnition of the AND (Equation 9) operation – a basic multilayer perceptron – and its corresponding regularizer in Table 11.
[Equation]
where H a 1 ∈ R d × 2 d , H a 2 ∈ R d × d and b a ∈ R
d are the parameters of the Neural Network.
Operation
[Table]
AND
Identity
[Equation]
Annihilator
[Equation]
Idempotence
[Equation]
Complementation
[Equation]
Comment .
Neural Logic Networks could be a good solution for Reasoning in KGs, especially after they have already been proven to be useful in other reasoning tasks (Shi et al., 2020).
There are many reasons why it might have not worked in this case: a wrong implementation, a unit mismatch between our loss and the regularizer values, or the regularizer being too constraining for the task.