E XPLAINABLE GNN-B ASED M ODELS OVER K NOWLEDGE G RAPHS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Neural Networks (GNNs) are often used to learn transformations of graph data.
While effective in practice, GNNs make predictions via numeric manipulations in an embedding space, so their output cannot be easily explained symbolically.
We propose a new family of GNN-based transformations of graph data that can be trained effectively, but where all predictions can be explained symbolically as logical inferences in Datalog—a well-known rule-based formalism.
We show how to encode an input knowledge graph into a graph with numeric feature vectors, process this graph using a GNN, and decode the result into an output knowledge graph.
We use a new class of monotonic GNNs (MGNNs) to ensure that this process is equivalent to a round of application of a set of Datalog rules.
We also show that, given an arbitrary MGNN, we can extract automatically a set of rules that completely characterises the transformation.
We evaluate our approach by applying it to classiﬁcation tasks in knowledge graph completion.
I NTRODUCTION
Knowledge graphs (KGs) (Hogan et al., 2021) are graph-structured knowledge bases describing an application domain as a collection of entities and their relationships.
They can be captured in formats such as the Resource Description Framework (RDF), and are used in applications such as Web search, recommendation, fraud detection, and data integration.
Many of these applications involve realising a graph transformation.
For example, in a recommender system we can represent user-item interactions as a graph (Aggarwal, 2016) and extend it with external KGs to improve accuracy, diversity, and interpretability of recommendations (Zhang et al., 2016; Wang et al., 2018; 2019a); the system then realises a transformation of the input graph into another graph containing recommended user-item interactions.
Moreover, a KG completion system (Rossi et al., 2021; Bordes et al., 2013; Teru et al., 2020) transforms an input graph into another graph containing relationships missing in the input.
Usually, the transformation is not given explicitly, but is learned from examples.
Graph Neural Networks (GNNs) (Scarselli et al., 2009; Liu & Zhou, 2020) are one of the most popular ML models for graph data.
They are usually applied to KGs in three stages (Hamilton et al., 2017).
First, the input KG is encoded into an embedding space as another graph whose vertices are labelled with numeric feature vectors.
Second, this graph is processed by a GNN with several layers; for each layer, the GNN updates the feature vectors based on the learned model parameters and the vectors of the previous layer.
Finally, the vectors of the output layer are decoded to the output KG.
While such approaches are effective in practice, GNNs compute predictions through numeric calculations in the embedding space, so the transformation can be difﬁcult to understand.
For example, it would be useful to know that a system recommended novel The Idiot because the user liked Crime and Punishment and both novels were written by Dostoevsky.
There is growing interest in explaining GNN predictions, and many methods have focused on identifying relevant subgraphs in the embedding space (Ying et al., 2019;
Luo et al., 2020; Lin et al., 2021).
Such methods, however, do not explain predictions symbolically —that is, by showing how each prediction can be derived from the input KG using rules of inference in a knowledge representation formalism.
Datalog (Abiteboul et al., 1995; Ceri et al., 1989) is a prominent formalism based on ‘if–then’ rules.
When applied to facts, the rules derive new facts.
For example, we can represent knowledge about literature using facts Author ( Dostoevsky , Crime and Punishment ) and Author ( Dostoevsky , The Idiot ) , and past interactions using fact Likes ( Crime and Punishment ) .
Then, the application of rule Author ( x, y 1 )
∧ Author ( x, y 2 ) ∧ Likes ( y 1 ) → Recommend ( y 2 ) to these facts derives the target fact Recommend ( The Idiot ) .
State-of-the-art Datalog engines (Motik et al., 2014) provide powerful methods for generating human-readable proofs explaining why a target fact follows from the input rules and the data.
Such proofs can help fostering trust in system predictions, ensuring compliance with norms, and enabling veriﬁcation of fairness standards.
The main contribution of this paper is a new family of GNN-based transformations of KGs that can be trained from examples as usual, but whose predictions can be explained symbolically using
Datalog rules.
Our transformation consists of the three steps mentioned earlier: encoding an input
KG, applying a GNN, and decoding the result back into an output KG.
These steps, however, were carefully crafted to ensure that the transformation can always be described symbolically.
We use a novel class of monotonic GNNs (MGNNs), which we designed to ensure an important property: when values in input feature vectors increase, no value of an output feature vector decreases.
To capture this intuition formally, we show that our transformation is monotonic under homomorphisms —a well-known property that captures key aspects of Datalog rule application.
In contrast to the related approaches (Teru et al., 2020; Hamaguchi et al., 2017; Schlichtkrull et al., 2018; Wang et al., 2019b), all predictions of our transformation can be explained using Datalog rules.
Speciﬁcally, for an arbitrary MGNN, we can produce an equivalent set of rules so that, on any KG, the rules and the MGNN-based transformation produce the same facts.
This opens the door to new ways of using MGNNs: when developing Datalog rules manually is difﬁcult, we can train an MGNN model in a supervised manner and use it to provide further predictions, but all such predictions can be explained using rules.
For example, we could train an MGNN on a graph containing examples of credit applications and then use it to support decision-making on new applications by applying the model to a different graph; this is analogous to how deep learning is already used in practice.
However, unlike the existing approaches, we can explicate the rules that are implicitly captured by the model and use them to fully explain each credit recommendation.
Thus, we integrate ML and symbolic AI in a single framework, allowing applications to use the best of both worlds.
We demonstrate the effectiveness of our approach by applying it to classiﬁcation tasks in KG completion.
Using well-known benchmarks, we compare our system’s performance with that of DRUM (Sadeghian et al., 2019) and AnyBURL (Meilicke et al., 2019), both state-of-the-art KG completion systems able to also generate rules.
We show that, in addition to providing an exact correspondence between the model and the extracted rules, our technique also offers competitive performance.
Proofs are given in the appendix, and the source code is available as supplementary material.
A N MGNN-B ASED T RANSFORMATION OF K NOWLEDGE G RAPHS
We now describe our GNN-based transformation of knowledge graphs.
In this paper, we see a KG as a dataset , which is a ﬁnite set of facts of the form A ( a ) or R ( a 1 , a 2 ) , where A is a unary predicate ,
R is a binary predicate , and a , a 1 , a 2 are constants .
Constants represent entities from an application domain, unary predicates typically represent entity types, and binary predicates represent relationships between entities.
For example, the statements that a user likes Crime and Punishment and it is a novel written by Fyodor Dostoevsky are represented as facts Likes ( Crime and Punishment ) and
Author ( Dostoevsky , Crime and Punishment ) .
KGs in formats such as RDF can be transformed into such a dataset in the obvious way.
For D a dataset, cn ( D ) is the set of constants occurring in D .
Our transformation is realised by an operator T M on datasets, which is parameterised by a GNN M .
A dataset D is ﬁrst encoded into a graph G D in the embedding space, where vertices are labelled with numeric feature vectors; then, G D is processed by M ; ﬁnally the result is decoded to the output dataset T M ( D ) .
In Section 3, we show that, by restricting M to be a GNN of a special form, called monotonic , we can ensure that T M is equivalent to a round of Datalog rule application; this, in turn, allows us to automatically extract an equivalent set of rules from T M .
The embedding space consists of coloured graphs , which extend the standard notion of undirected graphs in two ways.
First, each vertex is labelled by a numeric feature vector of some ﬁxed dimension; this is necessary in order to allow the coloured graph to be processed by M .
Second, each edge in the graph is assigned a colour, which allows us to distinguish different types of connections in the graph.
Figure 1 illustrates the encoding of the dataset D in the introduction.
First, we introduce vertices v Ds , v CP , and v I in G D to represent constants Dostoevsky , The Idiot , and Crime and Punishment .
The
D :
Dostoevsky
The Idiot
Crime and Punishment
Likes
Author Author
G D : v Ds v
I v CP v Ds,CP v CP,Ds v
Ds,I v I,Ds cc 1 cc 2 ccc 1 cccc
[Figure]
encoding decoding over
Since The and and Punishment do not in a in encoding of D contains neither v CP, nor v
Here, we on an order of to determine the corresponding positions; this is but ﬁxed in advance.
Third, we connect all vertices of G D referring to common constants with coloured edges, which allows us to distinguish different types of joins between facts.
For example, v Ds and v Ds,CP refer to Dostoevsky , so we connect them with an edge of colour c 1 , thus indicating that
Dostoevsky occurs ﬁrst in the constant pair of v Ds,CP .
Analogously, we connect v Ds,CP and v CP with an edge of colour c 2 .
The constant pair of v CP,Ds is the inverse of v Ds,CP ’s, so we connect them with an edge of colour c 3 .
Finally, we use edges of colour c 4 to connect vertices of G D representing constants occurring together in a fact, such as v CP and v Ds .
Formally, we ﬁx a set of four colours Col = { c 1 , c 2 , c 3 , c 4 } .
We assume that datasets and derived facts will use ﬁnitely many predicates, and ﬁx a list A 1 , . . .
, A (cid:15) , R (cid:15) +1 , . .
, R δ of available unary and binary predicates.
We consider vectors and matrices over the reals R and its nonnegative subset R + .
For v a vector, ( v ) i is the i -th element of v .
We apply scalar functions to vectors and matrices element-wise; for example, for v 1 , . .
. , v n vectors of equal dimension, max { v 1 , . .
. , v n } is the vector whose i -th element is max { ( v 1 ) i , . . .
, ( v n ) i } .
Finally, a function σ : R → R is monotonically increasing if x < y implies σ ( x ) ≤ σ ( y ) , and function σ is Boolean if it ranges over { 0 , 1 } .
The following deﬁnition describes the structure of our embedding space.
Deﬁnition 1.
A ( Col , δ ) - graph is a tuple (cid:104) V, { E c } c ∈ Col , λ (cid:105) , where V is a ﬁnite set of vertices; for each c ∈ Col , E c is a set of c -coloured undirected edges (possibly including self-loops) over V ; and labelling λ assigns to each v ∈ V a vector λ ( v ) of dimension δ .
If λ is implicitly understood, λ ( v ) is written as v .
For c ∈ Col and v ∈ V , N c ( v ) is the set of vertices connected to v by an edge in E c .
We are now ready to deﬁne our encoding of an arbitrary dataset D into a ( Col , δ ) -graph
G D .
Deﬁnition 2.
The encoding of a dataset D is the ( Col , δ ) -graph
G D that contains a vertex v a for each constant a ∈ cn ( D ) , and vertices v a,b and v b,a for all (not necessarily distinct) constants a and b that occur together in a binary fact in D .
For each vertex v a,b , G D contains an edge of colour cconnecting v a,b and v a , an edge of colour c 2 connecting v a,b and v b , an edge of colour c 3 connecting v a,b and v b,a , and an edge of colour c 4 connecting v a and v b .
For each v a , labelling ( v a ) i is 1 if
A i ( a ) ∈ D and 0 otherwise.
For each v a,b , labelling ( v a,b ) i is 1 if R i ( a, b ) ∈ D and 0 otherwise.
We process G D by a GNN.
As we explain in more detail Section 3, if we are to ensure that our transformation ‘mimics’ Datalog rule application, we cannot use an arbitrary GNN.
Instead, we introduce Monotonic Graph Neural Networks (MGNNs) , which restrict GNNs in a way that allows us to attain our objective.
Deﬁnition 3 also speciﬁes how to apply an MGNN to a coloured graph.
Deﬁnition 3.
A ( Col , δ ) - monotonic graph neural network ( MGNN ) M with L ≥ 1 layers is a tuple (cid:104){ A (cid:96) } 1 ≤ (cid:96) ≤ L , { B (cid:96)c } c ∈ Col , 1 ≤ (cid:96) ≤ L , { b (cid:96) } 1 ≤ (cid:96) ≤ L , σ, cls (cid:105) .
For each (cid:96) ∈ { 1 , . . .
, L } and each c ∈ Col , A (cid:96) and B (cid:96)c are matrices over R + of dimension m (cid:96) × n (cid:96) , and b (cid:96) is a vector over R of dimension n (cid:96) such that n 1 = m L = δ , and m (cid:96) − 1 = n (cid:96) for each (cid:96) ∈ { 2 , . . .
, L } .
Moreover, σ : R → R + is a monotonically increasing activation function, and cls : R + → { 0 , 1 } is a monotonically increasing classiﬁcation function.
The dimension of M is max { n 1 , . . .
, n L } .
Applying M to a ( Col , δ ) -graph G = (cid:104) V, { E c } c ∈ Col , λ (cid:105) induces the sequence λ 0 , . .
λ L of vertex labellings where λ 0 = λ , and for each (cid:96) ∈ { 1 , . . .
, L } and v ∈ V , the value of λ (cid:96) ( v ) is given by
[Equation]
The result M ( G ) of applying M to G is the ( Col , δ ) -graph with the same vertices and edges as G , but where each vertex v ∈ V is labelled by cls ( λ L ( v )) .
Intuitively, an MGNN consists of L layers, where each layer (cid:96) ∈ { 1 , . . .
, L } is deﬁned by a matrix
A (cid:96) , a matrix B (cid:96)c for each colour c , and a bias vector b (cid:96) ; these constitute the learnable parameters of the model.
In addition, an MGNN also uses functions σ and cls which are ﬁxed and given explicitly.
An MGNN transforms an arbitrary ( Col , δ ) -graph G into another ( Col , δ )
-graph M ( G ) as follows.
Each vertex v in G and each (cid:96) ∈ { 0 , . . .
, L } is associated with a labelling v (cid:96) .
Labelling v 0 is the labelling v of v in G , and, for each layer (cid:96) , vertex v is labelled by a feature vector v (cid:96) computed as speciﬁed in equation (1).
The feature vector of v in the resulting graph is computed by applying the classiﬁcation function cls to the ﬁnal labelling v L .
Thus, MGNNs instantiate the GNN framework by (i) taking the maximum of the features of a vertex’ neighbours (instead of the more commonly used sum or average), (ii) using nonnegative weights in matrices A (cid:96) and B (cid:96)c (but the elements of bias vectors b (cid:96) can be negative), and (iii) requiring the activation and classiﬁcation functions to be monotonically increasing.
We discuss the reasons for these restrictions in Section 3.
The result M ( G D ) of applying an MGNN to the encoding G D of a dataset D can be decoded into the output dataset of the transformation by essentially inverting the encoder (see Figure 1 for an example).
Thus, for each MGNN M , we obtain a distinct operator T M that is speciﬁed as follows.
Deﬁnition 4.
For M a ( Col , δ ) -MGNN
and D a dataset, T M ( D ) is the dataset containing the fact
A i ( a ) for each i ∈ { 1 , . . .
, (cid:15) } and each vertex v a such that ( v a ) i = 1 in M ( G D ) , and the fact
R i ( a, b ) for each i ∈ { (cid:15) + 1 , . . . , δ } and each vertex v a,b such that ( v a,b ) i = 1 in M ( G D ) .
Note that the dimensions of the matrices in M do not predetermine the kind of dataset to which
T M can be applied; in fact, M can be trained on some dataset and then applied to a different dataset over the same predicates.
This makes our transformation ideally suited for inductive settings in applications such as KG completion and recommendation (Teru et al., 2020;
Hamaguchi et al., 2017; van den Berg et al., 2017), where the datasets to which a trained model is applied may use constants (i.e., entities ) not mentioned in the datasets on which the model was trained.
E XTRACTING A N E QUIVALENT S ET OF D ATALOG R ULES
We now show that, for each MGNN M , we can automatically generate a set of Datalog rules P M such that T M and P M derive the same facts on any dataset D .
To this end, we ﬁrst recapitulate the basic deﬁnitions of Datalog and identify a property that characterises Datalog reasoning.
A (Datalog) atom is of the form
A ( t 1 ) or R ( t 1 , t 2 ) , where A and R are unary and binary predicates, respectively, and each t i is a constant or variable .
A (Datalog) rule is an implication of the form (2), where H is the atom called the rule’s head , all B i are atoms that together form the rule’s body , and each variable in H occurs in some B i .
[Equation]
A (Datalog) program is a ﬁnite set of rules.
A substitution ν is a mapping of ﬁnitely many variables to constants; for α an atom, αν is the result of replacing in α each variable x on which ν is deﬁned with ν ( x ) .
Each rule r of form (2) deﬁnes the immediate consequence operator T r on datasets: for
D a dataset, T r ( D ) contains fact Hν for each substitution ν satisfying { B 1 ν, . . .
B m ν } ⊆ D .
For
P a program, operator T P on datasets is deﬁned by T P ( D ) =
(cid:83) r ∈P T r ( D ) .
Our objective is to show that, for each MGNN M , operator T M ‘behaves analogously’ to Datalog rule application.
Deﬁnition 5 speciﬁes formally what this means in our setting.
Deﬁnition 5.
A homomorphism from a dataset D to a dataset D (cid:48) is a mapping h of constants to constants that is deﬁned at least on cn ( D ) and that satisﬁes h ( D )
⊆
D (cid:48) , where h ( D ) is the dataset obtained by replacing each constant a in D with h ( a ) and removing all duplicate facts.
An operator
T from datasets to datasets is monotonic under homomorphisms if, for all datasets D and D (cid:48) , each homomorphism from
For any constant-free program P , operator T P is monotonic under homomorphisms.
For example, let D =
{ A ( a ) , R ( a, b ) } and P = { S ( x, y )
← A ( x ) ∧ R ( x, y ) } , so T P ( D ) = { S ( a, b ) } .
First, operator T P is monotonic : adding facts to D cannot invalidate any consequence of T P , since the body of r still matches and the rule derives S ( a, b ) .
Second, operator T P is resilient to replacement of constants.
For example, let D (cid:48)
=
{ A ( c ) , R ( c, c ) } be obtained from D by replacing a and b with c .
Rule r ‘cannot tell’ that the constants have been replaced, deriving T P ( D (cid:48) )
= { S ( c, c ) } , which is obtained from T P ( D ) by the same replacement.
Monotonicity under homomorphisms combines both properties.
We next show that rules can be extracted from any operator satisfying this property.
Deﬁnition 6.
An operator T on datasets captures a rule or program α if T α ( D ) ⊆
T ( D ) for each dataset D .
Proposition 7.
Let T be an operator on datasets, let r be a constant-free rule of form (2) , let µ be a substitution that maps each variable of r to a distinct constant, and let D r = { B 1 µ, . . .
, B m µ } .
If
T is monotonic under homomorphisms and Hµ ∈ T ( D r ) , then T captures r .
Intuitively, if some operator T captures a rule r , then T r provides us with a lower bound for the consequences of T on an arbitrary dataset—on any dataset, T derives at least as many facts as r (but T can derive more facts).
Moreover, if T is monotonic under homomorphisms, Propositionprovides us with an effective way of checking whether T captures r : replace each variable in r with a distinct constant, collect the resulting body atoms in a dataset D r , apply T to D r , and ﬁnally check whether the result contains the replaced head atom of r .
Clearly, T captures a constant-free program
P if T captures each rule r ∈ P , so we can just apply this check to each r ∈ P to verify whether T captures P .
This provides us with the necessary machinery for rule extraction.
We next show that operator T M is monotonic under homomorphisms.
This shows that T M indeed ‘behaves like’ Datalog reasoning, and it will allow us to use Proposition 7 for rule extraction.
Theorem 8.
For each ( Col , δ ) -MGNN M , mapping T M is monotonic under homomorphisms.
Theorem 8 is a consequence of the restrictions on MGNNs mentioned after Deﬁnition 3.
For monotonicity, note that adding facts to a dataset D has the effect of possibly adding new vertices and edges and then ﬂipping from 0 to 1 some vector positions labelling the vertices of G D ; moreover, the elements of A (cid:96) and B (cid:96)c are nonnegative so the arguments of σ in equation (1) can only increase; ﬁnally, functions σ and cls are monotonically increasing so the elements of the feature vectors in M ( G D ) can only ﬂip from 0 to 1 , which is equivalent to deriving more facts in T M ( D ) .
For the resilience of constant merging, replacing, say, constant b with constant a in D is tantamount to merging vertices v b into v a in G D ; this will cause vertex v a to have fewer neighbours, but since we aggregate neighbours’ feature vectors by taking the maximum, this cannot change the result of equation (1).
Note that, if we aggregated neighbours using a sum (as is common in the literature), then elements of the feature vectors in M ( G D ) could decrease due to merging, which would invalidate Theorem 8.
Theorem 8 tells us that we can check whether T M captures a rule r effectively using Proposition 7.
This is a key step of our rule extraction algorithm.
In particular, we next show that it is sufﬁcient to consider only rules of a certain structure, and that the number of such rules is bounded and determined by the dimension of M , the number of layers of M , and the number of predicates δ .
Thus, to obtain a program P M deriving the same consequences as T M , we can, in principle, enumerate all such rules, apply Proposition 7 to each rule, and keep only the rules that pass this check.
To realise such procedure in practice, several optimisations are possible.
For example, the relevant space of rules can be signiﬁcantly reduced by exploiting the fact that rules can be subsumed (i.e., logically implied) by other rules; for instance, if we have determined that a rule B ( x ) ← A ( x ) has been captured, then rules of the form B ( x ) ← A ( x ) ∧ C , with C any conjunction, do not need to be considered since such rules are logically implied by the previous one and are hence redundant.
Deﬁnition 9 describes the syntactic structure of the rules that can fully characterise T M .
To this end, we use the well-known notion of a primal graph of a conjunction C of atoms, which is the undirected graph (possibly with self-loops) whose vertices are the variables of C , and where an edge connects each pair of (not necessarily distinct) variables x and y such that R ( x, y ) is in C for some R .
Note that the primal graph of an empty conjunction is also empty.
A primal graph is a (possibly empty) tree if it is connected and acyclic (and, in particular, without self-loops).
Deﬁnition 9.
Let c and d be nonnegative integers.
A constant-free conjunction C is ( c, d ) - treelike for a variable x if the primal graph of C is a tree that is either empty or can be rooted at x
(i.e., x occurs in C ) and, for each vertex v in the tree, the distance i of v from x is at most d and moreover v has at most c ( d − i ) children.
A ( c, d ) - tree-like rule is a rule of form A ( x ) ← C x or
R ( x, y ) ← C x ∧ B x,y ∧ C y , where C x and C y are ( c, d ) -tree-like conjunctions for x and y , respectively, with no variables in common, and B x,y is a nonempty constant-free conjunction of binary atoms of the form S ( x, y ) or S ( y, x ) .
For M a ( Col , δ ) -MGNN with L layers and dimension k (see Deﬁnition 3), let P M be the program that contains, up to variable renaming, each (3 k, L ) -tree-like rule that is captured by T M .
The primal graph of rule A ( x ) ∧ R ( x, y ) ∧ S ( x, z ) is a tree with root x and children y and z ; thus, it is (2,1)-tree like, but not (1,1)-tree-like (since x is at distance 0 from itself it can have at most one child).
Second, the primal graph of A ( x ) ∧ R ( x, y )
∧ S ( x, z ) ∧ T ( y, z ) is a triangle with nodes x , y , and z , so it is not ( c, d ) -tree-like for any c and d .
Theorem 10.
For each ( Col , δ ) -MGNN M and each dataset D , we have T M ( D ) =
T P M ( D ) .
Theorem 10 also describes the expressiveness (i.e., the kind of possible predictions) of T M .
For example, no MGNN can capture rules such as r =
R ( x, z ) ← R ( x, y )
∧ R ( y, z ) .
To see this, let D =
{ R ( a, b ) , R ( b, c ) } .
For an arbitrary MGNN M , Theorem 10 ensures that all predictions of
T M involving binary facts are made by rules of the form R ( x, y )
← C x ∧ B x,y ∧ C y ; however, no such rule can derive R ( a, c ) on D , so we necessarily have R ( a, c ) (cid:54)∈ T M ( D ) as well.
E VALUATION
We have evaluated our approach on KG completion tasks, which involve extending a KG seen as a dataset D to its complete version D (cid:48) by adding missing facts.
When seen as a classiﬁcation problem, the aim is to learn a Boolean function f ( · , · ) that takes a dataset D and a fact α over a ﬁxed set of unary and binary predicates such that f ( D, α ) returns true if and only if α ∈ D (cid:48) .
We consider the more general inductive variant of this task, where the testing dataset may contain constants not occurring in the training or validation datasets (but all predicates in the testing dataset are known before training).
Inductive KG completion is more general and challenging than the transductive variant (Bordes et al., 2013), where all constants in the testing data also occur in the training data.
Baselines.
We compared our approach with DRUM (Sadeghian et al., 2019) and AnyBURL (Meilicke et al., 2019), two state-of-the-art approaches for knowledge graph completion.
DRUM learns an end-to-end differentiable model that, given a dataset D and a fact α , returns a score representing the likelihood of α belonging to the completion of D ; predictions are then computed by applying a threshold to the score.
The system also produces a set of chain Datalog rules of the form H ( x 1 , x n ) ← B 1 ( x 1 , x 2 ) ∧ B 2 ( x 2 , x 3 ) ∧ · · · ∧ B n − 1 ( x n − 1 , x n ) , each associated with a conﬁdence score.
These extracted rules approximate the model’s predictions, but, unlike in our work, the exact relationship between the predictions and the result of applying the rules is unspeciﬁed.
AnyBURL uses path sampling techniques to generate chain rules of a slightly more general form (variables can be replaced by constants).
Each rule is associated with a score.
Given dataset D and fact α , AnyBURL makes its prediction by applying a threshold to a score computed by ﬁrst applying the rules on D once and then combining the probabilities associated to all rules deriving α .
Benchmarks and performance metrics.
We used the 12 KG completion benchmarks by Teru et al.
(2020), which are based on FB15K-
(Bordes et al., 2013), NELL-995 (Xiong et al., 2017), and WN18RR (Dettmers et al., 2018).
Each benchmark provides disjoint datasets T , V , and S for training, validation, and testing, respectively; statistics are shown in Table 1.
We evaluated performance of our systems as follows.
Each benchmark provides a method for randomly splitting its testing dataset S into disjoint subsets S I and S C ; intuitively, S I is the incomplete dataset to be completed with dataset S C .
Thus, we took ( S I , α ) for each fact α ∈ S C as a positive example for testing.
Moreover, we randomly sampled |S C | facts β 1 , . .
, β n over the predicates of
S I satisfying β i (cid:54)∈ S C , and we took each ( S I , β i ) as a negative example for testing.
Sampling was necessary because the set of possible facts not contained in S C is very large; moreover, random
[Table] [Table]
does not favour a speciﬁc way of constructing negative examples during training.
The number of positive and negative examples was the same to ensure that metrics are balanced.
For each system, we classiﬁed the positive and negative testing examples and computed the precision, recall, accuracy, and F1 score in the usual way, as well as the area under the precision-recall curve (AUC) by considering different classiﬁcation thresholds between 0 . 01 and 0 . 99 .
Training.
Our system is trained as a denoising autoencoder (Vincent et al., 2010).
The training dataset T is ﬁrst split with a 9:1 ratio into an incomplete dataset T I and a set T C of facts that should hold in the completion of T I .
To train our model, we used ( T I , α ) for each fact α ∈ T C as a positive example, and we obtained negative examples ( T I , β ) by sampling β analogously to how this was done for testing.
We consider MGNNs with 2 layers and ReLU as activation function.
We used cross-entropy loss with a logistic sigmoid on λ L ( v ) as the output probability, and we trained using Adam optimisation with the standard learning rate ( 0 . 01 ) and weight decay ( 5 × 10 − 4 ), and a maximum of 50 , 000 epochs.
We implemented the procedure in Pytorch Geometric v1.5.0, and ran it on a laptop running macOS 10.15.7 with 8 GB of RAM and an Intel Core i5 2.30 GHz CPU.
We trained the baseline systems using their (publicly available) code with their default conﬁgurations.
For each system and benchmark, we computed the accuracy on the validation dataset for a range of thresholds, and we used in our tests the threshold that maximises accuracy.
Evaluation results.
Table 2 shows the results of our evaluation.
As one can see, our system was overall very competitive, and it outperformed both baselines on most benchmarks based on NELL995.
Table 2 also shows the training times for DRUM and our system, which are are comparable; no times can be shown for AnyBURL because this system does not train a model.
Similarly to AnyBURL, our system attained extremely high precision scores; however, recall values were signiﬁcantly lower, especially for the benchmarks based on FB15K. This can be explained by the fact that, after randomly splitting the testing dataset S into subsets S I and S C , numerous facts of the form R ( a, b ) in S C use pairs of constants a and b that do not occur together in any fact in S I .
By Theorem 10 from Section 3, our transformation can never produce such facts in the output.
To conﬁrm our hypothesis that this is the main reason for low recall scores, we additionally split the testing dataset S so that a fact R ( a, b ) is placed into S C only if S I contains a fact of the form S ( a, b ) or S ( b, a ) for some predicate S .
The results for our system obtained in this way are shown in Table 3.
As one can see, precision scores remain high, whereas recall values increase signiﬁcantly; thus, accuracy, F1 score, and AUC values improve accordingly. Our system signiﬁcantly outperformed all
[Table] [Table]
[Table]
baselines in this setting; however, this way of splitting the data gives our system an unfair advantage in a comparison, and hence we did not present the results for the baselines in Table 3.
Rule Extraction.
For each benchmark, we used the algorithm in Appendix D to extract the set P of all non-redundant rules in program P M (c.f. Deﬁnition 9) with at most two body atoms.
Tableshows the numbers of such rules learned on each benchmark.
Since all rules of P are captured by T M , operator T P approximates T M ; in particular, T P ( D ) ⊆ T M ( D ) for any dataset D .
To measure the quality of the approximation, we applied T P to the test dataset of each benchmark. We observed that precision remained unchanged, while recall decreased only slightly (1.6% in average).
This shows that the generated “short” rules account for almost all predictions on the benchmarks. By analysing the rules in P , we noted that our transformation learned many rules describing sound inferences such as symmetry (e.g., sibling ( x, y ) ← sibling ( y, x ) on FB15K-237), inverse relations (e.g., capitalOfArea ( x, y )
← areaHasCapital ( y, x ) on FB15K-237), and relationship subsumption (e.g., athletePlayedForTeam ( x, y ) ← athleteLedSportsTeam ( x, y ) on NELL-995).
In addition, we generated rules using AnyBURL on each benchmark and checked how many of those are captured by our transformation; 2 Table 5 summarises our results.
The ﬁrst column provides the total number of rules generated by AnyBURL on each benchmark.
The second column indicates how many of those rules were tree-like as per Deﬁnition 9.
The third column shows how many rules from the second column are captured by our transformation per Deﬁnition 6.
Finally, the fourth column shows the number of AnyBURL rules r that are captured by T M on the speciﬁc dataset —that is, such that T r ( S I ) ⊆
T M ( S I ) .
As one can see, a signiﬁcant proportion of tree-like rules generated by AnyBURL are captured by T M .
Additionally, a signiﬁcant proportion of (not necessarily tree-shaped) rules generated by AnyBURL are captured by T M on the speciﬁc dataset, which partially explains why many of the predictions by AnyBURL are also obtained by T M .
R ELATED W ORK
GNN-based approaches to KG completion (Hamaguchi et al., 2017; Schlichtkrull et al., 2018; Xu et al., 2020;
Teru et al., 2020; Zhang et al., 2020) are becoming increasingly popular as they are more generally applicable than earlier solutions based on graph embeddings (Rossi et al., 2021).
Typically, the predictions made by these techniques cannot be explained symbolically.
Rule mining techniques for KGs can categorised in two groups.
Systems in the ﬁrst category learn a model on training data and provide ways to extract rules from it: DRUM and similar approaches (Yang et al., 2017; Sadeghian et al., 2019; Wang et al., 2020) use RNNs, while others use reinforcement learning (Xiong et al., 2017; Das et al., 2018), dynamic neural module networks (Rockt¨aschel & Riedel, 2017; Campero et al., 2018), or graph embeddings (Omran et al., 2018; Zhang et al., 2019).
However, the formal relationship between the model and the rules extracted from it is unspeciﬁed in these approaches; hence, model predictions can differ from the result of applying the rules.
Systems in the second category, such as AnyBURL, use training data to identify patterns of paths in the graph that are represented as rules (Gal´arraga et al., 2015; Meilicke et al., 2019; Ahmadi et al., 2020; Gu et al., 2020); hence, they do not involve training an ML model.
Inductive Logic Programming (ILP) generates rules given a dataset and examples of positive and negative inferences so that rule application yields all positive and no negative examples (Muggleton, 1991).
ILP techniques cannot typically handle noisy examples (Muggleton, 1991; Cropper &
Muggleton, 2014; Si et al., 2019; Raghothaman et al., 2020).
Other recent ILP systems, such as ∂ ILP (Evans & Grefenstette, 2018), achieve robustness to noise by interpreting the ILP task as a binary classiﬁcation problem and providing a differentiable implementation of deduction.
These approaches are similar to rule mining techniques, and they compare to our work analogously.
Furthermore, they focus on learning rules from small datasets and struggle with large-scale KGs.
Neuro-symbolic computation focuses on integrating Logic with neural networks (d’Avila Garcez et al., 2002).
Rule extraction methods have focused on identifying classes of rules for which the immediate consequence operator can be simulated by recurrent (H ¨ olldobler et al., 1999), ﬁbring (Bader et al., 2005), or feed-forward networks (Bader et al., 2007).
Their embedding methods are, however, different from ours, as they encode groups of facts as numeric vector components and they can only extract variable-free rules from trained networks.
Recently, Dong et al.
(2019) presented a network architecture that simulates application of arbitrary function-free ﬁrst-order rules to a dataset.
This work, however, does not provide an algorithm extracting rules from a trained model.
Existing methods for explaining the predictions of a GNN focus on identifying parts of the graph in the embedding space that are most relevant to a given prediction (Ying et al., 2019; Luo et al., 2020;
Lin et al., 2021).
These techniques are largely independent from the speciﬁcs of the model used.
In contrast, we focus on a speciﬁc type of GNN and provide a logical proof for each prediction.
GNNs were shown capable of expressing graph property tests such as isomorphism, existence of cliques, bipartiteness, and planarity (Xu et al., 2019; Morris et al., 2019;
Garg et al., 2020).
Moreover, ﬁrst-order logic queries expressible by GNNs correspond to the description logic ALCQ (Barcel´o et al., 2020).
Our work follows this line of research by relating GNNs and Datalog.
There has been work on neural learning of functions that are monotonic for the ordering on the real numbers You et al.
; Gupta et al.
; Wehenkel & Louppe (2019), and which rely on the use of non-negative matrices and monotonic activation functions.
Our work, however concerns
GNNs and the learning of monotonic functions for the ordering induced by the existence of homomorphisms between datasets, which is the notion of monotonicity in Logic and Data Management.
C ONCLUSION AND F UTURE W ORK
We presented a GNN-based transformation of datasets that mimics a round of application of Datalog rules.
The predictions made by our transformation can be explained symbolically, and we have shown our approach to be practically feasible.
We see many avenues for future work.
On the theoretical side, we shall develop extensions that capture more complex rules (e.g., transitivity), as well as nonmonotonic extensions such as negation or aggregation.
On the practical side, we shall establish links with the existing approaches to rule mining and ILP, thus allowing us to compare the performance of all of these heterogeneous approaches in a uniﬁed way.
A P ROOF OF P ROPOSITION
Proposition 7.
Let T be an operator on datasets, let r be a constant-free rule of form (2) , let µ be a substitution that maps each variable of r to a distinct constant, and let D r = { B 1 µ, . . .
, B m µ } .
If
T is monotonic under homomorphisms and Hµ ∈ T ( D r ) , then T captures r .
Proof.
Consider T , r , D r , and µ that satisfy the prerequisites of the proposition.
Moreover, consider an arbitrary dataset D and an arbitrary fact α ∈ T r ( D ) ; we show that α ∈ T ( D ) .
By the deﬁnition of operator T r , there exists a substitution ν such that { B 1 ν, . . .
B m ν } ⊆ D and Hν = α .
Now let h be the mapping of constants to constants such that h ( xµ ) =
xν for each variable x in occurring in r .
This deﬁnition of h is correct since µ maps each variable in r to a distinct constant.
Also, rule r is constant-free, so h ( D r )
=
h ( { B 1 µ, . . .
, B m µ } )
= { B 1 ν, . .
B m ν } ⊆ D holds; moreover, each constant in D r is obtained by instantiating a variable of r , so h is deﬁned on all of cn ( D r ) ).
In other words, h is a homomorphism from D r to D .
Since T is monotonic under homomorphisms by the assumption of the proposition, h is also a homomorphism from T ( D r ) to T ( D ) .
Moreover, Hµ ∈ T ( D r ) holds by our assumption, so the deﬁnition of homomorphisms ensures h ( Hµ ) ∈ T ( D ) .
Finally, atom H is constant-free, which together with the deﬁnition of h ensures h ( Hµ )
=
Hν .
Thus, we have Hν ∈ T ( D ) , as required.
B P ROOF OF T HEOREM
We prove Theorem 8 in two steps.
First, we ﬁrst introduce in Deﬁnition B1 a notion of homomorphisms on ( Col , δ ) -graphs that is analogous to the notion of homomorphisms on datasets.
Then, in
Lemma B2 we show that the transformation of ( Col , δ ) -graphs by an MGNN preserves such homomorphisms.
Theorem 8 ﬁnally combines this result with our encoding and decoding steps.
To formalise these results, we extend the order ≤ on real numbers to a partial order on vectors so that a 1 ≤ a 2 holds whenever ( a 1 ) i ≤ ( a 2 ) i holds for each position i .
Deﬁnition B1.
A label-respecting homomorphism from a ( Col , δ ) -graph (cid:104) V, { E c } c ∈ Col , λ (cid:105) to a ( Col , δ ) -graph
(cid:104) V (cid:48) , { E (cid:48)
c } c ∈ Col , λ (cid:48) (cid:105) is a mapping h : V → V (cid:48)
such that λ ( v ) ≤ λ (cid:48)
( h ( v )) for each vertex v ∈ V , and, for each colour c ∈ Col and each edge in E c between vertices v and w , vertices h ( v ) and h ( w ) are connected by an edge in E (cid:48) c .
Lemma B2.
For each ( Col , δ ) -MGNN M and all ( Col , δ ) -graphs G and G (cid:48) , each label-respecting homomorphism h from G to G (cid:48) is also a label-respecting homomorphism from M ( G ) to M ( G (cid:48) ) .
Proof.
Consider a ( Col , δ )
-MGNN M with components as speciﬁed in Deﬁnition 3, ( Col , δ ) -graphs
G = (cid:104) V, { E c } c ∈ Col , λ (cid:105) and G (cid:48) =
(cid:104) V (cid:48) , { E (cid:48) c } c ∈ Col , λ (cid:48) (cid:105) , and a label-respecting homomorphism h from
G to G (cid:48) .
Since M ( G ) and M ( G (cid:48) ) have the same edges as G and G (cid:48) , respectively, h satisﬁes the ﬁrst condition of a label-preserving homomorphism from M ( G ) to M ( G (cid:48) ) .
We show that h also satisﬁes the second condition.
To this end, let λ 0 , . . .
, λ L and λ (cid:48) 0 , . . .
, λ (cid:48)
L be the sequences of vertex labelling functions used in the computation of M ( G ) and M ( G (cid:48) ) , respectively.
We next show by induction on 0 ≤ (cid:96) ≤ L that λ (cid:96) ( v ) ≤ λ (cid:48) (cid:96) ( h ( v )) holds for each vertex v ∈ V 1 ; since classiﬁcation function cls is monotonically increasing, this will imply that cls ( λ L ( v )) ≤ cls ( λ (cid:48)
L ( h ( v ))) holds for each vertex v ∈ V 1 as well, which proves our claim.
The base case for (cid:96) = 0 holds immediately since λ 0 =
λ and λ (cid:48)
= λ (cid:48) .
For the inductive step, we assume that, for some (cid:96) − 1 , we have λ (cid:96) − 1 ( v ) ≤ λ (cid:48) (cid:96) − 1 ( h ( v )) for each v ∈ V 1 , and we show that λ (cid:96) ( v ) ≤ λ (cid:48) (cid:96) ( h ( v )) also holds for each v ∈ V 1 .
To this end, consider arbitrary i between 1 and the dimension of b (cid:96) .
By Deﬁnition 3, the i -th components of λ (cid:96) ( v ) and λ (cid:48) (cid:96) ( h ( v )) are equal to σ ( x 1 ) and σ ( x 2 ) , where x 1 and x 2 are as in (3) and (4) for ( A (cid:96) )
i and ( B (cid:96)c ) i the i -th rows of A (cid:96) and B (cid:96)c , respectively:
[Equation]
By the inductive hypothesis, we have λ (cid:96) − 1 ( v ) ≤ λ (cid:48) (cid:96) − 1 ( h ( v )) ; moreover, all numbers in A (cid:96) are nonnegative, so we clearly have ( A (cid:96) )
i λ
[Equation]
− 1 ( v ) ≤ ( A (cid:96) )
i λ (cid:48) (cid:96) − 1 ( h ( v )) .
In addition, mapping h satisﬁes the ﬁrst property of label-preserving homomorphisms so, for each colour c ∈ Col and each w ∈ N c
( v ) , we have h ( w ) ∈ N c ( h ( v )) ; moreover, by the inductive hypothesis, we have λ (cid:96) − 1 ( w ) ≤ λ (cid:48) (cid:96) − 1 ( h ( w )) .
By combining these two observations, we have
[Equation]
which, together with the fact that all numbers in ( B (cid:96)c ) i are nonnegative, implies
[Equation]
Consequently, we have x 1 ≤ x 2 .
Finally, activation function σ is monotonically increasing, so σ ( x 1 ) ≤ σ ( x 2 ) holds, which completes our proof of the inductive step.
Theorem 8.
For each ( Col , δ ) -MGNN M , mapping T M is monotonic under homomorphisms.
Proof.
Consider an arbitrary ( Col , δ ) -MGNN M , arbitrary datasets D and D (cid:48) , and an arbitrary homomorphism h from D to D (cid:48) ; thus, h ( D ) ⊆
D (cid:48) holds.
Now let G D and G D (cid:48) be ( Col , δ ) -graphs obtained by transforming D and D (cid:48) as in Deﬁnition 2.
Moreover, let g be the mapping from the vertices of G 1 to the vertices of G 2 deﬁned as follows:
• g ( v a ) = v h ( a ) for each vertex v a of G 1 ; and • g ( v a,b ) = v h ( a ) ,h ( b ) for each vertex v a,b of G
It is straightforward to see that g is a label-respecting homomorphism from G D to G D (cid:48) .
By
Lemma B2, mapping g is also a label-respecting homomorphism from M ( G D ) to M ( G D (cid:48) ) .
Thus, however, implies that h is also a homomorphism from T ( D ) to T ( D (cid:48) ) , as required.
C P ROOF OF T HEOREM
Theorem 10.
For each ( Col , δ ) -MGNN M and each dataset D , we have T M ( D ) =
T P M ( D ) .
Proof.
Consider arbitrary M , P M , and D as stated in the claim.
Let L be the number of layers of
M , and let k be the dimension of M .
Since operator T P M captures each rule in P M by deﬁnition, we clearly have T P M ( D ) ⊆
T M ( D ) .
For the converse, consider an arbitrary fact α ∈ T M ( D ) .
We next construct a rule r for which we prove r ∈ P M and α ∈ T r ( D ) .
Let G D be the ( Col , δ ) -graph obtained from D by the transformation in Deﬁnition 2, and let V D be the set of vertices of G D .
For each (cid:96) ∈ { 0 , . . .
, L } , let v (cid:96) be the vector labelling each v ∈ V D in the computation of M ( G D ) .
Moreover, let µ be a substitution that maps each variable to a distinct constant.
We next construct an atom H , a conjunction C , a substitution ν from the variables in C to cn ( D ) , a set U of fresh vertices (i.e., vertices not occurring in G D ) of the form u d and u d,e for d and e constants from the image of µ , and mappings ξ : U → V D and M c : U → 2 V D for each c ∈ Col .
We also assign each vertex in U to a level between 0 and L , and we identify a single vertex from U as the root vertex.
In the rest of this proof, we use letters a and b for the constants in D , letters d and e for the constants in the image of µ , possibly indexed letters v , w , and s for the vertices in V D , and possibly indexed letters u and t for the vertices in U .
Our construction is by induction from level L down to level 1 , and we prove in parallel the following properties:
P1.
for each vertex of the form u d ∈ U , for x the variable such that µ ( x ) =
d , and for a = ν ( x ) , we have ξ ( u d ) = v a ; P2.
for each vertex of the form u d,e ∈ U , for x and y variables such that µ ( x ) = d and µ ( y ) =
e , and for a = ν ( x ) and b = ν ( y ) , we have ξ ( u d,e ) = v a,b ; P3.
conjunction C satisﬁes ν ( C ) ⊆
D ; and P4.
if H is of the form A ( x ) , conjunction C is (3 k, L ) -tree-like for x ; otherwise it is of the form C x ∧ B x,y ∧ C y , where C x and C y are (3 k, L ) -tree-like conjunctions for x and y , respectively, and B x,y is a nonempty conjunction of atoms of the form S ( x, y ) or S ( y, x ) .
We initialise C as the empty conjunction, and we set ν , ξ , and each M c for c ∈ Col as the empty mappings.
For the induction base, we consider the form of the atom α ∈ T M ( D ) and proceed as follows; the base case also identiﬁes the root vertex.
Properties P1–P4 clearly hold after these steps
• If α is of the form A ( a ) , then V D contains vertex v a .
We introduce a fresh variable x , and we let d be the constant such that d = µ ( x ) .
We deﬁne ν ( x ) = a ; we deﬁne H =
A ( x ) ; we introduce vertex u d of level L ; we deﬁne ξ ( u d ) = v a ; and we make u d the root vertex.
Finally, we extend C with atom B ( x ) for each B ( a ) ∈ D .
Note that C can be empty if D does not contain a fact such as B ( a ) .
However, v a ∈ V D implies that constant a must occur in D in a fact of the form S ( a, b ) or S ( b, a ) .
Thus, N c ( v a ) (cid:54) = ∅ for c ∈ { c 1 , c 2 , c 4 } .
Since L ≥ 1 , vertex u d is processed in the inductive step below, which will extend C with an atom that contains variable x
If α is of the form R ( a, b ) , then V D contains vertices v a,b and v b,a ; due to the latter, D contains at least one fact of the form S ( a, b ) or S ( b, a ) , and so V D contains vertices v a and v b .
We introduce fresh variables x and y , and we let d and e be constants such that d = µ ( x ) and e = µ ( y ) .
We deﬁne ν ( x ) = a and ν ( y )
=
b ; we deﬁne H
= R ( x, y ) ; we introduce vertices u d , u e , u d,e , and u e,d of level L ; we deﬁne ξ ( u d ) = v a , ξ ( u e )
= v b , ξ ( u d,e ) = v a,b , and ξ ( u e,d ) = v b,a ; and we make u d,e the root vertex.
Moreover, we extend
C with an atom S ( x, y ) for each S ( a, b ) ∈ D , an atom S ( y, x ) for each S ( b, a ) ∈ D , an atom B ( x ) for each B ( a ) ∈ D , and an atom B ( y ) for each B ( b ) ∈ D ; note that, C cannot empty after this step.
Finally, we deﬁne M c ( u ) =
N c ( ξ ( u )) for each u ∈ { u d,e , u e,d } and colour c ∈ Col ; note that M c 1 ( u ) , M c 2 ( u ) , and M c 3 ( u ) are all singletons, and M c 4 ( u )
= ∅ .
Next, we consider each level (cid:96) with 1 ≤ (cid:96) ≤ L that has already been processed, and we consider each vertex of the form u d of level (cid:96) .
Let x be the variable such that µ ( x ) =
d , and let a = ν ( x ) ; property P1 ensures ξ ( u d ) = v a .
For each colour c ∈ { c 1 , c 2 , c 4 } , we choose a smallest set M c ( u d ) ⊆
N c ( v a ) such that, for each (cid:96) (cid:48) ∈ { 0 , . . .
, (cid:96) − 1 } and each i ∈ { 1 , . . .
, k } , there exists a vertex w ∈ M c ( u d ) satisfying
[Equation]
Note that set M c ( u d ) may not be unique, but any set satisfying (5) can be selected.
There are at most k distinct values of ( w (cid:96) (cid:48) )
i for each (cid:96) (cid:48) , so set M c ( u d ) contains at most k(cid:96) elements.
We also deﬁne M c 3 ( u d ) = ∅ .
We now consider each vertex v ∈ M c 1 ( u d ) ∪ M c 2 ( u d ) ∪ M c 4 ( u d ) .
By Definition 2, such v is of the form v a,b , v b,a , or v b for b ∈ cn ( D ) , depending on the set from which v is drawn.
Set V D clearly also contains vertices v b,a and v b if v ∈ M c 1 ( u d ) , and vertices v a,b and v b if v ∈ M c 2 ( u d ) .
Moreover, if v ∈ M c 4 ( u d ) , then V D clearly contains v b , and M c 4 ( u d ) ⊆
N c 4 ( v a ) ensures that D contains a fact of the form S ( a, b ) or S ( b, a ) ; thus, V D also contains v a,b and v b,a .
We introduce a fresh variable y , and we let e be the constant
such that e = µ ( y ) .
We deﬁne ν ( y )
=
b ; we introduce vertices u e , u d,e , and u e,d of level (cid:96) − 1 ; and we deﬁne ξ ( u e ) = v b , ξ ( u d,e )
= v a,b , and ξ ( u e,d )
= v b,a .
Moreover, we extend C with an atom S ( x, y ) for each S ( a, b ) ∈ D , an atom S ( y, x ) for each S ( b, a ) ∈ D , and an atom B ( y ) for each B ( b ) ∈ D .
Finally, we deﬁne M c ( u ) =
N c ( ξ ( u )) for each u ∈ { u d,e , u e,d } and colour c ∈ Col ; note that M c 1 ( u ) , M c 2 ( u ) , and M c 3 ( u ) are all singletons, and M c 4 ( u )
= ∅ .
Since each M c ( u d ) with c ∈ { c 1 , c 2 , c 4 } contains at most k(cid:96) vertices and we consider three colours, this step introduces at most 3 k(cid:96) fresh variables.
Moreover, note that each fresh variable y occurs in C in an atom of the form S ( x, y ) or S ( y, x ) .
Finally, it should be clear that P1–P4 hold after level (cid:96) is processed.
This completes our inductive construction.
At this point, either H =
A ( x ) and C is a nonempty (3 k, L ) -tree-like conjunction for x , or H =
R ( x, y ) and C =
C x ∧ B x,y ∧ C y where C x and C y are (3 k, L ) -tree-like conjunctions for x and y , respectively, and B x,y is a nonempty conjunction of atoms of the form S ( x, y ) or S ( y, x ) .
Thus, rule H ← C is of the form
A ( x ) ← C x or
R ( x, y ) ← C x ∧ B x,y ∧ C y .
To complete the proof, we next show this rule is captured by T M .
Then, Deﬁnition 9 ensures ensures H ← C ∈ P M ; together with Hν = α and ν ( C ) ⊆
D , we have α ∈ T P M ( D ) , which proves our claim.
We use Proposition 7 to show that H ← C is captured by T M .
To this end, let D C
= { αµ | α ∈ C } , and let G D C be the ( Col , δ ) -graph constructed using vertices in U obtained by applying the encoding from Deﬁnition 2 to dataset D C .
Finally, let u (cid:96) be the labelling associated with each vertex u ∈ U in the computation of M ( G D C ) .
The construction of C clearly ensures the following two properties for each colour c ∈ Col and each vertex u ∈ U :
R1.
for each vertex v ∈ M c ( u ) , there exists a vertex t ∈ N c ( u ) such
that ξ ( t ) = v ; and R2.
for each vertex t ∈ N c ( u ) in graph G D C , we have ξ ( t ) ∈ N c ( ξ ( u )) .
We now show by the induction on (cid:96) that, for each 0 ≤ (cid:96) ≤ L and each vertex u ∈ U whose level is at least (cid:96) , we have u (cid:96)
= v (cid:96) for v = ξ ( u ) .
For the base case (cid:96) = 0 , we have the following possibilities.
• Assume u is of the form u d .
Let x be the variable such that d = µ ( x ) , and let a
= ν ( x ) .
Property P1 ensures ξ ( u d ) = v a , and the construction of C ensures B ( x ) ∈ C
if and only if B ( a ) ∈ D ; thus, u 0 d = v 0 a holds.
• Assume u is of the form u d,e .
Let x
and y be variables such that µ ( x ) =
d and µ ( y ) =
e , and let a
= ν ( x ) and b = ν ( y ) .
Property P2 ensures ξ ( u d,e ) = v a,b , and the construction of C ensures S ( x, y ) ∈ C
if and only if S ( a, b ) ∈ D ; thus, u 0 d,e = v 0 a,b holds.
For the induction step, assume that the property holds for some (cid:96) − 1 , and consider an arbitrary vertex u ∈ U whose level is at least (cid:96) .
Let v = ξ ( u ) .
The induction assumption ensures u (cid:96) − 1 = v (cid:96) − 1 .
Also, for each colour c ∈ Col , our construction ensures that each vertex t ∈ N c ( u ) is of level at least (cid:96) − 1 , so the induction assumption ensures t (cid:96) − 1 = w (cid:96) − 1 for w = ξ ( t ) .
Finally, property R2 ensures that each t (cid:96) − 1 used to compute u (cid:96) is considered when computing v (cid:96) , and property R1 ensures that the maximal values of t (cid:96) − 1 are also present in the computation of u (cid:96) .
Thus, u (cid:96) = v (cid:96) holds.
To complete the proof, let u be the root vertex, and let v = ξ ( u ) .
The above property ensures u L = v L , and so cls ( u L ) =
cls ( v L ) as well.
Thus, we have µ ( H ) ∈ T M
( D C ) , so Propositionallows us to conclude that rule H ← C is captured by T M .
Please note that given a ( Col , δ ) -MGNN M , a dataset D , and a fact α ∈ T M ( D ) , the proof of
Theorem 10 shows how to identify a rule r ∈ P M such that α ∈ T r ( D ) .
A user interested in understanding why T M derived α from D can apply this procedure to extract a rule r that explains why α was derived.
Furthermore, a user can then search for shorter explanations by considering rules that subsume r and checking whether they are still captured by T M (using Proposition 7), and whether they still derive α from D .
D O PTIMISED RULE EXTRACTION ALGORITHM
In this section we describe the rule extraction algorithm that we have implemented and used to obtain the results in in Table 4 from Section 4.
The pseudocode for the algorithm is shown in Algorithm 1.
Our algorithm takes as input a trained model M and a number m and generates as output a list of (3 k, L ) -tree-like rules captured by T M with at most m atoms in the body, with k and L the dimension and number of layers of M respectively.
The algorithm relies on rule subsumption checks to ensure that redundant rules (i.e., logically implied by another rule) are not included in the output.
The algorithm starts by initialising an empty mapping head to bodies in Line 1.
This mapping will store pairs of the form ( S ( x, y ) , { C 1 , C 2 , . .
C n } ) , where each S ( x, y ) ← C i is a (3 k, L ) - tree-like rule captured by T M with no more than m body atoms.
In each iteration i of the loop in Lines 2–15, the algorithm will generate rules with exactly i atoms in the body.
To this end, the algorithm iterates over all ordered combinations of predicates with repetition of length i , which are generated by the auxiliary function C OMBINATIONS ( i ).
Then, in Line 4 , given the list of predicates pred list , the algorithm iterates over all conjunctions (up to variable renaming) that can be obtained by transforming each predicate in pred list into an atom, using a number of variables no greater than the size of pred list .
For example, if pred list is [ A, R ] , where A is unary and R is binary, then B ODIES ( pred list ) is
[Equation]
Lines 5 and 6 discard each conjunction that is not (3 k, L ) -tree-like, to avoid testing rules that do not have such form.
Lines 7-9 ground the conjunction body , apply T M to the result, and lift the resulting dataset back into a set of non-ground atoms.
In particular, E NCODE ( body ) grounds body by mapping each variable to a different constant, and then encodes the result into a ( Col , δ ) -labelledgraph as described in Section 2.
Function D
ECODE ( output graph ) performs the reverse operation,
Input m do 3: do 4: for body
B ODIES ( pred list do 5: if not T REE L IKE ( then 6: next ; 7: input graph ← E NCODE body ) 8: output graph ← M ( input
) 9: heads ← D ECODE ( output graph ) 10: for head do 11: body
) 12: for body
) do 13: if S then 14: break ; 15: head to add
) 16: extracted rules := [ ] 17: for ( head, bodies ) in head to bodies do 18: for in bodies do 19: and the result is a set heads of non-ground atoms, where each set element head is such that T M captures head ← body .
In Line 10, the algorithm iterates over each head in heads .
Line 11 rewrites the candidate rule head ← body to ensure that heads of extracted rules always use variables x and y in this order, and it makes the corresponding substitutions in the body of the candidate rule.
The result is rule new head ← new body .
Lines 12 and 14 check whether new head ← new body is subsumed by any rule previously stored in head to bodies .
If it is not, then Line 15 adds rule new head ← new body to head to bodies .
Speciﬁcally, conjunction new body is added to the set head to bodies ( new head ) .
Finally, Lines 16 to 19 process the mapping head to bodies and output all rules encoded in it.
Algorithm 1 satisﬁes the following property, which ensures its correctness.
Proposition D1.
Let E XTRACT
( M , m ) be the set of rules generated by Algorithm 1 for a ( Col , δ )
-MGNN M , of dimension k and L layers, and a number m ∈ N .
Then:
For every r ∈ E XTRACT ( M , m ) , r is a (3 k, L ) -tree-like rule r captured by T M with no more than m atoms in the body.
For every (3 k, L ) -tree-like rule r captured by T M with no more than m atoms in the body, there exists a rule in E XTRACT ( M , m ) that subsumes r (up to variable renaming).
Proof.
Item 1 is a direct consequence of Proposition 7 and the fact that Algorithm 1 only evaluates (3 k, L ) -tree-like rules with at most m atoms in the body.
To see that Item 2 holds, one can straightforwardly verify that if r is a (3 k, L ) -tree-like rule with at most m atoms in the body, then Lines 2of
Algorithm 1 check whether this rule is captured by T M ; if so, Lines 10-15 ensure that r is either in the output of the algorithm, or the output contains a rule that subsumes it. [Table]
In Section 4, we used Algorithm 1 with m = 2 to extract, for each benchmark, all the rules from the corresponding P M that have at most two atoms in the body and are not subsumed by another rule in the program.
Table 6 shows the classiﬁcation metrics obtained by using the extracted program to classify the positive and negative examples in the testing dataset of each benchmark.