T HE L OGICAL E XPRESSIVENESS OF G RAPH N EURAL N ETWORKS
A BSTRACT
The ability of graph neural networks (GNNs) for distinguishing nodes in graphs has been recently characterized in terms of the Weisfeiler-Lehman (WL) test for checking graph isomorphism.
This characterization, however, does not settle the issue of which Boolean node classiﬁers (i.e., functions classifying nodes in graphs as true or false) can be expressed by GNNs.
We tackle this problem by focusing on Boolean classiﬁers expressible as formulas in the logic FOC 2 , a well-studied fragment of ﬁrst order logic.
FOC 2 is tightly related to the WL test, and hence to
GNNs.
We start by studying a popular class of GNNs, which we call AC GNNs in which the features of each node in the graph are updated, in successive layers only in terms of the features of its neighbors.
We show that this class of GNNs is too weak to capture all FOC classiﬁers , and provide a syntactic characterization of the largest subclass of FOC classiﬁers that can be captured by
AC
-GNNs.
This subclass coincides with a logic heavily used by the knowledge representation community.
We then look at what needs to be added to AC GNNs for capturing all FOC 2 classiﬁers.
We show that it sufﬁces to add readout functions which allow to update the features of a node not only in terms of its neighbors, but also in terms of a global attribute vector.
We call GNNs of this kind ACR-GNNs
We experimentally validate our ﬁndings showing that on synthetic data conforming to FOC 2 formulas AC GNNs struggle to ﬁt the training data while ACR-GNNs can generalize even to graphs of sizes not seen during training.
I NTRODUCTION
Graph neural networks ( GNNs ) (Merkwirth & Lengauer, 2005; Scarselli et al., 2009) are a class of neural network architectures that has recently become popular for a wide range of applications dealing with structured data, e.g., molecule classiﬁcation, knowledge graph completion, and Web page ranking (Battaglia et al., 2018; Gilmer et al., 2017; Kipf & Welling, 2017; Schlichtkrull et al., 2018).
The main idea behind GNNs is that the connections between neurons are not arbitrary but reﬂect the structure of the input data.
This approach is motivated by convolutional and recurrent neural networks and generalize both of them (Battaglia et al., 2018).
Despite the fact that GNNs have recently been proven very efﬁcient in many applications, their theoretical properties are not yet well-understood.
In this paper we make a step towards understanding their expressive power by establishing connections between GNNs and well-known logical formalisms.
We believe these connections to be conceptually important, as they permit us to understand the inherently procedural behavior of some fragments of GNNs in terms of the more declarative ﬂavor of logical languages.
Two recent papers (Morris et al., 2019; Xu et al., 2019) have started exploring the theoretical properties of GNNs by establishing a close connection between GNNs and the Weisfeiler-Lehman ( WL ) test for checking graph isomorphism.
The WL test works by constructing a labeling of the nodes of the graph, in an incremental fashion, and then decides whether two graphs are isomorphic by comparing the labeling of each graph.
To state the connection between GNNs and this test, consider the simple GNN architecture that updates the feature vector of each graph node by combining it with the aggregation of the feature vectors of its neighbors.
We call such GNNs aggregate-combine GNNs , or AC-GNNs .
The authors of these papers independently observe that the node labeling produced by the WL test always reﬁnes the labeling produced by any GNN.
More precisely, if two nodes are labeled the same by the algorithm underlying the WL test, then the feature vectors of these nodes produced by any AC-GNN will always be the same.
Moreover, there are AC-GNNs that can reproduce the WL labeling, and hence AC-GNNs can be as powerful as the WL test for distinguishing nodes.
This does not imply, however, that AC-GNNs can capture every node classiﬁer —that is, a function assigning true or false to every node—that is reﬁned by the WL test.
In fact, it is not difﬁcult to see that there are many such classiﬁers that cannot be captured by AC-GNNs; one simple example is a classiﬁer assigning true to every node if and only if the graph has an isolated node.
Our work aims to answer the question of what are the node classiﬁers that can be captured by GNN architectures such as AC-GNNs.
To start answering this question, we propose to focus on logical classiﬁers —that is, on unary formulas expressible in ﬁrst order predicate logic (FO): such a formula classiﬁes each node v according to whether the formula holds for v or not.
This focus gives us an opportunity to link GNNs with declarative and well understood formalisms, and to establish conclusions about GNNs drawing upon the vast amount of work on logic.
For example, if one proves that two GNN architectures are captured with two logics, then one can immediately transfer all the knowledge about the relationships between those logics, such as equivalence or incomparability of expressiveness, to the GNN setting.
For AC-GNNs, a meaningful starting point to measure their expressive power is the logic FOC 2 , the two variable fragment of ﬁrst order predicate logic extended with counting quantiﬁers of the form ∃ ≥ N ϕ , which state that there are at least N nodes satisfying formula ϕ
(Cai et al., 1992).
Indeed, this choice of FOC 2 is justiﬁed by a classical result due to Cai et al.
(1992) establishing a tight connection between FOC 2 and WL: two nodes in a graph are classiﬁed the same by the WL test if and only if they satisfy exactly the same unary FOC 2 formulas.
Moreover, the counting capabilities of FOC 2 can be mimicked in FO (albeit with more than just two variables), hence FOC 2 classiﬁers are in fact logical classiﬁers according to our deﬁnition.
Given the connection between AC-GNNs and WL on the one hand, and that between WL and FOCon the other hand, one may be tempted to think that the expressivity of AC-GNNs coincides with that of FOC 2 .
However, the reality is not as simple, and there are many FOC 2 node classiﬁers (e.g., the trivial one above) that cannot be expressed by AC-GNNs.
This leaves us with the following natural questions.
First, what is the largest fragment of FOC 2 classiﬁers that can be captured by
AC-GNNs?
Second, is there an extension of AC-GNNs that allows to express all FOC 2 classiﬁers?
In this paper we provide answers to these two questions.
The following are our main contributions.
• We characterize exactly the fragment of FOC 2 formulas that can be expressed as ACGNNs.
This fragment corresponds to graded modal logic (de Rijke, 2000), or, equivalently, to the description logic ALCQ , which has received considerable attention in the knowledge representation community (Baader et al., 2003; Baader & Lutz, 2007).
• Next we extend the AC-GNN architecture in a very simple way by allowing global readouts , where in each layer we also compute a feature vector for the whole graph and combine it with local aggregations; we call these aggregate-combine-readout GNNs (ACR-GNNs).
These networks are a special case of the ones proposed by Battaglia et al.
(2018) for relational reasoning over graph representations.
In this setting, we prove that each FOCformula can be captured by an ACR-GNN
We experimentally validate our ﬁndings showing that the theoretical expressiveness of ACR-GNNs, as well as the differences between AC-GNNs and ACR-GNNs, can be observed when we learn from examples.
In particular, we show that on synthetic graph data conforming to FOC 2 formulas, ACGNNs struggle to ﬁt the training data while ACR-GNNs can generalize even to graphs of sizes not seen during training.
G RAPH N EURAL N ETWORKS
In this section we describe the architecture of AC-GNNs and introduce other related notions.
We concentrate on the problem of Boolean node classiﬁcation: given a (simple, undirected) graph
G =
( V, E ) in which each vertex v ∈ V has an associated feature vector x v , we wish to classify each graph node as true or false ; in this paper, we assume that these feature vectors are one-hot
The basic architecture for GNNs, and the one studied in recent studies on GNN expressibility (Morris et al., 2019; Xu et al., 2019), consists of a sequence of layers that combine the feature vectors of every node with the multiset of feature vectors of its neighbors.
Formally, let { AGG ( i ) } Li =1 and { COM ( i ) } Li =1 be two sets of aggregation and combination functions.
An aggregate-combine GNN (AC-GNN) computes vectors x ( i ) v for every node v of the graph G , via the recursive formula
[Equation]
where each x (0) v is the initial feature vector x v of v .
Finally, each node v of G is classiﬁed according to a Boolean classiﬁcation function CLS applied to x ( L ) v .
Thus, an AC-GNN with L layers is deﬁned as a tuple A = (cid:0) { AGG ( i ) } Li
=1 , { COM ( i ) } Li =1 , CLS (cid:1) , and we denote by A ( G, v ) the class (i.e., true or false ) assigned by A to each node v in G .
There are many possible aggregation, combination, and classiﬁcation functions, which produce different classes of GNNs (Hamilton et al., 2017; Kipf & Welling, 2017; Morris et al., 2019; Xu et al., 2019).
A simple, yet common choice is to consider the sum of the feature vectors as the aggregation function, and a combination function as
[Equation]
where C ( i ) and A ( i ) are matrices of parameters, b ( i ) is a bias vector, and f is a non-linearity function, such as relu or sigmoid.
We call simple an AC-GNN using these functions.
Furthermore, we say that an AC-GNN is homogeneous if all AGG ( i ) are the same and all COM ( i ) are the same (share the same parameters across layers).
In most of our positive results we construct simple and homogeneous GNNs, while our negative results hold in general (i.e., for GNNs with arbitrary aggregation, combining, and classiﬁcation functions).
The Weisfeiler-Lehman ( WL ) test is a powerful heuristic used to solve the graph isomorphism problem (Weisfeiler & Leman, 1968), or, for our purposes, to determine whether the neighborhoods of two nodes in a graph are structurally close or not.
Due to space limitations, we refer to (Cai et al., 1992) for a formal deﬁnition of the underlying algorithm, giving only its informal description: starting from a colored graph, the algorithm iteratively assigns, for a certain number of rounds , a new color to every node in the graph; this is done in such a way that the color of a node in each round has a one to one correspondence with its own color and the multiset of colors of its neighbors in the previous round.
An important observation is that the rounds of the WL algorithm can be seen as the layers of an AC-GNN whose aggregation and combination functions are all injective (Morris et al., 2019; Xu et al., 2019).
Furthermore, as the following proposition states, an AC-GNN classiﬁcation can never contradict the WL test.
Proposition 2.1 (Morris et al., 2019; Xu et al., 2019).
If the WL test assigns the same color to two nodes in a graph, then every AC-GNN classiﬁes either both nodes as true or both nodes as false .
C ONNECTION BETWEEN GNN S AND LOGIC 3. L OGICAL NODE CLASSIFIERS
Our study relates the power of GNNs to that of classiﬁers expressed in ﬁrst order (FO) predicate logic over (undirected) graphs where each vertex has a unique color (recall that we call these classiﬁers logical classiﬁers ).
To illustrate the idea of logical node classiﬁers, consider the formula
[Equation]
This formula has one free variable , x , which is not bounded by any quantiﬁer of the form ∃ or ∀ , and two quantiﬁed variables y and z .
In general, formulas with one free variable are evaluated over nodes of a given graph.
For example, the above formula evaluates to true exactly in those nodes v whose color is Red and that have both a Blue and a Green neighbor.
In this case, we say that node v of G satisﬁes α , and denote this by ( G, v ) | = α .
Formally, a logical (node) classiﬁer is given by a formula ϕ ( x ) in FO logic with exactly one free variable.
This formula classiﬁes as true those nodes v in G such that ( G, v ) | =
ϕ , while all other nodes (i.e., those with ( G, v ) (cid:54)| = ϕ ) are classiﬁed as false .
We say that a GNN classiﬁer captures a logical classiﬁer when both classiﬁers coincide over every node in every possible input graph.
Deﬁnition 3.1.
A GNN classiﬁer A captures a logical classiﬁer ϕ
( x ) if for every graph G and node v in G , it holds that A ( G, v ) =
true if and only if ( G, v ) | = ϕ .
3. L OGIC FOC
Logical classiﬁers are useful as a declarative formalism, but as we will see, they are too powerful to compare them to AC-GNNs.
Instead, for reasons we explain later we focus on classiﬁers given by formulas in FOC 2 , the fragment of FO logic that only allows formulas with two variables, but in turn permits to use counting quantiﬁers .
Let us brieﬂy introduce FOC 2 and explain why it is a restriction of FO logic.
The ﬁrst remark is that reducing the number of variables used in formulas drastically reduces their expressive power.
Consider for example the following FO formula expressing that x is a red node, and there is another node, y , that is not connected to x
and that has at least two blue neighbors, z 1 and z 2 : β ( x [Equation] =
[Equation]
(cid:3)(cid:1)
[Equation] The formula β ( x ) uses four variables, but it is possible to ﬁnd an equivalent one with just three: the trick is to reuse variable x and replace every occurrence of z 2 in β ( x ) by x .
However, this is as far as we can go with this trick: β ( x ) does not have an equivalent formula with less than three variables.
In the same way, the formula α ( x ) given in Equation (3) can be expressed using only two variables, x and y , simply by reusing y in place of z .
That being said, it is possible to extend the logic so that some node properties, such as the one deﬁned by β ( x ) , can be expressed with even less variables.
To this end, consider the counting quantiﬁer ∃ ≥ N for every positive integer N .
Analogously to how the quantiﬁer ∃ expresses the existence of a node satisfying a property, the quantiﬁer ∃ ≥ N expresses the existence of at least N different nodes satisfying a property.
For example, with ∃ ≥ 2 we can express β ( x ) by using only two variables by means of the classiﬁer
[Equation]
Based on this idea, the logic FOC 2 allows for formulas using all FO constructs and counting quantiﬁers, but restricted to only two variables.
Note that, in terms of their logical expressiveness, we have that FOC 2 is strictly less expressive than FO (as counting quantiﬁers can always be mimicked in FO by using more variables and disequalities), but is strictly more expressive than FO 2 , the fragment of
FO that allows formulas to use only two variables (as β ( x ) belongs to FOC 2 but not to FO 2 ).
The following result establishes a classical connection between FOC 2 and the WL test.
Together with Proposition 2.1, this provides a justiﬁcation for our choice of logic FOC 2 for measuring the expressiveness of AC-GNNs.
Proposition 3.2 (Cai et al., 1992).
For any graph G and nodes u, v in G , the WL test colors v and u the same after any number of rounds iff u and v are classiﬁed the same by all FOC 2 classiﬁers.
3. FOC 2 AND AC-GNN CLASSIFIERS
Having Propositions 2.1 and 3.2, one may be tempted to combine them and claim that every FOCclassiﬁer can be captured by an AC-GNN.
Yet, this is not the case as shown in Proposition 3.3 below.
In fact, while it is true that two nodes are declared indistinguishable by the WL test if and only if they are indistinguishable by all FOC 2 classiﬁers (Proposition 3.2), and if the former holds then such nodes cannot be distinguished by AC-GNNs (Proposition 2.1), this by no means tells us that
every
FOC 2 classiﬁer can be expressed as an AC-GNN.
Proposition 3.3.
There is an FOC 2 classiﬁer that is not captured by any AC-GNN.
One such FOC 2 classiﬁer is γ ( x ) in Equation (4), but there are inﬁnitely many and even simpler
FOC 2 formulas that cannot be captured by AC-GNNs.
Intuitively, the main problem is that an ACGNN has only a ﬁxed number L of layers and hence the information of local aggregations cannot travel further than at distance L of every node along edges in the graph.
For instance, the red node in γ ( x ) may be farther away than the node with the blue neighbours, which means that AC-GNNs would never be able to connect this information.
Actually, both nodes may even be in different connected components of a graph, in which case no number of layers would sufﬁce.
The negative result of Proposition 3.3 opens up the following important questions.
What kind of FOC 2 classiﬁers can be captured by AC-GNNs?
Can we capture FOC 2 classiﬁers with GNNs using a simple extension of AC-GNNs?
We provide answers to these questions in the next two sections.
T HE EXPRESSIVE POWER OF AC-GNN S
Towards answering our ﬁrst question, we recall that the problem with AC-GNN classiﬁers is that they are local, in the sense that they cannot see across a distance greater than their number of layers.
Thus, if we want to understand which logical classiﬁers this architecture is capable of expressing, we must consider logics built with similar limitations in mind.
And indeed, in this section we show that
AC-GNNs capture any FOC 2 classiﬁer as long as we further restrict the formulas so that they satisfy such a locality property.
This happens to be a well-known restriction of FOC 2 , and corresponds to graded modal logic (de Rijke, 2000) or, equivalently, to description logic ALCQ (Baader et al., 2003), which is fundamental for knowledge representation: for instance, the OWL 2 Web Ontology
Language (Motik et al., 2012; W3C OWL Working Group, 2012) relies on ALCQ .
The idea of graded modal logic is to force all subformulas to be guarded by the edge predicate E .
This means that one cannot express in graded modal logic arbitrary formulas of the form ∃ yϕ ( y ) , i.e., whether there is some node that satisﬁes property ϕ .
Instead, one is allowed to check whether some neighbor y of the node x where the formula is being evaluated satisﬁes ϕ .
That is, we are allowed to express the formula
∃ y
( E ( x, y )
∧ ϕ ( y )) in the logic as in this case ϕ ( y ) is guarded by
E ( x, y ) .
We can deﬁne this fragment of FO logic using FO syntax as follows.
A graded modal logic formula is either Col ( x ) , for Col a node color, or one of the following, where ϕ and ψ are graded modal logic formulas and N is a positive integer:
[Equation]
Notice then that the formula δ ( x ) :
= Red ( x ) ∧ ∃ y (cid:0) E ( x, y )
∧ Blue ( y ) (cid:1) is in graded modal logic, but the logical classiﬁer γ ( x ) in Equation (4) is not, because the use of ¬ E ( x, y ) as a guard is disallowed.
As required, we can now show that AC-GNNs can indeed capture all graded modal logic classiﬁers.
Proposition 4.1.
Each graded modal logic classiﬁer is captured by a simple homogeneous AC-GNN.
The key idea of the construction is that the vectors’ dimensions used by the AC-GNN to label nodes, represent the sub-formulas of the captured classiﬁer.
Thus, if a feature in a node is 1 then the node satisﬁes the corresponding sub-formula, and the opposite holds after evaluating L layers, where L is the “quantiﬁer depth” of the classiﬁer (which does not depend on the graph).
The construction uses simple, homogeneous AC-GNNs with the truncated relu non-linearity max(0 , min( x, 1)) .
The formal proof of Proposition 4.1, as well as other formal statements, can be found in the Appendix.
An interesting question that we leave as future work is to investigate whether the same kind of construction can be done with AC-GNNs using different aggregate and combine operators than the ones we consider here; for instance, using max instead of sum to aggregate the feature vectors of the neighbors, or using other non-linearity such as sigmoid, etc.
The relationship between AC-GNNs and graded modal logic goes further: we can show that graded modal logic is the “largest” class of logical classiﬁers captured by AC-GNNs.
This means that the only FO formulas that AC-GNNs are able to learn accurately are those in graded modal logic.
Theorem 4.2.
A logical classiﬁer is captured by AC-GNNs if and only if it can be expressed in graded modal logic.
The backward direction of this theorem is Proposition 4.1, while the proof of the forward direction is based on a recently communicated extension of deep results in ﬁnite model theory (Otto, 2019).
We point out that the forward direction holds no matter which aggregate and combine operators are considered, i.e., this is a limitation of the architecture for AC-GNNs, not of the speciﬁc functions that one chooses to update the features.
GNN S FOR CAPTURING FOC5. GNN S WITH GLOBAL READOUTS
In this section we tackle our second question: which kind of GNN architecture we need to capture all FOC 2 classiﬁers?
Recall that the main shortcoming of AC-GNNs for expressing such classiﬁers is their local behavior.
A natural way to break such a behavior is to allow for a global feature computation on each layer of the GNN.
This is called a global attribute computation in the framework of Battaglia et al.
Following the recent GNN literature (Gilmer et al., 2017; Morris et al., 2019; Xu et al., 2019), we refer to this global operation as a readout .
Formally, an aggregate-combine-readout GNN ( ACR-GNN ) extends AC-GNNs by specifying readout functions { READ ( i ) } Li =1 , which aggregate the current feature vectors of all the nodes in a graph.
Then, the vector x ( i ) v of each node v in G on each layer i , is computed by the following formula, generalizing Equation (1):
[Equation]
Intuitively, every layer in an ACR-GNN ﬁrst computes (i.e., “reads out”)
the aggregation over all the nodes in G ; then, for every node v , it computes the aggregation over the neighbors of v ; and ﬁnally it combines the features of v with the two aggregation vectors.
All the notions about ACGNNs extend to ACR-GNNs in a straightforward way; for example, a simple ACR-GNN uses the sum as the function READ ( i ) in each layer, and the combination function COM ( i ) ( x 1 , x 2 , x 3 ) =
f (cid:0) x 1 C ( i ) + x 2 A ( i ) + x 3 R ( i ) +
b ( i ) (cid:1) with a matrix R ( i ) , generalizing Equation (2).
5. ACR-GNN S AND FOC
To see how a readout function could help in capturing non-local properties, consider again the logical classiﬁer γ ( x ) in Equation (4), that assigns true to every red node v as long as there is another node not connected with v having two blue neighbors.
We have seen that AC-GNNs cannot capture this classiﬁer.
However, using a single readout plus local aggregations one can implement this classiﬁer as follows.
First, deﬁne by B the property “having at least 2 blue neighbors”.
Then an ACR-GNN that implements γ ( x ) can (1) use one aggregation to store in the local feature of every node if the node satisﬁes B , then (2) use a readout function to count how many nodes satisfying B exist in the whole graph, and (3) use another local aggregation to count how many neighbors of every node satisﬁy B .
Then γ is obtained by classifying as true every red node having less neighbors satisfying
B than the total number of nodes satisfying B in the whole graph.
It turns out that the usage of readout functions is enough to capture all non-local properties of FOC 2 classiﬁers.
Theorem 5.1.
Each FOC 2 classiﬁer can be captured by a simple homogeneous ACR-GNN.
The construction is similar to that of Proposition 4.1 and uses simple, homogeneous ACR-GNNs— that is, the readout function is just the sum of all the local node feature vectors.
Moreover, the readout functions are only used to deal with subformulas asserting the existence of a node that is not connected to the current node in the graph, just as we have done for classiﬁer γ ( x ) .
As an intermediate step in the proof, we use a characterization of FOC 2 using an extended version of graded modal logic, which was obtained by Lutz et al.
We leave as a challenging open problem whether FOC 2 classiﬁers are exactly the logical classiﬁers captured by ACR-GNNs.
5. C OMPARING THE NUMBER OF READOUT LAYERS
The proof of Theorem 5.1 constructs GNNs whose number of layers depends on the formula being captured—that is, readout functions are used unboundedly many times in ACR-GNNs for capturing different FOC 2 classiﬁers.
Given that a global computation can be costly, one might wonder whether this is really needed, or if it is possible to cope with all the complexity of such classiﬁers by performing only few readouts.
We next show that actually just one readout is enough.
However, this reduction in the number of readouts comes at the cost of severely complicating the resulting GNN.
Formally, an aggregate-combine GNN with ﬁnal readout (AC-FR-GNN) results out of using any number of layers as in the AC-GNN deﬁnition, together with a ﬁnal layer that uses a readout function, according to Equation (5).
Theorem 5.2.
Each FOC 2 classiﬁer is captured by an AC-FR-GNN.
The AC-FR-GNN in the proof of this theorem is not based on the idea of evaluating the formula incrementally along layers, as in the proofs of Proposition 4.1 and Theorem 5.1, and it is not simple (note that AC-FR-GNNs are never homogeneous).
Instead, it is based on a reﬁnement of the GIN architecture proposed by Xu et al.
(2019) to obtain as much information as possible about the local neighborhood in graphs, followed by a readout and combine functions that use this information to deal with non-local constructs in formulas.
The ﬁrst component we build is an AC-GNN that computes an invertible function mapping each node to a number representing its neighborhood (how big is this neighborhood depends on the classiﬁer to be captured).
This information is aggregated so that we know for each different type of a neighborhood how many times it appears in the graph.
We then use the combine function to evaluate FOC 2 formulas by decoding back the neighborhoods.
E XPERIMENTAL RESULTS
We perform experiments with synthetic data to empirically validate our results.
The motivation of this section is to show that the theoretical expressiveness of ACR-GNNs, as well as the differences between AC- and ACR-GNNs, can actually be observed when we learn from examples.
We perform two sets of experiments: experiments to show that ACR-GNNs can learn a very simple FOC 2 node classiﬁer that AC-GNNs cannot learn, and experiments involving complex FOC 2 classiﬁers that need more intermediate readouts to be learned.
We implemented our experiments in the PyTorch
Geometric library (Fey & Lenssen, 2019).
Besides testing simple AC-GNNs, we also tested the GIN network proposed by Xu et al.
(we consider the implementation by Fey & Lenssen (2019) and adapted it to classify nodes).
Our experiments use synthetic graphs, with ﬁve initial colors encoded as one-hot features, divided in three sets: train set with 5k graphs of size up to 50nodes, test set with 500 graphs of size similar to the train set, and another test set with 500 graphs of size bigger than the train set.
We tried several conﬁgurations for the aggregation, combination and readout functions, and report the accuracy on the best conﬁguration.
Accuracy in our experiments is computed as the total number of nodes correctly classiﬁed among all nodes in all the graphs in the dataset.
In every case we run up to 20 epochs with the Adam optimizer.
More details on the experimental setting, data, and code can be found in the Appendix.
We ﬁnally report results on a real benchmark (PPI) where we did not observe an improvement of ACR-GNNs over AC-GNNs.
Separating AC-GNNs and ACR-GNNs We consider a very simple FOC 2 formula deﬁned by α ( x ) :
= Red ( x ) ∧ ∃ y Blue ( y ) , which is satisﬁed by every red node in a graph provided that the graph contains at least one blue node.
We tested with line-shaped graphs and Erd¨os-Renyi (E-R) random graphs with different connectivities.
In every set (train and test)
we consider 50% of graphs not containing any blue node, and 50% containing at least one blue node (around 20% of nodes are in the true class in every set).
For both types of graphs, already single-layer ACR-GNNs showed perfect performance (ACR-1 in Table 1).
This was what we expected given the simplicity of the property being checked.
In contrast, AC-GNNs and GINs (shown in Table 1 as AC- L and GINL , representing AC-GNNs and GINs with L layers) struggle to ﬁt the data.
For the case of the line-shaped graph, they were not able to ﬁt the train data even by allowing 7 layers.
For the case of random graphs, the performance with 7 layers was considerably better.
In a closer look at the performance for different connectivities of E-R graphs, we found an improvement for AC-GNNs when we train them with more dense graphs (details in the Appendix).
This is consistent with the fact that AC-GNNs are able to move information of local aggregations to distances up to their
[Table] [Table]
number of layers.
This combined with the fact that random graphs that are more dense make the maximum distances between nodes shorter, may explain the boost in performance for AC-GNNs.
Complex FOC 2 properties
In the second experiment we consider classiﬁers
α i ( x ) constructed as
[Equation]
where ∃ [ N,M ] stands for “there exist between N and M nodes” satisfying a given property.
Observe that each α i ( x ) is in FOC 2 , as ∃ [ N,M ] can be expressed by combining ∃ ≥ N and ¬∃ ≥ M +1 .
We created datasets with E-R dense graphs and labeled them according to α 1 ( x ) , α 2 ( x ) , and α 3 ( x ) , ensuring in each case that approximately half of all nodes in our dataset satisfy every property.
Our experiments show that when increasing the depth of the formula (existential quantiﬁers with negations inside other existential quantiﬁers) more layers are needed to increase train and test accuracy (see Table 2).
We report ACR-GNNs performance up to 3 layers (ACR- L in Table 2) as beyond that we did not see any signiﬁcant improvement.
We also note that for the bigger test set, AC-GNNs and
GINs are unable to substantially depart from a trivial baseline of 50%.
We tested these networks with up to 10 layers but only report the best results on the bigger test set.
We also test AC-FR-GNNs with two and three layers (AC-FR- L in Table 2).
As we expected, although theoretically using a single readout gives the same expressive power as using several of them (Theorem 5.2), in practice more than a single readout can actually help the learning process of complex properties.
PPI We also tested AC- and ACR-GNNs on the Protein-Protein Interaction (PPI) benchmark (Zitnik & Leskovec, 2017).
We chose PPI since it is a node classiﬁcation benchmark with different graphs in the train set (as opposed to other popular benchmarks for node classiﬁcation such as Core or Citeseer that have a single graph).
Although the best results for both classes of GNNs on PPI were quite high (AC: 97.5 F1, ACR: 95.4 F1 in the test set), we did not observe an improvement when using ACR-GNNs.
Chen et al.
(2019) recently observed that commonly used benchmarks are inadequate for testing advanced GNN variants, and ACR-GNNs might be suffering from this fact.
F INAL REMARKS
Our results show the theoretical advantages of mixing local and global information when classifying nodes in a graph.
Recent works have also observed these advantages in practice, e.g., Deng et al.
(2018) use global-context aware local descriptors to classify objects in 3D point clouds, You et al.
(2019) construct node features by computing shortest-path distances to a set of distant anchor nodes, and Haonan et al.
(2019) introduced the idea of a “star node” that stores global information of the graph.
As mentioned before, our work is close in spirit to that of Xu et al.
and Morris et al.
(2019) establishing the correspondence between the WL test and GNNs.
In contrast to our work, they focus on graph classiﬁcation and do not consider the relationship with logical classiﬁers.
Regarding our results on the links between AC-GNNs and graded modal logic (Theorem 4.2), we point out that very recent work of Sato et al.
(2019) establishes close relationships between GNNs and certain classes of distributed local algorithms .
These in turn have been shown to have strong correspondences with modal logics (Hella et al., 2015).
Hence, variants of our Proposition 4.1 could be obtained by combining these two lines of work (but it is not clear if this combination would yield
AC-GNNs that are simple ).
However, these works do not investigate the impact of having non-local computations (such as the readouts that we consider), hence our results on the relationships between
FO an ACR-GNNs (Theorem 5.1 and 5.2) do not follow from these.
Morris et al.
(2019) also studied k -GNNs, which are inspired by the k -dimensional WL test.
In k -GNNs, graphs are considered as structures connecting k -tuples of nodes instead of just pairs of them.
We plan to study how our results on logical classiﬁers relate to k -GNNs, in particular, with respect to the logic FOC k that extends FOC 2 by allowing formulas with k variables, for each ﬁxed k > 1 .
Recent work has also explored the extraction of ﬁnite state representations from recurrent neural networks as a way of explaining them (Weiss et al., 2018; Koul et al., 2019; Oliva & LagoFern ´ andez, 2019).
We would like to study how our results can be applied for extracting logical formulas from GNNs as possible explanations for their computations. A CKNOWLEDGMENTS
This work was partly funded by the Millennium Institute for Foundational Research on Data 2 .
APPENDIX A P ROOF OF P ROPOSITION 3.
We ﬁrst recall the proposition.
Proposition 3.3.
There is an FOC 2 classiﬁer that is not captured by any AC-GNN.
Proof.
Consider the following FOC 2 node property α ( v ) :
=
Red ( v ) ∧ ∃ x Green ( x ) .
We will show by contradiction that there is no AC-GNN that captures α , no matter which aggregation, combining, and ﬁnal classiﬁcation functions are allowed.
Indeed, assume that A is an AC-GNN capturing α , and let L be its number of layers.
Consider the graph G that is a chain of L + 2 nodes colored Red, and consider the ﬁrst node v 0 in that chain.
Since A captures α , and since ( G, v 0 ) (cid:54)| =
α , we have that A labels v 0 with false , i.e., A ( G, v 0 )
= false .
Now, consider the graph G (cid:48) obtained from G by coloring the last node in the chain with Green (instead of Red).
Then one can easily show that A again labels v 0 by false in G (cid:48) .
But we have ( G (cid:48) , v 0 )
|
= α , a contradiction.
The above proof relies on the following weakness of AC-GNNs: if the number of layers is ﬁxed (i.e., does not depend on the input graph), then the information of the color of a node v cannot travel further than at distance L from v .
Nevertheless, we can show that the same holds even when we consider AC-GNNs that dispose of an arbitrary number of layers (for instance, one may want to run a homogeneous AC-GNN for f ( | E | ) layers for each graph G =
( V, E ) , for a ﬁxed function f ).
Assume again by way of contradiction that A is such an extended AC-GNN capturing α .
Consider the graph G consisting of two disconnected nodes v, u , with v colored Red and y colored Green.
Then, since ( G, v ) | = α , we have A ( G, v ) = true .
Now consider the graph G (cid:48) obtained from G by changing the color of u from Green to Red.
Observe that, since the two nodes are not connected, we will again have A ( G (cid:48) , v ) =
true , contradicting the fact that ( G (cid:48) , v ) (cid:54)| = α and that A is supposed to capture α .
By contrast, it is easy to see that this formula can be done with only one intermediate readout, using the technique in the proof of Theorem 5.1.
B P ROOF OF P ROPOSITION 4.
We ﬁrst recall the proposition.
Proposition 4.1.
Each graded modal logic classiﬁer is captured by a simple homogeneous AC-GNN.
We ﬁrst deﬁne formally the semantics of the graded modal logic (de Rijke, 2000) over simple undirected node-colored graphs (de Rijke, 2000), assuming the FO syntax introduced in the paper.
Deﬁnition B.1.
We deﬁne when a node v in a graph G satisﬁes a graded modal logic formula ϕ
( x ) , written as v | =
ϕ in G (where “in G ” may be omitted when clear), recursively as follows:
• if ϕ ( x ) =
Col ( x ) , then v | =
ϕ if and only if Col is the color of v in G , • if ϕ ( x ) =
ϕ (cid:48)
( x ) ∧ ϕ (cid:48)(cid:48)
( x ) , then v | =
ϕ if and only if v | = ϕ (cid:48) and v | =
ϕ (cid:48)(cid:48) , and similarly with ¬ ϕ (cid:48) ( x ) , and • if ϕ ( x ) = ∃ ≥ N ( E ( x, y ) ∧ ϕ (cid:48)
( y )) , then v | =
ϕ if and only if the set of nodes { u | u ∈ N G ( v ) and v | = ϕ (cid:48) } has cardinality at least N .
We can now proceed to the proof of the proposition.
Proof of Proposition 4.1.
Let ϕ ( x ) be a graded modal logic formula.
We will construct an ACGNN A ϕ that is further simple and homogeneous.
Let sub( ϕ ) =
( ϕ 1 , ϕ 2 , . . .
, ϕ L ) be an enumeration of the sub-formulas of ϕ such that if ϕ k is a subformula of ϕ (cid:96) then k ≤ (cid:96) .
The idea of the construction of A ϕ is to have feature vectors in R L such that every component of those vectors represents a different formula in sub( ϕ ) .
Then A ϕ will update the feature vector x
( i ) v of node v ensuring that component (cid:96) of x ( (cid:96) ) v gets a value 1 if and only if the formula ϕ (cid:96) is satisﬁed in node v .
We note that ϕ
= ϕ
L and thus, the last component of each feature vector after evaluating L layers in every node gets a value 1 if and only if the node satisﬁes ϕ .
We will then be able to use a ﬁnal classiﬁcation function CLS that simply extracts that particular component.
Formally, the simple homogeneous AC-GNN A ϕ has L layers and uses the aggregation and combine functions
[Equation]
where A , C ∈ R L × L , and b ∈ R L are deﬁned next, and σ is the truncated ReLU activation deﬁned by σ ( x ) =
min(max(0 , x ) , 1) .
The entries of the (cid:96) -th columns of A , C , and b depend on the sub-formulas of ϕ as follows:
Case 0 .
if ϕ (cid:96) ( x ) =
Col ( x ) with Col one of the (base) colors, then C (cid:96)(cid:96) = 1 , Case 1 .
if ϕ (cid:96) ( x ) =
ϕ j ( x ) ∧ ϕ k ( x ) then C j(cid:96) =
C k(cid:96)
= 1 and b (cid:96) = − 1 , Case 2 .
if ϕ (cid:96) ( x ) =
¬ ϕ k ( x ) then C k(cid:96)
= − 1 and b (cid:96) = 1 , Case 3 .
if ϕ (cid:96) ( x ) =
∃ ≥ N ( E ( x, y ) ∧ ϕ k ( y ))
then A k(cid:96)
= 1 and b (cid:96) =
− N + 1 , and all other values in the (cid:96) -th columns of A , C , and b are 0 .
We now prove that A ϕ indeed captures ϕ .
Let G =
( V, E ) be a colored graph.
For every node v in G we consider the initial feature vector x (0) v = ( x 1 , . . .
, x L ) such that x (cid:96) = 1 if sub-formula ϕ (cid:96) is the initial color assigned to v , and x (cid:96) = 0 otherwise.
By deﬁnition, AC-GNN A ϕ will iterate the aggregation and combine functions deﬁned above for L rounds ( L layers) to produce feature vectors x ( i ) v for every node v ∈ G and (cid:96) = 1 , . . .
, L as follows:
[Equation]
We next prove that for every ϕ (cid:96) ∈ sub( ϕ ) , every i ∈ { (cid:96), . . .
, L } , and every node v in G it holds that
[Equation]
where ( x ( i ) v ) (cid:96) is the (cid:96) -th component of x ( i ) v —that is, the (cid:96) -th component of x ( i ) v has a 1 if and only if v satisﬁes ϕ (cid:96) in G .
In the rest of the proof we will be continuously using the value of ( x ( i ) v ) (cid:96) whose general expression is
[Equation]
We proceed to prove (8) by induction on the number of sub-formulas of every ϕ (cid:96) .
If ϕ (cid:96) has one sub-formula, then ϕ (cid:96) ( x ) =
Col ( x ) with Col a base color.
We next prove that ( x (1) v ) (cid:96) =
if and only if v has Col as its initial color.
Since ϕ (cid:96) ( x ) =
Col ( x ) we know that C (cid:96)(cid:96)
= 1 and C k(cid:96)
=
for every k (cid:54) =
(cid:96) (see Case 0 above).
Moreover, we know that b (cid:96) =
and A k(cid:96)
=
for every k .
Then, from Equation (9) we obtain that
[Equation]
Then, given that ( x (0) v ) (cid:96) = 1 if the initial color of v is Col and ( x (0) v ) (cid:96) =
otherwise, we have that ( x (1) v ) (cid:96)
=
1 if ( G, v ) | = ϕ (cid:96) and ( x (1) v ) (cid:96) = 0 otherwise.
From this it is easy to prove that for every i ≥ 1 the vector ( x ( i ) v ) (cid:96) satisﬁes the same property.
Now assume that ϕ (cid:96) has more than one sub-formula, and assume that for every ϕ k with k < (cid:96) the property (8) holds.
Let i ≥ (cid:96) .
We are left to consider the following cases, corresponding to the cases for the shape of the formula above.
Case 1.
Assume that ϕ (cid:96) ( x ) =
ϕ j ( x ) ∧ ϕ k ( x ) .
Then C j(cid:96) =
C k(cid:96)
= 1 and b (cid:96) = − 1 .
Moreover, we have C m(cid:96)
=
for every m (cid:54) =
j, k and A n(cid:96)
=
for every n (see Case 2 above).
Then, from
Equation (9) we obtain that
[Equation]
Since the number of each proper sub-formula of ϕ (cid:96) is strictly less than both (cid:96) and i , by induction hypothesis we know that ( x ( i − 1) v ) j =
if and only if v | =
ϕ j and ( x ( i − 1) v )
j
=otherwise.
Similarly, ( x ( i − 1) v ) k
=
if and only if v | =
ϕ k and ( x ( i − 1) v ) k
= 0 otherwise.
Now, since ( x ( i ) v ) (cid:96) =
σ (( x ( i − 1) v )
j
+ ( x ( i − 1) v ) k − 1)
we have that ( x ( i ) v ) (cid:96)
=
if and only if ( x ( i − 1) v ) j
+( x ( i − 1) v ) k − 1 ≥ 1 that can only happen if ( x ( i − 1) v )
j = ( x ( i − 1) v ) k
= 1 .
Then ( x ( i ) v ) (cid:96) =
if and only if v | =
ϕ j and v | =
ϕ k —that is, if and only if v | =
ϕ (cid:96) (since ϕ (cid:96) ( x ) =
ϕ j ( x ) ∧ ϕ k ( x ) ), and ( x ( i ) v ) (cid:96) = 0 otherwise.
This is exactly what we wanted to prove.
Case 2.
Assume that ϕ (cid:96) ( x ) =
¬ ϕ k ( x ) .
Then C k(cid:96)
= − 1 and b (cid:96) = 1 .
Moreover, we have C m(cid:96)
=
for every m (cid:54) =
k and A n(cid:96)
=
for every n (see Case 2 above).
Then, from Equation (9) we obtain that
[Equation]
By induction hypothesis we know that ( x ( i − 1) v ) k
=
if and only if v | =
ϕ k and ( x ( i − 1) v ) k
= 0 otherwise.
Since ( x ( i ) v ) (cid:96) =
σ ( − ( x ( i − 1) v ) k +1)
we have that ( x ( i ) v ) (cid:96) =
if and only if 1 − ( x ( i − 1) v ) k
≥that can only happen if ( x ( i − 1) v ) k = 0 .
Then ( x ( i ) v ) (cid:96) =
if and only if v (cid:54)| = ϕ
k —that is, if and only if v | = ¬ ϕ k , which holds if and only if v | = ϕ (cid:96) , and ( x ( i ) v ) (cid:96) = 0 otherwise.
This is exactly what we wanted to prove.
Case 3.
Assume that ϕ (cid:96) ( x ) = ∃ ≥ N ( E ( x, y )
∧ ϕ k
( y )) .
Then A k(cid:96)
= 1 and b (cid:96) =
− N + 1 .
Moreover for every m we have that C m(cid:96)
=
(see Case 3 above).
Then, from Equation (9) we obtain that
[Equation]
x
[Equation]
i
[Equation]
[Equation]
x
[Equation]
i [Equation] [Equation] By induction hypothesis we know that ( x ( i − 1) u ) k
=
if and only if v | =
ϕ k and ( x ( i − 1) u ) k
=otherwise.
Then we can write ( x ( i ) v ) (cid:96) =
σ ( − N + 1 + m ) where m = |{ u | u ∈ N ( v ) and u | = ϕ k }| .
Thus, we have that ( x ( i ) v ) (cid:96) =
if and only if m ≥ N , that is if and only if there exists at least
N nodes connected with v that satisfy ϕ k , and ( x ( i ) v ) (cid:96) =
otherwise.
From that we obtain that ( x ( i ) v ) (cid:96) =
if and only if v | = ϕ (cid:96) since ϕ (cid:96) ( x ) =
∃ ≥ N ( E ( x, y ) ∧ ϕ k ( y )) , which is what we wanted to prove.
To complete the proof we only need to add a ﬁnal classiﬁcation after the L iterations of the aggregate and combine layers that simply classiﬁes a node v as true if the component of x ( L ) v corresponding to ϕ holds 1 .
C P ROOF OF T HEOREM 4.
We ﬁrst recall the theorem.
Theorem 4.2.
A logical classiﬁer is captured by AC-GNNs if and only if it can be expressed in graded modal logic.
Note that one direction follows immediately from Proposition 4.1, so we only need to show the following proposition.
Proposition C.1.
If a logical classiﬁer α is not equivalent to any graded modal logic formula, then there is no AC-GNN that captures α .
To prove this proposition, we will need the following deﬁnition, which is standard in modal logics theory.
Deﬁnition C.2.
Let G be a graph (simple, undirected and node-colored), v be a node in G , and L ∈
N .
The unravelling of v in G at depth L , denoted by Unr LG ( v ) , is the (simple undirected nodecolored) graph that is the tree having
– a node ( v, u 1 , . . .
, u i ) for each path ( v, u 1 , . . .
, u i ) in G with i ≤ L , – an edge between ( v, u 1 , . . .
, u i − 1 ) and ( v, u 1 , . . .
, u i ) when { u i − 1 , u i } is an edge in G (assuming that u 0 is v ), and – each node ( v, u 1 , . . .
, u i ) colored the same as u i in G .
We then observe the following.
Observation C.3.
Let G and G (cid:48) be two graphs, and v and v (cid:48) be two nodes in G and G (cid:48) , respectively.
Then for every L ∈ N , the WL test assigns the same color to v and v (cid:48) at round L if and only if there is an isomorphism between Unr LG ( v ) and Unr LG (cid:48)
( v (cid:48) ) sending v to v (cid:48) .
We will write Unr LG ( v ) (cid:39) Unr LG (cid:48)
( v (cid:48) ) to denote the existence of the isomorphism as in this observation.
To prove Proposition C.1,
we ﬁrst rephrase Proposition 2.1 in terms of unravellings.
Proposition C.4.
Let G and G (cid:48) be two graphs with nodes v in G and v (cid:48) in G (cid:48)
such that
Unr LG ( v ) (cid:39) Unr LG (cid:48) ( v (cid:48) ) for every L ∈ N .
Then for any AC-GNN A , we have A ( G, u )
=
A ( G (cid:48) , u (cid:48) ) .
Proof.
Follows directly from Proposition 2.1 and Observation C.3.
The crucial part of the proof of Proposition C.1 is the following non-trivial result, intuitively establishing that the fragment of unary FO formulas that only depend on the unravelling of a node is exactly the graded modal logic.
Theorem C.5 (Otto, 2019).
Let α be a unary FO formula.
If α is not equivalent to a graded modal logic formula then there exist two graphs G , G (cid:48) and two nodes v in G and u (cid:48) in G (cid:48) such that Unr LG ( v ) (cid:39) Unr LG (cid:48)
( v (cid:48) ) for every L ∈ N and such that u | = α in G but u (cid:48)
(cid:54)| = α in G (cid:48) .
Proof.
This directly follows from the van Benthem & Rosen characterization obtained in (Otto, 2019, Theorem 2.2) for ﬁnite structures (graphs), by noticing that for the notion of graded bisimulation ∼ # introduced in this note, we have that G, u ∼ # G (cid:48) , u (cid:48)
if and only if we have that Unr L G ( v ) (cid:39)
Unr LG (cid:48)
( v (cid:48) ) for every L ∈ N .
We point out here that the fact that the edge relation in G is undirected in our setting (as opposed to E being directed in (Otto, 2019)), and the fact that every node can only have one color in our setting (as opposed to being able to satisfy multiple “unary predicates” in (Otto, 2019)) are inessential, and that the proof of (Otto, 2019, Theorem 2.2) carries over to this setting.
We can now gather all of these to prove Proposition C.1.
Proof of Proposition C.1.
Let α be a logical classiﬁer (i.e., a unary FO formula) that is not equivalent to any graded modal logic formula.
Assume for a contradiction that there exists an AC-GNN A α that captures α .
Since α is not equivalent to any graded modal logic formula, by Theorem C.5 there exist two graphs G , G (cid:48) and two nodes v in G and u (cid:48) in G (cid:48) such that Unr LG ( v ) (cid:39) Unr LG (cid:48) ( v (cid:48) ) for every L ∈ N and such that ( (cid:63) )
u |
= α in G but u (cid:48)
(cid:54)| = α in G (cid:48) .
Since we have that Unr LG ( v ) (cid:39) Unr LG (cid:48)
( v (cid:48) ) for every L ∈ N , by Proposition C.4 we should have that A α ( G, u )
=
A α ( G (cid:48) , u (cid:48) ) .
But this contradicts ( (cid:63) ) and the fact that A α is supposed to capture α .
D P ROOF OF T HEOREM 5.
We ﬁrst recall the theorem.
Theorem 5.1.
Each FOC 2 classiﬁer can be captured by a simple homogeneous ACR-GNN.
To prove the theorem, we will use a characterization of the unary FOC 2 formulas provided by (Lutz et al., 2001) that uses a speciﬁc modal logic.
That logic is deﬁned via what are called modal parameters .
We adapt the deﬁnitions of (Lutz et al., 2001) to deal with simple undirected node-colored graphs.
Deﬁnition D.1.
A modal parameter is an expression built from the following grammar:
[Equation]
Given an undirected colored graph G =
( V, E ) and a node v of G , the interpretation of S on v is the set ε S ( v ) ⊆
V deﬁned inductively as follows:
– if S = id then ε S ( v ) :
= { v } ; – if S = e then ε S ( v ) :=
{ u | { u, v } ∈ E } ; – if S = S 1 ∪ S 2 then ε S ( v ) :
= ε S 1 ( v ) ∪ ε S 2 ( v ) ; – if S = S 1 ∩ S 2 then ε S ( v ) :
= ε S 1 ( v ) ∩ ε S 2 ( v ) ; – if S = ¬ S (cid:48)
then ε S ( v ) :
= V \ ε S ( v ) .
The modal logic EMLC consists of all the unary formulas that are built with the following grammar:
[Equation]
where C ranges over node colors, S over modal parameters, and N over N .
The semantics of the ﬁrst four constructs is deﬁned as expected, and for an undirected colored graph G =
( V, E ) and node v ∈ V , we have ( G, v ) | =
(cid:104) S (cid:105) ≥ N ϕ
if and only if there exist at least N nodes u in ε S ( v ) such that ( G, u ) | = ϕ .
Example D.2.
On an undirected graph G =
( V, E ) , the EMLC formula (cid:104)¬ e (cid:105) ≥ 2 ( (cid:104) e (cid:105) ≥ 3 Green ) holds on a node v ∈ V if v has at least two nonadjacent nodes u (and since our graphs have no self-loops, v could be u ) such that u has at least three green neighbors.
The following theorem is essentially a reformulation of (Lutz et al., 2001, Theorem 1) to our context (Lutz et al.
(2001) show this for FO 2 without counting quantiﬁers and for EMLC without counting, but an inspection of the proofs reveals that the result extends to counting quantiﬁers).
Theorem D.3 (Lutz et al., 2001, Theorem 1).
For every EMLC formula, there exists an equivalent FOC 2 unary formula.
Conversely, for every unary FOC 2 formula, there exists an equivalent
EMLC formula.
In order to simplify the proof, we will use the following lemma.
Lemma D.4.
Let ϕ be an EMLC formula.
Then there exists an EMLC formula ϕ (cid:48) equivalent to ϕ such that each modal parameter appearing in ϕ (cid:48) is one of the following:
a) id , thus representing the current node; b) e , thus representing the neighbours of the current node; c) ¬ e ∩ ¬ id , thus representing the nodes distinct from the current node and that are not neighbours of the current node; d) id ∪ e , thus representing the current node and its neighbors; e) ¬ id , thus representing all the nodes distinct from the current node: f)
¬ e , thus representing the nodes that are not neighbours of the current node (note that this includes the current node);
g) e ∪ ¬ e , thus representing all the nodes; h) e ∩ ¬ e , thus representing the emptyset.
Proof.
Let v be a node in a graph G , and consider the following three disjoint sets of nodes:
the singleton set consisting of v itself, 2.
the set of neighbors of v , 3.
the set of nodes that are not neighbors of v and that are not v .
These sets can be expressed by modal parameters: the ﬁrst is obtained by taking S
= i
d ; the second is obtained by taking S = e ; and the third is obtained by taking S =
¬
e ∩ ¬ id .
It is straightforward to verify by induction on S that, for any modal parameter S , if ε S ( v ) contains an element of one of the three sets, then it must contain all the elements of that set.
But then, this implies that a modal parameter can only represent a (possibly empty) disjoint union of these three sets.
Conversely, it is clear that any disjoint union over these three sets can be represented by a modal parameter.
It is then routine to check that the 8 cases (a)–(h) are obtained as all the 2 3 possible unions of these three sets (including the empty union, i.e., the emptyset).
For instance, case (f) is the union of sets 1 and 3.
Proof of Theorem 5.1.
The proof is similar to that of Proposition 4.1.
Let ϕ be an EMLC formula equivalent to the targeted FOC 2 unary formula that is of the form given by Lemma D.4, and let sub( ϕ ) =
( ϕ 1 , ϕ 2 , . . .
, ϕ L ) be an enumeration of the sub-formulas of ϕ such that if ϕ k is a subformula of ϕ (cid:96) then k ≤ (cid:96) .
We will build a simple homogeneous ACR-GNN A ϕ computing feature vectors x ( i ) v in R L such that every component of those vectors represents a different formula in sub( ϕ ) .
In addition, we will also make use of global feature vectors x
( i ) G in R L .
The GNN A ϕ will update the feature vector x ( i ) v of each node v in a graph ensuring that component (cid:96) of x ( i ) v gets a value 1 if and only if the formula ϕ (cid:96) is satisﬁed in node v (and 0 otherwise).
Similarly, x ( i ) G will be updated to make sure that every component represents the number of nodes in G that satisfy the corresponding subformula.
The readout and aggregate functions simply sum the input feature vectors.
When ϕ (cid:96) is of the form described by Cases 0–3 in the proof of Proposition 4.1, we deﬁne the (cid:96) -th columns of the matrices
A , C and bias b as in that proof, and the (cid:96) -th column of R (the matrix that multiplies the global readout feature vector) as the zero vector.
We now explain how we deﬁne their (cid:96) -th columns when ϕ (cid:96) is of the form (cid:104) S (cid:105) ≥ N ϕ k , according to the 8 cases given by Lemma D.4:
Case a. if ϕ (cid:96) =
(cid:104) i
d (cid:105) ≥ N ϕ k , then C k(cid:96)
=
if N = 1 and 0 otherwise; Case b. if ϕ (cid:96) =
(cid:104) e (cid:105) ≥ N ϕ k , then A k(cid:96)
= 1 and b (cid:96) =
− N + 1 ; Case c. if ϕ (cid:96) = (cid:104)¬ e ∩ ¬ id (cid:105) ≥ N ϕ k , then R k(cid:96)
= 1 and C k(cid:96)
=
A k(cid:96)
= − 1 and b (cid:96) =
− N + 1 ; Case d. if ϕ (cid:96) =
(cid:104) i
d ∪ e (cid:105) ≥ N ϕ k , then C k(cid:96)
= 1 and A k(cid:96)
= 1 and b (cid:96) =
− N + 1 ; Case e. if ϕ (cid:96) = (cid:104)¬ id (cid:105) ≥ N ϕ k , then R k(cid:96)
= 1 and C k(cid:96)
= − 1 and b (cid:96) =
− N + 1 ; Case f. if ϕ (cid:96) =
(cid:104)¬ e (cid:105) ≥ N ϕ k , then R k(cid:96)
= 1 and A k(cid:96)
= − 1 and b (cid:96) =
− N + 1 ; Case g. if ϕ (cid:96) =
(cid:104)
e ∪ ¬ e (cid:105) ≥ N ϕ k , then R k(cid:96)
= 1 and b (cid:96) =
− N + 1 ; Case h. if ϕ (cid:96) =
(cid:104) e ∩ ¬ e (cid:105) ≥ N ϕ k , then all
relevant values
are 0;
and all other values in the (cid:96) -th columns of A , C , R , and b are 0 .
The proof then goes along the same lines as the proof of Proposition 4.1.
E P ROOF OF T HEOREM 5.
We ﬁrst recall the theorem.
Theorem 5.2.
Each FOC 2 classiﬁer is captured by an AC-FR-GNN.
In the following proof we will use the machinery introduced in Appendices C and D. We will also make use of a particular AC-GNN with L layers, which we call A L primes , that maps every node v in a graph G to a natural number representing the complete unravelling of v of depth L in G (note that we do not claim that this AC-GNN can be realized in practice, this construction is mostly for theoretical purposes).
Let primes : N → N be the function such that primes( i ) is the i -th prime number indexed from 0 .
For instance, we have that primes(0)
= 2 , primes(1) = 3 , etc.
Now consider the function f( · , · ) that has as input a pair ( c, X ) where c ∈ N and X is a multiset of numbers in N , and produces a number in N as output, deﬁned as follows
[Equation]
It is not difﬁcult to prove that, as deﬁned above, f( · , · ) is an injective function.
Thus using the results by Xu et al.
(2019) (see the proof of their Theorem 3) we know that f can be used to implement the combine and aggregate operators of an AC-GNN such that for every graph G , after L layers, the color (natural number) assigned to every node in G has a one to one correspondence with the color assigned to that node in the L -th iteration of the WL test over G .
We call this AC-GNN A L primes .
Observation E.1.
We note that Xu et al.
(2019) also constructed an injective function that has ( c, X ) as inputs where c ∈ N and X is a multiset of elements in N (see their Lemma 5 and Corollary 6).
Nevertheless we cannot directly use that construction as it assumes the existence of a ﬁxed N such that the size of all multisets are bounded by N .
This would put also a bound of N on the maximum number of neighbors in the input graphs.
Thus we developed a new function (using an encoding based on prime numbers) to be able to deal with general graphs of unbounded degree.
Proof of Theorem 5.2.
Let α be an FOC 2 unary formula, and let ϕ be an equivalent EMLC formula that uses only modal parameters of the form given by Lemma D.4.
We construct an ACR-FR-GNN
A ϕ capturing ϕ
and hence α .
Let L be the quantiﬁer depth of ϕ (i.e., the deepest nesting of (cid:104) S (cid:105) ≥ N quantiﬁers).
For a subformula ϕ (cid:48) of ϕ , we also deﬁne the nesting depth nd ϕ
( ϕ (cid:48) ) of ϕ (cid:48) in ϕ to be the number of modal parameters under which ϕ (cid:48) is in ϕ .
The ﬁrst L − 1 layers of A ϕ are the same as those of A L − 1 primes , which do not use readouts.
With Observation C.3 at hand and using the fact that the inverses of the aggregation and combination functions of A L − 1 primes are computable, this ensures that, after L − 1 layers, for any graph G and node v in G , we can compute from A L − 1 primes ( G, v ) the unravelling Unr L − 1 G ( v ) .
Thus, we can assume without loss of generality (by modifying the last combination function for instance), that after L − 1 layers A ϕ computes Unr L − 1 G ( v ) in every node v of G .
We then use a readout whose output is a natural number representing the multiset {{ Unr L − 1 G ( v ) | v node in G }} ; for instance, we can encode this multiset using the same technique that we use for A primes .
Again, since this technique uses functions with computable inverses, we can assume without loss of generality that the output of this readout is actually the multiset {{ Unr L − 1 G ( v ) | v node in G }} .
Finally, we use a ﬁnal combination function COM ( L ) , that uses only the feature of the current node and the output of the readout—that is, the ﬁnal feature of a node v is COM ( L ) (Unr L − 1 G ( v ) , {{ Unr L − 1 G ( u ) | u node in G }} ) .
We now explain how we deﬁne COM ( L ) .
By induction on the structure of ϕ , for every subformula ϕ (cid:48) of ϕ , we do the following: for every node v in G and every node u in Unr L − 1 G ( v ) that is at depth (i.e., the distance from v )
at most nd ϕ
( ϕ (cid:48) ) in the tree
Unr L − 1 G ( v ) , we will label u by either ϕ (cid:48) or by ¬ ϕ (cid:48) .
We do so to ensure that ( (cid:63) ) for every node v in G and every node u
= ( v, u 1 , . . .
, u i ) in Unr L − 1 G ( v ) , we label u by ϕ (cid:48)
if and only if ( G, u i ) | = ϕ (cid:48) .
We explain our labeling process by induction on the structure of ϕ , and one can easily check in each case that ( (cid:63) ) will hold by induction.
Let v be a node in G and u be a node in Unr L − 1 G ( v ) that is at depth at most nd ϕ
( ϕ (cid:48) ) in the unravelling.
Case 1.
If ϕ (cid:48) is a color Col, we label u by ϕ (cid:48) if u is of that color, and by ¬ ϕ (cid:48) otherwise.
Case 2.
If ϕ (cid:48) is ϕ 1 ∧ ϕ 2 , then observe that we have nd ϕ ( ϕ (cid:48) )
=
nd ϕ ( ϕ 1 ) =
nd ϕ ( ϕ 2 ) , so that u is at depth at most both nd ϕ ( ϕ 1 ) and nd ϕ
ϕ 2 ) in the unravelling Unr L − 1 ( v ) .
Thus, we know that we have already labeled u by either ϕ 1 or ¬ ϕ 1 , and also by either ϕ 2 or ¬ ϕ 2 .
We then label u by ϕ (cid:48) if u is already labeled by ϕ 1 and ϕ 2 , and we label it by ¬ ϕ (cid:48) otherwise.
Case 3.
The case when ϕ (cid:48) is a negation is similar.
Case 4.
If ϕ (cid:48) is (cid:104) S (cid:105) ≥ N ϕ (cid:48)(cid:48) , then we only explain the case when the modal parameter S is ¬ e ∧ ¬ id , as the other cases work similarly.
First, observe that for every node v (cid:48) in G , we have labeled the root of Unr L − 1 G ( v (cid:48) ) by either ϕ (cid:48)(cid:48) or by ¬ ϕ (cid:48)(cid:48) : this is because the root of Unr L − 1 G ( v (cid:48) ) is always at depth 0 ≤ nd ϕ ( ϕ (cid:48)(cid:48) ) in Unr L − 1 G ( v (cid:48) ) .
Let m be the number of nodes u (cid:
) ∈
G such that we have labeled the root of Unr L − 1 G ( v (cid:48) ) by ϕ (cid:48)(cid:48) .
Next, note that for every children u (cid:48) of u in Unr L − 1 G ( v ) , we have that u (cid:48) is at depth
at most nd ϕ
( ϕ (cid:48)(cid:48) ) in Unr L − 1 G ( v ) , so that we have already labeled u (cid:48) by either ϕ (cid:48)(cid:48) or ¬ ϕ (cid:48)(cid:48) .
Let n be the number of children of u (in Unr L − 1 G ( v ) ) that we have labeled by ϕ (cid:48)(cid:48) .
Then we label u by ϕ (cid:48) if m − n ≥ N , and by ¬ ϕ (cid:48) otherwise.
We then simply deﬁne COM ( L ) (Unr L − 1 G ( v ) , {{ Unr L − 1 G ( u ) | u node in G }} ) to be 1 if the root of Unr L − 1 G ( v ) is labeled with ϕ , and 0 otherwise, which concludes the proof.
F D ETAILS ON THE EXPERIMENTAL SETTING AND RESULTS
All our code and data can be accessed online anonymously at https://anonymous.4open.
science/r/787222e2-ad5e-4810-a788-e80f0fe7eff0/
In all our experiments we tested different aggregate, combine and readout functions.
For aggregate and readout we only consider the sum, average, and max functions.
For the combine function we consider the following variants:
• COM 1 ( x , y , z ) =
f ( xA + yB + zC + b ) , • COM 2 ( x , y , z ) =
f ( MLP 1 ( x ) + MLP 2 ( y )
+ MLP 3 ( z ) + b ) , • COM 3 ( x , y , z ) =
MLP ( x + y + z + b ) , • COM 4 ( x , y , z ) =
MLP ( xA + yB + zC
+
b ) .
The above deﬁnitions are for ACR-GNNs.
For AC-GNNs we consider similar variants but without the z input.
We also used batch normalization in between every GNN and MLP layer.
We did not use any regularization.
When processing synthetic data we use a hidden size of 64 and trained with a batch-size of 128, and the Adam optimizer with PyTorch default parameters for 50 epochs.
We did not do any hyperparameter search besides changing the aggregation, combination, and readout functions.
For the activation functions we always used relu.
We observed a consistent pattern in which sum aggregator and readout produced better results compared with the others.
This is in line with our constructions in Proposition 4.1 and Theorem 5.1.
The choice of the combination function did not produce a signiﬁcant difference in the performance.
D ATA FOR THE EXPERIMENT WITH CLASSIFIER α ( x ) := R ED ( x ) ∧ ∃ y B LUE ( y )
For training and testing we constructed three sets of graphs: (a) Train set containing 5k graphs with nodes between 50 and 100, (b) Test set, same size, containing 500 graphs with the same number of nodes as in the train set (between 50 and 100 nodes), and (c) Test set, bigger size, containing 500 graphs with nodes between 100 and 200.
All graphs contain up to 5 different colors.
To force the models to try to learn the formula, in every set (train and test)
we consider 50% of graphs not containing any blue node, and 50% containing at least one blue node.
The number of blue nodes in every graph is ﬁxed to a small number (typically less than 5 nodes).
Moreover, to ensure that there is a signiﬁcant number of nodes satisfying the formula, we force graphs to contain at least 1/4 of its nodes colored with red .
The colors of all the other nodes are distributed randomly.
With all these restrictions, every dataset that we created had at least a 18% of nodes satisfying the property.
We consider two classes of graphs: line graphs and Erd¨os-Renyi graphs .
Line graphs these are connected graphs in which every node in the graph has degree 2 except for two nodes (the extreme nodes) that have degree 1.
To mimic the impossibility proof in Proposition 3.3 we put the blue nodes in one of the “sides” of the line, and the red nodes in the other “side”.
More speciﬁcally, consider the line graph with N nodes v 1 , . .
. , v N such that v i is connected with v i +1 .
Then, we ensure that every blue node appears in one of v 1 , . . .
, v N 2 and every red node appears in one of v N 2 +1 , . .
. , v N .
[Table]
[Table]
[Table]
Erd¨os-Renyi graphs These are random graphs in which one speciﬁes the number N of nodes and the number M of edges.
For this experiment we consider as extreme cases the case in which graphs contain the same number of nodes and edges and graphs in which the number of edges is twice the number of nodes.
Some statistics of the datasets are shown in Table 3.
E XPERIMENTS FOR DENSE E RD ¨ OS -R ENYI GRAPHS
We also took a closer look at the performance for different connectivities of random graphs (Table 4).
We deﬁne the set “Erd ¨ os-Renyi + k %” as a set of graphs in which the number of edges is k % larger than the number of nodes.
For example, “Erd ¨ os-Renyi + 100 %” contains random graphs in which the number of egdes doubles the number of nodes.
We see a consistent improvement in the performance of AC-GNNs and GINs when we train and test them with more dense graphs and more layers (Table 4).
D ATA FOR THE EXPERIMENT WITH CLASSIFIER α i ( x ) IN E QUATION (6)
For this case we only consider dense Erd¨os-Renyi synthetic graphs.
For the train set we consider graphs with nodes varying from 40 to 50 nodes and edges from 280 to 350 and similarly for the ﬁrst test set.
For the bigger test set, we consider graphs with nodes from 51 to 60 with edges ranging from 360 and 480 .
For labeling we consider the following formulas (starting from α 0 ( x ) :
=
Blue ( x ) ):
[Equation]
The choices of the intervals for every classiﬁer were for the pourpose of having approximately half of the nodes in the random graphs marked as true .
Statistics of the datasets are shown in Table 5.
[Table] [Table]
PPI E XPERIMENTS
We consider the standard train/validation/test split for this benchmarck (Fey & Lenssen, 2019).
We use a hidden size of 256 and the Adam optimizer for 500 epochs with early stopping when the validation set did not improve for 20 epochs.
We did not do any hyperparameter search besides changing the aggregation, combination
[Table]
and readout functions.
As opposed to the synthetic case, in this case we observed a better performance when the average or the max functions are used for aggregation.
Table 6 shows the best results for different layers (average of 10 runs).
As we can see,
ACR-GNNs do not imply an improvement over AC-GNNs for this benchmark
[Table]