G RAPH L OG : A B ENCHMARK FOR M EASURING L OGICAL G ENERALIZATION IN G RAPH N EURAL N ET -
WORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Relational inductive biases have a key role in building learning agents that can generalize and reason in a compositional manner.
While relational learning algorithms such as graph neural networks (GNNs) show promise, we do not understand their effectiveness to adapt to new tasks.
In this work, we study the logical generalization capabilities of GNNs by designing a benchmark suite grounded in ﬁrst-order logic.
Our benchmark suite, GraphLog, requires that learning algorithms perform rule induction in different synthetic logics, represented as knowledge graphs.
GraphLog consists of relation prediction tasks on 57 distinct procedurally generated logical worlds.
We use GraphLog to evaluate GNNs in three different setups: single-task supervised learning, multi-task (with pre-training), and continual learning.
Unlike previous benchmarks, GraphLog enables us to precisely control the logical relationship between the different worlds by controlling the underlying ﬁrst-order logic rules.
We ﬁnd that models’ ability to generalize and adapt strongly correlates to the availability of diverse sets of logical rules during multi-task training.
We also ﬁnd the severe catastrophic forgetting effect in continual learning scenarios, and
GraphLog provides a precise mechanism to control the distribution shift.
Overall, our results highlight new challenges for the design of GNN models, opening up an exciting area of research in generalization using graph-structured data.
I NTRODUCTION
Relational reasoning, or the ability to reason about the relationship between objects and entities, is considered one of the fundamental aspects of intelligence (Krawczyk et al., 2011; Halford et al., 2010), and is known to play a critical role in cognitive growth of children (Son et al., 2011; Farrington-Flint et al., 2007; Richland et al., 2010).
This ability to infer relations between objects/entities/situations, and to compose relations into higher-order relations, is one of the reasons why humans quickly learn how to solve new tasks (Holyoak and Morrison, 2012; Alexander, 2016).
The perceived importance of relational reasoning for generalization has fueled the development of several neural network architectures that incorporate relational inductive biases (Battaglia et al., 2016;
Santoro et al., 2017;
Battaglia et al., 2018).
Graph neural networks (GNNs), in particular, have emerged as a dominant computational paradigm within this growing area (Scarselli et al., 2008; Hamilton et al., 2017a;
Gilmer et al., 2017; Schlichtkrull et al., 2018; Du et al., 2019).
However, despite the growing interest in GNNs and their promise of relational generalization, we currently lack an understanding of how effectively these models can adapt and generalize across distinct tasks.
In this work, we study the task of logical generalization in the context of relational reasoning using
GNNs.
One example of such a reasoning task (from everyday life) can be in the context of a familygraph where the nodes are family members, and edges represent the relationships (brother, father, etc).
The objective is to learn logical rules, which are compositions of a speciﬁc form, such as “the son of my son is my grandson ”.
As new compositions of existing relations are discovered during the lifetime of a learner, (e.g., the son of my daughter is my grandson ), it should remember the old compositions, even as it learns new compositions (just like we humans do).
This simplistic example can be extended to the more complex (and yet practical) scenarios like identifying novel chemical compositions, or recommender systems, where agents have to learn and retain compositions of existing relations.
[Table]
Number of relations
[Table]
[Figure]
We study the effect of such generalization by analyzing the ability of GNNs in learning new relation compositions, leveraging ﬁrst-order logic.
In particular, we study how GNNs can induce logical rules and generalize by combining such rules in novel ways after training.
We propose a benchmark suite,
GraphLog, that is grounded in ﬁrst-order logic.
Figure 1 shows the setup of the benchmark.
Given a set of logical rules, we create a diverse set of logical worlds with a different subset of rules.
For each world (say W k ), we sample multiple knowledge graphs (say g ki ).
The learning agent should learn to induce the logical rules for predicting the missing facts in these knowledge graphs.
Using our benchmark, we evaluate the generalization capabilities of GNNs in the supervised setting by predicting inductively unseen combinations of known rules within a speciﬁc logical world .
We further analyze how various GNN architectures perform in the multi-task and continual learning scenarios, where they have to learn over a set of logical worlds with different underlying logics.
Our setup allows us to control the distribution shift by controlling the similarity between the different worlds , in terms of the overlap in logical rules between different worlds .
… r 2 ∧ r 3 ⟹ r 1 … …
Wr 4 ∧ r 2 ⟹ r 3 … …
Wr 1 ∧ r 2 ⟹ r… W n r 2 ∧ r 3 ⟹
rr 4 ∧ r 5 ⟹
rr 1 ∧ r 2 ⟹ r… … … …
g 1 i g
i g ni
r 4 ∧ r 2 ⟹
rr 4 ∧ r 5 ⟹
rr 1 ∧ r
[Figure]
r
[Figure]
(grounded in propositional logic) is
par-
titioned into overlapping subsets.
It is used to deﬁne unique worlds .
Within each world W k , several knowledge graphs g ki (governed by rule set of W k ) are generated.
Our analysis provides the following insights about logical generalization capabilities of GNNs:
• Two architecture choices for GNNs strongly (and positively) impact generalization: 1) incorporating multirelational edge features using attention, and 2) modularising GNN architecture to include a parametric representation function , to learn representations for the relations using a knowledge graph structure.
In the multi-task setting, training a model on a more diverse set of logical worlds improves generalization and adaptation performance.
• All the evaluated models exhibit catastrophic forgetting.
This indicates that the models are not learning transferable representations and compositions and just overﬁtting to the current task —highlighting the challenge of lifelong learning in context of logical generalization and GNNs.
B ACKGROUND AND R ELATED W ORK
GNNs .
Several GNN architectures have been proposed to learn representations of the graph inputs (Scarselli et al., 2008;
Duvenaud et al., 2015; Defferrard et al., 2016; Kipf and Welling, 2016;
Gilmer et al., 2017; Hamilton et al., 2017b; Schlichtkrull et al., 2018).
Previous works have focused on evaluating GNNs in terms of their expressive power (Barcel´o et al., 2019;
Morris et al., 2019; Xu et al., 2018), usefulness of features (Chen et al., 2019), and explaining their predictions (Ying et al., 2019).
Complementing these works, we evaluate GNN models on the task of logical generalization.
Knowledge graph completion .
Many knowledge graph datasets are available for relation prediction tasks (also known as knowledge base completion).
Prominent examples include Freebase15K (Bordes et al., 2013), WordNet (Miller, 1995), NELL (Mitchell and Fredkin, 2014), and YAGO (Suchanek et al., 2007;
Hoffart et al., 2011; Mahdisoltani et al., 2013).
Since these datasets are derived from real-world knowledge graphs, they are generally noisy and incomplete, and many facts are not available in the underlying knowledge bases (West et al., 2014; Paulheim, 2017).
Moreover, the underlying logical rules are often opaque and implicit (Guo et al., 2016), thus reducing the utility of these datasets for understanding the logical generalization capability of neural networks.
Procedurally generated datasets for reasoning .
Several procedurally generated benchmarks have been proposed to study the relational reasoning and compositional generalization properties of neural networks (Table 1).
These datasets provide a controlled testbed for evaluating compositional reasoning capabilities of neural networks in isolation.
Based on these insightful works, we enumerate the four key desiderata that, we believe, such a benchmark should provide:
Human interpretable rules should be used to generate the dataset.
The datasets should be diverse , and the compositional rules used to solve different tasks should be distinct, so that adaptation on a new task is not trivial.
The degree of similarity across the tasks should be conﬁgurable to evaluate the role of diversity in generalization.
The benchmark should test for compositional generalization , which is the ability to solve unseen combinations of existing rules, thus generalizing through the composition of known concepts.
The benchmark should support creating a large number of tasks and enable a more ﬁne-grained inspection of the generalization capabilities of the model in different setups, e.g., supervised learning, multi-task learning, and continual learning.
As shown in Table 1, GraphLog is unique in satisfying all of these desiderata.
We highlight that unlike previous works which have been largely focused on the image and text modalities, GraphLog is one of the unique attempts to test logical generalization on graph-structured data using Graph Neural Networks.
CLUTRR (Sinha et al., 2019) also provides similar underlying graphs in the text corpus, although GraphLog is different based on the following properties: • CLUTRR consists of a single rule world which in turn contains only 15 rules.
GraphLog extends the concept to a more general framework where a user can deﬁne how many rules can occur in a single world, as well as deﬁne multiple such worlds. • GraphLog allows building multiple worlds consisting of either the same, overlapping, or distinct set of rules - which allows practitioners to test multi-task and continual learning scenarios in minute detail by controlling the distribution shift, which is a key difference with CLUTRR. CLUTRR is built on a static set of relations (22 family relations) while GraphLog can contain any number of such relations since it’s a ﬂexible generator along with a dataset. Synthetic Graph Generation .
Synthetic graph generation is extensively studied using various neural generative approaches for scale-free random graphs (You et al., 2018; Liao et al., 2019). Unlike these approaches, GraphLog is built using a procedural graph generator, which allows us to control the logical validity of the query graphs, which is deﬁned in Section 3. Control over logical validity is easier in synthetically generated datasets, which are used extensively in Inductive Logic Programming (ILP) / Rule Learning context (Cornelio and Thost, 2019) 1 .
GraphLog on the other hand, is targeted towards Graph Neural Network evaluation, which also supports for richer distribution shift evaluation through Multi-task learning and Continual learning scenarios.
G RAPH L OG
Background and Terminology .
We leverage terminology and deﬁnitions from both knowledge graph and logic programming literature.
A graph G =
( V G , E G ) is a collection of a set of nodes V G and a set of edges E G between the nodes.
We assume that the graphs are relational , meaning that each edge between two nodes (say u and v ) is assigned a label , and can be denoted as r i ( u, v )
∈ E G .
A relation set R is a set of relations { r 1 , r 2 , ...
r K } .
In logic programming terms, nodes in the graph correspond to constants , and edges correspond to ground atoms .
Thus, as a slight abuse and mixture of notations, existence of an edge in the graph, i.e., r i ( u, v ) ∈
E G , is equivalent to background assumption that ground atom is true (i.e., that r i ( u, v ) ← ).
We deﬁne a rule set R as set of dyadic deﬁnite Datalog clauses (Abiteboul et al., 1995) of form:
[Equation]
Note that Equation 1 is also termed as chain-rule in logic literature. Following standard convention, we use upper-case to denote variables (e.g., the variables in Equation 1 can be substituted for nodes) and lower-case for constants (e.g., to denote nodes v ∈ V G ).
The relations r i , r j form the body while the relation r k forms the head of the rule.
Path-based Horn clauses of this form represent a limited and well-deﬁned subset of ﬁrst-order logic.
They encompass the types of logical rules learned by state-of-the-art rule induction systems for knowledge graph completion (Das et al., 2017; Evans and Grefenstette, 2017; Meilicke et al., 2018; Sadeghian et al., 2019; Teru et al., 2020; Yang et al., 2017;
Zhang et al., 2019) and are thus a useful synthetic test-bed.
We use p u,vG to denote a path from node u to v in a graph G (i.e., a sequence unique nodes connected by edges).
In logical terms, p u,vG corresponds to the conjunction of all the edge predicates in the path.
We construct graphs according to rules of the form in Equation 1 so that a path between two nodes will always imply a speciﬁc relation between these two nodes.
In other words, if we let P G denote the set of all ﬁnite paths in the graph G , then we have that
[Equation]
Thus, by following the path between two nodes, and applying rules of the form of Equationaccording to the edges of the path, we can always resolve the relationship between the nodes.
Problem Setup .
Given this setup, the task is to predict the relation between a pair of nodes.
In particular, we deﬁne a query ( g, u, v ) as follows: (i) the query subgraph g ⊂ G is a subgraph of the full graph G ; (ii) the query nodes are present in the query subgraph (i.e., u, v ∈ V g ); (iii) the edge between the query nodes is not present in the query subgraph, but the edge is present in the underlying graph (i.e., r ? ( u, v ) ∈
E G , r ?
( u, v ) / ∈ E g ) .
The goal is to infer the missing edge between query nodes r ?
( u, v ) .
We assume that the learning agent has access to a set of training queries g 1 , ..., g n ⊂ G to optimize a prediction model before being tested on a set of held-out queries g n +1 , ..., g n + n (cid:48) ⊂ G .
In knowledge graph terms, this is a form of link or relation prediction, as our goal is to infer the missing edge between two nodes.
Unlike most work on knowledge graph completion, we emphasize an inductive problem setup, where the query graph g in each training and testing example is unique, requiring generalization to unseen graphs (Teru et al., 2020).
Finally, note that we will sometimes refer to the full graphs G as world graphs , in order to distinguish them from the query graphs g ⊂ G .
We make this distinction because GraphLog involves link prediction over several different logical worlds , denoted W =
( G W , R W ) , with each deﬁned with by its own underlying world graph G W and rule set R W .
Our setup enables controlling the overlap between the rules in these different logical worlds, allowing for a unique ability to quantify the logical generalization capabilities of a learning agent.
3. D ATASET G ENERATION
As discussed in Section 2, we want our proposed benchmark to provide four key desiderata: (i) interpretable rules, (ii) diversity, (iii) compositional generalization and (iv) large number of tasks.
We describe how our dataset generation process ensures all four aspects.
Rule generation .
We create a set R of possible relations and use it to create a randomly generated rule set R , such two key constraints are satisﬁed: (i) No two rules in R can have the same body, ensuring that Equation 2 is satisﬁed.
(ii) Rules cannot have common relations among the head and body , ensuring the absence of cyclic dependencies in rules.
Generating the dataset using a consistent and well-deﬁned rule set ensures interpretability in the resulting dataset.
The full algorithm for rule generation is given in Appendix (Algorithm 1).
Graph generation .
The graph generation process has two steps: In the ﬁrst step, we sample overlapping subsets of R W ⊂ R to create individual worlds (as shown in Figure 1).
In each world, we recursively sample and use rules in R W to generate the world graph G W for that world.
This sampling procedure creates a diverse set of world graphs by considering only certain subsets of R .
By controlling the extent of overlap between the subsets of R (in terms of the number of common rules), we can precisely control the similarity between the different worlds.
The full algorithm for generating the world graphs and controlling the similarity between the worlds is given in Appendix (Algorithm 3 and Section A.2).
In the second step, the world graph G W in each world is used to sample a set of query graphs
G SW =
( g 1 , · · · g N ) for that speciﬁc world (shown as Step (a) in Figure 2).
A query graph g i is sampled from G W by sampling a pair of nodes ( u, v ) from G W and then by sampling a path p u,vG W .
The edge r i ( u, v ) between the source and sink node of the path provides the target relation for the learning model to predict.
To increase the complexity of the sampled g i graphs (beyond being simple paths), we also add nodes to g i by sampling neighbors of the nodes on p u,vG W , such that no other shorter path exists between u and v .
Algorithm 4 (in the Appendix) details our graph sampling approach.
Note that this sampling process leads to rich and complex graph structures, including cycles and motifs (see Figure 7(a) in Appendix for example of graphs).
GraphLog Dataset Suite .
We use the above data generation process to instantiate a dataset suite with 57 distinct logical worlds with 5000 query graphs per world (Figure 1).
The dataset is split into train, validation, and test worlds .
The query graphs within each world are also split into train, validation, and test sets (Table 2).
Though we instantiate 57 worlds , the GraphLog generator can instantiate an arbitrary number of worlds and has been included in the supplementary material.
3. S ETUPS SUPPORTED IN G RAPH L OG
Supervised learning .
A model is trained (and evaluated) on the train (and test split) of a world .
The number of rules grows exponentially with the number of relations K , making it impossible to train on all possible combinations of the relations.
We expect that a perfectly systematic model inductively generalizes to unseen combinations of relations by training only on a subset of combinations.
Multi-task learning .
GraphLog provides multiple logical worlds , with their training and evaluation splits.
A model is trained on (and evaluated on) the train (and test) splits of several worlds ( W 1 , · · · , W M ) .
GraphLog enables us to control the complexity of each world and similarity between the worlds to evaluate how model performance varies across similar vs. dissimilar worlds .
GraphLog is designed to study the effect of pre-training on adaptation.
In this setup, the model is pre-trained on the train split of multiple worlds ( W 1 , · · · , W M ) and ﬁne-tuned on the train split of the unseen heldout worlds ( W M +1 , · · · , W N ) .
The model is evaluated on the test split of the heldout worlds .
GraphLog enables mimicking in-distribution and out-of-distribution train (and test) scenarios and quantify the effect of multi-task pre-training for adaptation performance.
Continual learning .
The model is trained on a sequence of worlds .
Before training on a new world , it is evaluated on all the worlds that it has trained on so far.
Given the challenges involved in continual learning (Thrun and Pratt, 2012; Parisi et al., 2019; De Lange et al., 2019; Sodhani et al., 2019), we do not expect the models to perform well on the previous tasks.
Nonetheless, given that we are evaluating the models for relational reasoning and our datasets share relations , we expect the models to retain some knowledge of previous tasks.
We use the performance on the previous tasks to infer if the models learn to solve the relational reasoning tasks or just ﬁt to the current dataset distribution. Controlling similarity .
An unique feature of GraphLog is it allows practitioners to control the distribution shift when evaluating multi-task and continual learning baselines.
In GraphLog the diversity of worlds can be controlled via similarity. Concretely, the similarity between two worlds W i and W j is deﬁned as Sim ( W i , W j ) = |R i ∩ R j | , where W i and W j are the graph worlds and R i and R j are the set of rules associated with them. Thus GraphLog enables various training scenarios - training on highly similar worlds or training on a mix of similar and dissimilar worlds.
This ﬁne grained control allows GraphLog to mimic both in-distribution and out-of-distribution scenarios - during training and testing.
It also enables us to precisely categorize the effect of multi-task pre-training when the model needs to adapt to novel worlds.
M ODELS
In this section, we describe the message passing graph neural networks that we evaluate on GraphLog benchmark.
These models take as input the query ( g i , u, v ) and a learnable relation embedding r i ∈ R d (Step (d) and (e) in Figure 2), and output the relation as a k-way softmax (Step (f) in Figure 2).
We use both attention-based and non-attention based models.
Relational Graph Convolutional Network (RGCN) .
For a relational graph, the RGCN model (Schlichtkrull et al., 2018) is a natural choice for a baseline architecture.
In this approach, we iterate a series of message passing operations: h
[Equation]
, where h ( t ) u ∈ R d denotes representation of a node u at the t th layer of the model, T ∈ R
d r × d × d is a learnable tensor, r ∈ R d is relation representation, and N r i ( u ) denotes the neighbors of u (related by r i ).
× j , j = 1 , 2 , 3 denotes multiplication across the modes of the tensor.
RGCN model learns a relation-speciﬁc propagation matrix, speciﬁed by the interaction between relation embedding r i and shared tensor T .
Edge-based Graph Attention Network (Edge-GAT) .
Many recent works have highlighted the importance of the attention mechanism, especially in the context of relational reasoning (Vaswani et al., 2017; Santoro et al., 2018;
Schlag et al., 2019).
Motivated by this observation, we investigate an extended version of the GAT (Veliˇckovi´c et al., 2017), in order to handle the edge types.
We complement GAT by incorporating gating via an LSTM (Hochreiter and Schmidhuber, 1997) and where the attention is conditioned on both the incoming message (from the other nodes) and the relation embedding (of the other nodes): m N ( u ) =
(cid:80) r i ∈ R (cid:80) v ∈N ri ( u ) α (cid:16) h ( t − 1) u , h ( t − 1) v , r (cid:17) , h ( t ) u =
LSTM ( m N ( u ) , h ( t − 1) u ) .
Following the original GAT model, the attention function α uses concatenated input vectors.
We refer to this model as the Edge GAT ( E-GAT ) model.
Query and node representations .
We predict the relation for query ( g i , u, v ) by concatenating h ( K ) u , h ( K ) v (ﬁnal-layer query node embeddings) and applying a feedforward network (Step (f) in
Figure 2).
In absence of node features, we randomly initialize all the node embeddings in the GNNs.
[Figure] [Figure] [Equation] [Figure]
[Equation]
[Figure] [Equation] [Figure] Representation functions GNN models described above expect to learn d -dimensional relation representations.
We can further re-structure them notationwise into two modules: a) Representation module which is represented as a function f r : G W × R → R d , that maps logical relations within a particular world W to d -dimensional vector representations, and b) Composition module , which is a function f c : G W × V G W × V G W × R d ×| R | → R , that learns to compose the relation representations learned by f r and answer queries over a knowledge graph.
In this formulation, the relation embedding matrix r i ∈ R d used in RGCN and Edge-GAT can be thought of the output of a direct parameterization module ( Param ), which is devoid of any input and simply returns the embeddings from a lookup table.
Thus these two baselines can be redeﬁned as Param-RGCN and
Param-E-GAT models, where RGCN and E-GAT are the composition modules.
A major limitation of this approach is that the relation representations are optimized speciﬁcally for each logical world, and there is no inductive bias towards learning representations that can generalize.
Learning representations from the graph structure.
To deﬁne a more powerful representation function, we consider learning relation representations as a function of the graph G W underlying a logical world W . We consider an “extended” form of G W , [Table] [Figure]
[Table]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] denoted ˆ G W , which introduces new nodes (called edge-nodes ) corresponding to each edge in the original world graph G W . For an edge r ( u, v ) ∈ E G , the corresponding edge-node ( u − r − v ) is connected to only those nodes that were incident to it in the original graph (i e. nodes u and v ; see Figure 2, Step (b)).
This new graph ˆ G W only has one type of edge and comprises of nodes from both the original graph and from the set of edge-nodes.
We learn the relation representations by training a GNN model on ˆ G W and by averaging the edge-node embeddings corresponding to each relation type r i ∈ R .
(Step (c) in Figure 2).
For the GNN model, we consider the Graph Convolutional Network ( GCN ) (Kipf and Welling, 2016) and the Graph Attention Network ( GAT ) architectures.
The intuition behind creating the extended-graph is that the representation GNN function can learn the relation embeddings based on the structure of the complete relational graph G W .
Both the representation module and the composition modules are trained end-to-end to predict the relations.
Thus, we end up with four more models to evaluate GraphLog , namely GCN-RGCN , GCN-E-GAT , GAT-RGCN and
GAT-E-GAT .
E XPERIMENTS
We aim to quantify the performance of the different GNN models on the task of logical relation reasoning, in three setups: (i) Single Task Learning, (ii) Multi-Task Training and (iii) Continual
Learning.
Our experiments use the GraphLog benchmark with distinct 57 worlds (Section 3) anddifferent GNN models (Section 4).
In the main paper, we highlight the key results and provide the full results in the Appendix.
The code is included with the supplemental material.
5. S INGLE T ASK S UPERVISED L EARNING
We evaluate the models on all the 57 worlds , one model- world pair at a time (342 combinations).
With
GraphLog , it is difﬁcult for one model to outperform others on all the 57 datasets just by exploiting dataset-speciﬁc bias, thereby making the conclusions more robust.
We categorize the worlds in three categories of difﬁculty — easy , moderate and difﬁcult —based on the relative test performance of the models on each world and present the results in Figure 3.
(A complimentary formulation of difﬁculty is explained in Appendix A.4).
We observe that the models using E-GAT composition function always outperform the models using RGCN .
This conﬁrms our hypothesis about leveraging attention to improve the performance on relational reasoning tasks.
Interestingly, the relative ordering among the worlds , in terms of the test accuracy of the different models, is consistent irrespective of the model we use, highlighting the intrinsic ordering of difﬁculty of the different worlds in GraphLog.
5. M ULTI -T ASK T RAINING
Standard multi-task training .
First, we evaluate how changing similarity among training worlds affects test performance in the multi-task setup.
We train a model on eight worlds and test on
[Figure]
[Figure]
three distinct worlds.
In Table
[Figure] we observe that mix of similar and dissimilar worlds improves generalization capabilities of all the models.
Similar to the trends observed in the single task setup, the GAT-EGAT model consistently performs either as good as or better than other models.
The models using EGAT for the composition function perform better than the ones using the RGCN model.
Figure 4 shows how the models’ performance changes when we perform multi-task training on an increasingly large set of worlds .
Interestingly, we see that model performance improves as the number of worlds is increased from 10 to 20 but then begins to decline, indicating model capacity saturation in the presence of too many diverse worlds .
Multi-task pre
training .
In this setup, we pre-train the model on multiple worlds and adapt on a heldout world .
We study how the models’ adaption capabilities vary as the similarity between the training and the evaluation distributions changes.
Figure 5 considers the case of zero-shot adaptation and adaptation till convergence.
As we move along the x-axis , the zero-shot performance (shown with solid colors) decreases in all the setups.
This is expected as the similarity between the training and the evaluation distributions also decreases.
An interesting trend is that the model’s performance, after adaptation, increases as the similarity between the two distributions decreases.
This suggests that training over a diverse set of distributions improves adaptation capability.
5. C ONTINUAL L EARNING S ETUP
We evaluate knowledge retention capability by training GNNs on a sequence of worlds (arranged by relative similarity).
After convergence on any world , we report average of model’s performance on all previous worlds .
Figure 6(a) shows the rapid degradation in model’s performance on the previous worlds as it trains on new worlds , highlighting the limitation of current models for continual learning.
The role of the representation function .
We also investigate the model’s performance in a continual learning setup where the model learns only a world -speciﬁc representation function or a worldspeciﬁc composition function, and where the other module is shared across the worlds.
In Figure 6(b), we observe that sharing the representation function reduces the effect of catastrophic forgetting, but sharing the composition function does not have the same effect.
This suggests that the representation function learns representations that are useful across the worlds .
D ISCUSSION & C ONCLUSION
In this work, we propose GraphLog, a benchmark suite for evaluating the logical generalization capabilities of GNNs.
GraphLog is grounded in ﬁrst-order logic and provides access to a large number of diverse tasks that require compositional generalization to solve, including single task supervised learning, multi-task learning, and continual learning.
Our results highlight the importance of attention [Figure] mechanisms and modularity to achieve logical generalization, while also highlighting open challenges related to multi-task and continual learning in the context of GNNs.
A natural direction for future work is leveraging GraphLog for studying fast adaptation and meta-learning in the context of logical reasoning (e.g., via gradient-based meta-learning), as well as integrating state-of-the-art methods (e.g., regularization techniques) to combat catastrophic forgetting in the context of
A G RAPH L OG A. E XTENDED T ERMINOLOGY
In this section, we extend the terminology introduced in Section 3 to introduce the term descriptor .
A descriptor of a query graph g is computed by concatenating the relations occurring on the shortest path from the source and sink in the query ( u, v ) .
For example, if a query graph is described in the form of a list of edges such as g → (0 , 1 , r 1 ) , (1 , 2 , r 2 ) , (1 , 3 , r 3 ) , (2 , 4 , r 4 ) , where r i are the relations, then the descriptor of this graph given the query (0 , 4) would be D g
= r 1 , r 2 , r 4 .
In all our train, validation and test splits, we split graphs based on the unique set of descriptors, such as to maintain inductive reasoning aspect of the task.
A. D ATASET G ENERATION
This section follows up on the discussion in Section 3.1.
We describe all the steps involved in the dataset generation process.
Rule Generation .
In Algorithm 1, we describe the complete process of generating rules in GraphLog .
We require the set of K relations, which we use to sample the rule set R .
Then, we iterate through all possible combinations of relations in DataLog format to sample possible candidate rules.
We impose two constraints on the candidate rule: (i) No two rules in R can have the same body.
This ensures consistency between the rules.
(ii) Candidate rules cannot have common relations among the head and body .
This ensures absence of cycles.
We also add the inverse rule of our sampled candidate rule and check the same consistencies again.
We employ two types of unary Horn clauses to perform the closure of the available rules and to check the consistency of the different rules in R .
Using this process, we ensure that all generated rules are sound and consistent with respect to R .
World Sampling .
From the set of rules in R , we partition rules into buckets for different worlds (Algorithm 2).
We use a simple policy of bucketing via a sliding window of width w with stride s , to classify rules pertaining to each world.
For example, two such consecutive worlds can be generated as R t =
[ R i . . .
R i + w ] and R t +1 =
[ R i + s . . .
R i + w + s ] .
(Algorithm 2)
We randomly permute R before bucketing in-order.
Graph Generation .
This is a two-step process where ﬁrst we sample a world graph (Algorithm 3) and then we sample individual graphs from the world graph (Algorithm 4).
Given a set of rules R S , in the ﬁrst step, we recursively sample and apply rules in R S to generate a relation graph called world graph .
This sampling procedure enables us to create a diverse set of world graphs by considering only certain subsets (of R ) during sampling.
By controlling the extent of overlap between the subsets of
R (in terms of the number of rules that are common across the subsets), we can precisely control the similarity between the different world graphs .
By selecting subsets which have higher dissimilarity between each other, we introduce more diversity in terms of logical rules.
In the second step (Algorithm 4), the world graph is used to sample a set of query graphs G SW = { g 1 , · · · g N } .
A query graph g i is sampled from G W by sampling a pair of nodes ( u, v ) from G W and then by sampling a resolution path p u,vG W .
The edge r i ( u, v ) provides the target relation that the learning model has to predict.
Since the relation for the edge r i ( u, v ) can be resolved by composing the relations along the resolution path, the relation prediction task tests for the compositional generalization abilities of the models.
We ﬁrst sample all possible resolution paths and get their individual descriptors
D g i , which we split in training, validation and test splits.
We then construct the training, validation and testing graphs by ﬁrst adding all edges of an individual D g i to the corresponding query graph g i , and then sampling neighbors of
p g
i .
Concretely, we use Breadth First
Search (BFS) to sample the neighboring subgraph of each node u ∈ p
g
i with a decaying selection probability γ .
This allows us to create diverse input graphs while having precise control over its resolution by its descriptor D g i .
Splitting the dataset over these descriptor paths ensures inductive generalization.
A. C OMPUTING S IMILARITY
GraphLog provides precise control for categorizing the similarity between different worlds by computing the overlap of the underlying rules.
Concretely, the similarity between two worlds W i and
[Figure]
Input: Set of K relations { r i } K , K >
Deﬁne an empty rule set R for all r i ∈ { r i } K do for all r j ∈ { r i } K do for all r k ∈ { r i } K do
Deﬁne candidate rule t : [ r i , r j ] = ⇒
r k if Cyclical rule, i.e. r i ==
r k OR r j ==
r k then
Reject rule end if if t [ body ] (cid:54)∈ R then
Add t to R end if end for end for end for
Check and remove any further cyclical rules.
W j is deﬁned as Sim ( W i , W j ) =
|R i ∩R
j | , where W i and W j are the graph worlds and R i and R j are the set of rules associated with them.
Thus GraphLog enables various training scenarios - training on highly similar worlds or training on a mix of similar and dissimilar worlds.
This ﬁne grained control allows GraphLog to mimic both in-distribution and out-of-distribution scenarios - during training and testing.
It also enables us to precisely categorize the effect of multi-task pre-training when the model needs to adapt to novel worlds.
Algorithm
Require: Rule Set R S
Require: Number of worlds n w >
Require:
Number of rules per world w >
Require:
Overlapping increment stride s >for i = 0; i < |R S | − w ; do
R i
= R S
[ i ; i + w ] i
=
i + s end for
Algorithm
Require: Set of relations { r i } K , K >
Require:
Set of rules derived from { r i } K , |R| >
Require: Set rule selection probability gamma γ = 0 .
Set rule selection probability P [ R [ i ]]
= 1 , ∀ i ∈ |R|
Require: Maximum number of expansions s ≥
Require: Set of available nodes N , s.t.
| N | ≥
Require: Number of cycles of generation c ≥
Set WorldGraph set of edges G m = ∅ while | N | > 0 or c > 0 do
Randomly choose an expansion number for this cycle: steps = rand (2 , s )
Set added edges for this cycle E c = ∅ for all step in steps do if step =
then
With uniform probability, either:
Sample r t from R S [ head ] and sample u, v ∈ N without replacement, OR
Sample an edge ( u, r t , v ) from G m
Add ( u, r t , v ) to E c and G m else
Sample an edge ( u, r t , v ) from E c end if
Sample a rule R [ i ] from R following P s.t.
[ r i , r j ] = ⇒ r t
P [ R [ i ]] =
P [ R [ i ]] ∗ γ
Sample a new node y ∈ N without replacement
Add edge ( u, r i , y ) to E c and G m
Add edge ( y, r j , v ) t E c and G m end for if All rules in R is used atleast once then
Increment c by
Reset rule selection probability P [ R [ i ]]
= 1 ,
∀ i ∈ |R| end if end while
A. C OMPUTING DIFFICULTY
Recent research in multitask learning has shown evidence that models prioritize selection of difﬁcult tasks over easy tasks while learning to boost the overall performance (Guo et al., 2018).
Thus,
GraphLog also provides a method to examine how pretraining on tasks of different difﬁculty level affects the adaptation performance.
Due to the stochastic effect of partitioning of the rules, GraphLog consists of datasets with varying range of difﬁculty.
We use the supervised learning scores (Table 6) as a proxy to determine the the relative difﬁculty of different datasets.
We cluster the datasets such that tasks with prediction accuracy greater than or above 70% are labeled as easy difﬁculty, 50-70% are labeled as medium difﬁculty and below 50% are labeled as hard difﬁculty dataset.
We ﬁnd that the labels obtained by this criteria are consistent across the different models (Figure 3).
Graph properties affecting difﬁculty .
While we compute difﬁculty based on the proxy of supervised learning scores, we observe that the relative difﬁculty of the tasks are highly correlated with the number of descriptors (Section A.1) available for each task.
We can control the distribution of the dataset by explicitly requiring diversity in the descriptors.
Worlds having less number of descriptors are prone to be more difﬁcult for the task.
This is due to the fact that will less available descriptors with respect to the budget of data samples, our generation module samples the same set of descriptors while adding variable noise.
Thus, datasets with low descriptor count ends up with more relative noise.
This shows that for a learner, a dataset with enough variety among the resolution paths of the graphs with less noise is relatively easier to learn compared to the datasets which has less variation and more noise.
Require: Rule Set R S
Require: World Graph G m =
( V m , E m )
Require: Maximum Expansion length e >
Set Descriptor set S = ∅ for all u, v ∈ E m do
Get all walks Y ( u,v ) ∈
G m such that | Y ( u,v ) | ≤ e Get all descriptors D
Y ( u,v ) for all walks Y ( u,v )
Add D Y ( u,v ) to S end for
Set train graph set G train = ∅
Set test graph set G test = ∅
Split descriptors in train and test split, S train and S test for all D i ∈ S train or S test
do
Set source node u s =
D i [0] and sink node v s =
D i [ − 1]
Set prediction target t =
E m [ u s ][ v s ]
Set graph edges g i = ∅
Add all edges from D i to g i for all u, v ∈ D i do
Sample Breadth First Search connected nodes from u and v with decaying probability γ
Add the sampled edges to g i end for
Remove edges in g i which create shorter paths between u s and v s
Add ( g i , u s , v s , t ) to either G train or G test end for
B S UPERVISED LEARNING ON G RAPH L OG
We perform extensive experiments over all the datasets available in GraphLog (statistics given in
Table 6).
We observe that in general, for the entire set of 57 worlds, the GAT E-GAT model performs the best.
C M ULTITASK L EARNING C. M ULTITASK L EARNING ON DIFFERENT DATA SPLITS BY DIFFICULTY
[Table]
In Section A.4 we introduced the notion of difﬁculty among the tasks available in GraphLog .
Here, we consider a set of experiments where we perform multitask training and inductive testing on the worlds bucketized by their relative difﬁculty (Table 4).
We sample equal number of worlds from each difﬁculty bucket, and separately perform multitask training and testing.
We evaluate the average prediction accuracy on the datasets within each bucket.
We observe that the average multitask performance also mimics the relative task difﬁculty distribution.
We ﬁnd GAT-E-GAT model outperforms other baselines in Easy and Medium setup, but is outperformed by GAT-RGCN model in the Difﬁcult setup.
For each model, we used the same architecture and hyperparameter settings across the buckets.
Optimizing individually for each bucket may improve the relative performance.
[Figure] [Figure]
C. M ULTITASK P RE - TRAINING BY TASK SIMILARITY
In the main paper (Section 5.2) we introduce the setup of performing multitask pre-training on
GraphLog datasets and adaptation on the datasets based on relative similarity.
Here, we perform ﬁne-grained analysis of few-shot adapatation capabilities of the models.
We analyze the adaptation performance in two settings - when the adaptation dataset has complete overlap of rules with the training datasets ( group =1.0) and when the adaptation dataset has zero overlap with the training datasets ( group =0.0).
We ﬁnd RGCN family of models with a graph based representation function has faster adaptation on the dissimilar dataset, with GCN-RGCN showing the fastest improvement.
However on the similar dataset the models follow the ranking of the supervised learning experiments, with GAT-EGAT model adapting comparitively better.
C. M ULTITASK P RE - TRAINING BY TASK DIFFICULTY
Using the notion of difﬁculty introduced in Section A.4, we perform the suite of experiments to evaluate the effect of pre-training on Easy , Medium and Difﬁcult datasets.
Interestingly, we ﬁnd the performance on convergence is better on Medium and Hard datasets on pre-training, compared to the
Easy dataset (Table 5).
This behaviour is also mirrored in k-shot adaptation performance (Figure 9), where pre-training on Hard dataset provides faster adaptation performance on 4/6 models.
D C ONTINUAL L EARNING
A natural question arises following our continual learning experiments in Section 5.3 : does the order of difﬁculty of the worlds matter?
Thus, we perform an experiment following Curriculum Learning (Bengio et al., 2009) setup, where the order of the worlds being trained is determined by their relative difﬁculty (which is determined by the performance of models in supervised learning setup, Table 6, i.e., we order the worlds from easier worlds to harder worlds).
We observe that while the current task accuracy follows the trend of the difﬁculty of the worlds (Figure 10(a)), the mean of past accuracy is signiﬁcantly worse.
This suggests that a curriculum learning strategy might not be optimal to learn graph representations in a continual learning setting.
We also performed the same experiment with sharing only the composition and representation functions (Figure 10(b)), and observe similar trends where sharing the representation function reduces the effect of catastrophic forgetting.
[Table]
0.623 rule 524 train 5.87 18.011 22.202 Medium 0.
0.605 rule 194 train 4.29 11.459 13.037 Medium 0.
0.419 rule 306 train 4.14 11.238 12.919 Easy 0.
0.523 rule 149 train 3.58 11.238 13.549 Hard 0.
0.344 rule 224 train 4.14 11.371 13.403 Hard 0.
0.332 rule 224 train 3.82 12.661 15.105 Hard 0.
0.435 rule 205 train 3.59 11.345 13.293 Hard 0.
0.291 rule 147 train 3.
8.894 Hard 0.
0.181 rule 923 train 6.63 25.035 33.080 Easy 0.
0.618 rule 416 train 6.10 17.180 20.818 Easy 0.
0.702 rule 2024 train 8.63 34.059 45.985 Easy 0.
[Table]
[Table] [Table]
750 rule 272 train 4.
[Table] [Table]
754 Medium
[Table] [Table]
[Table] [Table]
[Table]
580 rule 422 train
[Table]
[Table] [Table]
681 Medium
[Table] [Table]
573 rule 383 train
[Table]
[Table] [Table]
111 Hard
[Table] [Table]
rule 879 train
[Table]
[Table] [Table]
152 Easy
[Table]
rule 278 train
[Table]
[Table] [Table]
[Table]
412 rule 352 train
[Table] [Table]
[Table] [Table]
rule
[Table] [Table]
[Table]
499 Easy
[Table]
rule 391 train
[Table]
[Table] [Table]
897 Medium
[Table] [Table]
[Table] [Table]
552 rule 144 train
[Table]
[Table] [Table]
Hard
[Table] [Table]
[Table] [Table]
389 rule 177 train
[Table]
[Table] [Table]
764 Hard
[Table] [Table]
[Table] [Table]
336 rule
[Table]
0.470 rule 95 train 2.
9.751 Hard 0.
0.395 rule 162 train 3.36 11.077 13.337 Medium 0.
0.437 rule 705 train 4.75 15.310 18.172 Hard 0.
0.450 rule 151 train 3.
9.127 10.001 Medium
[Table] [Table]
[Table] [Table]
[Table] [Table]
524 rule 2704 train 7.
[Table] [Table]
489 Easy
[Table] [Table]
[Table] [Table]
[Table] [Table]
749 rule 647 train 6.66 22.
[Table]
789 Easy
[Table] [Table]
[Table] [Table]
[Table]
611 rule 978 train 6.
[Table] [Table]
674 Easy
[Table] [Table]
[Table] [Table]
[Table]
0.753 rule 169 train 3.
9.983 11.177 Easy 0.
0.615 rule 286 train 3.99 12.274 16.117 Medium 0.
0.570 rule 332 valid 4.44 16.384 21.817 Easy 0.
0.689 rule
[Table]
Difﬁculty is calculated by taking the scores of the model (M1) and partitioning the worlds according to their accuracy ( ≥ 0 .
7 = Easy, ≥ 0 .
54 and < 0 .
7 = Medium, and < 0 .
=
Hard).
We provide both the mean of the raw accuracy scores for all models, as
well as the number of times the model is ranked ﬁrst in all the tasks.
[Figure]
E H YPERPARAMETERS AND E XPERIMENTAL S ETUP
In this section, we provide detailed hyperparameter settings for both models and dataset generation for the purposes of reproducibility.
The codebase and dataset used in the experiments are attached with the Supplementary materials, and will be made public on acceptance.
E. D ATASET H YPERPARAMS
We generate GraphLog with 20 relations or classes ( K ), which results in 76 rules in R S after consistency checks.
For unary rules, we specify half of the relations to be symmetric and other half to have their invertible relations.
To split the rules for individual worlds, we choose the number of rules for each world w = 20 and stride s = 1 , and end up with 57 worlds R 0 . . .
R 56 .
For each world R i , we generate 5000 training, 1000 testing and 1000 validation graphs.
E. M ODEL H YPERPARAMS
For all models, we perform hyper-parameter sweep (grid search) to ﬁnd the optimal values based on the validation accuracy.
For all models, we use the relation embedding and node embedding to be 200 dimensions.
Since the nodes in GraphLog does not have any features or attributes, we randomly initialize the embeddings in the GNN message passing layers for each epoch for all experiments.
We train all models with Adam optimizer with learning rate 0.001 and weight decay of 0.0001.
For supervised setting, we train all models for 500 epochs, and we add a scheduler for learning rate to decay it by 0.8 whenever the validation loss is stagnant for 10 epochs.
In multitask setting, we sample a new task every epoch from the list of available tasks.
Here, we run all models forepochs when we have the number of tasks ≤ 10 .
For larger number of tasks (Figure 4), we train by proportionally increasing the number of epochs compared to the number of tasks.
(2k epochs for 10 tasks, 4k epochs for 20 tasks, 6k epochs for 30 tasks, 8k epochs for 40 tasks and 10k epochs for
[Figure]
50 tasks).
For continual learning experiment, we train each task for 100 epochs for all models.
No learning rate scheduling is used for either multitask or continual learning experiments.
Individual model hyper-parameters are as follows:
• Representation functions : – GAT : Number of layers = 2, Number of attention heads = 2, Dropout = 0.
– GCN : Number of layers = 2, with symmetric normalization and bias, no dropout • Composition functions: – E-GAT : Number of layers = 6, Number of attention heads = 2, Dropout = 0.
– RGCN : Number of layers = 2, no dropout, with bias.