Original: Evaluating Graph Generative Models with Contrastively Learned Features
Revised: Evaluating Graph Generative Models with Contrastively Learned Features

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress
Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: A wide range of models have been proposed for Graph Generative Models, necessitating effective methods to evaluate their quality.
Revised: A wide range of models have been proposed for Graph Generative Models, necessitating effective methods to evaluate their quality.

Original: So far, most techniques use eithertraditional metrics based on subgraph counting, or the representations of randomlyinitialized Graph Neural Networks (GNNs).
Revised: So far, most techniques use eithertraditional metrics based on subgraph counting, or the representations of randomlyinitialized Graph Neural Networks (GNNs).

Original: We propose using representationsfrom contrastively trained GNNs, rather than random GNNs and show this givesmore reliable evaluation metrics.
Revised: We propose using representationsfrom contrastively trained GNNs, rather than random GNNs and show this givesmore reliable evaluation metrics.

Original: Neither traditional approaches nor GNN-basedapproaches dominate the other, however: we give examples of graphs that each approach is unable to distinguish.
Revised: Neither traditional approaches nor GNN-basedapproaches dominate the other, however: we give examples of graphs that each approach is unable to distinguish.

Original: We demonstrate that Graph Substructure Networks(GSNs), which in a way combine both approaches, are better at distinguishing thedistances between graph datasets.
Revised: We demonstrate that Graph Substructure Networks(GSNs), which in a way combine both approaches, are better at distinguishing thedistances between graph datasets.

Original:  Introduction
Revised:  Introduction

Original: Quantitative evaluation of generative models is challenging [45]; evaluating purely by visual inspection can introduce biases, particularly towards “precision” at cost of “recall” [37].
Revised: Quantitative evaluation of generative models is challenging [45]; evaluating purely by visual inspection can introduce biases, particularly towards “precision” at cost of “recall” [37].

Original: The traditionalmetric, likelihood, is also not only difficult to evaluate for implicit generative models on complex,highly structured datasets, but can also be a poor fit to users’ goals with generative modeling [37,45].
Revised: The traditionalmetric, likelihood, is also not only difficult to evaluate for implicit generative models on complex,highly structured datasets, but can also be a poor fit to users’ goals with generative modeling [37,45].

Original: Many quantitative proxy measures for the discrepancy between generator and real distributionshave thus been used in recent work on generative modeling of images, some of the most importantincluding the Inception Score (IS) [38], Fréchet Inception Distance (FID) [20], Precision/Recall (PR)[37], and Density/Coverage [32].
Revised: Many quantitative proxy measures for the discrepancy between generator and real distributionshave thus been used in recent work on generative modeling of images, some of the most importantincluding the Inception Score (IS) [38], Fréchet Inception Distance (FID) [20], Precision/Recall (PR)[37], and Density/Coverage [32].

Original: FID and PR measures in particular have been shown to correlatewith human judgments in some practical settings.
Revised: FID and PR measures in particular have been shown to correlatewith human judgments in some practical settings.

Original: All of these measures use representations extractedfrom CNNs pretrained on ImageNet classification [36].
Revised: All of these measures use representations extractedfrom CNNs pretrained on ImageNet classification [36].

Original: Nevertheless, it is recently shown thatself-supervised representations archives more reasonable ranking in terms of FID/Precision/Recall,while the ranking with ImageNet-pretrained embeddings often can be misleading [30].
Revised: Nevertheless, it is recently shown thatself-supervised representations archives more reasonable ranking in terms of FID/Precision/Recall,while the ranking with ImageNet-pretrained embeddings often can be misleading [30].

Original: These problems are exacerbated for the evaluation of generative models of graph data.
Revised: These problems are exacerbated for the evaluation of generative models of graph data.

Original: Graphs areoften used to represent concepts in a broad range of specialized domains, including various fieldsof engineering, bioinformatics, and so on, so that it is more difficult or perhaps impossible to find“generally good” features like those available for natural images via ImageNet models.
Revised: Graphs areoften used to represent concepts in a broad range of specialized domains, including various fieldsof engineering, bioinformatics, and so on, so that it is more difficult or perhaps impossible to find“generally good” features like those available for natural images via ImageNet models.

Original: Many methodsfor evaluating graph generative models are thus based on measuring discrepancies between statisticsof generic low-level graph features, including local measurements like degree distributions, clusteringcoefficient distributions, and four-node orbit counts [24, 52], or simple global properties such asspectra [25].
Revised: Many methodsfor evaluating graph generative models are thus based on measuring discrepancies between statisticsof generic low-level graph features, including local measurements like degree distributions, clusteringcoefficient distributions, and four-node orbit counts [24, 52], or simple global properties such asspectra [25].

Original: Differences between distributions of these features are generally measured via themaximum mean discrepancy (MMD) [15] or total variation (TV) distance.
Revised: Differences between distributions of these features are generally measured via themaximum mean discrepancy (MMD) [15] or total variation (TV) distance.

Original: More recent work [46] proposes instead extracting features via randomly initialized Graph Isomorphism Networks (GINs) [51].
Revised: More recent work [46] proposes instead extracting features via randomly initialized Graph Isomorphism Networks (GINs) [51].

Original: These features provide a representation for each graph, so that metricslike FID, Precision/Recall, and Density/Coverage can be estimated using these representations.
Revised: These features provide a representation for each graph, so that metricslike FID, Precision/Recall, and Density/Coverage can be estimated using these representations.

Original: For datasets with natural labels, we can alternatively find representations based on training a classifier;such features, however, are optimized for a very different task.
Revised: For datasets with natural labels, we can alternatively find representations based on training a classifier;such features, however, are optimized for a very different task.

Original: One significant challenge of graph data is that it is computationally difficult to even tell whether twographs are the same (are isomorphic to one another).
Revised: One significant challenge of graph data is that it is computationally difficult to even tell whether twographs are the same (are isomorphic to one another).

Original: GINs are known to be as powerful at detectingisomorphism as the standard Weisfeiler-Lehman (WL) test [51], but it is not known whether random
Revised: GINs are known to be as powerful at detectingisomorphism as the standard Weisfeiler-Lehman (WL) test [51], but it is not known whether random

Original: GIN representations have this ability.
Revised: GIN representations have this ability.

Original: Experimental evidence [51] shows that trained GIN networksare as powerful as the WL test, but have substantially worse power at random initialization.
Revised: Experimental evidence [51] shows that trained GIN networksare as powerful as the WL test, but have substantially worse power at random initialization.

Original: Unlike for distributions of natural images, however, the relevant graph features and particularly thesemantics of node or edge features varies widely between datasets.
Revised: Unlike for distributions of natural images, however, the relevant graph features and particularly thesemantics of node or edge features varies widely between datasets.

Original: On molecular graphs, addingany single edge will almost always violate physical constraints about stable atomic bonds; for houselayouts, the feasibility of adding an edge depends significantly on the rest of the layout and the typeof rooms (few front doors open into a bathroom, and direct connections between rooms on the firstand third floors are unlikely); Traditional metrics and random graph neural networks are both unlikelyto be able to capture these complex inter-dependencies functioning very differently across domains,and it similarly seems quite unlikely that pretraining on some “graph ImageNet” would be able tofind useful generic graph features.
Revised: On molecular graphs, addingany single edge will almost always violate physical constraints about stable atomic bonds; for houselayouts, the feasibility of adding an edge depends significantly on the rest of the layout and the typeof rooms (few front doors open into a bathroom, and direct connections between rooms on the firstand third floors are unlikely); Traditional metrics and random graph neural networks are both unlikelyto be able to capture these complex inter-dependencies functioning very differently across domains,and it similarly seems quite unlikely that pretraining on some “graph ImageNet” would be able tofind useful generic graph features.

Original: We therefore propose to train graph encoders on the same data used to train the generative model,using self-supervised contrastive learning to find meaningful feature extractors [18, 19, 34, 41, 44,48, 53–56], which we then use to compare the generated graphs to a test set.
Revised: We therefore propose to train graph encoders on the same data used to train the generative model,using self-supervised contrastive learning to find meaningful feature extractors [18, 19, 34, 41, 44,48, 53–56], which we then use to compare the generated graphs to a test set.

Original: The set of perturbationsintroduced in contrastive learning teaches the model which kinds of graphs should be consideredsimilar to one another.
Revised: The set of perturbationsintroduced in contrastive learning teaches the model which kinds of graphs should be consideredsimilar to one another.

Original: In this work we use types of perturbations traditionally used for trainingcontrastive learning methods on graphs, but point out that future work focusing on domain-specificmodeling can directly incorporate knowledge about which graphs should be considered similar bychoosing different perturbation sets.
Revised: In this work we use types of perturbations traditionally used for trainingcontrastive learning methods on graphs, but point out that future work focusing on domain-specificmodeling can directly incorporate knowledge about which graphs should be considered similar bychoosing different perturbation sets.

Original: Inspired by the theoretical results for contrastive learning of [50], we also propose two variations to ourrepresentation learning procedures.
Revised: Inspired by the theoretical results for contrastive learning of [50], we also propose two variations to ourrepresentation learning procedures.

Original: This upper bound shows that contrastively learned representationswork well for downstream tasks as long as the probability of data points yielding overlappingaugmentations is relatively large for within-class dataset pairs and relatively small for cross-classpairs.
Revised: This upper bound shows that contrastively learned representationswork well for downstream tasks as long as the probability of data points yielding overlappingaugmentations is relatively large for within-class dataset pairs and relatively small for cross-classpairs.

Original: Edit distances between graphs on typical training sets are large, however; we propose to usesubgraphs (“crops”) in our set of data augmentations for contrastive learning, which is more likelyto yield overlaps.
Revised: Edit distances between graphs on typical training sets are large, however; we propose to usesubgraphs (“crops”) in our set of data augmentations for contrastive learning, which is more likelyto yield overlaps.

Original: We also suggest enforcing a layer-wise Lipschitz constraint on feature extractors,which encourages similar graphs to have similar learned representations.
Revised: We also suggest enforcing a layer-wise Lipschitz constraint on feature extractors,which encourages similar graphs to have similar learned representations.

Original: We show experimentallythat both changes improve learning.
Revised: We show experimentallythat both changes improve learning.

Original: With all of these improvements, we further ask: can we theoretically guarantee that our learned GNNrepresentations outperform traditional local metrics?
Revised: With all of these improvements, we further ask: can we theoretically guarantee that our learned GNNrepresentations outperform traditional local metrics?

Original: We prove that we cannot: we give examplesof graphs easily distinguishable by local metrics that first-order GNNs cannot distinguish.
Revised: We prove that we cannot: we give examplesof graphs easily distinguishable by local metrics that first-order GNNs cannot distinguish.

Original: Yet theconverse is also true: we show graphs easily distinguishable by GNNs that appear equivalent to localmetrics.
Revised: Yet theconverse is also true: we show graphs easily distinguishable by GNNs that appear equivalent to localmetrics.

Original: We thus propose to use models based on Graph Substructure Networks (GSNs) [6], (usingnode degrees, and node clustering coefficients), and as a result, explicitly incorporating local metricsinto our models and surpassing the power of the WL test, and yield further improvements.
Revised: We thus propose to use models based on Graph Substructure Networks (GSNs) [6], (usingnode degrees, and node clustering coefficients), and as a result, explicitly incorporating local metricsinto our models and surpassing the power of the WL test, and yield further improvements.

Original:  Related Work
Revised:  Related Work

Original: Graph generative models.
Revised: Graph generative models.

Original: Our work is not particular to any type of graph generative model, asit focuses on simply evaluating samples; nonetheless, it is worth briefly reviewing some methods.
Revised: Our work is not particular to any type of graph generative model, asit focuses on simply evaluating samples; nonetheless, it is worth briefly reviewing some methods.

Original: Sequential generation models, such as GraphRNN [52] and GRAN [25], generate nodes and edges inan auto-regressive manner.
Revised: Sequential generation models, such as GraphRNN [52] and GRAN [25], generate nodes and edges inan auto-regressive manner.

Original: These models are efficient, but can struggle to capture global properties ofthe graphs, and require a predefined ordering.
Revised: These models are efficient, but can struggle to capture global properties ofthe graphs, and require a predefined ordering.

Original: One-shot models such as MolGAN [10], GraphDeconv[13], GraphVAE [23], and GraphNVP [27], on the other hand, generate all nodes and edges in oneshot; they can model long-range dependencies, but generally are not efficient and cannot scale tolarge graphs.
Revised: One-shot models such as MolGAN [10], GraphDeconv[13], GraphVAE [23], and GraphNVP [27], on the other hand, generate all nodes and edges in oneshot; they can model long-range dependencies, but generally are not efficient and cannot scale tolarge graphs.

Original: For a detailed overview of graph generative models, see [16].
Revised: For a detailed overview of graph generative models, see [16].

Original: Evaluating graph generative models.
Revised: Evaluating graph generative models.

Original: Graph generative models can be even more challenging toevaluate than visual or textual models, because it is generally more difficult for humans to judge thequality of a generated graph.
Revised: Graph generative models can be even more challenging toevaluate than visual or textual models, because it is generally more difficult for humans to judge thequality of a generated graph.

Original: The classic measure of the quality of a generative model, likelihood,also has significant issues with graphs: in addition to the kinds of issues that appear in generative
Revised: The classic measure of the quality of a generative model, likelihood,also has significant issues with graphs: in addition to the kinds of issues that appear in generative

Original: [Figure]
Revised: [Figure]

Original: models of images [45], the likelihood is particularly hard to evaluate on graphs where even checkingequality is quite difficult [33].
Revised: models of images [45], the likelihood is particularly hard to evaluate on graphs where even checkingequality is quite difficult [33].

Original: The most common method is to compute the Maximum Mean
Revised: The most common method is to compute the Maximum Mean

Original: Discrepancy (MMD) [15] between distributions of local graph statistics, including node degrees,cluster coefficients, and counts of orbits up to a particular size [25, 52].
Revised: Discrepancy (MMD) [15] between distributions of local graph statistics, including node degrees,cluster coefficients, and counts of orbits up to a particular size [25, 52].

Original: Global statistics, such aseigenvalues of the normalized graph Laplacian, are also used [52].
Revised: Global statistics, such aseigenvalues of the normalized graph Laplacian, are also used [52].

Original: These metrics, however, focusonly on low-level structure of the graph, and ignore any features that might be present on nodes oredges [46].
Revised: These metrics, however, focusonly on low-level structure of the graph, and ignore any features that might be present on nodes oredges [46].

Original: Choosing an appropriate kernel is also very important to consistency of these metrics[33].
Revised: Choosing an appropriate kernel is also very important to consistency of these metrics[33].

Original: These types of graph statistics might also encourage models to overfit to the training data, ratherthan truly learning the target distribution [40].
Revised: These types of graph statistics might also encourage models to overfit to the training data, ratherthan truly learning the target distribution [40].

Original: For unlabeled image-based generative models, most work focuses on metrics including the Inception Score (IS) [38], Fréchet Inception Distance (FID) [20], Precision/Recall (PR) [37], and
Revised: For unlabeled image-based generative models, most work focuses on metrics including the Inception Score (IS) [38], Fréchet Inception Distance (FID) [20], Precision/Recall (PR) [37], and

Original: Density/Coverage [32], all of which compare distributions in a fixed latent space (typically activationsof a late layer in an InceptionV3 [43] model trained on ImageNet).
Revised: Density/Coverage [32], all of which compare distributions in a fixed latent space (typically activationsof a late layer in an InceptionV3 [43] model trained on ImageNet).

Original: These methods are rarely adoptedin graph generative models, due to challenges with “general-purpose” graph models discussed in
Revised: These methods are rarely adoptedin graph generative models, due to challenges with “general-purpose” graph models discussed in

Original: Section 1.
Revised: Section 1.

Original: Thompson et al.
Revised: Thompson et al.

Original: [46] thus used random (untrained) graph encoders in these metrics.
Revised: [46] thus used random (untrained) graph encoders in these metrics.

Original: We discuss these methods in more detail in the appendix.
Revised: We discuss these methods in more detail in the appendix.

Original: Our work, inspired by Morozov et al.
Revised: Our work, inspired by Morozov et al.

Original: ’ssimilar proposal in image domains [31], explores the use of self-supervised contrastive training tofind representations that work better than random initializations.
Revised: ’ssimilar proposal in image domains [31], explores the use of self-supervised contrastive training tofind representations that work better than random initializations.

Original: Graph Contrastive Learning.
Revised: Graph Contrastive Learning.

Original: A popular method for self-supervised learning, contrastive learninggenerally aims to find a representation roughly invariant to various operations (e.g. for images, takingrandom crops, horizontal flipping, shifting colors) but able to identify different source data points.
Revised: A popular method for self-supervised learning, contrastive learninggenerally aims to find a representation roughly invariant to various operations (e.g. for images, takingrandom crops, horizontal flipping, shifting colors) but able to identify different source data points.

Original: Ideally, such a representation will be useful for downstream tasks not known when learning therepresentation.
Revised: Ideally, such a representation will be useful for downstream tasks not known when learning therepresentation.

Original: In graph settings, learned representations may be at the node, edge, or graph levels.
Revised: In graph settings, learned representations may be at the node, edge, or graph levels.

Original: Early works adopting contrastive learning to encode graphs used DeepInfoMax [21] loss to enforceconsistency between local (node) and global (graph) representations (e.g., DGI [48] and InfoGraph[41]).
Revised: Early works adopting contrastive learning to encode graphs used DeepInfoMax [21] loss to enforceconsistency between local (node) and global (graph) representations (e.g., DGI [48] and InfoGraph[41]).

Original: These methods did not use any specific augmentations and simply used distinct graphs asnegative examples.
Revised: These methods did not use any specific augmentations and simply used distinct graphs asnegative examples.

Original: Following works begin utilizing graph augmentations to further improve thelearned representations.
Revised: Following works begin utilizing graph augmentations to further improve thelearned representations.

Original: For example, MVGRL [18] uses graph diffusion and sub-graph inducing astwo types of augmentations, where as GCC [34] only uses multi-hop ego network inducing.
Revised: For example, MVGRL [18] uses graph diffusion and sub-graph inducing astwo types of augmentations, where as GCC [34] only uses multi-hop ego network inducing.

Original: Morerecent works use four types of graph augmentations including: feature masking, node dropping, edgeperturbation, and sub-graph inducing [19, 44, 53–56].
Revised: Morerecent works use four types of graph augmentations including: feature masking, node dropping, edgeperturbation, and sub-graph inducing [19, 44, 53–56].

Original: Among them, a few works rely on trial-anderror or heuristics to choose those augmentations for a given dataset (e.g., GraphCL [54], GRACE[55], and GCA [56]), whereas others introduce a policy network to learn to sample augmentationsand compute their parameters end-to-end along with graph representations (e.g., JOAO [53] and
Revised: Among them, a few works rely on trial-anderror or heuristics to choose those augmentations for a given dataset (e.g., GraphCL [54], GRACE[55], and GCA [56]), whereas others introduce a policy network to learn to sample augmentationsand compute their parameters end-to-end along with graph representations (e.g., JOAO [53] and

Original: LG2AR [19]).
Revised: LG2AR [19]).

Original:  GNNs Versus Local Metrics
Revised:  GNNs Versus Local Metrics

Original: Different methods for understanding graphs can “understand” the difference between graphs in verydifferent ways: a particular change to a graph might barely affect some features, while drasticallychanging others.
Revised: Different methods for understanding graphs can “understand” the difference between graphs in verydifferent ways: a particular change to a graph might barely affect some features, while drasticallychanging others.

Original: One extreme case is when a given metric cannot detect that two distinct (non- isomorphic) graphs are different.
Revised: One extreme case is when a given metric cannot detect that two distinct (non- isomorphic) graphs are different.

Original: Since graph isomorphism is a computationally difficult problem, weexpect that all efficiently computable graph representations “collapse” some pairs of input graphs.
Revised: Since graph isomorphism is a computationally difficult problem, weexpect that all efficiently computable graph representations “collapse” some pairs of input graphs.

Original: 1 Itis conceivable, however, that one method could be strictly more powerful than another.
Revised: 1 Itis conceivable, however, that one method could be strictly more powerful than another.

Original: For instance,since recent GNN models have overcome traditional models based on local metric representationsin a variety of problems [23, 47, 51], is it the case that GNNs are strictly more powerful than localmetrics?
Revised: For instance,since recent GNN models have overcome traditional models based on local metric representationsin a variety of problems [23, 47, 51], is it the case that GNNs are strictly more powerful than localmetrics?

Original: We show, constructively, that the answer is no: there are indeed graphs which GNNs can distinguishand local metrics cannot, but there are also graphs which local metrics can distinguish but first-order
Revised: We show, constructively, that the answer is no: there are indeed graphs which GNNs can distinguishand local metrics cannot, but there are also graphs which local metrics can distinguish but first-order

Original: GNNs cannot.
Revised: GNNs cannot.

Original: Figure 1(a) shows a pair of graphs with the same degree distribution, clustering coefficient, and four-node orbits, which can nonetheless be distinguished by GNNs (proof in Appendix B).
Revised: Figure 1(a) shows a pair of graphs with the same degree distribution, clustering coefficient, and four-node orbits, which can nonetheless be distinguished by GNNs (proof in Appendix B).

Original: On the other hand, the graphs in Fig. 1(b) have different clustering coefficient and smallest cycle,but first-order GNNs cannot tell them apart.
Revised: On the other hand, the graphs in Fig. 1(b) have different clustering coefficient and smallest cycle,but first-order GNNs cannot tell them apart.

Original: Thus, neither method strictly outperforms the other onall problems, and so there are theoretical generative models which perfectly match in GNN-basedrepresentations but differ in local metrics, and vice versa.
Revised: Thus, neither method strictly outperforms the other onall problems, and so there are theoretical generative models which perfectly match in GNN-basedrepresentations but differ in local metrics, and vice versa.

Original: This motivates our addition of local featuresto our graph representation models (Section 4).
Revised: This motivates our addition of local featuresto our graph representation models (Section 4).

Original: It is much easier to incorporated such hard-coded structures into GNNs than it would be to addlearning to feature metrics; in particular, counting higher-order local patterns quickly becomesprohibitively expensive, with a super-exponential time complexity.
Revised: It is much easier to incorporated such hard-coded structures into GNNs than it would be to addlearning to feature metrics; in particular, counting higher-order local patterns quickly becomesprohibitively expensive, with a super-exponential time complexity.

Original: GNNs can also easily handle nodeand/or edge features on the underlying graphs, which is far more difficult to add to local metrics.
Revised: GNNs can also easily handle nodeand/or edge features on the underlying graphs, which is far more difficult to add to local metrics.

Original: Another quality we would like in our graph representations, in addition to the ability to distinguishdistinct graphs, is some form of stability: if a distribution of graphs only changes slightly, we wouldlike our evaluation methods to give “partial credit” as opposed to a distribution where all graphsare dramatically different.
Revised: Another quality we would like in our graph representations, in addition to the ability to distinguishdistinct graphs, is some form of stability: if a distribution of graphs only changes slightly, we wouldlike our evaluation methods to give “partial credit” as opposed to a distribution where all graphsare dramatically different.

Original: (This is closely related to issues in training and evaluating image-basedgenerative models [1–3, 45].)
Revised: (This is closely related to issues in training and evaluating image-basedgenerative models [1–3, 45].)

Original: As previously discussed, the notion of “similar graphs” is very domaindependent, but traditional local metrics can be highly sensitive to changes like adding a single edge:
Revised: As previously discussed, the notion of “similar graphs” is very domaindependent, but traditional local metrics can be highly sensitive to changes like adding a single edge:

Original: Fig.
Revised: Fig.

Original: 1(c) shows an example where two graphs differing only by a single edge can have drasticallydifferent statistics.
Revised: 1(c) shows an example where two graphs differing only by a single edge can have drasticallydifferent statistics.

Original: By learning GNN representations, we can have some control over these types ofsmoothness properties; we exploit this explicitly in our methodology in Section 4.
Revised: By learning GNN representations, we can have some control over these types ofsmoothness properties; we exploit this explicitly in our methodology in Section 4.

Original:  Self-supervised Training of Graph Representations for Evaluation
Revised:  Self-supervised Training of Graph Representations for Evaluation

Original: Suppose we have two sets of graphs G train =
Revised: Suppose we have two sets of graphs G train =

Original: { G 1 , . . .
Revised: { G 1 , . . .

Original: G S } and G test =
Revised: G S } and G test =

Original: { G 1 , . . .
Revised: { G 1 , . . .

Original: G M } , eachsampled from the same data distribution p ( G ) .
Revised: G M } , eachsampled from the same data distribution p ( G ) .

Original: Also suppose that we have access to an unconditionalgraph generative model g ϕ ( . ) , which is trained on G train to learn the distribution of the observed setof graphs.
Revised: Also suppose that we have access to an unconditionalgraph generative model g ϕ ( . ) , which is trained on G train to learn the distribution of the observed setof graphs.

Original: We sample a set of generated graphs G gen = { G 1 , . . .
Revised: We sample a set of generated graphs G gen = { G 1 , . . .

Original: G N } ∼ p g ϕ ( G ) from this model.
Revised: G N } ∼ p g ϕ ( G ) from this model.

Original: In order to evaluate the quality of the sampled graphs (i.e., to decide whether the model g ϕ ( . ) hassuccessfully recovered the underlying distribution p ( G ) ), we can define a measure of divergence
Revised: In order to evaluate the quality of the sampled graphs (i.e., to decide whether the model g ϕ ( . ) hassuccessfully recovered the underlying distribution p ( G ) ), we can define a measure of divergence

Original: D ( G test , G gen ) to quantify the discrepancy between distributions of the real and generated graphs.
Revised: D ( G test , G gen ) to quantify the discrepancy between distributions of the real and generated graphs.

Original: One robust way to achieve this is to define the metric on latent vector representation spaces, andexpect representations of graphs rather than the original objects.
Revised: One robust way to achieve this is to define the metric on latent vector representation spaces, andexpect representations of graphs rather than the original objects.

Original: Thus, to use these metrics, we needto train a shared encoder f θ ( . ) and then compute the discrepancy as D ( f θ ( G test ) , f θ ( G gen )) .
Revised: Thus, to use these metrics, we needto train a shared encoder f θ ( . ) and then compute the discrepancy as D ( f θ ( G test ) , f θ ( G gen )) .

Original: Thereare a few such metrics well-studied in visual domains that can differentiate the fidelity and diversityof the model, and which we can adopt in graph domains.
Revised: Thereare a few such metrics well-studied in visual domains that can differentiate the fidelity and diversityof the model, and which we can adopt in graph domains.

Original: For evaluating image generative models, due to similar feature space across image datasets and alsoavailability of large-scale data, the trunk of a model trained over ImageNet with explicit supervisorysignals is usually chosen as the encoder.
Revised: For evaluating image generative models, due to similar feature space across image datasets and alsoavailability of large-scale data, the trunk of a model trained over ImageNet with explicit supervisorysignals is usually chosen as the encoder.

Original: However, it is not straightforward to adopt the same trick tograph-structured data: there are no ImageNet-scale graph datasets available, and more importantly,the semantics of graphs and their features vary wildly across commonly used graph datasets, far morethan occurs across distributions of natural images.
Revised: However, it is not straightforward to adopt the same trick tograph-structured data: there are no ImageNet-scale graph datasets available, and more importantly,the semantics of graphs and their features vary wildly across commonly used graph datasets, far morethan occurs across distributions of natural images.

Original: For instance, even datasets of molecular graphsmay use different feature sets to represent the molecules [17], in addition to the many cross-domainchallenges discussed in Section 1.
Revised: For instance, even datasets of molecular graphsmay use different feature sets to represent the molecules [17], in addition to the many cross-domainchallenges discussed in Section 1.

Original: Thus, it is not feasible to imagine a single “universal” graphrepresentation; we would like a general-purpose method for finding representations useful for a newgraph dataset.
Revised: Thus, it is not feasible to imagine a single “universal” graphrepresentation; we would like a general-purpose method for finding representations useful for a newgraph dataset.

Original: [Figure]
Revised: [Figure]

Original: Training with Graph Contrastive Learning To find expressive representation of real and generated graphs, we train the encoder using a contrastive objective.
Revised: Training with Graph Contrastive Learning To find expressive representation of real and generated graphs, we train the encoder using a contrastive objective.

Original: Assuming a set of rational augmentations T over G where each augmentation τ i ∈ T is defined as a function over graph G k that generatesan identity-preserving view of the graph: G + k
Revised: Assuming a set of rational augmentations T over G where each augmentation τ i ∈ T is defined as a function over graph G k that generatesan identity-preserving view of the graph: G + k

Original: = τ i ( G k ) , a contrastive framework with negativesampling strategy uses T to draw positive samples from the joint distribution p ( τ i ( G k ) , τ j ( G k )) inorder to maximize the agreement between different views of the same graph G k and to draw negativesamples from the product of marginals p ( τ i ( G k ))
Revised: = τ i ( G k ) , a contrastive framework with negativesampling strategy uses T to draw positive samples from the joint distribution p ( τ i ( G k ) , τ j ( G k )) inorder to maximize the agreement between different views of the same graph G k and to draw negativesamples from the product of marginals p ( τ i ( G k ))

Original: × p ( τ j ( G k ′ )) to minimize it for views from twodistinct graphs G k and G k ′ , k ̸ = k ′ .
Revised: × p ( τ j ( G k ′ )) to minimize it for views from twodistinct graphs G k and G k ′ , k ̸ = k ′ .

Original: As mentioned previously, we use a GIN architecture for our feature extractor.
Revised: As mentioned previously, we use a GIN architecture for our feature extractor.

Original: Other than the trainingprocess, the rest of our evaluation pipeline is similar to that of Thompson et al.
Revised: Other than the trainingprocess, the rest of our evaluation pipeline is similar to that of Thompson et al.

Original: [46], who userandom GIN weights.
Revised: [46], who userandom GIN weights.

Original: We consider two methods for training our GIN’s parameters: GraphCL [54]and InfoGraph [42].
Revised: We consider two methods for training our GIN’s parameters: GraphCL [54]and InfoGraph [42].

Original: GraphCL randomly samples augmentation from four possible augmentaiotnsincluding node dropping , edge perturbation , attribute masking , and sub-graph inducing based ona random walk to which we would like the representation to be roughly invariant.
Revised: GraphCL randomly samples augmentation from four possible augmentaiotnsincluding node dropping , edge perturbation , attribute masking , and sub-graph inducing based ona random walk to which we would like the representation to be roughly invariant.

Original: GraphCL usesnormalized temperature-scaled cross-entropy (NT-Xent) objective [7] to maximize the agreementbetween positive graph-level representations.
Revised: GraphCL usesnormalized temperature-scaled cross-entropy (NT-Xent) objective [7] to maximize the agreementbetween positive graph-level representations.

Original: InfoGraph works differently: it contrasts the graph-levelrepresentation with the representations of individual nodes, which encode neighborhood structureinformation.
Revised: InfoGraph works differently: it contrasts the graph-levelrepresentation with the representations of individual nodes, which encode neighborhood structureinformation.

Original: InfoGraph uses DeepInfoMax [21] objective to maximize the mutual informationbetween graph-level and node-level representations of each individual graph.
Revised: InfoGraph uses DeepInfoMax [21] objective to maximize the mutual informationbetween graph-level and node-level representations of each individual graph.

Original: Using Local Subgraph Counts as Input Features for GINs To build on the insight of Section 3,we also consider various methods for adding information about the local graph structure as nodefeatures, similarly to Graph Substructure Networks [6].
Revised: Using Local Subgraph Counts as Input Features for GINs To build on the insight of Section 3,we also consider various methods for adding information about the local graph structure as nodefeatures, similarly to Graph Substructure Networks [6].

Original: The simplest such method is to add the degreeof a node as an explicit feature for the GIN to consider.
Revised: The simplest such method is to add the degreeof a node as an explicit feature for the GIN to consider.

Original: We do this, but also add, by concatenation onnode features, higher-order local information as well: three-node and four-node clustering featuresfor each node.
Revised: We do this, but also add, by concatenation onnode features, higher-order local information as well: three-node and four-node clustering featuresfor each node.

Original: Four node clustering coefficient is calculated as:
Revised: Four node clustering coefficient is calculated as:

Original: [Equation]
Revised: [Equation]

Original: where N ( v ) denotes immediate neighbors of node v , deg ( v ) denotes the degree of node v , andq v ( u, w ) is the number of common neighbors of u and w , not counting v .
Revised: where N ( v ) denotes immediate neighbors of node v , deg ( v ) denotes the degree of node v , andq v ( u, w ) is the number of common neighbors of u and w , not counting v .

Original: Aggregating these featuresacross the whole graph would give information on distribution of 4 node orbits of the graph, but thisprovides more localized information across the graph that is nonetheless difficult or impossible for a
Revised: Aggregating these featuresacross the whole graph would give information on distribution of 4 node orbits of the graph, but thisprovides more localized information across the graph that is nonetheless difficult or impossible for a

Original: GIN to examine otherwise [8].
Revised: GIN to examine otherwise [8].

Original: Choice of Augmentations Wang et al.
Revised: Choice of Augmentations Wang et al.

Original: [50], building off work of Wang and Isola [49], studycontrastive learning in general settings (with an eye towards vision applications), and providean intriguing bound for the performance of downstream classifiers.
Revised: [50], building off work of Wang and Isola [49], studycontrastive learning in general settings (with an eye towards vision applications), and providean intriguing bound for the performance of downstream classifiers.

Original: To explain it, consider the“augmentation graph” of the training samples: if G i is the i th training example and G + i is a randomaugmentation of that example, we connect the nodes G i and G j in the augmentation graph if thereare feasible augmentations G + i and G + j such that G + i
Revised: To explain it, consider the“augmentation graph” of the training samples: if G i is the i th training example and G + i is a randomaugmentation of that example, we connect the nodes G i and G j in the augmentation graph if thereare feasible augmentations G + i and G + j such that G + i

Original: =
Revised: =

Original: G + j . 2 Wang et al.
Revised: G + j . 2 Wang et al.

Original: argue that downstreamlinear classifiers are likely to succeed if this augmentation graph has stronger intra-class connectivitythan it does cross-class connectivity, proving a tight connection between the two under a particularsetting with strong assumptions about this and related aspects of the setup.
Revised: argue that downstreamlinear classifiers are likely to succeed if this augmentation graph has stronger intra-class connectivitythan it does cross-class connectivity, proving a tight connection between the two under a particularsetting with strong assumptions about this and related aspects of the setup.

Original: Given the connection between the distribution metrics discussed in Appendix A.2 and classifier performance [e.g. 26, Section 4], if we accept the argument of Wang et al.
Revised: Given the connection between the distribution metrics discussed in Appendix A.2 and classifier performance [e.g. 26, Section 4], if we accept the argument of Wang et al.

Original: [50], evaluation metrics basedon a contrastively trained graph representation will give poor values (good classifier
Revised: [50], evaluation metrics basedon a contrastively trained graph representation will give poor values (good classifier

Original: performance)when generated samples are not well-connected to real samples in the augmentation graph, and viceversa.
Revised: performance)when generated samples are not well-connected to real samples in the augmentation graph, and viceversa.

Original: If we choose augmentations appropriately, this is sensible behavior.
Revised: If we choose augmentations appropriately, this is sensible behavior.

Original: Enforcing Lipschitz Layers in Representation Networks
Revised: Enforcing Lipschitz Layers in Representation Networks

Original: The prior line of reasoning also suggeststhat we should choose augmentations that are capable of making real graphs look like one another.
Revised: The prior line of reasoning also suggeststhat we should choose augmentations that are capable of making real graphs look like one another.

Original: Edit distances between graphs, however, are typically large on the datasets we consider, and soaugmentations based on adding or deleting individual nodes and/or edges will struggle to do this.
Revised: Edit distances between graphs, however, are typically large on the datasets we consider, and soaugmentations based on adding or deleting individual nodes and/or edges will struggle to do this.

Original: Thesame is true for many of the augmentations used on images, except – as Wang et al.
Revised: Thesame is true for many of the augmentations used on images, except – as Wang et al.

Original: [50] note – forcrop-type augmentations, where e.g. two different car pictures might become quite similar if we cropto just a wheel.
Revised: [50] note – forcrop-type augmentations, where e.g. two different car pictures might become quite similar if we cropto just a wheel.

Original: On graphs, an analogous operation is subgraph sampling, which we include in our
Revised: On graphs, an analogous operation is subgraph sampling, which we include in our

Original: GraphCL setup; InfoGraph already naturally looks at subgraph features as a core component.
Revised: GraphCL setup; InfoGraph already naturally looks at subgraph features as a core component.

Original: Taking this line of reasoning as well as the general motivations of contrastive learning further, it isalso natural to think that if we can inherently enforce “similar graphs” to have similar representations,this could improve the process of contrastive learning: we would save on needing to train the modelto learn these similarities, and it could help decrease the classifier performance for good generativemodels whose output graphs are legitimately near the distribution of target graphs.
Revised: Taking this line of reasoning as well as the general motivations of contrastive learning further, it isalso natural to think that if we can inherently enforce “similar graphs” to have similar representations,this could improve the process of contrastive learning: we would save on needing to train the modelto learn these similarities, and it could help decrease the classifier performance for good generativemodels whose output graphs are legitimately near the distribution of target graphs.

Original: A line of work on GANs in visual settings [1–3, 28, 29, 35] has made clear the importance of this typeof reasoning in losses for training generative models: the loss should smoothly improve as generatorsamples approach the target distribution, even if the supports differ.
Revised: A line of work on GANs in visual settings [1–3, 28, 29, 35] has made clear the importance of this typeof reasoning in losses for training generative models: the loss should smoothly improve as generatorsamples approach the target distribution, even if the supports differ.

Original: Viewing model evaluation metricsas a kind of “out-of-the-loop” loss function for training generative models – hyperparameter selectionand model development focusing on variants with better evaluation metrics – suggests that thesekinds of properties may be important for the problem of evaluation as well.
Revised: Viewing model evaluation metricsas a kind of “out-of-the-loop” loss function for training generative models – hyperparameter selectionand model development focusing on variants with better evaluation metrics – suggests that thesekinds of properties may be important for the problem of evaluation as well.

Original: We thus explicitly enforce the layers of our GIN to have a controlled Lipschitz constant, similarly toe.g.
Revised: We thus explicitly enforce the layers of our GIN to have a controlled Lipschitz constant, similarly toe.g.

Original: spectral normalization in GAN discriminators [29].
Revised: spectral normalization in GAN discriminators [29].

Original: To this end, we fix the λ Lipschitz factor to1 . 

Revised: To this end, we fix the λ Lipschitz factor to1 .

Original: in the experiments.
Revised: in the experiments.

Original: For each linear layer with weights W ℓ , we use projected gradient descent;after each update on the weights, if ∥ W ℓ ∥ > 1 . 0 , we update the value of the weights to W ℓ = W ℓ ∥ W ℓ ∥ .
Revised: For each linear layer with weights W ℓ , we use projected gradient descent;after each update on the weights, if ∥ W ℓ ∥ > 1 . 0 , we update the value of the weights to W ℓ = W ℓ ∥ W ℓ ∥ .

Original: This guarantees small changes in the graphs, such as adding/removing edges, or change in the inputfeatures, will not drastically change the final representation.
Revised: This guarantees small changes in the graphs, such as adding/removing edges, or change in the inputfeatures, will not drastically change the final representation.

Original:  Experimental Results
Revised:  Experimental Results

Original: In all of the experiments we train the model on the full dataset in an self-supervised manner.
Revised: In all of the experiments we train the model on the full dataset in an self-supervised manner.

Original: Following[46], we take the dataset and make perturbations on the dataset and see what is the trend in the measurements as the perturbation degree increases.
Revised: Following[46], we take the dataset and make perturbations on the dataset and see what is the trend in the measurements as the perturbation degree increases.

Original: We denote perturbation degree with r
Revised: We denote perturbation degree with r

Original: [Figure]
Revised: [Figure]

Original: and defineit for each type of perturbation.
Revised: and defineit for each type of perturbation.

Original: We use these type of perturbations:
Revised: We use these type of perturbations:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: • Mixing-Random:
Revised: • Mixing-Random:

Original: In perturbation with ratio r , we remove r chunk of the reference samples,and replace them with Erd ˝ os–Rényi (ER) graphs with the same ratio of edges.• Rewiring Edges: This perturbation, rewires each edge of the graph with probability r .
Revised: In perturbation with ratio r , we remove r chunk of the reference samples,and replace them with Erd ˝ os–Rényi (ER) graphs with the same ratio of edges.• Rewiring Edges: This perturbation, rewires each edge of the graph with probability r .

Original: Eachrewired edge, will change one of the sides of the edge with equal probability to another nodethat is not already connected to the stable node.• Mode Collapse and Mode Dropping:
Revised: Eachrewired edge, will change one of the sides of the edge with equal probability to another nodethat is not already connected to the stable node.• Mode Collapse and Mode Dropping:

Original: For these perturbation, first we cluster the graphsusing the WL-Kernel.
Revised: For these perturbation, first we cluster the graphsusing the WL-Kernel.

Original: First, we choose r ratio of clusters, then, for mode collapse, wereplace each graph on that dataset with the center of the cluster.
Revised: First, we choose r ratio of clusters, then, for mode collapse, wereplace each graph on that dataset with the center of the cluster.

Original: For, mode dropping, weremove the selected clusters and then for making size of the dataset fixed, we randomlyrepeat samples from other clusters.
Revised: For, mode dropping, weremove the selected clusters and then for making size of the dataset fixed, we randomlyrepeat samples from other clusters.

Original: For each experiment, we measure the Spearman rand correlation between the perturbation ratio, r ,and the value of the measurement.
Revised: For each experiment, we measure the Spearman rand correlation between the perturbation ratio, r ,and the value of the measurement.

Original: For measurements that supposed to decrease by the increase ofratio, we flip the values.
Revised: For measurements that supposed to decrease by the increase ofratio, we flip the values.

Original: As a result, in all experiments higher is better.
Revised: As a result, in all experiments higher is better.

Original: We gather the results amongdifferent datasets and several random seeds and plot them for distribution of the correlations.
Revised: We gather the results amongdifferent datasets and several random seeds and plot them for distribution of the correlations.

Original: Fordetailed experiments on the individual datasets see Appendix D.
Revised: Fordetailed experiments on the individual datasets see Appendix D.

Original: Datasets: Following [46], we use six diverse datasets ( three synthetic and three real-world) that arefrequently used in literature including: (1) Lobster [9] consisting of stochastic graphs with each nodeat most 2-hops away from a backbone path, (2) 2D Grid graphs [9, 25, 52], (3) Proteins [11] whererepresents connectivity among amino acids if their physical distance is less than a threshold, (4) 3-hop
Revised: Datasets: Following [46], we use six diverse datasets ( three synthetic and three real-world) that arefrequently used in literature including: (1) Lobster [9] consisting of stochastic graphs with each nodeat most 2-hops away from a backbone path, (2) 2D Grid graphs [9, 25, 52], (3) Proteins [11] whererepresents connectivity among amino acids if their physical distance is less than a threshold, (4) 3-hop

Original: Ego networks [52] extracted from the CiteSeer network [39] representing citation-based connectivityamong documents, (5) Community [52] graphs generated by Erd˝os–Rényi model [12], and (6) Zinc[22] is a dataset of attributed molecular graphs which allows to study sensitivity of metrics to changes in node/edge feature distributions.
Revised: Ego networks [52] extracted from the CiteSeer network [39] representing citation-based connectivityamong documents, (5) Community [52] graphs generated by Erd˝os–Rényi model [12], and (6) Zinc[22] is a dataset of attributed molecular graphs which allows to study sensitivity of metrics to changes in node/edge feature distributions.

Original: We follow the exact protocols used in [9, 25, 46, 52] as follows.
Revised: We follow the exact protocols used in [9, 25, 46, 52] as follows.

Original: We randomly sample 1000 graphs from Zinc dataset and use one-hot encoding for edge/node features.
Revised: We randomly sample 1000 graphs from Zinc dataset and use one-hot encoding for edge/node features.

Original: For community dataset, we set n = |V| / 2 , p = 0 . 3 , and add 0 .
Revised: For community dataset, we set n = |V| / 2 , p = 0 . 3 , and add 0 .

Original: 05 |V| inter-community edges withuniform probability.
Revised: 05 |V| inter-community edges withuniform probability.

Original: For all datasets, we use the node size range indicated in Table 4 in Appendix.
Revised: For all datasets, we use the node size range indicated in Table 4 in Appendix.

Original: Contrastive Training Versus Random GIN without Structural Features: In the first experiment,we compare the methods without adding structural features.
Revised: Contrastive Training Versus Random GIN without Structural Features: In the first experiment,we compare the methods without adding structural features.

Original: Results can be seen in Figure 3.
Revised: Results can be seen in Figure 3.

Original: Ingeneral we can see that in most measurements pretraining shows superior performance comparedto the random network.
Revised: Ingeneral we can see that in most measurements pretraining shows superior performance comparedto the random network.

Original: In this figure we have used GraphCL method.
Revised: In this figure we have used GraphCL method.

Original: Overall InfoGraph showssimilar results.
Revised: Overall InfoGraph showssimilar results.

Original: For InfoGraph results check Appendix D.
Revised: For InfoGraph results check Appendix D.

Original: In this experiments we use 10 randomseeds and gather data on all datasets.
Revised: In this experiments we use 10 randomseeds and gather data on all datasets.

Original: In Appendix D, same results separated for each dataset can beseen.
Revised: In Appendix D, same results separated for each dataset can beseen.

Original: In our experience, pretraining shows near perfect results on larger datasets, but for Lobster and
Revised: In our experience, pretraining shows near perfect results on larger datasets, but for Lobster and

Original: Grid datasets correlations are not near to 1 .
Revised: Grid datasets correlations are not near to 1 .

Original: Our observation is these measurements are moving withthe correct trend up to some perturbation ratio; but for example precision/recall become zero aftersome perturbation and keeps still.
Revised: Our observation is these measurements are moving withthe correct trend up to some perturbation ratio; but for example precision/recall become zero aftersome perturbation and keeps still.

Original: Our intuition here is because of highly regular structures in thesedatasets, model learns instantly to discrimate the real graphs from the fake ones very easily.
Revised: Our intuition here is because of highly regular structures in thesedatasets, model learns instantly to discrimate the real graphs from the fake ones very easily.

Original: And aftersmall amount of perturbation the perturbed graphs are all very far from the reference graphs.
Revised: And aftersmall amount of perturbation the perturbed graphs are all very far from the reference graphs.

Original: [Table]
Revised: [Table]

Original: Contrastive Training with Structural Information: In this experiment we concatenate nodedegrees and node clustering coefficients for each node feature.
Revised: Contrastive Training with Structural Information: In this experiment we concatenate nodedegrees and node clustering coefficients for each node feature.

Original: Here we can see that adding thesefeatures improves the models power for some measurements.
Revised: Here we can see that adding thesefeatures improves the models power for some measurements.

Original: However, on small datasets Grid and
Revised: However, on small datasets Grid and

Original: Lobster results become poorer.
Revised: Lobster results become poorer.

Original: Again intuition is more powerful model can easier distinguish maingraphs from the fake ones.
Revised: Again intuition is more powerful model can easier distinguish maingraphs from the fake ones.

Original: Break down of the results on the datasets in Appendix D proves this point.
Revised: Break down of the results on the datasets in Appendix D proves this point.

Original: Generative Models Benchmark
Revised: Generative Models Benchmark

Original: In this experiment, we use samples generated from the GRAN[25] after 25% of epochs, after full training, and also compare to a holdout set of the real data (half ofthe original dataset).
Revised: In this experiment, we use samples generated from the GRAN[25] after 25% of epochs, after full training, and also compare to a holdout set of the real data (half ofthe original dataset).

Original: We pretrain a model with full structural features and evaluate the criteria onthem.
Revised: We pretrain a model with full structural features and evaluate the criteria onthem.

Original: As expected, the early-training results are worse than post-training results, while the real datagives almost perfect results.
Revised: As expected, the early-training results are worse than post-training results, while the real datagives almost perfect results.

Original: Results are provided in Table 2.
Revised: Results are provided in Table 2.

Original: [Table]
Revised: [Table]

Original: Ablation Study: We analyze how layer normalization and using subgraph augmentations in
Revised: Ablation Study: We analyze how layer normalization and using subgraph augmentations in

Original: GraphCL reflects on the final results.
Revised: GraphCL reflects on the final results.

Original: We have used no structural feature setup and conductedthe experiment for mixing-random and mode-collapse experiments.
Revised: We have used no structural feature setup and conductedthe experiment for mixing-random and mode-collapse experiments.

Original: Figure 5, shows the results onthis task.
Revised: Figure 5, shows the results onthis task.

Original: The results prove that both of these improvements are essential for getting better results.
Revised: The results prove that both of these improvements are essential for getting better results.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: We have demonstrated that self-supervised pre-training of representations can yield significantlybetter metrics for graph evaluation than random ones, particularly when incorporating local graphfeatures with Lipschitz control, as inspired by theory.
Revised: We have demonstrated that self-supervised pre-training of representations can yield significantlybetter metrics for graph evaluation than random ones, particularly when incorporating local graphfeatures with Lipschitz control, as inspired by theory.

Original: We suggest graph generative modeling papersshould consider evaluating with these metrics in addition to or instead of their existing ones.
Revised: We suggest graph generative modeling papersshould consider evaluating with these metrics in addition to or instead of their existing ones.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?[N/A] A Preliminaries We now give further description of some background material.
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: A.1 Graph Neural Networks

Original: Suppose we have a set of graphs G = { G 1 , . . .

Original: G N } where each graph G =

Original: ( V , E , X , E ) consists of|V| nodes and |E| edges such that an edge e

Original: ij ∈ E connects nodes v i , v j ∈ V ( E ⊆ V × V ).

Original: If thegraph contains extra data beyond simply the connectivity structure, initial node features X ∈ R |V|× d xand/or edge features E ∈ R

Original: |V|×|V|× d e will also be available, where d x , and d e denote the dimensionof initial node and edge features, respectively.

Original: Message-passing GNNs learn low-dimensional noderepresentations h ( k ) v ∈ R d h , ∀ v ∈ V as follows:

Original: [Equation]

Original: where N ( v ) is a set of immediate neighbors of node v , ϕ and ψ are two arbitrary functions, and (cid:76) isthe aggregation function.

Original: We can use a read-out function to aggregate the learned node representationsinto a graph representation: h G = R ( { h v | v ∈ V} ) .

Original: Different instantiations of ϕ, ψ, (cid:76) results indifferent flavors of GNNs such as message passing neural network (MPNN) [14], graph convolutionalnetwork (GCN) [23], graph attention network (GAT)

Original: [47], and graph isomorphism network (GIN)[51].

Original: Particularly, GIN (in which ϕ and ψ are designed as MLP and identity functions, respectively)is shown to be as powerful at detecting isomorphism as the standard Weisfeiler-Lehman (WL) test.

Original: A.2 Evaluating Graph Generative Models in Latent Space

Original: Suppose we have two sets of graphs G train =

Original: { G 1 , . . .

Original: G S } and G test =

Original: { G 1 , . . .

Original: G M } , eachsampled from the same data distribution p ( G ) .

Original: Also suppose that we have access to an unconditionalgraph generative model g ϕ ( . ) , which is trained on G train to learn the distribution of the observed setof graphs.

Original: We sample a set of generated graphs G gen = { G 1 , . . .

Original: G N } ∼ p g ϕ ( G ) from this model.

Original: In order to evaluate the quality of the sampled graphs (i.e., to decide whether the model g ϕ ( . ) hassuccessfully recovered the underlying distribution p ( G ) ), we can define a measure of divergence

Original: D ( G test , G gen ) to quantify the discrepancy between distributions of the real and generated graphs.

Original: One robust way to achieve this is to define the metric on latent vector representation spaces, andexpect representations of graphs rather than the original objects.

Original: Thus, to use these metrics, weneed to train a shared encoder f θ ( . ) and then compute the discrepancy as D (cid:0) H t , H g (cid:1) where H t =f θ ( G test

Original: ) ∈ R M × d h and H g = f

Original: θ ( G gen ) ∈ R N × d h .

Original: There are a few such metrics well-studied invisual domains that can differentiate the fidelity and diversity of the model, and which we can adoptin graph domains.

Original: Fréchet Distance (FD) [20] models the graph representations as multivariate Gaussian distributionswith mean and covariance µ and C , and then computes the squared Wasserstein-2 distance betweenthem: D (cid:0) H t , H g (cid:1) =

Original: [-|| µ t − µ g || 
-]
Revised: [-|| µ t − µ g || 
-]

Original: + Tr (cid:0) C t + C g − 2( C t C g ) 1 / 2 (cid:1) .

Original: FD based on Inception v3 featuresof images has been shown to correlate with human judgments in some settings.

Original: It can also detectintra-class mode collapse [5].

Original: The estimator, however, has low variance high bias [4].

Original: Precision & Recall (PR) [37] measure constructs a hyper-sphere for each representation by extendinga radius to its kth nearest neighbor and then aggregates the hyper-spheres to define a manifold.

Original: Itthen computes the precision and recall based on generated samples that fall within the manifold ofreal samples and vice versa.

Original: PR is also shown to correlate with human judgments in visual domain.

Original: Itcan also detect mode collapse and mode dropping.

Original: Density & Coverage (DC) [32] is similar to PR but instead of aggregating the hyper-spheres, treatsthem independently.

Original: Density is computed as the average number of real hyper-spheres that a generatedsample falls in and Coverage is computed as the average number of generated hyper-spheres that areal samples falls in.

Original: DC is shown to be more robust compared to PR.

Original: Maximum Mean Discrepancy (MMD)

Original: [15] compares two distributions of any type, based on itemlevel comparison by a kernel function (e.g., polynomial kernel K ( x i , x j )

Original: =

Original: ( x ⊤ i x j + 1) p , Gaussian

Original: [Figure]

Original: ) (b) (c) (d) (e)

Original: [Figure]

Original: ) (g) (h) (i) (j) (k)

Original: (a) (b) . . .

Original: eeeeee' 1 eeeee'e'e'e'e'e'e'e' kernel K ( x i , x j ) = exp (cid:16)

Original: [Equation]

Original: (cid:17) , or linear kernel K ( x i , x j ) =

Original: x ⊤ i x j ).

Original: The usual estimator is

Original: [Equation]

Original: NM

Original: N (cid:88)

Original: i

Original: [Equation]

Original: M

Original: [Equation]

Original: [Equation]

Original: B Proofs

Original: Let C i,j denote the graph constructed by taking disjoint cycles of size i and a cycle of size j , thenconnecting the two with a single edge between one node from each side.

Original: Proposition 1.

Original: For any a, b, c, d satisfying 4 < a < c < d < b and a + b = c + d , C a,b and C c,d arenot distinguishable using local metrics, but there exists a GNN that can distinguish the two.

Original: Proof.

Original: First we will prove that local metrics can not identify these two class of graphs from eachother:

Original: Both graphs have a + b − 2 nodes of degree 2 and 2 nodes of degree 3 so the distribution of degreesis equal in both graphs.

Original: For clustering coefficient, none of the graphs has a cycle of size 3 , so it is 0for both graphs.

Original: We will count the number of different possible orbits of 4 nodes (all shown in Fig. 6), and show thecount is the same between two graphs.

Original: Orbits a-e occur in neither of the two graphs, so their counts are the same.

Original: Orbit f appears just twice in both graphs.

Original: For g and h: Consider an edge in an arbitrary graph, if we remove the edge and all nodes connectedto either sides of the edge, in the remaining graph selection of each two connected nodes with the firstedge will make pattern h, and two non-connected with the removed edge will make pattern g. Andthese are all patterns g and h that we can see in these graphs.

Original: Now, let us assume a mapping fromedges of C a,b to C c,d , by e i ↔ e ′ i for the named edges in figure 7.

Original: And assume any arbitrary mapping for other edges.

Original: If we call this mapping f , we can see that removing edge e and all its neighbors,leaves the same number of edges and nodes as removing f ( e ) and its neighbors for the second graph.

Original: This results as having the same number of patterns of g and h in both graphs.

Original: For pattern i, if we assume two connected edges and again remove all the nodes connected to thenodes of these edges, any of the remaining nodes + two removed edges will shape pattern i.

Original: Again bythe same mapping f , we can see that removing any of these two connected edges from the first graphleaves the same number of nodes as removing from the second graph.

Original: For pattern j, we can divide these patterns into two groups: 1) Patterns that do not include e 5 or e ′ 5 .2)

Original: Patterns that include these edges.

Original: For the first 1, we can assume we have two separate cycles oneach of the graphs.

Original: Each cycle of size n has n of these patterns, so first graph has a total of a + b ofthese patterns, and the second graph has a total of c + d = a + b of these patterns.

Original: So, these class ofpatterns have same amount of repetition in two graphs.

Original: For (2), we can easily see from the 7 there are8 of them in each of the graphs.

Original: So, the number of patterns of this type is also equal in both graphs.

Original: For pattern k, as two graphs have same number of nodes, we have (cid:0) nk (cid:1) possible selection of fournodes, and as the number of patterns of type a-j is equal between the graphs, pattern k should alsohave same number of repeat in both graphs.

Original: As a result, the distribution of local metrics is exactly same between two classes of graphs, and thusthese two graphs can not be distinguished using these metrics.

Original: Now, we prove that with a deep enough GNN, and good set of weights, we can distinguish these twographs.

Original: Assume all nodes have no information more than their degrees, at first all of the nodes excepttwo sides of bridge in each graph have the same message.

Original: There are two messages of degree 3, andrest of degree 2.

Original: In cycle with size a , after a steps, the initial node with degree 3, will finally receivea message that has started from itself, and so it would be different from any message that are alreadyspread in 3 other circles.

Original: As a result, there are at least two nodes with different messages betweentwo graphs after a steps.

Original: Thus two graphs are distinguishable using this GNN.

Original: For Fig.

Original: 1 (b), it is well-known that these two graphs can not be identified using WL-test, and thusresults of Xu et al.

Original: [51] imply that GNNs of order 1 can not distinguish these two graphs.

Original: On theother hand, for local structures, one of the graphs has two triangles and the other one has none.

Original: As aresult clustering coefficients of the nodes will be different.

Original: C Implementation Details and Hyperparameters

Original: In addition to the details here, code is available in the supplementary materials, and will be madepublic with release of the paper.

Original: C.1 Training Self-Supervised Methods

Original: For training GraphCL networks, we use node dropping and edge dropping probabilities of 0 .

Original: 1 for both.

Original: We use random walks of length 10 as subgraphs.

Original: We do not use node feature masking augmentations;here essentially all the datasets except for the ZINC do not have any node features, and we useconstant value of 1 . 0 for their node features.

Original: Removing just constant would leave some nodes withoutany features.

Original: For the experiments with structural features, the point was to see how the structuralfeatures work, as a result again we do not use feature dropping here as well.

Original: During the training, weuse structural features as default node features; this means that after augmentations like node/edgedropping, we do not recalculate the features for the new graph.

Original: This makes training process faster,but can affect the learning process in GraphCL only.

Original: The hyperparameters used for GINs are provided in 3.

Original: These hyperparameters are shared among

Original: GraphCL, InfoGraph and Random GIN models.

Original: For other hyperparameters of the self-supervisedtraining we follow the instructions from Sun et al.

Original: [42] and You et al.

Original: We train the self-supervisedmethods and evaluate them based on their criteria loss.

Original: After finding proper hyperparameters for thatdataset, we train the model with those hyperparameters and among all of the experiments for thatdataset we use the same model (no retraining for each experiment).

Original: [Table]

Original: C.2 Random GINs

Original: Experiments in [46] show that results on random GINs tend to have very small variance among different setups of the structures of the GINs.

Original: For fair evaluation, we used the same GIN hyperparametersamong all of the models for the same dataset.

Original: We use batch norm and orthogonal initialization asinstructed in the main paper.

Original: For each experiment with a different seed, we use a different randomlyinitialized GIN.

Original: For experiments with structural features, we give the same features as input forthe Random GIN as well.

Original: Random GIN with no structural features will be the same as the methodproposed in [46].

Original: C.3 Benchmark Experiment Setups

Original: We follow the exact instruction and public code released for [46] for the benchmarks.

Original: Step sizes forincreasing the r is set to 0 .

Original: For mixing random we remove r ratio of the graphs and for each Graph

Original: G =

Original: ( V, E ) that has been removed we replace it with and Erd ˝ os–Rényi (ER) graph with same numberof nodes and probability of connection as p = | E | ( | V | 2 ) .

Original: Rewiring probabilities are increasing from 0 .

Original: 0to 1 . 0 with step size of 0 . 01 .

Original: The number of steps for mode-dropping/mode-collapse experiments isas the number of clusters found by the clustering algorithm.

Original: Experiments for no structural featuresand with degree features are conducted with ten random seed for each dataset.

Original: For the clusteringfeatures we have used three random seed per dataset.

Original: Ablation studies are done using 5 random seedsper dataset.

Original: D Further Experiment ResultsD.1 Contrastive Training Versus Random GIN with Node Degree Features:

Original: In this experiment, similar to the first one, we compare random and pretrained models.

Original: The differenceis that we use just node degree features for the nodes, the easiest and fastest structural informationthat could be added to the model.

Original: Results are provided in Figure 8.

Original: D.2 Break Down of Experiments on Single Datasets

Original: Table 4 gives details of the datasets.

Original: Figs.

Original: 9 to 14 and Tables 5 to 10 break down the experiments withno structural features on individual datasets, Figs.

Original: 15 to 20 and Tables 11 to 16 those with degreefeatures

Original: [Table]
Revised: [N/A]

Original: and Figs.

Original: 21 to 26 and Tables 17 to 22 those with clustering features.

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: methods, with no structural features for lobster dataset

Original: [Table]

Original: [Figure]

Original: methods, with no structural features for grid dataset

Original: [Table]

Original: [Figure]

Original: methods, with no structural features for community dataset

Original: [Table]

Original: [Figure]

Original: methods, with no structural features for ego dataset

Original: [Table]

Original: [Figure]

Original: methods, with no structural features for proteins dataset

Original: [Table]

Original: [Figure]

Original: methods, with no structural features for zinc dataset

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: a

Original: [Table]

Original: n

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: R

Original: [Figure]

Original: n k

Original: C o rr e l a t i o n

Original: [Figure]

Original: D

Original: M

Original: [Figure]

Original: e a

Original: [Figure]

Original: M M

Original: [Figure]

Original: n k

Original: C o rr e l a t

Original: i

Original: [Figure]

Original: n

Original: [Figure]

Original: [Figure]

Original: FD

Original: MMD Lin

Original: [Figure]

Original: RBF

Original: [Figure]

Original: Mixing

Original: [Figure]

Original: Mean 1.

Original: 1.0 Median 1.

Original: Rewiring Edges Mean 0.

Original: 1.0 Median 0.

Original: Mode Collapse

Original: Mean -0.

Original: 1.0 Median -0.

Original: Mode Dropping Mean -0.

Original: 0.96 Median -0.

Original: GIN

Original: [Figure]

Original: Random

Original: Mixing Random Mean 1.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: C

Original: [Table]

Original: rr

Original: [Table]

Original: t i o n

Original: [Table]

Original: [Table]

Original: [Figure]

Original: t

Original: i

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]