Original: Evaluating Graph Generative Models with Contrastively Learned Features
Revised: Evaluating Graph Generative Models with Contrastively Learned Features

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddressemail
Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: A wide range of models have been proposed for Graph Generative Models, necessitating effective methods to evaluate their quality.
Revised: A wide range of models have been proposed for Graph Generative Models, necessitating effective methods to evaluate their quality.

Original: So far, most techniques use eithertraditional metrics based on subgraph counting, or the representations of randomlyinitialized Graph Neural Networks (GNNs).
Revised: So far, most techniques use eithertraditional metrics based on subgraph counting, or the representations of randomlyinitialized Graph Neural Networks (GNNs).

Original: We propose using representationsfrom constrastively trained GNNs, rather than random GNNs and show this givesmore reliable evaluation metrics.
Revised: We propose using representationsfrom contrastively trained GNNs, rather than random GNNs and show this givesmore reliable evaluation metrics.

Original: Neither traditional approaches nor GNN-basedapproaches dominate the other, however: we give examples of graphs that each approach is unable to distinguish.
Revised: Neither traditional approaches nor GNN-basedapproaches dominate the other, however: we give examples of graphs that each approach is unable to distinguish.

Original: We demonstrate that Graph Substructure Networks(GSNs), which in a way combine both approaches, are better at distinguishing thedistances between graph datasets.
Revised: We demonstrate that Graph Substructure Networks(GSNs), which in a way combine both approaches, are better at distinguishing thedistances between graph datasets.

Original:  Introduction
Revised:  Introduction

Original: Quantitative evaluation of generative models is challenging [46]; evaluating purely by visual inspection can introduce biases, particularly towards “precision” at cost of “recall” [38].
Revised: Quantitative evaluation of generative models is challenging [46]; evaluating purely by visual inspection can introduce biases, particularly towards “precision” at cost of “recall” [38].

Original: The traditionalmetric, likelihood, is also not only difficult to evaluate for implicit generative models on complex,highly structured datasets, but can also be a poor fit to users’ goals with generative modeling [38,46].
Revised: The traditionalmetric, likelihood, is also not only difficult to evaluate for implicit generative models on complex,highly structured datasets, but can also be a poor fit to users’ goals with generative modeling [38,46].

Original: Many quantitative proxy measures for the discrepancy between generator and real distributionshave thus been used in recent work on generative modeling of images, some of the most importantincluding the Inception Score (IS) [39], Fréchet Inception Distance (FID) [21], Precision/Recall (PR)[38], and Density/Coverage [33].
Revised: Many quantitative proxy measures for the discrepancy between generator and real distributionshave thus been used in recent work on generative modeling of images, some of the most importantincluding the Inception Score (IS) [39], Fréchet Inception Distance (FID) [21], Precision/Recall (PR)[38], and Density/Coverage [33].

Original: FID and PR measures in particular have been shown to correlatewith human judgments in some practical settings.
Revised: FID and PR measures in particular have been shown to correlatewith human judgments in some practical settings.

Original: All of these measures use representations extractedfrom CNNs pretrained on ImageNet classification [37].
Revised: All of these measures use representations extractedfrom CNNs pretrained on ImageNet classification [37].

Original: Nevertheless, it is recently shown thatself-supervised representations archives more reasonable ranking in terms of FID/Precision/Recall,while the ranking with ImageNet-pretrained embeddings often can be misleading [31].
Revised: Nevertheless, it is recently shown thatself-supervised representations archives more reasonable ranking in terms of FID/Precision/Recall,while the ranking with ImageNet-pretrained embeddings often can be misleading [31].

Original: These problems are exacerbated for the evaluation of generative models of graph data.
Revised: These problems are exacerbated for the evaluation of generative models of graph data.

Original: Graphs areoften used to represent concepts in a broad range of specialized domains, including various fieldsof engineering, bioinformatics, and so on, so that it is more difficult or perhaps impossible to find“generally good” features like those available for natural images via ImageNet models.
Revised: Graphs areoften used to represent concepts in a broad range of specialized domains, including various fieldsof engineering, bioinformatics, and so on, so that it is more difficult or perhaps impossible to find“generally good” features like those available for natural images via ImageNet models.

Original: Many methodsfor evaluating graph generative models are thus based on measuring discrepancies between statisticsof generic low-level graph features, including local measurements like degree distributions, clusteringcoefficient distributions, and four-node orbit counts [25, 54], or simple global properties such asspectra [26].
Revised: Many methodsfor evaluating graph generative models are thus based on measuring discrepancies between statisticsof generic low-level graph features, including local measurements like degree distributions, clusteringcoefficient distributions, and four-node orbit counts [25, 53], or simple global properties such asspectra [26].

Original: Differences between distributions of these features are generally measured via themaximum mean discrepancy (MMD) [15] or total variation (TV) distance.
Revised: Differences between distributions of these features are generally measured via themaximum mean discrepancy (MMD) [15] or total variation (TV) distance.

Original: More recent work [47] proposes instead extracting features via randomly initialized Graph Isomorphism Networks (GINs) [53].
Revised: More recent work [47] proposes instead extracting features via randomly initialized Graph Isomorphism Networks (GINs) [52].

Original: These features provide a representation for each graph, so that metricslike FID, Precision/Recall, and Density/Coverage can be estimated using these representations.
Revised: These features provide a representation for each graph, so that metricslike FID, Precision/Recall, and Density/Coverage can be estimated using these representations.

Original: For datasets with natural labels, we can alternatively find representations based on training a classifier;such features, however, are optimized for a very different task.
Revised: For datasets with natural labels, we can alternatively find representations based on training a classifier;such features, however, are optimized for a very different task.

Original: One significant challenge of graph data is that it is computationally difficult to even tell whether twographs are the same (are isomorphic to one another).
Revised: One significant challenge of graph data is that it is computationally difficult to even tell whether twographs are the same (are isomorphic to one another).

Original: GINs are known to be as powerful at detectingisomorphism as the standard Weisfeiler-Lehman (WL) test [53], but it is not known whether random
Revised: GINs are known to be as powerful at detectingisomorphism as the standard Weisfeiler-Lehman (WL) test [52], but it is not known whether random

Original: GIN representations have this ability.
Revised: GIN representations have this ability.

Original: Experimental evidence [53] shows that trained GIN networksare as powerful as the WL test, but have substantially worse power at random initialization.
Revised: Experimental evidence [52] shows that trained GIN networksare as powerful as the WL test, but have substantially worse power at random initialization.

Original: Unlike for distributions of natural images, however, the relevant graph features and particularly thesemantics of node or edge features varies widely between datasets.
Revised: Unlike for distributions of natural images, however, the relevant graph features and particularly thesemantics of node or edge features varies widely between datasets.

Original: On molecular graphs, addingany single edge will almost always violate physical constraints about stable atomic bonds; for houselayouts, the feasibility of adding an edge depends significantly on the rest of the layout and the typeof rooms (few front doors open into a bathroom, and direct connections between rooms on the firstand third floors are unlikely); Traditional metrics and random graph neural networks are both unlikelyto be able to capture these complex inter-dependencies functioning very differently across domains,and it similarly seems quite unlikely that pretraining on some “graph ImageNet” would be able tofind useful generic graph features.
Revised: On molecular graphs, addingany single edge will almost always violate physical constraints about stable atomic bonds; for houselayouts, the feasibility of adding an edge depends significantly on the rest of the layout and the typeof rooms (few front doors open into a bathroom, and direct connections between rooms on the firstand third floors are unlikely); Traditional metrics and random graph neural networks are both unlikelyto be able to capture these complex inter-dependencies functioning very differently across domains,and it similarly seems quite unlikely that pretraining on some “graph ImageNet” would be able tofind useful generic graph features.

Original: We therefore propose to train graph encoders on the same data used to train the generative model,using self-supervised contrastive learning to find meaningful feature extractors [19, 20, 35, 42, 45,50, 55–58], which we then use to compare the generated graphs to a test set.
Revised: We therefore propose to train graph encoders on the same data used to train the generative model,using self-supervised contrastive learning to find meaningful feature extractors [19, 20, 35, 42, 45,49, 54–57], which we then use to compare the generated graphs to a test set.

Original: The set of perturbationsintroduced in contrastive learning teaches the model which kinds of graphs should be consideredsimilar to one another.
Revised: The set of perturbationsintroduced in contrastive learning teaches the model which kinds of graphs should be consideredsimilar to one another.

Original: In this work we use types of perturbations traditionally used for trainingcontrastive learning methods on graphs, but point out that future work focusing on domain-specificmodeling can directly incorporate knowledge about which graphs should be considered similar bychoosing different perturbation sets.
Revised: In this work we use types of perturbations traditionally used for trainingcontrastive learning methods on graphs, but point out that future work focusing on domain-specificmodeling can directly incorporate knowledge about which graphs should be considered similar bychoosing different perturbation sets.

Original: Inspired by the theoretical results for contrastive learning of [52], we also propose two variations to ourrepresentation learning procedures.
Revised: Inspired by the theoretical results for contrastive learning of [51], we also propose two variations to ourrepresentation learning procedures.

Original: This upper bound shows that contrastively learned representationswork well for downstream tasks as long as the probability of data points yielding overlappingaugmentations is relatively large for within-class dataset pairs and relatively small for cross-classpairs.
Revised: This upper bound shows that contrastively learned representationswork well for downstream tasks as long as the probability of data points yielding overlappingaugmentations is relatively large for within-class dataset pairs and relatively small for cross-classpairs.

Original: Edit distances between graphs on typical training sets are large, however; we propose to usesubgraphs (“crops”) in our set of data augmentations for contrastive learning, which is more likelyto yield overlaps.
Revised: Edit distances between graphs on typical training sets are large, however; we propose to usesubgraphs (“crops”) in our set of data augmentations for contrastive learning, which is more likelyto yield overlaps.

Original: We also suggest enforcing a layer-wise Lipschitz constraint on feature extractors,which encourages similar graphs to have similar learned representations.
Revised: We also suggest enforcing a layer-wise Lipschitz constraint on feature extractors,which encourages similar graphs to have similar learned representations.

Original: We show experimentallythat both changes improve learning.
Revised: We show experimentallythat both changes improve learning.

Original: With all of these improvements, we further ask: can we theoretically guarantee that our learned GNNrepresentations outperform traditional local metrics?
Revised: With all of these improvements, we further ask: can we theoretically guarantee that our learned GNNrepresentations outperform traditional local metrics?

Original: We prove that we cannot: we give examplesof graphs easily distinguishable by local metrics that first-order GNNs cannot distinguish.
Revised: We prove that we cannot: we give examplesof graphs easily distinguishable by local metrics that first-order GNNs cannot distinguish.

Original: Yet theconverse is also true: we show graphs easily distinguishable by GNNs that appear equivalent to localmetrics.
Revised: Yet theconverse is also true: we show graphs easily distinguishable by GNNs that appear equivalent to localmetrics.

Original: We thus propose to use models based on Graph Substructure Networks (GSNs) [6], (usingnode degrees, and node clustering coefficients), and as a result, explicitly incorporating local metricsinto our models and surpassing the power of the WL test, and yield further improvements.
Revised: We thus propose to use models based on Graph Substructure Networks (GSNs) [6], (usingnode degrees, and node clustering coefficients), and as a result, explicitly incorporating local metricsinto our models and surpassing the power of the WL test, and yield further improvements.

Original:  Related Work
Revised:  Related Work

Original: Graph generative models.
Revised: Graph generative models.

Original: Our work is not particular to any type of graph generative model, asit focuses on simply evaluating samples; nonetheless, it is worth briefly reviewing some methods.
Revised: Our work is not particular to any type of graph generative model, asit focuses on simply evaluating samples; nonetheless, it is worth briefly reviewing some methods.

Original: Sequential generation models, such as GraphRNN [54] and GRAN [26], generate nodes and edges inan auto-regressive manner.
Revised: Sequential generation models, such as GraphRNN [53] and GRAN [26], generate nodes and edges inan auto-regressive manner.

Original: These models are efficient, but can struggle to capture global properties ofthe graphs, and require a predefined ordering.
Revised: These models are efficient, but can struggle to capture global properties ofthe graphs, and require a predefined ordering.

Original: One-shot models such as MolGAN [10], GraphDeconv[13], GraphVAE [24], and GraphNVP [28], on the other hand, generate all nodes and edges in oneshot; they can model long-range dependencies, but generally are not efficient and cannot scale tolarge graphs.
Revised: One-shot models such as MolGAN [10], GraphDeconv[13], GraphVAE [24], and GraphNVP [28], on the other hand, generate all nodes and edges in oneshot; they can model long-range dependencies, but generally are not efficient and cannot scale tolarge graphs.

Original: For a detailed overview of graph generative models, see [17].
Revised: For a detailed overview of graph generative models, see [17].

Original: Evaluating graph generative models.
Revised: Evaluating graph generative models.

Original: Graph generative models can be even more challenging toevaluate than visual or textual models, because it is generally more difficult for humans to judge thequality of a generated graph.
Revised: Graph generative models can be even more challenging toevaluate than visual or textual models, because it is generally more difficult for humans to judge thequality of a generated graph.

Original: The classic measure of the quality of a generative model, likelihood,also has significant issues with graphs: in addition to the kinds of issues that appear in generative
Revised: The classic measure of the quality of a generative model, likelihood,also has significant issues with graphs: in addition to the kinds of issues that appear in generative

Original: [Figure]
Revised: [Figure]

Original: models of images [46], the likelihood is particularly hard to evaluate on graphs where even checkingequality is quite difficult [34].
Revised: models of images [46], the likelihood is particularly hard to evaluate on graphs where even checkingequality is quite difficult [34].

Original: The most common method is to compute the Maximum Mean
Revised: The most common method is to compute the Maximum Mean

Original: Discrepancy (MMD) [15] between distributions of local graph statistics, including node degrees,cluster coefficients, and counts of orbits up to a particular size [26, 54].
Revised: Discrepancy (MMD) [15] between distributions of local graph statistics, including node degrees,cluster coefficients, and counts of orbits up to a particular size [26, 53].

Original: Global statistics, such aseigenvalues of the normalized graph Laplacian, are also used [54].
Revised: Global statistics, such aseigenvalues of the normalized graph Laplacian, are also used [53].

Original: These metrics, however, focusonly on low-level structure of the graph, and ignore any features that might be present on nodes oredges [47].
Revised: These metrics, however, focusonly on low-level structure of the graph, and ignore any features that might be present on nodes oredges [47].

Original: Choosing an appropriate kernel is also very important to consistency of these metrics[34].
Revised: Choosing an appropriate kernel is also very important to consistency of these metrics[34].

Original: These types of graph statistics might also encourage models to overfit to the training data, ratherthan truly learning the target distribution [41].
Revised: These types of graph statistics might also encourage models to overfit to the training data, ratherthan truly learning the target distribution [41].

Original: For unlabeled image-based generative models, most work focuses on metrics including the Inception Score (IS) [39], Fréchet Inception Distance (FID) [21], Precision/Recall (PR) [38], and
Revised: For unlabeled image-based generative models, most work focuses on metrics including the Inception Score (IS) [39], Fréchet Inception Distance (FID) [21], Precision/Recall (PR) [38], and

Original: Density/Coverage [33], all of which compare distributions in a fixed latent space (typically activationsof a late layer in an InceptionV3 [44] model trained on ImageNet).
Revised: Density/Coverage [33], all of which compare distributions in a fixed latent space (typically activationsof a late layer in an InceptionV3 [44] model trained on ImageNet).

Original: These methods are rarely adoptedin graph generative models, due to challenges with “general-purpose” graph models discussed in
Revised: These methods are rarely adoptedin graph generative models, due to challenges with “general-purpose” graph models discussed in

Original: Section 1.
Revised: Section 1.

Original: Thompson et al.
Revised: Thompson et al.

Original: [47] thus used random (untrained) graph encoders in these metrics.
Revised: [47] thus used random (untrained) graph encoders in these metrics.

Original: We discuss these methods in more detail in the appendix.
Revised: We discuss these methods in more detail in the appendix.

Original: Our work, inspired by Morozov et al.
Revised: Our work, inspired by Morozov et al.

Original: ’ssimilar proposal in image domains [32], explores the use of self-supervised contrastive training tofind representations that work better than random initializations.
Revised: ’ssimilar proposal in image domains [32], explores the use of self-supervised contrastive training tofind representations that work better than random initializations.

Original: Graph Contrastive Learning.
Revised: Graph Contrastive Learning.

Original: A popular method for self-supervised learning, contrastive learninggenerally aims to find a representation roughly invariant to various operations (e.g. for images, takingrandom crops, horizontal flipping, shifting colors) but able to identify different source data points.
Revised: A popular method for self-supervised learning, contrastive learninggenerally aims to find a representation roughly invariant to various operations (e.g. for images, takingrandom crops, horizontal flipping, shifting colors) but able to identify different source data points.

Original: Ideally, such a representation will be useful for downstream tasks not known when learning therepresentation.
Revised: Ideally, such a representation will be useful for downstream tasks not known when learning therepresentation.

Original: In graph settings, learned representations may be at the node, edge, or graph levels.
Revised: In graph settings, learned representations may be at the node, edge, or graph levels.

Original: DGI [50] and InfoGraph [42] adopt DeepInfoMax [22], and enforce consistency between local(node) and global (graph) representation.
Revised: DGI [49] and InfoGraph [42] adopt DeepInfoMax [22], and enforce consistency between local(node) and global (graph) representation.

Original: MVGRL [19] augments a graph via graph diffusion,and constructs two views by randomly sampling sub-graphs from the adjacency and diffusionmatrices.
Revised: MVGRL [19] augments a graph via graph diffusion,and constructs two views by randomly sampling sub-graphs from the adjacency and diffusionmatrices.

Original: GCC [35] uses sub-graph instance discrimination and contrasts sub-graphs from a multihop ego network.
Revised: GCC [35] uses sub-graph instance discrimination and contrasts sub-graphs from a multihop ego network.

Original: GraphCL [56] uses trial-and-error to hand-pick graph augmentations and thecorresponding parameters of each augmentation.
Revised: GraphCL [55] uses trial-and-error to hand-pick graph augmentations and thecorresponding parameters of each augmentation.

Original: JOAO [55] extends the GraphCL using a bi-levelmin-max optimization that learns to select the augmentations.
Revised: JOAO [54] extends the GraphCL using a bi-levelmin-max optimization that learns to select the augmentations.

Original: LG2AR [20] learns a policy to sampleaugmentations.
Revised: LG2AR [20] learns a policy to sampleaugmentations.

Original: GRACE [57] uses a similar approach to GraphCL to learn node representations.
Revised: GRACE [56] uses a similar approach to GraphCL to learn node representations.

Original: GCA [58] uses a set of heuristics to adaptively pick the augmentation parameters.
Revised: GCA [57] uses a set of heuristics to adaptively pick the augmentation parameters.

Original: BGRL [45] adopts
Revised: BGRL [45] adopts

Original: BYOL [16] and uses random augmentations to learn node representations.
Revised: BYOL [16] and uses random augmentations to learn node representations.

Original:  GNNs Versus Local Metrics
Revised:  GNNs Versus Local Metrics

Original: Different methods for understanding graphs can “understand” the difference between graphs in verydifferent ways: a particular change to a graph might barely affect some features, while drasticallychanging others.
Revised: Different methods for understanding graphs can “understand” the difference between graphs in verydifferent ways: a particular change to a graph might barely affect some features, while drasticallychanging others.

Original: One extreme case is when a given metric cannot detect that two distinct (nonisomorphic) graphs are different.
Revised: One extreme case is when a given metric cannot detect that two distinct (nonisomorphic) graphs are different.

Original: Since graph isomorphism is a computationally difficult problem, we expect that all efficiently computable graph representations “collapse” some pairs of input graphs.
Revised: Since graph isomorphism is a computationally difficult problem, we expect that all efficiently computable graph representations “collapse” some pairs of input graphs.

Original: 1 Itis conceivable, however, that one method could be strictly more powerful than another.
Revised: 1 Itis conceivable, however, that one method could be strictly more powerful than another.

Original: For instance,since recent GNN models have overcome traditional models based on local metric representationsin a variety of problems [24, 48, 53], is it the case that GNNs are strictly more powerful than localmetrics?
Revised: For instance,since recent GNN models have overcome traditional models based on local metric representationsin a variety of problems [24, 48, 52], is it the case that GNNs are strictly more powerful than localmetrics?

Original: We show, constructively, that the answer is no: there are indeed graphs which GNNs can distinguishand local metrics cannot, but there are also graphs which local metrics can distinguish but first-order
Revised: We show, constructively, that the answer is no: there are indeed graphs which GNNs can distinguishand local metrics cannot, but there are also graphs which local metrics can distinguish but first-order

Original: GNNs cannot.
Revised: GNNs cannot.

Original: Figure 1(a) shows a pair of graphs with the same degree distribution, clustering coefficient, and four-node orbits, which can nonetheless be distinguished by GNNs (proof in Appendix B).
Revised: Figure 1(a) shows a pair of graphs with the same degree distribution, clustering coefficient, and four-node orbits, which can nonetheless be distinguished by GNNs (proof in Appendix B).

Original: On the other hand, the graphs in Fig. 1(b) have different clustering coefficient and smallest cycle,but first-order GNNs cannot tell them apart.
Revised: On the other hand, the graphs in Fig. 1(b) have different clustering coefficient and smallest cycle,but first-order GNNs cannot tell them apart.

Original: Thus, neither method strictly outperforms the other onall problems, and so there are theoretical generative models which perfectly match in GNN-basedrepresentations but differ in local metrics, and vice versa.
Revised: Thus, neither method strictly outperforms the other onall problems, and so there are theoretical generative models which perfectly match in GNN-basedrepresentations but differ in local metrics, and vice versa.

Original: This motivates our addition of local featuresto our graph representation models (Section 4).
Revised: This motivates our addition of local featuresto our graph representation models (Section 4).

Original: It is much easier to incorporated such hard-coded structures into GNNs than it would be to addlearning to feature metrics; in particular, counting higher-order local patterns quickly becomesprohibitively expensive, with a super-exponential time complexity.
Revised: It is much easier to incorporated such hard-coded structures into GNNs than it would be to addlearning to feature metrics; in particular, counting higher-order local patterns quickly becomesprohibitively expensive, with a super-exponential time complexity.

Original: GNNs can also easily handle nodeand/or edge features on the underlying graphs, which is far more difficult to add to local metrics.
Revised: GNNs can also easily handle nodeand/or edge features on the underlying graphs, which is far more difficult to add to local metrics.

Original: Another quality we would like in our graph representations, in addition to the ability to distinguishdistinct graphs, is some form of stability: if a distribution of graphs only changes slightly, we wouldlike our evaluation methods to give “partial credit” as opposed to a distribution where all graphsare dramatically different.
Revised: Another quality we would like in our graph representations, in addition to the ability to distinguishdistinct graphs, is some form of stability: if a distribution of graphs only changes slightly, we wouldlike our evaluation methods to give “partial credit” as opposed to a distribution where all graphsare dramatically different.

Original: (This is closely related to issues in training and evaluating image-basedgenerative models [1–3, 46].)
Revised: (This is closely related to issues in training and evaluating image-basedgenerative models [1–3, 46].)

Original: As previously discussed, the notion of “similar graphs” is very domaindependent, but traditional local metrics can be highly sensitive to changes like adding a single edge:
Revised: As previously discussed, the notion of “similar graphs” is very domaindependent, but traditional local metrics can be highly sensitive to changes like adding a single edge:

Original: Fig.
Revised: Fig.

Original: 1(c) shows an example where two graphs differing only by a single edge can have drasticallydifferent statistics.
Revised: 1(c) shows an example where two graphs differing only by a single edge can have drasticallydifferent statistics.

Original: By learning GNN representations, we can have some control over these types ofsmoothness properties; we exploit this explicitly in our methodology in Section 4.
Revised: By learning GNN representations, we can have some control over these types ofsmoothness properties; we exploit this explicitly in our methodology in Section 4.

Original:  Self-supervised Training of Graph Representations for Evaluation
Revised:  Self-supervised Training of Graph Representations for Evaluation

Original: Suppose we have two sets of graphs G train =
Revised: Suppose we have two sets of graphs G train =

Original: { G 1 , . . .
Revised: { G 1 , . . .

Original: G S } and G test =
Revised: G S } and G test =

Original: { G 1 , . . .
Revised: { G 1 , . . .

Original: G M } , eachsampled from the same data distribution p ( G ) .
Revised: G M } , eachsampled from the same data distribution p ( G ) .

Original: Also suppose that we have access to an unconditionalgraph generative model g ϕ ( . ) , which is trained on G train to learn the distribution of the observed setof graphs.
Revised: Also suppose that we have access to an unconditionalgraph generative model g ϕ ( . ) , which is trained on G train to learn the distribution of the observed setof graphs.

Original: We sample a set of generated graphs G gen = { G 1 , . . .
Revised: We sample a set of generated graphs G gen = { G 1 , . . .

Original: G N } ∼ p g ϕ ( G ) from this model.
Revised: G N } ∼ p g ϕ ( G ) from this model.

Original: In order to evaluate the quality of the sampled graphs (i.e., to decide whether the model g ϕ ( . ) hassuccessfully recovered the underlying distribution p ( G ) ), we can define a measure of divergence
Revised: In order to evaluate the quality of the sampled graphs (i.e., to decide whether the model g ϕ ( . ) hassuccessfully recovered the underlying distribution p ( G ) ), we can define a measure of divergence

Original: D ( G test , G gen ) to quantify the discrepancy between distributions of the real and generated graphs.
Revised: D ( G test , G gen ) to quantify the discrepancy between distributions of the real and generated graphs.

Original: One robust way to achieve this is to define the metric on latent vector representation spaces, andexpect representations of graphs rather than the original objects.
Revised: One robust way to achieve this is to define the metric on latent vector representation spaces, andexpect representations of graphs rather than the original objects.

Original: Thus, to use these metrics, we needto train a shared encoder f θ ( . ) and then compute the discrepancy as D ( f θ ( G test  , f θ ( G gen ) Thereare a few such metrics well-studied in visual domains that can differentiate the fidelity and diversityof the model, and which we can adopt in graph domains.
Revised: Thus, to use these metrics, we needto train a shared encoder f θ ( . ) and then compute the discrepancy as D ( f θ ( G test ) , f θ ( G gen )) . Thereare a few such metrics well-studied in visual domains that can differentiate the fidelity and diversityof the model, and which we can adopt in graph domains.

Original: For evaluating image generative models, due to similar feature space across image datasets and alsoavailability of large-scale data, the trunk of a model trained over ImageNet with explicit supervisorysignals is usually chosen as the encoder.
Revised: For evaluating image generative models, due to similar feature space across image datasets and alsoavailability of large-scale data, the trunk of a model trained over ImageNet with explicit supervisorysignals is usually chosen as the encoder.

Original: However, it is not straightforward to adopt the same trick tograph-structured data: there are no ImageNet-scale graph datasets available, and more importantly,the semantics of graphs and their features vary wildly across commonly used graph datasets, far morethan occurs across distributions of natural images.
Revised: However, it is not straightforward to adopt the same trick tograph-structured data: there are no ImageNet-scale graph datasets available, and more importantly,the semantics of graphs and their features vary wildly across commonly used graph datasets, far morethan occurs across distributions of natural images.

Original: For instance, even datasets of molecular graphsmay use different feature sets to represent the molecules [18], in addition to the many cross-domainchallenges discussed in Section 1.
Revised: For instance, even datasets of molecular graphsmay use different feature sets to represent the molecules [18], in addition to the many cross-domainchallenges discussed in Section 1.

Original: Thus, it is not feasible to imagine a single “universal” graphrepresentation; we would like a general-purpose method for finding representations useful for a newgraph dataset.
Revised: Thus, it is not feasible to imagine a single “universal” graphrepresentation; we would like a general-purpose method for finding representations useful for a newgraph dataset.

Original: [Figure]
Revised: [Figure]

Original: Training with Graph Contrastive Learning To find expressive representation of real and generated graphs, we train the encoder using a contrastive objective.
Revised: Training with Graph Contrastive Learning To find expressive representation of real and generated graphs, we train the encoder using a contrastive objective.

Original: Assuming a set of rational augmentations T over G where each augmentation τ i ∈ T is defined as a function over graph G k that generatesan identity-preserving view of the graph: G + k
Revised: Assuming a set of rational augmentations T over G where each augmentation τ i ∈ T is defined as a function over graph G k that generatesan identity-preserving view of the graph: G + k

Original: = τ i ( G k ) , a contrastive framework with negativesampling strategy uses T to draw positive samples from the joint distribution p ( τ i ( G k ) , τ j ( G k )) inorder to maximize the agreement between different views of the same graph G k and to draw negativesamples from the product of marginals p ( τ i ( G k ))
Revised: = τ i ( G k ) , a contrastive framework with negativesampling strategy uses T to draw positive samples from the joint distribution p ( τ i ( G k ) , τ j ( G k )) inorder to maximize the agreement between different views of the same graph G k and to draw negativesamples from the product of marginals p ( τ i ( G k ))

Original: × p ( τ j ( G k ′ )) to minimize it for views from twodistinct graphs G k and G k ′ , k ̸ = k ′ .
Revised: × p ( τ j ( G k ′ )) to minimize it for views from twodistinct graphs G k and G k ′ , k ̸ = k ′ .

Original: As mentioned previously, we use a GIN architecture for our feature extractor.
Revised: As mentioned previously, we use a GIN architecture for our feature extractor.

Original: Other than the trainingprocess, the rest of our evaluation pipeline is similar to that of Thompson et al.
Revised: Other than the trainingprocess, the rest of our evaluation pipeline is similar to that of Thompson et al.

Original: [47], who userandom GIN weights.
Revised: [47], who userandom GIN weights.

Original: We consider two methods for training our GIN’s parameters: GraphCL [56]and InfoGraph [43].
Revised: We consider two methods for training our GIN’s parameters: GraphCL [55]and InfoGraph [43].

Original: GraphCL randomly samples augmentation from four possible augmentaiotnsincluding node dropping , edge perturbation , attribute masking , and sub-graph inducing based ona random walk to which we would like the representation to be roughly invariant..
Revised: GraphCL randomly samples augmentation from four possible augmentaiotnsincluding node dropping , edge perturbation , attribute masking , and sub-graph inducing based ona random walk to which we would like the representation to be roughly invariant.

Original: GraphCL usesnormalized temperature-scaled cross-entropy (NT-Xent) objective [7] to maximize the agreementbetween positive graph-level representations.
Revised: GraphCL usesnormalized temperature-scaled cross-entropy (NT-Xent) objective [7] to maximize the agreementbetween positive graph-level representations.

Original: InfoGraph works differently: it contrasts the graph-levelrepresentation with the representations of individual nodes, which encode neighborhood structureinformation.
Revised: InfoGraph works differently: it contrasts the graph-levelrepresentation with the representations of individual nodes, which encode neighborhood structureinformation.

Original: InfoGraph uses DeepInfoMax [22] objective to maximize the mutual informationbetween graph-level and node-level representations of each individual graph.
Revised: InfoGraph uses DeepInfoMax [22] objective to maximize the mutual informationbetween graph-level and node-level representations of each individual graph.

Original: Using Local Subgraph Counts as Input Features for GINs To build on the insight of Section 3,we also consider various methods for adding information about the local graph structure as nodefeatures, similarly to Graph Substructure Networks [6].
Revised: Using Local Subgraph Counts as Input Features for GINs To build on the insight of Section 3,we also consider various methods for adding information about the local graph structure as nodefeatures, similarly to Graph Substructure Networks [6].

Original: The simplest such method is to add the degreeof a node as an explicit feature for the GIN to consider.
Revised: The simplest such method is to add the degreeof a node as an explicit feature for the GIN to consider.

Original: We do this, but also add, by concatenation onnode features, higher-order local information as well: three-node and four-node clustering featuresfor each node.
Revised: We do this, but also add, by concatenation onnode features, higher-order local information as well: three-node and four-node clustering featuresfor each node.

Original: Four node clustering coefficient is calculated as:
Revised: Four node clustering coefficient is calculated as:

Original: [Equation]
Revised: [Equation]

Original: where q v ( u, w ) defined as the number of common neighbors of u and w , not counting v .
Revised: where N ( v ) denotes immediate neighbors of node v , deg ( v ) denotes the degree of node v , andq v ( u, w ) is the number of common neighbors of u and w , not counting v .

Original: Aggregatingthese features across the whole graph would give information on distribution of 4 node orbits of thegraph, but this provides more localized information across the graph that is nonetheless difficult orimpossible for a GIN to examine otherwise [8].
Revised: Aggregating these featuresacross the whole graph would give information on distribution of 4 node orbits of the graph, but thisprovides more localized information across the graph that is nonetheless difficult or impossible for a GIN to examine otherwise [8].

Original: Choice of Augmentations Wang et al.
Revised: Choice of Augmentations Wang et al.

Original: [52], building off work of Wang and Isola [51], studycontrastive learning in general settings (with an eye towards vision applications), and providean intriguing bound for the performance of downstream classifiers.
Revised: [51], building off work of Wang and Isola [50], studycontrastive learning in general settings (with an eye towards vision applications), and providean intriguing bound for the performance of downstream classifiers.

Original: To explain it, consider the“augmentation graph” of the training samples: if G i is the i th training example and G + i is a randomaugmentation of that example, we connect the nodes G i and G j in the augmentation graph if thereare feasible augmentations G + i and G + j such that G + i
Revised: To explain it, consider the“augmentation graph” of the training samples: if G i is the i th training example and G + i is a randomaugmentation of that example, we connect the nodes G i and G j in the augmentation graph if thereare feasible augmentations G + i and G + j such that G + i

Original: =
Revised: =

Original: G + j . 2 Wang et al.
Revised: G + j . 2 Wang et al.

Original: argue that downstreamlinear classifiers are likely to succeed if this augmentation graph has stronger intra-class connectivitythan it does cross-class connectivity, proving a tight connection between the two under a particularsetting with strong assumptions about this and related aspects of the setup.
Revised: argue that downstreamlinear classifiers are likely to succeed if this augmentation graph has stronger intra-class connectivitythan it does cross-class connectivity, proving a tight connection between the two under a particularsetting with strong assumptions about this and related aspects of the setup.

Original: Given the connection between the distribution metrics discussed in Appendix A.2 and classifier performance [e.g. 27, Section 4], if we accept the argument of Wang et al.
Revised: Given the connection between the distribution metrics discussed in Appendix A.2 and classifier performance [e.g. 27, Section 4], if we accept the argument of Wang et al.

Original: [52], evaluation metrics basedon a contrastively trained graph representation will give poor values (good classifier performance)when generated samples are not well-connected to real samples in the augmentation graph, and viceversa.
Revised: [51], evaluation metrics basedon a contrastively trained graph representation will give poor values (good classifier performance)when generated samples are not well-connected to real samples in the augmentation graph, and viceversa.

Original: If we choose augmentations appropriately, this is sensible behavior.
Revised: If we choose augmentations appropriately, this is sensible behavior.

Original: Enforcing Lipschitz Layers in Representation Networks
Revised: Enforcing Lipschitz Layers in Representation Networks

Original: The prior line of reasoning also suggeststhat we should choose augmentations that are capable of making real graphs look like one another.
Revised: The prior line of reasoning also suggeststhat we should choose augmentations that are capable of making real graphs look like one another.

Original: Edit distances between graphs, however, are typically large on the datasets we consider, and soaugmentations based on adding or deleting individual nodes and/or edges will struggle to do this.
Revised: Edit distances between graphs, however, are typically large on the datasets we consider, and soaugmentations based on adding or deleting individual nodes and/or edges will struggle to do this.

Original: Thesame is true for many of the augmentations used on images, except – as Wang et al.
Revised: Thesame is true for many of the augmentations used on images, except – as Wang et al.

Original: [52] note – forcrop-type augmentations, where e.g. two different car pictures might become quite similar if we cropto just a wheel.
Revised: [51] note – forcrop-type augmentations, where e.g. two different car pictures might become quite similar if we cropto just a wheel.

Original: On graphs, an analogous operation is subgraph sampling, which we include in our
Revised: On graphs, an analogous operation is subgraph sampling, which we include in our

Original: GraphCL setup; InfoGraph already naturally looks at subgraph features as a core component.
Revised: GraphCL setup; InfoGraph already naturally looks at subgraph features as a core component.

Original: Taking this line of reasoning as well as the general motivations of contrastive learning further, it isalso natural to think that if we can inherently enforce “similar graphs” to have similar representations,this could improve the process of contrastive learning: we would save on needing to train the modelto learn these similarities, and it could help decrease the classifier performance for good generativemodels whose output graphs are legitimately near the distribution of target graphs.
Revised: Taking this line of reasoning as well as the general motivations of contrastive learning further, it isalso natural to think that if we can inherently enforce “similar graphs” to have similar representations,this could improve the process of contrastive learning: we would save on needing to train the modelto learn these similarities, and it could help decrease the classifier performance for good generativemodels whose output graphs are legitimately near the distribution of target graphs.

Original: A line of work on GANs in visual settings [1–3, 29, 30, 36] has made clear the importance of this typeof reasoning in losses for training generative models: the loss should smoothly improve as generatorsamples approach the target distribution, even if the supports differ.
Revised: A line of work on GANs in visual settings [1–3, 29, 30, 36] has made clear the importance of this typeof reasoning in losses for training generative models: the loss should smoothly improve as generatorsamples approach the target distribution, even if the supports differ.

Original: Viewing model evaluation metricsas a kind of “out-of-the-loop” loss function for training generative models – hyperparameter selectionand model development focusing on variants with better evaluation metrics – suggests that thesekinds of properties may be important for the problem of evaluation as well.
Revised: Viewing model evaluation metricsas a kind of “out-of-the-loop” loss function for training generative models – hyperparameter selectionand model development focusing on variants with better evaluation metrics – suggests that thesekinds of properties may be important for the problem of evaluation as well.

Original: We thus explicitly enforce the layers of our GIN to have a controlled Lipschitz constant, similarly toe.g.
Revised: We thus explicitly enforce the layers of our GIN to have a controlled Lipschitz constant, similarly toe.g.

Original: spectral normalization in GAN discriminators [30].
Revised: spectral normalization in GAN discriminators [30].

Original: To this end, we fix the λ Lipschitz factor to1 . 

Revised: To this end, we fix the λ Lipschitz factor to1 .

Original: in the experiments.
Revised: in the experiments.

Original: For each linear layer with weights W ℓ , we use projected gradient descent;after each update on the weights, if ∥ W ℓ ∥ > 1 . 0 , we update the value of the weights to W ℓ = W ℓ ∥ W ℓ ∥ .
Revised: For each linear layer with weights W ℓ , we use projected gradient descent;after each update on the weights, if ∥ W ℓ ∥ > 1 . 0 , we update the value of the weights to W ℓ = W ℓ ∥ W ℓ ∥ .

Original: This guarantees small changes in the graphs, such as adding/removing edges, or change in the inputfeatures, will not drastically change the final representation.
Revised: This guarantees small changes in the graphs, such as adding/removing edges, or change in the inputfeatures, will not drastically change the final representation.

Original:  Experimental Results
Revised:  Experimental Results

Original: In all of the experiments we train the model on the full dataset in an self-supervised manner.
Revised: In all of the experiments we train the model on the full dataset in an self-supervised manner.

Original: Following[47], we take the dataset and make perturbations on the dataset and see what is the trend in themeasurements as the perturbation degree increases.
Revised: Following[47], we take the dataset and make perturbations on the dataset and see what is the trend in the measurements as the perturbation degree increases.

Original: We denote perturbation degree with r , and defineit for each type of perturbation.
Revised: We denote perturbation degree with r [Figure] and defineit for each type of perturbation.

Original: We use these type of perturbations:
Revised: We use these type of perturbations:

Original: [Figure]
Revised: [Figure]

Original: structural features.
Revised: [Figure]

Original: • Mixing-Random:
Revised: • Mixing-Random:

Original: In perturbation with ratio r , we remove r chunk of the reference samples,and replace them with Erd ˝ os–Rényi (ER) graphs with the same ratio of edges.• Rewiring Edges: This perturbation, rewires each edge of the graph with probability r .
Revised: In perturbation with ratio r , we remove r chunk of the reference samples,and replace them with Erd ˝ os–Rényi (ER) graphs with the same ratio of edges.• Rewiring Edges: This perturbation, rewires each edge of the graph with probability r .

Original: Eachrewired edge, will change one of the sides of the edge with equal probability to another nodethat is not already connected to the stable node.• Mode Collapse and Mode Dropping:
Revised: Each rewired edge, will change one of the sides of the edge with equal probability to another nodethat is not already connected to the stable node. • Mode Collapse and Mode Dropping:

Original: For these perturbation, first we cluster the graphsusing the WL-Kernel.
Revised: For these perturbation, first we cluster the graphsusing the WL-Kernel.

Original: First, we choose r ratio of clusters, then, for mode collapse, wereplace each graph on that dataset with the center of the cluster.
Revised: First, we choose r ratio of clusters, then, for mode collapse, wereplace each graph on that dataset with the center of the cluster.

Original: For, mode dropping, weremove the selected clusters and then for making size of the dataset fixed, we randomlyrepeat samples from other clusters.
Revised: For, mode dropping, weremove the selected clusters and then for making size of the dataset fixed, we randomlyrepeat samples from other clusters.

Original: For each experiment, we measure the Spearman rand correlation between the perturbation ratio, r ,and the value of the measurement.
Revised: For each experiment, we measure the Spearman rand correlation between the perturbation ratio, r ,and the value of the measurement.

Original: For measurements that supposed to decrease by the increase ofratio, we flip the values.
Revised: For measurements that supposed to decrease by the increase ofratio, we flip the values.

Original: As a result, in all experiments higher is better.
Revised: As a result, in all experiments higher is better.

Original: We gather the results amongdifferent datasets and several random seeds and plot them for distribution of the correlations.
Revised: We gather the results amongdifferent datasets and several random seeds and plot them for distribution of the correlations.

Original: Fordetailed experiments on the individual datasets see  D.
Revised: Fordetailed experiments on the individual datasets see Appendix D.

Original: Datasets: Following [47], we use six diverse datasets ( three synthetic and three real-world) that arefrequently used in literature including: (1) Lobster [9] consisting of stochastic graphs with each nodeat most 2-hops away from a backbone path, (2) 2D Grid graphs [9, 26, 54], (3) Proteins [11] whererepresents connectivity among amino acids if their physical distance is less than a threshold, (4) 3-hop
Revised: Datasets: Following [47], we use six diverse datasets ( three synthetic and three real-world) that arefrequently used in literature including: (1) Lobster [9] consisting of stochastic graphs with each nodeat most 2-hops away from a backbone path, (2) 2D Grid graphs [9, 26, 53], (3) Proteins [11] whererepresents connectivity among amino acids if their physical distance is less than a threshold, (4) 3-hop

Original: Ego networks [54] extracted from the CiteSeer network [40] representing citation-based connectivityamong documents, (5) Community [54] graphs generated by Erd˝os–Rényi model [12], and (6) Zinc[23] is a dataset of attributed molecular graphs which allows to study sensitivity of metrics to changesin node/edge feature distributions.
Revised: Ego networks [53] extracted from the CiteSeer network [40] representing citation-based connectivityamong documents, (5) Community [53] graphs generated by Erd˝os–Rényi model [12], and (6) Zinc[23] is a dataset of attributed molecular graphs which allows to study sensitivity of metrics to changes in node/edge feature distributions.

Original: We follow the exact protocols used in [9, 26, 47, 54] as follows.
Revised: We follow the exact protocols used in [9, 26, 47, 53] as follows.

Original: We randomly sample 1000 graphs from Zinc dataset and use one-hot encoding for edge/node features
Revised: We randomly sample 1000 graphs from Zinc dataset and use one-hot encoding for edge/node features.

Original: For community dataset, we set n = |V| / 2 , p = 0 . 3 , and add 0 .
Revised: For community dataset, we set n = |V| / 2 , p = 0 . 3 , and add 0 .

Original: 05 |V| inter-community edges withuniform probability.
Revised: 05 |V| inter-community edges withuniform probability.

Original: For all datasets, we use the node size range indicated in Table 1 in Appendix.
Revised: For all datasets, we use the node size range indicated in Table 4 in Appendix.

Original: Contrastive Training Versus Random GIN without Structural Features: In the first experiment,we compare the methods without adding structural features.
Revised: Contrastive Training Versus Random GIN without Structural Features: In the first experiment,we compare the methods without adding structural features.

Original: Results can be seen in Figure 3.
Revised: Results can be seen in Figure 3.

Original: Ingeneral we can see that in most measurements pretraining shows superior performance compared tothe random network.
Revised: Ingeneral we can see that in most measurements pretraining shows superior performance comparedto the random network.

Revised: In this figure we have used GraphCL method.

Revised: Overall InfoGraph showssimilar results.

Revised: For InfoGraph results check Appendix D.

Original: In this experiments we use 10 random seeds and gather data on all datasets.
Revised: In this experiments we use 10 randomseeds and gather data on all datasets.

Original: In  D, same results separated for each dataset can be seen.
Revised: In Appendix D, same results separated for each dataset can beseen.

Original: In our experience, pretraining shows nearperfect results on larger datasets, but for Lobster and Grid datasets correlations are not near to 1 .
Revised: In our experience, pretraining shows near perfect results on larger datasets, but for Lobster and Grid datasets correlations are not near to 1 .

Original: Ourobservation is these measurements are moving with the correct trend up to some perturbation ratio;but for example precision/recall become zero after some perturbation and keeps still.
Revised: Our observation is these measurements are moving withthe correct trend up to some perturbation ratio; but for example precision/recall become zero aftersome perturbation and keeps still.

Original: Our intuitionhere is because of highly regular structures in these datasets, model learns instantly to discrimate thereal graphs from the fake ones very easily.
Revised: Our intuition here is because of highly regular structures in thesedatasets, model learns instantly to discrimate the real graphs from the fake ones very easily.

Original: And after small amount of perturbation the perturbedgraphs are all very far from the reference graphs.
Revised: And aftersmall amount of perturbation the perturbed graphs are all very far from the reference graphs.

Original: Contrastive Training Versus Random GIN with Node Degree Features:

Original: In this experiment,similar to the first one, we compare random and pretrained models.

Original: The difference is that we use justnode degree features for the nodes, the easiest and fastest structural information that could be addedto the model.

Original: Results are provided in Figure 4.

Original: [Figure]
Revised: [Table]

Original: Contrastive Training with Structural Information of Node Clustering coefficients: In thisexperiment we concatenate node degrees and node clustering coefficients for each node feature.
Revised: Contrastive Training with Structural Information: In this experiment we concatenate nodedegrees and node clustering coefficients for each node feature.

Original: Herewe can see that adding these features improves the models power for some measurements.
Revised: Here we can see that adding thesefeatures improves the models power for some measurements.

Original: However,on small datasets Grid and Lobster results become poorer.
Revised: However, on small datasets Grid and Lobster results become poorer.

Original: Again intuition is more powerful modelcan easier distinguish main graphs from the fake ones.
Revised: Again intuition is more powerful model can easier distinguish maingraphs from the fake ones.

Original: Break down of the results on the datasets in Appendix D proves this point.
Revised: Break down of the results on the datasets in Appendix D proves this point.

Original: Ablation Study: We analyze how layer normalization and using subgraph augmentations in GraphCL reflects on the final results.
Revised: Generative Models Benchmark In this experiment, we use samples generated from the GRAN[26] after 25% of epochs, after full training, and also compare to a holdout set of the real data (half ofthe original dataset).

Original: We have used no structural feature setup and conducted
Revised: We pretrain a model with full structural features and evaluate the criteria onthem.

Original: [Figure] [Figure]
Revised: As expected, the early-training results are worse than post-training results, while the real datagives almost perfect results.

Original: k
Revised: Results are provided in Table 2.

Original: [Figure]
Revised: [Table]

Original: rr
Revised: Ablation Study: We analyze how layer normalization and using subgraph augmentations in

Original: [Figure] l a t i o n [Figure]
Revised: GraphCL reflects on the final results.

Original: the experiment for mixing-random and mode-collapse experiments.
Revised: We have used no structural feature setup and conductedthe experiment for mixing-random and mode-collapse experiments.

Original: Figure 6, shows the results onthis task.
Revised: Figure 5, shows the results onthis task.

Original: The results prove that both of these improvements are essential for getting better results.
Revised: The results prove that both of these improvements are essential for getting better results.

Revised: [Figure]

Revised: [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: We have demonstrated that self-supervised pre-training of representations can yield significantlybetter metrics for graph evaluation than random ones, particularly when incorporating local graphfeatures with Lipschitz control, as inspired by theory.
Revised: We have demonstrated that self-supervised pre-training of representations can yield significantlybetter metrics for graph evaluation than random ones, particularly when incorporating local graphfeatures with Lipschitz control, as inspired by theory.

Original: We suggest graph generative modeling papersshould consider evaluating with these metrics in addition to or instead of their existing ones.
Revised: We suggest graph generative modeling papersshould consider evaluating with these metrics in addition to or instead of their existing ones.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: of GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]