Original: T OWARDS E QUIVARIANT G RAPH C ONTRASTIVE L EARNING VIA C ROSS -G RAPH A UGMENTATION
Revised: T OWARDS E QUIVARIANT G RAPH C ONTRASTIVE L EARNING VIA C ROSS -G RAPH A UGMENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Leading graph contrastive learning (GCL) frameworks conform to the invariance mechanism by encouraging insensitivity to different augmented views of the same graph.
Revised: Leading graph contrastive learning (GCL) frameworks conform to the invariance mechanism by encouraging insensitivity to different augmented views of the same graph.

Original: Despite the promising performance, invariance worsens representation when augmentations cause aggressive semantics shifts.
Revised: Despite the promising performance, invariance worsens representation when augmentations cause aggressive semantics shifts.

Original: For example, dropping the super-node can dramatically change a social network’s topology.
Revised: For example, dropping the super-node can dramatically change a social network’s topology.

Original: In this case, encouraging invariance to the original graph can bring together dissimilar patterns and hurt the task of instance discrimination.
Revised: In this case, encouraging invariance to the original graph can bring together dissimilar patterns and hurt the task of instance discrimination.

Original: To resolve the problem, we get inspiration from equivariant self-supervised learning and propose Equivariant Graph Contrastive Learning (E-GCL) to encourage the sensitivity to global semantic shifts.
Revised: To resolve the problem, we get inspiration from equivariant self-supervised learning and propose Equivariant Graph Contrastive Learning (E-GCL) to encourage the sensitivity to global semantic shifts.

Original: Viewing each graph as a transformation to others, we ground the equivariance principle as a cross-graph augmentation – graph interpolation – to simulate global semantic shifts.
Revised: Viewing each graph as a transformation to others, we ground the equivariance principle as a cross-graph augmentation – graph interpolation – to simulate global semantic shifts.

Original: Without using annotation, we supervise the representation of cross-graph augmented views by linearly combining the representations of their original samples.
Revised: Without using annotation, we supervise the representation of cross-graph augmented views by linearly combining the representations of their original samples.

Original: This simple but effective equivariance principle empowers E-GCL with the ability of cross-graph discrimination.
Revised: This simple but effective equivariance principle empowers E-GCL with the ability of cross-graph discrimination.

Original: It shows significant improvements over the state-of-the-art GCL models in unsupervised learning and transfer learning.
Revised: It shows significant improvements over the state-of-the-art GCL models in unsupervised learning and transfer learning.

Original: Further experiments demonstrate E-GCL’s generalization to various graph pre-training frameworks.
Revised: Further experiments demonstrate E-GCL’s generalization to various graph pre-training frameworks.

Original: Code is available at https://anonymous.4open.science/r/E-GCL/
Revised: Code is available at https://anonymous.4open.science/r/E-GCL/

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph contrastive learning (GCL)
Revised: Graph contrastive learning (GCL)

Original: (You et al., 2020;
Revised: (You et al., 2020;

Original: Suresh et al., 2021; Xu et al., 2021) is a prevailing paradigm for self-supervised learning (Chen et al., 2020; Zbontar et al., 2021) on graph-structured data.
Revised: Suresh et al., 2021; Xu et al., 2021) is a prevailing paradigm for self-supervised learning (Chen et al., 2020; Zbontar et al., 2021) on graph-structured data.

Original: It typically pre-trains a graph neural network (GNN) (Dwivedi et al., 2020) without labeled data, in an effort to learn generalizable representations and boost the fine-tuning on downstream tasks.
Revised: It typically pre-trains a graph neural network (GNN) (Dwivedi et al., 2020) without labeled data, in an effort to learn generalizable representations and boost the fine-tuning on downstream tasks.

Original: The common theme across recent GCL studies is instance discrimination (Dosovitskiy et al., 2014;
Revised: The common theme across recent GCL studies is instance discrimination (Dosovitskiy et al., 2014;

Original: Purushwalkam & Gupta, 2020) — viewing each graph as a class of its own, and differing it from other graphs.
Revised: Purushwalkam & Gupta, 2020) — viewing each graph as a class of its own, and differing it from other graphs.

Original: It galvanizes representation learning to capture discriminative characteristics of graphs.
Revised: It galvanizes representation learning to capture discriminative characteristics of graphs.

Original: Towards this end, leading GCL works usually employ two key modules: graph augmentation and contrastive learning.
Revised: Towards this end, leading GCL works usually employ two key modules: graph augmentation and contrastive learning.

Original: Specifically, graph augmentation adopts the “intra-graph” strategy to create multiple augmented views of each graph, such as randomly dropping nodes (You et al., 2020) or adversarially perturbing edges (Suresh et al., 2021).
Revised: Specifically, graph augmentation adopts the “intra-graph” strategy to create multiple augmented views of each graph, such as randomly dropping nodes (You et al., 2020) or adversarially perturbing edges (Suresh et al., 2021).

Original: The views stemming from the same graph constitute the positive samples of this class, while the views of other graphs are treated as negatives.
Revised: The views stemming from the same graph constitute the positive samples of this class, while the views of other graphs are treated as negatives.

Original: Consequently, contrastive learning encourages the agreement between positive samples and the discrepancy between negatives.
Revised: Consequently, contrastive learning encourages the agreement between positive samples and the discrepancy between negatives.

Original: This procedure essentially imposes “invariance” (Purushwalkam & Gupta, 2020; Dangovski et al., 2022) upon representations — making the anchor graph’s representation invariant to its intra-graph augmentations (Figure 1a).
Revised: This procedure essentially imposes “invariance” (Purushwalkam & Gupta, 2020; Dangovski et al., 2022) upon representations — making the anchor graph’s representation invariant to its intra-graph augmentations (Figure 1a).

Original: Formally, let g be the anchor graph,
Revised: Formally, let g be the anchor graph,

Original: P be the groups of intra-graph augmentations, and ϕ ( · ) be the GNN encoder.
Revised: P be the groups of intra-graph augmentations, and ϕ ( · ) be the GNN encoder.

Original: The “invariance to intra-graph augmentations” mechanism states ϕ ( g ) =
Revised: The “invariance to intra-graph augmentations” mechanism states ϕ ( g ) =

Original: ϕ ( T p ( g )) , ∀ p ∈ P — the representation ϕ ( g ) is insensitive to the changes in augmentation p , where T p ( g ) is the action of augmentation p on graph g .
Revised: ϕ ( T p ( g )) , ∀ p ∈ P — the representation ϕ ( g ) is insensitive to the changes in augmentation p , where T p ( g ) is the action of augmentation p on graph g .

Original: We refer to works adopting this mechanism as Invariant Graph Contrastive Learning (I-GCL).
Revised: We refer to works adopting this mechanism as Invariant Graph Contrastive Learning (I-GCL).

Original: However, we argue that invariance to intra-graph augmentations alone is insufficient to improve the semantic quality of graph representations and boost the downstream performance:
Revised: However, we argue that invariance to intra-graph augmentations alone is insufficient to improve the semantic quality of graph representations and boost the downstream performance:

Original: [Table] [Table]
Revised: [Table] [Table]

Original: view1view
Revised: view1view

Original: [Table]
Revised: [Table]

Original: /eVN0r4pe9VypVkp1SqrOPJwAZdwDR7cQg3uoQEtYIDwDK/w5jw6L86787FszTmrmXP4A+fzB+udjQI=</latexit> C
Revised: /eVN0r4pe9VypVkp1SqrOPJwAZdwDR7cQg3uoQEtYIDwDK/w5jw6L86787FszTmrmXP4A+fzB+udjQI=</latexit> C

Original: [Figure]
Revised: [Figure]

Original: • Limiting the augmentations to local substructures of an individual graph is aggressive (Purushwalkam & Gupta 2020; Wang et al., 2022) insofar as the augmented views fragmentarily or even wrongly describe the characteristics of the anchor graph.
Revised: • Limiting the augmentations to local substructures of an individual graph is aggressive (Purushwalkam & Gupta, 2020; Wang et al., 2022) insofar as the augmented views fragmentarily or even wrongly describe the characteristics of the anchor graph.

Original: Take a molecule graph as an example.
Revised: Take a molecule graph as an example.

Original: After randomly dropping some nodes, one view could hold a cyano group ( -C ≡ N ) that determines the property of molecule hypertoxic, while another could corrupt this functional group.
Revised: After randomly dropping some nodes, one view could hold a cyano group ( -C ≡ N ) that determines the property of molecule hypertoxic, while another could corrupt this functional group.

Original: Thus, intra-graph augmentations are inadequate for presenting a holistic view of the anchor graph.
Revised: Thus, intra-graph augmentations are inadequate for presenting a holistic view of the anchor graph.

Original: • Worse still, aggressive augmentations easily make two positive views far from each other, but the invariance mechanism blindly forces their representations to be invariant.
Revised: • Worse still, aggressive augmentations easily make two positive views far from each other, but the invariance mechanism blindly forces their representations to be invariant.

Original: Considering the molecule graph’s views again ( cf.
Revised: Considering the molecule graph’s views again ( cf.

Original: Figure 1a), invariance-guided contrastive learning simply maximizes their representation agreement, regardless of the changes in the hypertoxic property.
Revised: Figure 1a), invariance-guided contrastive learning simply maximizes their representation agreement, regardless of the changes in the hypertoxic property.

Original: Therefore, it might amplify the negative impact of aggressive intra-graph augmentations and restrain representations from reflecting the instance semantics faithfully.
Revised: Therefore, it might amplify the negative impact of aggressive intra-graph augmentations and restrain representations from reflecting the instance semantics faithfully.

Original: To mitigate these negative influences, we get inspiration from the recent work on equivariant selfsupervised learning (E-SSL) (Dangovski et al., 2022).
Revised: To mitigate these negative influences, we get inspiration from the recent work on equivariant selfsupervised learning (E-SSL) (Dangovski et al., 2022).

Original: It splits the augmentations into two parts, to which representations should be insensitive and sensitive, and then establishes the invariance and equivariance mechanisms correspondingly.
Revised: It splits the augmentations into two parts, to which representations should be insensitive and sensitive, and then establishes the invariance and equivariance mechanisms correspondingly.

Original: The idea of “equivariance” is our focus, which makes representations aware of semantic changes caused by certain augmentations H .
Revised: The idea of “equivariance” is our focus, which makes representations aware of semantic changes caused by certain augmentations H .

Original: Here we formulate it as ϕ
Revised: Here we formulate it as ϕ

Original: ( T h ( g ))
Revised: ( T h ( g ))

Original: =
Revised: =

Original: T ′ h
Revised: T ′ h

Original: ( ϕ ( g )) , ∀ h ∈ H , where T h ( g ) and T ′ h ( ϕ ( g )) are the actions of augmentation h on graph g and representation ϕ
Revised: ( ϕ ( g )) , ∀ h ∈ H , where T h ( g ) and T ′ h ( ϕ ( g )) are the actions of augmentation h on graph g and representation ϕ

Original: ( g ) , respectively.
Revised: ( g ) , respectively.

Original: Jointly learning equivariance to sensitive augmentations
Revised: Jointly learning equivariance to sensitive augmentations

Original: H and invariance to insensitive augmentations P is promising to shield representations from the harms of aggressive augmentations.
Revised: H and invariance to insensitive augmentations P is promising to shield representations from the harms of aggressive augmentations.

Original: Nonetheless, it is hard, without domain knowledge (Dangovski et al., 2022; Chuang et al., 2022) or extensive testing (Dangovski et al., 2022), to tell apart sensitive and insensitive augmentations.
Revised: Nonetheless, it is hard, without domain knowledge (Dangovski et al., 2022; Chuang et al., 2022) or extensive testing (Dangovski et al., 2022), to tell apart sensitive and insensitive augmentations.

Original: To embody equivariance in GCL, we propose a simple but effective approach of Equivariant Graph
Revised: To embody equivariance in GCL, we propose a simple but effective approach of Equivariant Graph

Original: Contrastive Learning (E-GCL).
Revised: Contrastive Learning (E-GCL).

Original: E-GCL is an instantiation of E-SSL for graphs.
Revised: E-GCL is an instantiation of E-SSL for graphs.

Original: Unlike previous E-SSL works, E-GCL leaves existing intra-graph augmentations untouched, and creates new augmentations through the “cross-graph” strategy.
Revised: Unlike previous E-SSL works, E-GCL leaves existing intra-graph augmentations untouched, and creates new augmentations through the “cross-graph” strategy.

Original: Concretely, inspired by mixup (Guo & Mao, 2021; Zhang et al., 2018), the cross-graph augmentation interpolates the raw features of a pair of graphs ( i.e., T h ), while employing the same interpolation strategy on the graph labels that are portrayed by graph representations ( i.e., T ′ h ).
Revised: Concretely, inspired by mixup (Guo & Mao, 2021; Zhang et al., 2018), the cross-graph augmentation interpolates the raw features of two graphs ( i.e., T h ), while employing the same interpolation strategy on the graph labels that are portrayed by graph representations ( i.e., T ′ h ).

Original: The augmentations across instances not only maintain the holistic information on self-discrimination, but also are orthogonal to the intra-graph augmentations.
Revised: The augmentations across graphs not only maintain the holistic information on self-discrimination, but also are orthogonal to the intra-graph augmentations.

Original: On the top of intra- and cross-graph of augmentations, E-GCL separately builds the invariance and equivariance principles to guide the representation learning.
Revised: On the top of intraand cross-graph of augmentations, E-GCL separately builds the invariance and equivariance principles to guide the representation learning.

Revised: The equivariance to cross-graph augmentations diminishes the harmful invariance to aggressive augmentations that change global semantics.

Original: Integrating two principles enables representations to be sensitive to global semantic shifts across different graphs and insensitive to local substructure perturbations of single graphs.
Revised: Integrating two principles enables representations to be sensitive to global semantic shifts across different graphs and insensitive to local substructure perturbations of single graphs.

Original: Experiments show that E-GCL achieves promising performances to surpass current state-of-the-art GCL models, across diverse settings.
Revised: Experiments show that EGCL achieves promising performances to surpass current state-of-the-art GCL models, across diverse settings.

Original: We also demonstrate E-GCL’s generalization to various SSL backbones, including BarlowTwins (Zbontar et al., 2021), GraphCL (You et al., 2020) and SimSiam (Chen & He, 2021).
Revised: We also demonstrate E-GCL’s generalization to various SSL frameworks, including BarlowTwins (Zbontar et al., 2021), GraphCL (You et al., 2020) and SimSiam (Chen & He, 2021).

Original:  P RELIMINARIES : I NVARIANT G RAPH C ONTRASTIVE L EARNING
Revised:  P RELIMINARIES : I NVARIANT G RAPH C ONTRASTIVE L EARNING

Original: We begin by presenting the instance discrimination task and the invariance mechanism of I-GCL, and then introduce two key ingredients: graph augmentations and contrastive learning.
Revised: We begin by presenting the instance discrimination task and the invariance mechanism of I-GCL, and then introduce two key ingredients: graph augmentations and contrastive learning.

Original: Instance Discrimination.
Revised: Instance Discrimination.

Original: Let G = { g n } Nn =1 be the set of unlabeled graph instances.
Revised: Let G = { g n } Nn =1 be the set of unlabeled graph instances.

Original: We denote a graph instance g ∈ G by ( V , E ) involving the node set V and the edge set E .
Revised: We denote a graph instance g ∈ G by ( V , E ) involving the node set V and the edge set E .

Original: This graph structure can be represented as an adjacency matrix A ∈ { 0 , 1 } |V|×|V| , where A uv = 1 if the edge ( u, v ) ∈
Revised: This graph structure can be represented as an adjacency matrix A ∈ { 0 , 1 } |V|×|V| , where A uv = 1 if the edge ( u, v ) ∈

Original: E from node u to node v holds, otherwise A uv = 0 .
Revised: E from node u to node v holds, otherwise A uv = 0 .

Original: Moreover, each node v ∈ V could have d 1 -dimensional features x v ∈ R d 1 , while each edge ( u, v ) ∈
Revised: Moreover, each node v ∈ V could have d 1 -dimensional features x v ∈ R d 1 , while each edge ( u, v ) ∈

Original: E might have d 2 -dimensional features e uv ∈ R d 2 .
Revised: E might have d 2 -dimensional features e uv ∈ R d 2 .

Original: On the graph data G without annotations, contrastive self-supervised learning (SSL) aims to pre-train a graph encoder ϕ : G → R d that projects the graph instances to a d -dimensional space, so as to enhance the encoder’s representation ability and facilitate its fine-tuning in downstream tasks.
Revised: On the graph data G without annotations, contrastive self-supervised learning (SSL) aims to pre-train a graph encoder ϕ : G → R d that projects the graph instances to a d -dimensional space, so as to enhance the encoder’s representation ability and facilitate its fine-tuning in downstream tasks.

Original: Towards this end, a prevailing task of pre-training is instance discrimination (Dosovitskiy et al., 2014;
Revised: Towards this end, a prevailing task of pre-training is instance discrimination (Dosovitskiy et al., 2014;

Original: Purushwalkam & Gupta, 2020; Li et al., 2021) — treating each graph instance as one single class, and distinguishing it from the other graph instances.
Revised: Purushwalkam & Gupta, 2020; Li et al., 2021) — treating each graph instance as one single class, and distinguishing it from the other graph instances.

Original: Invariance.
Revised: Invariance.

Original: A leading solution to instance discrimination is to maximize the representation agreement between augmented views of the same graph, while minimizing the representation agreement between views of two different graphs.
Revised: A leading solution to instance discrimination is to maximize the representation agreement between augmented views of the same graph, while minimizing the representation agreement between views of two different graphs.

Original: It essentially encourages each instance’s representation to be invariant to the augmentations (Dangovski et al., 2022; Grill et al., 2020; Zbontar et al., 2021).
Revised: It essentially encourages each instance’s representation to be invariant to the augmentations (Dangovski et al., 2022; Grill et al., 2020; Zbontar et al., 2021).

Original: Mathematically, invariance can be described by groups (Dangovski et al., 2022; Kondor & Trivedi, 2018; Maron et al., 2019a).
Revised: Mathematically, invariance can be described by groups (Dangovski et al., 2022; Kondor & Trivedi, 2018; Maron et al., 2019a).

Original: Let P be a group of augmentations ( aka. transformations).
Revised: Let P be a group of augmentations ( aka. transformations).

Original: Invariance makes the encoder ϕ
Revised: Invariance makes the encoder ϕ

Original: insensitive to the actions T : P × G → G of the group P on the graphs G , formally:
Revised: insensitive to the actions T : P × G → G of the group P on the graphs G , formally:

Original: [Equation]
Revised: [Equation]

Original: where T p ( g ) :
Revised: where T p ( g ) :

Original: = T ( p, g ) is an action of applying the augmentation p on the instance g .
Revised: = T ( p, g ) is an action of applying the augmentation p on the instance g .

Original: Dictating invariance to the encoder will output the same representations for the original and augmented graphs.
Revised: Dictating invariance to the encoder will output the same representations for the original and augmented graphs.

Original: Probing into Equation (1), we find two key ingredients: intra-graph augmentation and contrastive learning, and will present their common practices in prior studies.
Revised: Probing into Equation (1), we find two key ingredients: intra-graph augmentation and contrastive learning, and will present their common practices in prior studies.

Original: Intra-graph Augmentation.
Revised: Intra-graph Augmentation.

Original: Typically, the augmentation group P is pre-determined to imply prior knowledge of graph data.
Revised: Typically, the augmentation group P is pre-determined to imply prior knowledge of graph data.

Original: Early studies (Hu et al., 2020; Qiu et al., 2020; You et al., 2020; Zhu et al., 2020) instantiate augmentations as randomly corrupting the topological structure, node features, or edge features of individual graph instances.
Revised: Early studies (Hu et al., 2020; Qiu et al., 2020; You et al., 2020; Zhu et al., 2020) instantiate augmentations as randomly corrupting the topological structure, node features, or edge features of individual graph instances.

Original: For example, AttrMasking (Hu et al., 2020) masks node and edge attributes, and applies an objective to reconstruct them.
Revised: For example, AttrMasking (Hu et al., 2020) masks node and edge attributes, and applies an objective to reconstruct them.

Original: GCC (Qiu et al., 2020) explores random walks over the anchor graph to create different subgraph views.
Revised: GCC (Qiu et al., 2020) explores random walks over the anchor graph to create different subgraph views.

Original: GraphCL (You et al., 2020) systematically investigates the combined effect of various random augmentations.
Revised: GraphCL (You et al., 2020) systematically investigates the combined effect of various random augmentations.

Original: Despite the success, random corruptions are too aggressive to maintain the semantic consistency (Guo & Mao, 2021) between the anchor graph and its augmented views.
Revised: Despite the success, random corruptions are too aggressive to maintain the semantic consistency (Guo & Mao, 2021) between the anchor graph and its augmented views.

Original: The invariance principle blindly ignores the semantic shift, thus easily pushing dissimilar patterns together and making a pernicious impact on the representation learning.
Revised: The invariance principle blindly ignores the semantic shift, thus easily pushing dissimilar patterns together and making a pernicious impact on the representation learning.

Original: Some follow-on studies (Zhu et al., 2021; Subramonian, 2021;
Revised: Some follow-on studies (Zhu et al., 2021; Subramonian, 2021;

Original: Suresh et al., 2021; Xu et al., 2021) learn augmentations instead to underscore salient substructures, so as to mitigate the semantic shift.
Revised: Suresh et al., 2021; Xu et al., 2021) learn augmentations instead to underscore salient substructures, so as to mitigate the semantic shift.

Original: For instance, GCA (Zhu et al., 2021) applies node centralities to discover important substructures in social networks.
Revised: For instance, GCA (Zhu et al., 2021) applies node centralities to discover important substructures in social networks.

Original: MICRO-Graph (Subramonian, 2021) learns chemically meaningful motifs to help the informative subgraph sampling.
Revised: MICRO-Graph (Subramonian, 2021) learns chemically meaningful motifs to help the informative subgraph sampling.

Original: More recently, AD-GCL (Suresh et al., 2021) adopts the idea of information bottleneck to adversarially learn the salient subgraphs.
Revised: More recently, AD-GCL (Suresh et al., 2021) adopts the idea of information bottleneck to adversarially learn the salient subgraphs.

Original: Contrastive Learning.
Revised: Contrastive Learning.

Original: Upon the augmented views, the contrastive learning objective is to classify whether they come from identical instances.
Revised: Upon the augmented views, the contrastive learning objective is to classify whether they come from identical instances.

Original: Specifically, it pulls the augmented views derived from the same instance ( i.e., positive samples) together and pushes the views of different instances ( i.e., negative samples) apart (Chen et al., 2020;
Revised: Specifically, it pulls the augmented views derived from the same instance ( i.e., positive samples) together and pushes the views of different instances ( i.e., negative samples) apart (Chen et al., 2020;

Original: He et al., 2020).
Revised: He et al., 2020).

Original: The common practices of this objective are
Revised: The common practices of this objective are

Original: InfoNCE (van den Oord et al., 2018), NCE (Misra & van der Maaten, 2020), and NT-Xent (Chen et al., 2020).
Revised: InfoNCE (van den Oord et al., 2018), NCE (Misra & van der Maaten, 2020), and NT-Xent (Chen et al., 2020).

Original: Here we consider the NT-Xent adopted by GraphCL.
Revised: Here we consider the NT-Xent adopted by GraphCL.

Original: Given a minibatch of graph instances { g i } Ni =1 , it first generates two different augmented views, denoted as { g 1 i | g 1 i
Revised: Given a minibatch of graph instances { g i } Ni =1 , it first generates two different augmented views, denoted as { g 1 i | g 1 i

Original: = T p 1 ( g ) , p 1 ∼ P} Ni = and { g 2 i | g 2 i
Revised: = T p 1 ( g ) , p 1 ∼ P} Ni = and { g 2 i | g 2 i

Original: = T p 2 ( g ) , p 2 ∼ P} Ni =1 , and then feeds them into the encoder to yield the representations as { z 1 i
Revised: = T p 2 ( g ) , p 2 ∼ P} Ni =1 , and then feeds them into the encoder to yield the representations as { z 1 i

Original: | z 1 i = ρ ( ϕ ( g 1 i )) }
Revised: | z 1 i = ρ ( ϕ ( g 1 i )) }

Original: Ni =1 and { z 2 i | z
Revised: Ni =1 and { z 2 i | z

Original: i
Revised: i

Original: =
Revised: =

Original: ρ ( ϕ ( g 2 i )) } Ni =1 , where ρ ( · ) is an MLP projection head.
Revised: ρ ( ϕ ( g 2 i )) } Ni =1 , where ρ ( · ) is an MLP projection head.

Original: Formally, the loss of NT-Xent is:
Revised: Formally, the loss of NT-Xent is:

Original: [Equation]
Revised: [Equation]

Original: where s ( · ) is the function of cosine similarity, and τ is a temperature hyperparameter.
Revised: where s ( · ) is the function of cosine similarity, and τ is a temperature hyperparameter.

Original: In a nutshell, the interplay between intra-graph augmentation and contrastive learning is tailor-made for invariance to make the encoder insensitive to differences between the anchor and augmented
Revised: In a nutshell, the interplay between intra-graph augmentation and contrastive learning is tailor-made for invariance to make the encoder insensitive to differences between the anchor and augmented

Original: [Figure]
Revised: [Figure]

Original: views.
Revised: views.

Original: In this work, we explore equivariance on cross-graph augmentations to make the encoder sensitive to the changes in self-discriminative information.
Revised: In this work, we explore equivariance on cross-graph augmentations to make the encoder sensitive to the changes in self-discriminative information.

Original:  M ETHODOLOGY : E QUIVARIANT G RAPH C ONTRASTIVE L EARNING
Revised:  M ETHODOLOGY : E QUIVARIANT G RAPH C ONTRASTIVE L EARNING

Original: Here we present the E-GCL framework, which imposes two principles — invariance to intra-graph augmentations (Figure 2 left) and equivariance to cross-graph augmentations (Figure 2 right) — on the representation learning, aiming to mitigate the potential limitations of I-GCL.
Revised: Here we present the E-GCL framework, which imposes two principles — invariance to intra-graph augmentations (Figure 2 left) and equivariance to cross-graph augmentations (Figure 2 right) — on the representation learning, aiming to mitigate the potential limitations of I-GCL.

Original: Next, we start with the concepts of equivariance and cross-graph augmentations.
Revised: Next, we start with the concepts of equivariance and cross-graph augmentations.

Original: 3. E QUIVARIANCE
Revised: 3. E QUIVARIANCE

Original: Inspired by the recent E-SSL studies (Dangovski et al., 2022; Chuang et al., 2022), we aim to patch invariance’s potential limitations with equivariance.
Revised: Inspired by the recent E-SSL studies (Dangovski et al., 2022; Chuang et al., 2022), we aim to patch invariance’s potential limitations with equivariance.

Original: Mathematically, with a group of augmentations H , the encoder ϕ ( · ) is said to be H -equivariant w.r.t.
Revised: Mathematically, with a group of augmentations H , the encoder ϕ ( · ) is said to be H -equivariant w.r.t.

Original: the actions T : H×G → G and T ′ :
Revised: the actions T : H×G → G and T ′ :

Original: H× R d → R d of the group
Revised: H× R d → R d of the group

Original: H applied on the graph space G and the representation space R d , if
Revised: H applied on the graph space G and the representation space R d , if

Original: [Equation]
Revised: [Equation]

Original: (3) where T h ( g ) is the action of applying the transformation h on the graph instance g , while T ′ h ( ϕ ( g )) is the action of h on the representation ϕ
Revised: (3) where T h ( g ) is the action of applying the transformation h on the graph instance g , while T ′ h ( ϕ ( g )) is the action of h on the representation ϕ

Original: ( g ) .
Revised: ( g ) .

Original: Equivariance requires that the representation ϕ ( g ) should be equivariant to the semantic changes in the graph g , i.e., any change applied to the graph should be faithfully reflected by the change of  representation.
Revised: H -equivariant requires that, given a transformation h ∈ H , h ’s influence on the graph should be faithfully reflected by the change of the graph’s representation.

Original: Taking Figure 1b as an example, givena transformation that perturbs the graph’s global semantics, the representation yielded by the equivariant encoder should transform in a definite way.
Revised: Taking Figure 1b as an example, given the graph’s global semantics are perturbed by graph interpolation, the representation yielded by the equivariant encoder should transform in a definite way.

Original: Jointly analyzing Equations (1) and (3), it can be shown that invariance is a special case of equivariance when setting T ′ h as the identity mapping
Revised: Jointly analyzing Equations (1) and (3), it can be shown that invariance is a special case of equivariance when setting T ′ h as the identity mapping.

Original: However, generalizing GCL to equivariance remains unexplored, thus a focus of our work.
Revised: However, generalizing GCL to equivariance remains unexplored, thus a focus of our work.

Original: Furthermore, as suggested in the recent E-SSL studies (Dangovski et al., 2022; Chuang et al., 2022), jointly imposing invariance to some transformations P and equivariance to other transformations H is promising to result in better representations than relying solely on one of them.
Revised: Furthermore, as suggested in the recent E-SSL studies (Dangovski et al., 2022; Chuang et al., 2022), jointly imposing invariance to some transformations P and equivariance to other transformations H is promising to result in better representations than relying solely on one of them.

Original: Here we term
Revised: Here we term

Original: P and
Revised: P and

Original: H as insensitive and sensitive transformations, respectively.
Revised: H as insensitive and sensitive transformations, respectively.

Original: For example, in computer vision, E-SSL (Dangovski et al., 2022) sets grayscale of images as P , while treating rotations as H ; in natural language processing, DiffCSE (Chuang et al., 2022) treats the model dropout as P , while using the word replacement as H .
Revised: For example, in computer vision, E-SSL (Dangovski et al., 2022) sets grayscale of images as P , while treating rotations as H ; in natural language processing, DiffCSE (Chuang et al., 2022) treats the model dropout as P , while using the word replacement as H .

Original: Clearly, it is of crucial importance to partition augmentations into P and
Revised: Clearly, it is of crucial importance to partition augmentations into P and

Original: H .
Revised: H .

Original: Nonetheless, these studies either conduct extensive testings on the impact of different partitions (Dangovski et al., 2022) which is time-consuming, or exploit domain knowledge to heuristically partition (Chuang et al., 2022) which might generalize poorly to other domains.
Revised: Nonetheless, these studies either conduct extensive testings on the impact of different partitions (Dangovski et al., 2022) which is time-consuming, or exploit domain knowledge to heuristically partition (Chuang et al., 2022) which might generalize poorly to other domains.

Original: Hence, it is infeasible to apply these strategies on graph augmentations.
Revised: Hence, it is infeasible to apply these strategies on graph augmentations.

Original: Worse still, different graph augmentations stem mostly from the perturbation of graph structures, thus highly likely to corrupt the same attributes of graphs.
Revised: Worse still, different graph augmentations stem mostly from the perturbation of graph structures, thus highly likely to corrupt the same attributes of graphs.

Original: Taking the graph g in Figure 1a as an example, masking the nitrogen N atom or dropping the
Revised: Taking the graph g in Figure 1a as an example, masking the nitrogen N atom or dropping the

Original: C ≡ N bond will both corrupt the cyano group and break the corresponding molecular properties.
Revised: C ≡ N bond will both corrupt the cyano group and break the corresponding molecular properties.

Original: In a nutshell, owing to (1) the common paradigm of structure corruption and (2) the risk of categorizing them all as insensitive augmentations, we conservatively argue that it is hard to partition graph augmentations into sensitive H and insensitive parts P .
Revised: In a nutshell, owing to (1) the common paradigm of structure corruption and (2) the risk of categorizing them all as insensitive augmentations, we conservatively argue that it is hard to partition graph augmentations into sensitive H and insensitive parts P .

Original: In this work, leaving partitioning untouched, we remain intra-graph augmentations as the insensitive transformations P and propose new augmentations across graphs as the sensitive transformations H .
Revised: In this work, leaving partitioning untouched, we remain intra-graph augmentations as the insensitive transformations P and propose new augmentations across graphs as the sensitive transformations H .

Original: 3. C ROSS - GRAPH A UGMENTATION
Revised: 3. C ROSS - GRAPH A UGMENTATION

Original: We first introduce graph interpolation (Guo & Mao, 2021) to create cross-graph augmentations as H .
Revised: We first introduce graph interpolation (Guo & Mao, 2021) to create cross-graph augmentations as H .

Original: Different from previous work, we propose an extension of graph interpolation for SSL (Section 3.3).
Revised: Different from previous work, we propose an extension of graph interpolation for SSL (Section 3.3).

Original: We also connect it to group theory and address its limitation of sensitivity to the relative permutation.
Revised: We also connect it to group theory and address its limitation of sensitivity to the relative permutation.

Original: Interpolating Graphs as Cross-graph Augmentations.
Revised: Interpolating Graphs as Cross-graph Augmentations.

Original: Given two graph instances g ∈ G and g ′ ∈ G , we employ mixup (Zhang et al., 2018), a simple yet effective linear interpolation approach, on the input features and class labels, respectively:
Revised: Given two graph instances g ∈ G and g ′ ∈ G , we employ mixup (Zhang et al., 2018), a simple yet effective linear interpolation approach, on the input features and class labels, respectively:

Original: [Equation]
Revised: [Equation]

Original: where y and y ′ separately denote the one-hot encodings to indicate the instance identities of g and g ′ in the instance discrimination task; λ ∼ Beta ( α, α ) ∈ [0 , 1] is the interpolation ratio sampled from a Beta distribution, in which α is a hyperparameter.
Revised: where y and y ′ separately denote the one-hot encodings to indicate the instance identities of g and g ′ in the instance discrimination task; λ ∼ Beta ( α, α ) ∈ [0 , 1] is the interpolation ratio sampled from a Beta distribution, in which α is a hyperparameter.

Original: This mixup strategy is initially proposed for supervised learning, aiming to put the interpolated samples in-between different classes and make the decision boundary robust to slightly corrupted samples (Verma et al., 2019; Zhang et al., 2018; 2021).
Revised: This mixup strategy is initially proposed for supervised learning, aiming to put the interpolated samples in-between different classes and make the decision boundary robust to slightly corrupted samples (Verma et al., 2019; Zhang et al., 2018; 2021).

Original: Despite the success of mixing image and text, it is challenging to interpolate graphs due to the structural differences between graph instances ( e.g., varying topologies and sizes).
Revised: Despite the success of mixing image and text, it is challenging to interpolate graphs due to the structural differences between graph instances ( e.g., varying topologies and sizes).

Original: To this end, we draw inspiration from the recent work (Guo & Mao, 2021) to perform linear interpolation between graphs.
Revised: To this end, we draw inspiration from the recent work (Guo & Mao, 2021) to perform linear interpolation between graphs.

Original: Specifically, with g =
Revised: Specifically, with g =

Original: ( V , E ) and g ′ =
Revised: ( V , E ) and g ′ =

Original: ( V ′ , E ′ ) , we mitigate their structural differences by padding virtual nodes and edges, which are associated with zero features 0 .
Revised: ( V ′ , E ′ ) , we mitigate their structural differences by padding virtual nodes and edges, which are associated with zero features 0 .

Original: Assuming |V| ≤ |V ′ | , g can be updated as a new graph with |V ′ | nodes, where the original node set
Revised: Assuming |V| ≤ |V ′ | , g can be updated as a new graph with |V ′ | nodes, where the original node set

Original: V remains unchanged but adds |V ′ | − |V| dummy virtual nodes, and the original nodes connect the virtual nodes with dummy virtual edges.
Revised: V remains unchanged but adds |V ′ | − |V| dummy virtual nodes, and the original nodes connect the virtual nodes with dummy virtual edges.

Original: Having padded two graphs to the same size, now we can directly add them up.
Revised: Having padded two graphs to the same size, now we can directly add them up.

Original: Before the interpolation, we first merge two node and edge sets as the new ones: ˜ V = V ∪ V ′ , ˜ E
Revised: Before the interpolation, we first merge two node and edge sets as the new ones: ˜ V = V ∪ V ′ , ˜ E

Original: = E ∪ E ′ .
Revised: = E ∪ E ′ .

Original: Then, ˜ g = λg + (1 − λ ) g ′ in Equation (4) is achieved by exerting linear interpolation on the adjacency matrices, node features, and edge features:
Revised: Then, ˜ g = λg + (1 − λ ) g ′ in Equation (4) is achieved by exerting linear interpolation on the adjacency matrices, node features, and edge features:

Original: [Equation]
Revised: [Equation]

Original: where A and A ′ are the adjacency matrices of g and g ′ after padding; x v and x ′ v are the features of node v ∈ ˜ V , which separately come from g and g ′ ; similarly, e uv and e ′ uv separately denote the features of edge ( u, v
Revised: where A and A ′ are the adjacency matrices of g and g ′ after padding; x v and x ′ v are the features of node v ∈ ˜ V , which separately come from g and g ′ ; similarly, e uv and e ′ uv separately denote the features of edge ( u, v

Original: ) ∈ ˜
Revised: ) ∈ ˜

Original: E from g and g ′ .
Revised: E from g and g ′ .

Original: Consequently, we generate a cross-graph augmentation.
Revised: Consequently, we generate a cross-graph augmentation.

Original: Connecting Cross-graph Augmentations to Groups.
Revised: Connecting Cross-graph Augmentations to Groups.

Original: In the language of groups, we can describe the cross-graph augmentation in Equation (4) as a group of transformations.
Revised: In the language of groups, we can describe the cross-graph augmentation in Equation (4) as a group of transformations.

Original: Given the input ( λ, g, g ′ ) , we can systemize the graph interpolation as two steps: (1) feature rescaling: ˆ g = λg , which rescales the node and edge features of g with the ratio λ ; (2) instance composition: ˜
Revised: Given the input ( λ, g, g ′ ) , we can systemize the graph interpolation as two steps: (1) feature rescaling: ˆ g = λg , which rescales the node and edge features of g with the ratio λ ; (2) instance composition: ˜

Original: g = C (ˆ g, ˆ g ′ )
Revised: g = C (ˆ g, ˆ g ′ )

Original: =
Revised: =

Original: ˆ g + ˆ g ′ , which adds the other rescaled graph ˆ g ′ =
Revised: ˆ g + ˆ g ′ , which adds the other rescaled graph ˆ g ′ =

Original: (1 − λ )
Revised: (1 − λ )

Original: g ′ .
Revised: g ′ .

Original: To construct a closed space for graph interpolation, we first define ˆ G =
Revised: To construct a closed space for graph interpolation, we first define ˆ G =

Original: { ˆ g | λ ∈ [ − 1 , 1] , g ∈ G} by performing feature rescaling on G to enable direct sampling of rescaled graphs.
Revised: { ˆ g | λ ∈ [ − 1 , 1] , g ∈ G} by performing feature rescaling on G to enable direct sampling of rescaled graphs.

Original: We allow λ < 0 to include the inverse elements of graphs.
Revised: We allow λ < 0 to include the inverse elements of graphs.

Original: Then, we generate I = <
Revised: Then, we generate I = <

Original: ˆ G > by combining the graphs in ˆ G via instance composition.
Revised: ˆ G > by combining the graphs in ˆ G via instance composition.

Original: We show that ( I , C ) forms a group in Appendix A.1.
Revised: We show that ( I , C ) forms a group in Appendix A.1.

Original: It is worth noting that each instance can be viewed as a transformation to others, i.e., C ( g, · ) :
Revised: It is worth noting that each instance can be viewed as a transformation to others, i.e., C ( g, · ) :

Original: = C g ( · ) , such that semantic shifts can be described via algebraic operators.
Revised: = C g ( · ) , such that semantic shifts can be described via algebraic operators.

Original: Group Averaging for Insensitivity to Relative Permutations.
Revised: Group Averaging for Insensitivity to Relative Permutation.

Original: Acute readers may notice that

Original:  Equation (5) can output different interpolations, when the node orders of one graph or padding positions of dummy nodes change.
Revised: Note that Equation (5) can output different interpolations, when the node orders of one graph or padding positions of dummy nodes change.

Original: Here we ascribe operations on “node orders” and “padding positions” to the “relative permutation” between two graphs g and g ′ .
Revised: We ascribe operations on “node orders” and “padding positions” to the “relative permutation” between two input graphs.

Revised: Unlike images and texts, the canonical permutations (orderings) of nodes in graphs are unlabeled.

Original: It is important to make the interpolations insensitive to relative permutations.
Revised: The default node permutations encode nothing useful about graph semantics (Xu et al., 2019; Hamilton et al., 2017). Thus, we enforce insensitivity to relative permutations by randomly permuting nodes in the bigger graph before graph interpolation. Next, we justify and develop this design with group averaging. Group averaging can make known architectures invariant to new symmetries (Puny et al., 2022;

Revised: Yarotsky, 2022).

Original: In the implementation, we randomly permute the node orders of the g (assuming | g | ≥ | g ′ | ) before interpolation — more formally, T P ◦ g , where P ∼ S n is a random permutation and T P is to permute g by P . 
Revised: It can be used for strict invariance to relative permutations. Let P ∼ S n be a random permutation, and T P ◦ g be permuting graph g by P . Assuming | g | ≥ | g ′

Original: Then, we obtain the graph interpolation as λT P ◦ g + (1 − λ ) g ′ and its representation as ϕ ( λT P ◦ g + (1 − λ ) g ′ ) .
Revised: | , we obtain the graph interpolation as λT P ◦ g + (1 − λ ) g ′ and its representation as ϕ ( λT P ◦ g + (1 − λ ) g ′ ) .

Original: To achieve insensitivity to relative permutations, we apply group averaging(Puny et al., 2022; Yarotsky, 2022) on the permutation operators:
Revised: To achieve strict invariance to relative permutations, we apply group averaging on the permutation operators:

Original: [Equation]
Revised: [Equation]

Original: where Φ is the function of group averaging.
Revised: where Φ is the function of group averaging.

Original: Φ is insensitive tothe relative permutation between g and g ′ , in the sense that Φ( λ, g, g ′ ) =
Revised: Φ is invariant to relative permutations between g and g ′ , in the sense that Φ( λ, g, g ′ ) =

Original: Φ( λ, T P ◦ g, g ′ )
Revised: Φ( λ, T P ◦ g, g ′ )

Original: =
Revised: =

Original: Φ( λ, g, T P ′ ◦
Revised: Φ( λ, g, T P ′ ◦

Original: g
Revised: g

Original: for all P, P ′ ∼ S n .
Revised: for all P, P ′ ∼ S n .

Original: Intuitively, it is achieved by averaging over all relative permutations.
Revised: Intuitively, it is achieved by averaging over all relative permutations.

Original: See Appendix A.2 for the proof.
Revised: See Appendix A.2 for the proof.

Original: However, the intractability of averaging over S n naturally arises as a problem.
Revised: However, the intractability of averaging over S n naturally arises as a problem.

Original: Following Murphy et al.
Revised: Following Murphy et al.

Original: (2019), our random permutation strategy ϕ
Revised: (2019), our random permutation strategy ϕ

Original: ( λT P ◦ g + (1 − λ ) g ′ ) is an unbiased estimator of Φ .
Revised: ( λT P ◦ g + (1 − λ ) g ′ ) is an unbiased estimator of Φ .

Original: Further, this strategy optimizes ρ ◦ ϕ toward an optima insensitive to the relative permutation.
Revised: Further, this strategy optimizes ρ ◦ ϕ toward an optima insensitive to the relative permutation.

Original: By Proposition 1, we conclude that using random permutation is a tractable surrogate for optimizing an invariant network to relative permutation.
Revised: By Proposition 1, we conclude that using random permutation is a tractable surrogate for optimizing an invariant network to relative permutation.

Revised: Appendix D.2 shows the influence of sampled permutation numbers.

Original: For simplicity, we still use λg + (1 − λ ) g ′ to represent graph interpolation in the rest text.
Revised: For simplicity, we still use λg + (1 − λ ) g ′ to represent graph interpolation in the rest text.

Original: Proposition 1.
Revised: Proposition 1.

Original: The contrastive loss with ϕ ( λT P ◦ g +(1 − λ ) g ′ )
Revised: The contrastive loss with ϕ ( λT P ◦ g +(1 − λ ) g ′ )

Original: upper bounds the loss of an invariant network to relative permutation 1 | S n | (cid:80) P ∈ S n ρ ( ϕ ( λ i T P ◦
Revised: upper bounds the loss of an invariant network to relative permutation 1 | S n | (cid:80) P ∈ S n ρ ( ϕ ( λ i T P ◦

Original: g i + (1 − λ i )
Revised: g i + (1 − λ i )

Original: g ′ i )) .
Revised: g ′ i )) .

Original: 3. E QUIVARIANCE TO C ROSS - GRAPH A UGMENTATIONS
Revised: 3. E QUIVARIANCE TO C ROSS - GRAPH A UGMENTATIONS

Original: Revisiting Equation (4), we can find the two terms of interpolation strategy align with the equivariance mechanism in Equation (3).
Revised: Revisiting Equation (4), we can find the two terms of interpolation strategy align with the equivariance mechanism in Equation (3).

Original: The semantic change caused by the feature interpolation is equivalently reflected by the label interpolation.
Revised: The semantic change caused by the feature interpolation is equivalently reflected by the label interpolation.

Original: Hence, we can instantiate the equivariance mechanism based on the feature and label interpolations.
Revised: Hence, we can instantiate the equivariance mechanism based on the feature and label interpolations.

Original: In the SSL setting, we interpolate graph representations as the alternative for label interpolation.
Revised: In the SSL setting, we interpolate graph representations as the alternative for label interpolation.

Original: Graph representations are derived from the encoder ϕ ( · ) with a global readout layer to summarize the graphs’ global semantics.
Revised: Graph representations are derived from the encoder ϕ ( · ) with a global readout layer to summarize the graphs’ global semantics.

Original: Hence, as shown by the right side of
Revised: Hence, as shown by the right side of

Original: Figure 2, we can parameterize equivariance approximately as:
Revised: Figure 2, we can parameterize equivariance approximately as:

Original: [Equation]
Revised: [Equation]

Original: Minimizing the distance between Equation (7)’s two sides allows the encoder to improve sensitivity to the global semantic shifts caused by cross-graph augmentations.
Revised: Minimizing the distance between Equation (7)’s two sides allows the encoder to improve sensitivity to the global semantic shifts caused by cross-graph augmentations.

Original: Although the strict equivariance is hardly guaranteed, experiments show that approaching Equation (7) can boost the performance on downstream tasks ( cf.
Revised: Although the strict equivariance is hardly guaranteed, experiments show that approaching Equation (7) can boost the performance on downstream tasks ( cf.

Original: Section 4.2).
Revised: Section 4.2).

Original: Furthermore, if the encoder is powerful enough to distinguish the interpolated graphs, it has a deterministic reflection in the representation space ϕ
Revised: Furthermore, if the encoder is powerful enough to distinguish the interpolated graphs, it has a deterministic reflection in the representation space ϕ

Original: ( G ) for a fixed transformation C ( g, · ) in graph space G (Non-trivial Equivariance (Dangovski et al., 2022)).
Revised: ( G ) for a fixed transformation C ( g, · ) in graph space G (Non-trivial Equivariance (Dangovski et al., 2022)).

Original: See Appendix A.1 for the proof.
Revised: See Appendix A.1 for the proof.

Original: Proposition 2.
Revised: Proposition 2.

Original: Assuming the encoder can detect the isomorphism of interpolated graphs, there exists a GNN encoder ϕ
Revised: Assuming the encoder can detect the isomorphism of interpolated graphs, there exists a GNN encoder ϕ

Original: that is non-trivially equivariant to the graph interpolation transformation.
Revised: that is non-trivially equivariant to the graph interpolation transformation.

Original: 3. I MPLEMENTING E-GCL
Revised: 3. I MPLEMENTING E-GCL

Original: This section details our implementation of E-GCL (Figure 2).
Revised: This section details our implementation of E-GCL (Figure 2).

Original: Specifically, given a minibatch of graph instances { g i } Ni =1 , we impose (1) invariance to intra-graph augmentation and (2) equivariance to cross-graph augmentation simultaneously on the shared encoder.
Revised: Specifically, given a minibatch of graph instances { g i } Ni =1 , we impose (1) invariance to intra-graph augmentation and (2) equivariance to cross-graph augmentation simultaneously on the shared encoder.

Original: Invariance.
Revised: Invariance.

Original: For the invariance principle, we follow the I-GCL paradigm to resort to the standard intra-graph augmentations P ( e.g., randomly dropping nodes in GraphCL), and create two augmented views of individual graphs: { g 1 i | g 1 i
Revised: For the invariance principle, we follow the I-GCL paradigm to resort to the standard intra-graph augmentations P ( e.g., randomly dropping nodes in GraphCL), and create two augmented views of individual graphs: { g 1 i | g 1 i

Original: =
Revised: =

Original: T p 1 ( g ) , p 1 ∼ P} Ni =1 and { g 2 i | g 2 i
Revised: T p 1 ( g ) , p 1 ∼ P} Ni =1 and { g 2 i | g 2 i

Original: = T p 2 ( g ) , p 2 ∼ P} Ni =1 .
Revised: = T p 2 ( g ) , p 2 ∼ P} Ni =1 .

Original: Consequently, the encoder ϕ ( · ) brings forth two representation lists: { z 1 i |
Revised: Consequently, the encoder ϕ ( · ) brings forth two representation lists: { z 1 i |

Original: z 1 i = ρ ( ϕ ( g 1 i )) }
Revised: z 1 i = ρ ( ϕ ( g 1 i )) }

Original: Ni =1 and { z 2 i | z
Revised: Ni =1 and { z 2 i | z

Original: i
Revised: i

Original: =
Revised: =

Original: ρ ( ϕ ( g 2 i )) } Ni =1 , in which ρ ( · ) is an MLP projector.
Revised: ρ ( ϕ ( g 2 i )) } Ni =1 , in which ρ ( · ) is an MLP projector.

Original: Equivariance.
Revised: Equivariance.

Original: For the equivariance principle, we first randomly shuffle the graphs in { g 2 i } Ni =1 , termed as { g 2 π ( i ) } Ni =1 , where π : [ N ] →
Revised: For the equivariance principle, we first randomly shuffle the graphs in { g 2 i } Ni =1 , termed as { g 2 π ( i ) } Ni =1 , where π : [ N ] →

Original: [ N ] is the function of random shuffling.
Revised: [ N ] is the function of random shuffling.

Original: Following the left side of Equation (7) and applying random permutations P i ∼ S n for all i ∈ [ N ] , we create the feature interpolations and then generate their representations as { z 3 i | z 3 i = ρ ( ϕ ( λT P i ◦ g 1 i +(1 − λ )
Revised: Following the left side of Equation (7) and applying random permutations P i ∼ S n for all i ∈ [ N ] , we create the feature interpolations and then generate their representations as { z 3 i | z 3 i = ρ ( ϕ ( λT P i ◦ g 1 i +(1 − λ )

Original: λg 2 π ( i ) ))
Revised: λg 2 π ( i ) ))

Original: } Ni =1 .
Revised: } Ni =1 .

Original: Meanwhile, according to the right side of Equation (7), we arrive at the representation interpolations as { z 4 i |
Revised: Meanwhile, according to the right side of Equation (7), we arrive at the representation interpolations as { z 4 i |

Original: z
Revised: z

Original: 4 i = ρ ( λϕ ( g 1 i ) + (1 − λ ) ϕ ( λg 2 π ( i ) ))
Revised: 4 i = ρ ( λϕ ( g 1 i ) + (1 − λ ) ϕ ( λg 2 π ( i ) ))

Original: } N i =1 .
Revised: } N i =1 .

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: Cooperative Game between Invariance and Equivariance.
Revised: Cooperative Game between Invariance and Equivariance.

Original: Based on the representations, we optimize the invariance and equivariance losses together with a weighting hyperparameter ω ∈ [0 , 1] :
Revised: Based on the representations, we optimize the invariance and equivariance losses together with a weighting hyperparameter ω ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: where ℓ ( · , · ) is the loss encouraging the insensitivity between augmented views of the same graph, which is determined by the SSL backbone, such as NT-Xent ( cf.
Revised: where ℓ ( · , · ) is the loss encouraging the insensitivity between augmented views of the same graph, which is determined by the SSL backbone, such as NT-Xent ( cf.

Original: Equation (2)) adopted by the
Revised: Equation (2)) adopted by the

Original: GraphCL.
Revised: GraphCL.

Original: Beyond contrastive learning, E-GCL is also applicable to various other SSL backbones, including BarlowTwins and SimSiam.
Revised: Beyond contrastive learning, E-GCL is also applicable to various other SSL backbones, including BarlowTwins and SimSiam.

Original: In a nutshell, the invariance loss underscores the insensitivity to intra-graph augmentations, while the equivariance loss induces the sensitivity to cross-graph augmentations.
Revised: In a nutshell, the invariance loss underscores the insensitivity to intra-graph augmentations, while the equivariance loss induces the sensitivity to cross-graph augmentations.

Original: The cooperative game between these two losses helps resolve the potential limitations of the conventional I-GCL paradigm, thus improving the expressive power of the encoder.
Revised: The cooperative game between these two losses helps resolve the potential limitations of the conventional I-GCL paradigm, thus improving the expressive power of the encoder.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we conduct experiments to answer the following research questions: RQ1: How effective is the proposed E-GCL in graph representation learning, and how does it generalize to existing SSL frameworks?
Revised: In this section, we conduct experiments to answer the following research questions: RQ1: How effective is the proposed E-GCL in graph representation learning, and how does it generalize to existing SSL frameworks?

Original: RQ2: What are the properties of E-GCL and the effects of its components?
Revised: RQ2: What are the properties of E-GCL and the effects of its components?

Original: In Appendix D.1, we present more ablation studies about 1) using G -mixup
Revised: In Appendix D.1, we present more ablation studies about 1) using G -mixup

Original: (Han et al., 2022), 2) interpolating representations at different positions, and 3) interpolating large and small graphs.
Revised: (Han et al., 2022), 2) interpolating representations at different positions, and 3) interpolating large and small graphs.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: Here we briefly introduce the baselines, datasets and evaluations.
Revised: Here we briefly introduce the baselines, datasets and evaluations.

Original: Details are in Appendix E. For a fair comparison, E-GCL uses the same intra-graph augmentation as GraphCL.
Revised: Details are in Appendix E. For a fair comparison, E-GCL uses the same intra-graph augmentation as GraphCL.

Original: If not noted, E-GCL employs a BarlowTwins backbone.
Revised: If not noted, E-GCL employs a BarlowTwins backbone.

Original: We study E-GCL’s generalization to other SSL frameworks later.
Revised: We study E-GCL’s generalization to other SSL frameworks later.

Original: Baselines .
Revised: Baselines .

Original: We compare E-GCL with the following state-of-the-art graph pre-training methods:graph2vec (Narayanan et al., 2017), Infomax (Veliˇckovi´c et al., 2019), InfoGraph (Sun et al., 2020), 
Revised: We compare E-GCL with the following state-of-the-art graph pre-training methods: Infomax (Veliˇckovi´c et al., 2019), InfoGraph (Sun et al., 2020), ContextPred (Hu et al., 2020),

Original: ContextPred (Hu et al., 2020), GraphCL (You et al., 2020), JOAO (You et al., 2021), AD-GCL (Suresh et al., 2021), GraphLOG (Xu et al., 2021), GraphMAE (Hou et al., 2022), and RGCL (Li et al., 2022).[Table] [Table]
Revised: GraphCL (You et al., 2020), JOAO (You et al., 2021), AD-GCL (Suresh et al., 2021), GraphLOG (Xu et al., 2021), GraphMAE (Hou et al., 2022), and RGCL (Li et al., 2022).

Original: [Table]

Original: Unsupervised Learning evaluates the pre-trained GNNs for prediction on the same dataset.
Revised: Unsupervised Learning evaluates the pre-trained GNNs for prediction on the same dataset.

Original: Following (You et al., 2020), we evaluate E-GCL on the eight TU datasets, including biochemical graphs and social networks.
Revised: Following (You et al., 2020), we evaluate E-GCL on the eight TU datasets, including biochemical graphs [Table] [Table] [Table]

Original: Specifically, we pre-train a three-layer GIN (Xu et al., 2019) and feed the generated graph representations into SVMs for evaluation.
Revised: and social networks. Specifically [Table] we pre-train a three-layer GIN (Xu et al., 2019) and feed the generated graph representations into SVMs for evaluation.

Original: We report the average values and standard deviations of accuracies (%) of five different runs, each of which corresponds to a 10-fold evaluation.
Revised: We report the average values and standard deviations of accuracies (%) of five different runs, each of which corresponds to a 10-fold evaluation.

Revised: Following (You et al., 2021), we report test accuracy of the epoch selected by validation set.

Original: Transfer Learning tests the pre-trained GNN’s transferability to downstream tasks.
Revised: Transfer Learning tests the pre-trained GNN’s transferability to downstream tasks.

Original: Following (Hu et al., 2020), we use the two million molecule samples from the ZINC15 dataset (Sterling & Irwin, 2015) for pre-training and eight multi-label classification datasets derived from the MoleculeNet (Wu et al., 2018) for fine-tuning.
Revised: Following (Hu et al., 2020), we use the two million molecule samples from the ZINC15 dataset (Sterling & Irwin, 2015) for pre-training and eight multi-label classification datasets derived from the MoleculeNet (Wu et al., 2018) for fine-tuning.

Original: Fine-tuning datasets are divided by scaffold split to create distribution shifts among the train/valid/test sets, so as to provide more realistic estimations of the molecule property prediction performance.
Revised: Fine-tuning datasets are divided by scaffold split to create distribution shifts among the train/valid/test sets, so as to provide more realistic estimations of the molecule property prediction performance.

Original: Following (Hu et al., 2020), we implement E-GCL with a five-layer
Revised: Following (Hu et al., 2020), we implement E-GCL with a five-layer

Original: GINE.
Revised: GINE.

Original: Further, we push the limits of E-GCL with the GraphTrans (Wu et al., 2021) backbone.
Revised: Further, we push the limits of E-GCL with the GraphTrans (Wu et al., 2021) backbone.

Original: GraphTrans stacks a four-layer Transformer (Vaswani et al., 2017) on top of the GINE to learn long range interactions.
Revised: GraphTrans stacks a four-layer Transformer (Vaswani et al., 2017) on top of the GINE to learn long range interactions.

Original: For evaluation, we pre-train a model and repeatedly fine-tune it on downstream datasets ten times.
Revised: For evaluation, we pre-train a model and repeatedly fine-tune it on downstream datasets ten times.

Original: We report the averages and standard deviations of ROC-AUC (%) scores.
Revised: We report the averages and standard deviations of ROC-AUC (%) scores.

Revised: Following (You et al., 2020), we report test set performance of the last epoch.

Original: 4. M AIN 
Revised: 4. M AIN R ESULTS (RQ1)

Original: [Equation] ESULTS

Original: [Equation]

Original: Unsupervised Learning Results .
Revised: Unsupervised Learning Results .

Original: Table 1a presents the unsupervised learning performance in TU datasets.
Revised: Table 1a presents the unsupervised learning performance in TU datasets.

Original: The last column denotes the improvement compared to a randomly initialized GNN.
Revised: The last column denotes the improvement compared to a randomly initialized GNN.

Original: E-GCL achieves the best performance in five out of eight datasets and the top three performances in the other three datasets.
Revised: E-GCL achieves the best performance in six out of eight datasets and the top three performances in the other two datasets.

Original: It also achieves the best average improvement of 6 . 70% compared to a randomly initialized GNN.
Revised: It also achieves the best average improvement of 7 . 43% compared to a randomly initialized GNN.

Original: We attribute E-GCL’s good performances to the equivariance principle of crossgraph augmentation.
Revised: We attribute E-GCL’s good performances to the equivariance principle of crossgraph augmentation.

Original: Other methods apply only the invariance principle of intra-graph augmentation, thus failing to generate representations as discriminative as E-GCL.
Revised: Other methods apply only the invariance principle of intra-graph augmentation, thus failing to generate representations as discriminative as E-GCL.

Original: Transfer Learning Results .
Revised: Transfer Learning Results .

Original: Table 1b presents the fine-tuning performances in transfer learning.
Revised: Table 1b presents the fine-tuning performances in transfer learning.

Original: EGCLs have achieved the best performances among all methods.
Revised: EGCLs have achieved the best performances among all methods.

Original: Specifically, E-GCL with GraphTrans achieves the best performances in four out of eight datasets and the best average improvement of 7 . 5% compared to a randomly initialized GNN.
Revised: Specifically, E-GCL with GraphTrans achieves the best performances in four out of eight datasets and the best average improvement of 5 . 5% compared to a randomly initialized GNN.

Original: When using the GINE backbone, E-GCL continues to show improvements over all the baseline models in average performance.
Revised: When using the GINE backbone, E-GCL continues to show improvements over all the baseline models in average performance.

Original: It shows that the equivariant pre-training of cross-graph augmentation gives E-GCL a better starting point for fine-tuning.
Revised: It shows that the equivariant pre-training of cross-graph augmentation gives E-GCL a better starting point for fine-tuning.

Original: Previous models apply only the intra-graph augmentation, which might bring together dissimilar patterns.
Revised: Previous models apply only the intra-graph augmentation, which might bring together dissimilar patterns.

Original: Notice that, E-GCL and other models’ improvements over previous works are not consistent across fine-tuning datasets.
Revised: Notice that, E-GCL and other models’ improvements over previous works are not consistent across fine-tuning datasets.

Original: We attribute the inconsistency to the Out of Distribution (OOD) evaluation setting in MoleculeNet.
Revised: We attribute the inconsistency to the Out of Distribution (OOD) evaluation setting in MoleculeNet.

Original: The validation set does not overlap the test set, which makes preventing overfitting and underfitting troublesome.
Revised: The validation set does not overlap the test set, which makes preventing overfitting and underfitting troublesome.

Original: We follow the evaluation protocol of previous works (You et al., 2020; Xu et al., 2021), which, however, does not guarantee the convergence of performance.
Revised: We follow the evaluation protocol of previous works (You et al., 2020; Xu et al., 2021), which, however, does not guarantee the convergence of performance.

Original: We leave a more stable fine-tuning for the OOD evaluation as future work.
Revised: We leave a more stable fine-tuning for the OOD evaluation as future work.

Original: In summary, E-GCL establishes a new state-of-the-art in unsupervised learning and transfer learning.
Revised: In summary, E-GCL establishes a new state-of-the-art in unsupervised learning and transfer learning.

Original: 4 BarlowTwins ( =0.4 ) 76.
Revised: 4 BarlowTwins ( =0.4 ) 76.

Original: A cc u r ac y ( % )  76.
Revised: A cc u r ac y ( % ) 76.93 76.68 76.5576.43 76.

Original: 4 GraphCL ( =0.2 )  76.
Revised: 4 GraphCL ( =0.3 ) 76.91 76.

Original: Backbone E-GCL (a) Different Beta ( α, α ) distributions.
Revised: Backbone E-GCL (a) Different Beta ( α, α ) distributions.

Original: 1.0 BarlowTwins ( =0.1 ) 76.56 76.68 76.69 76.85 76.4676. 1.0 GraphCL ( =0.1 ) 76.
Revised: 1.0 BarlowTwins ( =0.1 ) 76.55 76.63 76.69 76.93 76. 1.0 GraphCL ( =0.1 ) 76.70 76.65 76.91 76.75 76.8276.

Original: Backbone E-GCL
Revised: Backbone E-GCL

Original: [Figure]
Revised: [Figure]

Original: Generalization to Different SSL Frameworks .
Revised: Generalization to Different SSL Frameworks .

Original: To highlight E-GCL’s improvement and generalization ability, we apply the equivariance principle to three representative SSL frameworks of different flavors: GraphCL – GCL, SimSiam (Chen & He, 2021) – asymmetric Siamese Networks, and
Revised: To highlight E-GCL’s improvement and generalization ability, we apply the equivariance principle to three representative SSL frameworks of different flavors: GraphCL – GCL, SimSiam (Chen & He, 2021) – asymmetric Siamese Networks, and

Original: BarlowTwins – decorrelating feature dimensions (Table 2).
Revised: BarlowTwins – decorrelating feature dimensions (Table 2).

Original: We observe that equivariance consistently improves about 1% of SSL backbones’ average performances in both unsupervised learning and transfer learning.
Revised: We observe that equivariance consistently improves about 1% of SSL backbones’ average performances in both unsupervised learning and transfer learning.

Original: The results demonstrate the effectiveness of equivariance and its generalization ability to diverse SSL frameworks and different experimental settings.
Revised: The results demonstrate the effectiveness of equivariance and its generalization ability to diverse SSL frameworks and different experimental settings.

Original: 4. A NALYZING THE P ROPERTIES OF E-GCL (RQ2)
Revised: 4. A NALYZING THE P ROPERTIES OF E-GCL (RQ2)

Original: Hyper-parameter Sensitivity .
Revised: Hyper-parameter Sensitivity .

Original: Figure 3 presents E-GCL’s sensitivity with respect to the shape parameter α of Beta ( α, α ) distribution and the trade-off factor ω between invariance and equivariance.
Revised: Figure 3 presents E-GCL’s sensitivity with respect to the shape parameter α of Beta ( α, α ) distribution and the trade-off factor ω between invariance and equivariance.

Original: Shown by Figure 3a, α = 0 .
Revised: Shown by Figure 3a, α = 0 .

Original: 1 gives the best average performance for both the BarlowTwins and
Revised: 1 gives the best average performance for both the BarlowTwins and

Original: GraphCL backbones in unsupervised learning.
Revised: GraphCL backbones in unsupervised learning.

Original: We also observe that the optimal ω value differs for
Revised: We also observe that the optimal ω value differs for

Original: SSL backbones (Figure 3b).
Revised: SSL backbones (Figure 3b).

Original: The best ω for BarlowTwins and GraphCL are 0 .
Revised: The best ω for BarlowTwins and GraphCL are 0 .

Original: 4 and 0 . 2 , respectively.
Revised: 4 and 0 . 3 , respectively.

Original: When ω = 1 , the invariance loss vanishes and the performances drop to the lowest.
Revised: When ω = 1 , the invariance loss vanishes and the performances drop to the lowest.

Original: This demonstrates that the invariance mechanism and equivariance mechanism are complementary to each other and their cooperation makes for better graph representation learning.
Revised: This demonstrates that the invariance mechanism and equivariance mechanism are complementary to each other and their cooperation makes for better graph representation learning.

Original: [Figure]
Revised: [Figure]

Original: Training Dynamics of Alignment and Uniformity .
Revised: Training Dynamics of Alignment and Uniformity .

Original: To understand how E-GCL improves over I-GCL, we study their behaviors through the lens of alignment and uniformity losses (Wang & Isola, 2020), which constitute the asymptotic objective of contrastive learning.
Revised: To understand how E-GCL improves over I-GCL, we study their behaviors through the lens of alignment and uniformity losses (Wang & Isola, 2020), which constitute the asymptotic objective of contrastive learning.

Original: On a unit hypersphere, Alignment measures the closeness of the positive pairs and uniformity measures the evenness of the sample distribution.
Revised: On a unit hypersphere, Alignment measures the closeness of the positive pairs and uniformity measures the evenness of the sample distribution.

Original: We apply the contrastive backbone GraphCL with dropNode as the intra-graph augmentation and graph interpolation as the cross-graph augmentation.
Revised: We apply the contrastive backbone GraphCL with dropNode as the intra-graph augmentation and graph interpolation as the cross-graph augmentation.

Original: Figure 4 shows the losses on each type of augmented samples and their concatenations ( i.e., intra+cross aug).
Revised: Figure 4 shows the losses on each type of augmented samples and their concatenations ( i.e., intra+cross aug).

Original:  E-GCL achieves much better alignment and uniformity on cross-graph augmentations ⋆ than I-GCL, with a slight sacrifice of alignment on intra-augmentations .
Revised: Compared to E-GCL cross-aug, E-GCL intra+cross aug has better uniform loss but worse alignment loss. E-GCL achieves much better alignment and uniformity on cross-graph augmentations ⋆ than I-GCL, with a slight sacrifice of alignment on intra-augmentations .

Original: This is in expectation as E-GCL applies equivariance to explicitly optimize the cross-graph augmentations and trade-off the optimization of intra-graph augmentations.
Revised: This is in expectation as E-GCL applies equivariance to explicitly optimize the cross-graph augmentations and trade-off the optimization of intra-graph augmentations.

Original: Combining intra- and cross-graph augmentations, E-GCL achieves better alignment and uniformity than I-GCL , which explains the better performance.
Revised: Combining intra- and cross-graph augmentations, E-GCL achieves better alignment and uniformity than I-GCL , which explains the better performance.

Original:  C ONCLUSION AND F UTURE W ORKS
Revised:  C ONCLUSION AND F UTURE W ORKS

Original: In this paper, we propose Equivariant Graph Contrastive Learning (E-GCL) that combines equivariance and invariance to learn better graph representations.
Revised: In this paper, we propose Equivariant Graph Contrastive Learning (E-GCL) that combines equivariance and invariance to learn better graph representations.

Original: E-GCL encourages the sensitivity to global semantic shifts by grounding the equivariance principle as a cross-graph augmentation of graph interpolation.
Revised: E-GCL encourages the sensitivity to global semantic shifts by grounding the equivariance principle as a cross-graph augmentation of graph interpolation.

Original: This equivariance principle protects GNNs from aggressive intra-graph augmentations that can harmfully align dissimilar patterns and enables GNNs to discriminate cross-graph augmented samples.
Revised: This equivariance principle protects GNNs from aggressive intra-graph augmentations that can harmfully align dissimilar patterns and enables GNNs to discriminate cross-graph augmented samples.

Original: Extensive experiments in unsupervised learning and transfer learning demonstrate E-GCL’s significant improvements over state-of-the-art methods and its generalization ability to different SSL frameworks.
Revised: Extensive experiments in unsupervised learning and transfer learning demonstrate E-GCL’s significant improvements over state-of-the-art methods and its generalization ability to different SSL frameworks.

Original: In the future, we will explore more groundings of the equivariance principle in graphs.
Revised: In the future, we will explore more groundings of the equivariance principle in graphs.

Original: A P ROOF A. P ROOFS ON G RAPH I NTERPOLATION
Revised: A P ROOF A. P ROOFS ON G RAPH I NTERPOLATION

Original: Proof of that ( I , C ) forms a group .
Revised: Proof of that ( I , C ) forms a group .

Original: We first recall of the definition of ( I , C ) as follows:
Revised: We first recall of the definition of ( I , C ) as follows:

Original: When viewing g as the anchor being augmented, we can systemize the mixup as a combination of two steps: (1) feature rescaling: ˆ g = λg , which rescales the node and edge features of g with the ratio λ ; (2) instance composition: ˜
Revised: When viewing g as the anchor being augmented, we can systemize the mixup as a combination of two steps: (1) feature rescaling: ˆ g = λg , which rescales the node and edge features of g with the ratio λ ; (2) instance composition: ˜

Original: g = C (ˆ g, ˆ g ′ )
Revised: g = C (ˆ g, ˆ g ′ )

Original: =
Revised: =

Original: ˆ g + ˆ g ′ , which adds another rescaled graph ˆ g ′ .
Revised: ˆ g + ˆ g ′ , which adds another rescaled graph ˆ g ′ .

Original: Let ˆ G =
Revised: Let ˆ G =

Original: { ˆ g | λ ∈ [0 , 1] , g ∈ G} be the enlarged set of graphs via feature rescaling, I = < ˆ G > be the generated group by combining the graphs in ˆ G via instance composition.
Revised: { ˆ g | λ ∈ [0 , 1] , g ∈ G} be the enlarged set of graphs via feature rescaling, I = < ˆ G > be the generated group by combining the graphs in ˆ G via instance composition.

Original: We show that ( I , C ) forms a group.
Revised: We show that ( I , C ) forms a group.

Original: Proof.
Revised: Proof.

Original: I is a set of graphs and C ( · , · ) is a binary instance composition operation on I .
Revised: I is a set of graphs and C ( · , · ) is a binary instance composition operation on I .

Original: It suffices to show that (
Revised: It suffices to show that (

Original: I , C ) is a group if it satisfies the following conditions:
Revised: I , C ) is a group if it satisfies the following conditions:

Original: • Associativity Law .
Revised: • Associativity Law .

Original: C ( g 1 , C ( g 2 , g 3 ))
Revised: C ( g 1 , C ( g 2 , g 3 ))

Original: =
Revised: =

Original: g 1 +( g 2 + g 3 ) =
Revised: g 1 +( g 2 + g 3 ) =

Original: ( g 1 + g 2 )+ g 

Revised: ( g 1 + g 2 )+ g

Original: =
Revised: =

Original: C ( C ( g 1 , g 2 ) , g 3 ) for all
Revised: C ( C ( g 1 , g 2 ) , g 3 ) for all

Original: g 1 , g 2 , g 3 ∈ I .
Revised: g 1 , g 2 , g 3 ∈ I .

Original: Notice that, both resulting node feature matrices and adjacency matrices of both sides are equivalent.
Revised: Notice that, both resulting node feature matrices and adjacency matrices of both sides are equivalent.

Original: • Existence of Identity .
Revised: • Existence of Identity .

Original: Let e be the empty graph with no nodes and no edges, then C ( g, e ) =
Revised: Let e be the empty graph with no nodes and no edges, then C ( g, e ) =

Original: g =
Revised: g =

Original: C ( e, g ) for all g ∈ I , then the empty graph e is the identity.
Revised: C ( e, g ) for all g ∈ I , then the empty graph e is the identity.

Original: • Existence of Inverse .
Revised: • Existence of Inverse .

Original: For any graph g ∈
Revised: For any graph g ∈

Original: I , there exists a graph g − 1 ∈
Revised: I , there exists a graph g − 1 ∈

Original: I who has the same structure of g but with inverse values of node features and edge weights, aka.
Revised: I who has the same structure of g but with inverse values of node features and edge weights, aka.

Original: X
Revised: X

Original: g − 1 = − X g and A g − 1 = − A g , such that C ( g, g − 1 ) =
Revised: g − 1 = − X g and A g − 1 = − A g , such that C ( g, g − 1 ) =

Original: g + g − 1 = e .
Revised: g + g − 1 = e .

Original: Proposition 2.
Revised: Proposition 2.

Original: Assuming the encoder can detect the isomorphism of interpolated graphs, there exists a GNN encoder ϕ
Revised: Assuming the encoder can detect the isomorphism of interpolated graphs, there exists a GNN encoder ϕ

Original: that is non-trivially equivariant to the graph interpolation transformation.
Revised: that is non-trivially equivariant to the graph interpolation transformation.

Original: Proof.
Revised: Proof.

Original: For proof, it suffices to show that graph interpolation transformation satisfies the assumption of E-SSL’s (Dangovski et al., 2022)
Revised: For proof, it suffices to show that graph interpolation transformation satisfies the assumption of E-SSL’s (Dangovski et al., 2022)

Original: Non-trivial Equivariance Proposition.
Revised: Non-trivial Equivariance Proposition.

Original: We can then apply the
Revised: We can then apply the

Original: Non-trivial Equivariance Proposition to conclude the proof.
Revised: Non-trivial Equivariance Proposition to conclude the proof.

Original: The assumption states that, “ given P as the group of our interest, if ϕ ( T p ( g ))
Revised: The assumption states that, “ given P as the group of our interest, if ϕ ( T p ( g ))

Original: =
Revised: =

Original: ϕ ( T ′ p ( g ′ )) , then g = g ′ and p = p ′ for all p, p ′ ∈ P and g, g ′ ∈ G .”
Revised: ϕ ( T ′ p ( g ′ )) , then g = g ′ and p = p ′ for all p, p ′ ∈ P and g, g ′ ∈ G .”

Original: We rewrite the assumption with the graph interpolation formulation of equivariance: for all λg 1 , (1 − λ ) g 2 , λ ′ g ′ 1 , (1 − λ ′ )
Revised: We rewrite the assumption with the graph interpolation formulation of equivariance: for all λg 1 , (1 − λ ) g 2 , λ ′ g ′ 1 , (1 − λ ′ )

Original: g ′ 2 ∈
Revised: g ′ 2 ∈

Original: ˆ G with λ, λ ′ ∈ [0 , 1] , if ϕ ( λg 1 +(1 − λ ) g 2 )
Revised: ˆ G with λ, λ ′ ∈ [0 , 1] , if ϕ ( λg 1 +(1 − λ ) g 2 )

Original: =
Revised: =

Original: ϕ ( λ ′ g ′ 

Revised: ϕ ( λ ′ g ′

Original: +(1 − λ ′ )
Revised: +(1 − λ ′ )

Original: g ′ 2 ) , then either 1) λg 1 = λ ′ g ′ 1 , (1 − λ )
Revised: g ′ 2 ) , then either 1) λg 1 = λ ′ g ′ 1 , (1 − λ )

Original: g 2 = (1 − λ ′ )
Revised: g 2 = (1 − λ ′ )

Original: g ′ 2 , or 2) λg 1 = (1 − λ ′ )
Revised: g ′ 2 , or 2) λg 1 = (1 − λ ′ )

Original: g 2 , (1 − λg 2 ) =
Revised: g 2 , (1 − λg 2 ) =

Original: λ ′ g 1 .
Revised: λ ′ g 1 .

Original: We now prove this assumption.
Revised: We now prove this assumption.

Original: We consider graphs without edge features.
Revised: We consider graphs without edge features.

Original: We have assumed that GNN encoder can detect graph isomorphism of interpolated graphs.
Revised: We have assumed that GNN encoder can detect graph isomorphism of interpolated graphs.

Original: Therefore, we can infer the equivalence of the graphs based on the equivalence of graph embeddings.
Revised: Therefore, we can infer the equivalence of the graphs based on the equivalence of graph embeddings.

Original: Let ˜ g
Revised: Let ˜ g

Original: = λg 

Revised: = λg

Original: + (1 − λ )
Revised: + (1 − λ )

Original: g 2 and ˜ g ′
Revised: g 2 and ˜ g ′

Original: =
Revised: =

Original: λ ′ g ′ 

Revised: λ ′ g ′

Original: + (1 − λ ′ )
Revised: + (1 − λ ′ )

Original: g ′ 2 , we have ˜ g
Revised: g ′ 2 , we have ˜ g

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: g ′ because of ϕ (˜ g ) = ϕ
Revised: g ′ because of ϕ (˜ g ) = ϕ

Original: (˜ g ′ ) .
Revised: (˜ g ′ ) .

Original: Using the Intrusion-Freeness Theorem in (Guo & Mao, 2021), the graph interpolation transformation is invertible and the resulting original graph pair and mixup coefficient is unique, aka.
Revised: Using the Intrusion-Freeness Theorem in (Guo & Mao, 2021), the graph interpolation transformation is invertible and the resulting original graph pair and mixup coefficient is unique, aka.

Original: the equivalence of the interpolated graphs infers the equivalence of the original graphs and the interpolation coefficient λ .
Revised: the equivalence of the interpolated graphs infers the equivalence of the original graphs and the interpolation coefficient λ .

Original: Thus, we have either 1) λ
Revised: Thus, we have either 1) λ

Original: = λ ′ , g 1 = g ′ 1 , g 2 = g ′ 2 , or 2) λ
Revised: = λ ′ , g 1 = g ′ 1 , g 2 = g ′ 2 , or 2) λ

Original: = 1 − λ ′ , g 1 = g ′ 2 , g 2 = g 1 .
Revised: = 1 − λ ′ , g 1 = g ′ 2 , g 2 = g 1 .

Original: In either case, the assumption holds.
Revised: In either case, the assumption holds.

Original: Discussion .
Revised: Discussion .

Original: Proposition 2 relies on a powerful GNN encoder that can detect the isomorphism of interpolated graphs.
Revised: Proposition 2 relies on a powerful GNN encoder that can detect the isomorphism of interpolated graphs.

Original: This powerful GNN encoder exists both in theory and practice: 1) in theory, the universality of GNNs has been proved that they can approximate any function on graphs (Azizian &
Revised: This powerful GNN encoder exists both in theory and practice: 1) in theory, the universality of GNNs has been proved that they can approximate any function on graphs (Azizian &

Original: Lelarge, 2021); 2) in practice, Puny et al.
Revised: Lelarge, 2021); 2) in practice, Puny et al.

Original: (2022) have proposed a family of universal GNNs.
Revised: (2022) have proposed a family of universal GNNs.

Original: The possible limitation lies in the complexity of these methods (Puny et al., 2022; Maron et al., 2019b).
Revised: The possible limitation lies in the complexity of these methods (Puny et al., 2022; Maron et al., 2019b).

Original: We have tested E-GCL with the GIN and GraphTrans architectures.
Revised: We have tested E-GCL with the GIN and GraphTrans architectures.

Original: GIN is at most as powerful as the
Revised: GIN is at most as powerful as the

Original: 1-WL test (Leman & Weisfeiler, 1968; Azizian & Lelarge, 2021) and GraphTrans improves its ability to model long range interactions.
Revised: 1-WL test (Leman & Weisfeiler, 1968; Azizian & Lelarge, 2021) and GraphTrans improves its ability to model long range interactions.

Original: Our experiments show that GIN and GraphTrans is sufficient to demonstrate the improvement of E-GCL over previous methods.
Revised: Our experiments show that GIN and GraphTrans is sufficient to demonstrate the improvement of E-GCL over previous methods.

Original: Therefore, we leave the adoption of universal GNNs (Puny et al., 2022) as future work.
Revised: Therefore, we leave the adoption of universal GNNs (Puny et al., 2022) as future work.

Original: The Intrusion-Freeness Theorem (Guo & Mao, 2021) relies on the assumption in either Lemma 2 or
Revised: The Intrusion-Freeness Theorem (Guo & Mao, 2021) relies on the assumption in either Lemma 2 or

Original: Lemma 3 of their paper.
Revised: Lemma 3 of their paper.

Original: The assumption of their Lemma 2 states that: “ The node feature vectors for all graphs take values from a finite set and the values in the set are linearly independent ”.
Revised: The assumption of their Lemma 2 states that: “ The node feature vectors for all graphs take values from a finite set and the values in the set are linearly independent ”.

Original: This assumption holds strictly for chemical molecules (Hu et al., 2020), in which the node features are atom types that are finite and linearly independent ( i.e., you cannot combine two atoms to form another).
Revised: This assumption holds strictly for chemical molecules (Hu et al., 2020), in which the node features are atom types that are finite and linearly independent ( i.e., you cannot combine two atoms to form another).

Original: It is also automatically satisfied by anonymous social networks without any node information, i.e.,
Revised: It is also automatically satisfied by anonymous social networks without any node information, i.e.,

Original: COLLAB, RDT-B, RDT-M5K, IMDB-B in experiments.
Revised: COLLAB, RDT-B, RDT-M5K, IMDB-B in experiments.

Original: For other graphs with continuous node features such as word vectors, Lemma 3 provides a much weaker condition.
Revised: For other graphs with continuous node features such as word vectors, Lemma 3 provides a much weaker condition.

Original: Lemma 3 requires the linear independence of the entire node feature matrix for each graph in the dataset, which is more likely to hold in practice.
Revised: Lemma 3 requires the linear independence of the entire node feature matrix for each graph in the dataset, which is more likely to hold in practice.

Original: A. P ROOFS ON G ROUP A VERAGING
Revised: A. P ROOFS ON G ROUP A VERAGING

Original: Proof of that Φ( λ, g, g ′ ) is invariant to the relative permutation between g and g ′ in the sense that Φ( λ, g, g ′ ) =
Revised: Proof of that Φ( λ, g, g ′ ) is invariant to the relative permutation between g and g ′ in the sense that Φ( λ, g, g ′ ) =

Original: Φ( λ, T P ◦ g, g ′ )
Revised: Φ( λ, T P ◦ g, g ′ )

Original: =
Revised: =

Original: Φ( λ, g, T P ′ ◦
Revised: Φ( λ, g, T P ′ ◦

Original: g
Revised: g

Original: for all P, P ′ ∼ S n .
Revised: for all P, P ′ ∼ S n .

Original: Proof.
Revised: Proof.

Original: For all P ′ ∈ S n , we have
Revised: For all P ′ ∈ S n , we have

Original: [Equation]
Revised: [Equation]

Original: Let P ′′− 1 =
Revised: Let P ′′− 1 =

Original: P − 1 P ′ , we have P = P ′ P ′′ and
Revised: P − 1 P ′ , we have P = P ′ P ′′ and

Original: [Equation]
Revised: [Equation]

Original: We can similarly prove Φ( λ, g, g ′ )
Revised: We can similarly prove Φ( λ, g, g ′ )

Original: =
Revised: =

Original: Φ( λ, g, T P ′ ◦
Revised: Φ( λ, g, T P ′ ◦

Original: g
Revised: g

Original: Proposition 1.
Revised: Proposition 1.

Original: The contrastive loss with ϕ ( λT P ◦ g +(1 − λ ) g ′ )
Revised: The contrastive loss with ϕ ( λT P ◦ g +(1 − λ ) g ′ )

Original: upper bounds the loss of an invariant network to relative permutation 1 | S n | (cid:80) P ∈ S n ρ ( ϕ ( λ i T P ◦
Revised: upper bounds the loss of an invariant network to relative permutation 1 | S n | (cid:80) P ∈ S n ρ ( ϕ ( λ i T P ◦

Original: g i + (1 − λ i )
Revised: g i + (1 − λ i )

Original: g ′ i )) .
Revised: g ′ i )) .

Original: Proof.
Revised: Proof.

Original: The loss of NT-Xent is:
Revised: The loss of NT-Xent is:

Original: [Equation]
Revised: [Equation]

Original: (16) where ℓ pos aims to minimize the distance between positive views.
Revised: (16) where ℓ pos aims to minimize the distance between positive views.

Original: Let { ( λ 1 , g 1 , g ′ 1 ) , ( λ 2 , g 2 , g ′ 2 ) , ..., ( λ N , g N , g ′ N ) } be the batch of original graph pairs and the mixup coefficients.
Revised: Let { ( λ 1 , g 1 , g ′ 1 ) , ( λ 2 , g 2 , g ′ 2 ) , ..., ( λ N , g N , g ′ N ) } be the batch of original graph pairs and the mixup coefficients.

Original: If we use MSE to minimize the distance between the feature interpolation view and representation interpolation view and omit the temperature hyperparameter τ , ℓ pos can be written as:
Revised: If we use MSE to minimize the distance between the feature interpolation view and representation interpolation view and omit the temperature hyperparameter τ , ℓ pos can be written as:

Original: [Equation]
Revised: [Equation]

Original: where P i ∼ S n for all i ∈
Revised: where P i ∼ S n for all i ∈

Original: [ N ] .
Revised: [ N ] .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the last step is by Jensen’s inequality.
Revised: where the last step is by Jensen’s inequality.

Original: Notice that, the contrastive loss upper bounds the distance between ρ ( λ i ϕ ( g i )
Revised: Notice that, the contrastive loss upper bounds the distance between ρ ( λ i ϕ ( g i )

Original: + (1 − λ i ) ϕ
Revised: + (1 − λ i ) ϕ

Original: ( g ′ i )) and 1 | S n | (cid:80) P j ∈ S n ρ ( ψ ( λ i , T P j ◦ g
Revised: ( g ′ i )) and 1 | S n | (cid:80) P j ∈ S n ρ ( ψ ( λ i , T P j ◦ g

Original: i , g ′ i )) , which is the group averaging of ρ ◦ ψ .
Revised: i , g ′ i )) , which is the group averaging of ρ ◦ ψ .

Original: By the property of group averaging (Puny et al., 2022; Yarotsky, 2022; Murphy et al., 2019), it follows that| S n | (cid:80) P j ∈ S n ρ ( ψ ( λ i , T P j ◦ g
Revised: By the property of group averaging (Puny et al., 2022; Yarotsky, 2022; Murphy et al., 2019), it follows that| S n | (cid:80) P j ∈ S n ρ ( ψ ( λ i , T P j ◦ g

Original: i , g ′ i )) is invariant to relative permutation as well.
Revised: i , g ′ i )) is invariant to relative permutation as well.

Original: B R ELATED W ORKS
Revised: B R ELATED W ORKS

Original: We have briefly introduced GCL methods in Section 2.
Revised: We have briefly introduced GCL methods in Section 2.

Original: In this section, we first discuss E-GCL’s connections and differences with E-SSL and IfMixup.
Revised: In this section, we first discuss E-GCL’s connections and differences with E-SSL and IfMixup.

Original: Then, we present E-GCL’s relations to other graph mixup methods and geometric deep learning.
Revised: Then, we present E-GCL’s relations to other graph mixup methods and geometric deep learning.

Original: E-SSL.
Revised: E-SSL.

Original: E-GCL is inspired by the pioneer E-SSL works (Dangovski et al., 2022; Chuang et al., 2022) in CV and NLP.
Revised: E-GCL is inspired by the pioneer E-SSL works (Dangovski et al., 2022; Chuang et al., 2022) in CV and NLP.

Original: Dangovski et al.
Revised: Dangovski et al.

Original: (2022) find that, when the previous insensitive objective fail on certain augmentations, applying a sensitive objective to the same augmentations can improve performance.
Revised: (2022) find that, when the previous insensitive objective fail on certain augmentations, applying a sensitive objective to the same augmentations can improve performance.

Original: Specifically, they apply a sensitive objective on the four-fold rotations of images to improve existing SSL methods.
Revised: Specifically, they apply a sensitive objective on the four-fold rotations of images to improve existing SSL methods.

Original: In NLP, Chuang et al.
Revised: In NLP, Chuang et al.

Original: (2022) implement the sensitive objective as discriminating word replacement to improve sentence level embedding.
Revised: (2022) implement the sensitive objective as discriminating word replacement to improve sentence level embedding.

Original: Adapting E-SSL for graphs is challenging because existing intra-graph augmentations share the common paradigm of structure corruption, making it hard to categorize them into sensitive and insensitive augmentations.
Revised: Adapting E-SSL for graphs is challenging because existing intra-graph augmentations share the common paradigm of structure corruption, making it hard to categorize them into sensitive and insensitive augmentations.

Original: This works is different from previous E-SSL works that we introduce cross-graph augmentation to create global semantic shifts.
Revised: This works is different from previous E-SSL works that we introduce cross-graph augmentation to create global semantic shifts.

Original: By encouraging the sensitivity to cross-graph augmentation, we protect representations from the harmful aggressive intra-graph augmentations.
Revised: By encouraging the sensitivity to cross-graph augmentation, we protect representations from the harmful aggressive intra-graph augmentations.

Original: IfMixup.
Revised: IfMixup.

Original: In this work, we extend IfMixup (Guo & Mao, 2021) for cross-graph augmentation in SSL.
Revised: In this work, we extend IfMixup (Guo & Mao, 2021) for cross-graph augmentation in SSL.

Original: IfMixup mitigates the structural differences by padding virtual nodes for graph interpolation.
Revised: IfMixup mitigates the structural differences by padding virtual nodes for graph interpolation.

Original: It is previously developed for supervised learning.
Revised: It is previously developed for supervised learning.

Original: For SSL, we propose to supervise mixed graphs by the interpolation of the original graphs’ representations.
Revised: For SSL, we propose to supervise mixed graphs by the interpolation of the original graphs’ representations.

Original: We also connect graph mixup to group theory and address its limitation of sensitivity to the relative permutation.
Revised: We also connect graph mixup to group theory and address its limitation of sensitivity to the relative permutation.

Original: Graph Mixup.
Revised: Graph Mixup.

Original: Graph mixup has been a challenging task due to graphs’ irregular structures.
Revised: Graph mixup has been a challenging task due to graphs’ irregular structures.

Original: GraphMix (Verma et al., 2021) sidesteps the structural differences by mixing only node features.
Revised: GraphMix (Verma et al., 2021) sidesteps the structural differences by mixing only node features.

Original: Wang et al.
Revised: Wang et al.

Original: (2021) mixup the graph representations for graph classification.
Revised: (2021) mixup the graph representations for graph classification.

Original: G -Mixup (Han et al., 2022) samples adjacency matrices from the mixed graphons of two classes as graph mixup.
Revised: G -Mixup (Han et al., 2022) samples adjacency matrices from the mixed graphons of two classes as graph mixup.

Original: We opt out G -Mixup in our method due to its following limitations:
Revised: We opt out G -Mixup in our method due to its following limitations:

Original: • G -mixup does not support node feature mixup.
Revised: • G -mixup does not support node feature mixup.

Original: Their paper has no experiments on attributed graphs.
Revised: Their paper has no experiments on attributed graphs.

Original: Moreover, their instruction for sampling mixed node
Revised: Moreover, their instruction for sampling mixed node

Original: features
Revised: features

Original: is vague:
Revised: is vague:

Original: the
Revised: the

Original: instruction
Revised: instruction

Original: does not describe the sampling strategy and the used distribution.
Revised: does not describe the sampling strategy and the used distribution.

Original: • G -mixup does not scale to large graphs due to its O ( N 3 ) complexity ( N is the number of nodes).
Revised: • G -mixup does not scale to large graphs due to its O ( N 3 ) complexity ( N is the number of nodes).

Original: The
Revised: The

Original: high complexity is due to the SVD (Chatterjee, 2015) in graphon estimation.
Revised: high complexity is due to the SVD (Chatterjee, 2015) in graphon estimation.

Original: In comparison,
Revised: In comparison,

Original: IfMixup
Revised: IfMixup

Original: is
Revised: is

Original: scalable to large graphs with a linear complexity
Revised: scalable to large graphs with a linear complexity

Original: to
Revised: to

Original: edge
Revised: edge

Original: and
Revised: and

Original: node
Revised: node

Original: numbers O
Revised: numbers O

Original: E + N
Revised: E + N

Original: • G -mixup requires class labeling to estimate the graphon in each class.
Revised: • G -mixup requires class labeling to estimate the graphon in each class.

Original: In SSL
Revised: In SSL

Original: class labeling is
Revised: class labeling is

Original: unavailable.
Revised: unavailable.

Original: If we were to take risks and treated each graph as a class, the obtained graphon would be suboptimal due to the limited sample.
Revised: If we were to take risks and treated each graph as a class, the obtained graphon would be suboptimal due to the limited sample.

Original: In this case, it is outperformed by IfMixup (Table 3).
Revised: In this case, it is outperformed by IfMixup (Table 3).

Original: Geometric Deep Learning .
Revised: Geometric Deep Learning .

Original: Invariance and equivariance have been heavily studied under the scope of geometric deep learning (Bronstein et al., 2021).
Revised: Invariance and equivariance have been heavily studied under the scope of geometric deep learning (Bronstein et al., 2021).

Original: The goal is to explore geometric symmetries in neural architecture designs for effective weight sharing to reduce sample complexity (Cohen &
Revised: The goal is to explore geometric symmetries in neural architecture designs for effective weight sharing to reduce sample complexity (Cohen &

Original: Welling, 2016).
Revised: Welling, 2016).

Original: For example, generalizing the convolution operation from the Z 2 grids to the pgroup makes convolution equivariant to the four-fold rotation (Cohen & Welling, 2016); the message passing operation in GNNs maintains the node-level output equivariant to the permutation group
Revised: For example, generalizing the convolution operation from the Z 2 grids to the pgroup makes convolution equivariant to the four-fold rotation (Cohen & Welling, 2016); the message passing operation in GNNs maintains the node-level output equivariant to the permutation group

Original: S n (Battaglia et al., 2018).
Revised: S n (Battaglia et al., 2018).

Original: Exploring geometric symmetries like rotation and permutation have greatly improved performances in various applications, including galaxy morphology (Dieleman et al., 2015), point clouds (Chen et al., 2021; Zaheer et al., 2017), and spherical images (Cohen et al., 2018).
Revised: Exploring geometric symmetries like rotation and permutation have greatly improved performances in various applications, including galaxy morphology (Dieleman et al., 2015), point clouds (Chen et al., 2021; Zaheer et al., 2017), and spherical images (Cohen et al., 2018).

Original: Our work is different from geometric deep learning because we do not study neural architectures.
Revised: Our work is different from geometric deep learning because we do not study neural architectures.

Original: We study transformations that change the underlying semantics of graphs rather than their “poses” in the geometric space.
Revised: We study transformations that change the underlying semantics of graphs rather than their “poses” in the geometric space.

Original: C L IMITATIONS
Revised: C L IMITATIONS

Original: SSL is limited in that it has little knowledge of the downstream tasks.
Revised: SSL is limited in that it has little knowledge of the downstream tasks.

Original: Each type of intra-graph augmentation represents a human prior that performs differently on different downstream datasets (Purushwalkam & Gupta, 2020).
Revised: Each type of intra-graph augmentation represents a human prior that performs differently on different downstream datasets (Purushwalkam & Gupta, 2020).

Original: Our work grounds the equivariance mechanism as a domain agnostic cross-graph augmentation to facilitate representations with the sensitivity to global semantic shifts.
Revised: Our work grounds the equivariance mechanism as a domain agnostic cross-graph augmentation to facilitate representations with the sensitivity to global semantic shifts.

Original: Previous E-SSL works (Dangovski et al., 2022; Chuang et al., 2022) in CV and NLP divide existing data augmentations into two sets of sensitive augmentations and insensitive augmentations.
Revised: Previous E-SSL works (Dangovski et al., 2022; Chuang et al., 2022) in CV and NLP divide existing data augmentations into two sets of sensitive augmentations and insensitive augmentations.

Original: Our limitation is that we leave the existing intra-graph augmentations untouched as the insensitive augmentations, although the insensitivity to some aggressive intra-graph augmentations might diminish the sensitivity to cross-graph augmentations.
Revised: Our limitation is that we leave the existing intra-graph augmentations untouched as the insensitive augmentations, although the insensitivity to some aggressive intra-graph augmentations might diminish the sensitivity to cross-graph augmentations.

Original: However, disentangling the aggressive augmentations from the others requires extensive tests or domain knowledge.
Revised: However, disentangling the aggressive augmentations from the others requires extensive tests or domain knowledge.

Original: We leave it as future work.
Revised: We leave it as future work.

Original: Further, our equivariance branch is a patch to the limitations of the invariance branch.
Revised: Further, our equivariance branch is a patch to the limitations of the invariance branch.

Original: In future, we will explore GCL of the complete focus on equivariance without the invariance branch.
Revised: In future, we will explore GCL of the complete focus on equivariance without the invariance branch.

Original: Equivariance is a high-level mathematical concept unifying sensitivity and insensitivity.
Revised: Equivariance is a high-level mathematical concept unifying sensitivity and insensitivity.

Original: It has promising potential in graph representation learning.
Revised: It has promising potential in graph representation learning.

Original: Our work is limited that we explore the equivariance to a simple cross-graph augmentation of graph mixup.
Revised: Our work is limited that we explore the equivariance to a simple cross-graph augmentation of graph mixup.

Original: We believe there are other equivariance principles worth exploring.
Revised: We believe there are other equivariance principles worth exploring.

Original: The limitations and assumptions of the theoretical results have been discussed in Appendix A.
Revised: The limitations and assumptions of the theoretical results have been discussed in Appendix A.

Original: D E XPERIMENT D. A BLATION S TUDIES FOR G RAPH I NTERPOLATION
Revised: D E XPERIMENT D. A BLATION S TUDIES FOR G RAPH I NTERPOLATION

Original: [Table]
Revised: [Table]

Original: Comparison with G -mixup.
Revised: Comparison with G -mixup.

Original: Our E-GCL framework is agnostic to the graph mixup strategy for crossgraph augmentation.
Revised: Our E-GCL framework is agnostic to the graph mixup strategy for crossgraph augmentation.

Original: We also exploit G -Mixup
Revised: We also exploit G -Mixup

Original: (Han et al., 2022) for the cross-graph augmentation.
Revised: (Han et al., 2022) for the cross-graph augmentation.

Original: Different from the linear interpolation strategy of IfMixup (Guo & Mao, 2021), G -Mixup yields discrete adjacency matrices by sampling from the mixed graphons of two classes.
Revised: Different from the linear interpolation strategy of IfMixup (Guo & Mao, 2021), G -Mixup yields discrete adjacency matrices by sampling from the mixed graphons of two classes.

Original: To adapt G -Mixup to the SSL setting, we use their source code and treat each graph as a class.
Revised: To adapt G -Mixup to the SSL setting, we use their source code and treat each graph as a class.

Original: Grid search is conducted to tune the hyperparameters: α and ω .
Revised: Grid search is conducted to tune the hyperparameters: α and ω .

Original: Table 3 shows the performance comparison between different graph mixup strategies.
Revised: Table 3 shows the performance comparison between different graph mixup strategies.

Original: We do not compare performances on attributed graph datasets because
Revised: We do not compare performances on attributed graph datasets because

Original: [Table] [Table]
Revised: [Table] [Table]

Original: G -Mixup does not support mixing node features.
Revised: G -Mixup does not support mixing node features.

Original: We have discussed the limitations of G -mixup in the related works section (Appendix B)
Revised: We have discussed the limitations of G -mixup in the related works section (Appendix B)

Original: [Table]
Revised: [Table]

Original: Our linear interpolation strategy substantially outperforms G -mixup in 4 out of 4 TU datasets.
Revised: Our linear interpolation strategy substantially outperforms G -mixup in three out of four TU datasets.

Original: IfMixup also shows  better average performance than G -mixup.
Revised: IfMixup also shows 1 . 15% better average accuracy than G -mixup.

Original: While G -mixup improves over the
Revised: While G -mixup improves over the

Original: BarlowTwins baseline in the last three datasets, it performs worse in the COLLAB dataset.
Revised: BarlowTwins baseline in the last three datasets, it performs worse in the COLLAB dataset.

Original: Interpolation Position.
Revised: Interpolation Position.

Original: The cross-graph augmentation is supervised by the interpolation of the original graphs’ representations.
Revised: The cross-graph augmentation is supervised by the interpolation of the original graphs’ representations.

Original: We interpolate the representations before the projector to let the gradient mainly optimize the equivariance of the GNN encoder ϕ .
Revised: We interpolate the representations before the projector to let the gradient mainly optimize the equivariance of the GNN encoder ϕ .

Original: With this design, the encoder is trained to approach Equation (7), which is equivariance to cross-graph augmentation.
Revised: With this design, the encoder is trained to approach Equation (7), which is equivariance to cross-graph augmentation.

Original: However, strict equivariance is hardly achieved by GNNs.
Revised: However, strict equivariance is hardly achieved by GNNs.

Original: Therefore, we have the projector learn to ignore the subtle difference between the two sides of Equation (7) and reduce the task’s difficulty.
Revised: Therefore, we have the projector learn to ignore the subtle difference between the two sides of Equation (7) and reduce the task’s difficulty.

Original: Tablecompares the performances of not using equivariance and the performances of E-GCL when applying representation interpolation at different positions: before projector, after projector, and interpolating the cosine similarity scores.
Revised: Tablecompares the performances of not using equivariance and the performances of E-GCL when applying representation interpolation at different positions: before projector, after projector, and interpolating the cosine similarity scores.

Original: We verify that 1) before the projector is the optimal choice for both
Revised: We verify that 1) before the projector is the optimal choice for both

Original: GraphCL and BarlowTwins, and 2) using equivariance at all positions consistently outperforms No
Revised: GraphCL and BarlowTwins, and 2) using equivariance at all positions consistently outperforms No

Original: Equivariance.
Revised: Equivariance.

Original: More than better performance, before projector also allows our equivariance principle to work with a broader family of SSL frameworks (Zbontar et al., 2021;
Revised: More than better performance, before projector also allows our equivariance principle to work with a broader family of SSL frameworks (Zbontar et al., 2021;

Original: Bardes et al., 2022).
Revised: Bardes et al., 2022).

Original: Influence of Mixing Dummy Nodes.
Revised: Influence of Mixing Dummy Nodes.

Original: We pad dummy nodes to the original graphs to the same size before mixup.
Revised: We pad dummy nodes to the original graphs to the same size before mixup.

Original: As dummy nodes have zero features, adding them to original graphs does not introduce any noise into the interpolations.
Revised: As dummy nodes have zero features, adding them to original graphs does not introduce any noise into the interpolations.

Original: We demonstrate the neutral effect of dummy nodes by comparing the performances of mixing graphs with very different sizes and the original E-GCL in Table 5.
Revised: We demonstrate the neutral effect of dummy nodes by comparing the performances of mixing graphs with very different sizes and the original E-GCL in Table 5.

Original: The performance difference is only 0 .
Revised: The performance difference is only 0 .

Original: We deliberately create size differences between mixed graphs in the different size experiment.
Revised: We deliberately create size differences between mixed graphs in the different size experiment.

Original: Specifically, we sort the in-batch graphs by their sizes { g 1 , g 2 , ..., g N } , such that | g 1 | ≤ | g 2 | ≤ ...
Revised: Specifically, we sort the in-batch graphs by their sizes { g 1 , g 2 , ..., g N } , such that | g 1 | ≤ | g 2 | ≤ ...

Original: ≤ | g N | .
Revised: ≤ | g N | .

Original: Then, we mix the i -th graph with the ( N − i + 1) -th graph.
Revised: Then, we mix the i -th graph with the ( N − i + 1) -th graph.

Original: In this way, we create size differences in every batch such that the smallest graph g 1 will be mixed with the largest graph g N .
Revised: In this way, we create size differences in every batch such that the smallest graph g 1 will be mixed with the largest graph g N .

Original: We use the BarlowTwins backbone for the experiments.
Revised: We use the BarlowTwins SSL framework for the experiments.

Original: D. I NFLUENCE OF E QUIVARIANCE ON A GGRESSIVE A UGMENTATIONS .
Revised: D. I NFLUENCE OF S AMPLED P ERMUTATION N UMBERS

Revised: Experimental Settings.

Revised: In the existing experiments ( cf.

Revised: Section 4), we randomly permute one of the input graphs before graph interpolation.

Revised: This strategy is a 1-sample estimator of the group averaging and shows improvements over the state-of-the-art baselines.

Revised: Here we conduct ablation experiments to show that: 1) using random permutation improves performance; 2) using more samples to approximate group averaging improves performance, and the improvement of using more samples is only marginal and comes at the cost of increased complexity.

Revised: We conduct unsupervised learning experiments using the TU datasets.

Revised: We employ E-GCL with BarlowTwins framework, set α = 0 . 1 , and perform 5 experiments of different ω

Revised: = { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 .

Revised: 5 } values.

Revised: We report the average and max values of the mean accuracies (%) of these 5 different experiments.

Revised: The forward time is measured across 100 batches of size 128 on the COLLAB dataset

Revised: [Table]

Revised: [Table]

Revised: Experimental Results.

Revised: Table 6 shows that using random permutation consistently outperforms not using random permutation (No rand.

Revised: perm.).

Revised: Specifically, the 1-sample estimator shows better performance and adds no computational cost compared to No rand.

Revised: perm..

Revised: Further, it shows that using more samples to estimate the group averaging improves performance.

Revised: The 10-sample estimator gives the best performance.

Revised: However, the 10-sample estimator’s improvement is only marginal ( 0 .

Revised: 05% ∼ 0 .

Revised: compared to the 1-sample estimator

Revised: [Table]

Revised: but leads to almost three times increase in time complexity.

Revised: Thus, we recommend the 1-sample estimator in implementation.

Revised: D. I NFLUENCE OF E QUIVARIANCE ON A GGRESSIVE A UGMENTATIONS .

Revised: Intra-graph augmentations are problematic that they sometimes harmfully enforce insensitivity to semantically shifted graphs ( i.e., aggressive augmentations).

Revised: To patch the problem, cross-graph augmentations always enforce sensitivity to semantically shifted graphs that are generated by graph interpolation.

Revised: Consequently, the equivariance to cross-graph augmentations diminish the harmful invariance of aggressive intra-graph augmentations that change global semantics, leading to better performance.

Original: To justify that equivariance can mitigate the negative effect of aggressive augmentations, we conduct experiments w.r.t.
Revised: To justify that equivariance can mitigate the negative effect of aggressive augmentations, we conduct experiments w.r.t.

Original: Average Confusion Ratio (ACR) (Wang et al., 2022).
Revised: Average Confusion Ratio (ACR) (Wang et al., 2022).

Original: ACR is the metric to measure the ratio that the anchor graph’s nearest neighbors are the views of different graphs, rather than the other views of the same anchor.
Revised: ACR is the metric to measure the ratio that the anchor graph’s nearest neighbors are the views of different graphs, rather than the other views of the same anchor.

Original: Higher ACR indicates that the graph representations are less powerful to distinguish different graphs, thus reflecting worse negative influences of aggressive augmentations.
Revised: Higher ACR indicates that the graph representations are less powerful to distinguish different graphs, thus reflecting worse negative influences of aggressive augmentations.

Original: We use the GraphCL checkpoints from Table 2b and report the ACR scores on the ZINC15 dataset.
Revised: We use the GraphCL checkpoints from Table 2b and report the ACR scores on the ZINC15 dataset.

Original: As shown by Table 6, E-GCL achieves lower ACR than I-GCL.
Revised: As shown by Table 7, applying equivariance to cross-graph augmentation improves the ACR for GraphCL.

Original: This suggests that E-GCL’s equivariance principleachieves better cross-graph discrimination and mitigates the negative influences of aggressive augmentations, as compared to I-GCL.
Revised: This demonstrates that the equivariance principle mitigates the negative influences of aggressive augmentations, thus leading to better graph discrimination performance. D. R ESULTS IN THE F IRST S UBMISSION .

Original: [Table]
Revised: We include the results from our first submission for your reference (Table 8). We use Table 1 to replace Table 8 to report baseline performances under a consistent evaluation protocol.

Original: E I MPLEMENTATION D ETAILS E. E-GCL P SEUDO C ODE
Revised: E I MPLEMENTATION D ETAILS E. E-GCL P SEUDO C ODE

Original: Algorithm 1 presents the pseudo code of E-GCL. Algorithm
Revised: Algorithm 1 presents the pseudo code of E-GCL. [Table]

Original: [Table]
Revised: [Table]

Original: E. I MPLEMENTATION
Revised: E. I MPLEMENTATION

Original: We implement GNNs with the PyTorch Geometric library (Fey & Lenssen, 2019), which is opensource under the MIT license.
Revised: We implement GNNs with the PyTorch Geometric library (Fey & Lenssen, 2019), which is opensource under the MIT license.

Original: We conduct experiments using an NVIDIA V100 GPU (32 GB memory) on a server with a 40-core Intel CPU [Table]
Revised: We conduct experiments using an NVIDIA V100 GPU (32 GB memory) on a server with a 40-core Intel CPU.

Original: Baselines .
Revised: Baselines .

Original: For comparison, we report the performances of the following baseline methods:
Revised: For comparison, we report the performances of the following baseline methods:

Original: • graph2vec
Revised: • graph2vec

Original: (Narayanan et al., 2017) treats each graph as a document and employ a document embedding approach to learn graph embeddings.
Revised: (Narayanan et al., 2017) treats each graph as a document and employs a document embedding approach to learn graph embeddings.

Original: • Infomax (Veliˇckovi´c et al., 2019) maximizes the mutual information between the patch representations that summarize the subgraphs centered around nodes and the global readout of graphs.
Revised: • Infomax (Veliˇckovi´c et al., 2019) maximizes the mutual information between the patch representations that summarize the subgraphs centered around nodes and the global readout of graphs.

Original: • ContextPred (Hu et al., 2020) aims to predict the surrounding graph structures using the embeddings of the local subgraphs.
Revised: • ContextPred (Hu et al., 2020) aims to predict the surrounding graph structures using the embeddings of the local subgraphs.

Original: The prediction problem is formulated as a binary prediction with negative samples to resolve the intractability of predicting graph structures.
Revised: The prediction problem is formulated as a binary prediction with negative samples to resolve the intractability of predicting graph structures.

Original: InfoGraph (Sun et al., 2020) learns representations by maximizing the mutual information between graph-level representations and graph substructures of different scales, e.g., nodes and edges.
Revised: InfoGraph (Sun et al., 2020) learns representations by maximizing the mutual information between graph-level representations and graph substructures of different scales, e [Table] g., nodes and edges.

Original: • GraphCL
Revised: • GraphCL

Original: (You et al., 2020) explores the combination of intra-graph augmentations, such as node dropping, edge dropping, and subgraph, for contrastive graph representation learning.
Revised: (You et al., 2020) explores the combination of intra-graph augmentations, such as node dropping, edge dropping, and subgraph, for contrastive graph representation learning.

Original: • AD-GCL
Revised: • AD-GCL

Original: (Suresh et al., 2021) trains an augmenter to adversarially drop edges to remove redundant information.
Revised: (Suresh et al., 2021) trains an augmenter to adversarially drop edges to remove redundant information.

Original: • GraphLOG (Xu et al., 2021) utilizes clustering to contrast local instances and the corresponding hierarchical prototypes at every clustering layer.
Revised: • GraphLOG (Xu et al., 2021) utilizes clustering to contrast local instances and the corresponding hierarchical prototypes at every clustering layer.

Original: • JOAO (You et al., 2021) aims to automate the selection of graph augmentations via solving a bi-level optimization problem.
Revised: • JOAO (You et al., 2021) aims to automate the selection of graph augmentations via solving a bi-level optimization problem.

Original: • GraphMAE (Hou et al., 2022) explores GNN pretraining by reconstructing node features using a masked autoencoder.
Revised: • GraphMAE (Hou et al., 2022) explores GNN pretraining by reconstructing node features using a masked autoencoder.

Original: • RGCL (Li et al., 2022) learns a rationale generator to protect the salient features during data augmentation.
Revised: • RGCL (Li et al., 2022) learns a rationale generator to protect the salient features during data augmentation.

Original:  We do not compare with DGCL (Li et al., 2021) because their experiments follow a different protocol.
Revised: [Table] We do not compare with DGCL (Li et al., 2021) because their experiments follow a different protocol.

Original: DGCL selects GNN layers, dimension sizes and batch sizes based on the test set performance on each dataset.
Revised: DGCL selects GNN layers, dimension sizes and batch sizes based on the test set performance on each dataset.

Original: However, other baselines and our methods stick to the same GNN configuration for all the datasets.
Revised: However, other baselines and our methods stick to the same GNN configuration for all the datasets.

Original: Also, re-implementation is hard because the source code has not been released.
Revised: Also, re-implementation is hard because the source code has not been released.

Original: Augmentations .
Revised: Augmentations .

Original: Our intra-graph augmentation follows GraphCL
Revised: Our intra-graph augmentation follows GraphCL

Original: (You et al., 2020).
Revised: (You et al., 2020).

Original: We use dropNode for the unsupervised learning experiments and use both dropNode and subgraph for the transfer learning experiments.
Revised: We use dropNode for the unsupervised learning experiments and use both dropNode and subgraph for the transfer learning experiments.

Original: For the cross-graph augmentation of graph mixup, we include the self-loops of virtual nodes in the adjacency matrix due to better empirical performance.
Revised: For the cross-graph augmentation of graph mixup, we include the self-loops of virtual nodes in the adjacency matrix due to better empirical performance.

Original: Before graph mixup, we randomly shuffle the node order of one of the input graphs to have random relative permutations between input graphs, which leads to slightly better empirical performance.
Revised: Before graph mixup, we randomly shuffle the node order of one of the input graphs to have random relative permutations between input graphs, which leads to slightly better empirical performance.

Original: Implementation to Different SSL Frameworks .
Revised: Implementation to Different SSL Frameworks .

Original: For the BarlowTwins and SimSiam backbones, which use no negative samples, E-GCL follows strictly to their original loss functions.
Revised: For the BarlowTwins and SimSiam backbones, which use no negative samples, E-GCL follows strictly to their original loss functions.

Original: For the contrastive backbone GraphCL, E-GCL uses both intra-graph augmentations and cross-graph aug- mentations as negative samples to facilitate better cross-graph discrimination.
Revised: For the contrastive backbone GraphCL, E-GCL uses both intra-graph augmentations and cross-graph aug- mentations as negative samples to facilitate better cross-graph discrimination.

Original: Specifically, we have two types of embeddings for cross-graph augmentations: the feature interpolation embeddings { z 3 i | z 3 i = ρ ( ϕ ( λg 1 i + (1 − λ ) λg
Revised: Specifically, we have two types of embeddings for cross-graph augmentations: the feature interpolation embeddings { z 3 i | z 3 i = ρ ( ϕ ( λg 1 i + (1 − λ ) λg

Original: 2 π ( i ) ))
Revised: 2 π ( i ) ))

Original: } Ni =1 and representation interpolation embeddings { z 4 i | z 4 i = ρ ( λϕ ( g 1 i ) + (1 − λ ) ϕ ( λg 2 π ( i ) )) } Ni =1 .
Revised: } Ni = embeddings { z 4 i | z 4 i = ρ ( λϕ ( g 1 i ) + (1 − λ ) ϕ ( λg 2 π ( i ) )) } Ni =1 .

Original: To avoid overfitting to one type of cross-graph augmentations, we use half of each type as the anchor graphs and the other half as the negative samples.
Revised: To avoid overfitting to one type of cross-graph augmentations, we use half of each type as the anchor graphs and the other half as the negative [Table] [Table] [Table]

Revised: samples.

Original: Define m 3 j = z 3+ 1 ( j>N/ 2) j and m
Revised: Define m 3 j = z 3+ 1 ( j>N/ 2) j and m

Original: 4 j = z 4 − 1 ( j ≤ N/ 2) j , where 1 ( · ) is a binary indictor that returns 1 when the condition holds and returns 0 otherwise.
Revised: 4 j = z 4 − 1 ( j ≤ N/ 2) j , where 1 ( · ) is a binary indicator that returns 1 when the condition holds and returns 0 otherwise.

Original: { m 3 j } Nj =1 contains half of the feature interpolation embeddings and half of the representation interpolation embeddings; { m 4 j } Nj =1 contains the other half.
Revised: { m 3 j } Nj =1 contains half of the feature interpolation embeddings and half of the representation interpolation embeddings; { m 4 j } Nj =1 contains the other half.

Original: The loss function when using GraphCL backbone can be written as:
Revised: The loss function when using GraphCL backbone can be written as:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original:  [Table] [Table]
Revised: [Equation] [Table] [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: Notice that, we treat sample pairs that share partial graph identities ( e
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: g., ( m 3 j , z 1 j ) and ( m 3 j , m 4 π ( j ) )) as semi-positives and exclude them from negative samples.
Revised: [Equation] [Table] [Equation] Notice that, we treat sample pairs that share partial graph identities ( e.g., ( m 3 j , z 1 j ) and ( m 3 j , m 4 π ( j ) )) as semi-positives and exclude them from negative samples.

Original: Alignment and Uniformity Loss .
Revised: Alignment and Uniformity Loss.

Original: We use dropNode as the intra-graph augmentation and graph interpolation as the cross-graph augmentation.
Revised: We use dropNode as the intra-graph augmentation and graph interpolation as the cross-graph augmentation.

Original: We pre-train GraphCL for 1000 steps before evaluation.
Revised: We pre-train GraphCL for 1000 steps before evaluation.

Original: The GraphCL setup follows that in Table 8.
Revised: The GraphCL setup follows that in Table 10.

Original: We split the original pre-training dataset into two subsets
Revised: We split the original pre-training dataset into two subsets.

Original: [Table] The 80% subset is used for pre-training [Table] and 51200 samples from the other 20% subset are used for loss evaluation.
Revised: The 80% subset is used for pre-training, and 51200 samples from the other 20% subset are used for loss evaluation.

Original: E. H YPER - PARAMETERS
Revised: E. H YPER - PARAMETERS

Original: Unsupervised learning hyper-parameters are shown in Table 7.
Revised: Unsupervised Learning. The hyper-parameters are shown in Table 9.

Original: We use the same three layer GIN from (You et al., 2020).
Revised: We use the same three-layer GIN from (You et al., 2020).

Original: Following Zbontar et al.
Revised: Following Zbontar et al.

Original: (2021), the BarlowTwins backbone uses a threelayer MLP projector with hidden dimensions that are four times the input dimension.
Revised: (2021), the BarlowTwins backbone uses a threelayer MLP projector with hidden dimensions that are four times the input dimension.

Original: Following You et al.
Revised: Following You et al.

Original: , we use  learning rate 0 .
Revised: (2020), we use a learning rate 0 .

Original: 01 , batch size 128 and no weight decay.
Revised: 01 , batch size 128 and no weight decay.

Original: For E-GCL, we conduct grid search for α and ω in the following ranges α = { 0 . 1 , 0 . 4 , 1 , 2 , 4 } , ω
Revised: For E-GCL, we conduct grid search for α and ω in the following ranges α = { 0 . 1 , 0 . 4 , 1 , 2 , 4 } , ω

Original: = { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .
Revised: = { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .

Original: We train the GNN for 60 epochs and evaluate the generated embedding using non-linear SVMs every 10 epochs.
Revised: We train the GNN for 60 epochs and evaluate the generated embedding using non-linear SVMs every 10 epochs.

Original: Following You et al.
Revised: Following You et al.

Original: (2020), we search for the regularization parameter of SVMs in { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 , 1000 } .
Revised: (2020), we search for the regularization parameter of SVMs in { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 , 1000 } .

Original: We report performance from the epoch with the highest validation set performance.
Revised: Following (You et al., 2021), we report test accuracy of the epoch selected by validation set.

Original: Transfer learning hyper-parameters are shown in Table 8.
Revised: Transfer Learning. The hyper-parameters are shown in Table 10.

Original: We use a large batch size 2048 for
Revised: We use a large batch size 2048 for

Original: BarlowTwins and SimSiam to speed up pre-training.
Revised: BarlowTwins and SimSiam to speed up pre-training.

Original: The large batch size increases no complexity because BarlowTwins and SimSiam use no negative samples.
Revised: The large batch size increases no complexity because Barlowwins and SimSiam use no negative samples.

Original: Following Zbontar et al.
Revised: Following Zbontar et al.

Original: (2021), the
Revised: (2021), the

Original: BarlowTwins uses a three-layer MLP projector with hidden dimensions that are four times the input dimension.
Revised: BarlowTwins uses a three-layer MLP projector with hidden dimensions that are four times the input [Table] [Table] [Figure]

Revised: [Table]

Original: The SimSiam (Chen & He, 2021) uses a two-layer MLP projector and a two-layer MLP predictor.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: dimension.

Revised: The SimSiam (Chen & He, 2021) uses a two-layer MLP projector and a two-layer MLP predictor.

Original: Following Tian et al.
Revised: Following Tian et al.

Original: (2021), we let the predictor network use a learning rate that is ten times that for the GNN encoder and the projector.
Revised: [Table] we let the predictor network use a learning rate that is ten times that for the GNN encoder and the projector.

Original: The SimSiam backbone does not perform well with the default learning rate of 0 .
Revised: The SimSiam backbone does not perform well with the default learning rate of 0 .

Original: 001 and weight decay of 0 .
Revised: 001 and weight decay of 0 .

Original: Thus, we search for the SimSiam backbone’s learning rate and weight decay values in the following ranges: learning rate = { 1 e − 3 , 5 e − 4 } , weight decay = { 1 e − 4 , 1 e − 5 , 5 e − 5 } .
Revised: Thus [Table] we search for the SimSiam backbone’s learning rate and weight decay values in the following ranges: learning rate = { 1 e − 3 , 5 e − 4 } , weight decay = { 1 e − 4 , 1 e − 5 , 5 e − 5 } .

Original: We then fix the learning rate and weight decay and add the equivariance mechanism.
Revised: We then fix the learning rate and weight decay and add the equivariance mechanism.

Original: For E-GCL, we search for α and ω from a random subset of the following ranges α
Revised: For E-GCL, we search for α and ω from a random subset of the following ranges α

Original: =
Revised: =

Original: 0 . 1 , 0 . 4 , 1 , 2 , 4 } , ω
Revised: 0 . 1 , 0 . 4 , 1 , 2 , 4 } , ω

Original: = { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .
Revised: = { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .

Original: We do not conduct a grid search because the dataset is large.
Revised: We do not conduct a grid search because the dataset is large.

Original: For fine-tuning, the pre-trained model are re-trained 100 epochs on the transferred dataset.
Revised: For fine-tuning, the pre-trained model are re-trained 100 epochs on the transferred dataset.

Original: The fine-tuning learning rate is 1 e − 3 for GINE and 1 e − 4 for GraphTrans.
Revised: The fine-tuning learning rate is 1 e − 3 for GINE and 1 e − 4 for GraphTrans.

Revised: Following (You et al., 2020), we report test performance of the last epoch.

Original: Table 9 shows the configuration details of our used GNNs, in which the GINE is from (Hu et al., 2020) and the GraphTrans is from the Molpcba experiment of (Wu et al., 2021). 
Revised: Table 11 shows the configuration details of our used GNNs, in which the GINE is from (Hu et al., 2020) and the GraphTrans is from the Molpcba experiment of (Wu et al., 2021). [Table] [Table]

Revised: Baseline Hyperparameters.

Original: E. D ATASET S TATISTICS
Revised: We have re-evaluated some baselines to present a consistent experimental comparison with E-GCL. In the re-evaluation, we report the test performance selected by validation set for unsupervised learning; we report the last epoch test performance for transfer learning. When reproducing baselines, we change only the evaluation setting and leave other hyperparameters unchanged as much as possible. Table 12 summarizes hyperparameter details.

Revised: We now describe the process of selecting the baseline hyperparameters.

Revised: For unsupervised learning, we use the validation set to make decisions when a range of hyperparameters are provided in the original paper.

Revised: We use the same set of hyperparameters for all datasets.

Revised: For transfer learning, we fine-tune the same pre-training checkpoint for all downstream datasets for fair comparison.

Revised: E. D ATASET S TATISTICS

Original: Table 10 and Table 11 present the statistics of our used datasets.
Revised: Table 13 and Table 14 present the statistics of our used datasets.

Revised: F C OMPLEXITY A NALYSIS

Revised: The equivariant principle is computationally affordable.

Revised: In this section, we analyze the complexity of

Revised: GCL methods in two parts: 1) neural computation, and 2) data augmentation.

Revised: Symbols.

Revised: Formally, we define the symbols as follows: B ∈ Z + is the batch size; N ∈ Z

Revised: + is the number of nodes in a graph; E ∈ Z + is the number of edges in a graph;

Revised: L ∈ Z + is the number of

Revised: GNN layers; k ∈ (0 , 1) is the ratio of the cutted subgraph for subgraph augmentation; D ∈ Z + is the maximum degree of nodes in graph;

Revised: F ∈ Z + is the dimension of features of nodes and edges.

Revised: For graph interpolation of two graphs, let N , E , and D refer to the values of the bigger graph.

Revised: Complexity of Neural Computation.

Revised: We consider the complexity of GNN encoding and SSL loss function.

Revised: Here, we analyze the complexity when using the GIN architecture.

Revised: The E-GCL uses the

Revised: BarlowTwins framework.

Revised: Let O ( X ) =

Revised: O (2 BL ( NF 2 + EF )) be the complexity of encoding two batches of intra-graph augmentation graphs.

Revised: Shown by Table 15, E-GCL’s complexity is comparable to RGCL.

Revised: E-GCL’s complexity is at most twice of BarlowTwin.

Revised: The additional O ( X ) complexity of GNN encoding comes from encoding the cross-graph augmentation batch, whose size is at most the sum of the two intra-graph augmentations batches.

Revised: Similarly, E-GCL’s complexity of loss function is twice of that of BarlowTwins.

Revised: Complexity of Graph Augmentation.

Revised: Table 16 shows the complexity of some popular graph augmentations (You et al., 2020; Han et al., 2022).

Revised: Note that the complexity of our Graph Interpolation is linear to the graph size times the feature dimension.

Revised: It is lower than G -mixup and scalable to large graphs with thousands of nodes.

Revised: Although the complexity of graph interpolation is higher than

Revised: Drop Node and Drop Edge, it is scalable to large datasets.

Revised: In practice, a PyTorch dataloader with workers can process graph interpolation of 2048 chemical molecules in batches without putting GPU on wait.