Original: Decoupled Self-supervised Learning for Graphs
Revised: Decoupled Self-supervised Learning for Graphs

Original: Abstract
Revised: Abstract

Original: This paper studies the problem of conducting self-supervised learning for node representation learning on graphs.
Revised: This paper studies the problem of conducting self-supervised learning for node representation learning on graphs.

Original: Most existing self-supervised learning methods assume the graph is homophilous, where linked nodes often belong to the same class or have similar features.
Revised: Most existing self-supervised learning methods assume the graph is homophilous, where linked nodes often belong to the same class or have similar features.

Original: However, such assumptions of homophily do not always hold in real-world graphs.
Revised: However, such assumptions of homophily do not always hold in real-world graphs.

Original: We address this problem by developing a decoupled self-supervised learning (DSSL) framework for graph neural networks.
Revised: We address this problem by developing a decoupled self-supervised learning (DSSL) framework for graph neural networks.

Original: DSSL imitates a generative process of nodes and links from latent variable modeling of the semantic structure, which decouples different underlying semantics between different neighborhoods into the self-supervised learning process.
Revised: DSSL imitates a generative process of nodes and links from latent variable modeling of the semantic structure, which decouples different underlying semantics between different neighborhoods into the self-supervised learning process.

Original: Our DSSL framework is agnostic to the encoders and does not need prefabricated augmentations, thus is ﬂexible to different graphs.
Revised: Our DSSL framework is agnostic to the encoders and does not need prefabricated augmentations, thus is ﬂexible to different graphs.

Original: To effectively optimize the framework, we derive the evidence lower bound of the self-supervised objective and develop a scalable training algorithm with variational inference.
Revised: To effectively optimize the framework, we derive the evidence lower bound of the self-supervised objective and develop a scalable training algorithm with variational inference.

Original: We provide a theoretical analysis to justify that DSSL enjoys the better downstream performance.
Revised: We provide a theoretical analysis to justify that DSSL enjoys the better downstream performance.

Original: Extensive experiments on various types of graph benchmarks demonstrate that our proposed framework can achieve better performance compared with competitive baselines.
Revised: Extensive experiments on various types of graph benchmarks demonstrate that our proposed framework can achieve better performance compared with competitive baselines.

Original:  Introduction
Revised:  Introduction

Original: Graph-structured data is ubiquitous in the real world, such as social networks, knowledge graphs, and molecular structures.
Revised: Graph-structured data is ubiquitous in the real world, such as social networks, knowledge graphs, and molecular structures.

Original: In recent years, graph neural networks (GNNs) [16, 27, 52, 7, 60] have been proven to be powerful in node representation learning over graph-structured data.
Revised: In recent years, graph neural networks (GNNs) [16, 27, 52, 7, 60] have been proven to be powerful in node representation learning over graph-structured data.

Original: Typically, GNNs are trained with annotated labeled data in a supervised manner.
Revised: Typically, GNNs are trained with annotated labeled data in a supervised manner.

Original: However, collecting labeled data is expensive and impractical in many applications, especially for those requiring domain knowledge, such as medicine and chemistry [71, 20].
Revised: However, collecting labeled data is expensive and impractical in many applications, especially for those requiring domain knowledge, such as medicine and chemistry [71, 20].

Original: Moreover, supervised learning may suffer from problems of less-transferrable, over-ﬁtting and poor generalization when the task labels are scarce [10, 64].
Revised: Moreover, supervised learning may suffer from problems of less-transferrable, over-ﬁtting and poor generalization when the task labels are scarce [10, 64].

Original: Recently, self-supervised learning (SSL) provides a promising learning paradigm that reduces the dependence on manual labels in the image domain [5, 12, 13, 6].
Revised: Recently, self-supervised learning (SSL) provides a promising learning paradigm that reduces the dependence on manual labels in the image domain [5, 12, 13, 6].

Original: Compared to image data, there are unique challenges in designing self-supervised learning schemes for graph-structured data since nodes in the graph are correlated with each other rather than completely independent, and geometric structures are essential and heavily impact the performance in downstream tasks [27].
Revised: Compared to image data, there are unique challenges in designing self-supervised learning schemes for graph-structured data since nodes in the graph are correlated with each other rather than completely independent, and geometric structures are essential and heavily impact the performance in downstream tasks [27].

Original: A number of recent works [53, 18, 64, 69, 48, 65, 45] have studied graph self-supervised learning and conﬁrm that it can learn transferrable and generalizable node representations without any labels.
Revised: A number of recent works [53, 18, 64, 69, 48, 65, 45] have studied graph self-supervised learning and conﬁrm that it can learn transferrable and generalizable node representations without any labels.

Original: Typically, there are two main self-supervised schemes to capture structure information in graphs [53, 64, 45].
Revised: Typically, there are two main self-supervised schemes to capture structure information in graphs [53, 64, 45].

Original: The ﬁrst scheme is reconstructing the vertex adjacency following traditional network-embedding methods [26, 14, 17, 16, 46], which learns an encoder that imposes the topological closeness of nodes in the graph structure on latent representations.
Revised: The ﬁrst scheme is reconstructing the vertex adjacency following traditional network-embedding methods [26, 14, 17, 16, 46], which learns an encoder that imposes the topological closeness of nodes in the graph structure on latent representations.

Original: The key assumption behind this scheme is that neighboring nodes have similar representations [53, 46].
Revised: The key assumption behind this scheme is that neighboring nodes have similar representations [53, 46].

Original: However, this assumption over-emphasizes proximity [53, 64, 47] and does not always hold true for heterophilic and non-homophilous (mixing) graphs.
Revised: However, this assumption over-emphasizes proximity [53, 64, 47] and does not always hold true for heterophilic and non-homophilous (mixing) graphs.

Original: In comparison, contrastive learning methods [53, 18, 64, 69, 48, 65] construct two graph
Revised: In comparison, contrastive learning methods [53, 18, 64, 69, 48, 65] construct two graph

Original: Information
Revised: Information

Original: (NeurIPS
Revised: (NeurIPS

Original: [Figure]
Revised: [Figure]

Original: views via the stochastic augmentation and then learns representations by contrasting views with information maximization principle.
Revised: views via the stochastic augmentation and then learns representations by contrasting views with information maximization principle.

Original: While these contrastive methods can capture structure information without directly emphasizing proximity, their performance relies on topology augmentation [70, 50].
Revised: While these contrastive methods can capture structure information without directly emphasizing proximity, their performance relies on topology augmentation [70, 50].

Original: Importantly, conducting augmentation for non-homophilous graphs is relatively difﬁcult since linked nodes may be dissimilar, and nodes with high similarities might be farther away from each other.
Revised: Importantly, conducting augmentation for non-homophilous graphs is relatively difﬁcult since linked nodes may be dissimilar, and nodes with high similarities might be farther away from each other.

Original: Hence, the above problems pose an important and challenging research question: How to design an effective self-supervised scheme for node representation learning in non-homophilous graphs?
Revised: Hence, the above problems pose an important and challenging research question: How to design an effective self-supervised scheme for node representation learning in non-homophilous graphs?

Original: We approach this question by taking advantage of neighborhood strategies of nodes for the selfsupervised learning on non-homophilous graphs.
Revised: We approach this question by taking advantage of neighborhood strategies of nodes for the selfsupervised learning on non-homophilous graphs.

Original: Our key motivation is that nodes with similar neighborhood patterns should have similar representations.
Revised: Our key motivation is that nodes with similar neighborhood patterns should have similar representations.

Original: In other words, we expect that the neighborhood distributions can be exploited to distinguish node representations.
Revised: In other words, we expect that the neighborhood distributions can be exploited to distinguish node representations.

Original: Our assumption is more general than the standard homophily assumption as shown in Figure 1.
Revised: Our assumption is more general than the standard homophily assumption as shown in Figure 1.

Original: For instance, while the gender prediction in common dating networks lacks homophily [1], neighborhood distribution is very informative to the node gender labels, i.e., nodes with similar neighborhoods are likely to be similar.
Revised: For instance, while the gender prediction in common dating networks lacks homophily [1], neighborhood distribution is very informative to the node gender labels, i.e., nodes with similar neighborhoods are likely to be similar.

Original: While the motivation is straightforward, we are faced with two main challenges.
Revised: While the motivation is straightforward, we are faced with two main challenges.

Original: The ﬁrst challenge is capturing the local neighborhood distribution in a self-supervised learning manner.
Revised: The ﬁrst challenge is capturing the local neighborhood distribution in a self-supervised learning manner.

Original: The neighborhood distributions typically follow heterogeneous and diverse patterns.
Revised: The neighborhood distributions typically follow heterogeneous and diverse patterns.

Original: For instance, a node usually connects with others due to the complex interaction of different latent factors and therefore possesses distribution consisting of local mixing patterns wherein certain parts of the neighborhood are homophilous while others are non-homophilous.
Revised: For instance, a node usually connects with others due to the complex interaction of different latent factors and therefore possesses distribution consisting of local mixing patterns wherein certain parts of the neighborhood are homophilous while others are non-homophilous.

Original: The lack of supervision obstructs us from modeling the distribution of neighborhoods.
Revised: The lack of supervision obstructs us from modeling the distribution of neighborhoods.

Original: The second challenge is capturing the long-range semantic dependencies.
Revised: The second challenge is capturing the long-range semantic dependencies.

Original: As shown in Figure 1, in non-homophilous graphs, nodes with high semantic and local structural similarities might be farther away from each other.
Revised: As shown in Figure 1, in non-homophilous graphs, nodes with high semantic and local structural similarities might be farther away from each other.

Original: For this reason, global semantic information is the objective that we would incorporate for self-supervised learning.
Revised: For this reason, global semantic information is the objective that we would incorporate for self-supervised learning.

Original: This paper presents a new self-supervised framework, decoupled self-supervised learning (DSSL), to achieve a good balance between these two challenges.
Revised: This paper presents a new self-supervised framework, decoupled self-supervised learning (DSSL), to achieve a good balance between these two challenges.

Original: At the core of DSSL is the latent variables, which empower the model with the ﬂexibility to decouple the heterogeneous and diverse patterns in local neighborhood distributions and capture the global semantic dependencies in a coherent and synergistic framework.
Revised: At the core of DSSL is the latent variables, which empower the model with the ﬂexibility to decouple the heterogeneous and diverse patterns in local neighborhood distributions and capture the global semantic dependencies in a coherent and synergistic framework.

Original: Our contributions can be summarized as follows: (1) We propose a DSSL for performing self-supervised learning on non-homophilous graphs, which can leverage both useful local structure and global semantic information.
Revised: Our contributions can be summarized as follows: (1) We propose a DSSL for performing self-supervised learning on non-homophilous graphs, which can leverage both useful local structure and global semantic information.

Original: (2) We develop an efﬁcient training algorithm based on variational inference to simultaneously infer the latent variables and learn encoder parameters.
Revised: (2) We develop an efﬁcient training algorithm based on variational inference to simultaneously infer the latent variables and learn encoder parameters.

Original: (3) We analyze the properties of DSSL and theoretically show that the learned representations can achieve better downstream performance.
Revised: (3) We analyze the properties of DSSL and theoretically show that the learned representations can achieve better downstream performance.

Original: (4) We conduct experiments on real-world non-homophilous graphs, and the results demonstrate the effectiveness of our self-supervised learning framework.
Revised: (4) We conduct experiments on real-world non-homophilous graphs, and the results demonstrate the effectiveness of our self-supervised learning framework.

Original:  Related Work
Revised:  Related Work

Original: Non-homophilous Graphs.
Revised: Non-homophilous Graphs.

Original: Non-homophilous is known in many settings such as online transaction networks [36], dating networks [1] and molecular networks [68].
Revised: Non-homophilous is known in many settings such as online transaction networks [36], dating networks [1] and molecular networks [68].

Original: Recently, various GNNs [37, 68, 59, 29, 69, 8, 44, 62] have been proposed to deal with non-homophilous graphs with different methods such as potential neighbor discovery [37, 23, 22], adaptive message propagation [8, 59], exploring high-frequency signals [2] and higher-order message passing [68, 5].
Revised: Recently, various GNNs [37, 68, 59, 29, 69, 8, 44, 62] have been proposed to deal with non-homophilous graphs with different methods such as potential neighbor discovery [37, 23, 22], adaptive message propagation [8, 59], exploring high-frequency signals [2] and higher-order message passing [68, 5].

Original: Despite their success, they typically consider the semi-supervised setting and are trained with task-speciﬁc labeled data, while in
Revised: Despite their success, they typically consider the semi-supervised setting and are trained with task-speciﬁc labeled data, while in

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: practice, labels are often limited, expensive, and even inaccessible.
Revised: practice, labels are often limited, expensive, and even inaccessible.

Original: contrast, in this the problem of self-supervised learning: learning node representations without relying on labels.
Revised: contrast, in this the problem of self-supervised learning: learning node representations without relying on labels.

Original: Self-supervised Learning on Graphs.
Revised: Self-supervised Learning on Graphs.

Original: Self-supervised learning holds great promise for improving representations when labeled data are scarce [5, 12, 13, 19].
Revised: Self-supervised learning holds great promise for improving representations when labeled data are scarce [5, 12, 13, 19].

Original: Earlier combinations of GNNs and self-supervised learning involve GraphSAGE [16], VGAE [26], and Graphite [15], which typically follow the traditional network-embedding methods [38, 46, 14] and adopt the link reconstruction or random walk principle.
Revised: Earlier combinations of GNNs and self-supervised learning involve GraphSAGE [16], VGAE [26], and Graphite [15], which typically follow the traditional network-embedding methods [38, 46, 14] and adopt the link reconstruction or random walk principle.

Original: Since these methods over-emphasize node proximity at the cost of structural information [53, 45, 64], various graph contrastive learning methods have been proposed [53, 18, 64, 69, 48, 35], which aim to learn representations by contrasting representation under differently augmented views and have achieved promising performance.
Revised: Since these methods over-emphasize node proximity at the cost of structural information [53, 45, 64], various graph contrastive learning methods have been proposed [53, 18, 64, 69, 48, 35], which aim to learn representations by contrasting representation under differently augmented views and have achieved promising performance.

Original: However, they heavily rely on complex data- or task-speciﬁc augmentations, which are prohibitively difﬁcult for non-homophilous graphs.
Revised: However, they heavily rely on complex data- or task-speciﬁc augmentations, which are prohibitively difﬁcult for non-homophilous graphs.

Original: Our work differs from the above methods and aims to answer the question of how to design an effective self-supervised learning scheme for non-homophilous graphs.
Revised: Our work differs from the above methods and aims to answer the question of how to design an effective self-supervised learning scheme for non-homophilous graphs.

Original: Disentangled graph learning.
Revised: Disentangled graph learning.

Original: Our work is also related to but different from existing disentangled graph learning that aims to decouple latent factors in the graph.
Revised: Our work is also related to but different from existing disentangled graph learning that aims to decouple latent factors in the graph.

Original: vGraph [43] considers a mixture process to deﬁne the conditional probability on each edge.
Revised: vGraph [43] considers a mixture process to deﬁne the conditional probability on each edge.

Original: vGraph is a classical shallow network embedding algorithm.
Revised: vGraph is a classical shallow network embedding algorithm.

Original: However, we instead tackle the problem of self-supervised learning with
Revised: However, we instead tackle the problem of self-supervised learning with

Original: GNNs.
Revised: GNNs.

Original: There are a couple of works that explore the disentangled factors in the node-level [33, 32], edge-level [67] and graph-level [63].
Revised: There are a couple of works that explore the disentangled factors in the node-level [33, 32], edge-level [67] and graph-level [63].

Original: Whereas these methods require task-speciﬁc labels that can be extremely scarce.
Revised: Whereas these methods require task-speciﬁc labels that can be extremely scarce.

Original: By contrast, we address the problem of learning node representations on nonhomophilous graphs without labels.
Revised: By contrast, we address the problem of learning node representations on nonhomophilous graphs without labels.

Original: Disentangled contrastive learning [28] learns disentangled graph representations without labeled graphs.
Revised: Disentangled contrastive learning [28] learns disentangled graph representations without labeled graphs.

Original: [61] proposes the self-supervised graph-level representation learning with disentangled local and global structure [61].
Revised: [61] proposes the self-supervised graph-level representation learning with disentangled local and global structure [61].

Original: Nevertheless, their goal is to conduct graph-level classiﬁcation tasks, but we tackle a node-level representation problem.
Revised: Nevertheless, their goal is to conduct graph-level classiﬁcation tasks, but we tackle a node-level representation problem.

Original: [66, 9, 55] consider adding a clustering layer or a prototype clustering component to capture the global information.
Revised: [66, 9, 55] consider adding a clustering layer or a prototype clustering component to capture the global information.

Original: However, they are still based on contrastive learning [66, 9] or supervised learning [55].
Revised: However, they are still based on contrastive learning [66, 9] or supervised learning [55].

Original: By contrast, our framework is an unsupervised generative model and does not rely on graph augmentations and downstream labels.
Revised: By contrast, our framework is an unsupervised generative model and does not rely on graph augmentations and downstream labels.

Original: Moreover, we focus on non-homophilous graphs where connected nodes may not be similar to each other by decoupling the local diverse neighborhood context.
Revised: Moreover, we focus on non-homophilous graphs where connected nodes may not be similar to each other by decoupling the local diverse neighborhood context.

Original:  Decoupled Self-supervised Learning
Revised:  Decoupled Self-supervised Learning

Original: In this section, we describe our problem setting with respect to self-supervised learning and demonstrate our approach.
Revised: In this section, we describe our problem setting with respect to self-supervised learning and demonstrate our approach.

Original: An illustration of generative and inference processes is depicted in Figure 2 (a).
Revised: An illustration of generative and inference processes is depicted in Figure 2 (a).

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: We consider a graph G =
Revised: We consider a graph G =

Original: ( V , E ) , where V is a set of |V| =
Revised: ( V , E ) , where V is a set of |V| =

Original: N nodes and E ✓ V ⇥ V is a set of |E| edges between nodes.
Revised: N nodes and E ✓ V ⇥ V is a set of |E| edges between nodes.

Original: A 2 { 0 , 1 } N ⇥ N is the adjacency matrix of G .
Revised: A 2 { 0 , 1 } N ⇥ N is the adjacency matrix of G .

Original: The ( i, j ) -th
Revised: The ( i, j ) -th

Original: element A ij =if there exists an edge ( v i , v j ) between node v i and v j , otherwise A ij = 0 .
Revised: element A ij =if there exists an edge ( v i , v j ) between node v i and v j , otherwise A ij = 0 .

Original: The matrix X 2 R N ⇥ D describes node features.
Revised: The matrix X 2 R N ⇥ D describes node features.

Original: The i -th row of X , i.e., x i , is the feature of node v i .
Revised: The i -th row of X , i.e., x i , is the feature of node v i .

Original: Given the graph
Revised: Given the graph

Original: G =
Revised: G =

Original: ( X , A ) , the objective of self-supervised node representation learning is to learn an encoder function f ✓ ( X , A ) :
Revised: ( X , A ) , the objective of self-supervised node representation learning is to learn an encoder function f ✓ ( X , A ) :

Original: R N ⇥ N ⇥ R N ⇥ D !
Revised: R N ⇥ N ⇥ R N ⇥ D !

Original: R N ⇥ D 0 where ✓ denotes its parameters, such that the representation of node v i : f ✓ ( X , A )
Revised: R N ⇥ D 0 where ✓ denotes its parameters, such that the representation of node v i : f ✓ ( X , A )

Original: [ i ] , can be used for downstream tasks such as node classiﬁcation.
Revised: [ i ] , can be used for downstream tasks such as node classiﬁcation.

Original: 3.2 The Probabilistic Framework
Revised: 3.2 The Probabilistic Framework

Original: In this section, we introduce our framework, decoupled self-supervised learning, which can learn meaningful node representations on non-homophilous graphs by capturing the intrinsic graph structure.
Revised: In this section, we introduce our framework, decoupled self-supervised learning, which can learn meaningful node representations on non-homophilous graphs by capturing the intrinsic graph structure.

Original: The core idea of DSSL is to model the distributions of node neighbors via a mixture of generative processes in the representation space.
Revised: The core idea of DSSL is to model the distributions of node neighbors via a mixture of generative processes in the representation space.

Original: Speciﬁcally, we model the generation of neighbors by assuming each node has latent heterogeneous factors which are utilized to make connections to its different neighbors.
Revised: Speciﬁcally, we model the generation of neighbors by assuming each node has latent heterogeneous factors which are utilized to make connections to its different neighbors.

Original: Intuitively, the factor denotes various reasons behind why two nodes are connected in non-homophilous graphs.
Revised: Intuitively, the factor denotes various reasons behind why two nodes are connected in non-homophilous graphs.

Original: For instance, two nodes in a school network will be connected depending on some factors such as colleagues, friends, or classmates; in protein networks, even if they do not have similar features, different amino acid types are likely to be connected due to various interactions.
Revised: For instance, two nodes in a school network will be connected depending on some factors such as colleagues, friends, or classmates; in protein networks, even if they do not have similar features, different amino acid types are likely to be connected due to various interactions.

Original: Formally, let f ✓ ( X , A )
Revised: Formally, let f ✓ ( X , A )

Original: [ i ] = v i and f ✓ ( X , A )
Revised: [ i ] = v i and f ✓ ( X , A )

Original: [ j ] =
Revised: [ j ] =

Original: z j be the representations of nodes v i and v j , respectively.
Revised: z j be the representations of nodes v i and v j , respectively.

Original: Here we utilize different notations, i.e., v and z to distinguish between central and neighbor nodes since each node plays two roles: the node itself and a speciﬁc neighbor of other nodes.
Revised: Here we utilize different notations, i.e., v and z to distinguish between central and neighbor nodes since each node plays two roles: the node itself and a speciﬁc neighbor of other nodes.

Original: Our goal is to ﬁnd the encoder parameter ✓ which maximizes the likelihood of distribution p ( z j | v i ; ✓ ) on central node and its neighbor.
Revised: Our goal is to ﬁnd the encoder parameter ✓ which maximizes the likelihood of distribution p ( z j | v i ; ✓ ) on central node and its neighbor.

Original: To model the unobserved factors, we associate every node v i with a discrete latent variable k to indicate to which factor v i has.
Revised: To model the unobserved factors, we associate every node v i with a discrete latent variable k to indicate to which factor v i has.

Original: Assume that there are K factors in total, the log-likelihood of node neighbors v i can be written by marginalizing out the latent variables:
Revised: Assume that there are K factors in total, the log-likelihood of node neighbors v i can be written by marginalizing out the latent variables:

Original: [Equation]
Revised: [Equation]

Original: where N ( i ) is the set of out-neighbors of v i , the distribution p ✓ ( k | v i ) indicates the assignment of latent semantics over central node representation v i , and p ✓ ( z j | v i , k ) is the probability that node v i and its neighbor v j are connected under factor k .
Revised: where N ( i ) is the set of out-neighbors of v i , the distribution p ✓ ( k | v i ) indicates the assignment of latent semantics over central node representation v i , and p ✓ ( z j | v i , k ) is the probability that node v i and its neighbor v j are connected under factor k .

Original: Unlike previous works [46, 14, 26], which directly encourage nearby nodes to have similar representations, we provide an alternative way to model node neighbors and seek to decouple their latent relationship without any prior on neighbor partitions.
Revised: Unlike previous works [46, 14, 26], which directly encourage nearby nodes to have similar representations, we provide an alternative way to model node neighbors and seek to decouple their latent relationship without any prior on neighbor partitions.

Original: The similar high-level ideas of modeling latent variables between two nodes has also been explored in local augmentation in graphs (LA-GCN) [31] and vGraph [43].
Revised: The similar high-level ideas of modeling latent variables between two nodes has also been explored in local augmentation in graphs (LA-GCN) [31] and vGraph [43].

Original: Different from them, we consider modeling discrete relations in the learned representation space for self-supervised learning on non-homophilous graphs.
Revised: Different from them, we consider modeling discrete relations in the learned representation space for self-supervised learning on non-homophilous graphs.

Original: In Eq.
Revised: In Eq.

Original: (1), probabilities p ✓ ( k | v i ) and p ✓ ( z j | v i , k ) are not speciﬁed, and involve discrete latent variables.
Revised: (1), probabilities p ✓ ( k | v i ) and p ✓ ( z j | v i , k ) are not speciﬁed, and involve discrete latent variables.

Original: To make it solvable, we introduce the following generative process.
Revised: To make it solvable, we introduce the following generative process.

Original: Let µ k and ⌃ k be the mean and variance of the latent mixture component k , and ⇡ k be its corresponding mixture probability.
Revised: Let µ k and ⌃ k be the mean and variance of the latent mixture component k , and ⇡ k be its corresponding mixture probability.

Original: The generation of a node and its neighbor shown in Figure 2 (a) typically involves three steps: (1) draw a latent variable k from a categorical distribution p ( k ) on all mixture components, where p ( k ) is usually deﬁned as uniform distribution p ( k )
Revised: The generation of a node and its neighbor shown in Figure 2 (a) typically involves three steps: (1) draw a latent variable k from a categorical distribution p ( k ) on all mixture components, where p ( k ) is usually deﬁned as uniform distribution p ( k )

Original: = 1 K for unknown graphs and better generalization, (2) draw the central node representation v i from the Gaussian distribution p ✓ ( v i | k )
Revised: = 1 K for unknown graphs and better generalization, (2) draw the central node representation v i from the Gaussian distribution p ✓ ( v i | k )

Original: =
Revised: =

Original: N ( v i ; µ k , ⌃ k ), and (3) draw the neighbor representation z j from Gaussian distribution p ✓ ( z j | v i , k ) =
Revised: N ( v i ; µ k , ⌃ k ), and (3) draw the neighbor representation z j from Gaussian distribution p ✓ ( z j | v i , k ) =

Original: N ( z j ; µ z j , ⌃ j ) where the mean depends both on central representation and latent variable: µ z j = v i +  g ✓ ( k ) .
Revised: N ( z j ; µ z j , ⌃ j ) where the mean depends both on central representation and latent variable: µ z j = v i +  g ✓ ( k ) .

Original: Here the projector g ✓ ( · ) denotes another network that embeds latent variable k to the representation space, and  is the parameter that controls the strength of interpolation.
Revised: Here the projector g ✓ ( · ) denotes another network that embeds latent variable k to the representation space, and  is the parameter that controls the strength of interpolation.

Original: In practice, to reduce complexity, we consider using isotropic Gaussian, i.e., 8 k : ⌃ k
Revised: In practice, to reduce complexity, we consider using isotropic Gaussian, i.e., 8 k : ⌃ k

Original: = I  21 and 8 j : ⌃ j = I
Revised: = I  21 and 8 j : ⌃ j = I

Original:  22 where I is the identity matrix,  1 and  2 are hyperparameters.
Revised:  22 where I is the identity matrix,  1 and  2 are hyperparameters.

Original: In alignment with this generative process, the joint distribution of the edge and latent variable can be written as:
Revised: In alignment with this generative process, the joint distribution of the edge and latent variable can be written as:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, p ✓ ( v i | k ) can be viewed as the probability for node representation under the k -th mixture component.
Revised: Intuitively, p ✓ ( v i | k ) can be viewed as the probability for node representation under the k -th mixture component.

Original: Regarding p ✓ ( z j | v i , k ) =
Revised: Regarding p ✓ ( z j | v i , k ) =

Original: N ( z j ; µ z j , ⌃ j ) , this formulation is inspired by knowledge graph embedding [3], where two entities should be closed to each other under a certain relation operation.
Revised: N ( z j ; µ z j , ⌃ j ) , this formulation is inspired by knowledge graph embedding [3], where two entities should be closed to each other under a certain relation operation.

Original: However, unlike knowledge graph embedding, the relational context is a latent variable and not observed in our setting.
Revised: However, unlike knowledge graph embedding, the relational context is a latent variable and not observed in our setting.

Original: Intuitively, instead of enforcing the exact representation alignment of two linked nodes [26, 14, 17, 16, 46], our design can relax this strong homophily assumption and account for the semantic shift of representations between the central node and its neighbors.
Revised: Intuitively, instead of enforcing the exact representation alignment of two linked nodes [26, 14, 17, 16, 46], our design can relax this strong homophily assumption and account for the semantic shift of representations between the central node and its neighbors.

Original: Now, the posterior probability p ✓ ( k | v i ) in Eq.
Revised: Now, the posterior probability p ✓ ( k | v i ) in Eq.

Original: (1) can be derived by using Bayesian rules as follows:
Revised: (1) can be derived by using Bayesian rules as follows:

Original: [Equation]
Revised: [Equation]

Original: where µ = { µ k } Kk =1 can be treated as a set of trainable prototype representations, which are the additional distribution parameters.
Revised: where µ = { µ k } Kk =1 can be treated as a set of trainable prototype representations, which are the additional distribution parameters.

Original: This posterior probability represents the soft semantic assignment of the learned representation to the prototypes, which makes the node with similar semantic properties be close to its prototype and encode both the inter- and intra-cluster variation.
Revised: This posterior probability represents the soft semantic assignment of the learned representation to the prototypes, which makes the node with similar semantic properties be close to its prototype and encode both the inter- and intra-cluster variation.

Original: Substituting p ✓ ( k | v i ) and p ✓ ( z j | v i , k ) in Eq.
Revised: Substituting p ✓ ( k | v i ) and p ✓ ( z j | v i , k ) in Eq.

Original: (1) with the speciﬁed probabilities, we get ﬁnal objective L DSSL ( ✓ , µ ) .
Revised: (1) with the speciﬁed probabilities, we get ﬁnal objective L DSSL ( ✓ , µ ) .

Original: 3.3 Evidence Lower Bound
Revised: 3.3 Evidence Lower Bound

Original: Given the framework above, we are interested in: (i) learning the model parameters ✓ and µ by maximizing the log-likelihood
Revised: Given the framework above, we are interested in: (i) learning the model parameters ✓ and µ by maximizing the log-likelihood

Original: L DSSL ( ✓ , µ ) , and (ii) inferring the posterior of latent variable k for each observed links.
Revised: L DSSL ( ✓ , µ ) , and (ii) inferring the posterior of latent variable k for each observed links.

Original: However, it is computationally intractable to directly solve these two problems due to the latent variables.
Revised: However, it is computationally intractable to directly solve these two problems due to the latent variables.

Original: To solve this, we resort to amortized variational inference methods [25], and maximize the evidence lower-bound (ELBO) of Eq.
Revised: To solve this, we resort to amortized variational inference methods [25], and maximize the evidence lower-bound (ELBO) of Eq.

Original: (1), i.e.,
Revised: (1), i.e.,

Original: L
Revised: L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: ( ✓ ,  ,
Revised: ( ✓ ,  ,

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: where q  ( k | v i , z j ) is the introduced variational distribution parameterized by  and H ( · ) is the entropy operator.
Revised: where q  ( k | v i , z j ) is the introduced variational distribution parameterized by  and H ( · ) is the entropy operator.

Original: z j is included in this variational posterior, so the inference is also conditioned on the neighborhood information.
Revised: z j is included in this variational posterior, so the inference is also conditioned on the neighborhood information.

Original: We derive this ELBO in Appendix A.1.
Revised: We derive this ELBO in Appendix A.1.

Original: Maximizing this ELBO w.r.t.
Revised: Maximizing this ELBO w.r.t.

Original: { ✓ ,  , µ } is equivalent to (i) maximizing L DSSL ( ✓ , µ ) and to (ii) make variational q  ( k | v i , z j ) be close to true posterior.
Revised: { ✓ ,  , µ } is equivalent to (i) maximizing L DSSL ( ✓ , µ ) and to (ii) make variational q  ( k | v i , z j ) be close to true posterior.

Original: Plugging the parameterized probabilities into this ELBO, we obtain the following loss to minimize.
Revised: Plugging the parameterized probabilities into this ELBO, we obtain the following loss to minimize.

Original: See Appendix A.2 for corresponding derivations.
Revised: See Appendix A.2 for corresponding derivations.

Original: [Equation]
Revised: [Equation]

Original: where q  ( k | v i ) = 1 / | N ( i ) |
Revised: where q  ( k | v i ) = 1 / | N ( i ) |

Original: P j 2 N ( i ) q
Revised: P j 2 N ( i ) q

Original:  ( k | v i , z j ) is the posterior probability of semantic assignment for central node v i , by aggregating all its neighbors.
Revised:  ( k | v i , z j ) is the posterior probability of semantic assignment for central node v i , by aggregating all its neighbors.

Original: Thus, the ﬁrst term (denoted as L local ) in the loss encourages the model to reconstruct the local neighbors while considering different semantic shifts captured by latent variable k
Revised: Thus, the ﬁrst term (denoted as L local ) in the loss encourages the model to reconstruct the local neighbors while considering different semantic shifts captured by latent variable k

Original: (see Figure 2 (b)).
Revised: (see Figure 2 (b)).

Original: The second term (denoted as L global ) encourages the model to perform clustering with learned representation where possible, i.e., seeking to push the representation v i to its closest prototype cluster (see Figure 2 (c)).
Revised: The second term (denoted as L global ) encourages the model to perform clustering with learned representation where possible, i.e., seeking to push the representation v i to its closest prototype cluster (see Figure 2 (c)).

Original: The ﬁnal entropy term makes the model choose to have high entropy over q  ( k | v i , z j ) such that all of the K -channel losses must be low.
Revised: The ﬁnal entropy term makes the model choose to have high entropy over q  ( k | v i , z j ) such that all of the K -channel losses must be low.

Original: Overall, this loss derived from ELBO can capture global semantic similarities over neighborhoods and learn to decouple different latent patterns in the local neighbors.
Revised: Overall, this loss derived from ELBO can capture global semantic similarities over neighborhoods and learn to decouple different latent patterns in the local neighbors.

Original: Regarding the variaitonal distribution q  ( k | v i , z j ) , we model it as categorical distribution since k is a discrete multinomial variable.
Revised: Regarding the variaitonal distribution q  ( k | v i , z j ) , we model it as categorical distribution since k is a discrete multinomial variable.

Original: Speciﬁcally, the representations v i and z j are encoded to a combined representation and then q  ( k | v i , z j ) is determined by an output softmax inference head as follows:
Revised: Speciﬁcally, the representations v i and z j are encoded to a combined representation and then q  ( k | v i , z j ) is determined by an output softmax inference head as follows:

Original: [Equation]
Revised: [Equation]

Original: where h  denotes the inference network parameterized by  and [ · , · ] can be the element-wise product or concatenation operation.
Revised: where h  denotes the inference network parameterized by  and [ · , · ] can be the element-wise product or concatenation operation.

Original: h  ([ v i ; z j ])
Revised: h  ([ v i ; z j ])

Original: [ k ] indicates the k th element, i.e., the logit corresponding the latent context k .
Revised: [ k ] indicates the k th element, i.e., the logit corresponding the latent context k .

Original: Instead of introducing variational parameters individually, we consider the amortization inference [25, 58], which ﬁts a shared network to calculate each local parameter.
Revised: Instead of introducing variational parameters individually, we consider the amortization inference [25, 58], which ﬁts a shared network to calculate each local parameter.

Original: For the expectation terms in Eq.
Revised: For the expectation terms in Eq.

Original: (5), back-propagation through the discrete variable k is not di-
Revised: (5), back-propagation through the discrete variable k is not di-

Original: rectly feasible.
Revised: rectly feasible.

Original: We alleviate this by adopting the Straight-Through Gumbel-Softmax estimator [21], which provides a continuous differentiable approximation for drawing samples from a categorical distribution.
Revised: We alleviate this by adopting the Straight-Through Gumbel-Softmax estimator [21], which provides a continuous differentiable approximation for drawing samples from a categorical distribution.

Original: Speciﬁcally, for each sample, a latent cluster vector c 2 (0 , 1) K is drawn from:
Revised: Speciﬁcally, for each sample, a latent cluster vector c 2 (0 , 1) K is drawn from:

Original: [Equation]
Revised: [Equation]

Original: where ✏ [ k ] is i.i.d drawn from the Gumbel(0 , 1) distribution and  is a temperature.
Revised: where ✏ [ k ] is i.i.d drawn from the Gumbel(0 , 1) distribution and  is a temperature.

Original: With this reparameterization trick, we can obtain the surrogate E q  ( k | v i , z j )
Revised: With this reparameterization trick, we can obtain the surrogate E q  ( k | v i , z j )

Original: [ k v i +  g ✓ ( k ) 
Revised: [ k v i +  g ✓ ( k )

Original: z j k 22 ] ' E ✏ [ k v i +  g ✓ ( c )  z
Revised: z j k 22 ] ' E ✏ [ k v i +  g ✓ ( c )  z

Original: j k 22 ]
Revised: j k 22 ]

Original: and the gradients are estimated with Monte Carlo.
Revised: and the gradients are estimated with Monte Carlo.

Original: The expectation term over q  ( k | v i ) can be similarly estimated.
Revised: The expectation term over q  ( k | v i ) can be similarly estimated.

Original: Then, { ✓ ,  , µ } in Eq.
Revised: Then, { ✓ ,  , µ } in Eq.

Original: (5) can be efﬁciently solved by gradient descent.
Revised: (5) can be efﬁciently solved by gradient descent.

Original: 3.4 Algorithm Optimization
Revised: 3.4 Algorithm Optimization

Original: The overall optimization involves simultaneously training (1) the encoder f ✓ , (2) the projector g ✓ , (3) the inference predictor h  and (4) the prototypes µ .
Revised: The overall optimization involves simultaneously training (1) the encoder f ✓ , (2) the projector g ✓ , (3) the inference predictor h  and (4) the prototypes µ .

Original: The most canonical way to update the parameters is stochastic gradient descent.
Revised: The most canonical way to update the parameters is stochastic gradient descent.

Original: However, we observe that stochastically updating all parameters suffers from two problems: (1) The objective admits trivial solutions, e.g., outputting the same representation for all nodes in the optimization process.
Revised: However, we observe that stochastically updating all parameters suffers from two problems: (1) The objective admits trivial solutions, e.g., outputting the same representation for all nodes in the optimization process.

Original: (2) Updating prototypes without any constraints will lead to a degenerate solution, i.e., all nodes are assigned to a single cluster.
Revised: (2) Updating prototypes without any constraints will lead to a degenerate solution, i.e., all nodes are assigned to a single cluster.

Original: To address the issue of trivial solutions, inspired by the recent works [19, 13], we consider an asymmetric encoder architecture that includes online and target encoders.
Revised: To address the issue of trivial solutions, inspired by the recent works [19, 13], we consider an asymmetric encoder architecture that includes online and target encoders.

Original: Speciﬁcally, for each node pair ( v i , v j ) , the online encoder f ✓ produces the representation of the central node v i ; while the target encoder f ⇠ is used to produce the representation of its neighbor z j .
Revised: Speciﬁcally, for each node pair ( v i , v j ) , the online encoder f ✓ produces the representation of the central node v i ; while the target encoder f ⇠ is used to produce the representation of its neighbor z j .

Original: Importantly, the gradient of loss is only used to update the online encoder f ✓ while being blocked in the target encoder.
Revised: Importantly, the gradient of loss is only used to update the online encoder f ✓ while being blocked in the target encoder.

Original: The weights of the target encoder ⇠ are updated via the exponential moving average (EMA) of the online encoder ✓ :
Revised: The weights of the target encoder ⇠ are updated via the exponential moving average (EMA) of the online encoder ✓ :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where  ( · ) indicates a stochastic optimizer and ⌧ 2 [0 , 1] is the target decay rate.
Revised: ) where  ( · ) indicates a stochastic optimizer and ⌧ 2 [0 , 1] is the target decay rate.

Original: This update introduces an asymmetry between two encoders that prevents collapse to trivial solutions [13, 48].
Revised: This update introduces an asymmetry between two encoders that prevents collapse to trivial solutions [13, 48].

Original: To alleviate the second issue, besides the stochastic update, we also apply a global update for the prototype vectors µ = { µ i } K i =1 at the end of each training epoch to avoid a degenerate solution:
Revised: To alleviate the second issue, besides the stochastic update, we also apply a global update for the prototype vectors µ = { µ i } K i =1 at the end of each training epoch to avoid a degenerate solution:

Original: [Equation]
Revised: [Equation]

Original: i k 22 , where ⇡ i ( k ) = 1 / | N ( i ) |
Revised: i k 22 , where ⇡ i ( k ) = 1 / | N ( i ) |

Original: [Equation]
Revised: [Equation]

Original: The derivation is provided in Appendix A.3.
Revised: The derivation is provided in Appendix A.3.

Original: Intuitively, ⇡ i ( k ) reﬂects the degree of relevance of node v i to the k th prototype.
Revised: Intuitively, ⇡ i ( k ) reﬂects the degree of relevance of node v i to the k th prototype.

Original: Instead of only updating the prototypes in a mini-batch, we also aggregate all the representations as the prototype based on the soft assignment probability at the end of the epoch.
Revised: Instead of only updating the prototypes in a mini-batch, we also aggregate all the representations as the prototype based on the soft assignment probability at the end of the epoch.

Original: After the training is ﬁnished, we only keep the online encoder f ✓ for the downstream task.
Revised: After the training is ﬁnished, we only keep the online encoder f ✓ for the downstream task.

Original: Our full algorithm and network are provided in Appendix B.
Revised: Our full algorithm and network are provided in Appendix B.

Original: The details about time complexity of DSSL is given in Appendix C.1, which scales linearly in the size of edges.
Revised: The details about time complexity of DSSL is given in Appendix C.1, which scales linearly in the size of edges.

Original: 3.5 Theoretical Analysis
Revised: 3.5 Theoretical Analysis

Original: In this section, we provide an analysis of the proposed framework.
Revised: In this section, we provide an analysis of the proposed framework.

Original: We ﬁrst present the connection between the proposed objective and the mutual information maximization, then show that the learned representations by our objective provably enjoy a good downstream performance.
Revised: We ﬁrst present the connection between the proposed objective and the mutual information maximization, then show that the learned representations by our objective provably enjoy a good downstream performance.

Original: Due to the space limitation, all proofs of theorems and corollaries are provided in Appendix D.
Revised: Due to the space limitation, all proofs of theorems and corollaries are provided in Appendix D.

Original: We denote the random variable of the input graph as G and the downstream label as y .
Revised: We denote the random variable of the input graph as G and the downstream label as y .

Original: For clarity, we omit subscript i in what follows.
Revised: For clarity, we omit subscript i in what follows.

Original: From an information-theoretic learning perspective, a desirable way is to maximize the mutual information I ( v , y ) between the representation v and downstream label y .
Revised: From an information-theoretic learning perspective, a desirable way is to maximize the mutual information I ( v , y ) between the representation v and downstream label y .

Original: However, due to the lack of the downstream label, self-supervised learning resorts to maximizing
Revised: However, due to the lack of the downstream label, self-supervised learning resorts to maximizing

Original: I ( v , s ) where s is different designed self-supervised signal [51, 4, 56, 65, 11].
Revised: I ( v , s ) where s is different designed self-supervised signal [51, 4, 56, 65, 11].

Original: In our method, we have two self-supervised signals: the global semantic cluster information inferred by k and the local structural roles captured by the representations of the neighbors z
Revised: In our method, we have two self-supervised signals: the global semantic cluster information inferred by k and the local structural roles captured by the representations of the neighbors z

Original: = { z i | v i 2 N ( v ) } of node v .
Revised: = { z i | v i 2 N ( v ) } of node v .

Original: Then, we can interpret our objective in Eq.
Revised: Then, we can interpret our objective in Eq.

Original: (5) from the information maximization perspective:
Revised: (5) from the information maximization perspective:

Original: Theorem 1.
Revised: Theorem 1.

Original: Optimizing local and global terms in Eq.
Revised: Optimizing local and global terms in Eq.

Original: (5) is equivalent to maximizing the mutual information between the representation v and global signal k and maximizing the conditional mutual information between v and the local signal z , conditioned on global signal k .
Revised: (5) is equivalent to maximizing the mutual information between the representation v and global signal k and maximizing the conditional mutual information between v and the local signal z , conditioned on global signal k .

Original: Formally, we have:
Revised: Formally, we have:

Original: [Equation]
Revised: [Equation]

Original: This theorem suggests that we essentially combine both local structure and global semantic information as the self-supervised signal and maximize the mutual information between the representation v and their joint distribution ( k, z ) .
Revised: This theorem suggests that we essentially combine both local structure and global semantic information as the self-supervised signal and maximize the mutual information between the representation v and their joint distribution ( k, z ) .

Original: Next, we discuss how the learned representation affects the downstream task y based on the information bottleneck principle [51, 11, 56, 4].
Revised: Next, we discuss how the learned representation affects the downstream task y based on the information bottleneck principle [51, 11, 56, 4].

Original: The rationality of self-supervised learning is that the task-relevant information lies mostly in the shared information between the input and the self-supervised signals [51, 11, 4].
Revised: The rationality of self-supervised learning is that the task-relevant information lies mostly in the shared information between the input and the self-supervised signals [51, 11, 4].

Original: Speciﬁcally, we formulate our lightweight and reasonable assumption below, which serves as a foundation for our analysis.
Revised: Speciﬁcally, we formulate our lightweight and reasonable assumption below, which serves as a foundation for our analysis.

Original: Assumption 1.
Revised: Assumption 1.

Original: Nodes with similar labels should have similar "local structural roles" and "global semantic clusters.
Revised: Nodes with similar labels should have similar "local structural roles" and "global semantic clusters.

Original: " In this work, we equate "local structure" with the 1-hop neighborhood and "global semantic" with the clustering membership of a node.
Revised: " In this work, we equate "local structure" with the 1-hop neighborhood and "global semantic" with the clustering membership of a node.

Original: Formally, we have the task-relevant information y left in G except that in joint self-supervised signal ( z , k ) is relatively small: I ( G ; y | z , k )  ✏ .
Revised: Formally, we have the task-relevant information y left in G except that in joint self-supervised signal ( z , k ) is relatively small: I ( G ; y | z , k )  ✏ .

Original: Intuitively, this assumption indicates that most of the task-relevant information in the graph is contained in the self-supervised signal (local neighborhood and global semantic patterns).
Revised: Intuitively, this assumption indicates that most of the task-relevant information in the graph is contained in the self-supervised signal (local neighborhood and global semantic patterns).

Original: Based on this assumption, we have the following theorem, which reveals why the downstream tasks can beneﬁt from the learned representations learned by our objective function.
Revised: Based on this assumption, we have the following theorem, which reveals why the downstream tasks can beneﬁt from the learned representations learned by our objective function.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let v joint =

Original: [Equation]

Original: v

Original: [Equation]

Original: ; z , k ) , v

Original: [Equation]

Original: =

Original: [Equation] I [Equation] k ) , and v global = arg max v I ( v ; z ) .
Revised: Let v joint = arg max v I ( v ; z , k ) , v local = arg max v I ( v ; k ) , and v global = arg max v I ( v ; z ) .

Original: Formally, we have the following inequalities about the task-relevant information:
Revised: Formally, we have the following inequalities about the task-relevant information:

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 shows that the gap of task-relevant information between supervised representation v sup = arg max v I ( v , y ) and self-supervised representation v joint is ✏ .
Revised: Theorem 2 shows that the gap of task-relevant information between supervised representation v sup = arg max v I ( v , y ) and self-supervised representation v joint is ✏ .

Original: Thus, we can guarantee a good downstream performance as long as the Assumption 1 is satisﬁed.
Revised: Thus, we can guarantee a good downstream performance as long as the Assumption 1 is satisﬁed.

Original: It is noteworthy that jointly utilizing local structure and global semantics as the self-supervised signal is expected to contain more task information.
Revised: It is noteworthy that jointly utilizing local structure and global semantics as the self-supervised signal is expected to contain more task information.

Original: As further enlightenment, we can relate Eq.
Revised: As further enlightenment, we can relate Eq.

Original: (36) with the Bayes error rate [51]:
Revised: (36) with the Bayes error rate [51]:

Original: Corollary 1.
Revised: Corollary 1.

Original: Suppose that downstream label y is a M-categorical random variable.
Revised: Suppose that downstream label y is a M-categorical random variable.

Original: Then we have the upper bound for the downstream Bayes errors P e v
Revised: Then we have the upper bound for the downstream Bayes errors P e v

Original: [Equation]
Revised: [Equation]

Original: on learned representation v , where ˆ y is the estimation for label from our downstream classiﬁer:
Revised: on learned representation v , where ˆ y is the estimation for label from our downstream classiﬁer:

Original: [Equation]
Revised: [Equation]

Original: where Th( x ) = min { max { x, 0 } , 1  1 / | M |} is a thresholded operation [51].
Revised: where Th( x ) = min { max { x, 0 } , 1  1 / | M |} is a thresholded operation [51].

Original: Similarly, we can obtain the error upper bound of other representations v local and v global : RHS v local and RHS v global .
Revised: Similarly, we can obtain the error upper bound of other representations v local and v global : RHS v local and RHS v global .

Original: Then, we have inequalities on error upper bounds : RHS v joint  min(RHS v local , RHS v global ) .
Revised: Then, we have inequalities on error upper bounds : RHS v joint  min(RHS v local , RHS v global ) .

Original: This corollary says that our self-supervised signal has a tighter upper bound on the downstream Bayes error.
Revised: This corollary says that our self-supervised signal has a tighter upper bound on the downstream Bayes error.

Original: Thus, we can expect that the representation learned by our objective function, which utilizes both local structure and global semantic information, has superior performance on downstream tasks.
Revised: Thus, we can expect that the representation learned by our objective function, which utilizes both local structure and global semantic information, has superior performance on downstream tasks.

Original:  Experiments
Revised:  Experiments

Original: In this section, we empirically evaluate the proposed self-supervised learning method on several real-world graph datasets and analyze its behavior on graphs to gain further insights.
Revised: In this section, we empirically evaluate the proposed self-supervised learning method on several real-world graph datasets and analyze its behavior on graphs to gain further insights.

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Datasets.
Revised: Datasets.

Original: We perform experiments on widely-used homophilic graph datasets: Cora, Citeseer, and
Revised: We perform experiments on widely-used homophilic graph datasets: Cora, Citeseer, and

Original: Pubmed [42], as well as non-homophilic datasets: Texas, Cornell, Wisconsin [37], Penn94 and Twitch.
Revised: Pubmed [42], as well as non-homophilic datasets: Texas, Cornell, Wisconsin [37], Penn94 and Twitch.

Original: Penn94 and Twitch are two relatively large non-homophilous graph datasets proposed by [41, 29].
Revised: Penn94 and Twitch are two relatively large non-homophilous graph datasets proposed by [41, 29].

Original: We provide the detailed descriptions, statistics, and homophily measures of datasets in Appendix E.1.
Revised: We provide the detailed descriptions, statistics, and homophily measures of datasets in Appendix E.1.

Original: Baselines.
Revised: Baselines.

Original: To evaluate the effectiveness of DSSL, we consider the following representative unsupervised and self-supervised learning methods for the node representation task, including Deepwalk [38],
Revised: To evaluate the effectiveness of DSSL, we consider the following representative unsupervised and self-supervised learning methods for the node representation task, including Deepwalk [38],

Original: LINE [46], Struc2vec [40], GAE [26], VGAE [26], DGI [53], GraphCL [64], MVGRL [18] and
Revised: LINE [46], Struc2vec [40], GAE [26], VGAE [26], DGI [53], GraphCL [64], MVGRL [18] and

Original: BGRL [48].
Revised: BGRL [48].

Original: The detailed description of baselines and implementations are given in Appendix E.2.
Revised: The detailed description of baselines and implementations are given in Appendix E.2.

Original: Evaluation Protocol.
Revised: Evaluation Protocol.

Original: We consider two types of downstream tasks: node classiﬁcation and node clustering.
Revised: We consider two types of downstream tasks: node classiﬁcation and node clustering.

Original: For node classiﬁcation, we follow the standard linear-evaluation protocol on graphs [53, 48], where a linear classiﬁer is trained on top of the frozen representation, and test accuracy (ACC) is used as a proxy for representation quality.
Revised: For node classiﬁcation, we follow the standard linear-evaluation protocol on graphs [53, 48], where a linear classiﬁer is trained on top of the frozen representation, and test accuracy (ACC) is used as a proxy for representation quality.

Original: Forall datasets, we adopt the similar random split with a train/validation/test split ratio of 60% / 20% / 20% for the training of downstream linear classiﬁer following [37, 29].
Revised: For datasets, we adopt the similar random split with a train/validation/test split ratio of 48/32/20% for the training of downstream linear classiﬁer following [37]. Note that [37] claims that the ratios are 60/20/20%, which is different from the actual shared data splits as shown by [68, 34, 29].

Original: For the node clustering task, we perform K -means clustering on the obtained representations and set the number of clusters to the number of classes.
Revised: For the node clustering task, we perform K -means clustering on the obtained representations and set the number of clusters to the number of classes.

Original: We utilize the normalized mutual information (NMI) [54] as the evaluation metric for clustering.
Revised: We utilize the normalized mutual information (NMI) [54] as the evaluation metric for clustering.

Original: Setup.
Revised: Setup.

Original: For all self-supervised methods on graph neural networks, we consider a two-layer GCN [27] as the encoder unless otherwise stated, and randomly initialize parameters.
Revised: For all self-supervised methods on graph neural networks, we consider a two-layer GCN [27] as the encoder unless otherwise stated, and randomly initialize parameters.

Original: We run experiments with 10 random splits and report the average performance.
Revised: We run experiments with 10 random splits and report the average performance.

Original: We select the best conﬁguration of hyperparameters based on accuracy on the validation.
Revised: We select the best conﬁguration of hyperparameters based on accuracy on the validation.

Original: The detailed settings are given in Appendix E.2.
Revised: The detailed settings are given in Appendix E.2.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4.2 Overall Performance Comparison
Revised: 4.2 Overall Performance Comparison

Original: In this section, we conduct experiments on real-world graphs compared to state-of-the-art methods.
Revised: In this section, we conduct experiments on real-world graphs compared to state-of-the-art methods.

Original: Table 1 reports the average classiﬁcation accuracy with the standard deviation on node classiﬁcation after ten runs.
Revised: Table 1 reports the average classiﬁcation accuracy with the standard deviation on node classiﬁcation after ten runs.

Original: Since node clustering results have a similar tendency, we provide them in Appendix F.1.
Revised: Since node clustering results have a similar tendency, we provide them in Appendix F.1.

Original: From Table 1, we have the following observations: (1) Generally, our DSSL achieves the best performance on both node clustering and classiﬁcation tasks over the best baseline, demonstrating the effectiveness of DSSL on both homophilous and non-homophilous graphs and the robustness to different downstream tasks and graphs.
Revised: From Table 1, we have the following observations: (1) Generally, our DSSL achieves the best performance on both node clustering and classiﬁcation tasks over the best baseline, demonstrating the effectiveness of DSSL on both homophilous and non-homophilous graphs and the robustness to different downstream tasks and graphs.

Original: Especially, DSSL achieves a relative improvement of over 6% and 4% on Texas and Squirrel compared to the best baselines (2) Our DSSL cannot achieve the best performance on Cora.
Revised: Especially, DSSL achieves a relative improvement of over 6% and 4% on Texas and Squirrel compared to the best baselines (2) Our DSSL cannot achieve the best performance on Cora.

Original: This is reasonable since Cora is highly homophilous (see Appendix E.1), and the core design of augmentation in contrastive learning methods such as MVGRL and GraphCL enables them to be effective on homophilous graphs but failed on low-homophily settings.
Revised: This is reasonable since Cora is highly homophilous (see Appendix E.1), and the core design of augmentation in contrastive learning methods such as MVGRL and GraphCL enables them to be effective on homophilous graphs but failed on low-homophily settings.

Original: This supports our motivation in the introduction that it is relatively difﬁcult to design effective graph augmentation on non-homophilous graphs due to their heterogeneous and diverse patterns.
Revised: This supports our motivation in the introduction that it is relatively difﬁcult to design effective graph augmentation on non-homophilous graphs due to their heterogeneous and diverse patterns.

Original: (3) When compared with GAE, VGAE, and DGI, DSSL consistently and signiﬁcantly outperforms them.
Revised: (3) When compared with GAE, VGAE, and DGI, DSSL consistently and signiﬁcantly outperforms them.

Original: We deem that this improvement is mainly from the joint local semantic shift and global semantic clustering in DSSL, which is not included in existing methods.
Revised: We deem that this improvement is mainly from the joint local semantic shift and global semantic clustering in DSSL, which is not included in existing methods.

Original: [Figure]
Revised: [Figure]

Original: 4.3 Ablation Study and Parameter Analysis
Revised: 4.3 Ablation Study and Parameter Analysis

Original: In this section, we conduct an ablation study and investigate the sensitivity of hyper-parameters.
Revised: In this section, we conduct an ablation study and investigate the sensitivity of hyper-parameters.

Original: Ablation Study.
Revised: Ablation Study.

Original: We consider the following ablations: (A1)
Revised: We consider the following ablations: (A1)

Original: We remove the key component of DSSL: the local reconstruction loss ( w/o L local ).
Revised: We remove the key component of DSSL: the local reconstruction loss ( w/o L local ).

Original: (A2)
Revised: (A2)

Original: We remove global clustering loss to see if L local alone is still effective ( w/o L global ).
Revised: We remove global clustering loss to see if L local alone is still effective ( w/o L global ).

Original: (A3)
Revised: (A3)

Original: We remove the entropy term in Eq.
Revised: We remove the entropy term in Eq.

Original: (5) ( w/o entropy ).
Revised: (5) ( w/o entropy ).

Original: (A4)
Revised: (A4)

Original: We remove the semantic shift term g ✓ ( k ) in L local ( w/o semantic shift ).
Revised: We remove the semantic shift term g ✓ ( k ) in L local ( w/o semantic shift ).

Original: (A5)
Revised: (A5)

Original: We set posterior q  ( k | v i , z j ) = 1 /K as uniform distribution for each link ( w/ uniform posterior ).
Revised: We set posterior q  ( k | v i , z j ) = 1 /K as uniform distribution for each link ( w/ uniform posterior ).

Original: We show the ablation study results in Table 2.
Revised: We show the ablation study results in Table 2.

Original: L global alone does not provide much discriminative information, and it does not perform very well.
Revised: L global alone does not provide much discriminative information, and it does not perform very well.

Original: L local as a key component of DSSL alone produces better results
Revised: L local as a key component of DSSL alone produces better results

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: than the DGI baseline.
Revised: than the DGI baseline.

Original: We can also observe that considering semantic shift and personalized posterior can further improve the performance a lot which demonstrates our key motivations.
Revised: We can also observe that considering semantic shift and personalized posterior can further improve the performance a lot which demonstrates our key motivations.

Original: The full model (last row) achieves the best performance, which illustrates that different components in the proposed
Revised: The full model (last row) achieves the best performance, which illustrates that different components in the proposed

Original: DSSL are complementary to each other.
Revised: DSSL are complementary to each other.

Original: [Table]
Revised: [Table]

Original: Effect of Global Update on Prototypes .
Revised: Effect of Global Update on Prototypes .

Original: We conduct ablation studies to gain insights into the global updating of prototypes.
Revised: We conduct ablation studies to gain insights into the global updating of prototypes.

Original: As shown in Figure 3, we can observe that without Eq.
Revised: As shown in Figure 3, we can observe that without Eq.

Original: (9), the performance is not very good, and we are stuck in bad local optima.
Revised: (9), the performance is not very good, and we are stuck in bad local optima.

Original: As discussed above, a possible reason is that we will experience strong degeneracy if we only update prototype vectors with mini-batch training.
Revised: As discussed above, a possible reason is that we will experience strong degeneracy if we only update prototype vectors with mini-batch training.

Original: This ﬁgure also demonstrates that DSSL can converge within a few hundred steps, which is efﬁcient.
Revised: This ﬁgure also demonstrates that DSSL can converge within a few hundred steps, which is efﬁcient.

Original: Effect of Asymmetric Architecture .
Revised: Effect of Asymmetric Architecture .

Original: We then explore the effect of target decay rate ⌧ on the performance.
Revised: We then explore the effect of target decay rate ⌧ on the performance.

Original: Figureshows the learning curves of DSSL on Squirrel and Twitch.
Revised: Figureshows the learning curves of DSSL on Squirrel and Twitch.

Original: We can observe that the best result is achieved at ⌧ = 0 . 9 .
Revised: We can observe that the best result is achieved at ⌧ = 0 . 9 .

Original: When ⌧ = 1 . 0 , i.e., the target network is never updated,
Revised: When ⌧ = 1 . 0 , i.e., the target network is never updated,

Original: DSSL obtains a competitive result but is lower than ⌧ = 0 .
Revised: DSSL obtains a competitive result but is lower than ⌧ = 0 .

Original: This conﬁrms that slowly updating the target network is crucial in obtaining superior performance.
Revised: This conﬁrms that slowly updating the target network is crucial in obtaining superior performance.

Original: At the other extreme value ⌧ = 0 , the target network is the same as the online network, and DSSL demonstrates a degenerated performance.
Revised: At the other extreme value ⌧ = 0 , the target network is the same as the online network, and DSSL demonstrates a degenerated performance.

Original: Hyper-parameters Analysis.
Revised: Hyper-parameters Analysis.

Original: We investigate the hyper-parameters most essential to our framework design, i.e., the standard deviation  1 and  2 , the temperature of the Gumbel Softmax  , and the total number of factors K .
Revised: We investigate the hyper-parameters most essential to our framework design, i.e., the standard deviation  1 and  2 , the temperature of the Gumbel Softmax  , and the total number of factors K .

Original: The corresponding results are shown in Figure 4.
Revised: The corresponding results are shown in Figure 4.

Original:  21 resembles the temperature scaling in Eq.
Revised:  21 resembles the temperature scaling in Eq.

Original: We observe  21 is better to be selected from 0 .
Revised: We observe  21 is better to be selected from 0 .

Original: 6 to 1 . 0 , and a too small (e.g., 0 . 2 ) value may degenerate the performance in all three datasets.
Revised: 6 to 1 . 0 , and a too small (e.g., 0 . 2 ) value may degenerate the performance in all three datasets.

Original:  22 balances the local and global loss in Eq.
Revised:  22 balances the local and global loss in Eq.

Original: We can ﬁnd that having large values of  22 does not improve the performance, as the local loss plays an essential role in the proposed model.
Revised: We can ﬁnd that having large values of  22 does not improve the performance, as the local loss plays an essential role in the proposed model.

Original: Further, we observe that DSSL is not very sensitive to the Gumbel softmax temperature  , while a moderate hardness of the Softmax gives the best results.
Revised: Further, we observe that DSSL is not very sensitive to the Gumbel softmax temperature  , while a moderate hardness of the Softmax gives the best results.

Original: We also ﬁnd that as K increases from 2 to 8 , the performance of DGCL improves, which suggests the importance of decoupling latent factors.
Revised: We also ﬁnd that as K increases from 2 to 8 , the performance of DGCL improves, which suggests the importance of decoupling latent factors.

Original: However, training with large K will lead to a performance slightly drop.
Revised: However, training with large K will lead to a performance slightly drop.

Original: We provide results on other datasets in Appendix F.3.
Revised: We provide results on other datasets in Appendix F.3.

Original: Encoders Analysis.
Revised: Encoders Analysis.

Original: To further evaluate the effectiveness of the proposed DSSL, we consider the case where the self-supervised learning methods are implemented using another GNN encoder.
Revised: To further evaluate the effectiveness of the proposed DSSL, we consider the case where the self-supervised learning methods are implemented using another GNN encoder.

Original: Tableshows the results of the selected baseline with GAT as the encoder.
Revised: Tableshows the results of the selected baseline with GAT as the encoder.

Original: As shown in Table 3, our
Revised: As shown in Table 3, our

Original: DSSL performs better than all the compared methods in most cases, which once again proves the effectiveness of DSSL, and also shows that DSSL is broadly applicable to various GNN encoders.
Revised: DSSL performs better than all the compared methods in most cases, which once again proves the effectiveness of DSSL, and also shows that DSSL is broadly applicable to various GNN encoders.

Original: [Figure]
Revised: [Figure]

Original: 4.4 Visualization and Case Study
Revised: 4.4 Visualization and Case Study

Original: Local neighborhood Patterns .
Revised: Local neighborhood Patterns .

Original: We provide the visualization and case study results to understand how DSSL uncovers the latent patterns in local neighborhoods.
Revised: We provide the visualization and case study results to understand how DSSL uncovers the latent patterns in local neighborhoods.

Original: More results can be found in
Revised: More results can be found in

Original: Appendix F.4.
Revised: Appendix F.4.

Original: In Figure 5 (a), we calculate the cross-class neighborhood similarity (see Appendix E. for deﬁnition), which serves as the ground truth.
Revised: In Figure 5 (a), we calculate the cross-class neighborhood similarity (see Appendix E. for deﬁnition), which serves as the ground truth.

Original: If nodes with the same label share the same neighborhood distributions, the intra-class similarity should be high.
Revised: If nodes with the same label share the same neighborhood distributions, the intra-class similarity should be high.

Original: In Figure 5 (b), we calculate average posterior distribution
Revised: In Figure 5 (b), we calculate average posterior distribution

Original: q  ( k | v i ) over all nodes on each class and provide cosine similarity of them.
Revised: q  ( k | v i ) over all nodes on each class and provide cosine similarity of them.

Original: We can observe that our learned distribution shares a similar pattern with the cross-class neighborhood similarity, which shows that learned latent factors can capture the latent semantic information related to neighborhood patterns.
Revised: We can observe that our learned distribution shares a similar pattern with the cross-class neighborhood similarity, which shows that learned latent factors can capture the latent semantic information related to neighborhood patterns.

Original: In Figures 5 (c) and (d), we plot the subgraph of a randomly selected node, and the distribution q  ( k | v i , z j ) .
Revised: In Figures 5 (c) and (d), we plot the subgraph of a randomly selected node, and the distribution q  ( k | v i , z j ) .

Original: We use different colors to indicate different labels.
Revised: We use different colors to indicate different labels.

Original: We ﬁnd that similar neighbors over class generally have a similar distribution, while different types of links exhibit different latent distributions.
Revised: We ﬁnd that similar neighbors over class generally have a similar distribution, while different types of links exhibit different latent distributions.

Original: This observation matches our motivation that
Revised: This observation matches our motivation that

Original: DSSL can decouple the diverse semantics in the local neighborhoods.
Revised: DSSL can decouple the diverse semantics in the local neighborhoods.

Original: Global Semantic Patterns.
Revised: Global Semantic Patterns.

Original: We investigate how our learned semantic clusters perform on the longrange nodes.
Revised: We investigate how our learned semantic clusters perform on the longrange nodes.

Original: To show this, we study the learned cluster distribution q  ( k | v i ) for each node.
Revised: To show this, we study the learned cluster distribution q  ( k | v i ) for each node.

Original: To facilitate visualization, we randomly sample a sub-graph that contains high-hop neighborhoods from each dataset and use different colors to indicate different node labels.
Revised: To facilitate visualization, we randomly sample a sub-graph that contains high-hop neighborhoods from each dataset and use different colors to indicate different node labels.

Original: We then calculate the pair-wise similarity of the posteriors q  ( k | v i ) between nodes as shown in Figure 6.
Revised: We then calculate the pair-wise similarity of the posteriors q  ( k | v i ) between nodes as shown in Figure 6.

Original: We can observe that our learned semantic clusters exhibit similar patterns for the nodes in the same class.
Revised: We can observe that our learned semantic clusters exhibit similar patterns for the nodes in the same class.

Original: Some nodes exhibit similar semantic clusters regardless of the distance between them.
Revised: Some nodes exhibit similar semantic clusters regardless of the distance between them.

Original: For instance, node ID 6 exhibits a very similar posterior distribution to the same class node ID 19 in Pubmed, and node ID 3 exhibits similar posterior distribution to node ID 12 in Cornell, despite these nodes being ID 4 hops away.
Revised: For instance, node ID 6 exhibits a very similar posterior distribution to the same class node ID 19 in Pubmed, and node ID 3 exhibits similar posterior distribution to node ID 12 in Cornell, despite these nodes being ID 4 hops away.

Original: Since our DSSL seeks to pull the node representation to the inferred semantic clusters, we can learn the global node-node relationships in representations instead of favoring nearby nodes.
Revised: Since our DSSL seeks to pull the node representation to the inferred semantic clusters, we can learn the global node-node relationships in representations instead of favoring nearby nodes.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we study the problem of conducting self-supervised learning on non-homophilous graphs data.
Revised: In this paper, we study the problem of conducting self-supervised learning on non-homophilous graphs data.

Original: We present a novel decoupled self-supervised learning (DSSL) framework to decouple the diverse neighborhood context of a node in an unsupervised manner.
Revised: We present a novel decoupled self-supervised learning (DSSL) framework to decouple the diverse neighborhood context of a node in an unsupervised manner.

Original: Speciﬁcally, DSSL imitates the generative process of neighbors and explicitly models unobserved factors by latent variables.
Revised: Speciﬁcally, DSSL imitates the generative process of neighbors and explicitly models unobserved factors by latent variables.

Original: We show that DSSL can simultaneously capture the global clustering information and the local structure roles with the semantic shift.
Revised: We show that DSSL can simultaneously capture the global clustering information and the local structure roles with the semantic shift.

Original: We theoretically show that DSSL enjoys a good downstream performance.
Revised: We theoretically show that DSSL enjoys a good downstream performance.

Original: Extensive experiments on several graph datasets validated the superiority of DSSL and showed that DSSL could learn meaningful class-discriminative representations.
Revised: Extensive experiments on several graph datasets validated the superiority of DSSL and showed that DSSL could learn meaningful class-discriminative representations.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work was partially supported by the National Science Foundation (NSF) under grants number
Revised: This work was partially supported by the National Science Foundation (NSF) under grants number

Original: IIS-1909702 and IIS-1955851, Army Research Ofﬁce (ARO) under grant number W911NF-211-0198 and Department of Homeland Security under grant number E205949D.
Revised: IIS-1909702 and IIS-1955851, Army Research Ofﬁce (ARO) under grant number W911NF-211-0198 and Department of Homeland Security under grant number E205949D.

Original: The ﬁndings and conclusions in this paper do not necessarily reﬂect the view of the funding agency.
Revised: The ﬁndings and conclusions in this paper do not necessarily reﬂect the view of the funding agency.