Original: Decoupled Self-supervised Learning for Graphs
Revised: Decoupled Self-supervised Learning for Graphs

Original: Abstract
Revised: Abstract

Original: This paper studies the problem of conducting self-supervised learning for node representation learning on graphs.
Revised: This paper studies the problem of conducting self-supervised learning for node representation learning on graphs.

Original: Most existing self-supervised learning methods assume the graph is homophilous, where linked nodes often belong to the same class or have similar features.
Revised: Most existing self-supervised learning methods assume the graph is homophilous, where linked nodes often belong to the same class or have similar features.

Original: However, such assumptions of homophily do not always hold in real-world graphs.
Revised: However, such assumptions of homophily do not always hold in real-world graphs.

Original: We address this problem by developing a decoupled self-supervised learning (DSSL) framework for graph neural networks.
Revised: We address this problem by developing a decoupled self-supervised learning (DSSL) framework for graph neural networks.

Original: DSSL imitates a generative process of nodes and links from latent variable modeling of the semantic structure, which decouples different underlying semantics between different neighborhoods into the self-supervised learning process.
Revised: DSSL imitates a generative process of nodes and links from latent variable modeling of the semantic structure, which decouples different underlying semantics between different neighborhoods into the self-supervised learning process.

Original: Our DSSL framework is agnostic to the encoders and does not need prefabricated augmentations, thus is flexible to different graphs.
Revised: Our DSSL framework is agnostic to the encoders and does not need prefabricated augmentations, thus is ﬂexible to different graphs.

Original: To effectively optimize the framework, we derive the evidence lower bound of the self-supervised objective and develop a scalable training algorithm with variational inference.
Revised: To effectively optimize the framework, we derive the evidence lower bound of the self-supervised objective and develop a scalable training algorithm with variational inference.

Original: We provide a theoretical analysis to justify that DSSL enjoys the better downstream performance.
Revised: We provide a theoretical analysis to justify that DSSL enjoys the better downstream performance.

Original: Extensive experiments on various types of graph benchmarks demonstrate that our proposed framework can achieve better performance compared with competitive baselines.
Revised: Extensive experiments on various types of graph benchmarks demonstrate that our proposed framework can achieve better performance compared with competitive baselines.

Original:  Introduction
Revised:  Introduction

Original: Graph-structured data is ubiquitous in the real world, such as social networks, knowledge graphs and molecular structures.
Revised: Graph-structured data is ubiquitous in the real world, such as social networks, knowledge graphs, and molecular structures.

Original: In recent years, graph neural networks (GNNs) [16, 27, 52, 7, 60] have been proven to be powerful in node representation learning over graph-structured data.
Revised: In recent years, graph neural networks (GNNs) [16, 27, 52, 7, 60] have been proven to be powerful in node representation learning over graph-structured data.

Original: Typically, GNNs are trained with annotated labeled data in a supervised manner.
Revised: Typically, GNNs are trained with annotated labeled data in a supervised manner.

Original: However, collecting labeled data is expensive and impractical in many applications, especially for those requiring domain knowledge, such as medicine and chemistry [71, 20].
Revised: However, collecting labeled data is expensive and impractical in many applications, especially for those requiring domain knowledge, such as medicine and chemistry [71, 20].

Original: Moreover, supervised learning may suffer from problems of less-transferrable, over-fitting and poor generalization when the task labels are scarce [10, 64].
Revised: Moreover, supervised learning may suffer from problems of less-transferrable, over-ﬁtting and poor generalization when the task labels are scarce [10, 64].

Original: Recently, self-supervised learning (SSL) provides a promising learning paradigm that reduces the dependence on manual labels in the image domain [5, 12, 13, 6].
Revised: Recently, self-supervised learning (SSL) provides a promising learning paradigm that reduces the dependence on manual labels in the image domain [5, 12, 13, 6].

Original: Compared to image data, there are unique challenges in designing self-supervised learning schemes for graph-structured data since nodes in the graph are correlated with each other rather than completely independent, and geometric structures are essential and heavily impact the performance in downstream tasks [27].
Revised: Compared to image data, there are unique challenges in designing self-supervised learning schemes for graph-structured data since nodes in the graph are correlated with each other rather than completely independent, and geometric structures are essential and heavily impact the performance in downstream tasks [27].

Original: A number of recent works [53, 18, 64, 69, 48, 65, 45] have studied graph self-supervised learning and confirm that it can learn transferrable and generalizable node representations without any labels.
Revised: A number of recent works [53, 18, 64, 69, 48, 65, 45] have studied graph self-supervised learning and conﬁrm that it can learn transferrable and generalizable node representations without any labels.

Original: Typically, there are two main self-supervised schemes to capture structure information in graphs [53, 64, 45].
Revised: Typically, there are two main self-supervised schemes to capture structure information in graphs [53, 64, 45].

Original: The first scheme is reconstructing the vertex adjacency following traditional network-embedding methods [26, 14, 17, 16, 46], which learns an encoder that imposes the topological closeness of nodes in the graph structure on latent representations.
Revised: The ﬁrst scheme is reconstructing the vertex adjacency following traditional network-embedding methods [26, 14, 17, 16, 46], which learns an encoder that imposes the topological closeness of nodes in the graph structure on latent representations.

Original: The key assumption behind this scheme is that neighboring nodes have similar representations [53, 46].
Revised: The key assumption behind this scheme is that neighboring nodes have similar representations [53, 46].

Original: However, this assumption over-emphasizes proximity [53, 64, 47] and does not always hold true for heterophilic and non-homophilous (mixing) graphs.
Revised: However, this assumption over-emphasizes proximity [53, 64, 47] and does not always hold true for heterophilic and non-homophilous (mixing) graphs.

Original: In comparison, contrastive learning methods [53, 18, 64, 69, 48, 65] construct two graph 
Revised: In comparison, contrastive learning methods [53, 18, 64, 69, 48, 65] construct two graph Information (NeurIPS

Original: [Figure]
Revised: [Figure]

Original: views via the stochastic augmentation and then learns representations by contrasting views with information maximization principle.
Revised: views via the stochastic augmentation and then learns representations by contrasting views with information maximization principle.

Original: While these contrastive methods can capture structure information without directly emphasizing proximity, their performance relies on topology augmentation [70, 50].
Revised: While these contrastive methods can capture structure information without directly emphasizing proximity, their performance relies on topology augmentation [70, 50].

Original: Importantly, conducting augmentation for non-homophilous graphs is relatively difficult since linked nodes may be dissimilar, and nodes with high similarities might be farther away from each other.
Revised: Importantly, conducting augmentation for non-homophilous graphs is relatively difﬁcult since linked nodes may be dissimilar, and nodes with high similarities might be farther away from each other.

Original: Hence, the above problems pose an important and challenging research question: How to design an effective self-supervised scheme for node representation learning in non-homophilous graphs?
Revised: Hence, the above problems pose an important and challenging research question: How to design an effective self-supervised scheme for node representation learning in non-homophilous graphs?

Original: We approach this question by taking advantage of neighborhood strategies of nodes for the selfsupervised learning on non-homophilous graphs.
Revised: We approach this question by taking advantage of neighborhood strategies of nodes for the selfsupervised learning on non-homophilous graphs.

Original: Our key motivation is that nodes with similar neighborhood patterns should have similar representations.
Revised: Our key motivation is that nodes with similar neighborhood patterns should have similar representations.

Original: In other words, we expect that the neighborhood distributions can be exploited to distinguish node representations.
Revised: In other words, we expect that the neighborhood distributions can be exploited to distinguish node representations.

Original: Our assumption is more general than the standard homophily assumption as shown in Figure 1.
Revised: Our assumption is more general than the standard homophily assumption as shown in Figure 1.

Original: For instance, while the gender prediction in common dating networks lacks homophily [1], neighborhood distribution is very informative to the node gender labels, i.e., nodes with similar neighborhoods are likely to be similar.
Revised: For instance, while the gender prediction in common dating networks lacks homophily [1], neighborhood distribution is very informative to the node gender labels, i.e., nodes with similar neighborhoods are likely to be similar.

Original: While the motivation is straightforward, we are faced with two main challenges.
Revised: While the motivation is straightforward, we are faced with two main challenges.

Original: The first challenge is capturing the local neighborhood distribution in a self-supervised learning manner.
Revised: The ﬁrst challenge is capturing the local neighborhood distribution in a self-supervised learning manner.

Original: The neighborhood distributions typically follow heterogeneous and diverse patterns.
Revised: The neighborhood distributions typically follow heterogeneous and diverse patterns.

Original: For instance, a node usually connects with others due to the complex interaction of different latent factors and therefore possesses distribution consisting of local mixing patterns wherein certain parts of the neighborhood are homophilous while others are non-homophilous.
Revised: For instance, a node usually connects with others due to the complex interaction of different latent factors and therefore possesses distribution consisting of local mixing patterns wherein certain parts of the neighborhood are homophilous while others are non-homophilous.

Original: The lack of supervision obstructs us from modeling the distribution of neighborhoods.
Revised: The lack of supervision obstructs us from modeling the distribution of neighborhoods.

Original: The second challenge is capturing the long-range semantic dependencies.
Revised: The second challenge is capturing the long-range semantic dependencies.

Original: As shown in Figure 1, in non-homophilous graphs, nodes with high semantic and local structural similarities might be farther away from each other.
Revised: As shown in Figure 1, in non-homophilous graphs, nodes with high semantic and local structural similarities might be farther away from each other.

Original: For this reason, global semantic information is the objective that we would incorporate for self-supervised learning.
Revised: For this reason, global semantic information is the objective that we would incorporate for self-supervised learning.

Original: This paper presents a new self-supervised framework, decoupled self-supervised learning (DSSL), to achieve a good balance between these two challenges.
Revised: This paper presents a new self-supervised framework, decoupled self-supervised learning (DSSL), to achieve a good balance between these two challenges.

Original: At the core of DSSL is the latent variables, which empower the model with the flexibility to decouple the heterogeneous and diverse patterns in local neighborhood distributions and capture the global semantic dependencies in a coherent and synergistic framework.
Revised: At the core of DSSL is the latent variables, which empower the model with the ﬂexibility to decouple the heterogeneous and diverse patterns in local neighborhood distributions and capture the global semantic dependencies in a coherent and synergistic framework.

Original: Our contributions can be summarized as follows: (1) We propose a DSSL for performing self-supervised learning on non-homophilous graphs, which can leverage both useful local structure and global semantic information.
Revised: Our contributions can be summarized as follows: (1) We propose a DSSL for performing self-supervised learning on non-homophilous graphs, which can leverage both useful local structure and global semantic information.

Original: (2) We develop an efficient training algorithm based on variational inference to simultaneously infer the latent variables and learn encoder parameters.
Revised: (2) We develop an efﬁcient training algorithm based on variational inference to simultaneously infer the latent variables and learn encoder parameters.

Original: (3) We analyze the properties of DSSL and theoretically show that the learned representations can achieve better downstream performance.
Revised: (3) We analyze the properties of DSSL and theoretically show that the learned representations can achieve better downstream performance.

Original: (4) We conduct experiments on real-world non-homophilous graphs, and the results demonstrate the effectiveness of our self-supervised learning framework.
Revised: (4) We conduct experiments on real-world non-homophilous graphs, and the results demonstrate the effectiveness of our self-supervised learning framework.

Original:  Related Work
Revised:  Related Work

Original: Non-homophilous Graphs.
Revised: Non-homophilous Graphs.

Original: Non-homophilous is known in many settings such as online transaction networks [36], dating networks [1] and molecular networks [68].
Revised: Non-homophilous is known in many settings such as online transaction networks [36], dating networks [1] and molecular networks [68].

Original: Recently, various GNNs [37, 68, 59, 29, 69, 8, 44, 62] have been proposed to deal with non-homophilous graphs with different methods such as potential neighbor discovery [37, 23, 22], adaptive message propagation [8, 59], exploring high-frequency signals [2] and higher-order message passing [68, 5].
Revised: Recently, various GNNs [37, 68, 59, 29, 69, 8, 44, 62] have been proposed to deal with non-homophilous graphs with different methods such as potential neighbor discovery [37, 23, 22], adaptive message propagation [8, 59], exploring high-frequency signals [2] and higher-order message passing [68, 5].

Original: Despite their success, they typically consider the semi-supervised setting and are trained with task-specific labeled data, while in
Revised: Despite their success, they typically consider the semi-supervised setting and are trained with task-speciﬁc labeled data, while in

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: practice, labels are often limited, expensive, and even inaccessible.

Revised: contrast, in this the problem of self-supervised learning: learning node representations without relying on labels.

Original: Self-supervised Learning on Graphs.
Revised: Self-supervised Learning on Graphs.

Original: Self-supervised learning holds great promise for improving representations when labeled data are scarce [5, 12, 13, 19].
Revised: Self-supervised learning holds great promise for improving representations when labeled data are scarce [5, 12, 13, 19].

Original: Earlier combinations of GNNs and self-supervised learning involve GraphSAGE [16], VGAE [26], and Graphite [15], which typically follow the traditional network-embedding methods [38, 46, 14] and adopt the link reconstruction or random walk principle.
Revised: Earlier combinations of GNNs and self-supervised learning involve GraphSAGE [16], VGAE [26], and Graphite [15], which typically follow the traditional network-embedding methods [38, 46, 14] and adopt the link reconstruction or random walk principle.

Original: Since these methods over-emphasize node proximity at the cost of structural information [53, 45, 64], various graph contrastive learning methods have been proposed [53, 18, 64, 69, 48, 35], which aim to learn representations by contrasting representation under differently augmented views and have achieved promising performance.
Revised: Since these methods over-emphasize node proximity at the cost of structural information [53, 45, 64], various graph contrastive learning methods have been proposed [53, 18, 64, 69, 48, 35], which aim to learn representations by contrasting representation under differently augmented views and have achieved promising performance.

Original: However, they heavily rely on complex data- or task-specific augmentations, which are prohibitively difficult for non-homophilous graphs.
Revised: However, they heavily rely on complex data- or task-speciﬁc augmentations, which are prohibitively difﬁcult for non-homophilous graphs.

Original: Our work differs from the above methods and aims to answer the question of how to design an effective self-supervised learning scheme for non-homophilous graphs.
Revised: Our work differs from the above methods and aims to answer the question of how to design an effective self-supervised learning scheme for non-homophilous graphs.

Original: Disentangled graph learning.
Revised: Disentangled graph learning.

Original: Our work is also related to but different from existing disentangled graph learning that aims to decouple latent factors in the graph.
Revised: Our work is also related to but different from existing disentangled graph learning that aims to decouple latent factors in the graph.

Original: vGraph [43] considers a mixture process to define the conditional probability on each edge.
Revised: vGraph [43] considers a mixture process to deﬁne the conditional probability on each edge.

Original: vGraph is a classical shallow network embedding algorithm.
Revised: vGraph is a classical shallow network embedding algorithm.

Original: However, we instead tackle the problem of self-supervised learning with
Revised: However, we instead tackle the problem of self-supervised learning with

Original: GNNs.
Revised: GNNs.

Original: There are a couple of works that explore the disentangled factors in the node-level [33, 32], edge-level [67] and graph-level [63].
Revised: There are a couple of works that explore the disentangled factors in the node-level [33, 32], edge-level [67] and graph-level [63].

Original: Whereas these methods require task-specific labels that can be extremely scarce.
Revised: Whereas these methods require task-speciﬁc labels that can be extremely scarce.

Original: By contrast, we address the problem of learning node representations on nonhomophilous graphs without labels.
Revised: By contrast, we address the problem of learning node representations on nonhomophilous graphs without labels.

Original: Disentangled contrastive learning [28] learns disentangled graph representations without labeled graphs.
Revised: Disentangled contrastive learning [28] learns disentangled graph representations without labeled graphs.

Original: [61] proposes the self-supervised graph-level representation learning with disentangled local and global structure [61].
Revised: [61] proposes the self-supervised graph-level representation learning with disentangled local and global structure [61].

Original: Nevertheless, their goal is to conduct graph-level classification tasks, but we tackle a node-level representation problem.
Revised: Nevertheless, their goal is to conduct graph-level classiﬁcation tasks, but we tackle a node-level representation problem.

Original: [66, 9, 55] consider adding a clustering layer or a prototype clustering component to capture the global information.
Revised: [66, 9, 55] consider adding a clustering layer or a prototype clustering component to capture the global information.

Original: However, they are still based on contrastive learning [66, 9] or supervised learning [55].
Revised: However, they are still based on contrastive learning [66, 9] or supervised learning [55].

Original: By contrast, our framework is an unsupervised generative model and does not rely on graph augmentations and downstream labels.
Revised: By contrast, our framework is an unsupervised generative model and does not rely on graph augmentations and downstream labels.

Original: Moreover, we focus on non-homophilous graphs where connected nodes may not be similar to each other by decoupling the local diverse neighborhood context.
Revised: Moreover, we focus on non-homophilous graphs where connected nodes may not be similar to each other by decoupling the local diverse neighborhood context.

Original:  Decoupled Self-supervised Learning
Revised:  Decoupled Self-supervised Learning

Original: In this section, we describe our problem setting with respect to self-supervised learning and demonstrate our approach.
Revised: In this section, we describe our problem setting with respect to self-supervised learning and demonstrate our approach.

Original: An illustration of generative and inference processes is depicted in Figure 2 (a).
Revised: An illustration of generative and inference processes is depicted in Figure 2 (a).

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: We consider a graph G =
Revised: We consider a graph G =

Original: ( V , E ) , where V is a set of |V| =
Revised: ( V , E ) , where V is a set of |V| =

Original: N nodes and E ⊆ V × V is a set of |E| edges between nodes.
Revised: N nodes and E ✓ V ⇥ V is a set of |E| edges between nodes.

Original: A ∈ { 0 , 1 } N × N is the adjacency matrix of G .
Revised: A 2 { 0 , 1 } N ⇥ N is the adjacency matrix of G .

Original: The ( i, j ) -th
Revised: The ( i, j ) -th

Original: element A ij =if there exists an edge ( v i , v j ) between node v i and v j , otherwise A ij = 0 .
Revised: element A ij =if there exists an edge ( v i , v j ) between node v i and v j , otherwise A ij = 0 .

Original: The matrix X ∈ R N × D describes node features.
Revised: The matrix X 2 R N ⇥ D describes node features.

Original: The i -th row of X , i.e., x i , is the feature of node v i .
Revised: The i -th row of X , i.e., x i , is the feature of node v i .

Original: Given the graph
Revised: Given the graph

Original: G =
Revised: G =

Original: ( X , A ) , the objective of self-supervised node representation learning is to learn an encoder function f θ ( X , A ) :R N × N × R N × D
Revised: ( X , A ) , the objective of self-supervised node representation learning is to learn an encoder function f ✓ ( X , A ) :

Original: → R N × D ′ where θ denotes its parameters, such that the representation of node v i : f θ ( X , A )
Revised: R N ⇥ N ⇥ R N ⇥ D ! R N ⇥ D 0 where ✓ denotes its parameters, such that the representation of node v i : f ✓ ( X , A )

Original: [ i ] , can be used for downstream tasks such as node classification.
Revised: [ i ] , can be used for downstream tasks such as node classiﬁcation.

Original: 3.2 The Probabilistic Framework
Revised: 3.2 The Probabilistic Framework

Original: In this section, we introduce our framework, decoupled self-supervised learning, which can learn meaningful node representations on non-homophilous graphs by capturing the intrinsic graph structure.
Revised: In this section, we introduce our framework, decoupled self-supervised learning, which can learn meaningful node representations on non-homophilous graphs by capturing the intrinsic graph structure.

Original: The core idea of DSSL is to model the distributions of node neighbors via a mixture of generative processes in the representation space.
Revised: The core idea of DSSL is to model the distributions of node neighbors via a mixture of generative processes in the representation space.

Original: Specifically, we model the generation of neighbors by assuming each node has latent heterogeneous factors which are utilized to make connections to its different neighbors.
Revised: Speciﬁcally, we model the generation of neighbors by assuming each node has latent heterogeneous factors which are utilized to make connections to its different neighbors.

Original: Intuitively, the factor denotes various reasons behind why two nodes are connected in non-homophilous graphs.
Revised: Intuitively, the factor denotes various reasons behind why two nodes are connected in non-homophilous graphs.

Original: For instance, two nodes in a school network will be connected depending on some factors such as colleagues, friends, or classmates; in protein networks, even if they do not have similar features, different amino acid types are likely to be connected due to various interactions.
Revised: For instance, two nodes in a school network will be connected depending on some factors such as colleagues, friends, or classmates; in protein networks, even if they do not have similar features, different amino acid types are likely to be connected due to various interactions.

Original: Formally, let f θ ( X , A )
Revised: Formally, let f ✓ ( X , A )

Original: [ i ] = v i and f θ ( X , A )
Revised: [ i ] = v i and f ✓ ( X , A )

Original: [ j ] =
Revised: [ j ] =

Original: z j be the representations of nodes v i and v j , respectively.
Revised: z j be the representations of nodes v i and v j , respectively.

Original: Here we utilize different notations, i.e., v and z to distinguish between central and neighbor nodes since each node plays two roles: the node itself and a specific neighbor of other nodes.
Revised: Here we utilize different notations, i.e., v and z to distinguish between central and neighbor nodes since each node plays two roles: the node itself and a speciﬁc neighbor of other nodes.

Original: Our goal is to find the encoder parameter θ which maximizes the likelihood of distribution p ( z j | v i ; θ ) on central node and its neighbor.
Revised: Our goal is to ﬁnd the encoder parameter ✓ which maximizes the likelihood of distribution p ( z j | v i ; ✓ ) on central node and its neighbor.

Original: To model the unobserved factors, we associate every node v i with a discrete latent variable k to indicate to which factor v i has.
Revised: To model the unobserved factors, we associate every node v i with a discrete latent variable k to indicate to which factor v i has.

Original: Assume that there are K factors in total, the log-likelihood of node neighbors v i can be written by marginalizing out the latent variables:
Revised: Assume that there are K factors in total, the log-likelihood of node neighbors v i can be written by marginalizing out the latent variables:

Original: [Equation][Equation]
Revised: [Equation]

Original: ) where N ( i ) is the set of out-neighbors of v i , the distribution p θ ( k | v i ) indicates the assignment of latent semantics over central node representation v i , and p θ ( z j | v i , k ) is the probability that node v i and its neighbor v j are connected under factor k .
Revised: where N ( i ) is the set of out-neighbors of v i , the distribution p ✓ ( k | v i ) indicates the assignment of latent semantics over central node representation v i , and p ✓ ( z j | v i , k ) is the probability that node v i and its neighbor v j are connected under factor k .

Original: Unlike previous works [46, 14, 26], which directly encourage nearby nodes to have similar representations, we provide an alternative way to model node neighbors and seek to decouple their latent relationship without any prior on neighbor partitions.
Revised: Unlike previous works [46, 14, 26], which directly encourage nearby nodes to have similar representations, we provide an alternative way to model node neighbors and seek to decouple their latent relationship without any prior on neighbor partitions.

Original: The similar high-level ideas of modeling latent variables between two nodes has also been explored in local augmentation in graphs (LA-GCN) [31] and vGraph [43].
Revised: The similar high-level ideas of modeling latent variables between two nodes has also been explored in local augmentation in graphs (LA-GCN) [31] and vGraph [43].

Original: Different from them, we consider modeling discrete relations in the learned representation space for self-supervised learning on non-homophilous graphs.
Revised: Different from them, we consider modeling discrete relations in the learned representation space for self-supervised learning on non-homophilous graphs.

Original: In Eq.
Revised: In Eq.

Original: (1), probabilities p θ ( k | v i ) and p θ ( z j | v i , k ) are not specified, and involve discrete latent variables.
Revised: (1), probabilities p ✓ ( k | v i ) and p ✓ ( z j | v i , k ) are not speciﬁed, and involve discrete latent variables.

Original: To make it solvable, we introduce the following generative process.
Revised: To make it solvable, we introduce the following generative process.

Original: Let µ k and Σ k be the mean and variance of the latent mixture component k , and π k be its corresponding mixture probability.
Revised: Let µ k and ⌃ k be the mean and variance of the latent mixture component k , and ⇡ k be its corresponding mixture probability.

Original: The generation of a node and its neighbor shown in Figure 2 (a) typically involves three steps: (1) draw a latent variable k from a categorical distribution p ( k ) on all mixture components, where p ( k ) is usually defined as uniform distribution p ( k )
Revised: The generation of a node and its neighbor shown in Figure 2 (a) typically involves three steps: (1) draw a latent variable k from a categorical distribution p ( k ) on all mixture components, where p ( k ) is usually deﬁned as uniform distribution p ( k )

Original: = 1 K for unknown graphs and better generalization, (2) draw the central node representation v i from the Gaussian distribution p θ ( v i | k )=
Revised: = 1 K for unknown graphs and better generalization, (2) draw the central node representation v i from the Gaussian distribution p ✓ ( v i | k )

Original:  N ( v i ; µ k , Σ k ), and (3) draw the neighbor representation z j from Gaussian distribution p θ ( z j | v i , k ) =
Revised: = N ( v i ; µ k , ⌃ k ), and (3) draw the neighbor representation z j from Gaussian distribution p ✓ ( z j | v i , k ) =

Original: N ( z j ; µ z j , Σ j ) where the mean depends both on central representation and latent variable: µ z j = v i + βg θ ( k ) .
Revised: N ( z j ; µ z j , ⌃ j ) where the mean depends both on central representation and latent variable: µ z j = v i +  g ✓ ( k ) .

Original: Here the projector g θ ( · ) denotes another network that embeds latent variable k to the representation space, and β is the parameter that controls the strength of interpolation.
Revised: Here the projector g ✓ ( · ) denotes another network that embeds latent variable k to the representation space, and  is the parameter that controls the strength of interpolation.

Original: In practice, to reduce complexity, we consider using isotropic Gaussian, i.e., ∀ k : Σ k
Revised: In practice, to reduce complexity, we consider using isotropic Gaussian, i.e., 8 k : ⌃ k

Original: = I σ 21 and ∀ j : Σ
Revised: = I  21 and 8 j : ⌃ j = I

Original: j = I σ 22 where I is the identity matrix, σ 1 and σ 2 are hyperparameters.
Revised:  22 where I is the identity matrix,  1 and  2 are hyperparameters.

Original: In alignment with this generative process, the joint distribution of the edge and latent variable can be written as:
Revised: In alignment with this generative process, the joint distribution of the edge and latent variable can be written as:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, p θ ( v i | k ) can be viewed as the probability for node representation under the k -th mixture component.
Revised: Intuitively, p ✓ ( v i | k ) can be viewed as the probability for node representation under the k -th mixture component.

Original: Regarding p θ ( z j | v i , k ) =
Revised: Regarding p ✓ ( z j | v i , k ) =

Original: N ( z j ; µ z j , Σ j ) , this formulation is inspired by knowledge graph embedding [3], where two entities should be closed to each other under a certain relation operation.
Revised: N ( z j ; µ z j , ⌃ j ) , this formulation is inspired by knowledge graph embedding [3], where two entities should be closed to each other under a certain relation operation.

Original: However, unlike knowledge graph embedding, the relational context is a latent variable and not observed in our setting.
Revised: However, unlike knowledge graph embedding, the relational context is a latent variable and not observed in our setting.

Original: Intuitively, instead of enforcing the exact representation alignment of two linked nodes [26, 14, 17, 16, 46], our design can relax this strong homophily assumption and account for the semantic shift of representations between the central node and its neighbors.
Revised: Intuitively, instead of enforcing the exact representation alignment of two linked nodes [26, 14, 17, 16, 46], our design can relax this strong homophily assumption and account for the semantic shift of representations between the central node and its neighbors.

Original: Now, the posterior probability p θ ( k | v i ) in Eq.
Revised: Now, the posterior probability p ✓ ( k | v i ) in Eq.

Original: (1) can be derived by using Bayesian rules as follows:
Revised: (1) can be derived by using Bayesian rules as follows:

Original: [Equation]
Revised: [Equation]

Original: where µ = { µ k } Kk =1 can be treated as a set of trainable prototype representations, which are the additional distribution parameters.
Revised: where µ = { µ k } Kk =1 can be treated as a set of trainable prototype representations, which are the additional distribution parameters.

Original: This posterior probability represents the soft semantic assignment of the learned representation to the prototypes, which makes the node with similar semantic properties be close to its prototype and encode both the inter- and intra-cluster variation.
Revised: This posterior probability represents the soft semantic assignment of the learned representation to the prototypes, which makes the node with similar semantic properties be close to its prototype and encode both the inter- and intra-cluster variation.

Original: Substituting p θ ( k | v i ) and p θ ( z j | v i , k ) in Eq.
Revised: Substituting p ✓ ( k | v i ) and p ✓ ( z j | v i , k ) in Eq.

Original: (1) with the specified probabilities, we get final objective L DSSL ( θ, µ ) .
Revised: (1) with the speciﬁed probabilities, we get ﬁnal objective L DSSL ( ✓ , µ ) .

Original: 3.3 Evidence Lower Bound
Revised: 3.3 Evidence Lower Bound

Original: Given the framework above, we are interested in: (i) learning the model parameters θ and µ by maximizing the log-likelihood
Revised: Given the framework above, we are interested in: (i) learning the model parameters ✓ and µ by maximizing the log-likelihood

Original: L DSSL ( θ, µ ) , and (ii) inferring the posterior of latent variable k for each observed links.
Revised: L DSSL ( ✓ , µ ) , and (ii) inferring the posterior of latent variable k for each observed links.

Original: However, it is computationally intractable to directly solve these two problems due to the latent variables.
Revised: However, it is computationally intractable to directly solve these two problems due to the latent variables.

Original: To solve this, we resort to amortized variational inference methods [25], and maximize the evidence lower-bound (ELBO) of Eq.
Revised: To solve this, we resort to amortized variational inference methods [25], and maximize the evidence lower-bound (ELBO) of Eq.

Original: (1), i.e.,
Revised: (1), i.e.,

Revised: L

Original: [Equation]
Revised: [Equation]

Original: L
Revised: [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Revised: L

Original: where q ϕ ( k | v i , z j ) is the introduced variational distribution parameterized by ϕ and H ( · ) is the entropy operator.
Revised: [Equation] ( ✓ ,  , [Equation] ) =

Revised: [Equation]

Revised: where q  ( k | v i , z j ) is the introduced variational distribution parameterized by  and H ( · ) is the entropy operator.

Original: z j is included in this variational posterior, so the inference is also conditioned on the neighborhood information.
Revised: z j is included in this variational posterior, so the inference is also conditioned on the neighborhood information.

Original: We derive this ELBO in Appendix A.1.
Revised: We derive this ELBO in Appendix A.1.

Original: Maximizing this ELBO w.r.t.
Revised: Maximizing this ELBO w.r.t.

Original: { θ, ϕ, µ } is equivalent to (i) maximizing L DSSL ( θ, µ ) and to (ii) make variational q ϕ ( k | v i , z j ) be close to true posterior.
Revised: { ✓ ,  , µ } is equivalent to (i) maximizing L DSSL ( ✓ , µ ) and to (ii) make variational q  ( k | v i , z j ) be close to true posterior.

Original: Plugging the parameterized probabilities into this ELBO, we obtain the following loss to minimize.
Revised: Plugging the parameterized probabilities into this ELBO, we obtain the following loss to minimize.

Original: See Appendix A.2 for corresponding derivations.
Revised: See Appendix A.2 for corresponding derivations.

Original: [Equation]
Revised: [Equation]

Original: where q ϕ ( k | v i ) = 1 / | N ( i ) | (cid:80) j ∈ N ( i ) q ϕ ( k | v i , z j ) is the posterior probability of semantic assignment for central node v i , by aggregating all its neighbors.
Revised: where q  ( k | v i ) = 1 / | N ( i ) | P j 2 N ( i ) q  ( k | v i , z j ) is the posterior probability of semantic assignment for central node v i , by aggregating all its neighbors.

Original: Thus, the first term (denoted as L local ) in the loss encourages the model to reconstruct the local neighbors while considering different semantic shifts captured by latent variable k
Revised: Thus, the ﬁrst term (denoted as L local ) in the loss encourages the model to reconstruct the local neighbors while considering different semantic shifts captured by latent variable k

Original: (see Figure 2 (b)).
Revised: (see Figure 2 (b)).

Original: The second term (denoted as L global ) encourages the model to perform clustering with learned representation where possible, i.e., seeking to push the representation v i to its closest prototype cluster (see Figure 2 (c)).
Revised: The second term (denoted as L global ) encourages the model to perform clustering with learned representation where possible, i.e., seeking to push the representation v i to its closest prototype cluster (see Figure 2 (c)).

Original: The final entropy term makes the model choose to have high entropy over q ϕ ( k | v i , z j ) such that all of the K -channel losses must be low.
Revised: The ﬁnal entropy term makes the model choose to have high entropy over q  ( k | v i , z j ) such that all of the K -channel losses must be low.

Original: Overall, this loss derived from ELBO can capture global semantic similarities over neighborhoods and learn to decouple different latent patterns in the local neighbors.
Revised: Overall, this loss derived from ELBO can capture global semantic similarities over neighborhoods and learn to decouple different latent patterns in the local neighbors.

Original: Regarding the variaitonal distribution q ϕ ( k | v i , z j ) , we model it as categorical distribution since k is a discrete multinomial variable.
Revised: Regarding the variaitonal distribution q  ( k | v i , z j ) , we model it as categorical distribution since k is a discrete multinomial variable.

Original: Specifically, the representations v i and z j are encoded to a combined representation and then q ϕ ( k | v i , z j ) is determined by an output softmax inference head as follows:
Revised: Speciﬁcally, the representations v i and z j are encoded to a combined representation and then q  ( k | v i , z j ) is determined by an output softmax inference head as follows:

Original: [Equation]
Revised: [Equation]

Original: where h ϕ denotes the inference network parameterized by ϕ and [ · , · ] can be the element-wise product or concatenation operation.
Revised: where h  denotes the inference network parameterized by  and [ · , · ] can be the element-wise product or concatenation operation.

Original: h ϕ [ v i ; z j ])
Revised: h  ([ v i ; z j ])

Original: [ k ] indicates the k th element, i.e., the logit corresponding the latent context k .
Revised: [ k ] indicates the k th element, i.e., the logit corresponding the latent context k .

Original: Instead of introducing variational parameters individually, we consider the amortization inference [25, 58], which fits a shared network to calculate each local parameter.
Revised: Instead of introducing variational parameters individually, we consider the amortization inference [25, 58], which ﬁts a shared network to calculate each local parameter.

Original: For the expectation terms in Eq.
Revised: For the expectation terms in Eq.

Original: (5), back-propagation through the discrete variable k is not di-
Revised: (5), back-propagation through the discrete variable k is not di-

Original: rectly feasible.
Revised: rectly feasible.

Original: We alleviate this by adopting the Straight-Through Gumbel-Softmax estimator [21], which provides a continuous differentiable approximation for drawing samples from a categorical distribution.
Revised: We alleviate this by adopting the Straight-Through Gumbel-Softmax estimator [21], which provides a continuous differentiable approximation for drawing samples from a categorical distribution.

Original: Specifically, for each sample, a latent cluster vector c ∈ (0 , 1) K is drawn from:
Revised: Speciﬁcally, for each sample, a latent cluster vector c 2 (0 , 1) K is drawn from:

Original: [Equation]
Revised: [Equation]

Original: where ϵ [ k ] is i.i.d drawn from the Gumbel(0 , 1) distribution and γ is a temperature.
Revised: where ✏ [ k ] is i.i.d drawn from the Gumbel(0 , 1) distribution and  is a temperature.

Original: With this reparameterization trick, we can obtain the surrogate E q ϕ ( k | v i , z j )[ ∥ v i + βg θ ( k ) − z j ∥ 22 ] ≃ E ϵ
Revised: With this reparameterization trick, we can obtain the surrogate E q  ( k | v i , z j )

Original: [ ∥ v i + βg θ ( c ) − z j ∥ 22 ] and the gradients are estimated with Monte Carlo.
Revised: [ k v i +  g ✓ ( k )  z j k 22 ] ' E ✏ [ k v i +  g ✓ ( c )  z j k 22 ] and the gradients are estimated with Monte Carlo.

Original: The expectation term over q ϕ ( k | v i ) can be similarly estimated.
Revised: The expectation term over q  ( k | v i ) can be similarly estimated.

Original: Then, { θ, ϕ, µ } in Eq.
Revised: Then, { ✓ ,  , µ } in Eq.

Original: (5) can be efficiently solved by gradient descent.
Revised: (5) can be efﬁciently solved by gradient descent.

Original: 3.4 Algorithm Optimization
Revised: 3.4 Algorithm Optimization

Original: The overall optimization involves simultaneously training (1) the encoder f θ , (2) the projector g θ , (3) the inference predictor h ϕ and (4) the prototypes µ .
Revised: The overall optimization involves simultaneously training (1) the encoder f ✓ , (2) the projector g ✓ , (3) the inference predictor h  and (4) the prototypes µ .

Original: The most canonical way to update the parameters is stochastic gradient descent.
Revised: The most canonical way to update the parameters is stochastic gradient descent.

Original: However, we observe that stochastically updating all parameters suffers from two problems: (1) The objective admits trivial solutions, e.g., outputting the same representation for all nodes in the optimization process.
Revised: However, we observe that stochastically updating all parameters suffers from two problems: (1) The objective admits trivial solutions, e.g., outputting the same representation for all nodes in the optimization process.

Original: (2) Updating prototypes without any constraints will lead to a degenerate solution, i.e., all nodes are assigned to a single cluster.
Revised: (2) Updating prototypes without any constraints will lead to a degenerate solution, i.e., all nodes are assigned to a single cluster.

Original: To address the issue of trivial solutions, inspired by the recent works [19, 13], we consider an asymmetric encoder architecture that includes online and target encoders.
Revised: To address the issue of trivial solutions, inspired by the recent works [19, 13], we consider an asymmetric encoder architecture that includes online and target encoders.

Original: Specifically, for each node pair ( v i , v j ) , the online encoder f θ produces the representation of the central node v i ; while the target encoder f ξ is used to produce the representation of its neighbor z j .
Revised: Speciﬁcally, for each node pair ( v i , v j ) , the online encoder f ✓ produces the representation of the central node v i ; while the target encoder f ⇠ is used to produce the representation of its neighbor z j .

Original: Importantly, the gradient of loss is only used to update the online encoder f θ while being blocked in the target encoder.
Revised: Importantly, the gradient of loss is only used to update the online encoder f ✓ while being blocked in the target encoder.

Original: The weights of the target encoder ξ are updated via the exponential moving average (EMA) of the online encoder θ :
Revised: The weights of the target encoder ⇠ are updated via the exponential moving average (EMA) of the online encoder ✓ :

Original: [Equation]
Revised: [Equation]

Original:  where Γ ( · ) indicates a stochastic optimizer and τ ∈ [0 , 1] is the target decay rate.
Revised: [Equation] ) where  ( · ) indicates a stochastic optimizer and ⌧ 2 [0 , 1] is the target decay rate.

Original: This update introduces an asymmetry between two encoders that prevents collapse to trivial solutions [13, 48].
Revised: This update introduces an asymmetry between two encoders that prevents collapse to trivial solutions [13, 48].

Original: To alleviate the second issue, besides the stochastic update, we also apply a global update for the prototype vectors µ = { µ i } K i =1 at the end of each training epoch to avoid a degenerate solution: 
Revised: To alleviate the second issue, besides the stochastic update, we also apply a global update for the prototype vectors µ = { µ i } K i =1 at the end of each training epoch to avoid a degenerate solution: [Equation] i k 22 , where ⇡ i ( k ) = 1 / | N ( i ) |

Original: [Equation]
Revised: [Equation]

Original: The derivation is provided in Appendix A.3.
Revised: The derivation is provided in Appendix A.3.

Original: Intuitively, π i ( k ) reflects the degree of relevance of node v i to the k th prototype.
Revised: Intuitively, ⇡ i ( k ) reﬂects the degree of relevance of node v i to the k th prototype.

Original: Instead of only updating the prototypes in a mini-batch, we also aggregate all the representations as the prototype based on the soft assignment probability at the end of the epoch.
Revised: Instead of only updating the prototypes in a mini-batch, we also aggregate all the representations as the prototype based on the soft assignment probability at the end of the epoch.

Original: After the training is finished, we only keep the online encoder f θ for the downstream task.
Revised: After the training is ﬁnished, we only keep the online encoder f ✓ for the downstream task.

Original: Our full algorithm and network are provided in Appendix B.
Revised: Our full algorithm and network are provided in Appendix B.

Original: The details about time complexity of DSSL is given in Appendix C.1, which scales linearly in the size of edges.
Revised: The details about time complexity of DSSL is given in Appendix C.1, which scales linearly in the size of edges.

Original: 3.5 Theoretical Analysis
Revised: 3.5 Theoretical Analysis

Original: In this section, we provide an analysis of the proposed framework.
Revised: In this section, we provide an analysis of the proposed framework.

Original: We first present the connection between the proposed objective and the mutual information maximization, then show that the learned representations by our objective provably enjoy a good downstream performance.
Revised: We ﬁrst present the connection between the proposed objective and the mutual information maximization, then show that the learned representations by our objective provably enjoy a good downstream performance.

Original: Due to the space limitation, all proofs of theorems and corollaries are provided in Appendix D.
Revised: Due to the space limitation, all proofs of theorems and corollaries are provided in Appendix D.

Original: We denote the random variable of the input graph as G and the downstream label as y .
Revised: We denote the random variable of the input graph as G and the downstream label as y .

Original: For clarity, we omit subscript i in what follows.
Revised: For clarity, we omit subscript i in what follows.

Original: From an information-theoretic learning perspective, a desirable way is to maximize the mutual information I ( v , y ) between the representation v and downstream label y .
Revised: From an information-theoretic learning perspective, a desirable way is to maximize the mutual information I ( v , y ) between the representation v and downstream label y .

Original: However, due to the lack of the downstream label, self-supervised learning resorts to maximizing
Revised: However, due to the lack of the downstream label, self-supervised learning resorts to maximizing

Original: I ( v , s ) where s is different designed self-supervised signal [51, 4, 56, 65, 11].
Revised: I ( v , s ) where s is different designed self-supervised signal [51, 4, 56, 65, 11].

Original: In our method, we have two self-supervised signals: the global semantic cluster information inferred by k and the local structural roles captured by the representations of the neighbors z
Revised: In our method, we have two self-supervised signals: the global semantic cluster information inferred by k and the local structural roles captured by the representations of the neighbors z

Original: = { z i | v i ∈ N ( v ) } of node v .
Revised: = { z i | v i 2 N ( v ) } of node v .

Original: Then, we can interpret our objective in Eq.
Revised: Then, we can interpret our objective in Eq.

Original: (5) from the information maximization perspective:
Revised: (5) from the information maximization perspective:

Original: Theorem 1.
Revised: Theorem 1.

Original: Optimizing local and global terms in Eq.
Revised: Optimizing local and global terms in Eq.

Original: (5) is equivalent to maximizing the mutual information between the representation v and global signal k and maximizing the conditional mutual information between v and the local signal z , conditioned on global signal k .
Revised: (5) is equivalent to maximizing the mutual information between the representation v and global signal k and maximizing the conditional mutual information between v and the local signal z , conditioned on global signal k .

Original: Formally, we have:
Revised: Formally, we have:

Original: [Equation]
Revised: [Equation]

Original: This theorem suggests that we essentially combine both local structure and global semantic information as the self-supervised signal and maximize the mutual information between the representation v and their joint distribution ( k, z ) .
Revised: This theorem suggests that we essentially combine both local structure and global semantic information as the self-supervised signal and maximize the mutual information between the representation v and their joint distribution ( k, z ) .

Original: Next, we discuss how the learned representation affects the downstream task y based on the information bottleneck principle [51, 11, 56, 4].
Revised: Next, we discuss how the learned representation affects the downstream task y based on the information bottleneck principle [51, 11, 56, 4].

Original: The rationality of self-supervised learning is that the task-relevant information lies mostly in the shared information between the input and the self-supervised signals [51, 11, 4].
Revised: The rationality of self-supervised learning is that the task-relevant information lies mostly in the shared information between the input and the self-supervised signals [51, 11, 4].

Original: Specifically, we formulate our lightweight and reasonable assumption below, which serves as a foundation for our analysis.
Revised: Speciﬁcally, we formulate our lightweight and reasonable assumption below, which serves as a foundation for our analysis.

Original: Assumption 1.
Revised: Assumption 1.

Original: Nodes with similar labels should have similar "local structural roles" and "global semantic clusters.
Revised: Nodes with similar labels should have similar "local structural roles" and "global semantic clusters.

Original: " In this work, we equate "local structure" with the 1-hop neighborhood and "global semantic" with the clustering membership of a node.
Revised: " In this work, we equate "local structure" with the 1-hop neighborhood and "global semantic" with the clustering membership of a node.

Original: Formally, we have the task-relevant information y left in G except that in joint self-supervised signal ( z , k ) is relatively small: I ( G ; y | z , k ) ≤ ϵ .
Revised: Formally, we have the task-relevant information y left in G except that in joint self-supervised signal ( z , k ) is relatively small: I ( G ; y | z , k )  ✏ .

Original: Intuitively, this assumption indicates that most of the task-relevant information in the graph is contained in the self-supervised signal (local neighborhood and global semantic patterns).
Revised: Intuitively, this assumption indicates that most of the task-relevant information in the graph is contained in the self-supervised signal (local neighborhood and global semantic patterns).

Original: Based on this assumption, we have the following theorem, which reveals why the downstream tasks can benefit from the learned representations learned by our objective function.
Revised: Based on this assumption, we have the following theorem, which reveals why the downstream tasks can beneﬁt from the learned representations learned by our objective function.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let v joint =arg max v
Revised: Let v joint =

Original: [Equation]
Revised: [Equation]

Revised: v

Original: [Equation]
Revised: [Equation]

Original: ; z , k ) , vlocal =
Revised: ; z , k ) , v

Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: ; k ) , and v global = arg max v I ( v ; z ) .
Revised: k ) , and v global = arg max v I ( v ; z ) .

Original: Formally, we have the following inequalities about the task-relevant information:
Revised: Formally, we have the following inequalities about the task-relevant information:

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 shows that the gap of task-relevant information between supervised representation v sup = arg max v I ( v , y ) and self-supervised representation v joint is ϵ .
Revised: Theorem 2 shows that the gap of task-relevant information between supervised representation v sup = arg max v I ( v , y ) and self-supervised representation v joint is ✏ .

Original: Thus, we can guarantee a good downstream performance as long as the Assumption 1 is satisfied.
Revised: Thus, we can guarantee a good downstream performance as long as the Assumption 1 is satisﬁed.

Original: It is noteworthy that jointly utilizing local structure and global semantics as the self-supervised signal is expected to contain more task information.
Revised: It is noteworthy that jointly utilizing local structure and global semantics as the self-supervised signal is expected to contain more task information.

Original: As further enlightenment, we can relate Eq.
Revised: As further enlightenment, we can relate Eq.

Original: (36) with the Bayes error rate [51]:
Revised: (36) with the Bayes error rate [51]:

Original: Corollary 1.
Revised: Corollary 1.

Original: Suppose that downstream label y is a M-categorical random variable.
Revised: Suppose that downstream label y is a M-categorical random variable.

Original: Then we have the upper bound for the downstream Bayes errors P e v=
Revised: Then we have the upper bound for the downstream Bayes errors P e v

Original: E v (cid:2) 1 − max y ∈ y P (ˆ y =
Revised: [Equation]

Original: y | v ) (cid:3) on learned representation v , where ˆ y is the estimation for label from our downstream classifier:
Revised: on learned representation v , where ˆ y is the estimation for label from our downstream classiﬁer:

Original: [Equation]
Revised: [Equation]

Original: where Th( x ) = min { max { x, 0 } , 1 − 1 / | M |} is a thresholded operation [51].
Revised: where Th( x ) = min { max { x, 0 } , 1  1 / | M |} is a thresholded operation [51].

Original: Similarly, we can obtain the error upper bound of other representations v local and v global : RHS v local and RHS v global .
Revised: Similarly, we can obtain the error upper bound of other representations v local and v global : RHS v local and RHS v global .

Original: Then, we have inequalities on error upper bounds : RHS v joint ≤ min(RHS v local , RHS v global ) .
Revised: Then, we have inequalities on error upper bounds : RHS v joint  min(RHS v local , RHS v global ) .

Original: This corollary says that our self-supervised signal has a tighter upper bound on the downstream Bayes error.
Revised: This corollary says that our self-supervised signal has a tighter upper bound on the downstream Bayes error.

Original: Thus, we can expect that the representation learned by our objective function, which utilizes both local structure and global semantic information, has superior performance on downstream tasks.
Revised: Thus, we can expect that the representation learned by our objective function, which utilizes both local structure and global semantic information, has superior performance on downstream tasks.

Original:  Experiments
Revised:  Experiments

Original: In this section, we empirically evaluate the proposed self-supervised learning method on several real-world graph datasets and analyze its behavior on graphs to gain further insights.
Revised: In this section, we empirically evaluate the proposed self-supervised learning method on several real-world graph datasets and analyze its behavior on graphs to gain further insights.

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Datasets.
Revised: Datasets.

Original: We perform experiments on widely-used homophilic graph datasets: Cora, Citeseer, and
Revised: We perform experiments on widely-used homophilic graph datasets: Cora, Citeseer, and

Original: Pubmed [42], as well as non-homophilic datasets: Texas, Cornell, Wisconsin [37], Penn94 and Twitch.
Revised: Pubmed [42], as well as non-homophilic datasets: Texas, Cornell, Wisconsin [37], Penn94 and Twitch.

Original: Penn94 and Twitch are two relatively large non-homophilous graph datasets proposed by [41, 29].
Revised: Penn94 and Twitch are two relatively large non-homophilous graph datasets proposed by [41, 29].

Original: We provide the detailed descriptions, statistics, and homophily measures of datasets in Appendix E.1.
Revised: We provide the detailed descriptions, statistics, and homophily measures of datasets in Appendix E.1.

Original: Baselines.
Revised: Baselines.

Original: To evaluate the effectiveness of DSSL, we consider the following representative unsupervised and self-supervised learning methods for the node representation task, including Deepwalk [38],
Revised: To evaluate the effectiveness of DSSL, we consider the following representative unsupervised and self-supervised learning methods for the node representation task, including Deepwalk [38],

Original: LINE [46], Struc2vec [40], GAE [26], VGAE [26], DGI [53], GraphCL [64], MVGRL [18] and
Revised: LINE [46], Struc2vec [40], GAE [26], VGAE [26], DGI [53], GraphCL [64], MVGRL [18] and

Original: BGRL [48].
Revised: BGRL [48].

Original: The detailed description of baselines and implementations are given in Appendix E.2.
Revised: The detailed description of baselines and implementations are given in Appendix E.2.

Original: Evaluation Protocol.
Revised: Evaluation Protocol.

Original: We consider two types of downstream tasks: node classification and node clustering.
Revised: We consider two types of downstream tasks: node classiﬁcation and node clustering.

Original: For node classification, we follow the standard linear-evaluation protocol on graphs [53, 48], where a linear classifier is trained on top of the frozen representation, and test accuracy (ACC) is used as a proxy for representation quality.
Revised: For node classiﬁcation, we follow the standard linear-evaluation protocol on graphs [53, 48], where a linear classiﬁer is trained on top of the frozen representation, and test accuracy (ACC) is used as a proxy for representation quality.

Original: For all datasets, we adopt the similar random split with a train/validation/test split ratio of 60% / 20% / 20% for the training of downstream linear classifier following [37, 29].
Revised: For all datasets, we adopt the similar random split with a train/validation/test split ratio of 60% / 20% / 20% for the training of downstream linear classiﬁer following [37, 29].

Original: For the node clustering task, we perform K -means clustering on the obtained representations and set the number of clusters to the number of classes.
Revised: For the node clustering task, we perform K -means clustering on the obtained representations and set the number of clusters to the number of classes.

Original: We utilize the normalized mutual information (NMI) [54] as the evaluation metric for clustering.
Revised: We utilize the normalized mutual information (NMI) [54] as the evaluation metric for clustering.

Original: Setup.
Revised: Setup.

Original: For all self-supervised methods on graph neural networks, we consider a two-layer GCN [27] as the encoder unless otherwise stated, and randomly initialize parameters.
Revised: For all self-supervised methods on graph neural networks, we consider a two-layer GCN [27] as the encoder unless otherwise stated, and randomly initialize parameters.

Original: We run experiments with 10 random splits and report the average performance.
Revised: We run experiments with 10 random splits and report the average performance.

Original: We select the best configuration of hyperparameters based on accuracy on the validation.
Revised: We select the best conﬁguration of hyperparameters based on accuracy on the validation.

Original: The detailed settings are given in Appendix E.2.
Revised: The detailed settings are given in Appendix E.2.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4.2 Overall Performance Comparison
Revised: 4.2 Overall Performance Comparison

Original: In this section, we conduct experiments on real-world graphs compared to state-of-the-art methods.
Revised: In this section, we conduct experiments on real-world graphs compared to state-of-the-art methods.

Original: Table 1 reports the average classification accuracy with the standard deviation on node classification after ten runs.
Revised: Table 1 reports the average classiﬁcation accuracy with the standard deviation on node classiﬁcation after ten runs.

Original: Since node clustering results have a similar tendency, we provide them in Appendix F.1.
Revised: Since node clustering results have a similar tendency, we provide them in Appendix F.1.

Original: From Table 1, we have the following observations: (1) Generally, our DSSL achieves the best performance on both node clustering and classification tasks over the best baseline, demonstrating the effectiveness of DSSL on both homophilous and non-homophilous graphs and the robustness to different downstream tasks and graphs.
Revised: From Table 1, we have the following observations: (1) Generally, our DSSL achieves the best performance on both node clustering and classiﬁcation tasks over the best baseline, demonstrating the effectiveness of DSSL on both homophilous and non-homophilous graphs and the robustness to different downstream tasks and graphs.

Original: Especially, DSSL achieves a relative improvement of over 6% and 4% on Texas and Squirrel compared to the best baselines (2) Our DSSL cannot achieve the best performance on Cora.
Revised: Especially, DSSL achieves a relative improvement of over 6% and 4% on Texas and Squirrel compared to the best baselines (2) Our DSSL cannot achieve the best performance on Cora.

Original: This is reasonable since Cora is highly homophilous (see Appendix E.1), and the core design of augmentation in contrastive learning methods such as MVGRL and GraphCL enables them to be effective on homophilous graphs but failed on low-homophily settings.
Revised: This is reasonable since Cora is highly homophilous (see Appendix E.1), and the core design of augmentation in contrastive learning methods such as MVGRL and GraphCL enables them to be effective on homophilous graphs but failed on low-homophily settings.

Original: This supports our motivation in the introduction that it is relatively difficult to design effective graph augmentation on non-homophilous graphs due to their heterogeneous and diverse patterns.
Revised: This supports our motivation in the introduction that it is relatively difﬁcult to design effective graph augmentation on non-homophilous graphs due to their heterogeneous and diverse patterns.

Original: (3) When compared with GAE, VGAE, and DGI, DSSL consistently and significantly outperforms them.
Revised: (3) When compared with GAE, VGAE, and DGI, DSSL consistently and signiﬁcantly outperforms them.

Original: We deem that this improvement is mainly from the joint local semantic shift and global semantic clustering in DSSL, which is not included in existing methods.
Revised: We deem that this improvement is mainly from the joint local semantic shift and global semantic clustering in DSSL, which is not included in existing methods.

Original: [Figure]
Revised: [Figure]

Original: 4.3 Ablation Study and Parameter Analysis
Revised: 4.3 Ablation Study and Parameter Analysis

Original: In this section, we conduct an ablation study and investigate the sensitivity of hyper-parameters.
Revised: In this section, we conduct an ablation study and investigate the sensitivity of hyper-parameters.

Original: Ablation Study.
Revised: Ablation Study.

Original: We consider the following ablations: (A1)
Revised: We consider the following ablations: (A1)

Original: We remove the key component of DSSL: the local reconstruction loss ( w/o L local ).
Revised: We remove the key component of DSSL: the local reconstruction loss ( w/o L local ).

Original: (A2)
Revised: (A2)

Original: We remove global clustering loss to see if L local alone is still effective ( w/o L global ).
Revised: We remove global clustering loss to see if L local alone is still effective ( w/o L global ).

Original: (A3)
Revised: (A3)

Original: We remove the entropy term in Eq.
Revised: We remove the entropy term in Eq.

Original: (5) ( w/o entropy ).
Revised: (5) ( w/o entropy ).

Original: (A4)
Revised: (A4)

Original: We remove the semantic shift term g θ ( k ) in L local ( w/o semantic shift ).
Revised: We remove the semantic shift term g ✓ ( k ) in L local ( w/o semantic shift ).

Original: (A5)
Revised: (A5)

Original: We set posterior q ϕ ( k | v i , z j ) = 1 /K as uniform distribution for each link ( w/ uniform posterior ).
Revised: We set posterior q  ( k | v i , z j ) = 1 /K as uniform distribution for each link ( w/ uniform posterior ).

Original: We show the ablation study results in Table 2.
Revised: We show the ablation study results in Table 2.

Original: L global alone does not provide much discriminative information, and it does not perform very well.
Revised: L global alone does not provide much discriminative information, and it does not perform very well.

Original: L local as a key component of DSSL alone produces better results
Revised: L local as a key component of DSSL alone produces better results

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: visualization and case study results (best viewed on a computer screen and note that the latent distribution of each link need to be zoomed in to be better visible).
Revised: [Table] [Figure] [Table] [Figure]

Original: than the DGI baseline.
Revised: than the DGI baseline.

Original: We can also observe that considering semantic shift and personalized posterior can further improve the performance a lot which demonstrates our key motivations.
Revised: We can also observe that considering semantic shift and personalized posterior can further improve the performance a lot which demonstrates our key motivations.

Original: The full model (last row) achieves the best performance, which illustrates that different components in the proposed
Revised: The full model (last row) achieves the best performance, which illustrates that different components in the proposed

Original: DSSL are complementary to each other.
Revised: DSSL are complementary to each other.

Original: [Table]
Revised: [Table]

Original: Effect of Global Update on Prototypes .
Revised: Effect of Global Update on Prototypes .

Original: We conduct ablation studies to gain insights into the global updating of prototypes.
Revised: We conduct ablation studies to gain insights into the global updating of prototypes.

Original: As shown in Figure 3, we can observe that without Eq.
Revised: As shown in Figure 3, we can observe that without Eq.

Original: (9), the performance is not very good, and we are stuck in bad local optima.
Revised: (9), the performance is not very good, and we are stuck in bad local optima.

Original: As discussed above, a possible reason is that we will experience strong degeneracy if we only update prototype vectors with mini-batch training.
Revised: As discussed above, a possible reason is that we will experience strong degeneracy if we only update prototype vectors with mini-batch training.

Original: This figure also demonstrates that DSSL can converge within a few hundred steps, which is efficient.
Revised: This ﬁgure also demonstrates that DSSL can converge within a few hundred steps, which is efﬁcient.

Original: Effect of Asymmetric Architecture .
Revised: Effect of Asymmetric Architecture .

Original: We then explore the effect of target decay rate τ on the performance.
Revised: We then explore the effect of target decay rate ⌧ on the performance.

Original: Figureshows the learning curves of DSSL on Squirrel and Twitch.
Revised: Figureshows the learning curves of DSSL on Squirrel and Twitch.

Original: We can observe that the best result is achieved at τ = 0 . 9 .
Revised: We can observe that the best result is achieved at ⌧ = 0 . 9 .

Original: When τ = 1 . 0 , i.e., the target network is never updated,
Revised: When ⌧ = 1 . 0 , i.e., the target network is never updated,

Original: DSSL obtains a competitive result but is lower than τ = 0 .
Revised: DSSL obtains a competitive result but is lower than ⌧ = 0 .

Original: This confirms that slowly updating the target network is crucial in obtaining superior performance.
Revised: This conﬁrms that slowly updating the target network is crucial in obtaining superior performance.

Original: At the other extreme value τ = 0 , the target network is the same as the online network, and DSSL demonstrates a degenerated performance.
Revised: At the other extreme value ⌧ = 0 , the target network is the same as the online network, and DSSL demonstrates a degenerated performance.

Original: Hyper-parameters Analysis.
Revised: Hyper-parameters Analysis.

Original: We investigate the hyper-parameters most essential to our framework design, i.e., the standard deviation σ 1 and σ 2 , the temperature of the Gumbel Softmax γ , and the total number of factors K .
Revised: We investigate the hyper-parameters most essential to our framework design, i.e., the standard deviation  1 and  2 , the temperature of the Gumbel Softmax  , and the total number of factors K .

Original: The corresponding results are shown in Figure 4.
Revised: The corresponding results are shown in Figure 4.

Original: σ 21 resembles the temperature scaling in Eq.
Revised:  21 resembles the temperature scaling in Eq.

Original: We observe σ 21 is better to be selected from 0 .
Revised: We observe  21 is better to be selected from 0 .

Original: 6 to 1 . 0 , and a too small (e.g., 0 . 2 ) value may degenerate the performance in all three datasets.
Revised: 6 to 1 . 0 , and a too small (e.g., 0 . 2 ) value may degenerate the performance in all three datasets.

Original: σ 22 balances the local and global loss in Eq.
Revised:  22 balances the local and global loss in Eq.

Original: We can find that having large values of σ 22 does not improve the performance, as the local loss plays an essential role in the proposed model.
Revised: We can ﬁnd that having large values of  22 does not improve the performance, as the local loss plays an essential role in the proposed model.

Original: Further, we observe that DSSL is not very sensitive to the Gumbel softmax temperature λ , while a moderate hardness of the Softmax gives the best results.
Revised: Further, we observe that DSSL is not very sensitive to the Gumbel softmax temperature  , while a moderate hardness of the Softmax gives the best results.

Original: We also find that as K increases from 2 to 8 , the performance of DGCL improves, which suggests the importance of decoupling latent factors.
Revised: We also ﬁnd that as K increases from 2 to 8 , the performance of DGCL improves, which suggests the importance of decoupling latent factors.

Original: However, training with large K will lead to a performance slightly drop.
Revised: However, training with large K will lead to a performance slightly drop.

Original: We provide results on other datasets in Appendix F.3.
Revised: We provide results on other datasets in Appendix F.3.

Original: Encoders Analysis.
Revised: Encoders Analysis.

Original: To further evaluate the effectiveness of the proposed DSSL, we consider the case where the self-supervised learning methods are implemented using another GNN encoder.
Revised: To further evaluate the effectiveness of the proposed DSSL, we consider the case where the self-supervised learning methods are implemented using another GNN encoder.

Original: Tableshows the results of the selected baseline with GAT as the encoder.
Revised: Tableshows the results of the selected baseline with GAT as the encoder.

Original: As shown in Table 3, our
Revised: As shown in Table 3, our

Original: DSSL performs better than all the compared methods in most cases, which once again proves the effectiveness of DSSL, and also shows that DSSL is broadly applicable to various GNN encoders.
Revised: DSSL performs better than all the compared methods in most cases, which once again proves the effectiveness of DSSL, and also shows that DSSL is broadly applicable to various GNN encoders.

Original: [Figure]
Revised: [Figure]

Original: 4.4 Visualization and Case Study
Revised: 4.4 Visualization and Case Study

Original: Local neighborhood Patterns .
Revised: Local neighborhood Patterns .

Original: We provide the visualization and case study results to understand how DSSL uncovers the latent patterns in local neighborhoods.
Revised: We provide the visualization and case study results to understand how DSSL uncovers the latent patterns in local neighborhoods.

Original: More results can be found in
Revised: More results can be found in

Original: Appendix F.4.
Revised: Appendix F.4.

Original: In Figure 5 (a), we calculate the cross-class neighborhood similarity (see Appendix E. for definition), which serves as the ground truth.
Revised: In Figure 5 (a), we calculate the cross-class neighborhood similarity (see Appendix E. for deﬁnition), which serves as the ground truth.

Original: If nodes with the same label share the same neighborhood distributions, the intra-class similarity should be high.
Revised: If nodes with the same label share the same neighborhood distributions, the intra-class similarity should be high.

Original: In Figure 5 (b), we calculate average posterior distribution q ϕ ( k | v i ) over all nodes on each class and provide cosine similarity of them.
Revised: In Figure 5 (b), we calculate average posterior distribution q  ( k | v i ) over all nodes on each class and provide cosine similarity of them.

Original: We can observe that our learned distribution shares a similar pattern with the cross-class neighborhood similarity, which shows that learned latent factors can capture the latent semantic information related to neighborhood patterns.
Revised: We can observe that our learned distribution shares a similar pattern with the cross-class neighborhood similarity, which shows that learned latent factors can capture the latent semantic information related to neighborhood patterns.

Original: In Figures 5 (c) and (d), we plot the subgraph of a randomly selected node, and the distribution q ϕ ( k | v i , z j ) .
Revised: In Figures 5 (c) and (d), we plot the subgraph of a randomly selected node, and the distribution q  ( k | v i , z j ) .

Original: We use different colors to indicate different labels.
Revised: We use different colors to indicate different labels.

Original: We find that similar neighbors over class generally have a similar distribution, while different types of links exhibit different latent distributions.
Revised: We ﬁnd that similar neighbors over class generally have a similar distribution, while different types of links exhibit different latent distributions.

Original: This observation matches our motivation that
Revised: This observation matches our motivation that

Original: DSSL can decouple the diverse semantics in the local neighborhoods.
Revised: DSSL can decouple the diverse semantics in the local neighborhoods.

Original: Global Semantic Patterns.
Revised: Global Semantic Patterns.

Original: We investigate how our learned semantic clusters perform on the longrange nodes.
Revised: We investigate how our learned semantic clusters perform on the longrange nodes.

Original: To show this, we study the learned cluster distribution q ϕ ( k | v i ) for each node.
Revised: To show this, we study the learned cluster distribution q  ( k | v i ) for each node.

Original: To facilitate visualization, we randomly sample a sub-graph that contains high-hop neighborhoods from each dataset and use different colors to indicate different node labels.
Revised: To facilitate visualization, we randomly sample a sub-graph that contains high-hop neighborhoods from each dataset and use different colors to indicate different node labels.

Original: We then calculate the pair-wise similarity of the posteriors q ϕ ( k | v i ) between nodes as shown in Figure 6.
Revised: We then calculate the pair-wise similarity of the posteriors q  ( k | v i ) between nodes as shown in Figure 6.

Original: We can observe that our learned semantic clusters exhibit similar patterns for the nodes in the same class.
Revised: We can observe that our learned semantic clusters exhibit similar patterns for the nodes in the same class.

Original: Some nodes exhibit similar semantic clusters regardless of the distance between them.
Revised: Some nodes exhibit similar semantic clusters regardless of the distance between them.

Original: For instance, node ID 6 exhibits a very similar posterior distribution to the same class node ID 19 in Pubmed, and node ID 3 exhibits similar posterior distribution to node ID 12 in Cornell, despite these nodes being ID 4 hops away.
Revised: For instance, node ID 6 exhibits a very similar posterior distribution to the same class node ID 19 in Pubmed, and node ID 3 exhibits similar posterior distribution to node ID 12 in Cornell, despite these nodes being ID 4 hops away.

Original: Since our DSSL seeks to pull the node representation to the inferred semantic clusters, we can learn the global node-node relationships in representations instead of favoring nearby nodes.
Revised: Since our DSSL seeks to pull the node representation to the inferred semantic clusters, we can learn the global node-node relationships in representations instead of favoring nearby nodes.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we study the problem of conducting self-supervised learning on non-homophilous graphs data.
Revised: In this paper, we study the problem of conducting self-supervised learning on non-homophilous graphs data.

Original: We present a novel decoupled self-supervised learning (DSSL) framework to decouple the diverse neighborhood context of a node in an unsupervised manner.
Revised: We present a novel decoupled self-supervised learning (DSSL) framework to decouple the diverse neighborhood context of a node in an unsupervised manner.

Original: Specifically, DSSL imitates the generative process of neighbors and explicitly models unobserved factors by latent variables.
Revised: Speciﬁcally, DSSL imitates the generative process of neighbors and explicitly models unobserved factors by latent variables.

Original: We show that DSSL can simultaneously capture the global clustering information and the local structure roles with the semantic shift.
Revised: We show that DSSL can simultaneously capture the global clustering information and the local structure roles with the semantic shift.

Original: We theoretically show that DSSL enjoys a good downstream performance.
Revised: We theoretically show that DSSL enjoys a good downstream performance.

Original: Extensive experiments on several graph datasets validated the superiority of DSSL and showed that DSSL could learn meaningful class-discriminative representations.
Revised: Extensive experiments on several graph datasets validated the superiority of DSSL and showed that DSSL could learn meaningful class-discriminative representations.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work was partially supported by the National Science Foundation (NSF) under grants number
Revised: This work was partially supported by the National Science Foundation (NSF) under grants number

Original: IIS-1909702 and IIS-1955851, Army Research Office (ARO) under grant number W911NF-211-0198 and Department of Homeland Security under grant number E205949D.
Revised: IIS-1909702 and IIS-1955851, Army Research Ofﬁce (ARO) under grant number W911NF-211-0198 and Department of Homeland Security under grant number E205949D.

Original: The findings and conclusions in this paper do not necessarily reflect the view of the funding agency.A The Omitted Derivations A.1 The Evidence Lower-Bound
Revised: The ﬁndings and conclusions in this paper do not necessarily reﬂect the view of the funding agency.

Original: In this section, we provide the details of the lower-bound in Eq.

Original: By introducing the approximated posterior q ϕ ( k | v i , z j ) , the likelihood L DSSL ( θ, µ ) in Eq.

Original: (1) becomes:

Original: [Equation]

Original: where the third step uses Jensen’s inequality

Original: [Equation]

Original: A.2 The Optimization Loss

Original: In this section

Original: [Equation]

Original: we derive the optimization loss for learning parameters in Eq.

Original: Introducing the posterior in Eq.

Original: (3) and probabilities specification p θ ( z j | v i , k ) =

Original: N ( z j ; µ z j , Σ j ) to the negative of the evidence lower-bounded in Eq.

Original: (4), we have:

Original: [Equation]

Original: Since we consider isotropic Gaussian with Σ j = I σ 22 , we have:

Original: [Equation]

Original: Similarly, for N (cid:0) v i ; µ k , I σ 21 (cid:1) , we have:

Original: [Equation]

Original: where the last equivalence holds because we apply L2 normalization to v i and µ k , respectively.

Original: Substituting the corresponding terms in Eq.

Original: (14) with Eq.

Original: (15) and Eq.

Original: (17), and removing the constant terms that are irrelevant to model parameters { θ, ϕ, µ } , we have:

Original: [Equation]

Original: where q ϕ ( k | v i ) = 1 / | N ( i ) | (cid:80) j ∈ N ( i ) q ϕ ( k | v i , z j ) .

Original: Multiplying the constant 2 σ 22 for all terms and absorbing 2 σ 22 to the hyper-parameter σ 22 , we obtain the loss function in Eq.

Original: A.3 Derivation of Global Update

Original: In this section, we derive the global analytical update of prototypes.

Original: Specifically, we have:

Original: [Equation]

Original: In the second line, following [57], we treat the normalizing term as the constant.

Original: If we apply L normalization to µ k : ∥ µ k ∥ = 1 , we need to solve the Lagrangian of the objective function:

Original: [Equation]

Original: Take the gradient over µ k with respect to L ( µ k , β ) and set it to zero, we have:

Original: [Equation]

Original: where π i ( k ) =

Original: q ϕ ( k | v i ) .

Original: By taking the gradient with respect to multiplier β and setting it to zero, we have µ ⊤ k µ k = 1 .

Original: Combining µ ⊤ k µ k

Original: = 1 and Eq.

Original: (20), we have:

Original: [Equation]

Original: Further combining the above equation with Eq.

Original: (20), we obtain the analytical global update:

Original: [Equation]

Original: , where π

Original: [Equation]

Original: Input:

Original: Initialize

Original: Randomly select do

Original: Randomly sample end

Original: L ← Eq

Original: ← [ θ, ϕ, µ ] − Γ (cid:0) ∇ θ,ϕ, µ L (cid:1)

Original: Update ξ with

Original: Update

Original: B The Training Algorithm and Network Architecture B.1 The Training Algorithm of DSSL

Original: The overall training algorithm is shown in Algorithm 1.

Original: Concretely, for each iteration of DSSL, we randomly sample the nodes and their neighbors.

Original: Then, we calculate the loss function with Gumbel

Original: [Table]

Original: [Figure]

Original: softmax approximation.

Original: All online trainable parameters are updated with stochastic gradient descent (SGD), while those marked with ξ are the target network counterparts to be updated with exponential moving average.

Original: We apply a global update for the prototypes µ at the end of each training epoch.

Original: B.2 The Network Architecture of DSSL

Original: The DSSL framework is illustrated in Figure 7.

Original: DSSL proposes a self-supervised scheme that learns to encode the node using an online encoder along with a set of prototypes, inference projector g θ , and predictor h ϕ .

Original: A target encoder is used to produce presentations of neighbors and updated by exponential moving average.

Original: We stop gradients on the target encoder coming from the local loss to introduce asymmetry between the online and target encoders.

Original: C Efficiency Analysis C.1 Time Complexity Analysis

Original: We give the detailed time complexity per iteration of DSSL and compare it with representative methods on node representation learning such as GCA [69] and BGRL [48].

Original: For self-supervised learning, the time complexity mainly depends on two parts: the encoder and loss calculation.

Original: Here, we assume the backward pass to be approximately as costly as a forward pass following [48].

Original: Since our DSSL is agnostic to GNN encoders, we consider L -layer GCN as an example.

Original: A Llayer GCN [27] with d hidden dimensions has O (cid:0) dL |E| + Nd 2 L (cid:1) complexity where E is the number of edges, since it costs O ( d |E| ) to propagate message in each layer, and O (cid:0) Nd 2 (cid:1) to multiply by the weight matrix in each layer.

Original: Our DSSL additionally has the inference predictor and projector components in the forward pass, which costs O ( C predictor N + C projector N ) where C · are constants depending on architecture of the different components.

Original: The complexity of DSSL on loss calculation is O ( KNd ) + O ( |E| d ) where O ( KNd ) and O ( |E| d ) are for global and lo- cal losses, respectively.

Original: Thus the total time complexity per iteration for DSSL with GCN is

Original: O (cid:0) dL |E| + Nd 2 L + C predictor N + C projector N + KNd + |E| d (cid:1) .

Original: With the similar notations, we can express the time complexity of GCA as O (cid:0) dL |E| + Nd 2 L + C projector N + N 2 d (cid:1) and that of BGRL as O (cid:0) dL |E| + Nd 2 L + C predictor N + Nd (cid:1) .

Original: Typically, the architectures for predictor and projector are one-layer MLP for all methods.

Original: Thus the C predictor and C projector in different methods should be similar.

Original: Based on the analysis above, we can find that the time complexity of GCA scales quadratically in the size of the nodes, and both BGRL and our DSSL linearly increase with the number of nodes N when E is proportional to N .

Original: Thus the overall time complexity of our DSSL is at the same level as

Original: BGRL [48], which is a large-scale self-supervised learning method proposed recently.

Original: D Theoretical Analysis D.1 Proof of Theorem

Original: To prove Theorem 1, we first present three Lemmas:

Original: Lemma D.1.

Original: For any random variables X , Y , Z and V , we have the following relations:

Original: [Equation]

Original: Proof.

Original: [Equation]

Original: which completes the proof of this Lemma.

Original: Lemma D.2.

Original: For any random variables X and Z , the mutual information I ( X ; Z ) has the following variational lower bound:

Original: [Equation]

Original: where p θ ( Z | X ) is an introduced variational discriminator with θ representing the parameters.

Original: Proof.

Original: [Equation]

Original: Plugging the inequality above into Eq.

Original: (25) completes the proof.

Original: Lemma D.3.

Original: For any random variables X and Z , the mutual information I ( X, Z ) has the following contrastive lower bound:

Original: [Equation]

Original: where x and z are the sample instances, f θ ∈ R is a function parameterized by θ (e.g., a dot product between encoded representations of x and z ), and ˜ B is a set of samples drawn from a proposal distribution q ( ˜ B ) .

Original: The set ˜ B contains the one positive sample and | ˜ B| − 1 negative samples.

Original: Proof.

Original: The proof of this contrastive estimation bound can be founded in [39].

Original: Now, we are ready to prove our Theorem 1.

Original: We first restate Theorem 1:

Original: Theorem 1 .

Original: Optimizing local and global terms in Eq.

Original: (5) is equivalent to maximizing the mutual information between the representation v and global signal k and maximizing the conditional mutual information between v and the local signal z , conditioned on global signal k .

Original: Formally, we have:

Original: [Equation]

Original: Proof.

Original: According to Lemma D.1, for any v j ∈ N ( v ) we have:

Original: [Equation]

Original: Given the above, we further have the following:

Original: [Equation]

Original: For I ( v ; z j | k ) , we can relate it to our local loss L local in the main body of the paper:

Original: [Equation]

Original: where we use Lemma D.2 in the second line, and v and z

Original: j are the representations of node v and node z j through deterministic encoder f with input G =

Original: { X , A } .

Original: Note that the target encoder ξ is updated via the exponential moving average of the online encoder θ .

Original: Combining Eqs.

Original: (31) and (30), we have:

Original: [Equation]

Original: Our global loss is further related to contrastive estimation.

Original: Using Lemma D.3 on I ( v ; k ) , we have:

Original: [Equation]

Original: When ˜ B always includes all possible values of the latent variable k (i.e., | ˜ B| = K ) and they are uniformly distributed, and setting f θ ( v , k )

Original: = v ⊤ · µ k /σ 21 , we have:

Original: [Equation]

Original: Given the above, we have:

Original: [Equation]

Original: Combining Eqs.

Original: (31) and (35), and omitting the entropy term in Eq.

Original: (5) completes the proof.

Original: D.2 Proof of Theorem

Original: To prove Theorem 2, we first present the following Lemmas.

Original: Lemma D.4.

Original: For a representation v that is obtained with a deterministic GNN encoder f θ of input graph G with enough capacity, we have the data processing Markov chain: ( k, z ) ↔ y ↔ G → v .

Original: Proof.

Original: Since v = f θ ( G ) is a deterministic function of input graph G , we have the following conditional independence: ( k, z ) ⊥ v |G and y ⊥ v |G [11], which leads to the data processing Markov chain ( k, z ) ↔ y ↔ G → v .

Original: Thus, the proof is completed.

Original: Lemma D.5.

Original: By optimizing local and global terms in Eq.

Original: (5) , v joint is both minimal and sufficient: v joint is sufficient v joint = arg max v I ( v ; k, z ) ; v joint is minimal v joint = arg min v H ( v | k, z ) .

Original: Proof.

Original: We have I ( v ; z | k )

Original: =

Original: H ( v | k ) − H ( v | k, z ) , in which the entropy H ( v | k ) is a constant since p ( v | k ) is a Gaussian distribution.

Original: Thus, maximizing I ( v ; z | k ) is equivalent to minimize H ( v | k, z ) .

Original: Combining this with Theorem 1, one can directly conclude that v joint is sufficient and minimal.

Original: Next, we restate Theorem 2 and provide a complete proof with the help of the Lemmas above.

Original: Theorem 2 .

Original: Let v joint = arg max v I ( v ; z , k ) , v local = arg max v I ( v ; k ) , and v global = arg max v I ( v ; z ) .

Original: Formally, we have the following inequalities about the task-relevant information:

Original: [Equation]

Original: [Equation]

Original: Proof.

Original: Based on Lemma D.4, we have the following data processing inequality [49]:

Original: [Equation]

Original: Since v joint = arg max v I ( z , k ; v ) , we have: I ( k, z ; v joint ) =

Original: I ( k, z ; G ) and I ( k, z ; v joint ; y ) =

Original: I ( k, z ; G ; y ) .

Original: In addition, since v joint is minimal, we also have, I ( v joint ; y | k, z ) ≤ H ( v joint | k, z ) = 0 .

Original: Give the above, we have the following equality:

Original: [Equation]

Original: Thus, the mutual information gap between self-supervised representation v joint and supervised representation v sup is I ( G ; y | k, z ) .

Original: Based on the property of mutual information, we further have:

Original: [Equation]

Original: [Equation]

Original: Similarly, we have I ( G ; y | z ) ≥ I ( G ; y | k, z ) .

Original: Combining Eqs.

Original: (38) and (39), we have the following inequalities about the task-relevant information based on our Assumption 1:

Original: [Equation]

Original: y ) , I ( v global ; y )) ≥

Original: I ( G ; y ) − ϵ, (40) which completes the proof

Original: D.3 Proof of Corollary

Original: Restate Corollary 1:

Original: Corollary 1.

Original: Suppose that downstream label y is a M-categorical random variable.

Original: Then we have the upper bound for the downstream Bayes errors P e v =

Original: E v (cid:2) 1 − max y ∈ y P (ˆ y =

Original: y | v ) (cid:3) on learned representation v , where ˆ y is the estimation for label from our downstream classifier:

Original: [Equation]

Original: where Th( x ) = min { max { x, 0 } , 1 − 1 / | M |} is a thresholded operation [51].

Original: Similarly, we can obtain the error upper bound of other representations v local and v global : RHS v local and RHS v global .

Original: Then, we have inequalities on error upper bounds : RHS v joint ≤ min(RHS v local , RHS v global ) .

Original: Proof.

Original: To prove this Corollary, we use the following inequalities [49, 51]:

Original: [Equation]

Original: For H ( y | v joint ) and H ( y | v sup ) , we have the following relations:

Original: [Equation]

Original: [Equation]

Original: where we use Eq.

Original: (38) in the second equality.

Original: Combining Eqs.

Original: (42), (43) and (44), we have:

Original: [Equation]

Original: Further using that I ( G ; y | z , k ) ≤ I ( G ; y | z ) and I ( G ; y | z , k ) ≤ I ( G ; y | k ) in Eq.

Original: (39), we have

Original: [Equation]

Original: which completes the proof.

Original: [Table]

Original: E Experimental Details E.1 Datasets Description and Statistics

Original: In our experiments, we use the following real-world datasets.

Original: Cora, Citeseerm, and Pubmed are citation and high-homophily graphs, which are among the most widely used benchmarks for semi-supervised node classification [27, 16, 52].

Original: In these graphs, nodes are documents, and edges are citations.

Original: Each node is assigned a class label based on the research field.

Original: The features of each node are represented by a bag of words of its abstracts.

Original: Texas and Cornell are low-homophily graphs representing links between web pages of the corresponding universities and originally collected by the CMU WebKB projects, where nodes and edges represent web pages and hyperlinks, respectively.

Original: We use the pre-processed datasets by [37].

Original: Squirrel is the subgraph of web pages in Wikipedia discussing the related topics, collected by [41].

Original: The Squirrel graph is rather complex, with both homophily and heterophily combined.

Original: We also use the pre-processed datasets by [37]

Original: Penn94 is a friendship network from the Facebook 100 networks of university students from 2005 [29], where nodes represent students and are labeled with the reported gender of students.

Original: The node features are major, second major/minor, dorm/house, year, and high school.

Original: Twitch is a graph of relationships between accounts on the streaming platform Twitch.

Original: Node features are games liked, location, and streaming habits.

Original: Nodes are labeled with the explicit language used by a streamer.

Original: We utilize the pre-processed sub-graph Twitch-DE [30].

Original: In Twitch-DE, streamers that do not use explicit content (class 0) also often connect to streamers of class 1, which results in an overall non-homophilous structure.

Original: We utilize the homophily [68] and class-average homophily metrics proposed recently by [29] to measure the homophily level of graphs.

Original: Specifically, the edge-homophily is the proportion of edges that connect two nodes of the same class and the class-average homophily is defined as:

Original: [Equation]

Original: where [ a ] + = max( a, 0) and C is the total number of classes, and h k is the class-wise homophily:

Original: [Equation]

Original: in which d u is the number of neighbors of node u , and d ( k u ) u is the number of neighbors of u that have the same class label.

Original: Compared to the edge homophily metirc [68], this metric is less sensitive to the number of classes and the size of each class.

Original: The statistics of datasets are given in Table 4.

Original: We can find that the widely used citation graphs Cora, Citeseer, and Pubmed are highly homophilous, and other graphs show low homophily degrees and thus are non-homophilous.

Original: E.2 Baselines and Setup

Original: Deepwalk [38] is a network embedding method that generates random walks in the graph and then learns latent representations of nodes by treating walks as the equivalent of sentences with SkipGram.

Original: LINE [46] is an embedding model that preserves both the first-order and second-order proximities.

Original: Struc2vec [40] is the method for node representations by capturing the node structural identity.

Original: GAE [26] is a GCN encoder trained by reconstructing adjacency matrix.

Original: VGAE [26] is a probabilistic version of GAE and is trained by reconstructing the adjacency matrix.

Original: DGI [53] is an unsupervised node representation method that maximizes the mutual information between node representations and graph summary.

Original: GraphCL [64] learns node representations by contrasting different augmentations of graphs.

Original: MVGRL [18] is a learning method that contrasts encoders from neighbors and a graph diffusion.

Original: BGRL [48] is a graph representation learning method that learns node representations by predicting alternative augmentations of the graph.

Original: We used the official implementations released by the authors for all baselines.

Original: We ran our experiments on GeForce RTX 2080 Ti.

Original: In all our experiments, we use the Adam optimizer [24].

Original: The projector g θ and predictor h ϕ are both the Multilayer Perceptron (MLP) with a single hidden layer.

Original: We also used techniques like batch normalization and negative sampling on Penn94 and Twitch datasets.

Original: We use the same dataset splits and training procedure for all methods.

Original: We tune hyper-parameters for all models individually based on accuracy on the validation set and randomly initialize the model parameters.

Original: For all methods, the hyper-parameter search spaces are as follows: learning rate { 0 . 001 , 0 .

Original: 01 } , representation dimension { 16 , 32 , 64 } , L2 weight-decay { 5 e − 4 , 1 e − 4 , 5 e − 6 , 1 e − 6 } .

Original: For our DSSL, we tune the following hyper-parameters: σ∈ { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } , σ 22 ∈ { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } , λ ∈ { 0 . 2 , 0 . 4 , 0 .

Original: 8 , 1 . 0 } , β ∈ { 0 . 1 , 0 .

Original: E.3 Cross-Class Neighborhood Similarity

Original: To further demonstrate how DSSL uncovers the latent patterns in neighborhood distributions, we utilize the cross-class neighborhood similarity [34] to serves as the ground-truth for our case study experiments in section 4.4.

Original: Definition 1.

Original: (Cross-Class Neighborhood Similarity).

Original: Given a graph G and node labels, the cross-class neighborhood similarity between two classes c, c ′ is given by s (cid:0) c, c ′ (cid:1)

Original: =

Original: 1 |V c | | V c ′ | (cid:80) i ∈V c ,j ∈V c ′

Original: cos( d ( i ) , d ( j )) where V c indicates the set of nodes in class c and d ( i ) denotes the empirical histogram (over |C| classes) of node i’s neighbors’ labels, and the function cos( · , · ) measures the cosine similarity.

Original: This cross-class neighborhood similarity measures the neighborhood distributions between difference classes.

Original: If neighborhood patterns of different label nodes are different, the inter-class similarity will be low.

Original: In contrast, if nodes with the same label share the same neighborhood distributions, the intra-class similarity should be high.

Original: F Additional Experimental Results F.1 Node Clustering Results

Original: We provide the node clustering results in Table 5.

Original: As shown in Table 5, we can find that our DSSL can consistently improve the node clustering performance compared to the baselines on all datasets except the Cora.

Original: This observation once again verifies the effectiveness of DSSL in learning generalizable node representations.

Original: In addition, along with the node classification performance, these results show that DSSL can provide transferable and robust node representations for various downstream tasks.

Original: F.2 Additional Results on Synthetic Dataset

Original: In this section, we generate the synthetic graphs with various homophily ratios h to evaluate DSSL.

Original: Specifically, we consider the synthetic dataset: syn-products generated by [68].

Original: We refer the interested reader to [68] for the detailed generation process.

Original: The statistics for synthetic dataset syn-products are provided in Table 6.

Original: Roughly speaking, the synthetic graphs can be generated by controlling various edge-homophily ratios h where h = 1 indicts strong edge-homophily, and

Original: [Table]

Original: (cid:19) (cid:24)(cid:19) (cid:20)(cid:19)(cid:19) (cid:20)(cid:24)(cid:19) (cid:21)(cid:19)(cid:19) (cid:19)(cid:17)(cid:19) (cid:19)(cid:17)(cid:25) (cid:20)(cid:17)(cid:21) (cid:47)(cid:82) (cid:86)(cid:86) (cid:51)(cid:88)(cid:69)P(cid:72)(cid:71) (cid:26)(cid:19)(cid:17)(cid:19) (cid:26)(cid:21)(cid:17)(cid:24) (cid:26)(cid:24)(cid:17)(cid:19) (cid:26)(cid:26)(cid:17)(cid:24) (cid:27)(cid:19)(cid:17)(cid:19) (cid:27)(cid:21)(cid:17)(cid:24) (cid:36) (cid:70)(cid:70) (cid:88) (cid:85) (cid:68) (cid:70)(cid:92) (cid:3) (cid:11) (cid:8) (cid:12) h = 0 indicts perfect heterophily of generated graph.

Original: Here, we consider six edge-homophily ratios, h = [0 . 0 , 0 .

Original: We randomly split 25% of nodes into the training set, 25% into validation, and 50% into the test set for the downstream linear classifier.

Original: Table 7 gives the results on syn-products dataset.

Original: From this table, we have the following observations: (1) GAE achieves the best accuracy under strong homophily ( h = 1 ), but it is significantly less accurate than other methods in low-homophily settings.

Original: This is reasonable since GAE over-emphasizes proximity and the homophily assumption.

Original: (2) Our DSSL outperforms recent contrastive learning methods such as DGI and MVGRL, which shows the effectiveness of DSSL on non-homophily scenarios, and suggests that designing effective graph augmentation for those contrastive baselines on the non-homophilous graphs is difficult due to the heterogeneous and diverse neighbor pattern.

Original: (3) Our DSSL can generally achieve the best performance and effectively adapt to all levels of homophily.

Original: [Table]

Original: F.3 Additional Results on Ablation Study and Parameter Analysis

Original: Ablation Study .

Original: We further conduct experiments to study the effects of the global update on prototypes and asymmetric architecture on homophily datasets.

Original: Figures 8 and 9 show the ablation results on Cora, Citeseer and Pubmed.

Original: These results also echo the collapse issue as discussed in the main text: without the global update and asymmetric architecture, there is a significant performance drop on these three homophily datasets.

Original: Parameter Analysis .

Original: We provide more results on the parameter analysis.

Original: Figure 10 shows the results on Citeseer, Cornell, and Squirrel datasets.

Original: We can observe that having large values of σ 22 also does not improve the overall performance, and DSSL is not very sensitive to the Gumbel softmax temperature λ for these three datasets.

Original: We also find that as K increases from 2 to 8 ( 2 to 16 for

Original: Citeseer), the accuracy of DSSL improves.

Original: [Table]

Original: [Figure]

Original: F.4 Additional Results on Visualization and Case Study

Original: To further examine the learned latent factors, in Figure 11, we provide the additional case study and visualization results on other datasets.

Original: We can find that, in most cases, the latent factors share a similar distribution for the same type of link, which illustrates its effectiveness in decoupling the underlying latent semantic meaning of different neighbors.

Original: This also interprets the reason why capturing the semantic shift can improve performance.

Original: Figure 12 shows the results of long-range nodes on Cora and Texas datasets.

Original: We can also find that some nodes exhibit similar semantic clusters, regardless of the distance between them.

Original: These additional qualitative visualization results, along with our other experiment results, suggests that considering the semantic shift between local neighbors and capturing global semantics may be necessary for non-homophilous graphs.

Original: G Discussion and Broader impact

Original: In this paper, we proposed a new framework called decoupled self-supervised learning (DSSL) for unsupervised node representation learning on non-homophilous graphs.

Original: The theoretical analysis and extensive experiments show the effectiveness of our proposed DSSL.

Original: Limitation of the work .

Original: Despite the theoretical grounds and the promising experimental justifications, there is one limitation of the current work, which we hope to improve in future work: our theoretic analysis requires some assumptions on the relationship between the downstream labels and the neighborhood distribution.

Original: While, not surprisingly, we have to make some assumptions to expect good generalization for the unsupervised node representation learning, it is an interesting future direction to explore more relaxed assumptions than the ones used in this work.

Original: Potential negative societal impacts .

Original: In this work, we propose a self-supervised learning framework for node representation learning which does not rely on annotated labels, which might reduce the need for label annotation and thus makes a few individuals who focus on labeling or annotating data

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: unemployed.

Original: In addition, our learned representation may suffer from malicious adversaries who seek to extract node-level information about sensitive attributes.

Original: Thus, improving the robustness of DSSL is an interesting future work direction.