Original: Generalized Laplacian Eigenmaps
Revised: Generalized Laplacian Eigenmaps

Original: Abstract
Revised: Abstract

Original: Graph contrastive learning attracts/disperses node representations for similar/dissimilar node pairs under some notion of similarity.
Revised: Graph contrastive learning attracts/disperses node representations for similar/dissimilar node pairs under some notion of similarity.

Original: It may be combined with a low-dimensional embedding of nodes to preserve intrinsic and structural properties of a graph.
Revised: It may be combined with a low-dimensional embedding of nodes to preserve intrinsic and structural properties of a graph.

Original: COLES, a recent graph contrastive method combines traditional graph embedding and negative sampling into one framework.
Revised: COLES, a recent graph contrastive method combines traditional graph embedding and negative sampling into one framework.

Original: COLES in fact minimizes the trace difference between the within-class scatter matrix encapsulating the graph connectivity and the total scatter matrix encapsulating negative sampling.
Revised: COLES in fact minimizes the trace difference between the within-class scatter matrix encapsulating the graph connectivity and the total scatter matrix encapsulating negative sampling.

Original: In this paper, we propose a more essential framework for graph embedding, called Generalized Laplacian EigeNmaps (GLEN), which learns a graph representation by maximizing the rank difference between the total scatter matrix and the within-class scatter matrix, resulting in the minimum class separation guarantee.
Revised: In this paper, we propose a more essential framework for graph embedding, called Generalized Laplacian EigeNmaps (GLEN), which learns a graph representation by maximizing the rank difference between the total scatter matrix and the within-class scatter matrix, resulting in the minimum class separation guarantee.

Original: However, the rank difference minimization is an NP-hard problem.
Revised: However, the rank difference minimization is an NP-hard problem.

Original: Thus, we replace the trace difference that corresponds to the difference of nuclear norms by the difference of
Revised: Thus, we replace the trace difference that corresponds to the difference of nuclear norms by the difference of

Original: LogDet expressions, which we argue is a more accurate surrogate for the NP-hard rank difference than the trace difference.
Revised: LogDet expressions, which we argue is a more accurate surrogate for the NP-hard rank difference than the trace difference.

Original: While enjoying a lesser computational cost, the difference of LogDet terms is lower-bounded by the Affine-invariant
Revised: While enjoying a lesser computational cost, the difference of LogDet terms is lower-bounded by the Affine-invariant

Original: Riemannian metric (AIRM) andJesen-Bregman the LogDet Divergence (JBLD), and upper-bounded by AIRM scaled by the factor of √ m .
Revised: Riemannian metric (AIRM) and upper-bounded by AIRM scaled by the factor of √ m .

Original: We show  that GLEN offers favourable accuracy/scalability compared to state-of-the-art baselines.
Revised: We show on popular benchmarks/backbones that GLEN offers favourable accuracy/scalability compared to state-of-the-art baselines.

Original:  Introduction
Revised:  Introduction

Original: Laplacian Eigenmaps [3] and IsoMap [39] are graph embedding methods that reduce the dimensionality of data by assuming the data exists on a low-dimensional manifold.
Revised: Laplacian Eigenmaps [3] and IsoMap [36] are graph embedding methods that reduce the dimensionality of data by assuming the data exists on a low-dimensional manifold.

Original: The objective function in such models encourages node embeddings to lie near each other in the embedding space if nodes are close to each other in the original space.
Revised: The objective function in such models encourages node embeddings to lie near each other in the embedding space if nodes are close to each other in the original space.

Original: While the classical methods capture the related node pairs, they neglect modeling unrelated node pairs.
Revised: While the classical methods capture the related node pairs, they neglect modeling unrelated node pairs.

Original: In contrast, modern graph embedding models such as [38, 10, 47] and Graph Contrastive Learning (GCL) [40, 63, 11, 64, 62] are unified under the (Sampled) Noise Contrastive Estimation framework, called (Sampled)NCE [28, 23].
Revised: In contrast modern graph embedding models such as [35, 10, 44] and Graph Contrastive Learning (GCL) [37, 56, 11, 57, 55] are unified under the (Sampled) Noise Contrastive Estimation framework, called (Sampled)NCE [27, 23].

Original: Most of GCL methods do not incorporate the graph information into the loss but follow the setting from computer vision, i.e ., they assume that randomly drawn pairs should be dissimilar, whereas the original sample and its augmentations should be similar [42].
Revised: Most of GCL methods do not incorporate the graph information into the loss but follow the setting from computer vision, i.e ., they assume that randomly drawn pairs should be dissimilar, whereas the original sample and its augmentations should be similar [39].

Original: In contrast, COntrastive Laplacian EigenmapS (COLES) [62] is a framework which combines a (graph) neural network with Laplacian eigenmaps utilizing the graph Laplacian matrix within a contrastive loss.
Revised: In contrast, COntrastive Laplacian EigenmapS (COLES) [55] is a framework which combines a (graph) neural network with Laplacian eigenmaps utilizing the graph Laplacian matrix within a contrastive loss.

Original: Based on the NCE framework, COLES minimizes the trace difference of Laplacians.
Revised: Based on the NCE framework, COLES minimizes the trace difference of Laplacians.

Original: In this paper, we analyze the relation among within-class, between-class and total scatter matrices under the rank inequality, and prove that, under a simple assumption, the distance between any dissimilar (negative) samples would be greater/equal than the inter-class distance between their corresponding class centers.
Revised: In this paper, we analyze the relation among within-class, between-class and total scatter matrices under the rank inequality, and prove that, under a simple assumption, the distance between any dissimilar (negative) samples would be greater/equal than the inter-class distance between their corresponding class centers.

Original: Based on such a condition, we derive GLEN, a reformulation of graph embedding into a rank difference problem, which is a more general framework than other graph embedding frameworks, i.e ., under specific relaxations of the rank difference problem, we can recover different frameworks.
Revised: Based on such a condition, we derive GLEN, a reformulation of graph embedding into a rank difference problem, which is a more general framework than other graph embedding frameworks, i.e ., under specific relaxations of the rank difference problem, we can recover different frameworks.

Original: To that end, we demonstrate how to optimize the rank difference problem with a difference of LogDet expressions, a differentiable relaxation suitable for use with (graph) neural networks.
Revised: To that end, we demonstrate how to optimize the rank difference problem with a difference of LogDet expressions, a differentiable relaxation suitable for use with (graph) neural networks.

Original: We consider other surrogates of the rank difference problem, based on the Nuclear norm, γ -nuclear norm, Schatten norm, and the Geman norm.
Revised: We consider other surrogates of the rank difference problem, based on the Nuclear norm, γ -nuclear norm, Schatten norm, and the Geman norm.

Original: Moreover, we provide theoretical considerations regarding the low-rank optimization and connection to the Riemannian manifold in order to interpret our approach.
Revised: Moreover, we provide theoretical considerations regarding the low-rank optimization and connection to the Riemannian manifold in order to interpret our approach.

Original: In summary, our contributions are threefold:
Revised: In summary, our contributions are threefold:

Original: i. We propose a rank-based condition connecting within-class, between-class and total scatter matrices under which we provide the minimum class separation guarantee.
Revised: i. We propose a rank-based condition connecting within-class, between-class and total scatter matrices under which we provide the minimum class separation guarantee.

Original: We propose a loss function, Generalized Laplacian EigenNaps (GLEN), that realizes this condition.
Revised: We propose a loss function, Generalized Laplacian EigenNaps (GLEN), that realizes this condition.

Original: ii.
Revised: ii.

Original: As the rank difference problem is NP-hard, we consider a difference of LogDet surrogate to learn node embeddings, as opposed to the trace difference (an upper bound of the difference of LogDet terms) used by other graph embedding models.
Revised: As the rank difference problem is NP-hard, we consider a difference of LogDet surrogate to learn node embeddings, as opposed to the trace difference (an upper bound of the difference of LogDet terms) used by other graph embedding models.

Original: We also consider other surrogates.
Revised: We also consider other surrogates.

Original: iii.
Revised: iii.

Original: We study the distance between symmetric positive (semi-)definite matrices and the LogDet-based relaxation of GLEN.
Revised: We study the distance between symmetric positive (semi-)definite matrices and the LogDet-based relaxation of GLEN.

Original: While enjoying fewer computations, the difference of LogDet terms  enjoys the Affine-invariant Riemannian metric (AIRM) for a lower bound and AIRM scaled by √ m as an upper bound, and is lower-bounded by Jesen-Bregman LogDet Divergence (JBLD).
Revised: While enjoying fewer computations, the difference of LogDet terms of GLEN enjoys the Affine-invariant Riemannian metric (AIRM) for a lower bound and AIRM scaled by √ m as an upper bound.

Revised: We explain how GLEN connects to other graph embeddings.

Original:  Related Works
Revised:  Related Works

Original: Graph Embeddings.
Revised: Graph Embeddings.

Original: By assuming that the data lies on a low-dimensional manifold, graph embedding methods such as Laplacian Eigenmaps [3] and IsoMap [39] optimize low-dimensional data embeddings.
Revised: By assuming that the data lies on a low-dimensional manifold, graph embedding methods such as Laplacian Eigenmaps [3] and IsoMap [36] optimize low-dimensional data embeddings.

Original: These methods [5] construct a similarity graph by measuring the similarity of high-dimensional feature vectors and embed the nodes into a low-dimensional space.
Revised: These methods [5] construct a similarity graph by measuring the similarity of high-dimensional feature vectors and embed the nodes into a low-dimensional space.

Original: DeepWalk [32] uses truncated random walks to explore the graph structure, and the skip-gram model for word embedding to determine the embedding vectors of nodes.
Revised: DeepWalk [31] uses truncated random walks to explore the graph structure, and the skip-gram model for word embedding to determine the embedding vectors of nodes.

Original: By setting the walk length to one and using negative sampling [27], LINE [38] explores a similar idea with an explicit objective function while REFINE [59] imposes additional orthogonality constraints which deem REFINE extremely fast.
Revised: By setting the walk length to one and using negative sampling [26], LINE [35] explores a similar idea with an explicit objective function while REFINE [52] imposes additional orthogonality constraints which deem REFINE extremely fast.

Original: Node2Vec [9] interpolates between breadth- and depth-first sampling.
Revised: Node2Vec [9] interpolates between breadth- and depth-first sampling.

Original: COLES [62] unifies traditional graph embedding and negative sampling by introducing a positive contrastive term that captures the graph structure, and a negative contrastive random sampling.
Revised: COLES [55] unifies traditional graph embedding and negative sampling by introducing a positive contrastive term that captures the graph structure, and a negative contrastive random sampling.

Original: COLES solves the trace difference problem akin to traditional graph embedding models [46].
Revised: COLES solves the trace difference problem akin to traditional graph embedding models [43].

Original: In this paper, we propose a more general loss for graph embedding, i.e ., COLES solves the trace difference (Nuclear norms difference) relaxation of GLEN.
Revised: In this paper, we propose a more general loss for graph embedding, i.e ., COLES solves the trace difference (Nuclear norms difference) relaxation of GLEN.

Original: Graph embedding techniques [46] provide a general framework for dimensionality reduction such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), and Locality Preserving Projections (LPP) [12].
Revised: Graph embedding techniques [43] provide a general framework for dimensionality reduction such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), and Locality Preserving Projections (LPP) [12].

Original: All methods within this category can be considered as solving the same problem under varying assumptions, i.e ., maximising the intra- and inter-class separation by optimizing the trace difference, also used in metric learning [22].
Revised: All methods within this category can be considered as solving the same problem under varying assumptions, i.e ., maximising the intra- and inter-class separation by optimizing the trace difference, also used in metric learning [22].

Original: However, such a family of objective functions is not motivated by the guarantee on the minimum class separation between feature vectors from different categories.
Revised: However, such a family of objective functions is not motivated by the guarantee on the minimum class separation between feature vectors from different categories.

Original: GLEN, in its purest NP-hard form, provides the minimum class separation guarantee and can be realised by several formulations depending on chosen trade-offs.
Revised: GLEN, in its purest NP-hard form, provides the minimum class separation guarantee and can be realised by several formulations depending on chosen trade-offs.

Original: Unsupervised Representation Learning for Graph Neural Networks (GNN).
Revised: Unsupervised Representation Learning for Graph Neural Networks (GNN).

Original: Unsupervised
Revised: Unsupervised

Original: GNN training can be reconstruction-, contrastive- or diffusion-based.
Revised: GNN training can be reconstruction-, contrastive- or diffusion-based.

Original: To train a graph encoder in an unsupervised manner, GCN [17] minimizes a reconstruction error which only considers the similarity matrix and ignores the dissimilarity information.
Revised: To train a graph encoder in an unsupervised manner, GCN [17] minimizes a reconstruction error which only considers the similarity matrix and ignores the dissimilarity information.

Original: At various scales of the graph, contrastive methods determine the positive and negative sets.
Revised: At various scales of the graph, contrastive methods determine the positive and negative sets.

Original: For example, local-local CL and global-local CL strategies are highly popular.
Revised: For example, local-local CL and global-local CL strategies are highly popular.

Original: GraphSAGE [10], inspired by DeepWalk [32], uses the contrastive loss which encourages neighbor nodes to have similar representations, while preserving dissimilarity between representations of disparate nodes.
Revised: GraphSAGE [10], inspired by DeepWalk [31], uses the contrastive loss which encourages neighbor nodes to have similar representations, while preserving dissimilarity between representations of disparate nodes.

Original: DGI [40], inspired by Deep InfoMax (DIM) [13], uses an objective with global-local sampling strategy to maximize the Mutual Information (MI) between global and local graph embeddings.
Revised: DGI [37], inspired by Deep InfoMax (DIM) [13], uses an objective with global-local sampling strategy to maximize the Mutual Information (MI) between global and local graph embeddings.

Original: Augmented Multiscale Deep InfoMax (AMDIM) [2] maximizes
Revised: Augmented Multiscale Deep InfoMax (AMDIM) [2] maximizes

Original: MI between multiple data views.
Revised: MI between multiple data views.

Original: MVRLG [11] contrasts encodings from first-order neighbors and a graph diffusion.
Revised: MVRLG [11] contrasts encodings from first-order neighbors and a graph diffusion.

Original: Fisher-Bures Adversary GCN [36] treats the graph as generated w.r.t.
Revised: Fisher-Bures Adversary GCN [34] treats the graph as generated w.r.t.

Original: some observation noise.
Revised: some observation noise.

Original: COSTA [57] constructs the views by injecting features with the random noise.
Revised: COSTA [50] constructs the views by injecting features with the random noise.

Original: [Figure]
Revised: [Figure]

Original: However, such contrastive approaches often require thousands of epochs to converge and perform well.
Revised: However, such contrastive approaches often require thousands of epochs to converge and perform well.

Original: In addition, many contrastive losses have an exponential increase in memory overhead w.r.t.
Revised: In addition, many contrastive losses have an exponential increase in memory overhead w.r.t.

Original: the number of nodes.
Revised: the number of nodes.

Original: In contrast, our method does not explicitly use the local-local setting but the total scatter matrix, and thus saves computational and storage cost.
Revised: In contrast, our method does not explicitly use the local-local setting but the total scatter matrix, and thus saves computational and storage cost.

Original: Linear GNNs, i.e ., SGC [45] and S 2 GC [60], capture the neighborhood and increasingly larger neighborhoods of each node due to the diffusion, respectively.
Revised: Linear GNNs, i.e ., SGC [42] and S 2 GC [53], capture the neighborhood and increasingly larger neighborhoods of each node due to the diffusion, respectively.

Original: SGC and S 2 GC have no projection layer, thus the size of embeddings is equal to the input dimension.
Revised: SGC and S 2 GC have no projection layer, thus the size of embeddings is equal to the input dimension.

Original: GLEN can learn a projection layer in an unsupervised manner with a linear function or Multi-Layer Perceptron (MLP) applied to linear
Revised: GLEN can learn a projection layer in an unsupervised manner with a linear function or Multi-Layer Perceptron (MLP) applied to linear

Original: GNNs or any other GNN models [18, 36, 56], etc .
Revised: GNNs or any other GNN models [18, 34, 49], etc .

Original:  Preliminaries
Revised:  Preliminaries

Original: Notations.
Revised: Notations.

Original: Let G =( V, E ) be a simple, connected and undirected graph with n = | V | nodes and m = | E | edges.
Revised: Let G =( V, E ) be a simple, connected and undirected graph with n = | V | nodes and m = | E | edges.

Original: Let i ∈ { 1 , · · · , n } be the node index of G , and d j be the degree of node j of G .
Revised: Let i ∈ { 1 , · · · , n } be the node index of G , and d j be the degree of node j of G .

Original: Let W be the adjacency matrix, and D be the diagonal matrix containing degrees of nodes.
Revised: Let W be the adjacency matrix, and D be the diagonal matrix containing degrees of nodes.

Original: Let
Revised: Let

Original: X ∈ R n × d denote the node feature matrix where each node v is associated with a feature vector x v ∈ R d .
Revised: X ∈ R n × d denote the node feature matrix where each node v is associated with a feature vector x v ∈ R d .

Original: Let the normalized graph Laplacian matrix be defined as L = I − ˜ W ∈ S n + , a symmetric positive semi-definite matrix and ˜ W
Revised: Let the normalized graph Laplacian matrix be defined as L = I − ˜ W ∈ S n + , a symmetric positive semi-definite matrix and ˜ W

Original: = D − 1 / 2 WD − 1 / 2 .
Revised: = D − 1 / 2 WD − 1 / 2 .

Original: S m + ( + ) is a set of symmetric positive (semi- )definite matrices.
Revised: S m + ( + ) is a set of symmetric positive (semi- )definite matrices.

Original: Let Z = f Θ ( X ) ∈ R n × m be a generalized node embedding, i.e .,
Revised: Let Z = f Θ ( X ) ∈ R n × m be a generalized node embedding, i.e .,

Original: X could be identity matrix ( e.g ., no node attributes), f Θ ( X ) could be GNN or a linear function with parameters Θ .
Revised: X could be identity matrix ( e.g ., no node attributes), f Θ ( X ) could be GNN or a linear function with parameters Θ .

Original: Scalars/vectors/matrices are denoted by lowercase regular/lowercase bold/uppercase bold fonts.
Revised: Scalars/vectors/matrices are denoted by lowercase regular/lowercase bold/uppercase bold fonts.

Original: 3.1 Scatter Matrices
Revised: 3.1 Scatter Matrices

Original: Below are given standard definitions of scatter matrices, including the total scatter matrix S t ∈ S m + ( + ) , the within-class matrix S w ∈ S m + ( + ) , and between-class matrix S b ∈ S m + ( + ) :
Revised: Below are given standard definitions of scatter matrices, including the total scatter matrix S t ∈ S m + ( + ) , the within-class matrix S w ∈ S m + ( + ) , and between-class matrix S b ∈ S m + ( + ) :

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Let e be an n -dimensional vector with all coefficients equal one, I be an identity matrix, S t be the total scatter (covariance) matrix, and ¯ z ∈ R m be the mean of all samples.
Revised: Let e be an n -dimensional vector with all coefficients equal one, I be an identity matrix, S t be the total scatter (covariance) matrix, and ¯ z ∈ R m be the mean of all samples.

Original: Let µ y i ∈
Revised: Let µ y i ∈

Original: R m be the class center of the i -th sample and µ c ∈ R m be the c -th class center.
Revised: R m be the class center of the i -th sample and µ c ∈ R m be the c -th class center.

Original: Let the total number of categories be given by C , whereas n c be the number of samples for the c -th category.
Revised: Let the total number of categories be given by C , whereas n c be the number of samples for the c -th category.

Original: Let e c ∈ R n be a vector where a given coefficient indexed by node is equal one if its node is of class c , otherwise it is equal zero.
Revised: Let e c ∈ R n be a vector where a given coefficient indexed by node is equal one if its node is of class c , otherwise it is equal zero.

Original: We note that both S t and S w can take a form akin to Laplacian eigenmaps such that ˜ W t and ˜
Revised: We note that both S t and S w can take a form akin to Laplacian eigenmaps such that ˜ W t and ˜

Original: W w are the corresponding normalized adjacent matrices.
Revised: W w are the corresponding normalized adjacent matrices.

Original: Let us also define graph Laplacian matrices
Revised: Let us also define graph Laplacian matrices

Original: L t = I − ˜ W t ∈ S n + and L w =
Revised: L t = I − ˜ W t ∈ S n + and L w =

Original: I − ˜ W w ∈ S n + which will be used in the sequel.
Revised: I − ˜ W w ∈ S n + which will be used in the sequel.

Original: Importantly, let us assume that a graph Laplacian matrix L containing graph links could be seen as a noisy version of
Revised: Importantly, let us assume that a graph Laplacian matrix L containing graph links could be seen as a noisy version of

Original: L w in which all nodes of a given class c connect under the weight equal 1 /n c .
Revised: L w in which all nodes of a given class c connect under the weight equal 1 /n c .

Original: Observe that S t =
Revised: Observe that S t =

Original: S w + S b .
Revised: S w + S b .

Original: Thus, Rank( S t ) ≤ Rank( S w ) + Rank( S b ) due to the rank inequality.
Revised: Thus, Rank( S t ) ≤ Rank( S w ) + Rank( S b ) due to the rank inequality.

Original: Below we highlight the condition underpinning the subsequent motivation:
Revised: Below we highlight the condition underpinning the subsequent motivation:

Original: Condition 1.
Revised: Condition 1.

Original: Rank( S t ) =
Revised: Rank( S t ) =

Original: Rank( S w ) + Rank( S b ) .
Revised: Rank( S w ) + Rank( S b ) .

Original: 3.2 Motivation
Revised: 3.2 Motivation

Original: Figure 1 shows some three optimal solutions for Condition 1.
Revised: Figure 1 shows some three optimal solutions for Condition 1.

Original: The rank of between-class scatter matrix S b for the whole dataset is at most C − 1 (where C is the number of classes).
Revised: The rank of between-class scatter matrix S b for the whole dataset is at most C − 1 (where C is the number of classes).

Original: Since
Revised: Since

Original: Rank( AB ) ≤ min(Rank( A ) , Rank( B )) , we have † Rank( S − 1 w S b ) ≤ Rank( S b ) ≤ C − 1 .
Revised: Rank( AB ) ≤ min(Rank( A ) , Rank( B )) , we have † Rank( S − 1 w S b ) ≤ Rank( S b ) ≤ C − 1 .

Original: The rank is the number of non-zero eigenvalues of a matrix so S − 1 w S b has at most C − 1 non-zero eigenvalues.
Revised: The rank is the number of non-zero eigenvalues of a matrix so S − 1 w S b has at most C − 1 non-zero eigenvalues.

Original: Condition 1 implies that Rank( S − 1 w S b ) = 0 results in the minimum class separation guarantee under that condition.
Revised: Condition 1 implies that Rank( S − 1 w S b ) = 0 results in the minimum class separation guarantee under that condition.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let the feature dimension be larger than the class number ( i.e ., m > C ) and Conditionhold.
Revised: Let the feature dimension be larger than the class number ( i.e ., m > C ) and Conditionhold.

Original: Then, the minimum class separation is equal to the distance between class centers.
Revised: Then, the minimum class separation is equal to the distance between class centers.

Original: In other words, the distance between any two vectors z i and z j with labels y i ̸
Revised: In other words, the distance between any two vectors z i and z j with labels y i ̸

Original: = y j is greater/equal the distance between class centers µ y i and µ y j :
Revised: = y j is greater/equal the distance between class centers µ y i and µ y j :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As S w is the orthogonal complement of S b , i.e ., S − 1 w S b = 0 , S w + S b = U Σ U ⊤ , S w =
Revised: As S w is the orthogonal complement of S b , i.e ., S − 1 w S b = 0 , S w + S b = U Σ U ⊤ , S w =

Original: U 1: k Σ 1: k U ⊤ 1: k and S b =
Revised: U 1: k Σ 1: k U ⊤ 1: k and S b =

Original: U k
Revised: U k

Original: +1: m Σ k
Revised: +1: m Σ k

Original: +1: m U ⊤ k
Revised: +1: m U ⊤ k

Original: +1: m where 1 ≤ k < m .
Revised: +1: m where 1 ≤ k < m .

Original: Let z i = µ y i +
Revised: Let z i = µ y i +

Original: U ⊤ ϵ
Revised: U ⊤ ϵ

Original: i
Revised: i

Original: where ϵ i is the representation under the basis U and ϵ ( k +1: m ) ,i = 0 because only top k components 1: k represent S w .
Revised: where ϵ i is the representation under the basis U and ϵ ( k +1: m ) ,i = 0 because only top k components 1: k represent S w .

Original: Thus, the orthogonal projection U k +1: m fulfills ∥
Revised: Thus, the orthogonal projection U k +1: m fulfills ∥

Original: U k +1: m ( z i − z j ) ∥ 2 ≤ ∥
Revised: U k +1: m ( z i − z j ) ∥ 2 ≤ ∥

Original: z
Revised: z

Original: i − z j ∥ 2 .
Revised: i − z j ∥ 2 .

Original: Moreover, U k +1: m ( z i − µ y i )
Revised: Moreover, U k +1: m ( z i − µ y i )

Original: =
Revised: =

Original: U k
Revised: U k

Original: +1: m ( U ⊤ ϵ i )
Revised: +1: m ( U ⊤ ϵ i )

Original: = 0 .
Revised: = 0 .

Original: That is, all { z i : y i =
Revised: That is, all { z i : y i =

Original: c } are projected onto the mean µ c .
Revised: c } are projected onto the mean µ c .

Original: Thus, the inequality in Eq.
Revised: Thus, the inequality in Eq.

Original: 2 holds.
Revised: 2 holds.

Original: Theorem 1 guarantees the worst inter-class distance § .
Revised: Theorem 1 guarantees the worst inter-class distance § .

Original: Figure 1 shows some cases that meet Condition 1.
Revised: Figure 1 shows some cases that meet Condition 1.

Original: Figure 1a shows the case for which the class centers collapse to a single point and thus the inter-class distance equals zero (collapse of the feature space).
Revised: Figure 1a shows the case for which the class centers collapse to a single point and thus the inter-class distance equals zero (collapse of the feature space).

Original: Figures 1b and 1c show other cases.
Revised: Figures 1b and 1c show other cases.

Original:  Methodology
Revised:  Methodology

Original: Condition 1 points to a promising research direction in learning discriminative feature spaces.
Revised: Condition 1 points to a promising research direction in learning discriminative feature spaces.

Original: However, optimizing over the rank is NP-hard and non-differentiable.
Revised: However, optimizing over the rank is NP-hard and non-differentiable.

Original: In what follows, we provide the formulation of Generalized Laplacian EigeNmaps (GLEN) and its relaxation, which is differentiable.
Revised: In what follows, we provide the formulation of Generalized Laplacian EigeNmaps (GLEN) and its relaxation, which is differentiable.

Original: 4.1 Generalized Laplacian Eigenmaps
Revised: 4.1 Generalized Laplacian Eigenmaps

Original: As solving Condition 1 is NP-hard, we propose a relaxation where Rank( S t ) is encouraged to be as large as possible (bounded by the feature dimension m ).
Revised: As solving Condition 1 is NP-hard, we propose a relaxation where Rank( S t ) is encouraged to be as large as possible (bounded by the feature dimension m ).

Original: On the contrary, if Rank( S t ) ≈ Rank( S b ) then the small Rank( S w ) limits the feature diversity.
Revised: On the contrary, if Rank( S t ) ≈ Rank( S b ) then the small Rank( S w ) limits the feature diversity.

Original: In the extreme case, if Rank( S w ) = 0 , the feature representation collapses.
Revised: In the extreme case, if Rank( S w ) = 0 , the feature representation collapses.

Original: Larger 

Revised: Larger

Original: <
Revised: <

Original: Rank( S b ) ≤ C − 1 improves the inter-class diversity.
Revised: Rank( S b ) ≤ C − 1 improves the inter-class diversity.

Original: We propose a new Generalized Laplacian EigeNmaps (GLEN) framework for unsupervised network embedding.
Revised: We propose a new Generalized Laplacian EigeNmaps (GLEN) framework for unsupervised network embedding.

Original: In the most general form, GLEN maximizes the difference of rank terms:
Revised: In the most general form, GLEN maximizes the difference of rank terms:

Original: [Equation]
Revised: [Equation]

Original: As the general matrix Rank Minimization Problem (RMP) [7] is NP-hard and so is the difference of rank terms in Eq.
Revised: As the general matrix Rank Minimization Problem (RMP) [7] is NP-hard and so is the difference of rank terms in Eq.

Original: 3, we relax this problem by the difference of LogDet terms that serve as a surrogate of the NP-hard problem.
Revised: 3, we relax this problem by the difference of LogDet terms that serve as a surrogate of the NP-hard problem.

Original: Appendix H derives GLEN from the SampledNCE framework.
Revised: Appendix I derives GLEN from the SampledNCE framework.

Original: GLEN (LogDet relaxation).
Revised: GLEN (LogDet relaxation).

Original: I. Define:
Revised: I. Define:

Original: [Equation]
Revised: [Equation]

Original: where λ ≥ 0 controls the impact of log det( S w ) .
Revised: where λ ≥ 0 controls the impact of log det( S w ) .

Original: If λ = 0 , δ ( · ) encourages Rank( f Θ ( X ))
Revised: If λ = 0 , δ ( · ) encourages Rank( f Θ ( X ))

Original: = m .
Revised: = m .

Original: II.
Revised: II.

Original: Let S t = f Θ ( X )
Revised: Let S t = f Θ ( X )

Original: ⊤ L t f Θ ( X ) and S
Revised: ⊤ L t f Θ ( X ) and S

Original: w = f Θ ( X )
Revised: w = f Θ ( X )

Original: ⊤ L w f Θ ( X ) .
Revised: ⊤ L w f Θ ( X ) .

Original: Then the LogDet relaxation becomes:
Revised: Then the LogDet relaxation becomes:

Original: [Equation]
Revised: [Equation]

Original: where I ensures I + αf Θ ( X ) ⊤ L f Θ ( X ) > 0 as f Θ ( X ) ⊤ L f Θ ( X ) may be S m + leading to det( f Θ ( X )
Revised: where I ensures I + αf Θ ( X ) ⊤ L f Θ ( X ) > 0 as f Θ ( X ) ⊤ L f Θ ( X ) may be S m + leading to det( f Θ ( X )

Original: ⊤ L f Θ ( X ))
Revised: ⊤ L f Θ ( X ))

Original: = 0 .
Revised: = 0 .

Original: Thus, we use log det( I + α S ) as a smooth surrogate for Rank( S ) .
Revised: Thus, we use log det( I + α S ) as a smooth surrogate for Rank( S ) .

Original: Proposition 1.
Revised: Proposition 1.

Original: Let σ ( S ) be the vector of eigenvalues of matrix S ∈ S
Revised: Let σ ( S ) be the vector of eigenvalues of matrix S ∈ S

Original: m
Revised: m

Original: + ( + ) , and Eig( S ) be a diagonal matrix with σ ( S ) as its diagonal.
Revised: + ( + ) , and Eig( S ) be a diagonal matrix with σ ( S ) as its diagonal.

Original: Let S , S ′ ∈ S m + ( + ) and α > 0 .
Revised: Let S , S ′ ∈ S m + ( + ) and α > 0 .

Original: Then, δ ( S , S ′ ; α, λ )
Revised: Then, δ ( S , S ′ ; α, λ )

Original: =
Revised: =

Original: δ (Eig( S ) , Eig( S ′ ); α, λ ) , i.e ., δ ( · ) depends on eigenvalues rather than eigenvectors of S and S ′ .
Revised: δ (Eig( S ) , Eig( S ′ ); α, λ ) , i.e ., δ ( · ) depends on eigenvalues rather than eigenvectors of S and S ′ .

Original: Proof.
Revised: Proof.

Original: The proof follows from the equality det( I + α S ) =
Revised: The proof follows from the equality det( I + α S ) =

Original: (cid:81)
Revised: (cid:81)

Original: i σ i
Revised: i σ i

Original: ( I + α S )
Revised: ( I + α S )

Original: = (cid:81)
Revised: = (cid:81)

Original: i (1 + ασ i ( S ))
Revised: i (1 + ασ i ( S ))

Original: =
Revised: =

Original: det( I + α Eig( S )) .
Revised: det( I + α Eig( S )) .

Original: Thus δ ( S , S ′ ; α, λ ) = log det( I + α S ) − λ log det( I + α S ′ )
Revised: Thus δ ( S , S ′ ; α, λ ) = log det( I + α S ) − λ log det( I + α S ′ )

Original: = log det( I + α Eig( S ))
Revised: = log det( I + α Eig( S ))

Original: − λ log det( I + α Eig( S ′ ))
Revised: − λ log det( I + α Eig( S ′ ))

Original: =
Revised: =

Original: δ (Eig( S ) , Eig( S ′ ); α, λ ) .
Revised: δ (Eig( S ) , Eig( S ′ ); α, λ ) .

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: Below, we compare our approach and other methods by looking at (i) the low-rank optimization and (ii) the non-Euclidean distances between symmetric positive (semi-)definite matrices.
Revised: Below, we compare our approach and other methods by looking at (i) the low-rank optimization and (ii) the non-Euclidean distances between symmetric positive (semi-)definite matrices.

Original: 5.1 Nuclear Norm vs . LogDet for Rank Minimization
Revised: 5.1 Nuclear Norm vs . LogDet for Rank Minimization

Original: Claim 1.
Revised: Claim 1.

Original: COLES [62] is a convex relaxation (using the nuclear norm) of the rank difference in Eq.
Revised: COLES [55] is a convex relaxation (using the nuclear norm) of the rank difference in Eq.

Original: 3: Θ
Revised: 3: Θ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Θ
Revised: Θ

Original: [Equation]
Revised: [Equation]

Original: where Tr (cid:0) f Θ ( X ) ⊤ L t f Θ ( X ) (cid:1) =
Revised: where Tr (cid:0) f Θ ( X ) ⊤ L t f Θ ( X ) (cid:1) =

Original: ∥
Revised: ∥

Original: S t ∥ ∗ and Tr (cid:0) f Θ ( X )
Revised: S t ∥ ∗ and Tr (cid:0) f Θ ( X )

Original: ⊤ L w f Θ ( X ) (cid:1) =
Revised: ⊤ L w f Θ ( X ) (cid:1) =

Original: ∥
Revised: ∥

Original: S w ∥ ∗ .
Revised: S w ∥ ∗ .

Original: The nuclear norm ∥ · ∥ ∗ can be regarded as the ℓ 1 norm over singular values.
Revised: The nuclear norm ∥ · ∥ ∗ can be regarded as the ℓ 1 norm over singular values.

Original: As the ℓ 1 norm induces sparsity, the nuclear norm encourages sparse singular values leading to low-rank solutions.
Revised: As the ℓ 1 norm induces sparsity, the nuclear norm encourages sparse singular values leading to low-rank solutions.

Original: If f Θ ( X ) ⊤ f Θ ( X ) is restricted to be diagonal, ∥ f Θ ( X )
Revised: If f Θ ( X ) ⊤ f Θ ( X ) is restricted to be diagonal, ∥ f Θ ( X )

Original: ⊤ f Θ ( X ) ∥ ∗ =
Revised: ⊤ f Θ ( X ) ∥ ∗ =

Original: ∥
Revised: ∥

Original: Diag (cid:0) f Θ ( X ) ⊤ f Θ ( X ) (cid:1)
Revised: Diag (cid:0) f Θ ( X ) ⊤ f Θ ( X ) (cid:1)

Original: ∥and the nuclear norm surrogate for the rank minimization reduces to the ℓ 1 norm surrogate for the cardinality (rank) minimization.
Revised: ∥and the nuclear norm surrogate for the rank minimization reduces to the ℓ 1 norm surrogate for the cardinality (rank) minimization.

Original: However, for the m -dimensional embedding, the solution of trace difference lies on a subspace of dimension less than m − 1 [3].
Revised: However, for the m -dimensional embedding, the solution of trace difference lies on a subspace of dimension less than m − 1 [3].

Original: Thus, the constraint Ω( f Θ ( X ))
Revised: Thus, the constraint Ω( f Θ ( X ))

Original: =
Revised: =

Original: B prevents the dimensional collapse, i.e ., f Θ ( X ) ⊤ f Θ ( X ) =
Revised: B prevents the dimensional collapse, i.e ., f Θ ( X ) ⊤ f Θ ( X ) =

Original: I .
Revised: I .

Original: Compared with the trace-based relaxation, LogDet is more suitable for cardinality minimization as it is less sensitive to large singular values.
Revised: Compared with the trace-based relaxation, LogDet is more suitable for cardinality minimization as it is less sensitive to large singular values.

Original: Also, the difference of LogDet terms does not require decorrelation of features to prevent the dimensional collapse.
Revised: Also, the difference of LogDet terms does not require decorrelation of features to prevent the dimensional collapse.

Original: We discuss this matter in Appendix A.
Revised: We discuss this matter in Appendix A.

Original: In our case, the difference of LogDet terms is always bounded by the difference of trace terms as follows.
Revised: In our case, the difference of LogDet terms is always bounded by the difference of trace terms as follows.

Original: Proposition 2.
Revised: Proposition 2.

Original: Given an embedding matrix f Θ ( X ) ∈ R n × m , a fixed small constant α > 0 , we have the following inequality:
Revised: Given an embedding matrix f Θ ( X ) ∈ R n × m , a fixed small constant α > 0 , we have the following inequality:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Proposition 2 is also related to the inequality Rank( S ) ≤ log det( I + S ) ≤
Revised: Proposition 2 is also related to the inequality Rank( S ) ≤ log det( I + S ) ≤

Original: Tr( S ) [7].
Revised: Tr( S ) [7].

Original: 5.2 Distance between Symmetric Positive (Semi-)Definite Matrices.
Revised: 5.2 Distance between Symmetric Positive (Semi-)Definite Matrices.

Original: Below, we provide a perspective on non-Euclidean distances between matrices from S m
Revised: Below, we provide a perspective on non-Euclidean distances between matrices from S m

Original: + ( + ) to compare the proposed method with other graph embeddings, e.g ., Laplacian Eigenmaps [3] and COLES [62].
Revised: + ( + ) to compare the proposed method with other graph embeddings, e.g ., Laplacian Eigenmaps [3] and COLES [55].

Original: For clarity, we also reformulate the Laplacian eigenmaps and COLES into forms in Prop. 3 and 4.
Revised: For clarity, we also reformulate the Laplacian eigenmaps and COLES into forms in Prop. 3 and 4.

Original: Proposition 3.
Revised: Proposition 3.

Original: Laplacian Eigenmaps [3] method equals to maximizing the Frobenius norm:
Revised: Laplacian Eigenmaps [3] method equals to maximizing the Frobenius norm:

Original: [Equation]
Revised: [Equation]

Original: Proposition 4.
Revised: Proposition 4.

Original: Contrastive Laplacian Eigenmaps [62] equals to maximizing the difference of
Revised: Contrastive Laplacian Eigenmaps [55] equals to maximizing the difference of

Original: Frobenius norm terms:
Revised: Frobenius norm terms:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Note that Eq.
Revised: Note that Eq.

Original: 9 encourages the linear kernel matrix f Θ ( X ) f Θ ( X ) ⊤ to be close to ˜ W w while Eq.encourage the linear kernel matrix to be far from the ˜ W w at the same time.
Revised: 9 encourages the linear kernel matrix f Θ ( X ) f Θ ( X ) ⊤ to be close to ˜ W w while Eq.encourage the linear kernel matrix to be far from the ˜ W w at the same time.

Original: Our loss follows the non-Euclidean geometry.
Revised: Our loss follows the non-Euclidean geometry.

Original: Below, we demonstrate the relation of Eq.
Revised: Below, we demonstrate the relation of Eq.

Original: 4 to the
Revised: 4 to the

Original: Affine-invariant Riemannian metric (AIRM).
Revised: Affine-invariant Riemannian metric (AIRM).

Original: Indeed, our loss function is bounded from both sides by
Revised: Indeed, our loss function is bounded from both sides by

Original: AIRM and AIRM scaled by √ m respectively.
Revised: AIRM and AIRM scaled by √ m respectively.

Original: Proposition 5.
Revised: Proposition 5.

Original: Let σ ( S ) be the vector of eigenvalues of S , for any matrix S t , S w ∈ S m + ( + ) , we have: [Equation] log
Revised: Let σ ( S ) be the vector of eigenvalues of S , for any matrix S t , S w ∈ S m + ( + ) , we have: ∥ log(

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given A = I + S t and B = I + S w , we have: log
Revised: Given A = I + S t and B = I + S w , we have: log

Original: [Equation]
Revised: [Equation]

Original: We have Tr( A ) =
Revised: We have Tr( A ) =

Original: ∥ σ ( A ) ∥ 1 , ∥
Revised: ∥ σ ( A ) ∥ 1 , ∥

Original: A ∥ F = ∥ σ ( A ) ∥ 2 and ∥ x ∥ 2 ≤ ∥ x ∥ 1 ≤ √ m ∥ x
Revised: A ∥ F = ∥ σ ( A ) ∥ 2 and ∥ x ∥ 2 ≤ ∥ x ∥ 1 ≤ √ m ∥ x

Original: ∥ 2 .
Revised: ∥ 2 .

Original: Thus, Eq. 4 is  an approximation of AIRM  between the total scatter matrix and the within-class matrix.Prop. 6 relates the LogDet difference to the regularized Jensen Bregman LogDet Divergence (JBLD). Definition 1.
Revised: Thus, Eq. 4 is trying to find a mapping function maximizing an approximation of AIRM distance between the total scatter matrix and the within-class matrix.

Original: Given A , B ∈ S m + ( + ) , the regularized Jensen Bregman LogDet Divergence is:

Original: [Equation]

Original: A , B

Original: [Equation]

Original: α

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: where I can be interpreted as a regularization to prevent det( A )

Original: [-= 
-]
Revised: [-= 
-]

Original: and log det( A ) =

Original: −∞ .

Original: Proposition 6.

Original: For any S w , S t ∈ S m + ( + ) and α ≥ 0 , we have:

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Also, δ (cid:16) S t , S w ; α, 12 (cid:17) ≥ J ( S w , S b ; α ) − m log 12 where the equality holds if log det( I + α S b )

Original: = 0 .

Original: Proof.

Original: Based on Eq.

Original: 14, we notice JBLD is a lower bound of our loss:

Original: [Equation] 5.3 Relationship of the LogDet model to the Schatten norm
Revised: 5.3 Relationship of the LogDet model to the Schatten norm

Original: Below we demonstrate the relationship between the LogDet, Trace and Rank operators, respectively, under the Schatten norm [29] framework.
Revised: Below we demonstrate the relationship between the LogDet, Trace and Rank operators, respectively, under the Schatten norm [28] framework.

Original: Essential is the following family of objective functions:
Revised: Essential is the following family of objective functions:

Original: [Equation]
Revised: [Equation]

Original: where σ i ( S ) , i = 1 , . .
Revised: where σ i ( S ) , i = 1 , . .

Original: , m , are the eigenvalues of either S t ∈ S m
Revised: , m , are the eigenvalues of either S t ∈ S m

Original: + ( + ) or S w ∈ S m + ( + ) , which are the total scatter matrix and the within scatter matrix from our experiments, respectively.
Revised: + ( + ) or S w ∈ S m + ( + ) , which are the total scatter matrix and the within scatter matrix from our experiments, respectively.

Original: Moreover, we define a normalization constant c where c = 1 or c = log( α + γ ) as detailed below.
Revised: Moreover, we define a normalization constant c where c = 1 or c = log( α + γ ) as detailed below.

Original: Given c = 1 , we have:
Revised: Given c = 1 , we have:

Original: [Equation]
Revised: [Equation]

Revised: ,γ ( S ) where S γ,p ( S ) = (cid:18) m (cid:88)

Revised: i =

Original:  From the asymptotic analysis, we conclude that  LogDet is an arbitrarily accurate rational approximation of ℓ 0 (the  pseudo-norm counting non-zero elements) over the eigenvalues ofS .
Revised: ( σ i ( S [Equation] γ ) p ) (cid:19) 1 /p . From the asymptotic analysis, we conclude that the LogDet is an arbitrarily accurate rational approximation of ℓ 0 (the so-called pseudo-norm counting non-zero elements) over the eigenvalues of

Revised: S .

Original: The case p = 1 yields the nuclear norm (trace) which makes the ‘smoothed’ rank difference of GLEN become equivalent of COLES.
Revised: The case p = 1 yields the nuclear norm (trace) which makes the ‘smoothed’ rank difference of GLEN become equivalent of COLES.

Original: The opposing limit case,  p = 0, recovers the LogDet formula.
Revised: The opposing limit case, denoted as p = 0 recovers the LogDet formula.

Original: One can also recover the exact Rank from the LogDet formulation by:
Revised: One can also recover the exact Rank from the LogDet formulation by:

Original: [Equation]
Revised: [Equation]

Original: This is apparent because:
Revised: This is apparent because:

Original: [Equation]
Revised: [Equation]

Original:  Experiments
Revised:  Experiments

Original: We evaluate GLEN (itsLogDet relaxation) onthe transductive  node classification task and node clustering.
Revised: We evaluate GLEN (its relaxation) on transductive and inductive node classification tasks and node clustering.

Original: GLEN is compared to popular unsupervised, contrastive, and (semi-)supervised approaches.
Revised: GLEN is compared to popular unsupervised, contrastive, and (semi-)supervised approaches.

Original: Except for the classifier, unsupervised models do not use labels.
Revised: Except for the classifier, unsupervised models do not use labels.

Original: To learn similarity/dissimilarity, contrastive models employ the contrastive setting.
Revised: To learn similarity/dissimilarity, contrastive models employ the contrastive setting.

Original: Labels are used in training the projection layer and classifier in semi-supervised models.
Revised: Labels are used to train the projection layer and classifier in semi-supervised models.

Original: A fraction of nodes ( i.e ., 5 or 20 per class) used in training are labeled in the semi-supervised setting.
Revised: A fraction of nodes ( i.e ., 5 or 20 per class) used for training are labeled for semi-supervised setting.

Original: A SoftMax classifier is used by (semi-)supervised models, while a logistic regression classifier is used by unsupervised and contrastive approaches.
Revised: A SoftMax classifier is used for (semi-)supervised models, while a logistic regression classifier is used for unsupervised and contrastive approaches.

Original: See
Revised: See

Original: Appendix E for implementation details.
Revised: Appendix E for implementation details.

Original: Datasets.
Revised: Datasets.

Original: GLEN is evaluated on four citation networks: Cora, Citeseer, Pubmed, Cora Full [17, 4] for transductive setting.
Revised: GLEN is evaluated on four citation networks: Cora, Citeseer, Pubmed, Cora Full [17, 4] for transductive setting.

Original: We also employ the large scale Ogbn-arxiv from OGB [14].
Revised: We also employ the large scale Ogbn-arxiv from OGB [14].

Original: See Appendix D for details of datasets.
Revised: See Appendix D for details of datasets.

Original: Metrics.
Revised: Metrics.

Original: As fixed data splits [48] often  benefit models that overfit, we average results over 50 random splits for each dataset.
Revised: As fixed data splits [45] often on transductive models benefit models that overfit, we average results over 50 random splits for each dataset.

Original: We evaluate performance for 5 and 20 samples per class.
Revised: We evaluate performance for 5 and 20 samples per class.

Original: Nonetheless, we also evaluate our model on the standard splits.
Revised: Nonetheless, we also evaluate our model on the standard splits.

Original: Baseline models.
Revised: Baseline models.

Original: We group baseline models into unsupervised, contrastive and (semi-)supervised methods, and implement them in the same framework/testbed.
Revised: We group baseline models into unsupervised, contrastive and (semi-)supervised methods, and implement them in the same framework/testbed.

Original: Contrastive methods include DeepWalk [32], GCN+SampledNCE developed as an alternative to GraphSAGE+SampledNCE [10],
Revised: Contrastive methods include DeepWalk [31], GCN+SampledNCE developed as an alternative to GraphSAGE+SampledNCE [10],

Original:  Graph2Gauss [4], SCE [54], DGI [40], GRACE [63], GCA [64], GraphCL [49] and COLES [62], which are our main competitors.
Revised: [Table] Graph2Gauss [4], SCE [47], DGI [37], GRACE [56], GCA [57], GraphCL [46] and COLES [55], which are our main competitors.

Original: Note that GRACE, GCA and GraphCL are based on multi-view and data augmentation, and GraphCL is mainly intended for graph classification.
Revised: Note that GRACE, GCA and GraphCL are based on multi-view and data augmentation, and GraphCL is mainly intended for graph classification.

Original: We do not study graph classification as it requires advanced node pooling with mixed- or high-order statistics [43, 19, 20].
Revised: We do not study graph classification as it requires advanced node pooling with mixed- or high-order statistics [40, 19, 20].

Original: We compare results with representative (semi-)supervised GCN [17], GAT [40] and MixHop [1] models.
Revised: We compare results with representative (semi-)supervised GCN [17], GAT [37] and MixHop [1] models.

Original: SGC and S 2 GC are unsupervised spectral filter networks.
Revised: SGC and S 2 GC are unsupervised spectral filter networks.

Original: They do not have any learnable parameters.
Revised: They do not have any learnable parameters.

Original: COLES and GLEN can be regarded as dimension reduction techniques for SGC and
Revised: COLES and GLEN could be regarded as dimension reduction techniques for SGC and

Original: S 2 GC, thus we compare them to PCA-S 2 GC and RP-S 2 GC, which use PCA and random projections to obtain the projection layer.
Revised: S 2 GC, thus we compare them to PCA-S 2 GC and RP-S 2 GC, which use PCA and random projections to obtain the projection layer.

Original: We set hyperparameters based on the settings described in prior papers.
Revised: We set hyperparameters based on the settings described in prior papers.

Original: 6.1 Transductive Learning
Revised: 6.1 Transductive Learning

Original: Below, we consider transductive learning where all nodes are available in the training process.
Revised: In this section, we consider transductive learning where all nodes are available in the training process.

Original: COLES vs .
Revised: COLES vs .

Original: GLEN.
Revised: GLEN.

Original: Table 1 shows the performance of GLEN vs .
Revised: Table 1 shows the performance of GLEN vs .

Original: COLES on two different backbones, i.e ., GCN and S 2 GC.
Revised: COLES on two different backbones, i.e ., GCN and S 2 GC.

Original: On both backbones, GLEN shows non-trivial improvements on all four datasets.
Revised: On both backbones, GLEN shows non-trivial improvements on all four datasets.

Original: GLEN-S 2 GC outperforms  COLES by up to 4.6%.
Revised: GLEN-S 2 GC outperforms the COLES by up to 4.6%.

Original: Table 2 evaluates GLEN on Cora, Citeseer,
Revised: Table 2 evaluates GLEN on Cora, Citeseer,

Original: PubMed on the standard splits instead of the random splits.
Revised: PubMed on the standard splits instead of the random splits.

Original: See Appendix G for comparisons to additional contrastive learning frameworks.
Revised: See Appendix G for comparisons to additional contrastive learning frameworks.

Original: Contrastive Embedding Baselines vs .
Revised: Contrastive Embedding Baselines vs .

Original: GLEN.
Revised: GLEN.

Original: Table 1 shows that GLEN-GCN and GLEN-S 2 GC outperform unsupervised models.
Revised: Table 1 shows that GLEN-GCN and GLEN-S 2 GC outperform unsupervised models.

Original: In particular, GLEN-GCN outperforms GCN+SampledNCE on all four datasets, which shows that GLEN has an advantage over the SampledNCE framework.
Revised: In particular, GLEN-GCN outperforms GCN+SampledNCE on all four datasets, which shows that GLEN has an advantage over the SampledNCE framework.

Original: In addition, GLEN-S 2 GC outperforms the best contrastive baseline DGI by up to 3.4%.
Revised: In addition, GLEN-S 2 GC outperforms the best contrastive baseline DGI by up to 3.4%.

Original: On Cora withtraining samples, GLEN-S 2 GC outperforms S 2 GC by 6.8%.
Revised: On Cora withtraining samples, GLEN-S 2 GC outperforms S 2 GC by 6.8%.

Original: Finally, Table 3 shows that GLEN-S 2 GC (small number of trainable parameters) outperforms other methods on the challenging Ogbn-arxiv.
Revised: Finally, Table 3 shows that GLEN-S 2 GC (small number of trainable parameters) outperforms other methods on the challenging Ogbn-arxiv.

Original: Semi-supervised GNNs vs .
Revised: Semi-supervised GNNs vs .

Original: GLEN.
Revised: GLEN.

Original: Table 1 shows that the contrastive GCN baselines perform worse than semi-supervised variants, especially when 20 labeled samples per class are available.
Revised: Table 1 shows that the contrastive GCN baselines perform worse than semi-supervised variants, especially when 20 labeled samples per class are available.

Original: In contrast, GLEN-GCN outperformed the semi-supervised GCN on Cora by 10% and 3.4% givenand 20 labeled samples per class.
Revised: In contrast, GLEN-GCN outperformed the semi-supervised GCN on Cora by 10% and 3.4% givenand 20 labeled samples per class.

Original: GLEN-GCN also outperforms GCN on Citeseer and Pubmed by 9.9% and 5.2% given 5 labeled samples per class.
Revised: GLEN-GCN also outperforms GCN on Citeseer and Pubmed by 9.9% and 5.2% given 5 labeled samples per class.

Original: These results show the superiority of GLEN on four datasets when the number of samples per class is 5.
Revised: These results show the superiority of GLEN on four datasets when the number of samples per class is 5.

Original: Even for 20 labeled samples per class,
Revised: Even for 20 labeled samples per class,

Original: GLEN-S 2 GC outperforms the best semi-supervised baselines on all four datasets e.g ., by 3.3% on
Revised: GLEN-S 2 GC outperforms the best semi-supervised baselines on all four datasets e.g ., by 3.3% on

Original: Cora.
Revised: Cora.

Original: Semi-supervised models ( e.g ., GAT and MixHop) are affected by the low number of labeled samples, which is consistent with [25].
Revised: Semi-supervised models ( e.g ., GAT and MixHop) are affected by the low number of labeled samples, which is consistent with [25].

Original: The accuracy of GLEN-GCN and GLEN-S 2 GC is unaffected.
Revised: The accuracy of GLEN-GCN and GLEN-S 2 GC is unaffected.

Original: Unsupervised GNNs vs .
Revised: Unsupervised GNNs vs .

Original: GLEN.
Revised: GLEN.

Original: SGC and S 2 GC are unsupervised linear networks based on spectral filters which do not use labels (except for the classifier).
Revised: SGC and S 2 GC are unsupervised linear networks based on spectral filters which do not use labels (except for the classifier).

Original:  GLEN helps both methods reduce the dimension and achievegood discriminative features.
Revised: As a dimension reduction method, GLEN helps both methods reduce the dimension and achieve discriminative features.

Original: Table 1 shows thatGLEN-S 2 GC outperforms
Revised: Table 1 shows that

Original:  RP-S 2 GC and PCA-S 2 GC under the same projection size.
Revised: GLEN-S 2 GC outperforms RP-S 2 GC and PCA-S 2 GC under the same projection size.

Original: GLEN-S 2 GC also outperforms the unsupervised S 2 GC baseline (high-dimensional representation). 
Revised: GLEN-S 2 GC also outperforms the unsupervised S 2 GC baseline (high-dimensional representation). [Table] [Table] [Table]

Revised: [Table]

Original: 6.2 Node Clustering
Revised: 6.2 Node Clustering

Original: We compare GLEN-GCN and GLEN-S 2 GC with three types of clustering methods:
Revised: We compare GLEN-GCN and GLEN-S 2 GC with three types of clustering methods:

Original: i. Methods that use only node features, e.g ., k-means and spectral clustering (spectral-f) construct a similarity matrix with the node features by alinear kernel.
Revised: i. Methods that use only node features e g ., k-means and spectral clustering (spectral-f) construct a similarity matrix with the node features by a

Revised: linear

Revised: kernel.

Original: ii.
Revised: ii.

Original: Structural clustering methods that only use the graph structure: spectral clustering (spectral-g) that takes the graph adjacency matrix as the similarity matrix, and DeepWalk [32] [Table]
Revised: Structural clustering methods that only use the graph structure: spectral clustering (spectral-g) that takes the graph adjacency matrix as the similarity matrix, and DeepWalk [31].

Original: iii.
Revised: iii.

Original: Attributed graph clustering methods that use node features and the graph: Graph Autoencoder (GAE), Graph Variational Autoencoder (VGAE)
Revised: Attributed graph clustering methods that use node features and the graph: Graph Autoencoder (GAE), Graph Variational Autoencoder (VGAE)

Original: ], Adversarially Regularized Graph Autoencoder (ARGE), Var.
Revised: [17], Adversarially Regularized Graph Autoencoder (ARGE), Var.

Original: Graph Autoencoder (ARVGE) [31] , SGC  S 2 GC [60], COLES [62].
Revised: Graph Autoencoder (ARVGE) [30], SGC [42] , S 2 GC [53], COLES [55].

Original: We measure and report the clustering Accuracy (Acc), Normalized Mutual Information (NMI) and macro F1-score (F1).
Revised: We measure and report the clustering Accuracy (Acc), Normalized Mutual Information (NMI) and macro F1-score (F1).

Original: We run each method 10 times on Cora, CiteSeer and PubMed.
Revised: We run each method 10 times on Cora, CiteSeer and PubMed.

Original: Wefollow [55] and set the number of propagation steps to 8 for SGC, S 2 GC, COLES-S 2 GC and COLES-S 2 GC.
Revised: We set the number of propagation steps to 8 for SGC, S 2 GC, COLES-S 2 GC and COLES-S 2 GC following [48].

Original: Table 4 shows that GLEN-S 2 GC outperforms other methods in all cases, whereas GLEN-GCN outperforms COLES-GCN, COLES-GCN (Stiefel) and contrastive GCN on all datasets.
Revised: Table 4 shows that GLEN-S 2 GC outperforms other methods in all cases, whereas GLEN-GCN outperforms COLES-GCN, COLES-GCN (Stiefel) and contrastive GCN on all datasets.

Original: 6.3 Comparison of Surrogates of Rank
Revised: 6.3 Comparison of Surrogates of Rank

Original: Table 5  shows results on four additional surrogates of Rank ( S ) :
Revised: Table 5 above shows results on four additional surrogates of Rank ( S ) :

Original: • Nuclear norm: R NN ( S ) =
Revised: • Nuclear norm: R NN ( S ) =

Original: (cid:80) i σ i ( S ) .
Revised: (cid:80) i σ i ( S ) .

Original: • γ -nuclear norm [16]: R γ -NN =
Revised: • γ -nuclear norm [16]: R γ -NN =

Original: (cid:80) i (1+ γ ) σ i ( S ) γ
Revised: (cid:80) i (1+ γ ) σ i ( S ) γ

Original: + σ i ( S ) .
Revised: + σ i ( S ) .

Original: • S p norm [29]: R S p
Revised: • S p norm [28]: R S p

Original: = (cid:80) i σ i ( S ) p .
Revised: = (cid:80) i σ i ( S ) p .

Original: • Geman norm [8]: R Geman = (cid:80) i σ i ( S ) γ + σ i ( S ) .
Revised: • Geman norm [8]: R Geman = (cid:80) i σ i ( S ) γ + σ i ( S ) .

Original: 6.4 Transductive One-shot Learning for Image Recognition
Revised: 6.4 Transductive One-shot Learning on Image Classification Datasets

Original: The most common setting in few-shot learning [35, 37, 50, 26, 53, 51, 52] is the inductive setting.
Revised: The most common setting in FSL is the inductive setting.

Original: In such a scenario, only samples in the support set can be used to fine-tune the model or learn a function for the inference of query labels.
Revised: In such a scenario, only samples in the support set can be used to fine-tune the model or learn a function for the inference of query labels.

Original: In contrast, in the transductive scenario, the model has access to all the query data (unlabeled) that needs to be classified.
Revised: In contrast, in the transductive scenario, the model has access to all the query data (unlabeled) that needs to be classified.

Original: EASE [61] is a transductive few-shot learner for so-called episodic image classification.
Revised: EASE [54] is a transductive few-shot learner for so-called episodic image classification.

Original: Given feature matrix Z ∈ R n × m from a CNN backbone (ResNet-12), EASE minimizes Tr ( UZ ⊤ L w ZU ⊤ ) −
Revised: Given feature matrix Z ∈ R n × m from a CNN backbone (ResNet-12), EASE minimizes Tr ( UZ ⊤ L w ZU ⊤ ) −

Original: Tr ( UZ ⊤ L t ZU ⊤ ) (subject to UU ⊤ = I ) in order to learn a linear projection U .
Revised: Tr ( UZ ⊤ L t ZU ⊤ ) (subject to UU ⊤ = I ) in order to learn a linear projection U .

Original: We extendEASE with the objective of GLEN to  learn the linear projection U by minimizing log det( UZ ⊤ L w ZU ⊤ ) − log det( UZ ⊤ L t ZU ⊤ ) (subject to UU ⊤ =
Revised: We extend GLEN to EASE to learn the linear projection U by minimizing log det( UZ ⊤ L w ZU ⊤ ) − log det( UZ ⊤ L t ZU ⊤ ) (subject to UU ⊤ =

Original: I .
Revised: I .

Original: We also apply the S p norm instead of LogDet.
Revised: We also apply the S p norm instead of log det .

Original: Table 6 shows the results of EASE with the LogDet-based relaxation and the S p -norm based relaxation of GLEN.
Revised: Tableshows the results of EASE based on the LogDet and the S p -norm based relaxations of GLEN.

Original: For the simplicity ofthe experiment, we use soft k-means rather than Sinkhorn k-means as in theoriginal EASE pipeline.
Revised: For the simplicity of experiment, we use soft k-means rather than Sinkhorn k-means as in the EASE pipeline.

Original: See EASE [61] for the experimental setting
Revised: Please refer to EASE [54] for the experimental setup of one-shot learning.

Original: [Table] We evaluate our approach on four few-shot classification benchmarks, mini-ImageNet [41], tieredImageNet [33], CUB [44] [Table] and CIFAR-FS [21].
Revised: We evaluate our approach on four few-shot classification benchmarks, mini-ImageNet [38], tieredImageNet [32], CUB [41], and CIFAR-FS [21].

Original: The performance numbers are given as accuracy (%) and the 0.95 confidence intervals are reported.
Revised: The performance numbers are given as accuracy % and the 0.95 confidence intervals are reported.

Original: We use publicly available pre-trained ResNet-12 [30] that is trained on the base-class training split. [Table] [Table] [Table]
Revised: We use publicly available pre-trained ResNet-12 [29] that are trained on the base class training set.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Scalability.
Revised: Scalability.

Original: GraphSAGE and DGI require neighbor sampling with redundant forward/backward steps (long runtime).
Revised: GraphSAGE and DGI require neighbor sampling with redundant forward/backward steps (long runtime).

Original: In contrast, GLEN-S 2 GC enjoys a simple implementation with low memory usage/low runtime.
Revised: In contrast, GLEN-S 2 GC enjoys a simple implementation with low memory usage/low runtime.

Original: For graphs with over  thousandsof nodes and  millions edges (OGB benchmark),
Revised: For graphs with over 100 thousands nodes and 10 millions edges (Reddit),

Original: GLEN runs fast on NVIDIA 1080 GPU.
Revised: GLEN runs fast on NVIDIA 1080 GPU.

Original: Even on larger graph benchmarks, GLEN is fast as it optimizes the total scatter and the within-class matrices whose size depends onthe embedding size rather than the node number.
Revised: Even on larger graph benchmarks, GLEN is fast as it optimizes the total scatter and the within-class matrices whose size depends on embedding size rather than the node number.

Original: The runtime of GLEN-S 2 GC is also favourable in comparison to multi-view augmentation-based GraphCL.
Revised: The runtime of GLEN-S 2 GC is also favourable in comparison to multi-view augmentation-based GraphCL.

Original: Specifically, GLEN-S 2 GC took 0.54s, 0.3s, 5.3s and 15.4s on Cora, Citeseer, Pubmed and Cora Full, respectively.
Revised: Specifically, GLEN-S 2 GC took 0.54s, 0.3s, 5.3s and 15.4s on Cora, Citeseer, Pubmed and Cora Full, respectively.

Original: GraphCL took 110.19s, 101.0s, ≥ 8 h and ≥ 8 h respectively.
Revised: GraphCL took 110.19s, 101.0s, ≥ 8 h and ≥ 8 h respectively.

Original: Although the LogDet difference is somewhat slower than the trace difference in forward/backward propagation, it converges faster, thus enjoying a similar low runtime.
Revised: Although the LogDet difference is somewhat slower than the trace difference in forward/backward propagation, it converges faster, thus enjoying a similar low runtime.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we model contrastvie learning as a rank difference problem to approximate the condition that the rank of total scatter matrix should equal the sum of ranks of within-scatter and between-scatter matrices.
Revised: In this paper, we model contrastvie learning as a rank difference problem to approximate the condition that the rank of total scatter matrix should equal the sum of ranks of within-scatter and between-scatter matrices.

Original: We relax this NP-hard problem with a differentiable difference of LogDet terms.
Revised: We relax this NP-hard assumption with a differentiable difference of LogDet terms.

Original: We also show two perspectives on GLEN and the existing methods based on the low-rank optimization and distance between symmetric positive (semi-)definite  matrices.
Revised: We also show two perspectives on GLEN and the existing methods based on the low-rank optimization and distance between symmetric positive (semi-)definite matrices matrices.

Original: In low-rank optimization, we explain why the LogDet difference is a better surrogate  to optimize rank difference compared to the trace difference.
Revised: In low-rank optimization, we explain why the LogDet difference is a better surrogate function to optimize rank difference compared to the trace difference.

Original: We also show that our solution encourages linear kernel of embeddings become the geometric mean between the total scatter matrix and the within-classscatter matrix.
Revised: We also show that our solution encourages linear kernel of embeddings become the geometric mean between the total scatter matrix and the within-class matrix.

Original: GLEN works well with many backbones and outperforms many unsupervised, contrastive and (semi-)supervised methods.
Revised: GLEN works well with many backbones outperforming many unsupervised, contrastive and (semi-)supervised methods.

Revised: Acknowledgments and Disclosure of Funding

Revised: We thank reviewers for stimulating questions that helped us improve several aspects of our analysis.

Revised: Hao Zhu is supported by an Australian Government Research Training Program (RTP) Scholarship.

Revised: Piotr Koniusz is supported by CSIRO’s Machine Learning and Artificial Intelligence Future Science

Revised: Platform (MLAI FSP).

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?.
Revised: [Yes] See Section ?.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: i. For
Revised: i. For

Original: all
Revised: all

Original: authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] ii.
Revised: [Yes] ii.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] iii.
Revised: [Yes] iii.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details ( e.g ., data splits, hyperparameters, how they were chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars ( e.g ., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used ( e.g ., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] iv.
Revised: [Yes] iv.

Original: If you are using existing assets ( e.g ., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: v. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: v. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] Generalized Laplacian Eigenmaps (Supplementary Material)
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: Acknowledgments and Disclosure of Funding

Original: We thank reviewers for stimulating questions that helped us improve several aspects of our analysis.

Original: Hao Zhu is supported by an Australian Government Research Training Program (RTP) Scholarship.

Original: Piotr Koniusz is supported by CSIRO’s Machine Learning and Artificial Intelligence Future Science

Original: Platform (MLAI FSP).

Original: A Spectrum Analysis

Original: Below we analyze the spectrum of embedding matrix f Θ ( X ) for several different objectives to show that GLEN enjoys better embedding properties than COLES [72].

Original: Without decorrelation term ∥ f Θ ( X ) ⊤ f Θ ( X ) − I ∥ 2 F used by COLES, the trace-based objective in

Original: Eq.

Original: 6 optimization may results in the so-called dimensional collapse which means that the vast majority of singular values of embeddings are zeroed and only one or two leading components are non-zero.

Original: Fig.

Original: 2b shows that in COLES the sixth singular value and the following singular values are almost equal zero, whereas the first singular value is over 2600 in value, thus dominating the spectrum.

Original: Fig.

Original: 2a shows that our GLEN based on the LogDet difference of terms performs better than COLES.

Original: Firstly, the first leading singular value is not as large as COLES (only 80) so it does not dominate the spectrum.

Original: The next singular values follow a gradual decline.

Original: It is worth noting that even with the decorrelation term ∥ f Θ ( X ) ⊤ f Θ ( X ) − I ∥ 2 F used by COLES, it only helps the first five leading singular values, and the first singular value is still larger than the counterpart of GLEN (393 vs . 80).

Original: [Figure]

Original: For comparison purposes, we truncate the range of singular values for COLES.

Original: The first singular value of COLES is 2650 (very large value).

Original: The range of singular values with the use of decorrelation term is 393.

Original: The decorrelation term penalizes ∥ f Θ ( X ) ⊤ f Θ ( X ) − I ∥ 2 F .

Original: Notice that the spectrum of

Original: GLEN is more balanced.

Original: B Nuclear Norm vs . LogDet for Rank Minimization

Original: In our paper, to facilitate the analysis we assume Rank( S ) ≤ log det( I + S ) ≤

Original: Tr( S ) .

Original: Below we clarify the necessary conditions for this inequality to hold true.

Original: Proposition 7.

Original: Let S ∈ S

Original: m + ( + ) , if Tr( S ) ≥

Original: m then Rank( S ) ≤ log det( I + S ) ≤

Original: Tr( S ) .

Original: Proof.

Original: It is easy to observe that Rank( S ) ≤

Original: Tr( S ) if Tr( S ) ≥ m .

Original: Moreover, we have Tr( S ) >

Original: Tr(log

Original: ( I + S ))

Original: = log det( I + S ) .

Original: Let Rank( S ) = n ≤ m ≤ Tr( S ) , then we have log(1+Tr( S )) ≤ (cid:80) mi log(1+ σ i ( S ))

Original: ≤ (cid:80) mi log( I +Tr( S ) /m ) and (cid:80) mi log(1+Tr( S ) /m ) ≥ (cid:80)

Original: ni log(1+ m/n ) ≥ (cid:80) mi log(1 + m/m ) ≥ Rank( S ) .

Original: C Gradient Analysis

Original: Proposition 8.

Original: The objective of optimization, f Θ ( X ) ⊤ L t = f Θ ( X ) ⊤ L w , is achieved when the following gradient rule is zeroed.

Original: Proof.

Original: [Equation]

Original: [Equation]

Original: log

Original: [Equation]

Original: ∂

Original: [Equation]

Original: log det (cid:0) I + αf Θ ( X ) ⊤ L w f Θ ( X ) (cid:1) ∂ Z = 0 ⇔ (cid:0) I + αf Θ ( X )

Original: ⊤ L t f Θ ( X ) (cid:1) − 1 f Θ ( X )

Original: ⊤ L t − (cid:0)

Original: I + αf Θ ( X )

Original: ⊤ L w f Θ ( X ) (cid:1) − 1 f Θ ( X )

Original: ⊤ L w = 0 .

Original: D Reproducibility

Original: [Table]

Original: D.1 Datasets

Original: Cora is a well-known citation network that is labelled by paper topic.

Original: The majority of approaches report on a subset of this dataset.

Original: The Cora dataset contains 2708 scientific publications divided into seven categories.

Original: The citation network has 5429 links.

Original: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding dictionary word.

Original: The dictionary contains 1433 distinct words.

Original: Cora Full is made up of 19793 scientific publications divided into seventy categories.

Original: The citation network has 65311 links.

Original: The dictionary containsdistinct words.

Original: CiteSeer contains 3312 scientific papers divided into six categories.

Original: The citation network haslinkages.

Original: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the relevant dictionary word.

Original: The dictionary has 3703 distinct terms.

Original: Pubmed is comprised of 19717 scientific papers on diabetes from the PubMed database, grouped into one of three categories.

Original: The citation network has 44338 linkages.

Original: Each publication in the dataset is described by a TF/IDF weighted word vector drawn from a vocabulary of 500 distinct terms.

Original: Ogbn-arxiv is the citation network represented by a directed graph of all Computer Science (CS) arXiv papers indexed by MAG [69].

Original: Each node represents an arXiv paper, and each directed edge indicates that one paper references another.

Original: Each paper includes a 128-dimensional feature vector generated by averaging the embeddings of words in the title and abstract.

Original: Individual word embeddings are produced by running the skip-gram model [68] over the MAG corpus.

Original: E Implementation Details

Original: GLEN is implemented in PyTorch.

Original: The propagation procedure is implemented with sparse-dense matrix multiplications.

Original: The codes of GCN, COLES-GCN, SGC, COLES-SGC, S 2 GC and COLESS 2 GC are also implemented with PyTorch.

Original: The weight matrices of classifier are initialized with

Original: Glorot normal initializer.

Original: We employ Adam [66] to optimize parameters of the proposed methods and adopt early stopping to control the training epochs based on the validation loss.

Original: For all experiments, we use Adam to optimize Eq.

Original: 5 because our method does not have a closed-form solution as COLES does.

Original: All the experiments in this paper are conducted on a single NVIDIA GeForce RTX 1080 with

Original: GB memory on a PC with Unbuntu 18.04.

Original: We use Python 3.7.3, PyTorch 1.10.0, and CUDA 10.2.

Original: E.1 Hyperparameters

Original: Tables 8 and 9 summarize hyperparameters.

Original: GLEN is not highly sensitive to different hyperparameters.

Original: The only hyperparameter for SGC and S 2 GC is the aggregation step K .

Original: As a result, we use K =for the majority of benchmarks.

Original: Except for Ogbn-arxiv, all contrastive-based approaches use logistic regression as the classifier.

Original: It is worth noting that we do not modify any logistic regression parameters and instead use the default settings.

Original: Ogbn-arxiv exhibits non-linear feature characteristics because

Original: Bag-of-Words was used by the authors to form embeddings.

Original: As a result, the MLP classifier is chosen for GLEN-S 2 GC on this dataset.

Original: We specifically preserve the MLP classifier configuration from the baseline.

Original: The learning rate for the MLP is 0.005 and the dropout rate is 0.4.

Original: [Table]

Original: [Table]

Original: F Limitation and Broader Impact

Original: Although we propose a promising criterion to learn discriminant features, the rank difference problem is NP-hard.

Original: Choosing and especially solving high-quality surrogates to the rank difference of terms is still an unsolved optimization problem that requires a further work within the optimization community.

Original: This paper discusses the design of feature space from a critical point of view.

Original: To this end, it proposes a novel generalized graph embedding framework which is very different compared to the traditional graph embedding frameworks ( e.g ., Laplacian Eigenmaps, Linear Discriminant Analysis or Contrastive Laplacian Eigenmaps).

Original: The scope of this framework is not limited to graph embedding.

Original: GLEN can help many downstream works which require learning discriminant representation in an unsupervised manner to improve the performance of downstream tasks.

Original: Node classification, object category recognition, metric learning and other downstream tasks can accelerate downstream training thanks to GLEN.

Original: More importantly, GLEN-S 2 GC is a lightweight energy-friendly network compared to heavy GCN-based GCL (see ‘Scalability’ section).

Original: Regarding negative impacts, we are not aware of any.

Original: GCL is an unsupervised learning paradigm which, by itself, cannot lead to any negative impact.

Original: G Additional Results

Original: Comparison with other frameworks.

Original: Below we redefine Local Preserving Projection (LPP) [65] and Deep Spectral Clustering (DSC) [67] within the COLES and GLEN frameworks.

Original: All models below are based on the S 2 GC backbone:

Original: • DSC extends Deep Spectral Clustering Learning [67] by minimizing Tr ( f Θ ( X )

Original: ⊤ L w f Θ ( X )) where f Θ is a two-layer neural network (MLP).

Original: Kindly note this is non-contrastive learning that only uses L w .

Original: • LPP is an extension of Locality Preserving Projections [65] that learns an orthogonal linear projection by minimizing Tr ( UX ⊤ L w XU ⊤ )) .

Original: Notice this is non-contrastive learning that only uses L w .

Original: • COLES-LPP is defined by us as minimizing Tr ( UX ⊤ L t XU ⊤ ))

Original: − Tr ( UX ⊤ L w XU ⊤ )) .

Original: • COLES-DSC minimizes

Original: Tr ( f Θ ( X ) ⊤ L t f Θ ( X ))

Original: − Tr ( f Θ ( X ) ⊤ L w f Θ ( X )) .

Original: Two MLP layers are added (as in the DSC model above) between the S 2 GC backbone and the loss (some models [70, 71] precede S 2 GC with an MLP).

Original: Their layer parameters are added to Θ .

Original: • GLEN-LPP is defined as log det( UX ⊤ L t XU ⊤ )) − log det( UX ⊤ L w XU ⊤ )) .

Original: • GLEN log det( f Θ ( X ) ⊤ L t f Θ ( X ))

Original: − log det( f Θ ( X ) ⊤ L w f Θ ( X )) also uses two MLP layers as in the DSC model.

Original: Table 10 below shows the results.

Original: [Table]
Revised: [N/A]

Original: H Derivation of GLEN from the SampledNCE framework

Original: COLES [72] extends Laplacian Eigenmaps:

Original: [Equation]

Original: where Z = [ z 1 , · · ·

Original: [Table]

Original: z n ] and Ω( Z ) are constraints ( i

Original: [Table]

Original: e ., orthogonality) by expanding the SampledNCE formulation:

Original: [Equation]

Original: Symbols p n and p d are negative and positive sampling distributions respectively.

Original: Moreover, s Θ ( v, u ) = log exp( u ⊤ v ) =

Original: u ⊤ v and ˜ s Θ ( v, u ′ )

Original: =

Original: log exp( − u ′⊤ v )

Original: = − u ′⊤ v are similarity measures, whereas η ≥ 0 controls the impact of negative sampling.

Original: GLEN generalizes SampledNCE, a framework for contrastive learning with positive and negative sampling, which relies on two terms:

Original: [Equation]

Original: The above two terms are evaluated over two different distributions u ∼ p

Original: d ( u | v ) (nodes u from the adjacency matrix) and u ′ ∼ p n ( u ′ | v ) (nodes u ′ from random negative adjacency matrix).

Original: Let us consider the positive sampling term (the negative sampling term can be expanded in the similar way).

Original: Let p d ( v ) =

Original: 1 √ D

Original: vv and p d ( u | v ) =

Original: ˆ W uv √ D uu where ˆ W is an unnormalized adjacency matrix and D is its degree matrix.

Original: Let W be the degree normalized matrix.

Original: Notice u and v are indexes of embeddings u and v .

Original: Let s ( · ) be defined as in COLES.

Original: Then:

Original: [Equation]

Original: where ϕ ( · ) is a pooling function, i.e ., ϕ ( M )

Original: =

Original: Tr ( M ) yields COLES:

Original: [Equation]

Original: where z i ⊥

Original: z

Original: [-j imposes orthogonality constraints of Laplacian eigenmaps and δ ( z ) = 
-]
Revised: [-j imposes orthogonality constraints of Laplacian eigenmaps and δ ( z ) = 
-]

Original: if z = 0 and δ ( z )

Original: [-= 
-]
Revised: [-= 
-]

Original: if z ̸ = 0 .

Original: Finally, think that rows of Z contain all u (and v ).

Original: We let the pooling operator ϕ ( · ) operate on the entire spectrum under a general aggregation scheme.

Original: A very general operator is ϕ ( M ) =

Original: Rank ( M ) from which we can recover the original trace (nuclear norm) of COLES or LogDet of GLEN, γ -nuclear, S p , and Geman norm, respectively.

Original: COLES uses the following expression based on SampledNCE:

Original: [Equation]

Original: where W + is a normalized adjacent matrix and W l, − are κ normalized randomized k -regular graphs (adjacency matrices), while L w and L t are the corresponding Laplacian matrices.

Original: Negative random sampling is represented by W − , e.g ., a randomized k -regular graph or several such graphs.

Original: If we sample κ → ∞ randomized k -regular graphs (adjacency matrices of size n × n ) (each row receives 1 with probability k/n ), the expectation of randomized graph (adjacency matrix) is given as:

Original: [Equation]

Original: (29) which by itself is a fully-connected graph with the graph Laplacian matrix L t

Original: =

Original: I − kn 11 ⊤ .

Original: We simply set k = 1 to use 1 -regular graphs for negative sampling so L t

Original: = I − 1 n 11 ⊤ .

Original: Thus, our contrastive term is equivalent of the total scatter matrix S t known from the Linear Discriminant

Original: Analysis, i.e ., S t =

Original: Z ⊤ ( I − 1 n 11 ⊤ ) Z

Original: =

Original: Z ⊤ L t Z .

Original: The positive sampling is encoded by the graph adjacency matrix L w .

Original: Thus, our GLEN is given as:

Original: [Equation]

Original: More precisely, if Z is produced by an encoder, i.e ., Z = f Θ ( X ) then we optimize the above problem w.r.t.

Original: Θ .