Original: E FFICIENT BLOCK CONTRASTIVE LEARNING VIA PARAMETER - FREE META - NODE APPROXIMATION
Revised: E FFICIENT BLOCK CONTRASTIVE LEARNING VIA PARAMETER - FREE META - NODE APPROXIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Contrastive learning has recently achieved remarkable success in many domains including graphs.
Revised: Contrastive learning has recently achieved remarkable success in many domains including graphs.

Original: However contrastive loss, especially for graphs, requires a large number of negative samples which is unscalable and computationally prohibitive with a quadratic time complexity.
Revised: However contrastive loss, especially for graphs, requires a large number of negative samples which is unscalable and computationally prohibitive with a quadratic time complexity.

Original: Sub-sampling is not optimal and incorrect negative sampling leads to sampling bias.
Revised: Sub-sampling is not optimal. Incorrect negative sampling leads to sampling bias.

Original: In this work, we propose a meta-node based approximation technique that can (a) proxy all negative combinations (b) in quadratic cluster size time complexity, (c) at graph level, not node level, and (d) exploit graph sparsity.
Revised: In this work, we propose a meta-node based approximation technique that can (a) proxy all negative combinations (b) in quadratic cluster size time complexity, (c) at graph level, not node level, and (d) exploit graph sparsity.

Original: By replacing node-pairs with additive cluster-pairs, we compute the negatives in cluster-time at graph level.
Revised: By replacing node-pairs with additive cluster-pairs, we compute the negatives in cluster-time at graph level.

Original: The resulting Proxy approximated meta-node Contrastive (PamC) loss, based on simple optimized GPU operations, captures the full set of negatives, yet is efficient with a linear time complexity.
Revised: The resulting Proxy approximated meta-node Contrastive (PamC) loss, based on simple optimized GPU operations, captures the full set of negatives, yet is efficient with a linear time complexity.

Original: By avoiding sampling, we effectively eliminate sample bias.
Revised: By avoiding sampling, we effectively eliminate sample bias.

Original: We meet the criterion for larger number of samples, thus achieving block-contrastiveness, which is proven to outperform pair-wise losses.
Revised: We meet the criterion for larger number of samples, thus achieving block-contrastiveness, which is proven to outperform pair-wise losses.

Original: We use learnt soft cluster assignments for the meta-node constriction, and avoid possible heterophily and noise added during edge creation.
Revised: We use learnt soft cluster assignments for the meta-node construction, and avoid possible heterophily and noise added during edge creation.

Original: Theoretically, we show that real world graphs easily satisfy conditions necessary for our approximation.
Revised: Theoretically, we show that real world graphs easily satisfy conditions necessary for our approximation.

Original: Empirically, we show promising accuracy gains over state-of-the-art graph clustering on 6 benchmarks.
Revised: Empirically, we show promising accuracy gains over state-of-the-art graph clustering on 6 benchmarks.

Original: Importantly, we gain substantially in efficiency; up to 3x in trainingtime, 1.8x in inference time and over 5x in GPU memory reduction. 
Revised: Importantly, we gain substantially in efficiency; up to 2x in training time and over 5x in GPU memory reduction. The code is publicly available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Discriminative approaches based on contrastive learning has been outstandingly successful in practice (Guo et al., 2017; Wang & Isola, 2020), achieving state-of-the-art results (Chen et al., 2020a) or at times outperforming even supervised learning (Logeswaran & Lee, 2018; Chen et al., 2020b).
Revised: Discriminative approaches based on contrastive learning has been outstandingly successful in practice (Guo et al., 2017; Wang & Isola, 2020), achieving state-of-the-art results (Chen et al., 2020a) or at times outperforming even supervised learning (Logeswaran & Lee, 2018; Chen et al., 2020b).

Original: Specifically in graph clustering, contrastive learning can outperform traditional convolution and attention-based Graph Neural Networks (GNN) on speed and accuracy (Kulatilleke et al., 2022).
Revised: Specifically in graph clustering, contrastive learning can outperform traditional convolution and attention-based Graph Neural Networks (GNN) on speed and accuracy (Kulatilleke et al., 2022).

Original: While traditional objective functions encourage similar nodes to be closer in embedding space, their penalties do not guarantee separation of unrelated graph nodes (Zhu et al., 2021).
Revised: While traditional objective functions encourage similar nodes to be closer in embedding space, their penalties do not guarantee separation of unrelated graph nodes (Zhu et al., 2021a).

Original: Differently, many modern graph embedding models (Hamilton et al., 2017; Kulatilleke et al., 2022), use contrastive objectives.
Revised: Differently, many modern graph embedding models (Hamilton et al., 2017; Kulatilleke et al., 2022), use contrastive objectives.

Original: These encourage representation of positive pairs to be similar, while making features of the negatives apart in embedding space (Wang & Isola, 2020).
Revised: These encourage representation of positive pairs to be similar, while making features of the negatives apart in embedding space (Wang & Isola, 2020).

Original: A typical deep model consists of a trainable encoder that generates positive and negative node embedding for the contrastive loss (Zhu et al., 2021).
Revised: A typical deep model consists of a trainable encoder that generates positive and negative node embedding for the contrastive loss (Zhu et al., 2021a).

Original: It has been shown that convolution is computationally expensive and may not be necessary for representation learning (Chen et al., 2020a).
Revised: It has been shown that convolution is computationally expensive and may not be necessary for representation learning (Chen et al., 2020a).

Original: As the requirement for contrastive loss is simply an encoder, recently researchers have been able to produce state-of-the-art results using simpler and more efficient MLP based contrastive loss implementations (Hu et al., 2021; Kulatilleke et al., 2022).
Revised: As the requirement for contrastive loss is simply an encoder, recently researchers have been able to produce state-of-the-art results using simpler and more efficient MLP based contrastive loss implementations (Hu et al., 2021; Kulatilleke et al., 2022).

Original: Thus, there is a rapidly expanding interest and scope for contrastive loss based models.
Revised: Thus, there is a rapidly expanding interest and scope for contrastive loss based models.

Original: We consider the following specific but popular (Hu et al., 2021; Kulatilleke et al., 2022) form of contrastive loss where τ is the temperature parameter, γ ij is the relationship between nodes i, j and the loss for the i th node is:
Revised: We consider the following specific but popular (Hu et al., 2021; Kulatilleke et al., 2022) form of contrastive loss where τ is the temperature parameter, γ ij is the relationship between nodes i, j and the loss for the i th node is:

Original: [Equation]
Revised: [Equation]

Original: When no labels are present, sampling of positive and negative nodes plays a crucial role (Kipf &
Revised: When no labels are present, sampling of positive and negative nodes plays a crucial role (Kipf &

Original: Welling, 2016) and is a key implementation detail in contrastive methods (Velickovic et al., 2019).
Revised: Welling, 2016) and is a key implementation detail in contrastive methods (Velickovic et al., 2019).

Original: Positive samples in graphs are typically connected by edges (Kulatilleke et al., 2021), similar to words in a sentence in language modelling (Logeswaran & Lee, 2018).
Revised: Positive samples in graphs are typically connected by edges (Kulatilleke et al., 2021), similar to words in a sentence in language modelling (Logeswaran & Lee, 2018).

Original: Often data augmentation is used to generate positive samples; Chen et al.
Revised: Often data augmentation is used to generate positive samples; Chen et al.

Original: (2020b) used crop, colouring, blurring.
Revised: (2020b) used crop, coloring, blurring.

Original: However, it is harder to obtain negative samples.
Revised: However, it is harder to obtain negative samples.

Original: With no access to labels, negative counterparts are typically obtained via uniform sampling (Park et al., 2022), via synthesizing/augmenting (Chen et al., 2020b) or adding noise.
Revised: With no access to labels, negative counterparts are typically obtained via uniform sampling (Park et al., 2022), via synthesizing/augmenting (Chen et al., 2020b) or adding noise.

Original: Also, in graphs, adjacency information can be exploited to derive negatives (Hu et al., 2021; Kulatilleke et al., 2022) for feature contrastion.
Revised: Also, in graphs, adjacency information can be exploited to derive negatives (Hu et al., 2021; Kulatilleke et al., 2022) for feature contrastion.

Original: However, while graphs particularly suited for contrastive learning, to be effective, a large number of negative samples must be used, along with larger batch sizes and longer training compared to its supervised counterparts (Chen et al., 2020b).
Revised: However, while graphs particularly suited for contrastive learning, to be effective, a large number of negative samples must be used (Wang & Isola, 2020) (e.g., 65536 in He et al. (2020)), along with larger batch sizes and longer training compared to its supervised counterparts (Chen et al., 2020b).

Revised: Prior work has used data augmentation-based contrastive methods Zhu et al.

Original: Unlike other domains, such as vision, negative sample generation brings only limited benefits to graphs (Chuang et al., 2020; Zhu et al., 2021).
Revised: (2020; 2021b), negative samples using asymmetric structures Thakoor et al. or avoided negative samples altogether via feature-level decorrelation Zhang et al. (2021b). While Thakoor et al. (2021); Zhang et al.

Revised: (2021b) address complexity and scalability, as seen in Appendix Table 4, their performance can be further improved.

Revised: Unlike other domains, such as vision, negative sample generation brings only limited benefits to graphs (Chuang et al., 2020; Zhu et al., 2021a).

Original: To understand this phenomenon, observe the raw embedding of USPS image dataset, in the top row of Figure 4 which looks already clustered.
Revised: To understand this phenomenon, observe the raw embedding of USPS image dataset, in the top row of Figure 7 which looks already clustered.

Original: A direct consequence of this is that graphs are more susceptible to sampling bias (Chuang et al., 2020; Zhu et al., 2021).
Revised: A direct consequence of this is that graphs are more susceptible to sampling bias (Chuang et al., 2020; Zhu et al., 2021a).

Original: Thus, graph contrastive learning approaches suffer from insufficient negatives and the complex task of sample generation in addition to O ( N 2 ) time complexity required to contrast every negative node.
Revised: Thus, graph contrastive learning approaches suffer from insufficient negatives and the complex task of sample generation in addition to O ( N 2 ) time complexity required to contrast every negative node.

Original: However, what contrastive loss exactly does remain largely a mystery (Wang & Isola, 2020).
Revised: However, what contrastive loss exactly does remain largely a mystery (Wang & Isola, 2020).

Original: For example, Arora et al.
Revised: For example, Arora et al.

Original: (2019)’s
Revised: (2019)’s

Original: analysis based on the assumption of latent classes provides good theoretical insights, yet their explanation on representation quality dropping with large number of negatives is inconsistent with experimental findings (Chen et al., 2020b).
Revised: analysis based on the assumption of latent classes provides good theoretical insights, yet their explanation on representation quality dropping with large number of negatives is inconsistent with experimental findings (Chen et al., 2020b).

Original: Contrastive loss is seen as maximizing mutual information (MI) between two views.
Revised: Contrastive loss is seen as maximizing mutual information (MI) between two views.

Original: Yet, contradictorily, tighter bound on MI can lead to poor representations (Wang & Isola, 2020).
Revised: Yet, contradictorily, tighter bound on MI can lead to poor representations (Wang & Isola, 2020).

Revised: Motivation : Prior work has approximated the task in order to approximate the loss.

Revised: SwAV (Caron et al., 2020) learns to predict a node prototype code of an augmented view from the other view.

Revised: GRCCA (Zhang et al., 2021a) maps augmented graphs to prototypes using k-means for alignment.

Revised: PCL (Li et al., 2020) assigns several prototypes of different granularity to an image enforcing its representation to be more similar to its corresponding prototype.

Revised: However, all these works use some form of data augmentation which assumes that the task-relevant information is not significantly altered and require computationally expensive operations.

Original: Motivation : Wang & Isola (2020) identifies alignment and uniformity as key properties of contrastive loss: alignment encourages encoders to assign similar features to similar samples; uniformity encourages a feature distribution that preserves maximal information.
Revised: Wang & Isola (2020) identifies alignment and uniformity as key properties of contrastive loss: alignment encourages encoders to assign similar features to similar samples; uniformity encourages a feature distribution that preserves maximal information.

Original: It is fair to assume that latent clusters are dissimilar.
Revised: It is fair to assume that latent clusters are dissimilar.

Original: Even with the rare possibility of two identical cluster centers initially, one will usually change or drift apart.
Revised: Even with the rare possibility of two identical cluster centers initially, one will usually change or drift apart.

Original: It is intuitive that cluster centers should be uniformly distributed in the hyperspace, similar to nodes, in order to preserve as much information of the data as possible.
Revised: It is intuitive that cluster centers should be uniformly distributed in the hyperspace, similar to nodes, in order to preserve as much information of the data as possible.

Original: Uniformly distributing points on a hyperspace is defined as minimizing the total pairwise potential w.r.t.
Revised: Uniformly distributing points on a hyperspace is defined as minimizing the total pairwise potential w.r.t.

Original: a certain kernel function and is well-studied (Wang & Isola, 2020).
Revised: a certain kernel function and is well-studied (Wang & Isola, 2020).

Original: Thus, we are naturally motivated to use the cluster centers as meta-nodes for negative contrastion.
Revised: Thus, we are naturally motivated to use the cluster centers as meta-nodes for negative contrastion.

Original: By aggregation, all its constituent nodes cab be affected.
Revised: By aggregation, all its constituent nodes cab be affected.

Original: Thus, we avoid sampling, effectively eliminate sample bias, and also meet the criterion of larger number of samples.
Revised: Thus, we avoid sampling, effectively eliminate sample bias, and also meet the criterion of larger number of samples.

Original: Learned soft cluster assignments can avoid possible heterophily and add robustness to noise in edge construction.
Revised: Learned soft cluster assignments can avoid possible heterophily and add robustness to noise in edge construction.

Original: In this work, we propose a novel contrastive loss, PamC, which uses paramaterless proxy metanods to approximate negative samples.
Revised: In this work, we propose a novel contrastive loss, PamC, which uses paramaterless proxy metanods to approximate negative samples.

Original: Our approach indirectly uses the full set of negative samples and yet is efficient with a time complexity of O ( N ) .
Revised: Our approach indirectly uses the full set of negative samples and yet is efficient with a time complexity of O ( N ) .

Original: Not only does PamCGC, based on PamC, outperform or match previous work, but it is also simpler than any prior negative sample generation approach, faster and uses relatively less GPU memory.
Revised: Not only does PamCGC, based on PamC, outperform or match previous work, but it is also simpler than any prior negative sample generation approach, faster and uses relatively less GPU memory.

Original: It can be incorporated to any contrastive learning-based clustering model with minimal modifications, and works with diverse data, as we demonstrate using benchmark datasets from image, text and graph modalities.
Revised: It can be incorporated to any contrastive learning-based clustering model with minimal modifications, and works with diverse data, as we demonstrate using benchmark datasets from image, text and graph modalities.

Revised: 𝑓𝑐 𝑒0 𝑓𝑐 𝑒0 𝑓𝑐 𝑒 𝑓𝑐 𝑧 𝑓𝑐 𝑑0 𝑓𝑐 𝑑0 𝑓𝑐 𝑑 𝑛𝑜𝑑𝑒 𝑓𝑒𝑎𝑡𝑢𝑟𝑒𝑠 𝑥 𝑧 𝐶𝑙𝑢𝑠𝑡𝑒𝑟𝐶𝑒𝑛𝑡𝑒𝑟𝑠 𝜇 ( - ) pivot

Original: To summarize, our main contributions are:
Revised: Distribution (Q 2 ) 𝑦 𝑄 𝑃 ҧ𝑥 pre training 𝑀𝑆𝐸 𝑥, ҧ𝑥 𝐴𝐶𝐶,𝑁𝑀𝐼,𝐹𝐼,𝐴𝑅𝐼 𝑧 𝐶𝑙𝑢𝑠𝑡𝑒𝑟𝑖𝑛𝑔 −𝑆𝑒𝑙𝑓 𝑠𝑢𝑝𝑒𝑟𝑣𝑖𝑠𝑖𝑜𝑛 𝑅𝑒𝑎𝑙 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 𝑐𝑒𝑛𝑡𝑒𝑟𝑠 −𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠 𝐾𝐿 𝑃 || 𝑄 𝐺𝑟𝑎𝑝ℎ𝑒𝑑𝑔𝑒𝑠 −𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠 Real Cluster Centres ҧ𝜇 0 0 01 1 11 1 1 𝑥 𝑖 𝑥 𝑗 ∝ 𝑖𝑗 − 𝛼 𝑖𝑗 ∙ log(exp(𝑠𝑖𝑚 𝑥 𝑖 ,𝑥 𝑗 ) +log(exp(𝑠𝑖𝑚 ҧ𝜇 𝑎 , ҧ𝜇 𝑏 )

Revised: Inter-cluster cluster centers − repulsive

Revised: [Figure]

Original:  • We introduce an efficient novel parameter-free proxy, PamC, for negative sample approximation that is scalable, computationally efficient and able to include all samples.
Revised: Members are linked to [Figure] To summarize, our main contributions are: • We introduce an efficient novel parameter-free proxy, PamC, for negative sample approximation that is scalable, computationally efficient and able to include all samples.

Original: It works with diverse data, including graphs.
Revised: It works with diverse data, including graphs.

Original: We claim PamC is the first to implicitly use the whole graph with O ( N ) time complexity, in addition to further 3-fold gains.
Revised: We claim PamC is the first to implicitly use the whole graph with O ( N ) time complexity, in addition to further 3-fold gains.

Original: • We provide theoretical proof and show that real world graphs always satisfies the necessary conditions, and that PamCGC is block-contrastive, known to outperform pair-wise losses.
Revised: • We provide theoretical proof and show that real world graphs always satisfies the necessary conditions, and that PamCGC is block-contrastive, known to outperform pair-wise losses.

Original: • Extensive experiments on 6 benchmark datasets show PamCGC, using proposed PamC, is on par with or better than state-of-the-art graph clustering methods in accuracy while achieving 3x trainingtime, 1.8 inference time and 5x GPU memory efficiency.
Revised: • Extensive experiments on 6 benchmark datasets show PamCGC, using proposed PamC, is on par with or better than state-of-the-art graph clustering methods in accuracy while achieving 2x training time and 5x GPU memory efficiency.

Original: 𝑓𝑐 𝑒0 𝑓𝑐 𝑒0 𝑓𝑐 𝑒 𝑓𝑐 𝑧 𝑓𝑐 𝑑0 𝑓𝑐 𝑑0 𝑓𝑐 𝑑 𝑛𝑜𝑑𝑒 𝑓𝑒𝑎𝑡𝑢𝑟𝑒𝑠 𝑥 𝑧 𝐶𝑙𝑢𝑠𝑡𝑒𝑟𝐶𝑒𝑛𝑡𝑒𝑟𝑠 𝜇 ( - ) pivot

Original: Distribution (Q 2 ) 𝑦 𝑄 𝑃 ҧ𝑥 pre training 𝑀𝑆𝐸 𝑥, ҧ𝑥 𝐴𝐶𝐶,𝑁𝑀𝐼,𝐹𝐼,𝐴𝑅𝐼 𝑧 𝐶𝑙𝑢𝑠𝑡𝑒𝑟𝑖𝑛𝑔 −𝑆𝑒𝑙𝑓 𝑠𝑢𝑝𝑒𝑟𝑣𝑖𝑠𝑖𝑜𝑛 𝑅𝑒𝑎𝑙 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 𝑐𝑒𝑛𝑡𝑒𝑟𝑠 −𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠 𝐾𝐿 𝑃 || 𝑄 𝐺𝑟𝑎𝑝ℎ𝑒𝑑𝑔𝑒𝑠 −𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠

Original: Real Cluster Centres ҧ𝜇 0 0 01 1 11 1 1 𝑥 𝑖 𝑥 𝑗

Original: ∝

Original: 𝑖𝑗 − 𝛼 𝑖𝑗 ∙ log(exp(𝑠𝑖𝑚 𝑥 𝑖 ,𝑥 𝑗 ) +log(exp(𝑠𝑖𝑚 ҧ𝜇 𝑎 , ҧ𝜇 𝑏 )

Original: Inter-cluster cluster centers

Original: [Figure]

Original: to

Original: [Figure]

Original:  I MPLEMENTATION
Revised:  I MPLEMENTATION

Original: First we describe PamC, which is our parameter-free proxy to efficiently approximate the negative samples, as shown in Figure 1.
Revised: First we describe PamC, which is our parameter-free proxy to efficiently approximate the negative samples, as shown in Figure 1.

Original: Next, we introduce PamCGC, a self-supervised model based on
Revised: Next, we introduce PamCGC, a self-supervised model based on

Original: PamC to simultaneously learn discriminative embeddings and clusters.
Revised: PamC to simultaneously learn discriminative embeddings and clusters.

Original: 2. N EGATIVE SAMPLE APPROXIMATION BY META - NODE PROXIES
Revised: 2. N EGATIVE SAMPLE APPROXIMATION BY META - NODE PROXIES

Original: Contrastive loss makes positive or connected nodes closer and negative or unconnected nodes further away in the feature space (Kulatilleke et al., 2022).
Revised: Contrastive loss makes positive or connected nodes closer and negative or unconnected nodes further away in the feature space (Kulatilleke et al., 2022).

Original: However, in order to be effective, all negative nodes need to be contrasted with x i which is computationally expensive.
Revised: However, in order to be effective, all negative nodes need to be contrasted with x i which is computationally expensive.

Original: A cluster centre is formed by combining all member nodes, and can be seen as an aggregated representation, or a proxy, of its compositional elements.
Revised: A cluster center is formed by combining all member nodes, and can be seen as an aggregated representation, or a proxy, of its compositional elements.

Original: Motivated by this, we use the cluster centres to enforce negative contrastion.
Revised: Motivated by this, we use the cluster centers to enforce negative contrastion.

Original: Specifically, we contrast every cluster centre ˆ µ i with every cluster centre ˆ µ j where i ̸ = j .
Revised: Specifically, we contrast every cluster center ˆ µ i with every cluster center ˆ µ j where i ̸ = j .

Original: Following Arora et al.
Revised: Following Arora et al.

Original: ; Chuang et al.
Revised: ; Chuang et al.

Original: (2020), we assume an underlying set of discrete latent classes C which represents semantic content, i.e., similar nodes x i , x j are in the same latent class ˆ µ .
Revised: (2020), we assume an underlying set of discrete latent classes C which represents semantic content, i.e., similar nodes x i , x j are in the same latent class ˆ µ .

Original: Thus, we derive our proxy for negative samples as:
Revised: Thus, we derive our proxy for negative samples as:

Original: [Equation]
Revised: [Equation]

Original: Note that, ℓ proxy contains no i or j terms!
Revised: Note that, ℓ proxy contains no i or j terms!

Original: resulting in three fold gains.
Revised: resulting in three fold gains.

Original: Firstly, we replace (cid:80) Ni =1 , with a more efficient (cid:80) Ca =1 where N ≫ C , typically many magnitudes, in almost all datasets, as evident from Table 1.
Revised: Firstly, we replace (cid:80) Ni =1 , with a more efficient (cid:80) Ca =1 where N ≫ C , typically many magnitudes, in almost all datasets, as evident from Table 1.

Original: Secondly, the ℓ proxy is at graph level with time complexity of O ( N ) rather than an instance level O ( N 2 ) .
Revised: Secondly, the ℓ proxy is at graph level with time complexity of O ( N ) rather than an instance level O ( N 2 ) .

Original: Finally, given real world graphs (especially larger graphs,) are sparse, a sparse implantation for the positives, using edge-lists, will result in a third efficiency gain, which is only possible by not having to operate on the negatives explicitly.
Revised: Finally, given real world graphs (especially larger graphs,) are sparse, a sparse implantation for the positives, using edge-lists, will result in a third efficiency gain, which is only possible by not having to operate on the negatives explicitly.

Original: Note that a prerequisite of the proxy approximation is the availability of labels to construct the learned cluster centers ˆ µ , which we explain in the next section.
Revised: Note that a prerequisite of the proxy approximation is the availability of labels to construct the learned cluster centers ˆ µ , which we explain in the next section.

Original: Thus, the complete graph level contrastive loss can be expressed as:
Revised: Thus, the complete graph level contrastive loss can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: Theoretical explanation.
Revised: Theoretical explanation.

Original: The standard contrastive loss uses Jensen-Shannon divergence, which yields log 2 constant and vanishing gradients for disjoint distributions of positive and negative sampled pairs (Zhu et al., 2021).
Revised: The standard contrastive loss uses Jensen-Shannon divergence, which yields log 2 constant and vanishing gradients for disjoint distributions of positive and negative sampled pairs (Zhu et al., 2021a).

Original: However, in the proposed method, positive pairs are necessarily edgelinked (either explicitly or via influence (Kulatilleke et al., 2022)), and unlikely to be disjoint.
Revised: However, in the proposed method, positive pairs are necessarily edge-linked (either explicitly or via influence (Kulatilleke et al., 2022)), and unlikely to be disjoint.

Original: Using meta-nodes for negatives, which are compositions of multiple nodes, lowers the possibility of disjointness.
Revised: Using meta-nodes for negatives, which are compositions of multiple nodes, lowers the possibility of disjointness.

Original: An algorithm using the average of the positive and negative samples in blocks as a proxy instead of just one point has a strictly better bound due to Jensen’s inequality getting tighter and is superior compared to their equivalent of element-wise contrastive counterparts (Arora et al., 2019).
Revised: An algorithm using the average of the positive and negative samples in blocks as a proxy instead of just one point has a strictly better bound due to Jensen’s inequality getting tighter and is superior compared to their equivalent of element-wise contrastive counterparts (Arora et al., 2019).

Original: The computational and time cost is a direct consequence of node level contrastion.
Revised: The computational and time cost is a direct consequence of node level contrastion.

Original: Given,
Revised: Given,

Original: N ≫ clusters , we circumvent the problem of large N by proposing a proxy-ed negative contrastive objective that operates directly at the cluster level.
Revised: N ≫ clusters , we circumvent the problem of large N by proposing a proxy-ed negative contrastive objective that operates directly at the cluster level.

Original: Establishing mathematical guarantee
Revised: Establishing mathematical guarantee

Original: : Assume node embeddings Z = { z 1 , z 2 , z 3 . . .
Revised: : Assume node embeddings Z = { z 1 , z 2 , z 3 . . .

Original: z N } , clusters µ = { µ 1 , µ 2 . . .
Revised: z N } , clusters µ = { µ 1 , µ 2 . . .

Original: µ C } , a label assignment operator label( z i )
Revised: µ C } , a label assignment operator label( z i )

Original: such that µ a = (cid:80) Ni =1 1 [ i ∈ label( z i )=
Revised: such that µ a = (cid:80) Ni =1 1 [ i ∈ label( z i )=

Original: a ] · z i , a temperature hyperparameter τ and,similarity( i, j, z i , z j ) = sim ( z i , z j )
Revised: a ] · z i , a temperature hyperparameter τ and,

Original: [Equation]
Revised: [Equation]

Original: Using sim( z i , z j ) as the shorthand notation for similarity( i, j,
Revised: Using sim( z i , z j ) as the shorthand notation for similarity( i, j,

Original: z i , z j ) , the classic contrastive loss is:
Revised: z i , z j ) , the classic contrastive loss is:

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can express the cluster based contrastive loss as:
Revised: Similarly, we can express the cluster based contrastive loss as:

Original: [Equation]
Revised: [Equation]

Original: As 0 ≤ sim ≤ 1 . 0 , we establish the condition for our inequality as;
Revised: As 0 ≤ sim ≤ 1 . 0 , we establish the condition for our inequality as;

Original: [Equation]
Revised: [Equation]

Original: We provide the full derivationin Appendix A.1.
Revised: We provide the full derivation

Original:  As C > 1 (minimum 2 are needed for a cluster), and log ( x ) : x > 0 is strictly increasing, N > 1 + ( C − 1)
Revised: [Figure] As C > 1 (minimum 2 are needed for a cluster), and log ( x ) : x > 0 is strictly increasing, N > 1 + ( C − 1)

Original: e τ is the necessary condition, which is easily satisfied for nearly all real world datasets and as seen in Figure 2 for multiple τ temperatures.
Revised: e τ is the necessary condition, which is easily satisfied for nearly all real world datasets and as seen in Figure 2 for multiple τ temperatures.

Original: Thus, as loss NN > loss CC , loss NN upper bounds loss CC , the more efficient variant.
Revised: Thus, as loss NN > loss CC , loss NN upper bounds loss CC , the more efficient variant.

Original: Additionally loss CC benefits from block-contrastiveness (Arora et al., 2019), achieves a lower minima and uses the fullest possible negative information.
Revised: Additionally loss CC benefits from block-contrastiveness (Arora et al., 2019), achieves a lower minima and uses the fullest possible negative information.

Original: We also show, experimentally, that minimizing loss CC results in effective, and sometimes better, representations for downstream tasks.
Revised: We also show, experimentally, that minimizing loss CC results in effective, and sometimes better, representations for downstream tasks.

Original: 2. C ONSTRUCTING THE META - NODE CLUSTER CENTRES ( ˆ µ )
Revised: 2. C ONSTRUCTING THE META - NODE CLUSTER CENTERS ( ˆ µ )

Original: In order to derive the real cluster centers ˆ µ , which is distinct from the learnt cluster centers µ , we simply aggregate all the node embedding z of a cluster using its label .
Revised: In order to derive the real cluster centers ˆ µ , which is distinct from the learnt cluster centers µ , we simply aggregate all the node embedding z of a cluster using its label .

Original: Even with unlabeled data, label() can be accomplished using predicted soft labels.
Revised: Even with unlabeled data, label() can be accomplished using predicted soft labels.

Original: The intuition here is that, during backpropagation, the optimization process will update the constituent node embeddings, z , to incorporate negative distancing.
Revised: The intuition here is that, during backpropagation, the optimization process will update the constituent node embeddings, z , to incorporate negative distancing.

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: where label(c) is either the ground-truth or learnt soft labels.
Revised: where label(c) is either the ground-truth or learnt soft labels.

Original: Accordingly, our proxy can equally be used in supervised and unsupervised scenarios and has a wider general applicability as an improvement of the contrastive loss at large.
Revised: Accordingly, our proxy can equally be used in supervised and unsupervised scenarios and has a wider general applicability as an improvement of the contrastive loss at large.

Original: Finlay, Equation 8 can be realized with softmax () and mean () operations, which are well optimized GPU primitives in any machine learning framework.
Revised: Finlay, Equation 8 can be realized with softmax () and mean () operations, which are well optimized GPU primitives in any machine learning framework.

Original: We provide a reference pytorch implementation.
Revised: We provide a reference pytorch implementation.

Original: 2. O BTAINING THE SOFT LABELS
Revised: 2. O BTAINING THE SOFT LABELS

Original: Graph clustering is essentially unsupervised.
Revised: Graph clustering is essentially unsupervised.

Original: To this end, following Xie et al.
Revised: To this end, following Xie et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: (2017); Wang et al.
Revised: (2017); Wang et al.

Original: Kulatilleke et al.
Revised: Kulatilleke et al.

Original: (2022), we use probability distribution derived softlabels and a self-supervision mechanism for cluster enhancement.
Revised: (2022), we use probability distribution derived softlabels and a self-supervision mechanism for cluster enhancement.

Original: Specifically, we obtain soft cluster assignments probabilities q iu for embedding z i and cluster centre µ u .
Revised: Specifically, we obtain soft cluster assignments probabilities q iu for embedding z i and cluster center µ u .

Original: In order to handle differently scaled clusters and be computationally convenient (Wang et al., 2019), we use the student’s t -distribution (Maaten & Hinton, 2008) as a kernel for the similarity measurement between the embedding and centroid:[Equation]
Revised: In order to handle differently scaled clusters and be computationally convenient (Wang et al., 2019), we use the student’s t -distribution (Maaten & Hinton, 2008) as a kernel for the similarity measurement between the embedding and centroid:

Original: [Equation]
Revised: [Equation]

Original: where, η is the Student’s t -distribution’s degree of freedom.
Revised: where, η is the Student’s t -distribution’s degree of freedom.

Original: Cluster centers µ are initialized by
Revised: Cluster centers µ are initialized by

Original: K -means on embeddings from the pre-trained AE.
Revised: K -means on embeddings from the pre-trained AE.

Original: We use Q =
Revised: We use Q =

Original: [ q iu ] as the distribution of the cluster assignments of all samples, and η =1 for all experiments following Bo et al.
Revised: [ q iu ] as the distribution of the cluster assignments of all samples, and η =1 for all experiments following Bo et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original:  Kulatilleke et al
Revised: (2021); Kulatilleke et al.

Original: [Equation]
Revised: [Table]

Original: Nodes closer in embedding space to a cluster center has higher soft assignment probabilities in
Revised: Nodes closer in embedding space to a cluster center has higher soft assignment probabilities in

Original: Q .
Revised: Q .

Original: A target distribution P that emphasizes the confident assignments is obtained by squaring and normalizing Q , given by :
Revised: A target distribution P that emphasizes the confident assignments is obtained by squaring and normalizing Q , given by :

Original: [Equation]
Revised: [Equation]

Original: where (cid:80) i q iu is the soft cluster frequency of centroid u .
Revised: where (cid:80) i q iu is the soft cluster frequency of centroid u .

Original: Following Kulatilleke et al.
Revised: Following Kulatilleke et al.

Original: (2022), we minimize the KL divergence between Q and P distributions, which forces the current distribution Q to approach the more confident target distribution P .
Revised: (2022), we minimize the KL divergence between Q and P distributions, which forces the current distribution Q to approach the more confident target distribution P .

Original: KL divergence updates models more gently and lessens severe disturbances on the embeddings (Bo et al., 2020).
Revised: KL divergence updates models more gently and lessens severe disturbances on the embeddings (Bo et al., 2020).

Original: Further, it can accommodate both the structural and feature optimization targets of
Revised: Further, it can accommodate both the structural and feature optimization targets of

Original: PamCGC.
Revised: PamCGC.

Original: We self-supervise cluster assignments 1 by using distribution Q to target distribution P , which then supervises the distribution Q in turn by minimizing the KL divergence as:
Revised: We self-supervise cluster assignments 1 by using distribution Q to target distribution P , which then supervises the distribution Q in turn by minimizing the KL divergence as:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: The final proposed model, after incorporating PamC contrastive objective with self-supervised clustering, where α > 0 controls structure incorporation and β > 0 controls cluster optimization is:
Revised: The final proposed model, after incorporating PamC contrastive objective with self-supervised clustering, where α > 0 controls structure incorporation and β > 0 controls cluster optimization is:

Original: [Equation]
Revised: [Equation]

Original: 2. C OMPLEXITY A NALYSIS
Revised: 2. C OMPLEXITY A NALYSIS

Original: Given input data dimension d and AE layer dimensions of d 1 , d 2 , · · · , d L , following Kulatilleke et al.
Revised: Given input data dimension d and AE layer dimensions of d 1 , d 2 , · · · , d L , following Kulatilleke et al.

Original: (2022), O AE =
Revised: (2022), O AE =

Original: O ( Nd 2 d 21 ...
Revised: O ( Nd 2 d 21 ...

Original: d 2 L/ 2 ) for PamCGC-AE.
Revised: d 2 L/ 2 ) for PamCGC-AE.

Original: Assuming K clusters, from Equation 9, the time complexity is O cluster
Revised: Assuming K clusters, from Equation 9, the time complexity is O cluster

Original: =
Revised: =

Original: O ( NK + N log N ) following Xie et al.
Revised: O ( NK + N log N ) following Xie et al.

Original: For PamC, we only compute ∥ z ∥ 22 and z i · z j for the actual positive edges E using sparse matrix resulting in a time complexity O + = O
Revised: For PamC, we only compute ∥ z ∥ 22 and z i · z j for the actual positive edges E using sparse matrix resulting in a time complexity O + = O

Original: ( NEd z ) , linear with the number of edges E , with d z embedding dimension.
Revised: ( NEd z ) , linear with the number of edges E , with d z embedding dimension.

Original: For the negatives, we use the meta-node based negatives O − =
Revised: For the negatives, we use the meta-node based negatives O − =

Original: O ( CC ) where C is the meta-node.
Revised: O ( CC ) where C is the meta-node.

Original: Note that, for real graphs, N ≫ C in many magnitudes.
Revised: Note that, for real graphs, N ≫ C in many magnitudes.

Original: Thus, the overall time complexity is linearly related to the number of samples and edges.
Revised: Thus, the overall time complexity is linearly related to the number of samples and edges.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate PamCGC on transductive node clustering comparing to state-of-the-art self-supervised, contrastive and (semi-)supervised methods.
Revised: We evaluate PamCGC on transductive node clustering comparing to state-of-the-art self-supervised, contrastive and (semi-)supervised methods.

Original: Datasets.
Revised: Datasets.

Original: Following Bo et al.
Revised: Following Bo et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: (2021); Kulatilleke et al.
Revised: (2021); Kulatilleke et al.

Original: (2022), experiments are conducted on six common clustering benchmarks, which includes one image dataset (USPS (Le Cun et al., 1990)), one sensor data dataset (HHAR (Stisen et al., 2015)), one text dataset (REUT (Lewis et al., 2004)) and three citation graphs (ACM 2 , CITE 4 , and DBLP 3 ).
Revised: (2022), experiments are conducted on six common clustering benchmarks, which includes one image dataset (USPS (Le Cun et al., 1990)), one sensor data dataset (HHAR (Stisen et al., 2015)), one text dataset (REUT (Lewis et al., 2004)) and three citation graphs (ACM 2 , CITE 4 , and DBLP 3 ).

Original: For the non-graph data, we use  undirected k -nearest neighbour (KNN (Altman, 1992)) to generate adjacency matrix A following
Revised: For the non-graph data, we use training method. undirected k -nearest neighbour (KNN (Altman, 1992)) to generate adjacency matrix A following

Original: Bo et al.
Revised: Bo et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: Table 1 summarizes the datasets.
Revised: Table 1 summarizes the datasets.

Original: Baseline Methods.
Revised: Baseline Methods.

Original: We compare with multiple models.
Revised: We compare with multiple models.

Original: K-means (Hartigan & Wong, 1979) is a classical clustering method using raw data.
Revised: K-means (Hartigan & Wong, 1979) is a classical clustering method using raw data.

Original: AE (Hinton & Salakhutdinov, 2006) applies K-means to deep representations learned by an auto-encoder.
Revised: AE (Hinton & Salakhutdinov, 2006) applies K-means to deep representations learned by an auto-encoder.

Original: DEC (Xie et al., 2016) clusters data in a jointly optimized feature space.
Revised: DEC (Xie et al., 2016) clusters data in a jointly optimized feature space.

Original: IDEC (Guo et al., 2017) enhances DEC by adding KL divergence-based reconstruction loss.
Revised: IDEC (Guo et al., 2017) enhances DEC by adding KL divergence-based reconstruction loss.

Original: Following models exploit graph structures during clustering: SVD (Golub &
Revised: Following models exploit graph structures during clustering: SVD (Golub &

Original: Reinsch, 1971) applies singular value decomposition to the adjacency matrix.
Revised: Reinsch, 1971) applies singular value decomposition to the adjacency matrix.

Original: DGI (Velickovic et al., 2019) learns embeddings by maximizing node MI with the graph.
Revised: DGI (Velickovic et al., 2019) learns embeddings by maximizing node MI with the graph.

Original: GAE (Kipf & Welling, 2016) combines convolution with the AE.
Revised: GAE (Kipf & Welling, 2016) combines convolution with the AE.

Original: ARGA (Pan et al., 2018) uses an adversarial regularizer to guide the embeddings learning.
Revised: ARGA (Pan et al., 2018) uses an adversarial regularizer to guide the embeddings learning.

Original: Following deep graph clustering jointly optimize embeddings and graph clustering:
Revised: Following deep graph clustering jointly optimize embeddings and graph clustering:

Original: DAEGC (Wang et al., 2019), uses an attentional neighbor-wise strategy and clustering loss.
Revised: DAEGC (Wang et al., 2019), uses an attentional neighbor-wise strategy and clustering loss.

Original: SDCN (Bo et al., 2020), couples DEC and GCN via a fixed delivery operator and uses feature reconstruction.
Revised: SDCN (Bo et al., 2020), couples DEC and GCN via a fixed delivery operator and uses feature reconstruction.

Original: AGCN (Peng et al., 2021), extends SDCN by adding an attention-based delivery operator and uses multi scale information for cluster prediction.
Revised: AGCN (Peng et al., 2021), extends SDCN by adding an attention-based delivery operator and uses multi scale information for cluster prediction.

Original: CGC (Park et al., 2022) uses a multi-level, hierarchy based contrastive loss.
Revised: CGC (Park et al., 2022) uses a multi-level, hierarchy based contrastive loss.

Original: SCGC and SCGC* (Kulatilleke et al., 2022) uses block contrastive loss with an AE and MLP respectively.
Revised: SCGC and SCGC* (Kulatilleke et al., 2022) uses block contrastive loss with an AE and MLP respectively.

Original: The only difference between SCGC*et [Table] and PamCGC is the novel PamC loss, Also as SCGC* is the current state-of-the-art.
Revised: The only difference between SCGC* and PamCGC is the novel PamC loss, Also as SCGC* is the current state-of-the-art.

Original: Thus, it is used as the benchmark.
Revised: Thus, it is used as the benchmark.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: Following Bo et al.
Revised: Following Bo et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: (2021), we use Accuracy (ACC),
Revised: (2021), we use Accuracy (ACC),

Original: Normalized Mutual Information (NMI), Average Rand Index (ARI), and macro F1-score (F1) for evaluation.
Revised: Normalized Mutual Information (NMI), Average Rand Index (ARI), and macro F1-score (F1) for evaluation.

Original: For each, larger values imply better clustering.
Revised: For each, larger values imply better clustering.

Original: 3. I MPLEMENTATION
Revised: 3. I MPLEMENTATION

Original: The positive component of our loss only requires the actual connections and can be efficiently represented by sparse matrices.
Revised: The positive component of our loss only requires the actual connections and can be efficiently represented by sparse matrices.

Original: Further, the negative component of the loss is graph-based, and not instance based, thus needs to be computed only once per epoch.
Revised: Further, the negative component of the loss is graph-based, and not instance based, thus needs to be computed only once per epoch.

Original: Thus, by decoupling the negatives, our loss is inherently capable of batching and is trivially parallelizable.
Revised: Thus, by decoupling the negatives, our loss is inherently capable of batching and is trivially parallelizable.

Original: Computation of the negative proxy, which is only C · C does not even require a GPU!
Revised: Computation of the negative proxy, which is only C · C does not even require a GPU!

Original: For fair comparison, we use the same 500 − 500 − 2000 − 10 AE dimensions as in Guo et al.
Revised: For fair comparison, we use the same 500 − 500 − 2000 − 10 AE dimensions as in Guo et al.

Original: Bo et al.
Revised: Bo et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: (2021); Kulatilleke et al.
Revised: (2021); Kulatilleke et al.

Original: (2022) and the same pre-training procedure,
Revised: (2022) and the same pre-training procedure,

Original: i.e.
Revised: i.e.

Original: 30 epochs; learning rate of 10 − 3 for USPS, HHAR, ACM, DBLP and 10 − 4 for REUT and CITE; batch size of 256 .
Revised: 30 epochs; learning rate of 10 − 3 for USPS, HHAR, ACM, DBLP and 10 − 4 for REUT and CITE; batch size of 256 .

Original: We made use of the publicly available pre-trained AE from Bo et al.
Revised: We made use of the publicly available pre-trained AE from Bo et al.

Original: We use a once computed edge-list for training, which is not needed during inference.
Revised: We use a once computed edge-list for training, which is not needed during inference.

Original: For training, for each dataset, we initialize the cluster centers from K -means and repeat the experiments 10 times with 200 epochs to prevent extreme cases.
Revised: For training, for each dataset, we initialize the cluster centers from K -means and repeat the experiments 10 times with 200 epochs to prevent extreme cases.

Original: We cite published accuracy results from Bo et al.
Revised: We cite published accuracy results from Bo et al.

Original: Peng et al.
Revised: Peng et al.

Original: (2021); Kulatilleke et al.
Revised: (2021); Kulatilleke et al.

Original: for other models.
Revised: for other models.

Original: For all timing and memory experiments, we replicate the exact same training loops, including internal evaluation metric calls, when measuring performance for fair comparison.
Revised: For all timing and memory experiments, we replicate the exact same training loops, including internal evaluation metric calls, when measuring performance for fair comparison.

Original: Our code will be made publicly available.
Revised: Our code will be made publicly available.

Original: 3.Q UANTITATIVE R ESULTS
Revised: 3. R ESULTS

Original: We show our hyperparameters in Table 1.
Revised: We show our hyperparameters in Table 1.

Original: Comparison of results with state-of-the-art graph and non-graph datasets are in Table 2 and Table 3, respectively.
Revised: Comparison of results with state-of-the-art graph and non-graph datasets are in Table 2 and Table 3, respectively.

Original: For the graph data, PamCGC is stateof-the-art for DBLP.
Revised: For the graph data, PamCGC is stateof-the-art for DBLP.

Original: A paired-t test shows ACM and CITE results to be best for both SCGC* and
Revised: A paired-t test shows ACM and CITE results to be best for both SCGC* and

Original: PamCGC.
Revised: PamCGC.

Original: In non-graph results, PamCGC comes second best in USPS image data.
Revised: In non-graph results, PamCGC comes second best in USPS image data.

Original: While results for
Revised: While results for

Original: HHAR are somewhat lagging, PamCGC is the best for REUT.
Revised: HHAR are somewhat lagging, PamCGC is the best for REUT.

Original: Generally we achieve better results on the natural graph datasets; ACM, DBLP and CITE, while being competitive on other modalities. 
Revised: Generally we achieve better results on the natural graph datasets; ACM, DBLP and CITE, while being competitive on other modalities. We present the qualitative results in Appendix A.4. et

Original: [Table] [Figure]
Revised: [Table] [Table]

Original:  P ERFORMANCE
Revised: 3. P ERFORMANCE

Original: In Figure 3 we compare the GPU based training, inference times and GPU memory.
Revised: In Figure 3 we compare the GPU based training time and GPU memory.

Original: Our model times also include the time taken for the cumulative influence computation.
Revised: Our model times also include the time taken for the cumulative influence computation.

Original: For all the datasets, PamCGC is superior by 3.6x trainingtime, 1.8x inference time and 5.3x GPU memory savings.
Revised: For all the datasets, PamCGC is superior by 2.2x training time and 5.3x GPU memory savings.

Original: Especially, for larger datasets USPS, HHAR and REUT, PamCGC uses 5.2,7.7,8.7x less GPU memory.
Revised: Especially, for larger datasets USPS, HHAR and REUT, PamCGC uses 5.2,7.7,8.7x less GPU memory.

Original: Note that SCGC only differs from PamCGC by its use of the novel proxy-ed PamC to which we solely attribute the time and memory savings.
Revised: Additionally, we used CITE dataset (3327 nodes) to create synthetics nodes. For a scale factor n , as contact nodes n times, along with edge-lists.

Original: We use UMAP (McInnes et al., 2018), in Figure 4, to get a visual understanding of the raw and learnt embedding spaces.
Revised: Figure 3(right) shows the scaled edges and nodes for scale factors 5 , 10 , 15 · · · 45 and the GPU memory and training time for 1 epoch on Google colab T4 GPU with 16GB memory.

Original: Except for USPS, which is a distinct set of 0 · · · 9 handwritten digits (raw 1), we see that all other datasets produce quite indistinguishable clusters.
Revised: Without PamC, scales over 5 is not possible due to running out of memory. With PamC over x45 (150,000 nodes) is possible.

Original: Clustering is nearly non-existent in the (last 3) graph datasets. This clearly shows a characteristic difference in graph data, which can lead to high samplings bias.
Revised: GPU and memory increase is liner confirming the theoretical time complexity. We used CITE as it is a very common dataset.

Original: Note that ˆ µ ̸ = µ for any meta-node.
Revised: We used synthetic node creation to capture variation over node size. Appendix A.8 shows GPU time breakup. Appendix A.6 shows the CITE dataset results with PamC when scaled from 1 . . . 20 in steps of 1.

Original: 3. A BLATION STUDY
Revised: 3. A BLATION STUDY

Original: To investigate PamCs ability to generalize to other models, we incorporate it to SDCN and AGCN models, modified for contrastive loss.
Revised: To investigate PamCs ability to generalize to other models, we incorporate it to SDCN and AGCN models, modified for contrastive loss.

Original: Figure 5 shows the GPU training time and accuracy.
Revised: Figure 4 shows the GPU training time and accuracy.

Original: As
Revised: As

Original: PamC is a loss function, there is no difference in the inference times.
Revised: PamC is a loss function, there is no difference in the inference times.

Revised: As expected, training times

Revised: [Figure]

Revised: Note that SCGC* only differs from PamCGC by its use of the novel proxy-ed PamC to which we solely attribute the time and memory savings

Original: As expected, training times are significantly shorter, with (often better) training accuracy due to block contrastiveness.
Revised: [Figure] are significantly shorter, with (often better) training accuracy due to block contrastiveness.

Original: Note that
Revised: Note that

Original: PamC only improves loss computation efficiency.
Revised: PamC only improves loss computation efficiency.

Original: Majority of the SDCN and AGCN computation time is spent in their GNNs convolution operations.[Figure] [Figure]
Revised: Majority of the SDCN and AGCN computation time is spent in their GNNs convolution operations.

Original: For dblp, usps, reut accuracy is actually better.

Original: We also carry out extensive experimentation to assess the behavior of hyperparameters.
Revised: We also carry out extensive experimentation to assess the behavior of hyperparameters.

Original: PamC is robust to changes in hyperparameter values and performs best with a learning rate of 0 .
Revised: PamC is robust to changes in hyperparameter values and performs best with a learning rate of 0 .

Original: 001 , as shown in Appendix A.2.
Revised: 001 , as shown in Appendix A.2.

Original: Further, PamC accuracy against all hyperparameter combinations is generally equal or better than the less efficient non proxy-ed contrastive loss variant, as seen in Appendix A.3.
Revised: Further, PamC accuracy against all hyperparameter combinations is generally equal or better than the less efficient non proxy-ed contrastive loss variant, as seen in Appendix A.3.

Original: 3. F UTURE WORK
Revised: 3. F UTURE WORK

Original: Our parameter-free proxy-ed contrastive loss uses the full positive edge information which, as some of our experiments has shown, is redundant.
Revised: Our parameter-free proxy-ed contrastive loss uses the full positive edge information which, as some of our experiments has shown, is redundant.

Original: For example, USPS gives similar results with 40% positive edges removed.
Revised: For example, USPS gives similar results with 40% positive edges removed.

Original: An algorithm to drop un-informative edges may result in further efficiency improvements, which we leave for future work.
Revised: An algorithm to drop un-informative edges may result in further efficiency improvements, which we leave for future work.

Original: While theoretically possible, it would be interesting to see how our proxy-ed contrastive loss works with semi or fully supervised data.
Revised: While theoretically possible, it would be interesting to see how our proxy-ed contrastive loss works with semi or fully supervised data.

Original: Further study is needed to explore how hard cluster centers effect the optimization process.
Revised: Further study is needed to explore how hard cluster centers effect the optimization process.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we present an efficient parameter-free proxy approximation to incorporate negative samples in contrastive loss for joint clustering and representation learning.
Revised: In this work, we present an efficient parameter-free proxy approximation to incorporate negative samples in contrastive loss for joint clustering and representation learning.

Original: We eliminate sample bias, achieve block contrastiveness and 0( N ) .
Revised: We eliminate sample bias, achieve block contrastiveness and 0( N ) .

Original: Our work is supported by theoretical proof and empirical results.
Revised: Our work is supported by theoretical proof and empirical results.

Original: We improve considerably over previous methods accuracy, speed and memory usage.
Revised: We improve considerably over previous methods accuracy, speed and memory usage.

Original: Our approach differs from prior self-supervised clustering by the proxy mechanism we use to incorporate all negative samples efficiently.
Revised: Our approach differs from prior self-supervised clustering by the proxy mechanism we use to incorporate all negative samples efficiently.

Original: The strength of this simple approach indicates that, despite the increased interest in graphs, effective contrastive learning remains relatively unexplored.
Revised: The strength of this simple approach indicates that, despite the increased interest in graphs, effective contrastive learning remains relatively unexplored.

Original: A A PPENDIX A. P ROOFS OF T HEORETICAL R ESULTS - D ERIVATION OF E QUATION
Revised: A A PPENDIX A. P ROOFS OF T HEORETICAL R ESULTS - D ERIVATION OF E QUATION

Original: Assume node embeddings Z = { z 1 , z 2 , z 3 . . .
Revised: Assume node embeddings Z = { z 1 , z 2 , z 3 . . .

Original: z N } , clusters µ = { µ 1 , µ 2 . . .
Revised: z N } , clusters µ = { µ 1 , µ 2 . . .

Original: µ C } , a label assignment operator label( z i )
Revised: µ C } , a label assignment operator label( z i )

Original: such that µ a = (cid:80) Ni =1 1 [ i ∈ label( z i )=
Revised: such that µ a = (cid:80) Ni =1 1 [ i ∈ label( z i )=

Original: a ] · z i , a hyperparameter τ related to the temperature in contrastive loss and similarity
Revised: a ] · z i , a hyperparameter τ related to the temperature in contrastive loss and similarity

Original: [Equation]
Revised: [Equation]

Original: We use sim( z i , z j ) as the shorthand notation for similarity( i, j, z i , z j ) interchangeably for brevity.
Revised: We use sim( z i , z j ) as the shorthand notation for similarity( i, j, z i , z j ) interchangeably for brevity.

Original: We begin with Equation 1, which is the popular form of contrastive loss (Hu et al., 2021; Kulatilleke et al., 2022).
Revised: We begin with Equation 1, which is the popular form of contrastive loss (Hu et al., 2021; Kulatilleke et al., 2022).

Original: With τ as the temperature parameter, γ ij the relationship between nodes i, j , the loss for the i th can be expanded as:
Revised: With τ as the temperature parameter, γ ij the relationship between nodes i, j , the loss for the i th can be expanded as:

Original: [Equation]
Revised: [Equation]

Original: where, the first part on the right corresponds to the negative node contrasting portion and the second portion contrasts the positives for node i .
Revised: where, the first part on the right corresponds to the negative node contrasting portion and the second portion contrasts the positives for node i .

Original: From Equation 14, for all nodes N , we take to negative node contrasting portion, by averaging over N nodes to obtain:
Revised: From Equation 14, for all nodes N , we take to negative node contrasting portion, by averaging over N nodes to obtain:

Original: [Equation]
Revised: [Equation]

Original: Note our use of the more concise sim() and the compact e notation over exp() interchangeably for compactness reasons.
Revised: Note our use of the more concise sim() and the compact e notation over exp() interchangeably for compactness reasons.

Original: We expand Equation 15, together with e 0 = 1 in cases where i =
Revised: We expand Equation 15, together with e 0 = 1 in cases where i =

Original: j , as:
Revised: j , as:

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we can express the cluster based contrastive loss as:
Revised: we can express the cluster based contrastive loss as:

Original: [Equation]
Revised: [Equation]

Original: with the following expansion:
Revised: with the following expansion:

Original: [Equation]
Revised: [Equation]

Original: If, loss minNN > loss maxCC
Revised: If, loss minNN > loss maxCC

Original: [Equation]
Revised: [Equation]

Original: we have loss NN loss CC > 1 .
Revised: we have loss NN loss CC > 1 .

Original: Next we show the conditions necessary for establishing this inequality.
Revised: Next we show the conditions necessary for establishing this inequality.

Original: As 0 ≤ sim ≤ 1 .
Revised: As 0 ≤ sim ≤ 1 .

Original: [Equation]
Revised: [Equation]

Original: we obtain the min using sim min = 0 :
Revised: we obtain the min using sim min = 0 :

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can obtain the max , using sim max = 1 . 0 : loss
Revised: Similarly, we can obtain the max , using sim max = 1 . 0 : loss

Original: [Equation]
Revised: [Equation]

Original: Combining Equation 19 and Equation 20, we establish the necessary condition for our inequality,
Revised: Combining Equation 19 and Equation 20, we establish the necessary condition for our inequality,

Original: Equation 7 as;
Revised: Equation 7 as;

Original: [Equation]
Revised: [Equation]

Original: This derivation is used in Section 2.1, where we show how the condition is almost always satisfied in real graphs.
Revised: This derivation is used in Section 2.1, where we show how the condition is almost always satisfied in real graphs.

Original: As a result, loss NN upper bounds loss CC .
Revised: As a result, loss NN upper bounds loss CC .

Original: Note that a lower loss is better.
Revised: Note that a lower loss is better.

Original: [Table]
Revised: [Figure]

Original: [Table]

Revised: VS A CCURACY

Original: [Figure]
Revised: [Figure]

Original: A. H YPERPARAMETER BEHAVIOUR WITH AND WITHOUT P AM C
Revised: A. H YPERPARAMETER BEHAVIOUR WITH AND WITHOUT P AM C

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original:  hhar 0.65 0.
Revised: on hhar 0.65 0.

Original: A CC _ AV G 0.
Revised: A CC _ AV G 0.

Original: 0.001 LR on reut 0.76 0.
Revised: 0.001 LR on reut 0.76 0.

Original: A CC _ AV G 0.
Revised: A CC _ AV G 0.

Original: 0.001 LR on acm 0.84 0.
Revised: 0.001 LR on acm 0.84 0.

Original: A CC _ AV G 0.
Revised: A CC _ AV G 0.

Original: 0.001 LR on dblp 0.50 0.55 0.
Revised: 0.001 LR on dblp 0.50 0.55 0.

Original: A CC _ AV G 0.
Revised: A CC _ AV G 0.

Original: 0.001 LR on cite 0.64 0.
Revised: 0.001 LR on cite 0.64 0.

Original: A CC _ AV G
Revised: A CC _ AV G

Revised: A. Q UALITATIVE R ESULTS

Revised: [Figure]

Revised: We use UMAP (McInnes et al., 2018), in Figure 7, to get a visual understanding of the raw and learnt embedding spaces.

Revised: Except for USPS, which is a distinct set of 0 · · · 9 handwritten digits (raw 1), we see that all other datasets produce quite indistinguishable clusters.

Revised: Clustering is nearly non-existent in the (last 3) graph datasets.

Revised: This clearly shows a characteristic difference in graph data, which can lead to high samplings bias.

Revised: Note that ˆ µ ̸ = µ for any meta-node

Revised: [Figure]

Revised: A. D ATASET SIZE VS GPU MEMORY AND TIME , WITH AND WITHOUT P AM C

Revised: [Figure]

Revised: A. I MPROVED GPU MEMORY AND TRAINING TIME ON SYNTHETIC CITE DATASET

Revised: T i m e pe r t r a i n i ng epo c h ( s )

Revised: G P U m e m o r y pe r

Revised: ep

Revised: c

Revised: oh

Revised: [Figure]

Revised: A. R ESULTS FROM SELECTED WORKS ON THE CITE DATASET

Revised: [Table]

Revised: A. GPU WORKLOAD BREAKDOWN WITH AND WITHOUT P AM C.

Revised: [Table]