From: ICLR.cc/2023/Conference/Paper516/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: *Summary*: The paper proposed a bayesian framework to avoid collapse in an online clustering framework. The framework is evaluated on online clustering of different image benchmarks where it shows improved clustering performance without collapse.

*Strengths*: (1) The paper proposes a way to avoid collapse in online clustering without using data augmentation. This is widely applicable to many domains that rely on online clustering for representation learning. (2)  The authors extensively analyze the clustering quality in this work on several datasets. This is valuable. 

*Weaknesses*: (1) While online clustering, as noted by the authors too, is mainly used for representation learning, the paper does not contain experiments that analyze this property. The authors mainly analyze the "clustering quality" while not analyzing the underlying representations. This issue is raised by many reviewers too, and I believe is an important weakness of this work as it does not answer whether this method is a useful online deep-clustering method. (2) A uniform distribution of the clusters is a central assumption in this work which is violated in many natural datasets. The claims in this work are validated on balanced datasets.
justification_for_why_not_higher_score: All reviewers raised questions about the analysis and presentation of this work. I am also not convinced of its utility since the authors over-index on clustering quality as a metric.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper516/Reviewer_pkxS
title: Response to rebuttal
comment: 1) The distinction between online and offline methods: Author's response is valid. Authors should cite the papers in this comment and lead readers to these papers in case there is any confusion. 

2) "well-documented that data augmentation improves results, but it is limited to cases where we know the semantic symmetries": In this case, authors should run experiments on the dataset where we don't know the transformations and show efficacy of their method on those datasets. It is clear that the current method may not be too useful for images where we know what the good transformations are. 

In any case, authors have addressed some of my confusion but there are still major improvements in clarity and experiments that are needed. I am improving my score from 1 to 3.

From: ICLR.cc/2023/Conference/Paper516/Authors
title: Author response
comment: We thank reviewer VqHu for their comments. Below is our response.

**Their is no evidence why hard assignments are better to regularize than soft assignments.**
As shown in Section 4.3, soft assignment regularization can produce undesirable results, where the soft probabilities over clusters for each point as squeezed to be close to each other without changing the argmax. What we want to enforce is that each cluster has equal numbers of points assigned to it, this is for example what is specified in the ground truth labels (the datasets we consider are equally partitioned). Uniform soft assignments is not an end in itself, and, as we show, both empirically and theoretically, neither is it a suitable means to encouraging uniform hard assignments

**What is the advantage of not requiring data augmentation?**
Data augmentation limits applicability to cases where we know the identity-preserving transformations to apply. Methods that do not use data augmentation are therefore more widely applicable.

**The experimental design does not seem very fair. The results of the comparison methods in this paper are not the same as they should be in the original papers. It seems that the results are underestimated, and such experimental comparisons are unacceptable and do not give a reasonable explanation.** 
We have implemented the methods for avoiding collapse as they are described in the corresponding papers. The results are not the same because we are not implementing/investigating the proposed deep clustering models themselves, but rather a single component, namely the partition support technique for avoiding collapse. We test these existing partition support techniques in the same context that we test our proposed partition support method.

**The results of the proposed method seem below the simple k-means. The performance of the proposed method should be discussed.** 
Firstly, Kmeans is (in its usual form) an offline clustering algorithm. Secondly, in experiments that we have run, Kmeans performs worse than the figures we report. Thirdly, most importantly, the relative performance to Kmeans does not matter so much, because are proposing and investigating just a single component of online deep clustering models, and we test this component without additional techniques to improve performance. The cluster accuracy figures are only reported to demonstrate that our proposed partition support method performs better than existing partition support methods.

**Reporting variance in cluster sizes is not common, give more details of it.**
We believe this is straightforward. It is just the definition of variance applied to the sizes of the clusters after clustering the dataset. It is a measure of collapse because high variance means some clusters are much larger than others.

**Too similar to simple k-means, can’t see the novelty, needs to be presented in more detail
We are proposing a partition support method. The clustering model that we use to test this proposed method we are not claiming to be novel. Note that this model we use is also very different to Kmeans because it is online and uses a deep encoder network. 

**The reviewer supposes the paper has poor reproducibility.**
We would be grateful for some experimental details that are missing, and we can add them to the paper.

From: ICLR.cc/2023/Conference/Paper516/Authors
title: Author response
comment: We thank reviewer Zch2 for their comments. Below is our response.

**The assumption of the same isotropic variance across all clusters is infeasible.**
This is a simple modelling choice that works well in practice and produces the intuitive optimization problem of Eq. (4).

**There is no ablation study on the sensitivity of parameters, such as the performance of the method under different variance.**
Thank you for this suggestion, we will explore it in a future work.

**The method is vaguely described, and doesn’t have a clear intuition.**
A significant part of the paper is devoted to describing the method in detail. The supplementary material contains even more detailed derivations and proofs. The intuition of the method is given in 3 different ways: (1) from the initial assumption of maximizing the posterior given a uniform prior: a uniform prior clearly discourages the collapsed solution in which almost all points are in the same cluster, as this is very far from uniform, (2) from the comparison with maximizing conditional entropy: again this discourages the collapsed solution, because such a solution has low entropy, and (3) from the direct penalization to large clusters that results from our rigorously derived optimization problem: this point is explicated at the top of page 5.

**The reference Zhan et al. Online Deep Clustering for Unsupervised Representation Learning is missing.**
This paper is already cited and described in our related work section.

**The experiments are hard to follow. For example, what is the meaning of hard variance and soft variance in Table 2?**
Hard variance is the variance in cluster size, where the size of each cluster is taken as the number of points that are assigned to that cluster. Soft cluster variance is the variance in cluster size, where the size of each cluster is taken as the sum of all the partial (soft) assignments to that cluster. We have added a line to the paper to clarify this.

**The experimental results do not directly support the conclusion claimed in the work.**
The results clearly show that our method produces the highest entropy in cluster assignments and the lowest variance in cluster sizes. These two figures are direct measures of collapse (low entropy and high variance indicate collapse), so the results directly show that our method is the best at avoiding collapse.

**The Bayesian framework is not new in data clustering. The following paper shows an example. Duan et al. Bayesian Distance Clustering.**
Clustering is a broad area of study and “Bayesian” refers to a wide variety of techniques. We certainly would not claim to be the first to use a Bayesian approach in a context involving clustering. What is new is the use of a Bayesian approach to prevent collapse in online deep clustering. The paper you mention, for example, does not involve deep learning, i.e., it is not in the area of deep clustering. It is quite different from our paper, it uses a different Bayesian approach to solve a different problem (robustness of model-based clustering). 

**The submission does not provide source code. The experimental results are not easy to reproduce.**
Code will be available on publication.

From: ICLR.cc/2023/Conference/Paper516/Authors
title: Author response
comment: We thank reviewer pkxS for their comments. Below is our response.

**The distinction between online and offline methods is vague, any deep clustering method that uses k-means to cluster can be online. Can authors add a table where they can select recent deep clustering methods and classify them into online and offline methods?**
The definition between offline and online clustering is straightforward, and not invented by us. Online clustering is a function from individual data points, or batches of data points, to cluster labels for each point. Offline clustering is a function from an entire dataset to cluster labels for each point. Online clustering could, for example, be performed with very limited memory, where each point (or batch thereof) was forgotten entirely after being clustered, both during training and inference. K-means, using Lloyd’s algorithm, is an offline clustering method. There are modifications to make it online [1-3], but these are not used, to our knowledge, by any deep clustering models that are based on kmeans.

[1] Liberty, E., Sriharsha, R. and Sviridenko, M., 2016. An algorithm for online k-means clustering. In 2016 Proceedings of the 18th Workshop on Algorithm Engineering and Experiments (ALENEX) (pp. 81-89). 
[2] Badawi, A. and Bilal, M., 2019. High-level synthesis of online k-means clustering hardware for a real-time image processing pipeline. Journal of Imaging, 5(3), p.38.
[3] Cohen-Addad, V., Guedj, B., Kanade, V. and Rom, G., 2021, March. Online k-means clustering. In International Conference on Artificial Intelligence and Statistics (pp. 1126-1134). PMLR.

**Some related work is missing.**
The first two papers you list are relevant. Thanks for these references, they are now included. The third paper is a clustering method only, it makes no use of deep learning. It is not relevant to our paper. We already cite the fourth paper.

**There should be an explicit definition of \theta_2 in equation 1.**
Thanks. We have added an explicit definition.

**Shouldn't \mu_k be inferred in this objective  of equation 1?**
No. There are two components to training, learning the parameters \theta_1 and \theta_2, and assigning cluster labels to each batch. Eq. (1) shows how to optimize the parameters given assignments, so that equation assumes we already have the assignments. The rest of Section 3 then shows how to compute the assignments for each batch, given the encodings and cluster centroids.

**How is the assumed uniform prior different from assumptions of prior methods? Not all the state of the art methods need stronger assumption of uniformity across the batch [3].**
The novelty of our method is not the assumption of a uniform prior (although the SK method, which is commonly used, has no probabilistic basis at all). Rather, our novelty is the use of this uniform prior in a Bayesian framework, which shows clearly the effect on the optimization problem of assuming a uniform prior.

**Combination assignment has some relation to deep clustering using consensus. Can authors comment more on this?**
We are not sure we understand this comment. Is this referring to the paper “Representation learning for clustering via building consensus”? That paper uses several techniques for improving clustering accuracy, which are unrelated to the topic of our paper. The method that it uses to prevent collapse is SK, which we already discuss at length.

**Where is Y in equation 4? How is Y transformed to Q in equation 5?**
Y, which is equal to (k_1, ..., k_N)  is under the argmin sign in Eq. (4). Does this answer your question? It can be seen from the definitions of Y and Q that they uniquely determine each other: in the ith row of Q, k_i=1 and all other entries are 0.

**Authors should explicitly mention the assumption that the number of clusters is known.**
Thanks. We have added this to the updated paper.

**Some methods get accuracy of 84%+ on CIFAR10 [1-4]. Does this mean data augmentation is too powerful and shouldn't be ignored?**
It is clear, and well-documented that data augmentation improves results, but it is limited to cases where we know the semantic symmetries, i.e., we know what transformations will leave the identity of the data point unchanged.

From: ICLR.cc/2023/Conference/Paper516/Authors
title: Author response
comment: We thank reviewer uDkk for their comments. Below is our response.

**Learned embeddings should be analysed using linear probes.**
This is a somewhat different angle to that taken in our paper, as it evaluates the representation produced by the encoder rather than the clustering model itself. We can explore adding this to the paper.

**Experiments should be performed to investigate varying the prior over clusters, rather than just having a uniform distribution.**
We can explore adding this to the paper. 

**Kmeans/agglomerative clustering should be used as a baseline.**
Kmeans/agglomerative clustering are not, in their usual form, online clustering models, so they are not relevant to the phenomenon of collapse, which is explored in our paper. 

**How is K chosen?**
K is set to the ground-truth number of classes, as is the case for all models that we compare to. We have added a line to the paper to clarify this.

**References should be placed in the main paper.**
Thanks, this is fixed in the updated paper.

From: ICLR.cc/2023/Conference/Paper516/Reviewer_uDkk
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes an online clustering objective that seeks to balance between assigning a datapoint to the closest centroid and keeping the cluster sizes similar, with the goal to prevent cluster collapse. Experimentation across datasets show good clustering performance and less susceptibility to collapse.
strength_and_weaknesses: Strengths: 

The paper is written in a clear, straight-forward manner, and provides good background and prior work references.

The paper presents a clear connection between a greedy assignment for each datapoint when posed as a set of clusters each represented as a multivariate normal with equal prior, and shows the relationship to maximization of the entropy of cluster labels.

Further, the paper presents the observation that methods that encourage high soft label assignment variance are not generally able to also encourage high hard label assignment. 

Evaluation for clustering is performed on CIFAR10, CIFAR100, FashionMNIST, and STL. Clustering accuracies show improvement with the proposed method. 

However, in some ways there are more analysis that could be performed: 

a) While there is evaluation of the produced clusters, there is not any substantial analysis on the quality of the learned embeddings that are produced. This is commonly performed e.g., by learning a simple classifier on top of the learned embeddings, as for example in Caron et al. 

b) The loss implicitly assumes equal distributions across the centroid. It would be interesting to explore the results when this assumption is violated. For example, some classes having many examples and others having few. 

c) There are perhaps additional other simple clustering baselines that could be used, such as k-means, agglomerative clustering, etc.

d) It's unclear to me how K (the number of clusters to use) was selected. Is it assumed that K is known for each dataset? If so, what happens when K is unknown?

clarity,_quality,_novelty_and_reproducibility: The paper is written in a mostly clear style. Some areas for improvement: Fig. 1 has a great deal of wasted space. 

Small nit:  Place the references were placed in the main paper rather than in the supplementary materials. 

summary_of_the_review: The paper presents a motivated idea for avoiding cluster collapse in online clustering, and shows in experimentation that this method is less likely to collapse. While effective, I would like to see more analysis of the results and the learned embedding representations. Even more convincing would be to see the impact of this method in a semi supervised training framework for feature representation learning. 

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper516/Reviewer_pkxS
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Authors point out that successful existing models for image clustering have employed various techniques, most of which require data augmentation or which aim to make the average soft assignment across the dataset the same for each cluster. Authors propose a method that does not use data augmentation and give empirical results on four image datasets. 
strength_and_weaknesses: Strengths:
1) Authors give clear intuition of their method. 
2) Authors give empirical results on four image datasets

Weaknesses:
1) Paper is not clearly written. Equations are not easy to understand or misuse some of the notations. 
2) Some of the important statements are vague (please check the summary of the review part for details). 
3) Assumptions are not explicitly stated. 
4) Empirical evidence is very weak and actually can conclude that data augmentation is important. 

clarity,_quality,_novelty_and_reproducibility: 1) Clarity and Quality: Paper is not clearly written and is sometimes vague when comparing to other methods. In some places, mathematical notations are not clear in equations. 
2) Novelty: Paper does propose novel objective but I am not sure if it will be used in image clustering because of poor empirical performance. 
summary_of_the_review: 1) Distinction between online and offline methods is very vague. Any deep clustering method that uses k-means to cluster can be online. Can authors add a table where they can select recent deep clustering methods and classify them into online and offline methods? 

2) Related work is missing some of the recent papers. Here are just a few of them [1-4].

3) What is \theta_2 in equation 1? One can infer indirectly but having \theta_2 there will make the objective clear. 

4) Shouldn't \mu_k be inferred in this objective (equation 1)? 

5) "We assume a uniform prior over K cluster label" How is this any different than assumption that some of the other state-of-the art methods make? Not all the state of the art methods need stronger assumption of uniformity across the batch [3]. Comparison of this part is vague and authors should clarify which of the existing methods don't require this assumption. Can authors clarify this?

6) Combination assignment has some relation to deep clustering using consensus. Can authors comment more on this?

7) Where is Y in equation 4? How is Y transformed to Q in equation 5? 

8) Proposed method (like most state-of-the-art methods) assumes that the number of clusters is known. Authors should explicitly mention this in the paper.  

9) Experimental results are not really impressive compared to state-of-the-art results. For example, there are methods which get accuracy of 84%+ on CIFAR10 [1-4]. Does this mean data augmentation is too powerful and shouldn't be ignored? 

[1] Van Gansbeke, Wouter, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. "Scan: Learning to classify images without labels." In European conference on computer vision, pp. 268-285. Springer, Cham, 2020.
[2] Deshmukh, Aniket Anand, Jayanth Reddy Regatti, Eren Manavoglu, and Urun Dogan. "Representation learning for clustering via building consensus." Machine Learning (2022): 1-38.
[3] Ma, Zhengrui, Zhao Kang, Guangchun Luo, Ling Tian, and Wenyu Chen. "Towards clustering-friendly representations: subspace clustering via graph filtering." In Proceedings of the 28th ACM international conference on multimedia, pp. 3081-3089. 2020.
[4]Niu, Chuang, Hongming Shan, and Ge Wang. "Spice: Semantic pseudo-labeling for image clustering." arXiv preprint arXiv:2103.09382 (2021).

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper516/Reviewer_Zch2
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The work solves the problem of online deep clustering with a Bayesian framework. One of the issues in the previous studies on deep clustering is collapse. The proposed Bayesian framework avoids collapse and does not require data augmentation for data clustering. Experiments are conducted on four images datasets to validate the performance of the method.
strength_and_weaknesses: Strength：
1）The proposed method does not require data augmentation. 
2）A Bayesian framework for online data clustering is proposed.

Weakness：
1）The assumption of the same isotropic variance across all clusters is infeasible.
2）There is no ablation study on the sensitivity of parameters, such as the performance of the method under different variance.
3）The intuition of the method avoiding collapse is unclear. The work needs a detailed justification on the effectiveness of the method.
4）Some reference is missing, like

Zhan et al. Online Deep Clustering for Unsupervised Representation Learning


clarity,_quality,_novelty_and_reproducibility: 1) The problem definition is clear, but the method is vague about how it solves the collapse in data clustering. 
2) The experiments are hard to follow. For example, what is the meaning of `hard variance’ and `soft variance’ in Table 2? The experimental results do not directly support the conclusion claimed in the work.
3) The Bayesian framework is not new in data clustering. The following paper shows an example.
Duan et al. Bayesian Distance Clustering
4) The submission does not provide source code. The experimental results are not easy to reproduce.

summary_of_the_review: The submission provides a method of Bayesian framework for online data clustering. However, the method is less persuasive on its design and the reported results. The assumption on the variance of all clusters is probably incorrect. Besides, some important studies are not compared in experiments. Also, the ablation study on the sensitivity of parameters is missing. Overall, the reviewer believes the submission is not ready to publish.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper516/Reviewer_VqHu
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a clustering method without data augmentation which regularizes hard assignments. The main contribution of this work can be basically summarized into one point: a Bayesian framework of hard assignments in online deep clustering models.
strength_and_weaknesses: Strength:
The paper attempts a method of clustering with hard regularization. The structure of the paper is complete and readable. 
Weaknesses:
1/ The motivation of this paper is not compellent. According to the paper, soft assignments are not an accurate measure of collapse, which is why the author regards hard assignments as the only way to solve such a problem. While the paper cannot give strict evidence to prove such a viewpoint. Besides, the reason why the proposed method without DA is confusing and unreasonable. The reviewer wonders about the author's deep thinking of it.
2/ The experimental design does not seem very fair. The results of the comparison methods in this paper are not the same as they should be in [1], [2], [3]. It seems that the results are underestimated, and such experimental comparisons are unacceptable and do not give a reasonable explanation. Besides, the results of the proposed method seem below the simple k-means. The performance of the proposed method should be discussed.
3/ The metric of variance in cluster size is not common. The reviewer wonders about more details of such a metric.
4/ The reviewer hopes to see a necessary pipeline figure of the proposed method.
[1] Caron M, Misra I, Mairal J, et al. Unsupervised learning of visual features by contrasting cluster assignments[J]. Advances in Neural Information Processing Systems, 2020, 33: 9912-9924.
[2] Hu W, Miyato T, Tokui S, et al. Learning discrete representations via information maximizing self-augmented training[C]//International conference on machine learning. PMLR, 2017: 1558-1567.
[3] Zhong H, Chen C, Jin Z, et al. Deep robust clustering by contrastive learning[J]. arXiv preprint arXiv:2008.03030, 2020.

clarity,_quality,_novelty_and_reproducibility: 1/ The clarity of this paper is good.
2/ The quality of this paper is below the borderline of the conference.
3/ The novelty of this paper is not sufficient. The nature of the proposed method is similar to simple k-means. The reviewer cannot see the core novelty of the proposed method itself which needs to be presented in detail.
4/ The reviewer supposes the paper has poor reproducibility.

summary_of_the_review: This paper proposes a clustering method without data augmentation which regularizes hard assignments. The main contribution of this work can be basically summarized into one point: a Bayesian framework of hard assignments in online deep clustering models. According to the paper, the proposed approach seems to show relatively good performance, but whether its performance exceeds the existing method still needs to be discussed. It is recommended that the author could consider the aspects presented in the weaknesses.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough