From: ICLR.cc/2023/Conference/Paper2344/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This manuscript proposes a new method to quantify the complexity of an image using hierarchical clustering guided by the minimum description length principle (MDL). The method can correctly distinguish signal from noise and produces a zero score for a white-noise image. Experiments were conducted on various datasets such as ImageNet and MNIST.

The reviewers raised the below concerns: 
- The computational complexity of the method is high.
- The method depends on GMM initialization and MDL does not seem effective.
- The method involves careful hyperparameter tuning.
- An application of image complexity is missing.

Given the current form of the submission that is below the bar of ICLR, the meta reviewer cannot recommend acceptance at this stage. The authors are suggested to revise the paper following the reviewers' comments.
justification_for_why_not_higher_score: N/A
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: The reviewers were aware of the authors' rebuttal, saying that "we are not proposing our method to solve some specific applied problem. Rather, it is to facilitate understanding of the concept ...". An application of image complexity is missing, which is not sufficient to justify acceptance.

From: ICLR.cc/2023/Conference/Paper2344/Authors
title: Author response
comment: We thank reviewer WmZd for their comments. Below are our responses.

**Our proposed metric has high computational complexity.**
Comparing our run time to that of previous methods is not quite a fair comparison, because our method is not merely a modification of existing methods to improve them slightly, it is a completely different approach with completely different properties. In particular, it consistently and reliably succeeds at assigning white noise low complexity, whereas all existing methods completely fail at this. 

**In most experiments the K_max=8 value is selected, which suggests that the MDL peak occurs for values of K higher than 8.**
Thank you for taking the time to read the supplementary material in detail. This is true, but it does not matter whether we use the ‘true’ value of K, i.e., the value that gives the overall lowest MDL. What matters is that, within the range of K that we consider, MDL can select different values for different images, in a systematic way, which leads to a systematic difference in complexity scores. For example, real-world images are normally, as you note, found to have K_max=8, whereas noise images have K_max=1. This leads to a systematic difference in the score assigned to ImageNet images vs. white noise images.

**The proposed metric requires several hyperparameters just to compute a quality measure.**
What we propose is not just a quality measure, it is a new way to conceive of the concept of complexity. This is a fundamental concept, not just in computer science but also in many other fields of study; and, in the context in which we conduct our experiments, there currently does not exist a good method for quantifying it. 

**No practical application of the method to a problem related to image complexity is given, applications are mentioned but the proposed method is not applied in such applications.**
The main purpose of our paper is to better understand the nature of complexity, by quantifying it in a way that better agrees with our intuition. We mention applications to highlight that practical problems often do come up against the question of what complexity is, but we are not proposing our method to solve some specific applied problem. Rather, it is to facilitate understanding of the concept, and so it is in that context that we evaluate it experimentally, by showing its performance on a diversity of settings and image types. 

**GMM training depends on initialization, a deterministic metric would be preferrable.**
Our method conceives complexity as having a complex composition, i.e., taking a large number of steps to compose. We argue, and our results demonstrate that this is the correct way to conceive complexity, however, exploring possible compositions is inherently taxing computationally, so we make recourse to approximation via clustering. When measuring a complex property (which complexity itself is), we would argue that it is not feasible to expect to be able to compute an exact deterministic metric. This arises in other areas of machine learning when trying to quantify a subtle concept, e.g., disentanglement in deep learning representations is often measured via the training of a classifier [1-2]

[1] Ridgeway, K. and Mozer, M.C., 2018. Learning deep disentangled embeddings with the f-statistic loss. Advances in Neural Information Processing Systems, 31.
[2] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S. and Lerchner, A., 2016. beta-VAE: Learning basic visual concepts with a constrained variational framework.

**What is \alpha_max and \alpha_min (eq. 4)?**
Thanks; we have added the missing definition of these terms.

From: ICLR.cc/2023/Conference/Paper2344/Authors
title: Author response
comment: We thank reviewer gwjr for their comments. Below are our responses.

**Complexity analysis and comparison with other baseline methods is needed.**
Total processing time is ∼10s per image. This can be reduced to ∼4s by only checking up to 5 clusters, with essentially the same results. We also expect that processing multiple images with the same model could speed up performance, as the clustering model for one image could be initialized to the fit clustering model of previous images, to give a warm start. This is slower than other methods, which all take < 0.5s per image. However, this is not quite a fair comparison, because our method is not merely a modification of existing methods to improve them slightly, it is a completely different approach with completely different properties. In particular, it consistently and reliably succeeds at assigning white noise low complexity, whereas all existing methods completely fail at this.

**It is hard to tell the advantage of the proposed complexity on clean data. Scores should be shown for individual images.**
Scores for individual images, and worked examples, are provided in the supplementary material. We do not include them in the main paper, because we instead chose to evaluate by comparing classes of images. As we argue at the beginning of Section 4, we do not feel that the scores for individual images are the best way to assess the quality of an image complexity measure, because there is too much variability and subjectivity. Rather, comparing different classes of images (e.g., different datasets) is more reliable.

**There is no study performed for the number of clusters K.**
The exploration of different values for K, along with a rigorous method for choosing the best value, is a central part of our method. This is described in detail in Section 3.1.

From: ICLR.cc/2023/Conference/Paper2344/Authors
title: Author response
comment: We thank reviewer p7WR for their comments. Below are our responses.

**It would be better to show some individual images from ImageNet.**
Scores, along with worked examples, for individual images are provided in the supplementary material. We do not include them in the main paper, because we instead chose to evaluate by comparing classes of images. As we argue at the beginning of Section 4, we do not feel that the scores for individual images are the best way to assess the quality of an image complexity measure, because there is too much variability and subjectivity. Rather, comparing different classes of images (e.g., different datasets) is more reliable.

**It is unclear in what tasks/applications we need to measure image complexity.**
At the beginning of Section 2, we have suggested several tasks that make use of image complexity measures. The two most important are remote sensing and psychometrics. It is relevant to remote sensing especially in the context of searching for extraterrestrial life from satellite images, as this involves identifying images that depict some meaningful structure, as opposed to near-uniform textures or random noise. It is relevant psychometrically, because it moves towards being able to quantify our intuitive notion of complexity. It gives a procedure for computing a number that (modulo the variability and subjectivity of assessing complexity, which we have designed our experiments so as to limit) assigns a high number to images that we see as complex, and a low number to images that we see as simple. To our knowledge, there does not currently exist a procedure for doing this.

**From Table 1, apart from white noise, the entropy method also works well.**
We would argue that the assignment of a high score to noise is a cardinal failing of a complexity metric, and that, even if it assigns reasonable scores to non-noisy images, it cannot be thought of as capturing what we mean by complexity. 

**What is the computational cost to calculate the proposed score?**
Total processing time is ∼10s per image. This can be reduced to ∼4s by only checking up to 5 clusters, with essentially the same results. We also expect that processing multiple images with the same model could speed up performance, as the clustering model for one image could be initialized to the fit clustering model of previous images, to give a warm start.

**Does varying the size of the object in an image change complexity score?**
Thanks for this suggestion. We will explore this and include the results if completed in time.

From: ICLR.cc/2023/Conference/Paper2344/Reviewer_p7WR
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This manuscript proposes a new method to quantify the complexity of an image.
The proposed method is based on hierarchical clustering guided by the minimum description length principle (MDL). 
Different from existing works (e.g., entropy or fractal dim), the method can correctly distinguish signal from noise and produces a zero score for a white-noise image. 
Experiments are conducted on various datasets such as complex ImageNet and easy MNIST. 
strength_and_weaknesses: Pros 
- the method is clearly elaborated and well-motivated.
- experiment results show the method can reasonably reflect the complexity of images from different datasets.
- studies of adding various gaussian noise (Table 3) and varying image resolutions are interesting.  

Cons
- it is unclear what score for each image instance is in one dataset. For example, it would be better to show some instances in Imagenet and see if these images are with different complexity scores.  
- it is unclear in what tasks/applications we need to measure image complexity. 
- from Table 1, apart from white noise, the entropy method also works well. It raises another question what is the computational cost to calculate the proposed score?

Other suggestions
- assume there is an object in one image, e.g., a cat. when varying this cat's sizes such as 8x8,16x16, 32x32, or 48x48 (the image is fixed-size, 64x64), what changes will their scores make?

clarity,_quality,_novelty_and_reproducibility: the method is new and clearly introduced.
summary_of_the_review: Overall, the manuscript is well written. My main concern is what applications or tasks we need to measure image complexity.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2344/Reviewer_gwjr
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes to measure the complexity of images by a hierarchy of their patches using MDL clustering. Experiments are conducted on different image data sets to show that the proposed method distinguish images from different data sets.
strength_and_weaknesses: Strength:

1. The paper is well-organized and easy to follow. 
2. The experiments show that the proposed method won't assign a high complexity to noisy images.

Weakness:

1. Since the algorithm evaluates a dataset's complexity based on each image's evaluation, it would be necessary for the authors to provide some complexity analysis of the proposed method and compare it with the complexity of other baseline methods. 

2. The evaluation is limited. The experiment results show that the proposed methods won't assign high complexity to noisy images. However, it's hard to tell the advantage of the proposed complexity on clean data. Since there is no ground truth for image complexity, the authors should show some comparisons on some sample images to show that on what kind of image patterns proposed method would assign a higher complexity than compared methods. 

3. No study is performed for the number of clusters K. Since clustering is the key step for proposed methods, it would be necessary to study if the choice of K will influence the evaluation results on different datasets. 
clarity,_quality,_novelty_and_reproducibility: This paper is well written, and the proposed method is incrementally novel. While no source code is provided, it may not be difficult to implement the proposed method based on its description.
summary_of_the_review: Due to the issues mentioned in the weakness, I feel that this paper is below the acceptance bar of ICLR in its current form.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2344/Reviewer_WmZd
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The paper presents a method for computing 'image complexity'. The proposed technique applies patch-based clustering at various scales using GMM and MDL and computes the entropy of the partition in each scale. The total entropy is the sum of the scale entropies.
strength_and_weaknesses: Strength
The proposed idea is interesting and suggests a novel way of assessing the variation in image content. 

Weaknesses
- A major concern is related to the computational complexity of the method which is very high compared to other approaches.
- The method depends on GMM initialization and MDL does not seem to be quite effective (in most experiments the K_max=8 value is selected).
- The method involves several hyperparameters and makes the GMM assumption.
 - A practical application of the method to a problem related to image complexity is missing. 
clarity,_quality,_novelty_and_reproducibility: The proposed idea is interesting and suggests a novel way of assessing the variation in image content. However, there are several concerns regarding the implementation of the idea. 
The paper is well written in general, although presentation quality could be improved.
The authors provide the source code of their method.
summary_of_the_review: The paper presents a method for computing 'image complexity'. The proposed technique applies patch-based clustering at various scales using GMM and MDL. 
and computes the entropy of the partition in each scale. The total entropy is the sum of the scale entropies. 
The paper is well written in general and relies on an interesting idea.
However there are several concerns to be addressed:
1) Execution time is much higher compared to other methods.
2) GMM training depends on initialization, thus the final result could be different for different executions of the algorithm. I would prefer a deterministic image quality measure. 
3) The MDL criterion does not seem to be sufficiently  effective for estimating the number of components. In most examples presented in the appendix, the Kmax value (=8) was selected. This indicates that the MDL peak may occur for values of K higher than 8.
4) The method involves several hyperparameters such as K_max, patch size, number of scales, \epsilon. Many hyperparameters should be set and several assumptions are made (eg. Gaussian modelling) for just computing a quality measure.
5) What is \alpha_max and \alpha_min (eq. 4)? 
6) Several applications related to 'image complexity' are mentioned in the beginning of the section 2. However, there is no use of the proposed approach in any of those applications.  
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold