Paper: CutSharp: A Simple Data Augmentation Method for Learned Image Compression (ICLR-2024)

Reviewer comment (Experiments):
First of all, I am not sure if data augumentation is really necessary for image compression at the first place. Image compression is an unsupervised task, and the data is very very easy to acquire. Any image dataset can be directly used. And there are aboundant large datasets, such as Openimage (9,000k) and LAION-5b (5,000,000k), that have not been used by image compression at all. Instead of performing data augumentation on a small dataset (the authors use 300k subset of Openimage), directly use a larger dataset seems to be the most straightforward way. Afterall, natural image dataset are already very very large. For example, the authors train the model with 90 epochs on 300k images, which means that any image dataset with 27,000k is enough to provide a different image for each iter. And obvious candidates for this task would include LAION-400M (400,000k), LAION-5B high res (170,000k) and LAION-5B (5,000,000k).

Author response:
### W1. Necessity of data augmentation for unsupervised tasks
- We believe that data augmentation is necessary to efficiently use the same dataset, even for unsupervised tasks.
- In this vein, CutSharp consistently improves performance compared to no data augmentation (albeit slight), when using the same training dataset.
- [Please refer to Table 1] Furthermore, we showed that just collecting data is not the answer, because the performance trained using top40-60% dataset (size: 60K) is better than the performance trained using entire dataset (size: 300K). It implies that large dataset size does not guarantee higher performance for LIC.

Reviewer comment (Evaluation):
Second, the empirical results is too weak to justify that the data augumentation. The best improvement for BD-BR is -0.66\%, which is too marginal. For example, a -5.0\% improvement in BD-BR is approximately 0.2 dB improvement in BD-PSNR. And a BD-BR of -0.66\%, without reporting standard deviation, is too weak to support a paper.

Author response:
### W2. Slight improvement of CutSharp
- [Please refer to Table 4] We admit that CutSharp did not bring much performance improvement. Nevertheless, please note that CutSharp is an augmentation technique with consistent improvement across various LIC models.
- We will update the standard deviation of the results for 3 runs.

Reviewer comment (Experiments):
It would be much more convincing if the authors are dealing with special images that are hard to acquire. For example, medical image, remote sensing image. Currently the authors only test their methods on natural image, which is too cheap to acquire and eliminates the need for data augumentation.

Author response:
### Q1&Q2. Specific-domain where acquiring data is hard
- Thank you for the insightful comment! However, we think that this comment focuses on ``domain'' for LIC, which is slightly different from the direction our paper is aiming for.
- This is because we addressed B-RGB-SD as an intrinsic characteristic of an image (i.e., sample-wisely), but domain is an intrinsic characteristic of a cluster of images.
- In addition, we should design four settings for addressing this comment:
  - Setting1. training dataset - natural (i.e., acquiring data is easy), test dataset - natural
  - Setting2. training dataset - natural (i.e., acquiring data is easy), test dataset - speical
  - Setting3. training dataset - speical (i.e., acquiring data is hard), test dataset - natural
  - Setting4. training dataset - speical (i.e., acquiring data is hard), test dataset - speical
- Although the comparison between above settings would be greatly impactful, it is likely to be a new work. Rather, we would like to state that through our work, data augmentation is generally necessary for LIC.

Reviewer comment (Experiments):
The current empirical results is too weak, and the authors should provide an improvement of 2\% to 5\% improvement of BD-BR to support a paper, this might be achieved by chaning the domain of image.

Author response:
### Q1&Q2. Specific-domain where acquiring data is hard
- Thank you for the insightful comment! However, we think that this comment focuses on ``domain'' for LIC, which is slightly different from the direction our paper is aiming for.
- This is because we addressed B-RGB-SD as an intrinsic characteristic of an image (i.e., sample-wisely), but domain is an intrinsic characteristic of a cluster of images.
- In addition, we should design four settings for addressing this comment:
  - Setting1. training dataset - natural (i.e., acquiring data is easy), test dataset - natural
  - Setting2. training dataset - natural (i.e., acquiring data is easy), test dataset - speical
  - Setting3. training dataset - speical (i.e., acquiring data is hard), test dataset - natural
  - Setting4. training dataset - speical (i.e., acquiring data is hard), test dataset - speical
- Although the comparison between above settings would be greatly impactful, it is likely to be a new work. Rather, we would like to state that through our work, data augmentation is generally necessary for LIC.

Reviewer comment (Experiments):
Authors propose using RGB standard deviation of RGB blocks to get a measure for the difficulty of an image, and show in Fig 2. how this correlates to bpp/PSNR. They report a person correlation. The weakness lies in no baselines given: if we have a codec (e.g., even just JPEG) we can trivially calculate bpp/PSNR and with two or more images we could fit a line. How does that compare to the proposed method?

Author response:
### W1. No baselines of B-RGB-SD
- [Please refer to Section 4 (Figure 2)] We aimed to establish a correlation between compression difficulty (e.g., bpp/psnr after training LIC) and the intrinsic characteristics of the image before training (i.e., codec-independently). This is important because it would enable us to design image augmentation techniques.
- [Please refer to Appendix A] With this intention, we introduced Image-wise RGB-SD (I-RGB-SD) and Block-wise RGB-SD (B-RGB-SD) for the Kodak dataset. As we mentioned in Section 4, the Pearson correlations between B-RGB-SD and bpp/psnr are 0.8957/-0.7989. However, the Pearson correlations between I-RGB-SD and bpp/psnr are 0.0692/0.0023.
- For comparison, a measure that has a stronger correlation with compression performance is better. Therefore, B-RGB-SD is a better measure than I-RGB-SD.
- Furthermore, we calculate the Pearson correlations between block-wise entropy (calculated by skimage.measure.shannon_entropy) and bpp/psnr. The values are 0.7521/-0.7682, which are weaker than the correlations between B-RGB-SD and bpp/psnr. (will be updated.)

Reviewer comment (Novelty):
In Fig.4, we also don't see a novel insight: training on hard examples improves hard inference, but makes easy images worse, and vice-versa. This is just classical ML: you are good where you train.

Author response:
### W2. Utilization of B-RGB-SD
- [Please refer to Table 1 and Figure 4] From the results, we would like to highlight that B-RGB-SD is a pretty accurate measure to estimate compression-related difficulty, rather than strong actionable message.
- As you mentioned, "training on *easy* examples improves *easy* inference, and training on *hard* examples improves *hard* inference" is undoubted.
- However, without the use of "B-RGB-SD," we could not explicitly demonstrate this hypothesis for LIC.
- [Please refer to Section 5] Therefore, we proposed a new augmentation method by considering B-RGB-SD of training dataset, inspired by the fact that a small portion of an image has relatively higher B-RGB-SD than the rest of it, in general.

Reviewer comment (Evaluation):
The CutSharp augmentation provides very minuscule gains only (Table 4) and in Fig 5 we see no obvious visual difference, which is not surprising given the tiny PSNR differences (e.g., 0.05dB between a) and d)).

Author response:
### W3. Slight improvement of CutSharp
- [Please refer to Figure 5 and Table 4] We admit that there is not much difference in quality comparison because CutSharp did not bring much performance improvement. Nevertheless, please note that CutSharp is an augmentation technique that consistently improves the performance of various LIC models.

Reviewer comment (Experiments):
Generally, people use augmentation because labeling is hard. In image compression, one can just get more data, as no labels are required. It would have been insightful to double the training set and see how that affects BD-rate gains. The reviewer would be surprised if CutSharp gets bigger gains than those obtained from doubling the dataset.

Author response:
### Q1. Necessity of data augmentation for unsupervised tasksd
- We agree that collecting data is not difficult for unsupervised tasks.
- [Please refer to Table 1] However, we demonstrated that just increasing dataset size is not the answer, because the performance trained using top40-60% dataset (size: 60K) is better than the performance trained using entire dataset (size: 300K). Rather, we suggest B-RGB-SD should be considered, even for collecting data for LIC.
- Furthermore, data augmentation like CutSharp can improve performance even when using the same dataset.
- In addition, we will update the results of CutSharp using random 50% of the Open Images dataset, following your comments.

Reviewer comment (Novelty):
My major concern is the limited technical novelty and contribution of the paper. CutSharp is a simple idea but just a variant of the sharpening method -- sharpen the random region of an image. It compensates for the imbalance of B-RGB-SD of the cropped image.

Author response:
- [Please refer to our general response] We believe that our work has valuable contributions as the first data-centric approach for LIC.
- With regard to CutSharp, although we use the basic augmentation technique of "Sharpening" and adopt "Cut" idea, we analyzed existing augmentation techniques for LIC tasks and adopted "Cut" idea based on our B-RGB-SD analysis and intuition.

Reviewer comment (Experiments):
The experiments in the paper are not convincing and the overall performance of CutSharp is not strong enough. First of all, the improvement of CutSharp on Kodak (-0.39%) and CLIC (-0.40%) is worse than Sharpening=0.50 on Kodak (-0.46%) and ColorJitter=0.2 on CLIC (-0.63%).  The visualization of learned motion patterns in Fig. 5 seems not appealing enough.

Author response:
- [Please refer to Table 2 and Table 3] We would like to highlight that CutSharp can improve for both datasets, although the improvement is slight. The improvement on both datasets was not achieved when using only one fixed augmentation.
- [Please refer to Figure 5 and Table 4] We admit that there is not much difference in quality comparison because CutSharp did not bring much performance improvement. Nevertheless, please note that CutSharp is an augmentation technique with consistent improvement across LIC models.

Reviewer comment (Evaluation):
The analysis of B-RGB-SD is not convincing enough. The experiments reveal that B-RGB-SD is more effective in capturing the compression-related difficulty of an image. However, it is not clear how B-RGB-SD influences the compression performance.

Author response:
- [Please refer to Figure 3] We showed that on average, as the B-RGB-SD of the training dataset increases (i.e., blue → orange → green → red → purple), the LIC models tend to target higher PSNR at higher BPP, even if objective functions are the same.
- [Please refer to Figure 4] Next, we provided a sample-wise assessment of the trade-off between rate and distortion by calculating the BD-rate.

Reviewer comment (Experiments):
Sec.5.2 said "In other words, during training, the original image is cropped to a smaller size, which is more likely to have a low B-RGB-SD because typically only a small portion of the original image has high B-RGB-SD." -> How did you come to this conclusion? More experiments are suggested to support this claim.

Author response:
- [Please refer to Section 5.2] Firstly, we strongly apologize for providing incorrect information. We would like to modify sentences from "In other words, during training, the original image is cropped to a smaller size, which is more likely to have a low B-RGB-SD because typically only a small portion of the original image has high B-RGB-SD. Therefore, to compensate this imbalance, we apply Sharpening to an arbitrary part of the cropped image during training, to increase the B-RGB-SD." to "Typically, only a small portion of the original image has relatively higher B-RGB-SD than the rest of it, as described in Table 6 in Appendix A. Therefore, to imitate this imbalance, we apply Sharpening to an arbitrary part of the cropped image during training, to increase the B-RGB-SD."
- [Please refer to Appendix A (Table 6)] We provided B-RGB-SD histogram (before averaging) of Kodak dataset. In general, the histograms of images are negatively-skewed distributions, and this property is the intuition behind CutSharp.

Reviewer comment (Experiments):
If the claim of Q1 is true, the problem of imbalance on B-RGB-SD may come from the crop process during training. The crop process leads to low B-RGB-SD samples. I am wondering what if you drop the low B-RGB-SD samples?

Author response:
- In Section 5.2, low B-RGB-SD was expressed negatively, but I would like to correct it in Q1 and apologize once again.
- [Please refer to Table 1] Regardless of Q1, Table 1 provides a rough answer to this question. Using the training dataset with top 40-60% B-RGB-SD brings performance improvement, which implies that training samples with high or low B-RGB-SD can be harmful.
- We will update the results using different training dataset setups (e.g., top0-80%, top10-90%, top20-80%, etc).

Reviewer comment (Evaluation):
The finding "_models trained on a subset of dataset, based on B-RGB-SD can outperform models trained on the entire dataset_" is meaningless. It is just some kind of overfitting. The test dataset, either Kodak or CLIC, only contain a few dozen images. When dividing the entire dataset into different groups according to the B-RGB-SD metric, one of the divided subsets happens to be closer to the test set in some of the statistical properties (e.g, B-RGB-SD). So it is no doubt that the model trained on this subset will lead to a better result on the test set. Figure 4 can only prove that the subset of top 20-40% B-RGB-SD is closer to Kodak than other subsets and the subset of top 60-80% B-RGB-SD is the one closest to CLIC dataset.

Author response:
- [Please refer to Figure 4] As you mentioned, a smaller distribution shift between the training dataset and test sample undoubtedly leads to performance improvement: training on *easy* examples improves *easy* inference, and training on *hard* examples improves *hard* inference.
- However, without the use of "B-RGB-SD," we could not explicitly demonstrate this hypothesis for LIC.
- In other words, B-RGB-SD is a measure that captures compression-related difficulty (i.e., easy or hard), which we have highlighted through our work.
- [Please refer to Section 5] Therefore, we proposed a new augmentation method by considering B-RGB-SD of training dataset, rather than matching distribution between training and test datasets, because we realized the infeasibility and challenges of matching distributions.

Reviewer comment (Theory):
Another major weakness of this paper is lack of motivation. Only the data augmentation method (Cutsharp) is given, but there is no explanation as to why it is designed this way.

Author response:
- [Please refer to Section 5.1] We described the details and ineffectiveness of four color-related augmentation techniques with B-RGB-SD: positive/negative ColorJitter, Blurring, and Sharpening. Among these techniques, Sharpening exhibits the lowest rate–distortion performance degradation for both Kodak and CLIC-P datasets.
- [Please refer to Table 2, Table 3, and Table 4] Nevertheless, Sharpening could not improve the rate–distortion performance on both datasets. Therefore, we borrowed the "Cut" idea, inspired by the fact that a small portion of an image has relatively higher B-RGB-SD than the rest of it, in general (as described Table 6). Finally, this leads to consistent improvement.