From: ICLR.cc/2023/Conference/Paper1737/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1737/Authors
title: Rebuttal for reviewer XMRc
comment: # Rebuttal for reviewer XMRc
## 1. The channel-wise normalization [a] and k-nearest neighbor clustering [b] are proposed many years ago. Both of them are widely used in the computer vision field. Therefore, the novelty of the paper is limited.
### **Response**: 
The major contribution of this work lies in identifying and analyzing for the first time a fundamental problem of feature acquisition bias in deep learning, and a novel and robust feature acquisition network (RFANet) is proposed to address this intrinsic weakness of current FSFGIC models.

Normalization and k-nearest neighbor are common components in metric-learning based image classification methods such as in DN4, CovaMNet, BSNet, and ATL-Net. These two components are not contributions of this work.

[a] Wu, Zuxuan, et al. "Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

[b] Boiman, Oren, Eli Shechtman, and Michal Irani. "In defense of nearest-neighbor based image classification." 2008 IEEE conference on computer vision and pattern recognition. IEEE, 2008.

## 2. The authors should provide evidence to support the claim that DA operations increase the quantization noise of images.
### **Response**: 
When performing data augmentation on digital images, the new image is slightly different from the original image as a digital image is represented by values quantized to integers at locations quantized to image grids. Such quantization noise often causes image blur and loss of information.

## 3. The authors should compare with more latest works in the experiment, such as [c][d]. [c] PaCL: Part-level Contrastive Learning for Fine-grained Few-shot Image Classification, ACM MM 2022. In fact, [d] achieves better performance than the proposed approach in the CUB dataset.
### **Response**: 
Our paper was submitted to ICLR2023 (Sept 28, 2022) before [c] was published on October 10, 2022.

FRN [d] is a state-of-the-art method, which is used as one of the benchmarks in this paper (see Table 8). In our paper, the standard “Categories Sampler” experimental setting is used in the same way as used by the other 8 benchmarks (see Table 8), such as DN4, BSNet, and AT-Net. The results reported in paper [d] is obtained in a different experimental setting not comparable to any other 8 benchmarks and our method, thus cannot be directly used for performance comparison. The FRN [d] results in Table 8 are obtained by us, using the code provided by the authors of [d] under the standard “Categories Sampler” experimental setting. Our obtained results of FRN method on the CUB dataset for the 5-way 1-shot and 5-way 5-shot FSFGIC tasks are 78.07 and 89.33 respectively, which are the same results of the FRN method as reported in the article [e]. In summary, the performance of our method is significantly better than [d] on the CUB dataset as shown in Table 8.

[c] Wang, Chuanming, Huiyuan Fu, and Huadong Ma. "PaCL: Part-level Contrastive Learning for Fine-grained Few-shot Image Classification." Proceedings of the 30th ACM International Conference on Multimedia. 2022.

[d] Wertheimer, Davis, Luming Tang, and Bharath Hariharan. "Few-shot classification with feature map reconstruction networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[e] Yangfan Li and Chunjiang Bian. Few-shot fine-grained ship classification with a foreground-aware feature map reconstruction network. IEEE Transactions on Geoscience and Remote Sensing, 60:1–12, 2022

## 4. The ablation experiments should study why the normalization and geometric average operations contribute to performance improvement.
### **Response**: 
For our proposed geometric average measure, it has already been described in detail in the ablation studies of the article (Influence of n DA operations on the query image for FSFGI) as follows:
```
Based on the backbone of Conv64F, four different n (n belongs to {2,3,4,5}) are selected for performing 5-way 1-shot and 5-way 5-shot FSFGIC tasks on the Stanford Dogs and Standford Cars datasets with a fixed parameter k (k=1). The results on the 5-way 1-shot and 5-way 5-shot tasks are shown in Table 5. It is observed from Table 5 that when n equals 3, the proposed RFANet achieves the allover best classification performance. Therefore, DA operations on the query image with n=3 is recommended for our designed architecture.
```

Normalization operation is a widely used standard component in most metric-learning based methods such as DN4, CovaMNet, BSNet, and ATL-Net. It is not a contribution of this paper.

From: ICLR.cc/2023/Conference/Paper1737/Authors
title: Rebuttal for reviewer wpgM
comment: # Rebuttal for reviewer wpgM
## 1. Some words (e.g. “bias in features”, “quantization noise”) need more explanation at the time it was first presented.
### **Response**: 
- **Bias in features** denotes the deviation of features from different instances (such as augmented versions) of each input image.
- **Quantization noise**: Image mapping processes, such as applying DA operations, introduce errors because a digital image is quantized to image grids, which is named quantization noise in this paper.

These have been added in the revised version.

## 2. The language expression is a little repeated. There are some minor issues. e.g. In Table 4 the first column are wrongly written as the same.
### **Response**: These have been corrected. Thank you very much!

From: ICLR.cc/2023/Conference/Paper1737/Authors
title: Rebuttal for reviewer jYXE
comment: # Rebuttal to Reviewer jYXE
## 1. In the paper, authors present a PFANet for FSFG task to address the feature quantization bias, and three contributions are summarized. However, it is quite confusing about the specific definition of feature information, as well as the differences from contribution (1) and (3). The phrase "feature information" and "feature bias" are too general without the careful descriptions.

### **Response**:

- **Feature information** denotes the features obtained from different instances (such as augmented versions) of each input image.

- **Feature bias** denotes the deviation of features from different instances (such as augmented versions) of each input image.

These have been added in the revised version.

## 2. For the novelty part, seems authors propose to involve data augmentation to enhance the stability of the feature extraction, which is not novel in FSFG tasks.
### **Response**: This work identified and analyzed for the first time a fundamental problem, feature acquisition bias, in deep learning, and a novel and robust feature acquisition network (RFANet) is proposed to address this intrinsic problem in deep learning.

## 3. For experiments, 5-way experiments are conducted, what about 20-way (the case when class dictionary is large)?
### **Response**: As suggested, the 20-way 1-shot and 20-way 5-shot FSFGIC experiments on the Stanford Cars dataset have now been conducted and added into the article as follows:

```
The proposed RFANet has also been compared with DN4 on the Stanford Cars dataset for 20-way 1-shot and 20-way 5-shot FSFGIC tasks. 127 classes, 20 classes, and 49 classes of Stanford Cars dataset are randomly selected for training, validation, and testing respectively. Their classification accuracies are summarized in Table 7, consistently demonstrating the superior performance of the proposed method when class number increases.
```
Table 7: Comparison results on the Stanford Cars dataset based on Conv64F (20-way Accuracy (%) is evaluated.). 
| Model | Backbone | 1-shot | 5-shot |
|:---:|:---:|:---:|:---:|
| DN4 | Conv64F | 37.67$\pm$0.59 | 69.58$\pm$0.34 |
| RFANet | Conv64F | **48.55$\pm$0.48** | **77.65$\pm$0.31** |
## 4. If the performance gains are obtained by the data augmentation, ablation analysis should be conducted regarding which type of augmentation possess the most gain.

### **Response**:  The significant gains of classification accuracy obtained in this paper come from the RFANet’s capability in addressing the feature acquisition bias problem in current deep learning networks for FSFGIC tasks. 

As suggested, the proposed RFANet architecture with different DA has been conducted as follows:
```
Additional experiments are conducted for investigating the effect of different DA operations. Three different operations (i.e., random cropping, random horizontal flipping, and color jittering) are selected individually for performing 5-way 1-shot and 5-way 5-shot FSFGIC tasks respectively on the Stanford Cars dataset. Their results are shown in Table 6. It is observed that the proposed RFANet is more effective when applied on random cropping than on flipping and jittering.
```
Table 6: Comparison results on the Stanford Cars dataset based on Conv64F (5-way Accuracy (%) is evaluated.). 
| Model | 1-shot | 5-shot |
|:---:|:---:|:---:|
| RFANet (random cropping) | **68.19$\pm$0.67** | **89.38$\pm$0.36** |
| RFANet (random horizontal flipping) |  67.63$\pm$0.67 | 86.77$\pm$0.35 |
| RFANet (color jittering) |  64.60$\pm$0.66 | 87.59$\pm$0.39 |

From: ICLR.cc/2023/Conference/Paper1737/Authors
title: Rebuttal for reviewer DzHP
comment: # Rebuttal for Reviewer DzHP
## 1. The weakness is the ad-hoc definition of the features in Sec. 3.3.1. Although the authors present an ablation study, I am not convinced that Sec. 3.3.1 is more than a heuristic selection.
### **Response:** In this work, we identify and analyze for the first time a fundamental problem of existing metric-learning based FSFGIC methods which fail to effectively address the bias in feature acquisition from each input image that causes misclassification. The feature descriptors obtained by a convolutional embedding module from different DA operations on the same image have better consistency than those obtained from different images under different DA operations. To guide a convolution-based embedding model to obtain robust image feature information, the regularized feature descriptors of the augmented support image and the regularized feature descriptors of the n augmented query images are employed for similarity calculation for FSFGIC. This is still an open question and the proposed method is a first attempt on how to design a network that can effectively address the feature acquisition bias in deep learning.

## 2. I am also kind of puzzled that 1-NN performs best. I wonder whether there is a relation to Goering et al.: Nonparametric Part Transfer for Fine-grained Recognition, CVPR 2014. However, in the end, the performance justifies the approach.
### **Response:** In this work, 1-NN achieves the best classification performance on Stanford Dogs and Stanford Cars datasets for both 5-way 1-shot and 5-way 5-shot FSFGIC tasks. We also found that 1-NN performs the best in the DN4 method [A] and NDPNet method [B]

[A]  Li, Wenbin, et al. "Revisiting local descriptor based image-to-class measure for few-shot learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[B] Zhang, Weichuan, et al. "NDPNet: A novel non-linear data projection network for few-shot fine-grained image classification." arXiv preprint arXiv:2106.06988 (2021).

From: ICLR.cc/2023/Conference/Paper1737/Reviewer_DzHP
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors report problems in data augmentation due to quantization noise. This introduces bias in the model, which is even more severe in fine-grained classification. They present a method that solves the dilemma between data augmentation and bias for a specific few-shot fine-grained classification model, namely c-way k-shot tasks. The idea is to augment each query image n-times and the support image only once. For all images, a feature embedding is generated by a CNN (for example, Resnet). Given a feature similarity based on the cosine distance, a k-NN selection is made that lead to an image-to-class similarity following a paper by Li (2019). The method is evaluated on five different fine-grained datasets. The performance is convincing compared to SotA methods. 
strength_and_weaknesses: The strength of the paper is its simplicity in terms of use and the performance demonstrated in the experimental evaluation. Furthermore, as far as I can say, RFANet can be integrated into any episodic training mechanism, which is another plus for this work.

The weakness is the ad-hoc definition of the features in Sec. 3.3.1. Although the authors present an ablation study, I am not convinced that Sec. 3.3.1 is more than a heuristic selection. I am also kind of puzzled that 1-NN performs best. I wonder whether there is a relation to Goering et al.: Nonparametric Part Transfer for Fine-grained Recognition, CVPR 2014. However, in the end, the performance justifies the approach. 


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. I like the experimental evaluation of 5 different tasks that support the authors' claim. However, only originality is moderate—especially the performance of 1-NN instead of the general case of k-NN puzzles me a bit. An explanation of this empirical result would have been an additional benefit of the paper.
summary_of_the_review: The paper convinces by the performance of an idea successfully implemented to FSFGIC. However, the concept itself is only moderately original and novel. There are no new theoretical concepts involved. The relation to the bias remains unclear to me. This term is misleading in the title. In addition, I would argue for better data augmentation to avoid quantization errors instead of such a new feature-generating network for end-to-end learning. 

After rebuttal and discussion with the other reviewers, I feel this paper has no strong theoretical contribution. On the other side, this lets me expect that the method and results can be easily reproduced, and the simplicity of a method can also be a strength. However, I would only see space for this paper if no other paper is available with a stronger methodological contribution.  
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1737/Reviewer_jYXE
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: In this paper, authors propose to address the feature acquisition bias for the few shot fine grained image classification task targeted for the metric learning based framework. Experiments are conducted on several benchmark datasets to validate the performance.
strength_and_weaknesses: Strength:
Authors analyze the fundamental problems of feature bias via the data augmentation, which motivates the proposed framework. Extensive ablation studies are conducted to analyze the important model components and key parameters.

Weakness:
The overall novelty is limited, and identified fundamental issues are somewhat unclear, as well as the reported experiments cannot fully support the claims.
clarity,_quality,_novelty_and_reproducibility: The overall writing is decent with the detailed introduction of related works, and clear descriptions of the proposed framework.

As mentioned in the weakness part, the overall novelty is somewhat limited, which is summarized in the next part.

The proposed model should be able to reproduce with the firm understanding of FSFG task.
summary_of_the_review: In the paper, authors present a PFANet for FSFG task to address the feature quantization bias, and three contributions are summarized. However, it is quite confusing about the specific definition of feature information, as well as the differences from contribution (1) and (3). The phrase 'feature information' and 'feature bias' are too general without the careful descriptions.

For the novelty part, seems authors propose to involve data augmentation to enhance the stability of the feature extraction, which is not novel in FSFG tasks.

For experiments, 5-way experiments are conducted, what about 20-way (the case when class dictionary is large)?

If the performance gains are obtained by the data augmentation, ablation analysis should be conducted regarding which type of augmentation possess the most gain.

---------------------

Post feedback comments:

I have read through the feedback and other reviewers' comments, my main concerns at the first round of review is that the somewhat limited novelty regarding applying the data augmentation for FS classification and some missing experimental analysis for proposed components. I can see that during the feedback, authors have supplied additional experiments regarding the larger shot/way size, and abolition analysis of augmentation types, and improved their writing. With the updated version, the main weakness is still the novelty of utilizing the data augmentation (at least from the current version, the novelty is not that significant). Based on the overall feedback and revised manuscript, I'm willing to raise my score accordingly.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1737/Reviewer_wpgM
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper explores the problems brought by data augmentation (DA) in few-shot fine-grained image classification (FSFGIC) and proposes a robust feature acquisition network (RFANet) as solution, which represents the similarity measurement strategy in consideration of several query images of different DA. The contributions include identifying and analyzing the problems caused by DA in FSFGIC, presenting RFANet which can guide model to learn a robust feature representation.
strength_and_weaknesses: Strength
1. The paper flows logically and clearly, which can always explain my doubts arose in the reading process. 
2. The problems brought by DA revealed in the paper may inspire future work.
3. A new architecture is proposed with clear motivations.
4. Plenty of experiments on five benchmarks have been done to demonstrate the effectiveness of the method.
5. The extensive experiments in appendix show the detail of fair comparison.

Weaknesses
1. Some words (e.g. “bias in features”, “quantization noise”) need more explanation at the time it was first presented.
2. The language expression is a little repeated.
3. There are some minor issues. e.g. In Table 4 the first column are wrongly written as the same.

clarity,_quality,_novelty_and_reproducibility: The quantity of experiments and SOTA performances, as well as inspiration show the quality of the work. The paper writing with great clarity since different parts of the paper are arranged logically. The paper firstly identify and analyse the problems caused by DA, which demonstrates its originality.
summary_of_the_review: Overall, the paper reveals the fundamental problem of feature bias caused by data augmentation, and proposes RFANet to address the issue. I recommend the paper due to its good quality representing by the its adequate experiments and good performance, and inspiration for future work by originally analyzing the feature bias problem of DA.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1737/Reviewer_XMRc
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Existing fine-grained image classification methods use data augmentation operations to avoid overfitting. The authors claim that DA operation increases the quantization noise of images and leads to bias in the obtained feature. To overcome this, the authors propose to conduct channel-wise normalization and k-nearest neighbor clustering to capture more robust image features.
strength_and_weaknesses: Strength:
1. The idea of employing normalization operations to reduce the noise brought by data augmentation is simple and straightforward. The overall framework is clearly demonstrated in Figure2.
2. The proposed method is clearly explained in the paper. The manuscript is easy to follow.

Weakness:
1. The channel-wise normalization [a] and k-nearest neighbor clustering [b] are proposed many years ago. Both of them are widely used in the computer vision field. Therefore, the novelty of the paper is limited.
2. The authors should provide evidence to support the claim that DA operations increase the quantization noise of images.
3. The authors should compare with more latest works in the experiment, such as [c][d]. In fact, [d] achieves better performance than the proposed approach in the CUB dataset.
4. The ablation experiments should study why the normalization and geometric average operations contribute to performance improvement.

[a] Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation, CVPR 2018.
[b] In defense of nearest-neighbor based image classification, CVPR 2008.
[c] PaCL: Part-level Contrastive Learning for Fine-grained Few-shot Image Classification, ACM MM 2022.
[d] Few-Shot Classification with Feature Map Reconstruction Networks, CVPR 2021.
clarity,_quality,_novelty_and_reproducibility: The proposed approach is well is clearly explained in the paper. The writing is fluent and easy to understand. However, the idea of the paper lacks originality.
summary_of_the_review: The idea of the paper lacks novelty. The contribution is incremental. Therefore, I believe that this paper should be rejected.

======================================================================================================

Thank the authors for addressing my concerns. After reading the authors’ responses and peer reviews, I decided to keep my score of 3. My main concern is the novelty. The authors claim that they propose a novel feature acquisition network (RFANet) to address the feature acquisition bias of current FSFGIC models. However, the authors also say that the normalization and k-nearest neighbor are not their contributions, which is the main approach written in section 3. Therefore, I believe that the novelty of this paper is limited.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough