From: ICLR.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Reject
comment: Summary: The given work studies one-shot object detection, and demonstrates an array of experiments that show that by increasing the number of categories in training data, the model can get better at one-shot detection. 

Pros:
- Well written
- Presents many experiments that are interesting
- Improves SOTA one-shot performance on COCO by using more data.

Cons:
- Authors did not test their claims on a variety of model architectures
- Similar conclusions have been made in prior art.
- Conclusions are intuitive. As more categories are added, the likelihood of reducing semantic dissimilarity to a new novel category is quite high.
- Overall contribution is currently limited.

Reviewers are unanimous in their decision. Authors did not alleviate concerns of reviewer. AC recommends authors take all feedback into consideration and submit to another venue or workshop.

From: ICLR.cc/2022/Conference/Paper3702/Authors
title: Rebuttal
comment: The reviewers find our paper “well motivated” (djNN) and our experiment on Pascal VOC “smart” (fCN3), but criticize that our findings are (A) not novel, (B) intuitive and (C) that our paper lacks a modelling contribution. We would like to address these points of critique:


**(A) The findings are not novel:** As the reviewers correctly note, previous papers [Sbai et. al. 2020, Jiang et. al. 2020, Fan et. al. 2020] have also observed that more categories can help in few-shot tasks. We here add three important insights:
1. This improvement is driven by the number of categories, not the number of samples (Figure 5B).
2. It comes from a closing of the generalization gap: The models perform similar overall but generalize better to novel categories (Figures 1 & 5A).
3. The closing of the generalization gap and the additional value large models provide on broad datasets (Table 3) signal a difference in mechanism; from memorizing the training categories to learning a function that is applicable to all kinds of objects.


**(B) The findings are intuitive:** We absolutely agree with the reviewers that our findings are intuitive. But we don’t agree that this reduces their value.
1. Intuition is great to form hypotheses, but scientific conclusions require evidence. We here present this evidence through extensive experiments with multiple models and datasets.
2. Are intuitive results less relevant? If the trend we find continues and the generalization gap further shrinks with more categories, few-shot detection may become as easy for machines as it is for us. Judging by the few-shot capabilities of BigTransfer [Koleshnikov et. al. 2019], CLIP [Radford et. al. 2021] and GPT-3 [Brown et. al. 2020], all of which are trained on huge, broad datasets, this is quite plausible. At the end of the day the goal is to develop models that can detect all sorts of objects. With these models and datasets readily available, should we not use them and build upon them? At the end of the day the goal is to develop models that can detect all sorts of objects. Whether we achieve this by using great methods or simply broad datasets is not really relevant.


**(C) The paper lacks a modelling contribution:** The hypothesis of the paper is that training on wider datasets improves one-shot generalization. The central findings are that the number of categories determines relative performance (45% on COCO -> 89% on LVIS) and that models trained on broader datasets generalize better (SotA and improved relative performance on COCO). Of course new models are still required to improve performance in absolute terms, but the point of the paper is to demonstrate that we need a paradigm shift in how we train and evaluate one-shot detection models.

From: ICLR.cc/2022/Conference/Paper3702/Reviewer_fCN3
summary_of_the_paper: This paper studies one-shot 2D object detections. The major conclusion is that when keeping the number of training images fixed, increasing the number of training categories can significantly increase the one-shot performance.  This paper demonstrated this conclusion empirically by doing controlled experiments on COCO, Objects365, and LIVS. 

Inspired by this observation, this paper improves the state-of-the-art one-shot detection performance on COCO from 22.0 to 27.5 AP50 by training on LIVS. 
Other two related conclusions in this paper:
- PASCAL VOC is not suitable for evaluating one-shot object detection algorithms. The reason may be that the number of instances per image is low(2.9 ins/img on average). The algorithm only needs to recognize foreground objects rather than objects of target categories. 
- Only when the training data is challenging enough, increasing the model size and training time can help improve one-shot performance.  

main_review: Strengths:

1. The paper designs a smart experiment to demonstrate that PASCAL VOC is too easy/biased for evaluating one-shot object detection algorithms. The experiment shows that assigning black reference images does not degrade the performance on PASCAL VOC too much. 
2. The primary conclusion that increasing the number of training categories significantly influences one-shot performance is clearly stated. Experiments for the major conclusion are valid. (Section 4.3, 4.4, Figure 4,5). 
3. This paper improves stoa one-shot performance on COCO, though by training on LIVS. 

------------------------
Weakness:

1. By training on LIVS, this paper obtains state-of-the-art performance. But training on LIVS may not be fair compared with others. 

2. The major conclusion is not very non-trivial because previous work has shown that increasing the number of categories for pretraining can obtain better representations, like [1](figure 1) (It is not the exact same conclusion and scenarios, but it is similar.).  Meanwhile, similar conclusions like increasing the number of categories for recognitions increase the learning difficulty dramatically. (figure 2 in [1] ) 

3. There is some sort of overclaiming in some of this paper's sentences regarding the performance. Like in the introduction: 'This generalization gap can be almost closed by increasing the number of categories used for training.'  But the performance of one-shot detectors mentioned in table 4 is still below 30 AP50. 

One thing I am curious about is that how training on Objects365 can affect the one-shot performance on COCO compared with current state-of-the-art results. 

[1] Mahajan D, Girshick R, Ramanathan V, et al. Exploring the limits of weakly supervised pretraining[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 181-196.
summary_of_the_review: I will not recommend accepting the paper. Mainly because of the limited contribution. I will vote for strong reject if I find similar references demonstrating similar conclusions in one-shot detection scenarios.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

From: ICLR.cc/2022/Conference/Paper3702/Reviewer_djNN
summary_of_the_paper: - The paper considers the problem of one-shot object detection, meaning, the model is asked to detect unseen categories, based on only one provided a template.

- The main discovery made by the authors is that, to generalise better, the model should be trained with data from as many categories as possible, given the same budget on number of samples for training.

- Architecture-wise, a Siamese Faster R-CNN is adapted. 
main_review: - Strength
    - The paper is generally well-written and well-motivated.
    - I appreciate the authors for validating the fact that PASCAL is not suitable for one-shot detection, given the task simply becomes salient object detection.

- Weakness:
    - The contribution of the paper is limited.
    - The paper simply raises the discovery that training more categories can be beneficial for generalisation on one-shot detection, which is probably not that surprising, given the architecture used is based on matching, and more importantly, while increasing the number of training categories, it becomes more likely that the training and held-out categories are subcategories of one root.

    - As for architecture, the authors use Siamese Faster R-CNN, similar to tracking tasks, and class-agnostic countings.

summary_of_the_review: - Overall, I think the discovery is well-made, it shines light on one perspective, that is, using more diverse training data can be beneficial for generalisation,  however, it misses the other element on, how can we further boost the generalisation from an architecture or model design perspective.

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

From: ICLR.cc/2022/Conference/Paper3702/Reviewer_b9Xw
summary_of_the_paper: This paper studies the effect of category number in the one-shot object detection task. In the testing of one-shot detection, there exists a performance gap between the base (training) classes and held-out classes. It is claimed that this performance gap can be largely closed by increasing the number of categories used for training. And the number of categories is more crucial than the number of samples per category. Experiments are conducted on VOC, COCO, Object365, and LVIS using a Siamese-style detector to verify the claims.
main_review: # Strengths:
- State-of-the-art performance is achieved on COCO one-shot detection benchmark by training the detector on LVIS.
- This paper is well written. The presentation is easy to follow.

# Weaknesses:
- The study on the effect of category number in machine learning tasks has been conducted in previous works, e.g., Sun et al. (2017), which weakens the novelty of this paper.
- A major concern is on the split of train categories and held-out categories. As the category number increases, the train categories have higher probability to contain more categories semantically related to the held-out categories. With more semantically related categories to train on, it is reasonable that the detector can have better performance on held-out. For example, adding more cat species in the train categories may better help improve the detection accuracy of a held-out “ragdoll” category than adding more vehicle categories. Therefore, the large number of categories may not be the root cause of reduced generalization gap, but instead semantic relation may be the key. Currently, it seems there is one single fixed train/held-out split in all the experiments. Have the authors tried with other split? For example, does adding more unrelated categories still improve the performance?

summary_of_the_review: On the one hand, this paper is well written and produces sota performance. However, I am concerned about the novelty and the root cause for the improvements. I am leaning towards reject for now and would like to see the response from authors.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

From: ICLR.cc/2022/Conference/Paper3702/Reviewer_wPts
summary_of_the_paper: This paper shows that the key to reduce the generalization gap between base classes and novel classes is to increase the number of training categories, instead of training samples. The authors did many experiments on four existing datasets (PASCAL, COCO, Objects365, and LVIS) with Siamese Faster R-CNN to verify this point. Experiments show that with more categories in the training set, the generalization gap will be nearly closed. Finally, the author proposes that future data sets should focus on the diversity of categories.
main_review: Strengths:
1.	The paper conducts many experiments to illustrate the impact of different factors on the performance of base classes and novel classes, including the number of training categories, the number of samples, the model capacity and the training time, etc. 
2.	The paper emphasizes that more categories are more beneficial to the task of FSOD, including shrinking the generalization gap, and improving the performance on novel classes. The authors suggest that the future datasets should focus on the diversity of categories.
3.	The paper is well-written.

Weaknesses:
1.	The paper's major point of view has been observed in previous works, and it is very intuitive. For example, a similar point of view has already been pointed out in [1]. See section 5.4 “More Categories vs. More Samples?” and Table 7 in [1]. Moreover, a high-diversity dataset has been proposed and corresponding experiments have been carried to verify the importance of the number of categories. However, except for the related work section that mentions [1], the other parts of this paper do not mention [1], and do not compare with [1]. 

2.	The main point of the paper is not convincing: 
a) The authors only conducted experiments on metric-based methods (called example-based visual search task in this paper) and ignored finetuning-based models. However, more and more finetuning-based models has been proposed, such as TFA[2]. The authors should validate their views in both types of methods, instead of just one of them.
b) If there exists domain shift from training categories to testing categories, more training categories will not bring benefits. Many previous works performed COCO->VOC cross-domain experiments [3][4], where images from COCO are used for training and images from VOC are used for testing. Results show that the performance on VOC novel classes is worse than that when using base classes in PASCAL dataset, even COCO has more categories.

3.	The contribution of the paper is limited. The paper does not propose a new dataset, nor does it propose a novel method, but uses existing datasets to draw a conclusion, and this conclusion was also drawn in the experiments of previous works. And this conclusion is only verified on the metric-based methods.


[1] Fan, Qi, et al. "Few-shot object detection with attention-RPN and multi-relation detector." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[2] Wang, Xin, et al. "Frustratingly Simple Few-Shot Object Detection." International Conference on Machine Learning. 2020.
[3] Wang, Yu-Xiong, Deva Ramanan, and Martial Hebert. "Meta-learning to detect rare objects." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
[4] Kang, Bingyi, et al. "Few-shot object detection via feature reweighting." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

Some details:
1.	In related work, the paper classifies [1] as one-shot object detection, which conflicts with the title of [1] and the previous classification method.
2.	In the introduction, the authors claim that one-shot object detection problem can be decomposed into three subproblems. However, there are many methods that are not based on RPN[3][4], where class-agnostic object proposal is not needed.. Therefore, making such claim is inappropriate.

summary_of_the_review: Considering that the contribution of this article does not meet the requirements of ICLR, I do not recommend to accept it.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.