Paper: Towards Understanding the Effect of Pretraining Label Granularity (ICLR-2024)

Reviewer comment (Theory):
In Definition 4.2 regarding 'hard samples', this paper characterizes them based on the introduction of random noise. However, merely adding random noise doesn't necessarily make a sample challenging to classify. Learning with noise is different from learning with hard samples. Prior research typically defines hard samples as those with significant classification loss, e.g., boot-strapping or hard negative mining.

Author response:
To clarify, our definition of hard samples relies on two factors. Firstly, we assume that common features are not present in these samples. As we know from the theory, common features are very easy for the neural networks to learn, and the neural networks  indeed exhibit a tendency towards learning these common features over learning the rarer fine-grained features. Therefore, by removing common features from an input sample, the sample becomes challenging to the neural network. Second, the addition of extra noise, on top of removing the common features, is to make the example even more "confusing" to the neural network, thus making our theoretical result more pronounced. The interpretation of the noisy patches, as detailed in Section 3.2, is to simulate "distracting irrelevant patterns" in real-world inputs.

Reviewer comment (Writing):
It is suggested to use \citep rather than \cite in the latex.

Author response:
Thank you for pointing this out! We have updated \cite to \citep in our paper now.

Reviewer comment (Experiments):
Testing on more datasets. The current results are verified on a single cross-dataset pair which not hold for other dataset pairs. There are some datasets are studied in low-shot learning could be used in this sceneries.

Author response:
**W1a.** "Testing on more datasets", "... hard, costly, and usually "impossible" to obtain the used hierarchy for large-scale dataset"

**A**: As is discussed in the related work section (Section 2.2), there are already a number of empirical works that pretrain large models with highly fine-grained labels (albeit with lower quality than human labels) for improving model generalization. For example, [1,2] use noisy hashtags from Instagram as pretraining labels, [3,4]  apply clustering on the data first and then treat the cluster IDs as pretraining labels, and [5] uses the queries from image search results, etc.

Our work, however, deviates from empirically demonstrating this already-known observation again, but rather is focused on mathematically explaining this phenomenon. 
Our experimental work is comparable to, if not more extensive than, that of the majority of papers in the area of deep learning theory. For reference, please see the following representative papers [6 - 8].


**W1b.** "There are some datasets are studied in low-shot learning could be used in this sceneries"

**A**: Thank you for this suggestion. We will take these datasets into consideration in our future work.

Reviewer comment (Experiments):
Studying how to obtain the hierarchy/fine-grained labels for unlabeled datasets. It's hard, costly, and usually "impossible" to obtain the used hierarchy for large-scale dataset; therefore, it's important to have discussion and analysis here. The paper currently provided a simple study on Section 5.2. However, I expect more analysis such as how should we decide class-level for unlabeled dataset. A probably related paper here is Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy.

Author response:
**W2.** "Studying how to obtain the hierarchy/fine-grained labels for unlabeled datasets"

**A**: Thank you for this suggestion. We are indeed very interested in exploring methods of generating hierarchy/fine-grained labels for datasets that do not have human labels. For example, one approach is to utilize large language models such as [9] to decompose the coarse-grained labels into fine-grained categories. Subsequently, visual question answering (VQA) models such as [10, 11] can be employed to automatically classify the input samples in a fine-grained manner. We are also interested in further understanding the limitations of training DNNs on large-scale fine-grained but noisy data. Lastly, we hope our results may give motivation for others in the community to develop innovative techniques for obtaining fine-grained labels for unlabeled datasets.

Reviewer comment (Writing):
You should use \citep not \cite in most place of citations.

Author response:
"You should use \citep not \cite in most place of citations."

**A**: Thank you for pointing this out! We have updated \cite to \citep in our paper now.

Reviewer comment (Theory):
I am not totally sure that the idealized setup considered here makes much sense in practice. For example, Jain et al. (2023) claim that fine-grained labels are often hard and expensive to obtain and going in the coarse --> fine-grained direction is equally valuable. Moreover, when pretraining on large-scale datasets, (e.g., Mahajan et al. 2018), I believe it is often not clear what the label hierarchy is (or if it even exists).

Author response:
The fine-to-coarse transfer learning setting is already widely used in practice (and getting increasingly popular), especially for training large models. As we discussed in the related work section (Section 2.2), there are more and more works that pretrain large models with highly fine-grained labels and the downstream classification task is coarser in label granularity. For example, [1,2] use noisy hashtags from Instagram as pretraining labels, [3,4]  apply clustering on the data first and then treat the cluster IDs as pretraining labels, [5] uses the queries from image search results, etc. 

Despite this growing trend, to the best of our knowledge, there is no theoretical work that justifies the benefits of (pre-)training deep neural networks (DNNs) with such fine-grained labels. Our work aims to fill this gap by providing a theoretical understanding of this phenomenon that already exists in practice.

Additionally, we are indeed very interested in studying how to generate fine-grained labels for datasets that do not have human labels. For example, we could first use large language models such as [6] to decompose each coarse-grained label into fine-grained labels, and then use visual question answering (VQA) models such as [7,8] to automatically classify the input samples in a fine-grained manner. We are also interested in further understanding the limitations of training DNNs on large-scale fine-grained but noisy data. Lastly, we hope our results may give motivation for others in the community to develop innovative techniques for obtaining fine-grained labels for unlabeled datasets.

Reviewer comment (Experiments):
The other concern that I have is related to the transition from theoretical contributions to empirical experiments. I am not sure if the experiments on ImageNet and iNaturalist are sufficient to support the presented theory (could you please elaborate a bit on that if that is the case).

Author response:
The purpose of the theoretical study is to *explain* the empirical observations we made, in particular, the observations we summarized in the introduction of this paper: "Under certain basic conditions on the pretraining and target label functions, DNNs pretrained at reasonably high label granularities tend to generalize better in downstream classification tasks than those pretrained at low label granularities." Therefore, our large-scale experiments are not intended to validate the theory, but it is the other way around. In particular, we chose ImageNet and iNaturalist because they are widely used benchmarks in the literature. Our experiments confirm that the observed phenomenon extends to these popular benchmarks. Our theoretical results provide a mathematically rigorous explanation for the empirical observations.

Reviewer comment (Experiments):
One suggestion (that should be doable and easy to implement) would be to generate synthetic data and confirm that Theorems 4.1 and 4.2 hold on it, exploring and explaining the impact of the different parameters needed by your theory.

Author response:
As stated above, the primary purpose of this work is to use our study on the 2-layer neural network to provide rigorous support for our claim that pretraining at reasonably high label granularity is beneficial for generalization. The theoretical results we presented are asymptotic in nature and they serve to qualitatively confirm the empirical observation we made. It is important to note that our approach aligns with many current deep-learning-theoretic works. For instance, [9-11] referenced below are example theory papers that are similar in structure to ours, but contain significantly fewer experimental investigations.

Reviewer comment (Experiments):
Q1: Just to confirm: When fine-tuning, you do fine-tune the whole network, i.e., you do not keep the feature extractor fixed. It is a bit surprising to me that regardless of the pretraining granularity (e.g. on Fig. 1 and Table 1), the fine-tuned model does not catch up with the baseline training, assuming that sufficient time for finetuning is given.

Author response:
We do not keep the feature extractor fixed during finetuning.

Given that our work represents an initial step towards understanding the effect of pretraining label granularity for DNNs, we want to stick to the common practices as much as possible. Early stopping the pretraining and finetuning phase is a common practice in the field.

More specifically, in the iNaturalist experiments, we allowed the finetuning phase to run for 90 epochs, matching the same length of the pretraining phase (90 epochs is the standard training length for ImageNet-like datasets), therefore the finetuning time is substantial. In the ImageNet21k->ImageNet1k experiments, we adopted the procedure established in the original ViT paper [12].

Furthermore, it is natural for a finetuned model to behave differently from a baseline model. We know that in deep learning, due to the extreme non-convexity of the optimization landscape, initialization of weights in a deep neural network can significantly impact the training and generalization performance of the model. In our situation, pretraining at different label granularities lead to different stochastic gradient descent (SGD) trajectories of the neuron weights, so these different SGD trajectories will very likely head towards different local minima. Finetuning does not necessarily push the neuron weights to the local minima that the baseline models end up in.

Reviewer comment (Experiments):
Q2: Is the granularity solely determined by the number of classes or number of classes AND class level in hierarchy? Do we assume that all classes in a given (pre)training dataset are at the same hierarchy level? What if we mix classes from different levels in the class hierarchy during the pretraining?

Author response:
Our definition of "label granularity" is the number of classes. In our experiments, all classes in a given (pre)training dataset are indeed at the same hierarchy level.

Thank you for the suggestion, we will take this factor into consideration in our future work. For this work, we are taking the first step in understanding the effect of pretraining label granularity, so we chose to stick with the most basic setting of all classes being in the same hierarchy level.

Reviewer comment (Theory):
Q4: If I understand correctly, the theorems require that the neural networks are trained only on "easy" samples. Why is that the case? If it is indeed needed, how can you distinguish between easy and hard samples during training?

Author response:
As discussed in our paper, "hard" samples are generally rare in natural image datasets. Our theoretical result is intended to present the "feature-learning bias" of a neural network in an exaggerated fashion. Therefore, examining the case of "no hard training examples at all" serves as a natural starting point for theoretical analysis. In our theory, the main distinction between easy and hard samples is whether or not they contain patches that are dominated by common features: easy samples contain these patches, while hard samples do not. These theoretical considerations do not imply that we advocate for training real DNNs only on easy samples in practice.

Reviewer comment (Evaluation):
Q6: On iNaturalist (Fig. 4) why do you only report validation errors but omit the final accuracies? Are the accuracies consistent with the shown figure?

Author response:
Error rate and accuracy are complementary metrics that add up to 100\%, so reporting error rate is equivalent to reporting accuracy. When referring to "final accuracies", we presume you mean the accuracies on the test set. We should note that the dataset we worked with does not contain a test set (with ground truth labels). Thus we can only rely on the validation set to assess the model's generalization performance. Also note that this is in fact a common practice in the community. For example, people only report the validation error/accuracy on the popular ImageNet1k dataset to reflect the generalization performance of their models.

Reviewer comment (Theory):
Minor: In your examples (Sec 4) you consider mainly binary task (i.e., 2 coarse level classes). Can the theory and the theorems be extended to the multi-class setup?

Author response:
Yes, our theory can be extended to the multi-class setup easily. We chose to present the binary-class case to maintain clarity and avoid further complicating the notations.