Paper: Hard View Selection for Contrastive Learning (ICLR-2024)

Reviewer comment (Theory):
Lack of Theoretical Analysis: The paper seems to focus on empirical validation but doesn't provide a theoretical foundation for why the "hard view selection" approach should work, which could limit its scientific rigor.

Author response:
Thank you for your feedback. We also thought about ways to incorporate a theoretical foundation and validation to our manuscript. Given the nature of our study however, we found it challenging to find an easy enough framework where math/theory could be applied, especially given the dynamics and assumptions of the various baselines HVS is applied to in our manuscript. Note also that, to our best knowledge, little to no theory exists for the considered baselines in this paper. We appreciate your point on enhancing scientific rigor and would be grateful for any suggestions or insights you might have on how to integrate theoretical analysis more effectively into our work.

Reviewer comment (Theory):
Unclear Impact on Convergence: Introducing harder samples could potentially slow down the convergence of the training process.

Author response:
Across all our experiments, we fortunately observed no slow-down in convergence during training. More broadly speaking, the training loss pattern remained identical to the baseline training. The only difference to the baseline trainings we were able to observe is that the training loss is slightly higher (see Fig. 7 in the supplementary material for an example of the training loss with/without HVS). This might encourage better generalization.

Reviewer comment (Evaluation):
A detailed analysis of the HVS overhead in terms of running time/ wall clock/ FLOPs/ throughput along with memory requirement for each the baselines DINO (2x overhead), SimSiam(1.55x overhead), SimCLR. Preferably in a format of a table.

Author response:
We have addressed the concern on computation overhead in the common comment to all reviewers. Please let us know if your concerns persist so that we can further discuss them with you.

Reviewer comment (Evaluation):
Extending on the above point, it seems since the overhead is added due to HSV, a more fair choice of x-axis in table 1 (and other tables) should be training time instead of epochs, as we see in table 1, that algorithm's performance increases with longer epoch training. For example assuming DINO + HSV is twice slower than DINO, it would be great to see a performance comparison between DINO+HSV at epoch 100 with DINO at epoch 200 (similarity with other baselines).

Author response:
We have addressed the concern on computation overhead in the common comment to all reviewers. Please let us know if your concerns persist so that we can further discuss them with you.

Reviewer comment (Experiments):
As HSV is proposed as a general view sampling strategy it might be helpful to see some results on other relevant baselines like SimCLR_v2, DINO_v2, Swav. If some of the recent methods require huge amounts of compute or dataset, showing some results on ImageNet with lower training epochs would give the reader a better idea about the efficacy of the hard view selection strategy.

Author response:
We thought carefully and long about the selection of baselines for our paper. Indeed, there exist other baselines as well. For DINOv2, we kindly note that it was not pretrained on ImageNet-1k but on the unreleased LVD-142M dataset and so there would be no way to add HVS and compare against it fairly. For SimCLRv2, we note that only a simplified version has been open-sourced by the authors [1]. For example, the public code of v2 does incorporate contrastive learning which is used in v1. In order for us to show that our method works with the original (contrastive) v1 loss, we decided against using v2. In the case of Swav, we notice that it comes from the same authors as DINO which is the more recent and better-performing method that, similar to Swav, also incorporates the multi-crop technique. 

Regarding the reviewer’s comment on including a more recent baseline, we hope we were able to address it by adding the popular iBOT baseline (see general comment to all reviewers above). We believe this result again underpins the general applicability of HVS since it also works well with a MIM objective.

[1] https://github.com/google-research/simclr/issues/115

Reviewer comment (Novelty):
It could be a good idea to relate and mention in the sub-section of “Taking a Closer Look at the Intersection over Union” with the idea of “local-to-global” correspondences (as introduced in SimCLR: Global and local views/Adjacent views and DINO: global and student view). Qualitatively, looking at Fig2, it does look like a HSV selected view pair comprises of global view (larger area crop) and student view (small area crop).

Author response:
We thank the reviewer once again for this valuable feedback point. Indeed, this is a rich addition to the existing description and builds a useful analogy to SimCLR as well as DINO through the local-to-global correspondences and multi-crop settings. We have added this to the end of the mentioned subsection in the paper.

Reviewer comment (Reproducibility):
How many seeds are used for Table 1 average computation. “Ref: averaged over multiple seeds unless otherwise mentioned”

Author response:
Thank you for pointing this out. Our results are averages of 3 seeds (except for iBOT where we currently average over 2 seeds). We have updated the mentioned sentence in the paper to be more specific.

Reviewer comment (Evaluation):
It is stated in the paper that SimSiam+HSV has 1.55x overhead and DINO+HSV has 2x overhead. Since DINO uses much more views (10 views leading to 128 max view pairs for forward call) than SimSiam (2 views that leads to 4 view pairs for forward call). What is the time taken for doing 1 DINO forward call and 1 backward call as the reviewer was expecting the DINO+HSV to be slower than 2x.

Author response:
Did we answer your question with the common comment to all reviewers and with the additional method-specific speed information provided there? Please do let us know if your concerns persist so that we can provide the missing information.

Reviewer comment (Experiments):
The reviewer was wondering about the combination of HSV with random view pair selection in order to reduce the computational overhead. Maybe having 50% random and 50% HSV samples for training would be nice ablation experiment to shed light on the requirement of HSV adversarial pair selection. If this works without decreasing the performance with a large margin, then it would be interesting to see some ablation on tuning this percentage of HSV samples in training.

Author response:
We appreciate the suggestion of this interesting idea. In fact, we have tested this for the min IoU policy (see Table 4 in the paper; denoted as “alternating”) where it helped reduce the inferior performance difference to the baseline. Indeed, applying an alternating strategy to HVS would help with the speed overhead and likely reduce the computation overhead by approx. 50%. However, we hypothesize it would also reduce the final performance noticeably. In any case, this is an interesting idea to test and which we have also added to the list of actionable steps (see common comment ) to take for reducing the computational overhead 

Overall, we thank the reviewer again for the valuable feedback. Hopeful that we have addressed and mitigated your initial concerns, we would like to ask you if you could increase your score (or, alternatively, we would be glad for any points we should address that would raise your score).

Reviewer comment (Novelty):
As shown in Figure-3, positive pairs used for computing loss overlap less thanks to the proposed method, and this facilitates better representations. Tian 2020b already shows this phenomenon, i.e. there is a sweat spot in the IoU ratio which leads to the "optimal" performance. It is not clear what more this paper offers. One distinct angle in this paper is the fact that multiple positive pairs are utilized during training (although loss is computed over only 1 pair in the end). I wonder what would happen if loss was computed over all crops, by simple averaging or weighted averaging (depending on the difficulty of a pair).

Author response:
**As shown in Figure-3, positive pairs used for computing loss overlap less thanks to the proposed method, and this facilitates better representations. Tian 2020b already shows this phenomenon, i.e. there is a sweat spot in the IoU ratio which leads to the "optimal" performance. It is not clear what more this paper offers.**

We thank the reviewer for pointing out the relevance of Tian2020b. Indeed, we agree with the reviewer that in hindsight we should have contrasted our paper in greater detail with Tian2020b and regret to not have done that in the initial version of the paper. Since also Reviewer #a1az has brought up this point, we are copy-pasting our response here, which we believe discusses the similarities and differences to Tian2020b comprehensively:

First, Tian2020b leverages the information of views to characterize good views for a given task based on the mutual information principle and use this framework to discover sweet spots for data augmentation hyperparameters. These data augmentation hyperparameters are widely adopted by SSL approaches and our baselines. Additionally, Tian2020b use their insights about good views to then adversarially learn flow-based models that generate novel color spaces for the small STL-10 dataset. After the views have been learned offline, they then perform standard contrastive learning on these generated views. 

This approach is different to HVS on multiple fronts: with HVS we propose a method that can be integrated *online* into CL and non-CL pipelines without the use of a potentially instable adversarial learning objective. Our method is dependent on the model state and on individual samples since it is integrated online into the learning procedure, which is different to Tian2020b since their approach is not entangled with the model state. 

While Tian2020b ablates also based on distance metrics in their study, they do not include the IoU metric. Moreover, another central difference is that we use the distance metric as a tool to better understand the selection of HVS and to discover that model-state and image-dependence is essential, where Tian2020b uses it as a motivation for their framework. Furthermore, our ablation studies include experiments that quantify the difficulty of predicting good/hard pairs, which Tian2020b does not have. In summary, we view Tian2020b as complementary to our work, particularly in the context of their derived data augmentation policies and believe that HVS could even be combined with the findings of Tian2020b. However, it's important to underscore the limited overlap in methodology and experimental design between the two approaches. HVS in the end could be seen as an early work that shows that for good views, we need to derive better sampling methods that can dynamically adapt to the actual state of training.

We have now added the expanded discussion on Tian2020b from above to the Introduction and Related Work sections of our paper. 

**[...] One distinct angle in this paper is the fact that multiple positive pairs are utilized during training (although loss is computed over only 1 pair in the end). I wonder what would happen if loss was computed over all crops, by simple averaging or weighted averaging (depending on the difficulty of a pair).**

Thank you for this idea on averaging the losses. Please allow us to make sure that the method was correctly understood by the reviewer. We emphasize that the loss is computed for all pairs individually but only backpropagated for one pair (namely the one with the highest loss). Let us make sure we understood your suggestion correctly: instead of forwarding all view pairs and discarding all except the pair with the highest loss, you are suggesting to keep all pair losses and compute an average across all pair losses or smart averaging (by weighing the pairs with some difficulty score)? Assuming we understood your suggestion, we believe it would significantly ramp up the computation cost of the method since it would effectively multiply the batch size (since all activations for all pairs would have to be kept in memory for the backward pass). While applying HVS with a weighted loss that amplifies the gradient updates proportional to some difficulty measure sounds very exciting, but since we have no experience in this regard we lack the intuition about the consequences of such a modification with regard to training stability. Please let us know if we misunderstood your suggestion so that we can engage in a follow-up discussion.

Reviewer comment (Experiments):
The computational overhead introduced by the proposed method ("about a factor of 1.55× for SimSiam") is a bit unfair for the baselines. I wonder if this extra compute time can be used in favor of other models too, e.g. by training models longer. Cause longer training schedules often bring substantial gains for self-supervised methods. Also, it should be noted that the proposed method has seen more samples (due to encoding multiple pairs) which already impacted for instance batch-norm statistics (although loss is not backpropagated over unused pairs). Then it would be nice to see baselines processed 4x more samples.

Author response:
**The computational overhead introduced by the proposed method ("about a factor of 1.55× for SimSiam") is a bit unfair for the baselines. I wonder if this extra compute time can be used in favor of other models too, e.g. by training models longer. Cause longer training schedules often bring substantial gains for self-supervised methods. Also, it should be noted that the proposed method has seen more samples (due to encoding multiple pairs) which already impacted for instance batch-norm statistics (although loss is not backpropagated over unused pairs). Then it would be nice to see baselines processed 4x more samples.**

Thank you for communicating this critique with us. We have addressed the concern on computation overhead in the common comment to all reviewers. Please let us know if your concerns persist so that we can further discuss them with you.

Regarding the concern about the models having seen more samples, we kindly disagree with the reviewer. First, the DINO and iBOT ViT-S models do not use batch norm but layer norm layers which do not keep track of running statistics and those experiments still show the same improvements. Moreover, even when batch norm layers are used, the batch norm learnable parameters are not affected by the HVS forward passes since we disable gradients for the selection step and hence, the BN parameters are not updated during the selection. Note also, that the running mean and variance buffers are only used for inference, see the pytorch documentation [1] which states:

“*Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation.*”

During training, we keep our model in model.train() state so the running mean and variance are not used. Consequently, the HVS models do not have an advantage over the baselines in terms of having seen more samples.

[1] https://pytorch.org/docs/master/generated/torch.nn.BatchNorm2d.html?highlight=batchnorm2d#torch.nn.BatchNorm2d


We express our gratitude once more for the constructive feedback provided. Aiming to have effectively addressed your concerns, we kindly inquire whether there is an opportunity to increase your score. Alternatively, we welcome any additional points for improvement that would allow for a score increase.

Reviewer comment (Theory):
In addition to the point above, it actually seems that the proposed method is better defined for non-contrastive losses. In fact, for contrastive losses (which contrast the attraction of the positives with the repulsion of the negatives) the method is not guaranteed to find the hardest possible set of positives and negatives. This would require exploring all the possible combinations of negatives, which is intractable. A better idea would be to use all the negatives from all the views at the denominator. In that case, since the denominator works as a max at low temperatures, you would be “guaranteed” to have the hardest negative. This is not described and discussed in the paper except one sentence at the end of 3.3.

Author response:
**In addition to the point above, it actually seems that the proposed method is better defined for non-contrastive losses. In fact, for contrastive losses (which contrast the attraction of the positives with the repulsion of the negatives) the method is not guaranteed to find the hardest possible set of positives and negatives. This would require exploring all the possible combinations of negatives, which is intractable. A better idea would be to use all the negatives from all the views at the denominator. In that case, since the denominator works as a max at low temperatures, you would be “guaranteed” to have the hardest negative. This is not described and discussed in the paper except one sentence at the end of 3.3.**

We agree with the reviewer that this point is described vaguely and requires more clarity in the paper. We describe this in greater detail in the Appendix under “I.1 SimCLR”. If we understood the reviewer right, yes, that is correct and we updated I.1 with the following description for more clarity:

“*For each iteration, we evaluate all possible view pairs and contrast each view against every other example in the mini-batch. Intuitively, the pair that yields the highest loss is selected, which is the pair that at the same time minimizes the numerator and maximizes the denominator in the above equation. In other words, the hardest pair is the one, that has the lowest similarity with another augmented view of itself and the lowest dissimilarity with all other examples.*”

Did this address your concern and did we understand you correctly? We hope we were able to give some clarification and are happy to discuss further suggestions.

Reviewer comment (Reproducibility):
In algorithm 1, the authors report that they first organize the views into pairs and then forward each pair. This entails forwarding every view multiple times (one view can be in more than one pair). This is confusing and does not make sense at all. It would be better to first forward the views separately and then compare them 2 by 2. Can the authors please explain why they first create the pairs and then forward?

Author response:
**In algorithm 1, the authors report that they first organize the views into pairs and then forward each pair. This entails forwarding every view multiple times (one view can be in more than one pair). This is confusing and does not make sense at all. It would be better to first forward the views separately and then compare them 2 by 2. Can the authors please explain why they first create the pairs and then forward?**

Thanks, you are right. In practice, we indeed only forward each view once to generate its embedding before we create the pairs over the embeddings. We fully agree with the reviewer that Algorithm 1 does not reflect this properly. For didactic reasons, we thought that it would be more understandable to a reader if the view pairs were first generated. However, after considering the reviewer’s feedback, we agree it would be better to stay closer to the actual (more efficient) implementation and updated Algorithm 1 accordingly. Thank you for pointing this out.

Reviewer comment (Experiments):
Why not just make all comparisons and backprop all of them? Why is this not ablated?

Author response:
**Why not just make all comparisons and backprop all of them? Why is this not ablated?**

While we agree with the reviewer that the result of this experiment would certainly be interesting and valuable, its execution would require significantly larger compute resources because instead of selecting one pair from many and only computing the gradients for this pair, one would now have to compute and hold all gradients for all pairs which would drive up the memory and speed costs.

Reviewer comment (Experiments):
Figure 4 shows that the manual min IoU policy works for SimSiam but not for DINO. Maybe it’s just that multi-crop does not play well with min IoU. I am not fully convinced by this ablation.

Author response:
**Figure 4 shows that the manual min IoU policy works for SimSiam but not for DINO. Maybe it’s just that multi-crop does not play well with min IoU. I am not fully convinced by this ablation.**

We thank the reviewer for this constructive feedback and would like to explain the ideas behind this mentioned ablation:

Being aware of the computation expenses introduced by HVS, we thought about ways to simplify the method while maintaining its performance improvements. For this, we have adopted the idea of using distance metrics such as the IoU to incorporate into a simple but generally applicable learning strategy such as HVS is. 

As the reviewer pointed out rightly, the min IoU policy did not show the improvements across all baselines (specifically DINO) that HVS is able to provide. However, we emphasize that this policy was derived from running the SimSiam baseline. Since the main goal of this ablation was to be able to come up with a method that is generally applicable and transferable, we deemed it irrelevant to derive another policy with the DINO baseline. In our view, it is also much less straightforward to derive and apply such a policy in the multi-crop setting (let alone in other settings such as iBOT’s Masked Image Modelling objective) than it is to simply apply HVS. Instead, our interpretation of this ablation is that HVS’ model state and image-dependent selection of pairs is essential for the success of such a method. As a result, and agreeing with the reviewer, we believe that a manually derived min IoU policy (such as in the case of SimSiam) does not transfer to other baselines that entail variations in the augmentation pipeline (such as multi-crop). 

We hope we were able to convince the reviewer about the usefulness of this ablation and added the above explanation to the paper in Section “5.2 Q2: Can a Manual Augmentation Policy be Inferred?”.

Reviewer comment (Evaluation):
Throughout all the tables the performance improvements are small. Once factored in the ~2x slowdown, there seems to be almost no difference. This is not great, also considering that the proposed method probably uses a lot more memory than the vanilla method, and it is also more complex to implement. In practice, I would rather train longer with a well-known vanilla method than using the proposed method.

Author response:
**Throughout all the tables the performance improvements are small. Once factored in the ~2x slowdown, there seems to be almost no difference. This is not great, also considering that the proposed method probably uses a lot more memory than the vanilla method, and it is also more complex to implement. In practice, I would rather train longer with a well-known vanilla method than using the proposed method.**

We thank the reviewer for these constructive points. Hoping that we were able to address the critique regarding time and memory complexity of HVS in the general comment to all reviewers above, we additionally emphasize that implementing HVS is very simple and we invite the reviewer to briefly take a look at our code. All it takes to integrate HVS is to change the dataloader to sample “num_views” images [1] (in contrastive/non-contrastive learning one does this anyway since we always need two views at a minimum) and integrate our simple select function that is called in the train loop [2]. We are currently preparing the integration of HVS into pytorch through a pull-request where HVS could be integrated as a decorator for the train function and further ease the adoption of it.

We point out again that so far we have only seen evidence pointing to the fact that longer trainings with HVS also continue to improve the performance so that one can expect that the performance difference to the baseline is maintained.

[1] https://anonymous.4open.science/r/hard-view-selection/data.py (Line 31)

[2] https://anonymous.4open.science/r/hard-view-selection/pretrain.py (Line 196)

Reviewer comment (Experiments):
In addition, there exist more efficient ways to learn representations nowadays (e.g. CLIP). There are many papers that use multiple views for CLIP as well. Why not testing them?

Author response:
**In addition, there exist more efficient ways to learn representations nowadays (e.g. CLIP). There are many papers that use multiple views for CLIP as well. Why not testing them?**

Thank you for pointing this out. While we agree with the reviewer that other, multi-modal approaches exist to learn representations in a self-supervised way, we are unsure about the claim that those are necessarily more efficient.

It's worth noting that our current focus in this work has been on vision-only approaches, and our choice of baselines reflects that emphasis. We kindly ask the reviewer to consider the achievements and contributions made in this study rather than dwelling on aspects from other areas such as multi-modal self-supervised learning. Regarding the potential adoption of HVS in multi-modal settings, we acknowledge its relevance and importance and would view it as a valuable avenue for future exploration. We hope the reviewer can appreciate the intriguing results we have presented in this work, and we sincerely value the constructive feedback provided.

Reviewer comment (Novelty):
There are many other papers that treat this topic. They were not discussed. A few examples: "On the Importance of Asymmetry for Siamese Representation Learning", "What Makes for Good Views for Contrastive Learning?". These papers were also not compared with.

Author response:
**There are many other papers that treat this topic. They were not discussed. A few examples: "On the Importance of Asymmetry for Siamese Representation Learning", "What Makes for Good Views for Contrastive Learning?". These papers were also not compared with.**

We thank the reviewer for pointing out our shortcoming on referencing relevant related work. While we agree with the reviewer that we do not include the first mentioned paper, we do discuss Tian2020b (which is “What Makes for Good Views for Contrastive Learning”) in our paper. However, we agree with the reviewer that our paper would benefit from a deeper discussion about the similarities and differences with Tian2020b. First, Tian2020b leverages the information of views to characterize good views for a given task based on the mutual information principle and use this framework to discover sweet spots for data augmentation hyperparameters. These data augmentation hyperparameters are widely adopted by SSL approaches and our baselines. Additionally, Tian2020b use their insights about good views to then adversarially learn flow-based models that generate novel color spaces for the small STL-10 dataset. After the views have been learned offline, they then perform standard contrastive learning on these generated views. 

This approach is different to HVS on multiple fronts: with HVS we propose a method that can be integrated *online* into CL and non-CL pipelines without the use of a potentially instable adversarial learning objective. Our method is dependent on the model state and on individual samples since it is integrated online into the learning procedure, which is different to Tian2020b since their approach is not entangled with the model state. While Tian2020b ablates also based on distance metrics in their study, they do not include the IoU metric. Moreover, another central difference is that we use the distance metric as a tool to better understand the selection of HVS and to discover that model-state and image-dependence is essential, where Tian2020b uses it as a motivation for their framework. Furthermore, our ablation studies include experiments that quantify the difficulty of predicting good/hard pairs, which Tian2020b does not have. 

In summary, we view Tian2020b as complementary to our work, particularly in the context of their derived data augmentation policies and believe that HVS could even be combined with the findings of Tian2020b. However, it's important to underscore the limited overlap in methodology and experimental design between the two approaches. HVS in the end could be seen as an early work that shows that for good (or hard) views, we need to derive better sampling methods that can dynamically adapt to the actual state of training.

Regarding the first mentioned paper, “On the Importance of Asymmetry for Siamese Representation Learning”, we are doubtful about the relatedness to our work after reviewing it. The focus of this paper appears to be on improving representation learning by controlling encoding variance in the latent space of Siamese networks, exploring their findings with different augmentation strategies such as multi-crop. While we acknowledge the potential for combining HVS with their approach, the immediate connection to our work may require further clarification. If the reviewer identifies relevant points of intersection between our work and the mentioned paper, we are open to incorporating those insights. 

We have now added the expanded discussion on Tian2020b from above to the Introduction and Related Work sections of our paper. We thank the reviewer once again for the valuable feedback. Hopeful that we have addressed and mitigated the initial concerns, we would like to ask you if you could increase your score (or, alternatively, we would be glad for any points we should address that would raise your score).