Paper: Addressing Sample Inefficiency in Multi-View Representation Learning (ICLR-2024)

Reviewer comment (Experiments):
Several Non-contrastive methods such as BYOL and SimSiam or DINO have not involved any negative samples but they are not adequately compared.

Author response:
c. **Comparisons with Other SSL Methods**: Regarding the comparisons with other Self-Supervised Learning (SSL) methods like BYOL, SimSiam, and DINO, while we agree that a comprehensive comparison is valuable, we believe these comparisons are outside the scope of this work. In particular, we seek to establish theoretical foundations for correlation-based non-contrastive SSL. We aim to extend our theoretical framework to other families of non-contrastive SSL, with an eye toward practical recommendations to improve these methods.

Reviewer comment (Experiments):
The experimental setting is also not sufficient since it only trained for 100 epochs on CIFAR-10 and 50 epochs on STL-10, some methods with more augmentations can have a fast convergence, but for longer training that might be diminished. This setting is not practical where most SSL methods are not converged (as shown in the SimSiam paper). Please see [4] for reference where all SSL methods are conducted at least 1000 epochs.

Author response:
We thank the reviewer for sharing solo-learn [1] as a reference for several SSL algorithms pretrained for 1000 epochs. However, we would like to highlight that while solo-learn pretrains for 1000 epochs on CIFAR-10, they consider only Resnet-18 architectures that are computationally much cheaper than the Resnet-50 used in our experiments, making it difficult to run 1000-epoch pretraining on our compute budget. Nevertheless, we worked hard over the last few days to incorporate the reviewer’s feedback. We have updated Fig. 4 & 5 with 100-epoch pretraining on STL-10 and 400-epoch pretraining on CIFAR-10 (see Appendix A). Note that our core claims still hold, i.e., using multiple augmentations leads to good downstream task performance earlier in training. 


| **Algorithm**                          | **Best Accuracy (%)** | **Epoch of Best Accuracy** |
|:--------------------------------------:|:---------------------:|:-----------------------------:|
| Barlow-Twins (2-augs) w/ pdim=256      $\hspace{1cm}$ | 92.04 ± 0.16       $\hspace{1.5cm}$   | 400                      $\hspace{1cm}$     |
| Barlow-Twins (4-augs) w/ pdim=256      $\hspace{1cm}$ | 92.39 ± 0.17        $\hspace{1.5cm}$  | 340                      $\hspace{1cm}$     |
| Barlow-Twins (8-augs) w/ pdim=256      $\hspace{1cm}$ | 92.64 ± 0.10       $\hspace{1.5cm}$  | 140                      $\hspace{1cm}$     |

Reviewer comment (Evaluation):
The paper claims that more augmentations would help the representation learning quality but some figures show that 8 augmentations perform worse than 4 augmentations, this may be contradicting to that claim.

Author response:
We would like to clarify the misunderstanding regarding the performance of 8 augmentations. 
- First, Fig. 4 demonstrates the utility of multiple augmentations in improving convergence, i.e., learning better features earlier in training using the full dataset. We have added results for 8 augmentations while pretraining with BarlowTwins loss on CIFAR-10 to demonstrate that **increasing the number of augmentations leads to better downstream performance earlier in training** (see Appendix A), in line with our claims.
- On the other hand, Fig. 5 inspects the limits of SSL feature learning with fewer unique unlabeled samples. To this end, we vary the dataset size such that the total number of samples (after augmentations) in each epoch is the same, i.e., we compare training on 100% dataset with 2 augmentations, 50% dataset with 4 augmentations, and 25% dataset with 8 augmentations. The reviewer is correct in pointing out that training with 25% of the dataset with 8 augmentations can slightly hurt downstream performance compared to training with 50% dataset with 4 augmentations. We note this is not a discrepancy between our theory and empirical results. Instead, this indicates that while **multiple augmentations can recover competitive performance with fewer samples**, there is a **limit to their utility in the extremely low data regime**. To further highlight this tradeoff between downstream task performance and the number of unique unlabeled samples, we vary the number of augmentations and dataset size to develop an improved Pareto frontier, as shown in Fig. 6.

Reviewer comment (Novelty):
A large part of this submission has already been covered by two prior works ([1] in ICLR last year and [2] on arXiv in June), and there is nothing particularly new in this submission.

Author response:
1. ### Note on ICLR review guidelines:

Most of the reviewer’s criticism of our theoretical contribution relies on [2] Zhai et al paper available on arXiv in June 2023. However, this contradicts the ICLR guidelines: “consider papers contemporaneous if they are published (available in online proceedings) within the last four months” (https://iclr.cc/Conferences/2024/ReviewerGuide#FAQ). The same goes for [4] Cabannes et al. 2023, published at ICML 2023 on July 03, 2023 \[[proceedings](https://proceedings.mlr.press/v202/).\]

Reviewer comment (Writing):
The writing of this submission is quite confusing at times, especially in the mathematical part, which is not very rigorous and needs significant improvement.

Author response:
2. ### Feedback on theoretical typos:

We thank the reviewer for pointing out some of the typos in the theory sections of our paper. Indeed, as the reviewer pointed out, we have a typo in the definition of $T_M$, which should follow the definition in the text (see the line below). We will update the writing in the theory section of our manuscript, per the reviewer’s suggestions, but we would like to point out that these do not change the core takeaway of our work. Moreover, we feel that the reviewer’s feedback either focuses on minor points that detract from a holistic evaluation of our work or lacks specific constructive feedback on improving our work. We would therefore request the reviewer to kindly offer more specific points or concerns they may have regarding the theoretical arguments presented in our work.

Reviewer comment (Novelty):
The authors might have overlooked prior works [3, 4], in particular [4], which comes from the same group as VICReg and covers most results about VICReg presented in this submission.

Author response:
3. ### Misinterpretation of the scope of our work:
The reviewer cites several references ([1], [3], [4]) as superior or more definitive than our findings. However, these references predominantly offer theoretical and potential insights without concrete experimental validation. Furthermore, [1] & [3] specifically deal with contrastive learning frameworks (as the reviewer also notes). In the related work section, we mention how our work builds on insights from theoretical advances in contrastive SSL and cite [1]. While we agree with the reviewer that these works present useful theoretical insights, we strongly disagree that these works are “superior” to our work. On the contrary, we believe that our work presents complementary viewpoints, thereby improving the overall quality of scientific know-how in this research direction. For example, Section 6.2 in [4] mentions, “However, when the total amount $m \times n$ of pre-processed data is held fixed, it is generally better to process many inputs with two views $m = 2$, rather than a few inputs with many augmentations.” Our results (Fig. 5 & 6) show that this statement is not generally true in practical settings and that a tradeoff exists where multiple augmentations can sometimes yield better performance with fewer unlabeled samples. Therefore, it is indeed possible to use multiple augmentations to improve the sample efficiency of SSL algorithms, contrary to the recommendation provided by [4]. Unfortunately, the reviewer has currently overlooked these empirical contributions of our work.

We would like to reiterate that our work is a timely research (evidenced by concurrent submissions addressing similar questions) that marries theoretical insights with practical recommendations, thereby bridging the gap between SSL theory and practice. Specifically, we focus on non-contrastive SSL algorithms like BarlowTwins and VICReg. Based on the reviewer’s comments, we agree that it would be interesting to adopt a similar approach as our work and extend existing theoretical literature to present practical recommendations that apply to more general SSL algorithms. Therefore, our work, in addition to the concurrent works (mentioned by the reviewer), contributes to a better understanding of SSL algorithms and paves the way for more compute and sample-efficient SSL frameworks. However, that is outside the scope of our current paper.

In conclusion, we urge the reviewer to reassess our paper, considering its scope, content, and contributions rather than comparing/contrasting it against external publications, especially those not aligned with ICLR guidelines. We believe our work represents a significant step forward in SSL research by presenting a holistic perspective from theoretical insights to practical recommendations, thereby drawing the attention of SSL practitioners to recent theoretical developments.

Reviewer comment (Novelty):
The nature of other SSL algorithms such as MAE is incorrectly stated as an "open problem" (page 9), as it has already been addressed by [2].

Author response:
1. ### Note on ICLR review guidelines:

Most of the reviewer’s criticism of our theoretical contribution relies on [2] Zhai et al paper available on arXiv in June 2023. However, this contradicts the ICLR guidelines: “consider papers contemporaneous if they are published (available in online proceedings) within the last four months” (https://iclr.cc/Conferences/2024/ReviewerGuide#FAQ). The same goes for [4] Cabannes et al. 2023, published at ICML 2023 on July 03, 2023 \[[proceedings](https://proceedings.mlr.press/v202/).\]

Reviewer comment (Theory):
The definitions of $k^{DAB}$ and $T_M$ seem strange, which might be why the authors cannot prove that V(F) is the linear span of the top-d eigenfunctions. Specifically, $T_M f(x) = \int f(x_0) p(x_0|x) dp(x_0)$ yields two $p(x_0)$ in the numerator. A better definition could be $T_M f(x) = \int f(x_0) dp(x_0|x)$. Section 2.2 of [2] should be consulted, replacing $k^{DAF}, k^{DAB}$ with $K_A, K_X$ defined there.

Author response:
2. ### Feedback on theoretical typos:

We thank the reviewer for pointing out some of the typos in the theory sections of our paper. Indeed, as the reviewer pointed out, we have a typo in the definition of $T_M$, which should follow the definition in the text (see the line below). We will update the writing in the theory section of our manuscript, per the reviewer’s suggestions, but we would like to point out that these do not change the core takeaway of our work. Moreover, we feel that the reviewer’s feedback either focuses on minor points that detract from a holistic evaluation of our work or lacks specific constructive feedback on improving our work. We would therefore request the reviewer to kindly offer more specific points or concerns they may have regarding the theoretical arguments presented in our work.

Reviewer comment (Novelty):
The submission’s title refers to "addressing sample inefficiency", but the term "sample efficiency" is not clearly defined. Section 4.3 is the only part addressing it, showing experimentally that more diverse augmentations require fewer samples. This corresponds to a known result from [2] that stronger augmentations (including more diverse ones) lower sample complexity.

Author response:
3. ### Misinterpretation of the scope of our work:
The reviewer cites several references ([1], [3], [4]) as superior or more definitive than our findings. However, these references predominantly offer theoretical and potential insights without concrete experimental validation. Furthermore, [1] & [3] specifically deal with contrastive learning frameworks (as the reviewer also notes). In the related work section, we mention how our work builds on insights from theoretical advances in contrastive SSL and cite [1]. While we agree with the reviewer that these works present useful theoretical insights, we strongly disagree that these works are “superior” to our work. On the contrary, we believe that our work presents complementary viewpoints, thereby improving the overall quality of scientific know-how in this research direction. For example, Section 6.2 in [4] mentions, “However, when the total amount $m \times n$ of pre-processed data is held fixed, it is generally better to process many inputs with two views $m = 2$, rather than a few inputs with many augmentations.” Our results (Fig. 5 & 6) show that this statement is not generally true in practical settings and that a tradeoff exists where multiple augmentations can sometimes yield better performance with fewer unlabeled samples. Therefore, it is indeed possible to use multiple augmentations to improve the sample efficiency of SSL algorithms, contrary to the recommendation provided by [4]. Unfortunately, the reviewer has currently overlooked these empirical contributions of our work.

We would like to reiterate that our work is a timely research (evidenced by concurrent submissions addressing similar questions) that marries theoretical insights with practical recommendations, thereby bridging the gap between SSL theory and practice. Specifically, we focus on non-contrastive SSL algorithms like BarlowTwins and VICReg. Based on the reviewer’s comments, we agree that it would be interesting to adopt a similar approach as our work and extend existing theoretical literature to present practical recommendations that apply to more general SSL algorithms. Therefore, our work, in addition to the concurrent works (mentioned by the reviewer), contributes to a better understanding of SSL algorithms and paves the way for more compute and sample-efficient SSL frameworks. However, that is outside the scope of our current paper.

In conclusion, we urge the reviewer to reassess our paper, considering its scope, content, and contributions rather than comparing/contrasting it against external publications, especially those not aligned with ICLR guidelines. We believe our work represents a significant step forward in SSL research by presenting a holistic perspective from theoretical insights to practical recommendations, thereby drawing the attention of SSL practitioners to recent theoretical developments.

Reviewer comment (Novelty):
In the summary, the authors claim the submission provides "a fresh theoretical analysis", but most results are already known, and the only theorem is Theorem 3.1. The work seems more empirical or heuristic, with similar experiments present in prior work.

Author response:
3. ### Misinterpretation of the scope of our work:
The reviewer cites several references ([1], [3], [4]) as superior or more definitive than our findings. However, these references predominantly offer theoretical and potential insights without concrete experimental validation. Furthermore, [1] & [3] specifically deal with contrastive learning frameworks (as the reviewer also notes). In the related work section, we mention how our work builds on insights from theoretical advances in contrastive SSL and cite [1]. While we agree with the reviewer that these works present useful theoretical insights, we strongly disagree that these works are “superior” to our work. On the contrary, we believe that our work presents complementary viewpoints, thereby improving the overall quality of scientific know-how in this research direction. For example, Section 6.2 in [4] mentions, “However, when the total amount $m \times n$ of pre-processed data is held fixed, it is generally better to process many inputs with two views $m = 2$, rather than a few inputs with many augmentations.” Our results (Fig. 5 & 6) show that this statement is not generally true in practical settings and that a tradeoff exists where multiple augmentations can sometimes yield better performance with fewer unlabeled samples. Therefore, it is indeed possible to use multiple augmentations to improve the sample efficiency of SSL algorithms, contrary to the recommendation provided by [4]. Unfortunately, the reviewer has currently overlooked these empirical contributions of our work.

We would like to reiterate that our work is a timely research (evidenced by concurrent submissions addressing similar questions) that marries theoretical insights with practical recommendations, thereby bridging the gap between SSL theory and practice. Specifically, we focus on non-contrastive SSL algorithms like BarlowTwins and VICReg. Based on the reviewer’s comments, we agree that it would be interesting to adopt a similar approach as our work and extend existing theoretical literature to present practical recommendations that apply to more general SSL algorithms. Therefore, our work, in addition to the concurrent works (mentioned by the reviewer), contributes to a better understanding of SSL algorithms and paves the way for more compute and sample-efficient SSL frameworks. However, that is outside the scope of our current paper.

In conclusion, we urge the reviewer to reassess our paper, considering its scope, content, and contributions rather than comparing/contrasting it against external publications, especially those not aligned with ICLR guidelines. We believe our work represents a significant step forward in SSL research by presenting a holistic perspective from theoretical insights to practical recommendations, thereby drawing the attention of SSL practitioners to recent theoretical developments.

Reviewer comment (Novelty):
Given the popularity of foundation models and representation learning, the literature review is insufficiently thorough, and the authors should expand it before claiming originality or novelty.

Author response:
3. ### Misinterpretation of the scope of our work:
The reviewer cites several references ([1], [3], [4]) as superior or more definitive than our findings. However, these references predominantly offer theoretical and potential insights without concrete experimental validation. Furthermore, [1] & [3] specifically deal with contrastive learning frameworks (as the reviewer also notes). In the related work section, we mention how our work builds on insights from theoretical advances in contrastive SSL and cite [1]. While we agree with the reviewer that these works present useful theoretical insights, we strongly disagree that these works are “superior” to our work. On the contrary, we believe that our work presents complementary viewpoints, thereby improving the overall quality of scientific know-how in this research direction. For example, Section 6.2 in [4] mentions, “However, when the total amount $m \times n$ of pre-processed data is held fixed, it is generally better to process many inputs with two views $m = 2$, rather than a few inputs with many augmentations.” Our results (Fig. 5 & 6) show that this statement is not generally true in practical settings and that a tradeoff exists where multiple augmentations can sometimes yield better performance with fewer unlabeled samples. Therefore, it is indeed possible to use multiple augmentations to improve the sample efficiency of SSL algorithms, contrary to the recommendation provided by [4]. Unfortunately, the reviewer has currently overlooked these empirical contributions of our work.

We would like to reiterate that our work is a timely research (evidenced by concurrent submissions addressing similar questions) that marries theoretical insights with practical recommendations, thereby bridging the gap between SSL theory and practice. Specifically, we focus on non-contrastive SSL algorithms like BarlowTwins and VICReg. Based on the reviewer’s comments, we agree that it would be interesting to adopt a similar approach as our work and extend existing theoretical literature to present practical recommendations that apply to more general SSL algorithms. Therefore, our work, in addition to the concurrent works (mentioned by the reviewer), contributes to a better understanding of SSL algorithms and paves the way for more compute and sample-efficient SSL frameworks. However, that is outside the scope of our current paper.

In conclusion, we urge the reviewer to reassess our paper, considering its scope, content, and contributions rather than comparing/contrasting it against external publications, especially those not aligned with ICLR guidelines. We believe our work represents a significant step forward in SSL research by presenting a holistic perspective from theoretical insights to practical recommendations, thereby drawing the attention of SSL practitioners to recent theoretical developments.

Reviewer comment (Experiments):
The experimental results presented in the paper are less than convincing and appear to involve an unfair comparison. Figure 4, in particular, showcases curves that have not converged. A fair comparison should ensure that all models are optimized and have reached convergence.

Author response:
1. ### On Experimental Results and Convergence:

We thank the reviewer for pointing out this apparent lack of clarity in the result figures. We would like to reiterate that our main claim is that using more augmentations will lead to faster convergence, i.e., learning better features earlier in pretraining. Our current results demonstrate that using multiple augmentations leads to better linear readout performance earlier in training, both in terms of full passes over the dataset (epochs) and wall clock time (see Fig 4). We incorporated the feedback and **ran new experiments where we pretrain each model to 100 epochs** (see updated Fig. 4 & 5) to align with standard practice in SSL pretraining of ResNet50 networks. However, the reviewer’s concerns are valid in settings with sufficient compute budget to pretrain models till convergence and select the best-performing checkpoint. To address this concern, we **have run additional experiments wherein we pretrain ResNet50 models with BarlowTwins on CIFAR-10 up to 400 epochs** and have added these new results (see Appendix A). In line with our claims, using multiple augmentations leads to similar (or even slightly better) performance much earlier in training.

| **Algorithm**                          | **Best Accuracy (%)** | **Epoch of Best Accuracy** |
|:--------------------------------------:|:---------------------:|:-----------------------------:|
| Barlow-Twins (2-augs) w/ pdim=256      $\hspace{1cm}$ | 92.04 ± 0.16      $\hspace{1.5cm}$    | 400          $\hspace{1cm}$                 |
| Barlow-Twins (4-augs) w/ pdim=256      $\hspace{1cm}$ | 92.39 ± 0.17      $\hspace{1.5cm}$    | 340           $\hspace{1cm}$                |
| Barlow-Twins (8-augs) w/ pdim=256      $\hspace{1cm}$ | 92.64 ± 0.10       $\hspace{1.5cm}$   | 140            $\hspace{1cm}$               |

Reviewer comment (Novelty):
The utilization of multiview is not a novel concept, and it appears to be an inferior approach when compared to the multi-crop technique employed in SwAV. SwAV has thoroughly investigated this and found that using full views can be less effective due to increased memory overhead. Surprisingly, this paper does not address memory usage and effective computation, which raises questions about the validity of its claims. If each iteration consumes more computational resources and requires additional memory, it is expected to converge faster, making it a critical factor to consider.

Author response:
2. ### Innovation and Utility of Multiview Technique:

We agree with the reviewer that using multiple views is a well-established heuristic in contrastive SSL frameworks like SwAV. However, our core contribution is two fold:
- We provide theoretical principles that explain why a multiview approach is useful in practice while extending this approach to non-contrastive learning setups like BarlowTwins and VICReg.
- Furthermore, our results demonstrate that the multiview approach not only yields good performance but is also useful for learning good features from fewer unlabeled samples. As highlighted by Reviewer QPCx our insights present “a strategy that reduces the data needed to learn good representations may benefit the community.” 

We would also like to note that SwAV presents a useful empirical approach to use multiple augmentations and would be a special case of our theoretical framework, wherein using crops of smaller size leads to a particular choice of mapping $M$ (and correspondingly operator $T_M$). Nevertheless, we have also **added some new experiments** (see Appendix A) where we adopt a SwAV-style approach, i.e., 2 augmentations of full resolution and additional augmentations of smaller resolution, and demonstrate minor performance gains (at lesser compute cost than full-view augmentations) over the vanilla 2-view approach. While the decreased performance gain (as opposed to using full views for all augmentations) could stem from an implicit bias over the scale of details in learned features from the particular choice of $M$, these results still align with our theoretical conclusions regarding the utility of multiple augmentations. Taken together, we would like to disagree with the reviewer’s claim about the inferiority of our method compared to SwAV and instead present SwAV as a special (compute-efficient) case of our framework.

3. ### Memory Usage and Computation Efficiency:

The reviewer's point on memory usage and computation efficiency is well-taken. Our omission of an in-depth discussion on this aspect was due to space constraints and our primary focus on the theoretical aspects of SSL. However, we agree that this is an important factor and will add a section discussing computational resources and memory usage in the context of our proposed methods. Would such an addition address the reviewer's concerns in this area?

Reviewer comment (Presentation):
There seems to be a lack of a clear connection between the discussion of the graphs and the paper's main idea, which can make the paper's arguments less coherent.

Author response:
4. ### Connection Between Graphs and Main Idea:

We thank the reviewer for pointing out this apparent lack of clarity. The main idea of our paper is to make SSL more compute and sample efficient, i.e., to learn good features with a lower compute footprint from fewer unique unlabelled samples. Our results demonstrate that contrary to dominant heuristics for NC-SSL pretraining,  it is possible to **learn good features with smaller projector dimensionality** (Fig. 3), thereby reducing the compute footprint of our models. Furthermore, our theoretical results suggest the **potential of using multiple augmentations to learn good features with fewer passes over the dataset**, i.e., training epochs (Fig. 4) as well as **with fewer unique unlabeled samples** (Fig. 5). Therefore, we believe these results are strongly aligned with the main idea in the paper but are happy to add further clarification in relevant sections. We would greatly appreciate specific feedback from the reviewer on this point.

Reviewer comment (Experiments):
What happens if you repeat the experiments from Figure 5 with {4, 8} (and even more?) data augmentations but with 100% of the training data?

Author response:
We thank the reviewer for this valuable suggestion. Accordingly, we add new experiments with 8 augmentations with Barlow-Twins on the full CIFAR-10 dataset (Appendix A). 

| Algorithm                            | #augs=2           | #augs=4           | #augs=8           |
| :------------------------------------: | :-----------------: | :-----------------: | :-----------------: |
| Barlow-Twins (CIFAR-10) w/ pdim=256       $\hspace{1cm}$ |    86.43 $\pm$ 0.72   $\hspace{1cm}$ | 91.73 $\pm$ 0.16  $\hspace{1cm}$  | 92.71 $\pm$ 0.19  $\hspace{1cm}$  |
| Barlow-Twins (CIFAR-10) w/ pdim=8192     $\hspace{1cm}$ |    85.44 $\pm$ 0.54    $\hspace{1cm}$ | 91.40 $\pm$ 0.32  $\hspace{1cm}$  | 92.40 $\pm$ 0.13 $\hspace{1cm}$  |

From the above table, it’s clear that increasing the number of augmentations improves downstream performance in line with our claim. However, using more augmentations also increases the per-epoch time, thereby increasing the overall experiment time. In other words, a tradeoff exists between the performance and the compute time. This tradeoff is further demonstrated in Fig. 6 of our manuscript, where we present a Pareto frontier of performance vs compute time by varying the number of augmentations and the fraction of the dataset. Therefore, given our computing budget, we have restricted our experiments to 8 augmentations.

In summary, we appreciate the constructive feedback and agree that the aspects highlighted for improvement are crucial for a more comprehensive understanding of non-contrastive SSL techniques. We are committed to exploring these areas and believe they will significantly enhance the depth and applicability of our research.