Paper: Self-Supervised Learning with the Matching Gap (ICLR-2024)

Reviewer comment (Evaluation):
[Evaluation] The evaluation section could have been more comprehensive. For example, when strong data augmentations are involved (Table 1), only several SSL baselines are included, e.g. MoCo-v3, DINO. The settings shown in Table 1 are also not consistent, e.g. different number of epochs, which makes it difficult to interpret the results, e.g. could the proposed method match DINO’s performance when trained for the same number of epochs? Also, it would be beneficial to include architectures beyond ViT-B(L)/16, e.g. ViT-S, CNN, etc.

Author response:
■ In presenting (and evaluating) MG, our focus was not to reach SOTA performance, but to suggest a simple alternative to contrastive SSL losses that can naturally avoid collapse. We do show that a “lean” implementation of the MG loss leads to comparable results to SOTA.
Instead of chasing SOTA, we used our compute budget to explore MG’s performance over diverse settings (decreasing amount of augmentations, removing momentum encoder and predictor heads, or other experiments in section A.3 “Additional ablation studies”, results over smaller datasets (CIFAR-10, CIFAR-100, and SVHN) using ResNet-50 as backbone). 
We found that MG is consistently near or at the top, despite being much simpler conceptually than most approaches. Following your (and other reviewers’ comments), we are now running additional experiments, to study impact of #epochs.

Reviewer comment (Evaluation):
[Performance] With strong augmentations, the proposed method shows no benefit compared to SOTA methods. The method outperforms several SSL approaches with weaker augmentations. However, at least from the application perspective, it is unclear to the reviewer what are the advantages of using only weak augmentations, especially when the training epochs are the same.

Author response:
■ As mentioned in our introduction, increasing the variety/strength of augmentations can be seen as a way to mitigate the tendency of contrastive losses to *force* instances of the same image to collapse in representation space. However, as nicely presented by Assran et al. 2023 the reliance on hand-crafted data augmentations is limiting and may introduce unwanted biases. With that, we find that attaining good performance with a smaller number of epochs, and a smaller set of augmentations, is a worthy direction. Besides, this can be also useful in settings (e.g. biomedical data) where the number of “natural” augmentations is more limited than in vision tasks (and where even testing the hypotheses that many augmentations are needed, or not, is harder).

Reviewer comment (Evaluation):
[Claim] Some of the claims are not well-justified. For example, i) compared to CL that may “collapse representations for very different images”, the proposed method learns diverse representations for different views of the same image; ii) stronger data augmentations could help mitigate the representation collapsing problem. In order to justify the claims, the authors may measure/compare the mean distance of different views of the same image, or different samples of the same classes, across different models (i.e. the proposed method and baselines) and settings (i.e. strong/weak augmentations).

Author response:
■ We thank the reviewer for this suggestion. We are now evaluating statistical properties of the representation space of MG and alternative approaches, under different augmentations of the same image. We will update the response once we have results, and hope to include it in a revised manuscript version within the rebuttal period.

Reviewer comment (Evaluation):
The advantages of the single-level optimization in MG loss over the bi-level optimization in IOT loss are not provided clearly. Figure 2 shows that MG loss slightly underperforms but is competitive with IOT loss. I wonder if it improves the training speed/convergence or reduces the memory consumption?

Author response:
■ Our MG approach will be at least twice as fast when it comes to computing the loss and its gradient, since only requires a forward pass. It will also be more memory efficient if one chooses to unroll the Sinkhorn solutions, or even faster if one chooses to differentiate implicitly the OT solution (see e.g. Luise et al. 2018), which requires solving a $n\times n$ linear system.

That being said, in the large-scale setting that we have used — notice that IOT (Shi et al. 2023) was never evaluated on ImageNet —  the overhead of running a ViT-B architecture far dominates the costs of running Sinkhorn when using a small batch size of 128, a fairly large entropic regularization ($\epsilon$). The loss computation, whether MG or IOT, is negligible. The speedup of MG vs. IOT did not, therefore, play an important factor in the overall compute cost. As we mention in the conclusion, this tradeoff will change when using simpler feature architectures, larger batch sizes, or, as mentioned in the conclusion, considering other OT approaches that cannot be differentiated (e.g. low-rank solvers).

Reviewer comment (Experiments):
Unfair comparisons. The implementation of the experiments largely followed the setting of Dino, which used two global crops and ten local crops by default. However, some of the baseline methods, e.g., MoCov3 and I-JEPA, used only two global crops, making it unfair to directly compare the performance with MG loss on the default setting.

Author response:
■ In presenting MG our goal was not to suggest a combination of tricks that could reach SOTA performance, but rather to provide a conceptually simple alternative to contrastive SSL losses, with the hope that, when applied in  “naively”, its performance will be consistently comparable or better than far more complex approaches that use several tricks. We believe we have delivered in that sense.

Following this objective, we have used our computation budget to explore a variety of settings that showcase the robustness of MG, the simplifications it allows for, as well as its shortcomings. 

We indeed relied on reported performances for baseline methods in the most comparable setting (e.g. matching backbone architecture). Here, it is important to note that while indeed the number of crops provides an advantage for MG in a majority of cases MG is evaluated over a **significantly lower number of epochs**.

At last, specifically concerning the comparison to I-JEPA, Assran et al. (2023) present the new setting, Image-based Joint-Embedding Predictive Architecture, in which the construction is by definition different: for each image, given a context block, predict the representations of various target blocks, where $M$ (typically setting $M=4$) non-overlapping targets are chosen. Hence, it is not straightforward to define an exact _fair_ comparison between these settings.

Reviewer comment (Evaluation):
Even under the potentially unfair comparison, the performance of MG loss is only comparable and sometimes even inferior to the contrastive loss.

Author response:
■ In presenting MG our goal was not to suggest a combination of tricks that could reach SOTA performance, but rather to provide a conceptually simple alternative to contrastive SSL losses, with the hope that, when applied in  “naively”, its performance will be consistently comparable or better than far more complex approaches that use several tricks. We believe we have delivered in that sense.

Following this objective, we have used our computation budget to explore a variety of settings that showcase the robustness of MG, the simplifications it allows for, as well as its shortcomings. 

We indeed relied on reported performances for baseline methods in the most comparable setting (e.g. matching backbone architecture). Here, it is important to note that while indeed the number of crops provides an advantage for MG in a majority of cases MG is evaluated over a **significantly lower number of epochs**.

At last, specifically concerning the comparison to I-JEPA, Assran et al. (2023) present the new setting, Image-based Joint-Embedding Predictive Architecture, in which the construction is by definition different: for each image, given a context block, predict the representations of various target blocks, where $M$ (typically setting $M=4$) non-overlapping targets are chosen. Hence, it is not straightforward to define an exact _fair_ comparison between these settings.

Reviewer comment (Writing):
Some notations are used without first introduced, e.g., $c(\cdot,\cdot)$ in Introduction and $t(\cdot,\cdot)$ in Sec. 3.

Author response:
■ $c(\cdot,\cdot)$ is simply a cost, this was defined at the beginning of Section 2.2, we now mention it earlier in the intro when first introduced. The function $t(\cdot,\cdot)$ was defined in Equation 3 (using the $:=$ symbol). We have now split it to make it more visible and clear.

Reviewer comment (Novelty):
The novelty of this paper is rather limited. This article only uses a previously proposed technique to improve the computational complexity of inverse OT-based contrast loss in the optimization process. I do not find any new insights related to the field of contrastive learning.

Author response:
■ We respectfully disagree, our method was not proposed previously, and our goal is not to improve the computational complexity of inverse OT. Improving inverse OT would require carrying out a more efficient differentiation of $\rm{argmins}$ solutions (using e.g. unrolling or the implicit function theorem). Instead, our method **bypasses** those challenges **entirely** thanks to a different formulation, resulting in a much simpler optimization.

Reviewer comment (Writing):
This paper is really hard to follow. There are many mathematical symbols and proper nouns that lack explanation. For example, what is t in eq. 6 and 7, what is bistochastic matrices, and what are the difference between matching cost, measuring agreement, matching gap, and optimality gap?

Author response:
■ We regret that the reviewer found the paper hard to follow. 
The function $t$ is defined in Equation 3 ($:= $), in the revised manuscript we have made this definition more apparent; 
The definition of bistochastic matrices is provided in Section 2.2, $\mathcal{B}_n$, and is a standard concept (see. E.g. https://en.wikipedia.org/wiki/Bistochastic_matrix);
The “matching cost” is defined in Equation 3;
“Measuring agreement” is used in a sentence to describe the IOT loss (“Inspiration: Measuring Agreement using Inverse OT.”). This refers directly to a sentence above, _“In such cases, OT can be used as a way to quantify whether locations (xi, yi) are in agreement with the diagonal pairing”_;
The _matching gap_ is the name of the method we introduce; 
The optimality gap, or “gap to optimality” is a common concept in optimization, quantifying to what extent a solution is not optimal. We use this reference to highlight that the matching gap is simply the objective value of $J_n$ vs. the best possible objective value, as in Equation 7.

Reviewer comment (Evaluation):
The experimental results cannot verify the effectiveness of the proposed method. First, the performance gain is pretty small. Second, there are many cases where the proposed method obtains a bad result.

Author response:
■ In presenting the MG loss we did not claim (or aim) for SOTA performance. The MG was applied without any attempt to introduce new "tricks" that are often used to push performance in SSL. We did not perform extensive hyper-parameter scans, or architecture fine-tuning, and therefore cannot attain SOTA performances obtained typically using more effort in this regard. 

Instead, our goal was to present experiments over various settings, exploring the robustness, the shortcomings, and the ability of the MG to remove some of those tricks. It is therefore expected that MG does not always come on top in any setting. Instead, we want to highlight that MG is consistently near or at the top, despite being much simpler conceptually than most approaches and being an "upgrade" (set-based, as mentioned by Reviewer tJTN) of classical  InfoNCE. 

We do, however, respectfully disagree with the reviewer, as we believe the method does not obtain _bad_ results. With that, if at any place the reviewer feels that we have oversold our method, we are of course keen to tone down any claim they find problematic.

Reviewer comment (Writing):
The "A Link between InfoNCE and the Matching Gap" part and the "Our Contribution: Single Level Optimization with the Matching Gap" part are so vague that I have read them many times without understanding the logical relationship.

Author response:
■ Did you find the sections themselves vague, or the interplay between them? If any of our attempts at making this intuitive failed, we regret this and ask instead the reviewer to focus on the equations.
 
The section "Our contribution: Single Level [...]" presents the crux of the method. It provides all the information required to form our loss, presented in Eq. (6,7), borrowing notations and definitions introduced in Eq. (3,4).

The section "A Link between InfoNCE and the Matching Gap" positions our contributions in the literature. It mentions that the InfoNCE loss and its gradients, in Equations (1,2), should simply be contrasted to (9,10) in our approach. (1,2) and (9,10) are structurally identical, except for the weights, which are computed adaptively, with an OT twist, in (9,10).

Reviewer comment (Theory):
How do you get eq. 8 from eq. 7?

Author response:
■ We simply expanded the notation $t$ that was introduced in Eq. 3. There was a small typo, highlighted by reviewer **1FzB**, that might have caused confusion. We apologize for this, and corrected it in the revised manuscript.

Reviewer comment (Experiments):
Experimentation - I think that this new form of loss would be better justified if there would be empirical evidence that supports the intuitions (in addition to the standard benchmarking and ablations). It would perhaps be interesting to see how the embeddings of an augmented batch behave, in comparison to standard NCE, or some statistics of that kind.

Author response:
We agree, and we have been working on such a presentation.

Reviewer comment (Theory):
The formulation is restricted to the 2-view setting. While this is simple, it would be interesting to know whether there are effective generalizations to multi-view settings.

Author response:
This is indeed a very interesting avenue for further research.

Reviewer comment (Experiments):
There is no specification or discussion regarding batch size, which has an important role in contrastive learning. Supposedly, the compact computation and 2-view setup could allow for larger batch sizes. It would be interesting to see how performance scales with batch-size.

Author response:
Thanks for this great suggestion. As can be seen in our general response above, we have investigated this aspect. We reproduce these results below for convenience. Using **MG** in a setting strictly identical to that reported in Table 1 (300 epochs over ImageNet), we obtain the following accuracies

| $n$ |  Linear Accuracy|
|-----| ---------------|
| 32 | 75.43 %  |
| 48 | 75.83 % |
| 64 | 76.29 % |
| 96 | 76.71 % |
| 128 | 76.70 % |
| 192 | 73.24 % |

As can be seen in these results, performance is fairly stable across $n$ sizes, and maybe suggest that for larger $n$, one might need to readjust $\varepsilon$ values. We are still running results for even smaller sizes (e.g. 16, 24) and will report them.

Reviewer comment (Theory):
Several minor inaccuracies (which don't affect the analysis or correctness): (i) Bistochastic should be non-negative (ii) Should be <P,logP> in Equation 3 (without the -1) (iii) Last row of the loss equation is wrong, resulting in a matrix rather than a value: should probably be \eps<P,logP> instead of \eps\logP.

Author response:
Many thanks for spotting these typos:
- (i) yes, we have added the "+"
- (ii) this "-1" is a convention that is adopted, for instance, in the computational OT book (e.g. https://arxiv.org/pdf/1803.00567.pdf Eq. 4.1, p.57). This helps rewrite the dual more elegantly, and also plays a role for unbalanced formulations, where the generalized KL is used.
- (iii) apologies for this ugly typo, now corrected.

Reviewer comment (Experiments):
I understand (and am in favor of) the limited budget experimentation. Did you ablate on number of epochs, within the budget, to see if the dimnishing returns behavior is comparable to other methods?

Author response:
Yes, we have ran this experiment, as reported above. As mentioned in the general remark, the performance did not improve markedly when going from 300 to 600, when keeping all other hyperparameters unchanged ($76.5 %$)

Reviewer comment (Experiments):
Due to the approach that does not require positives to converge to the same point - Perhaps there is actually room for more aggressive augmentation, that can exploit a richer extension of the training data?

Author response:
This is a good remark. However, as we debate in the paper, we took the opposite direction, and did instead try to get rid of aggressive augmentations, to simplify the overall pipeline (see e.g. Table 3). Our intuition is that agressive augmentations are a form of regularization that is needed with InfoNCE.

Reviewer comment (Novelty):
As I know, DINOv2 and SwAV also use the optimal transport algorithm to solve contrastive learning. But I can not find the discussion about such methods in related work. I would like to find the discussion about the difference between Matching Gap and SwAV/DINO.

Author response:
■  Thanks for pointing this out. We agree that it is worth adding a discussion on this, which we did in the updated pdf, paragraph “Related works” in Section 3. To develop: optimal transport does indeed plays a role in SwAV (Caron et al. 2020) and DINOv2 (Oquab et al. 2023), where optimal transport, and more specifically the *Sinkhorn algorithm*, is used as a differentiable *proxy* to obtain a *balanced* $k$-means clustering of representations (i.e. each cluster must capture a similar amount of mass).  Note that in DINO (Caron et al. 2021), Sinkhorn was replaced with Softmax (similar to a single Sinkhorn iteration), but was later reintroduced in DINOv2, following Ruan et al. (2022). Importantly, in both SwAV and DINOv2, OT is used as a low-level clustering routing, only *indirectly* involved in the final loss, which is a more standard KL-based quantity, on a discretization using those clusters. With that, because OT is used at an intermediate step, this requires, as in Inverse OT (Shi et al. 2023), differentiating the Sinkhorn iterations.

Reviewer comment (Experiments):
The author only conducts downstream experiments on transfer classification. Many self-supervised learning methods evaluate the downstream detection (COOC, VOC) and segmentation (COCO, aed20k) performance. I think the simple downstream classification task is not enough.

Author response:
■ We thank the reviewer for making these suggestions. We are now conducting these evaluations and hope to include them in an updated version within the rebuttal period.

Many thanks for getting back to us.

We just wanted to mention that we are still in the process of adding additional experiments. We have added a few runs following reviewer **iFzB**'s questions, and are currently doing our best to handle your request to evaluate other downstream tasks. Thanks for your patience.