Paper: Poly-View Contrastive Learning (ICLR-2024)

Reviewer comment (Reproducibility):
What is the exact loss function the paper used to define the poly-view contrastive learning? It seems the Eq. (22) is the poly-view contrative loss, whereas it is in a very high level abstract and implicit form, making it hard to interprete how to compute the empirical loss for the terms, and why M=2 links to SimCLR. I recommend to make the loss in a more explicit form of empirical losses and interpretation (e.g., what is the used sufficient statistics for M=2 for SimCLR? )

Author response:
### *W1.1: What is the loss for PVC?*
Minimizing the PVC losses maximizes the MI lower bounds of Theorem 2.2. Based on your comment we have clarified the corresponding equations, making it clear what the loss is. Additionally, the full pseudo-code for all loss computations is provided in Appendix F.2.5, allowing any reader to follow the loss computation line by line and reproduce our results.

### *W1.2: How does M=2 link to SimCLR?*
In Appendix E.2 we show how $M=2$ gives SimCLR for PVC. We have now added an equivalent derivation for sufficient statistics.

Reviewer comment (Evaluation):
Empirical evidence lacks suitable interpretation and linkage to the significance of the theorems. For significance, I mean how we can use the theorem takeaways to practically improve the SSL algorithms? I expect to see the evidence on larger dataset with mainstream architecture such as ResNet and ViT/transformer with longer epochs.

Author response:
### *W2.1: Empirical evidence lacks suitable interpretation and linkage to the significance of the theorems. For significance, I mean how we can use the theorem takeaways to practically improve the SSL algorithms?*
The PVC and Sufficient Statistic algorithms are a direct consequence of Theorems 2.2 and 2.4 respectively. These theorems tell you how to construct a Poly-View loss function. Without the theorems, one can use heuristics and develop Multi-Crop, which Poly-View methods outperform in all of our experiments.

In the synthetic case (Section 3.1) we observe experiments directly following theorems from Section 2. For example, increasing multiplicity $M$ increases One-vs-Rest MI as derived in Section 2.3.1 on data processing inequality. We also observe One-vs-Rest MI empirically converges to $\mathcal I(x_\alpha; c)$ which, as discussed in Section 2.3.1, is the upper bound for One-vs-Rest MI.

The ImageNet1k experiments follow similarly. Geometric and Sufficient Statistics with higher multiplicity $M$ provide tighter MI bounds and a better proxy for InfoMax, resulting in improved performance when holding other hyperparameters fixed (Figure 3a).

### *W.2: I expect to see the evidence on larger dataset with mainstream architecture such as ResNet and ViT/transformer with longer epochs*
Please see Figure 3a for training and evaluation on ImageNet1k dataset, with a ResNet50 architecture trained for up to 1024 epochs.

Reviewer comment (Experiments):
It is unclear to me if the multiplicity of views simply benefits from more equivalent of epochs (in the experiments) or whether the exposure to the number of data has been constrained to be exactly same between the poly-view contrastive learning and other baselines.

Author response:
### *W.3: It is unclear to me if the multiplicity of views simply benefits from more equivalent epochs (in the experiments) or whether the exposure to the number of data has been constrained to be exactly the same between the poly-view contrastive learning and other baselines*
We were also concerned about this, and chose to focus on it in Section 3.2.

For Figure 3b, we train $M=2$ SimCLR models and $M=8$ Poly-View for 64, 128, 256, 512 and 1024 epochs. We do this either by holding the total number of views fixed (Fixed Batch. bottom row) which means we drop the batch size $B$ with increasing $M$, or allow total views to increase (Growing Batch, top row).

1. To answer whether it is the total number of samples seen by the model, we can look at the leftmost column of Figure 3a. We see that Poly-View methods always outperform SimCLR.
2. To answer whether it is the total number of updates seen by the model, we can look at the center column of Figure 3a. We see that Poly-View methods always outperform SimCLR.
3. To answer whether it is the total compute spent on the model, we can look at the right column of Figure 3a. Poly-View methods outperform SimCLR only in the Fixed Batch case, but not in the Growing Batch.

If *effective epochs* corresponds to *total steps* then in the above scenario 2 we see that for the same amount of effective epochs, Poly-View methods still outperform non-Poly-View methods.

We have also added a fourth way of comparing models in Appendix F.2.4 where we show the Total FLOPs used to produce each model. The conclusion is consistent with comparisons based on Relative Compute.

Please let us know if anything else is still not clear.

Reviewer comment (Theory):
The assumption 3 in section 2.4.1 is kind of strong.

Author response:
### *W1: Exponential family assumption is strong*

Our representations are given by a neural network and are therefore finite-dimensional (2048 in the ResNet50 case), typical in self-supervised learning. As the sufficient statistics is a map to the representation space, according to the Fisher-Darmois-Koopman-Pitman (FDKP) theorem, for smooth nowhere vanishing probability densities, a finite-dimensional sufficient statistic exists if and only if the densities are from an exponential family (https://ieeexplore.ieee.org/document/4048925).
I.e. the sufficient statistics are distributed according to the exponential family because they map to finite-dimensional representations.

The assumption of finite-dimensional representations is not strong in the context of self-supervised learning. 
The exponential families that follow from this assumption are widely used in machine learning due to the availability of efficient methods as well as their tractability and interpretability. Many traditional distributions are in the exponential family, e.g. normal, exponential, log-normal, gamma, chi-squared, beta, Dirichlet, Bernoulli, categorical, Poisson, geometric, inverse Gaussian, von Mises, and von Mises-Fisher distributions.
Finally, in self-supervised learning, the exponential families have appeared in many recent works, e.g. https://arxiv.org/abs/2102.08850, https://arxiv.org/abs/2203.07004, and https://proceedings.mlr.press/v89/hyvarinen19a.html.

We have reworded assumption 3 to make it clear it is a consequence of modeling representations using a neural network, and *not* an assumption. Thank you for highlighting this to us.

Reviewer comment (Experiments):
The experiments shown in section 3 display the relative compute of algorithms and show One-vs-Rest objectives could beat simCLR with the same relative compute, how about the real training complexity. And how does it perform on real downstream tasks.

Author response:
### *W3/Q1: Computation time is not evaluated by real time*

In Figure 3 we evaluate computation time using:
- Total training epochs  
- Total updates
- Relative Compute

These metrics capture different notions of time, as choosing only one or two of these metrics paints an incomplete picture. E.g., displaying only total updates does not take into account the practical consideration that a single update for one setting may require more computation than a single update for another, which would be captured by also including Relative Compute.

The three metrics above were also chosen as they are ML framework agnostic and hardware agnostic. Wall-clock time on the other hand can vary according to CUDA version, GPU model/compute capability network interconnect (e.g. infiniband/EFA availability). This makes wall-clock time difficult to use reproducibly.

Based on your comment, we have included total FLOPs, another popular, reproducible measure of computation time, in Appendix F.2.4, to complete the picture. We have also added a comment that, assuming an optimal hardware setting and fully-parallel implementation, total updates is proportional to wall-clock time, which we hope addresses your primary concern. Thank you again for your comment.

Reviewer comment (Theory):
I cannot fully understand why the One-vs-Rest loss could effectively reduce the training epoch and batch size.

Author response:
### *Q2: Why does One-vs-Rest reduce 1) training epochs and 2) batch size*
1. One-vs-Rest for high multiplicity $M$ is a better proxy for InfoMax (Section 2.1 and Equation 27), and so increasing $M$ gives a better signal to noise. There is also an optimizer scaling law interpretation (https://arxiv.org/abs/2102.12470, https://arxiv.org/abs/2307.13813) where we can trade 2 steps of 2 views per sample with one step of 4 views per sample.
2. Based on 1., One-vs-Rest means you can maintain performance if you reduce your batch size if you increase $M$ accordingly.

Reviewer comment (Evaluation):
An extension of simCLR loss outperforms the carefully designed SUFFICIENT STATISTICS loss in section 3, does it mean that the poly-view contrastive learning works mainly because the superiority of simCLR loss?

Author response:
### *Q3: Does PCV outperform sufficient statistics because of superiority of SimCLR loss?*
Sufficient statistics with $M=2$ is SimCLR (Section 2.4.2 and a new section in Appendix E.2), i.e. sufficient statistics also has the benefits of SimCLR. Additionally, all experiments show sufficient statistics with $M>2$ outperforms SimCLR ($M=2$). What we do observe is that PVC-Geometric outperforms sufficient statistics, but the reason cannot be due to SimCLR for the above reasons and is an interesting direction for future work.