Paper: On the Provable Advantage of Unsupervised Pretraining (ICLR-2024)

Reviewer comment (Theory):
The primary weakness of this paper stems from assumption 3.2, which appears to be an overly robust condition. The core challenge of the paper lies in discerning the (constraint) relationship between (x, s) and (x, z). By introducing a new parameter, $\kappa$, to simply constrain this relationship, the paper sidesteps the intricacy of this challenge. While the existence of such a parameter in the three sub-cases presented in the paper is addressed, its presence in more complex scenarios remains questionable. The use of such a strong assumption might detract from the general applicability of the results.

Author response:
**Q1: Over-reliance on Assumption 3.2**

A: While most prior works focused on specific applications, this paper aims to identify generic structural condition which allows unsupervised pretraining to be beneficial to learning downstream tasks. We view informative conditions (assumption 3.2) as one of the first steps towards understanding in general scenarios. We agree that assumption 3.2 is only a sufficient condition, and identifying weaker assumptions is an interesting future direction.

Reviewer comment (Theory):
A significant concern arises from the implications of assumption 3.2. Based on the assumption, it can be inferred that given
$$\mathbb{P}\_{\phi}(x, s) = \mathbb{P}\_{\phi^*}(x, s),$$
there exists a transformation $T_1$ such that
$$\mathbb{P}\_{T\_1 \circ \phi}(x, z) = \mathbb{P}\_{\phi^*}(x, z).$$
However, when considering mixed uniform distributions over specific cubical sets, these assumptions are immediately invalidated. This limitation highlights potential scenarios where the stated assumption may not be practically applicable.

Author response:
**Q2: Potential Pitfalls**

A: If we understand the example correctly, we can consider 1-dimensional case, reviewer is saying that for (a) a mixture of uniform distribution over [0, 1] and [1, 2] with mixing weights 0.5 and 0.5, versus (b) a mixture of uniform distribution over [0, 0.5] and [0.5, 2] with mixing weights 0.25 and 0.75; two models have different joint distribution of $(x, z)$ but have the same marginal distribution on $x$, so our assumption is violated. However, we would argue that our assumption in this setting is violated for a good reason: as unsupervised pretraining only observes $x$, it has no ability to distinguish (a) model vs (b) model, that is, unsupervised pretraining will not be helpful in this situation.

Reviewer comment (Theory):
It would be beneficial for the authors to provide a more thorough justification or relaxation of assumption 3.2, possibly exploring weaker conditions that achieve similar results.

Author response:
**Q1: Over-reliance on Assumption 3.2**

A: While most prior works focused on specific applications, this paper aims to identify generic structural condition which allows unsupervised pretraining to be beneficial to learning downstream tasks. We view informative conditions (assumption 3.2) as one of the first steps towards understanding in general scenarios. We agree that assumption 3.2 is only a sufficient condition, and identifying weaker assumptions is an interesting future direction.

Reviewer comment (Theory):
How do the assumptions and theorems presented in the paper translate to real-world applications? Are there any practical scenarios where these findings could be directly applied?

Author response:
**Q3: Practical applications**

A: The results of this paper (a) can be directly applied to a number of practical applications; and (b) can be used to provide strong insights to certain advanced practical problems. For (a), for instance, the factor model example in this paper has been widely used in statistics, dealing with dependent measurements in real world applications including genomics, neuroscience, economics, finance, etc [1]. For (b), our setting directly fits the standard pretraining setup (e.g., in large language models) which uses a large amount of unlabeled data to pretrain a deep neural network as a representation, and then uses a small amount of labeled data to only fine-tunes the linear head for downstream tasks. This paper considers the MLE algorithm for pretraining, which provides insights to the success of many practical algorithms that use proxies for MLE (such as optimizing the variational lower bound, as in the case of variational autoencoder) due to computational constraints.

[1] Fan, J., Wang, K., Zhong, Y., & Zhu, Z. (2021). Robust high dimensional factor models with applications to statistical machine learning. Statistical science: a review journal of the Institute of Mathematical Statistics, 36(2), 303.

Reviewer comment (Experiments):
All of the examples are in relatively simple settings (linear functions, separated Gaussian mixtures). Is this due to the way the bounds scale with model/distribution complexity?

Author response:
We choose these representative examples to illustrate the core ideas and showcase the applicability of our generic framework. While our theoretical framework does apply to more advanced/complex applications, the computation of model complexity and informative condition coefficients of these applications can be quite involved and technical, which may qualify for a separate paper by themselves (we note for instance, there are many papers in the literature focusing on just computing the Rademacher complexity, or PAC bayes bound for certain architecture of neural networks). We leave those as exciting future works.

Reviewer comment (Theory):
Could you explain how you derive the excess risk for the baseline supervised algorithm?

Author response:
The excess risk bound for baseline supervised algorithm follows from standard risk bound (see, e.g., Section 4.1.2 and Theorem 4.10 in [1]) in supervised learning by plugging in $G_{\Phi,\Psi}$ (defined at the end of page 3 of our paper) as the function class.  

[1]Wainwright, Martin J. High-dimensional statistics: A non-asymptotic viewpoint. Vol. 48. Cambridge university press, 2019.

Reviewer comment (Theory):
The main assumption (Assumption 3.2) is not fully justified. The zero-informative counter-example is solid, but justifying the informative condition by showing "if some example is zero-informative, then the unsupervised pretraining could be useless" is a tautology. In short, the paper's conclusion of the superiority of unsupervised pretraining is built on the presumption of believing the pretraining to be useful.

Author response:
This paper aims to identify the structural condition which allows unsupervised pretraining to be beneficial to learning downstream tasks. Information theoretically speaking, pretraining is only helpful when pretraining tasks “correlates” with downstream tasks. In the setup of this paper, this means pretraining objectives must “correlate” with learning representation $z$ (since $z$ is the intermediate quantity connects pretraining to downstream). One contribution of this paper is to give a natural, mathematical quantification of this “correlation”. We did so using density estimation and TV distance, which gives rise to informative conditions (Assumption 3.2). We agree that Assumption 3.2 is only a sufficient condition — weaker conditions possibly exist. We view this as a first step towards identifying richer and more general conditions for this important problem in the future.

While we do not yet have formal lower bound on $\kappa$, we would like to argue that certain $\kappa$ dependency is necessary in risk bound $\kappa \sqrt{\frac{C_{\Phi}}{m}}+ \sqrt{\frac{C_{\Psi}}{n}}$ in worst cases. That is, the alternative risk bound $\sqrt{\frac{C_{\Phi}}{m}}+ \sqrt{\frac{C_{\Psi}}{n}}$ without $\kappa$ is not possible: we still consider the zero-informative example where $z=Ax$ + noise and $x$ is sampled from standard Gaussian with matrix $A$ unknown. In this case, the unsupervised pretraining step only observes the distribution of $x$, which information theoretically does not reveal anything about $A$ and representation $z$. We assume the downstream task is very simple — the map from $z$ to label $y$ is known (which can be realized by choosing a function class that only contains a single function; this makes $C_{\Psi} = 0$ ). We consider the case where the number of unlabeled data $m$ is extremely large and goes to infinity. Then the alternative risk bound would imply that even without any labeled data, we can still achieve 0 excess risk. We know this is not possible since we still need to learn $A$ from noisy labeled data.

Reviewer comment (Theory):
This paper shows three instances for its framework: factor models, Gaussian mixture models, and contrastive learning. But the last needs more careful handling since it is usually considered a self-supervised learning task.

Author response:
We view contrastive learning as a general class of approaches to learning that focuses on extracting meaningful representations by contrasting positive and negative pairs of instances. Our model captures this core idea, and fits this description. While we agree that our model would generate contrastive loss that is slightly different from the loss used in practice (such as in CLIP), such slight mismatch is not uncommon in theory work due to the requirement of rigorous math treatment — the difference in loss also exists for prior theoretical work for contrastive learning such as Haochen et al. and [1]. We believe that bridging this gap, and consider theory that involves practical contrastive loss is an important future direction.

[1] Saunshi, N., Plevrakis, O., Arora, S., Khodak, M. and Khandeparkar, H., 2019, May. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning (pp. 5628-5637). PMLR.

Reviewer comment (Theory):
The third example, contrastive learning (Section 6), seems to differ from the mainstream contrastive learning modeling. A common approach is to assume access to positive pairs $(x_i, x_i^+)$ from the same latent class and negative pairs $(x_i, x_i^-)$, while this paper models it in a somewhat "reverse" way: each sample of the pair $(x, x^\prime)$ is i.i.d. (which is often not satisfied in practice; see HaoChen et al. 2021), and the positive/negative signs are logistically distributed according to their latent similarity. I am not sure if this model is representative of a standard contrastive learning model, and it would be great if the authors could provide more explanations.

Author response:
We view contrastive learning as a general class of approaches to learning that focuses on extracting meaningful representations by contrasting positive and negative pairs of instances. Our model captures this core idea, and fits this description. While we agree that our model would generate contrastive loss that is slightly different from the loss used in practice (such as in CLIP), such slight mismatch is not uncommon in theory work due to the requirement of rigorous math treatment — the difference in loss also exists for prior theoretical work for contrastive learning such as Haochen et al. and [1]. We believe that bridging this gap, and consider theory that involves practical contrastive loss is an important future direction.

[1] Saunshi, N., Plevrakis, O., Arora, S., Khodak, M. and Khandeparkar, H., 2019, May. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning (pp. 5628-5637). PMLR.