Paper: Towards a statistical theory of data selection under weak supervision (ICLR-2024)

Reviewer comment (Theory):
Theorem 1 asserts that $\rho_{\mathrm{unb}}/\rho_{\mathrm{nr}}$ can grow arbitrarily large by selecting the feature vectors' distribution in an adversarial manner. However, can the same be said for $\rho_{\mathrm{nr}}/\rho_{\mathrm{unb}}$? What happens when the feature vectors' distribution is more generic, such as Gaussian? Without additional guarantees in these respects, the theorem may lack substantial significance.

Author response:
**Theorem 1.** *This theorem is not about adversarial examples*, but random i.i.d. ones (“there exists a distribution”). Further, by inspection of the proof (Example G.2), the distribution is very simple and natural (as ‘natural’ as the Gaussian distribution). We simply take feature vectors $x_i$ that are generated as $x_i=r \cdot u_i$, where $u_i$ is uniform on the sphere, and $r$ is a scalar that is independent of $u_i$ and has a power law tail. Gaussian with identity covariance does not work for this example because (informally) 'all Gaussian vectors look the same'. In a sense, this is what makes Gaussian data an unrealistic model in many applications: they have way less variability than real data.

Reviewer comment (Writing):
In Theorem 2: "an non-empty" -> "a non-empty".

Author response:
We also fixed the typographical error pointed by the reviewer in the updated version. Thank you for catching it.

Reviewer comment (Theory):
**Why $\Omega:\mathbb{R}^p\rightarrow \mathbb{R}$**: In other words, why the dimensionality of parameter $\theta$ has been assumed to be the same as that of the input vector $\boldsymbol{x}$. Obviously, this assumption makes sense when using a linear model. However, does it alter the generality of the presented framework in any shape or form?

Author response:
**Dimensionality of the parameter.** The theory of Section 4 accommodates for a number of parameters (indicated by $p$) different from the number of input dimensions ($d$). The theory of Section 5 assumes a generalized linear model, and therefore $p=d$.

Reviewer comment (Experiments):
**In Figure 1, how do you justify the reduction in estimation error after subsampling?** The way I have understood the experiment: "Full data" curve uses all the 34345 data samples with **true** labels. On the other hand, the proposed scheme (in the case of a weak surrogate model)  
1) First, uses only 1472 samples (with true labels) from another fraction of the dataset in order to train a surrogate model.  
2) You use the above-mentioned surrogate model to select, say, 50% of the dataset (without seeing their true labels, right?), and then  
3) The training procedure considers the true labels, only for the above-mentioned selected samples, and uses them for training the main model.  

At the end, the main model outperforms the "Full data" curve. Is it because the surrogate model has been trained too good? Can you please also report the estimation error of the weak and strong surrogate models? Also, it should be noted that Theorem 2 only shows that there "exists" cases for which $\rho_{\mathrm{nr}}$ is not monotonic. But it does not prove error can become lower than that of a full data ERM.

Author response:
**Explaining Figure 1.** The experiment settings of the Figure 1 that you described are correct. The test accuracy of the weak (strong) surrogate model is 87% (91%). Weaker models can perform better than stronger ones, depending on $N/p$, see Figure 9 Appendix O. 
We agree that these results are surprising. Indeed, we were motivated to work out the theory in order to demystify these results. In summary:

- The high-dimensional theory of Section 5 and Appendix L captures this phenomenon. See in particular Figures 2 and 7, where (in some of the settings) the same phenomenon is observed both under misspecification and imperfect surrogate estimation.
- Theorem 2 also proves the same phenomenon in a low-dimensional setting. (In this case the surrogate is assumed to be perfect, but it is elementary to see that using parameters estimated by a number of samples much smaller than $d$ yields the same result).
- The reason for this is that the intuition “every sample is useful” is only correct if the data distribution is perfectly matched to the loss function (e.g. logistic and logistic). As soon as this does not hold (which is the case in most real data applications), not all datapoints improve the estimation error. 

**Theorem 2.** The theorem states that the error is strictly increasing for gamma in the interval $(\gamma_0,1)$. In particular, this means that the error at $\gamma=\gamma_0$ is strictly smaller than the error at $\gamma=1$. Recalling the definition of $\gamma$, the theorem proves that the (test) error achieved by training on subsampled data is strictly smaller than the error on the full sample ERM. We also note that:

- For brevity, we state the theorem in the form “there exists.” However, the proof is constructive and provides data distributions for which data selection improves over full sample ERM.
- As mentioned above, the same claims can be shown to hold if the surrogate model is trained on a vanishing fraction of the n samples for which labels are provided.

Reviewer comment (Theory):
**Assumption A.1 (in Proposition B.1)**: I am a little confused here... In Proposition 4.1, do we have to assume that A.1 (of Proposition B.1) holds? Or the subsampling strategy that is guaranteed to minimize $\rho_{\mathrm{unb}}$ automatically satisfies this condition? (i.e., forcing the minimizers of $R_S$ and $R$ to coincide with each other)

Author response:
**Proposition 4.1.** First of all, we apologize for the somewhat awkward formulation here. We were led to this by the attempt to defer to appendices material that is pedagogically useful, but is not novel.
 
Second, if the assumption is not satisfied, then the subsampling scheme is (roughly speaking) ‘much worse’ than (say) random subsampling. (Technically, the procedure is not consistent.) In other words, we could write a longer form of this proposition without this assumption, and establish that schemes that do not satisfy it are ‘very bad.’

Finally, under many models, satisfying these assumptions is fairly easy.

Reviewer comment (Writing):
**What (or should I say: Where) is $b(x)$ in Proposition 4.3**?

Author response:
**Proposition 4.3.** Thanks, this is a typographical error, please see the updated version.

Reviewer comment (Writing):
Page 1 second paragraph "close to $n$": $n$ is not defined at this point. It is a bit weird to say close to $n$.

Author response:
**Close to $n$.** Here $n$ is the target  subsample size. However, since we study probabilistic data selection methods as well, we allow for the subsampled size to have small fluctuations around $n$. (We could of course eliminate these fluctuations, at the cost of making the whole paper clumsier.)

Reviewer comment (Writing):
In (1.3): Is $\ell_{\text{test}}$ a new loss function? Or should $\ell_{\text{test}}$ be $\ell$?

Author response:
**$\ell_{test}$.**  This is the loss used for testing. This needs not to be the same as for training (eg cross-entropy for training and classification for testing).

Reviewer comment (Evaluation):
How does the provided results in semi-supervised learning (where one has a small labeled dataset and many unlabeled examples but uses both for training) compare with the setting of the paper?

Author response:
**Semi-supervised learning.** Semi-supervised learning addresses the same problem as the one in our paper: learning under a limited ‘labeling budget.’ However a formal comparison is at the moment difficult because of the differences between the two settings. In semi-supervised learning, the subset of labeled examples is given (not chosen algorithmically) and the learner tries to make the best use of unlabeled examples.

In our setting, we are allowed to choose which examples to label, but once this is done, learning is carried out uniquely on the labeled examples. The two models are both interesting, but motivated by different use cases.

On the other hand, we believe it should be possible (and interesting) to define a unified setting that captures both scenarios. We expect that in this setting, a mixture of the two strategies will be required.

Reviewer comment (Evaluation):
The improvement in the ERM generalization using a well-selected subsample holds even for imperfect surrogate models (from Figure 1). Is there a result that compares the weakness of the surrogate model with the improvement (for some specific examples)?

Author response:
**Imperfect surrogate models.** Empirically, Figure 9 in Appendix O carries out a comparison of surrogate models of different strength.
Mathematically, formalizing what 'optimality' means in the context of imperfect surrogates is far from obvious. This requires formalizing the notion of uncertainty about the true model.
We provides two types of rigorous results for imperfect surrogates:
- *Low-dimensional.* This is mainly contained in Appendix K and summarized in Section 4.5. In particular, we prove that using a weaker model can yield better data selection (in minimax sense) than using the perfect surrogate (See Theorem 7 and Example K.2).
- *High-dimensional.* This is mainly contained in Appendix L and summarized in Section 5. In this case we fix an imperfect surrogate and associated data selection procedure, and characterize the resulting generalization error of data learned on data selected by that procedure (See Theorems 4 and 8). We study the effect of surrogate model misspecification (Figure 2) and estimation error in the surrogate model (Figure 7).

Reviewer comment (Theory):
There appears to be a dependency of some calculated values on the value of optimal parameters that are to be estimated (\theta^*) starting from the equation 4.2. It was unclear for me whether we can use estimates of such parameters and how correct would be the final results when the assumptions are violated or some values are replaced with estimates (in case if closed form solutions are unavailable).

Author response:
**Dependency on estimated parameters.** Indeed, some of our estimates depend on unobserved parameters (as pointed out in Remark C.1). However, empirical versions of these quantities can be plugged in. Under the stated assumptions, corrections due to this preplacement are negligible. We refrain from making this explicit at each passage uniquely to avoid further notational burden.

Reviewer comment (Theory):
To clarify the assumptions, how applicable is the method to models with multiple local minimas? (non-convex losses and models)

Author response:
**Non-convexity and related questions.** We present two sets of results:
- *Low-dimensional asymptotics*: These are not limited to convex methods, but they do assume that the global optimizer is used for estimation.
- *High-dimensional asymptotics*: these assume a (generalized) linear model, and convex loss.

The referee raises an interesting question, which we rephrase as: 

*"What is the impact of data selection on the global optimization landscape in non-convex settings?"*

This is a very difficult question and indeed more basic questions are widely open in non-convex statistical settings. A few remarks:
- If subsampling is unbiased then we expect that uniform convergence results for the landscape [e.g. Mei, Bai, Montanari Ann. Stat. 2018] can be applied to show that the whole landscape under subsampling is 'as good as' the original one.
- For biased subsampling, the same uniform convergence results will imply that the landscape can be characterized in terms of a certain 'modified population landscape'. The latter can be 'worse' or 'better' than the original landscape, and which one will depend on a case-by-case analysis. 
- For models outside the uniform convergence regime (e.g. highly overparameterized models) the above arguments will not apply, but uniform convergence of the landscape is often too strong of a requirement, and convergence in a neighborhood of the optimization trajectory is sufficient.

Overall, we think there is some promise for extending the present results to capture the effects of data selection on non-convex optimization.

Reviewer comment (Theory):
Related to the previous question, how applicable is the method to various non-linear models? For example, any model that contains non-trivial non-linearity, such as a two-layer network?

Author response:
**Non-convexity and related questions.** We present two sets of results:
- *Low-dimensional asymptotics*: These are not limited to convex methods, but they do assume that the global optimizer is used for estimation.
- *High-dimensional asymptotics*: these assume a (generalized) linear model, and convex loss.

The referee raises an interesting question, which we rephrase as: 

*"What is the impact of data selection on the global optimization landscape in non-convex settings?"*

This is a very difficult question and indeed more basic questions are widely open in non-convex statistical settings. A few remarks:
- If subsampling is unbiased then we expect that uniform convergence results for the landscape [e.g. Mei, Bai, Montanari Ann. Stat. 2018] can be applied to show that the whole landscape under subsampling is 'as good as' the original one.
- For biased subsampling, the same uniform convergence results will imply that the landscape can be characterized in terms of a certain 'modified population landscape'. The latter can be 'worse' or 'better' than the original landscape, and which one will depend on a case-by-case analysis. 
- For models outside the uniform convergence regime (e.g. highly overparameterized models) the above arguments will not apply, but uniform convergence of the landscape is often too strong of a requirement, and convergence in a neighborhood of the optimization trajectory is sufficient.

Overall, we think there is some promise for extending the present results to capture the effects of data selection on non-convex optimization.

Reviewer comment (Theory):
Is it possible to use gradient-based and similar approximate procedures that require low subsampling error over the whole parameters space to converge (difference between Loss under subsampled data and full data to be small not only at the optimal values of parameters)?

Author response:
**Non-convexity and related questions.** We present two sets of results:
- *Low-dimensional asymptotics*: These are not limited to convex methods, but they do assume that the global optimizer is used for estimation.
- *High-dimensional asymptotics*: these assume a (generalized) linear model, and convex loss.

The referee raises an interesting question, which we rephrase as: 

*"What is the impact of data selection on the global optimization landscape in non-convex settings?"*

This is a very difficult question and indeed more basic questions are widely open in non-convex statistical settings. A few remarks:
- If subsampling is unbiased then we expect that uniform convergence results for the landscape [e.g. Mei, Bai, Montanari Ann. Stat. 2018] can be applied to show that the whole landscape under subsampling is 'as good as' the original one.
- For biased subsampling, the same uniform convergence results will imply that the landscape can be characterized in terms of a certain 'modified population landscape'. The latter can be 'worse' or 'better' than the original landscape, and which one will depend on a case-by-case analysis. 
- For models outside the uniform convergence regime (e.g. highly overparameterized models) the above arguments will not apply, but uniform convergence of the landscape is often too strong of a requirement, and convergence in a neighborhood of the optimization trajectory is sufficient.

Overall, we think there is some promise for extending the present results to capture the effects of data selection on non-convex optimization.

Reviewer comment (Experiments):
There is an extensive theory of data selection in the case when labels are known (in that case there are methods that guarantee low multiplicative or additive error over whole parameters space, far from optimum, when comparing full and subsampled datasets); would it be possible to evaluate the proposed model against such prior art, for example, considering a "perfect oracle" that exactly predicts the labels in the proposed setting?

Author response:
**Known labels.** Our experience is that our weakly supervised approach compares well with data selection methods that make use of the actual labels. If the reviewer has in mind a specific algorithm of this type that would be meaningful to use as a benchmark, we would be interested in carrying out a comparison.

R2: Thanks for the concrete suggestions. We agree that this is an interesting comparison.

We carried out **preliminary simulations** in the setting of Figure 1 using the base algorithm of [Munteanu et al. 2018]. (We modified their method minimally, to allow for varying subsampling rates.)

The results are reported in the Figure 1 in the newly submitted supplementary material file (titled: "Towards a Statistical Theory of Data Selection Under Weak Supervision: Response to Reviewers"): we emphasize that these are preliminary results and we need to carry out a more careful study before including this comparison in the paper.

Below we reproduce the text from the response:
In our simulations, the method of [Munteanu et al. 2018] does not behave better than random subsampling. Notice that the simulations of that paper are all carried out in a very low-dimensional (underparametrized) regime in which the sample size $n$ is much larger than the number of parameters $p$: in all of their examples $n>2000\times p$. In contrast, we work in a higher dimensional setting, in which the model is either moderately underparametrized, or is overparametrized.
We also point out that, the coreset approach presents a fundamental limitation. Indeed, it aims at approximating as well as possible the full sample empirical risk minimization problem. As a consequence, it will never outperform the full sample ERM, while our approach does.