Paper: Out-Of-Domain Unlabeled Data Improves Generalization (ICLR-2024)

Reviewer comment (Novelty):
The comparison with related works isn't thorough in the sense that the paper mentions these related works but doesn't provide any comparison of their current work with it. For e.g. the paper doesn't mention that the works of Carlini et al, Carmon et. al. etc. were for the adversarial robustness setting. It would make the contributions stand out more clearly if there is precise comparison as to how earlier work is different from the current.

Author response:
**The comparison with related works isn't thorough**: In the final version, we will incorporate additional explanations and provide a more detailed comparison with existing works to ensure that our contribution is distinctly highlighted.

Reviewer comment (Evaluation):
There are multiple anomalies that are unexplained in the experiments. Perhaps addressing that would make the results less surprising. Few examples: i) In the simulated data, it would helpful to know roughly the least number of labeled samples required to obtain the same accuracy as the optimal. ii) In the real dataset case, the different distribution case achieves higher accuracy for fewer labeled samples. It is not clear why this column shouldn't be the same or worse as the same distribution case since it is using only the labeled samples that are drawn both from the NCT-CRC-HE-100K dataset?

Author response:
**There are multiple anomalies that are unexplained in the experiments**: We appreciate the reviewer for highlighting this issue. In our final version, we will conduct additional experiments and provide a detailed discussion of these anomalies. Regarding the two examples mentioned by the reviewer: i) We will include these values in our revised version. ii) The purpose of the table was not to compare the two settings where unlabeled samples came from the same or different distributions, but rather to illustrate the impact of using more unlabeled samples in each case. Therefore, we used different test sets and search spaces for hyperparameter search in these two settings, resulting in different accuracies in these two parts. However, we recognize the need for clarity and will regenerate the table, accompanied by an explanation comparing these two settings in our final version.

Reviewer comment (Experiments):
It should be possible to empirically verify the dimension dependence?

Author response:
**It should be possible to empirically verify the dimension dependence?**: Yes, we will add some experiments regarding the relationship between the number of samples and dimension in our final version.

Reviewer comment (Writing):
On Pg5, what is "crowded areas"? This notion makes an appearance in the conclusion as well, but there is no explanation about what this is?

Author response:
**What is "crowded areas"?**: By "crowded areas," we are referring to locations where the probability density function exhibits higher values. Our constraints, based on Distributionally Robust Optimization (DRO) and relying solely on unlabeled data, inherently force the classifier boundaries to maintain a distance from these identified areas.

Reviewer comment (Theory):
In Theorem 4.1, why is the expectation only w.r.t P_0 since we also get unlabeled samples from P_1. If this is included in the high probability statement, then that seems strange because I would assume the algorithm randomness to be separately considered from the unlabeled dataset randomness.

Author response:
**In Theorem $4.1$, why is the expectation only w.r.t $P_0$ since we also get unlabeled samples from $P_1$. If this is included in the high probability statement, then that seems strange because I would assume the algorithm randomness to be separately considered from the unlabeled dataset randomness.**: The expectation in Theorem 4.1 is solely with respect to $P_0$, as these terms constitute the definition of generalization risk within the distribution for which we aim to develop a proficient classifier. Although we utilize labeled samples from $P_0$ and unlabeled samples from $P_1$ to construct a robust classifier, the stochastic nature associated with these samples is reflected in the upper bound provided for the Rademacher complexity and the high probability statement.

Reviewer comment (Theory):
Is there any intuition for why the error in robust loss increases with \gamma?

Author response:
**Is there any intuition for why the error in robust loss increases with $\gamma$?**: The parameter $\gamma$ exhibits an inverse relationship with the robustness of the classifier, indicating that a lower value of $\gamma$ leads to a more robust classifier with reduced risk. Additionally, it is a well-established fact that a more robust classifier tends to have lower variance compared to a less robust one. Consequently, a higher value of $\gamma$ corresponds to increased variance and a less favorable upper bound.

Reviewer comment (Writing):
Minor comment: I found the notation in (1), \hat{R}(\theta, S) confusing and also not used later (unless in the proofs). Isn't this just R(\theta, \hat{P}^{m}_S)?

Author response:
**Minor comment: I found the notation in (1), $\hat{R}(\theta, S)$ confusing and also not used later (unless in the proofs). Isn't this just $R(\theta, \hat{P}^{m}_S)$ ?**: The reviewer's observation is accurate. We will rectify this issue to eliminate any potential confusion.

Reviewer comment (Theory):
The paper's linear Gaussian mixture model is very restrictive.

Author response:
It is crucial to emphasize that the algorithm and methodology outlined in the paper do not rely on specific assumptions about the distributions of the data, and consequently are NOT confined to a Gaussian mixture model. Our approach is inherently general. While conducting a theoretical analysis in the paper, we chose to assess the algorithm's performance within the framework of a Gaussian mixture model for two primary reasons: i) this assumption is prevalent in the existing literature [p1][p2][p3], thereby facilitating comparisons between our theoretical advancements and prior findings (a point also underscored by Reviewer v7JJ). Additionally, ii) employing a Gaussian mixture model allows us to derive mathematically explicit bounds, enabling us to elucidate the potentially surprising aspects of our findings.

[p1] Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K. and Madry, A., 2018. Adversarially robust generalization requires more data. Advances in neural information processing systems, 31.

[p2] Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, J.C. and Liang, P.S., 2019. Unlabeled data improves adversarial robustness. Advances in neural information processing systems, 32.

[p3] Alayrac, J.B., Uesato, J., Huang, P.S., Fawzi, A., Stanforth, R. and Kohli, P., 2019. Are labels required for improving adversarial robustness?. Advances in Neural Information Processing Systems, 32.

Reviewer comment (Theory):
When n=0, no out-of-domain samples are utilized and the problem reduces to simple ERM. But in Theorem 4.2, when n=0, the dependence of the error on dimension is d^{3/8}, meaning that this reduction in the exponent of the dimension is not related to the utilization of out-of-domain samples. Furthermore, in the case of n=0, why is the non-robust error better than error obtained through ERM?

Author response:
Let us clarify this issue. When we do not use any unlabeled samples (i.e., $n=0$), our bound becomes $\mathcal{O}\left((\frac{d}{m})^{3/8}\right)$, which is actually worse than the upper-bound of ERM ($^*$), $\mathcal{O}\left((\frac{d}{m})^{1/2}\right)$, and thus the ERM algorithm gives a better generalization bound. It's also important to highlight that these bounds are meaningful only when $\frac{d}{m} \leq 1$. Additionally, in the presence of unlabeled samples, there is no requirement for $m$ to be greater than $d$ in our method. The upper-bound then becomes $\mathcal{O}\left(\left(\frac{d}{m + n}\right)^{3/8}\right)$ , and it tends to zero as $n$ approaches infinity. We will further elucidate this matter in the final version to avoid any ambiguity. Specifically, we acknowledge that Remark 4.3 may have contributed to this misunderstanding. Therefore, we are committed to revising its statement in the final version to ensure clarity.

(*) The performance gap between our method and ERM when $n=0$ can be effectively minimized to zero by intentionally diminishing the impact of the constraint (reducing $\lambda$ in our primary optimization criteria) when $n$ is small. However, it is important to note that the primary emphasis of this study is on extreme regimes, particularly when $n\gg m$. Consequently, we have chosen to omit these secondary derivations to enhance the overall readability of the paper.