Paper: Tight Rates in Supervised Outlier Transfer Learning (ICLR-2024)

Reviewer comment (Reproducibility):
The algorithm proposed in Section 4.8 requires as input the VC dimension of the hypothesis class. However, in practice, the exact VC dimension may be unknown. Could you please give some practical suggestions on using this algorithm when the exact VC dimension is unknown?

Author response:
We first note that the assumption of known VC is common in theoretical works and in fact we know very few results where such an assumption is not made. This is because, the exact VC or an upper-bound thereof is known for many of the common hypothesis classes encountered in the theoretical literature, e.g., linear to polynomial classes, classification trees, fixed architecture Neural networks, etc. 

However, we agree with the reviewer that in practice, we might not know the tightest upper-bound for a given choice of models, and theory works on uniform concentration then prescribes the use of notions such as Empirical Rademacher Complexity (ERC) which may be estimable in principle (but often not practical). All our upper bounds hold also under ERC (this is evident in our concentration arguments), and we will make sure to emphasize this.

Reviewer comment (Theory):
The notation in inequality (4.1) is a little redundant. Since $h_{S, \alpha}^*$ is a solution to the source problem, the difference between the expected error of any $h$ in the hypothesis class and of $h_{S, \alpha}^*$ w.r.t. the source distribution must be non-negative. So, there is no need to use the max function.

Author response:
It appears that the reviewer must have missed a nuance in the definition; the notation is not actually redundant: $h^*_{S,\alpha}$ is the best classifier among $\mathcal{H}_{\alpha} = \\{h\in \mathcal{H}: R_0(h)\leq \alpha\\}\$. 

However, in the definition, $h$ belongs to a larger class, namely $ \mathcal{H}_{\alpha+r} =\\{h\in \mathcal{H}: R_0(h)\leq \alpha+r\\}$. Hence, the difference of the risks can be negative.

Reviewer comment (Novelty):
The theoretical results mainly rely on the previous techniques.

Author response:
Neyman-Pearson classification, as discussed in the first part of the paper, is inherently different from traditional classification. Already, as argued in the first set of results, widely differing distributions may admit the same optimal classifiers (see Section 3). 
More generally, while some of the tools may appear similar, the analysis requires fundamentally different approaches. For instance a main difficulty arises from the fact that excess risks could be negative as explained above, due to the fact that a learner may operate on a different subset $\mathcal{H}_{a+r}$ of $\mathcal{H}$ since it can only estimate $\mathcal{H}_a$ from data.

Reviewer comment (Experiments):
There is no experiment.

Author response:
We refer the reviewer to our answer to a similar question by Reviewer 3fJQ (question 2). Namely, we do not claim to propose a new procedure. Rather the procedure is of a theoretical nature and mainly serves to establish that the lower-bound of Theorem 4.4 is attainable in principle, even without prior knowledge of distributional conditions. We re-emphasize that our main aim in this work is theoretical, and is to shed some needed insights on the limits of performance in outlier transfer.

Reviewer comment (Writing):
There are several typos, including: Page 4, in the 5th line in Section 3, the source and target problem are denoted by the same notation.

Author response:
Thanks for bringing the typos to our attention. We fixed them and uploaded a revised version.

Reviewer comment (Writing):
There are several typos, including: Page 4, in the last line, the LHS and RHS of the second to last inequality are the same.

Author response:
Thanks for bringing the typos to our attention. We fixed them and uploaded a revised version.

Reviewer comment (Writing):
There are several typos, including: Page 6, in the 5th line in Section 4.3, $n_S$ should be $n_T$.

Author response:
Thanks for bringing the typos to our attention. We fixed them and uploaded a revised version.

Reviewer comment (Presentation):
As far as I can tell, this paper does not actually follow the ICLR LaTeX template. For instance, the margins don't appear correct? Please fix this.

Author response:
Thank you for pointing this out; we had not noticed at first that one of the packages we had included unfortunately interacted with the ICLR margin. We have now corrected that and uploaded a revised version.

Reviewer comment (Experiments):
There are no numerical experiments. I think this paper would improve dramatically with experimental results, especially on real data, and especially on showing how well the adaptive method in Section 4.8 works in practice.

Author response:
The adaptive method is in fact of a theoretical nature and not practical at the moment since it relies on 0-1 risk minimization. It only serves to support our theoretical findings, namely that, even without knowledge of the discrepancy between distributions, it is possible in principle to achieve nearly the same optimal rate as an oracle. As a first theoretical work on the subject of outlier transfer, our aim is to yield such initial insights towards developing practical procedures with similar guarantees, e.g., by relying on properly designed surrogate losses for N-P classification. We do hope however that the reviewer will still appreciate the solid foundation we are laying towards such a practical goal. Our main aims at this point are theoretical---i.e., understanding achievable rates in this under-studied transfer setting---and we therefore never claimed to be proposing an algorithm at this point. We will do our best to clarify this further.

Reviewer comment (Novelty):
Detailed discussion of how applied researchers address this outlier transfer problem in practice would be helpful to provide some point of reference (even if these existing approaches lack guarantees): for instance, even getting a rough understanding of whether there are common conceptual ideas used would be helpful or if actually the methods are just completely different (if so, maybe some discussion of what the key conceptual differences are would be helpful).

Author response:
Various heuristics have been proposed so far, most of which boiling down to the simplest idea of combining both samples and hoping for the best; as such very little is understood at the moment, which motivates our work. We agree with you that such discussion will really enhance the paper, and we will follow your suggestion.

Reviewer comment (Theory):
If the size is large, will the results have special asymptotic properties?

Author response:
Thank you for the very nuanced question. Unfortunately, besides the fact that our finite-sample results imply consistency, in fact, strong consistency (since success probabilities are exponential), we did not consider other notions of asymptotic convergence, so it is unclear to us at this point whether any special asymptotic phenomena may arise.