Paper: Towards the Fundamental Limits of Knowledge Transfer over Finite Domains (ICLR-2024)

Reviewer comment (Novelty):
The setting of "knowledge transfer" seems to be fancy, but according to the definition, the teacher distribution is considered as "ground truth" that is already given, thus the "hard labels" regime seems to fall in the common statistical learning framework. The reviewer is not sure whether tabular settings under the usual statistical learning framework have been studied.

Author response:
- As mentioned at the end of our Section 1.1, previous works [1, 2] similar to our $\mathsf{H}\text{ard }\mathsf{L}\text{abels}$ setting consider the teacher to be also an estimator that learning from samples generated via the Bayes (ground truth) probability. [1, 2] also largely fall into the ERM (empirical risk minimization) framework. **In contrast**, motivated by modern (proprietary) language (teacher) models, we treat the teacher model to be the Bayes (ground truth) probability and our target of interest is just a kind of symmetric *closeness metric* between the student and the teacher, and thus is different from the ERM-related ones that mainly concern *generalization bounds* or the comparison between the generalization bound of the student and that of the teacher.

- The term "knowledge transfer" we considered is to transfer any information from the teacher classifier to the student classifier as we mentioned in our Introduction section, any portion of $\{\pi^\star(\cdot|s_i)\}_{i=1}^n$ is considered as privileged (i.e., additional) information.

- At the very beginning of Section 3 (for $\mathsf{H}\text{ard }\mathsf{L}\text{abels}$), we already emphasize that 
  
  > "This is equivalent to the standard setting for estimating the conditional density $\pi^\star(\cdot|\cdot)$"

Reviewer comment (Theory):
The bounds pertaining to other scenarios discussed in the paper can also be derived using mathematical methods similar to the ones mentioned above.

Author response:
Let us respectfully emphasize some key points to accurately deliver the messages as follows.

- The learner (student model) does not learn $\rho\times\pi^\star$, and only aim to learn $\pi^\star(\cdot|\cdot)$, which is a **discrete** and **conditional** density. $\rho\times\pi^\star$ is purely a probability distribution indexed by $(s, a)$ pairs, and learning a good $\widehat{\rho\times\pi^\star}$ **can not** imply learning a good $\widehat{\pi^\star}$.

- Thus, the target of interest is the conditional total variation $\mathsf{TV}(\hat\pi, \pi^\star|\rho)$, while as we mentioned in the paper, an uniform $\rho$ is indeed good to provide some intuitions about the worst-case upper bound **in expectation** in Theorem 3.2.

- We respectfully disagree with the reviewer on our contributions to understanding the method efficiency of **the later two** settings. When certain portion of $\{\pi^\star(\cdot|s_i)\}_{i=1}^n$ is provided to the student, vanilla concentration bounds concerning only samples (such as Hoeffding's inequality, the Angluin-Valiant bound, Bernstein's inequality, Bennett's inequality, or McDiarmid's theorem) are **no longer applicable**, so our novel technical roadmap is instead inspired by *the Missing Mass Analysis* [1].

- We also respectfully disagree with the reviewer on our contributions to understanding the problem complexity of **each** setting. The three information-theoretic lower bounds **can not** be easily derived from standard concentration bounds. For the first case, the constructive proof of Theorem 3.1 is derived from Assouad's hypercube reduction. For the later two cases, Theorem 4.1 and Theorem 5.1 **can not** even be derived via standard reductions like Le Cam's, Assound's, or Fano's [2], because these three standard methods can only tackle the case with no ground truth leakage, i.e., they can only deal with estimators of $\pi^\star$  that learn from purely samples from $\rho\times\pi^\star$. In contrast, our proofs of Theorem 4.1 and Theorem 5.1 are based on two different and explicit constructions of Bayes priors over $\rho\times\pi^\star$ and the reduction from bounding minimax risk to bounding worse-case Bayes risk from below [3]. Thus, all the three constructive proofs for Theorem 3.1, Theorem 4.1, and Theorem 5.1 , are not derived from standard concentration bounds, and are **different** from each other, even from an intuitive perspective.

- Also, we want to mention an important contribution related to **instance-dependent**/benign-case analysis in Theorem 3.2: the complexity measure $\xi(\pi^\star)$, which provably explains the $O(1/n)$ rate of $\mathsf{CE_{sgl}}$ rather than $O(1/\sqrt{n})$ **in expectation** when $\xi(\pi^\star)\asymp n^{-1}\text{poly}(|\mathcal S|, |\mathcal A|)$.

- Another contribution is that the while the worst-case **high-probability** upper bound in Theorem 3.2 has $\text{poly}\log(|\mathcal S|, |\mathcal A|)$ factors, the worst-case **upper bound** in expectation in Theorem 3.2 **does not**.

Reviewer comment (Theory):
It would be more intriguing if the paper explored more intricate scenarios for $\mathcal{S}$ and $\mathcal{A}$. For example, one would naturally expect other (and more interesting) notions of complexity (such as VC-dimension, etc.) of distribution families $\rho$ and $\pi^*$ to appear in the bounds, instead of relying solely on the mere cardinality of the query-answer spaces.

Author response:
**A2**: We answer this question in two fold. First, the rationale behind considering **discrete** $\rho$ and $\pi^\star$ is that modern language models are often classifiers of **tokens**, with tokens or a collection of tokens as input. Second, we have added some discussions and conjectures (for log-linear or general softmax $\pi^\star$) on function approximation in the face of large state space in Appendix H (highlighted in blue), which builds a quantitative relation between the $\ell_2$-distance of model parameters and the conditional total variation between models, and leave the complete characterization as future work.

Reviewer comment (Experiments):
The tabular setting is quite basic which does not capture many practical settings when the state space is large.

Author response:
Thank you for your appreciation and we have added some discussions and conjectures (for log-linear or general Softmax $\pi^\star$) on function approximation in the face of large $\mathcal S$ in Appendix H (highlighted in blue), which builds a quantitative relation between the $\ell_2$-distance of model parameters and the conditional total variation between models.

Reviewer comment (Theory):
How can these guarantees generalize to more practical settings where the state space is large? (speculation is fine)

Author response:
Thank you for your appreciation and we have added some discussions and conjectures (for log-linear or general Softmax $\pi^\star$) on function approximation in the face of large $\mathcal S$ in Appendix H (highlighted in blue), which builds a quantitative relation between the $\ell_2$-distance of model parameters and the conditional total variation between models.