Paper: Learning to Reject with a Fixed Predictor: Application to Decontextualization (ICLR-2024)

Reviewer comment (Experiments):
The experiment setting is not well explained and the settings are not comprehensive. As the authors mentioned, the LLM predictors are their main focus but in the experiments, there is only one type of LLM. As T5X is a family of models, there are other choices and more architectures from other families will be more comprehensive for evaluating the proposed loss.

Author response:
We kindly ask the reviewer to elaborate on which part of the experiment setting was not sufficiently well explained. While it is true that there generally exist many LLM architectures, it would not be feasible to evaluate all of them. By showing very strong results on T5X XXL, a standard model, we demonstrate that there is at least one architecture for which our surrogate loss is very useful.

Reviewer comment (Experiments):
To train the rejector still required to label new information, it is hard to distinguish whether the improvement is from the additional information or the loss function.

Author response:
Since the baseline using the cross-entropy loss had access to these new labels as well, the improvement is necessarily from the loss function.

Reviewer comment (Theory):
According to my understanding, applying the surrogate loss requires labelling the output from the given model. Then the model is further trained by the learned surrogate loss. Thus some extra information is introduced. Can the Cross entropy loss and Maxprob have the same information?

Author response:
The cross entropy baseline did have the same information - we finetuned a model using the cross-entropy loss *on the annotations*. However, it would not be possible for Maxprob to also benefit from this information because the annotations form a binary classification task, while the predictor (from which Maxprob is derived) is trained to generate decontexualizations.

Reviewer comment (Writing):
What is the format of the ejector?

Author response:
The format for the rejector is described in section 7.2 (surrogate loss), and it is trained on the decontextualization rejection task described in section 7.1. At a high level, its input is a sentence and a candidate decontextualization, and its output is a real number whose sign corresponds to acceptance or rejection.

Reviewer comment (Evaluation):
For the std comparison it is not very clear whether rejection loss is much better than Maxprob in Figure 4 and in Figure 5.

Author response:
We respectfully disagree with the reviewer. The curve for the rejection loss lies above that of Maxprob in both figures, and in Figure 4, it lies *significantly* above that of Maxprob.

Reviewer comment (Experiments):
The method is only applied to the decontextualization problem when it actually seems to have much more broad applicability. I can think of several LLM applications where the ability to increase precision by rejecting would be useful. For example, we could try to learn a rejector that cuts down on hallucinations in summarization or text simplification. More exploration of this technique beyond the decontextualization problem would make it more impactful.

Author response:
**Response:** As discussed in the conclusion, annotation for the LLM setting is expensive and this limits our empirical evaluation. We therefore provided additional experimental evaluation on image classification tasks, where checking the correctness of the predictor is easy. We leave further experimental evaluation for future work, but observe very strong results on our current LLM experiments.

Reviewer comment (Theory):
Additionally, to the best of our knowledge, minimizing the cross-entropy loss does not have any proven guarantee with respect to our main objective: minimizing the induced rejection loss.

Author response:
**Response:** Our surrogate losses benefit from $\mathcal{R}$-consistency bounds with the hypothesis set $\mathcal{R}$ extending beyond the family of all measurable functions $\mathcal{R}_{\mathrm{all}}$. This includes, for example, the family of linear models and one-hidden-layer neural networks considered by Awasthi et al. (2022). For simplicity, we presented results only for the family of all measurable functions. However, our proofs and results can be directly extended to other hypothesis sets. In the final version, we will include these $\mathcal{R}$-consistency bounds.

While the cost-sensitive surrogate loss benefits from Bayes-consistency, it does not benefit from $\mathcal{R}$-consistency bounds. As pointed out by Awasthi et al. (2022), Bayes-consistency does not supply any information about learning with a typically restricted hypothesis set, which of course would not contain all measurable functions. Furthermore, it provides no guarantee for approximate minimizers since the rate of that asymptotic convergence could be arbitrarily slow.

Reviewer comment (Theory):
When $\mathcal{R} = \mathcal{R}\_{all}$, can't we use a cost-sensitive surrogate loss plus usual (Bayes) consistency results? I.e., I don't understand the following claim in Appendix C given that the paper's results only consider $\mathcal{R} = \mathcal{R}\_{all}$: (i) There is a lack of any H-consistency bound guarantees for cost-sensitive surrogate losses with respect to the induced rejection loss.

Author response:
We have uploaded a revised version with a newly added Appendix D. The main results (on p.18), Theorem 7 and Corollary 8, demonstrate that our surrogate losses benefit from $\mathcal{R}$-consistency bounds with any bounded hypothesis set, including the family of linear models and one-hidden-layer ReLU networks considered in (Awasthi et al., 2022) as special cases. In the initial version, we mentioned before Section 5.5 "Similar results can be derived for other families of functions $\mathcal{R}$, such as that of linear functions or neural networks with one hidden-layer as in (Awasthi et al., 2022)", but did not include them for simplicity. We believe these newly added results will further strengthen the theoretical part of the paper and address the reviewer's concerns.

Note that there is a lack of similar guarantees for cost-sensitive surrogate losses with respect to the induced rejection loss. Furthermore, our surrogate loss outperforms the cost-sensitive baseline in our image classification experiments.

Reviewer comment (Experiments):
No ablation on $\alpha$ even though it's important in the bound (Thm 5), and different experiments use different values (e.g., 4 in main text experiments, 3.5 in appendix vision experiments).

Author response:
**Response:** In our experiments on image classification, the value of $\alpha$ is selected using cross-validation, a method typically applicable in practice. In our future work, we aim to explore a more principled approach for choosing $\alpha$. A promising direction involves establishing data-dependent bounds, which would act as a useful guide for selecting $\alpha$ across different datasets.

Reviewer comment (Novelty):
Why do we care about $\mathcal{H}$-consistency even though we only consider the space $\mathcal{R}_{all}$ of all measurable functions? It would be helpful to further emphasize this earlier in the paper as a major difference / novelty / reason why more naive baselines don't work (see earlier comment under weaknesses).

Author response:
**Response:** Our surrogate losses benefit from $\mathcal{R}$-consistency bounds with the hypothesis set $\mathcal{R}$ extending beyond the family of all measurable functions $\mathcal{R}_{\mathrm{all}}$. This includes, for example, the family of linear models and one-hidden-layer neural networks considered by Awasthi et al. (2022). For simplicity, we presented results only for the family of all measurable functions. However, our proofs and results can be directly extended to other hypothesis sets. In the final version, we will include these $\mathcal{R}$-consistency bounds.

While the cost-sensitive surrogate loss benefits from Bayes-consistency, it does not benefit from $\mathcal{R}$-consistency bounds. As pointed out by Awasthi et al. (2022), Bayes-consistency does not supply any information about learning with a typically restricted hypothesis set, which of course would not contain all measurable functions. Furthermore, it provides no guarantee for approximate minimizers since the rate of that asymptotic convergence could be arbitrarily slow.

Reviewer comment (Writing):
Small typo in display at bottom of pg4; $\le 0$ should be in the subscript of $\mathbb{I}$.

Author response:
**Response:** Thank you, we will fix this in the final version.

Reviewer comment (Writing):
Small typo in display at bottom of pg4; not enough parentheses.

Author response:
**Response:** Thank you, we will fix this in the final version.

Reviewer comment (Writing):
Minor semantics/notation: $r(x,f(x))$ reads like a score for rejecting, so the notation reads like $r>0 \implies$ reject, when it's actually being used in the opposite sense. This tripped me up several times. I know this notation is inherited from Cortes et al. but I think it's a bit confusing.

Author response:
**Response:** We are indeed staying consistent with the notation from the literature.

Reviewer comment (Reproducibility):
Although GitHub repo links and other identifying information cannot be written in a paper under review, I did not see any indication of the intention to make the code public, nor is it provided in supplementary materials.

Author response:
We are hoping to release code for the final version and are waiting for approval.

Reviewer comment (Theory):
Page 4. First equation/inequality continued. The bound becomes a lower bound, not an upper bound.

Author response:
As pointed out by reviewer KRjg, the first inequality indeed holds. Thus, we are still giving an upper bound. We will replace the second inequality with equality (we were simply staying consistent with Cortes et al.). The last relation in the derivation holds because a max of two positive real numbers is upper-bounded by their sum.

Reviewer comment (Theory):
Page 4. First equation/inequality continued. The last relation was not checked. That might still hold despite this, but the reasoning needs to be clarified.

Author response:
As pointed out by reviewer KRjg, the first inequality indeed holds. Thus, we are still giving an upper bound. We will replace the second inequality with equality (we were simply staying consistent with Cortes et al.). The last relation in the derivation holds because a max of two positive real numbers is upper-bounded by their sum.

Reviewer comment (Experiments):
There is no test set. Only train and validation, where cross-validation is used so the training algorithm sees all data.

Author response:
We perform k-fold cross validation, and thus the training algorithm does not see the data it is tested on. Additionally, in the decontextualization experiment with LLMs, we do not tune any hyper-parameters for our proposed method.

Reviewer comment (Evaluation):
Page 5. It is said that "underlying scores are not favourable for that precision level". Why is that?

Author response:
We would like to point out that we have instead written “not *necessarily* favorable.” This is because the same scores are used for each precision level; the learning algorithm was not trained to target a specific precision level. On the other hand, in our surrogate loss, we are targeting a specific precision level through the value of $c$.

Reviewer comment (Evaluation):
These are possibly standard deviation bars in figures 4 and 5. How were they generated? Is it from different folds of cross-validation?

Author response:
As described in their respective captions, the standard deviations in Figure 4 are from the different cross-validation splits, and the standard deviations in Figure 5 are from different training runs.

Reviewer comment (Experiments):
Are Maxprob and cross-entropy trained on different models? Why is that?

Author response:
These are very slightly different - Maxprob was derived from an existing decontextualization model, and we chose to use T5X for all of our fine-tuning as it was the closest model available to us.

Reviewer comment (Presentation):
Page 4. Last equation "<=0" should be in the subscript.

Author response:
Thank you, we will fix these in the final version.

Reviewer comment (Presentation):
Please recheck format for citations: some citations use "et al." while others list all authors.

Author response:
Thank you, we will fix these in the final version.