Paper: Selective Prediction via Training Dynamics (ICLR-2024)

Reviewer comment (Evaluation):
Added storage overhead for the intermediate models.

Author response:
The reviewer is right that our approach requires storage of additional models and also has an increased inference time cost due to forward-propagating inputs through all models. However, deep ensembles, the best competing approach to date, also requires storage of multiple models as well as an increased inference time cost. Many past works from the selective classification domain [1,2,3,4] do not consider Deep Ensembles explicitly as a competing method. In summary, we observe that the added storage and inference time cost do contribute to better selective prediction performance.

**References**

[1]  Feng, Leo, et al. "Towards Better Selective Classification." The Eleventh International Conference on Learning Representations. 2022.

[2] Huang, Lang, Chao Zhang, and Hongyang Zhang. "Self-adaptive training: beyond empirical risk minimization." Advances in neural information processing systems 33 (2020): 19365-19376.

[3] Liu, Ziyin, et al. "Deep gamblers: Learning to abstain with portfolio theory." Advances in Neural Information Processing Systems 32 (2019).

[4] Geifman, Yonatan, and Ran El-Yaniv. "Selectivenet: A deep neural network with an integrated reject option." International conference on machine learning. PMLR, 2019.

Reviewer comment (Evaluation):
Added inference cost for the prediction using the intermediate models compared to other selective prediction models that only require one forward pass through the architecture and the gating mechanism.

Author response:
The reviewer is right that our approach requires storage of additional models and also has an increased inference time cost due to forward-propagating inputs through all models. However, deep ensembles, the best competing approach to date, also requires storage of multiple models as well as an increased inference time cost. Many past works from the selective classification domain [1,2,3,4] do not consider Deep Ensembles explicitly as a competing method. In summary, we observe that the added storage and inference time cost do contribute to better selective prediction performance.

**References**

[1]  Feng, Leo, et al. "Towards Better Selective Classification." The Eleventh International Conference on Learning Representations. 2022.

[2] Huang, Lang, Chao Zhang, and Hongyang Zhang. "Self-adaptive training: beyond empirical risk minimization." Advances in neural information processing systems 33 (2020): 19365-19376.

[3] Liu, Ziyin, et al. "Deep gamblers: Learning to abstain with portfolio theory." Advances in Neural Information Processing Systems 32 (2019).

[4] Geifman, Yonatan, and Ran El-Yaniv. "Selectivenet: A deep neural network with an integrated reject option." International conference on machine learning. PMLR, 2019.

Reviewer comment (Experiments):
Have you plotted other baselines for the Figure 2 to see what impact these baselines have compared to SPTD?

Author response:
We provide this extended experiment as part of the updated PDF in [Figure 8](https://transfer.sh/91kwPXIxRw/fig8.png). We see that all methods reliably improve over the SR baseline. At the same time, we notice that SAT and DE still assign higher confidence away from the data due to limited use of decision boundary oscillations. SPTD addresses this limitation and assigns more uniform uncertainty over the full data space.

Reviewer comment (Experiments):
How do you select which of the intermediate checkpoints should be used for the inference stage? It is possible to design clever selection strategies during training to reduce the storage and inference cost rather than storing intermediate points at fixed checkpoint intervals.

Author response:
Our method relies on first computing a very detailed approximation of the training dynamics and then subsampling (see *Checkpoint Selection Strategy* on page 8) and weighting the subsampled checkpoints using $v_t = (\frac{t}{T})^k$. You can think of the weighting as a prioritization to which part of the training dynamics to pay attention to. As we show below, our weighting is an appropriate choice for the problems we consider and outperforms alternative non-convex or uniform weighting strategies. We agree with the reviewer that other weighting strategies with potentially stronger performance might exist. However, these weightings might require deliberate distributional assumptions. Our work demonstrates that $v_t = (\frac{t}{T})^k$ with a flexible choice of $k$ enables SOTA selective classification performance across a wide range of datasets without much hyper-parameter tuning.

Reviewer comment (Experiments):
Have you tried other weighting schemes than (t/T)^k?

Author response:
We have experimented with other weightings beside $v_t = (\frac{t}{T})^k$ with $k \in [0,\infty)$ but found that our choice of $v_t$ performs best across our experimental panel. This weighting encourages us to be particularly sensitive to late disagreements while at the same time incorporating diversity from models across the full training spectrum. We have updated the submission with a new experiment shown in [Figure 13](https://transfer.sh/u9s4eb2GXl/fig13.png) in which we also consider concave weightings $k \in (0,1]$ as well as a uniform weighting assigning the same weight to all checkpoints. It is evident that our weighting choice performs better than other approaches. Finally, we want to remind the reviewer that, as described in Section 3.2, our choice for $v_t$ is not arbitrary but inspired by recent results from sample difficulty [1,2,3].

**References**:

[1] Baldock, Robert, Hartmut Maennel, and Behnam Neyshabur. "Deep learning through the lens of example difficulty." Advances in Neural Information Processing Systems 34 (2021): 10876-10889.

[2] Zhang, Chiyuan, et al. "Understanding deep learning (still) requires rethinking generalization." Communications of the ACM 64.3 (2021): 107-115.

[3] Jiang, Ziheng, et al. "Characterizing structural regularities of labeled data in overparameterized models." arXiv preprint arXiv:2002.03206 (2020).

Reviewer comment (Evaluation):
Have you compared the inference cost of STPD with other methods (which only require one forward pass through the network and some gating mechanism)?

Author response:
As is typical for ensemble-based methods, our method naturally incurs a higher inference-time cost compared to models that do not require access to multiple models. While an ensemble consisting of $M$ models incurs approximately $M$ times the inference cost of a model only relying on a single forward pass, we remark that ensemble-based methods (Deep Ensembles, Selective Prediction Training Dynamics, as well as their combination) provide significantly stronger selective accuracy at a fixed coverage level than single forward pass methods (see Table 1).

Reviewer comment (Evaluation):
The numerical improvement over baseline (Table 1) seems very small.

Author response:
We understand the reviewer’s concern about the small improvements shown in Table 1. At the same time, we would ask the reviewer to put the presented results into context: 
- Many past works from the selective classification domain [1,2,3,4] do not consider Deep Ensembles explicitly as a competing method. Since we mainly consider our method as an alternative to DE, these improvements can appear small. 
- Moreover, many past works [2,3] do not explicitly accuracy-align models at full coverage which can give the proposed methods an unfair head-start and as a result overestimate the method’s effectiveness. We make sure to compare all methods on an equal footing by disentangling selective prediction performance from gains in overall utility. We highlight the presence of accuracy alignment in the caption of Table 1.
- Finally, we highlight that our method’s advantages extend beyond the SPTD results reported in Table 1. This includes (i) ambiguity w.r.t the training stage; (ii) retroactive applicability; as well as (iii) composability with existing SC approaches.

**References**

[1]  Feng, Leo, et al. "Towards Better Selective Classification." The Eleventh International Conference on Learning Representations. 2022.

[2] Huang, Lang, Chao Zhang, and Hongyang Zhang. "Self-adaptive training: beyond empirical risk minimization." Advances in neural information processing systems 33 (2020): 19365-19376.

[3] Liu, Ziyin, et al. "Deep gamblers: Learning to abstain with portfolio theory." Advances in Neural Information Processing Systems 32 (2019).

[4] Geifman, Yonatan, and Ran El-Yaniv. "Selectivenet: A deep neural network with an integrated reject option." International conference on machine learning. PMLR, 2019.

Reviewer comment (Evaluation):
Distribution on g evaluation (Figure 4) is only done for the proposed method. If other baselines have the same pattern, the relative efficacy of the method becomes questionable

Author response:
We provide an extended study on all methods in [Figure 10](https://transfer.sh/T4s7BpVbov/fig10.png). Since all methods are designed to address the selective prediction problem, they all manage to separate correct from incorrect points (albeit at varying success rates). We see that SPTD spreads the scores for incorrect points over a wide range with little overlap. We observe that for SR, incorrect and correct points both have their mode at approximately the same location which hinders performative selective classification. Although SAT and DE show larger bumps at larger score ranges for incorrect points, the separation with correct points is weaker as correct points also result in higher scores (i.e. longer blue tail) more often than for SPTD.

Reviewer comment (Experiments):
Have the authors try the method on OOD datasets? Can it reliably reject OOD samples?

Author response:
The reviewer is right that the field of OOD detection is an important discipline in trustworthy ML related to SC. We therefore have already provided preliminary evidence in [Figure 15](https://transfer.sh/emelUeYzsw/fig15.png) in the Appendix that our method can be used for detecting OOD examples (and also adversarial examples). While these results are already encouraging, we remark that adversarial and OOD samples are less well defined as incorrect data points and can come in a variety of different flavors (i.e., various kinds of attacks or various degrees of OOD-ness). As such, we strongly believe that future work is needed to determine whether a training-dynamics-based approach to SC can be reliably used for OOD and adversarial sample identification. In particular, a study of the exact observed training dynamics for both types of samples seems vital to ensure improved detectability.  We remark that in our extended synthetic experiments in [Figure 8](https://transfer.sh/91kwPXIxRw/fig8.png), we observe that SAT and DE assign higher confidence away from the data due to limited use of decision boundary oscillations. SPTD addresses this limitation and assigns more uniform uncertainty over the full data space, enabling better OOD detectability.

Reviewer comment (Experiments):
The results are presented in a way that gives the impression that one can control the coverage. How is it possible in practice? While the threshold can be adjusted to meet a certain coverage level on the training set, it is unclear how this could be done for the unseen test set. Setting tau algorithmically should be part of the solution.

Author response:
> First, the results are presented in a way that gives the impression that one can control the coverage. [...]

As part of our evaluation, we directly compute the accuracy/coverage tradeoff on the test set. This is consistent with how prior works evaluate their selective classification approaches [1,2,3,4] and allows us to highlight relative performance improvements of our method over competing works.

**References**

[1]  Feng, Leo, et al. "Towards Better Selective Classification." ICLR. 2022.

[2] Huang, Lang, Chao Zhang, and Hongyang Zhang. "Self-adaptive training: beyond empirical risk minimization." NeurIPS 33 (2020).

[3] Liu, Ziyin, et al. "Deep gamblers: Learning to abstain with portfolio theory." NeurIPS 32 (2019).

[4] Geifman, Yonatan, and Ran El-Yaniv. "Selectivenet: A deep neural network with an integrated reject option." ICML, 2019.

Reviewer comment (Evaluation):
If the main point of exploiting the checkpoint model is mainly to help with uncertainty calibration, what makes it perform better than methods that do so explicitly? A more insightful discussion on this is needed.

Author response:
> Second, if I understand the main point correctly, the exploitation of the checkpoint model is mainly to help with uncertainty calibration. [...]

The reviewer is right that our method helps with uncertainty calibration. Although we think that the precise reason for the effectiveness of our particular method (in particular theoretical guarantees) should be part of future work, we do have some evidence that our method provides improved results due to diversity ensembling. Note that while model ensembling can be achieved in various different ways, many past works have found that a key ingredient to well-performing ensembling is sufficient diversity between ensemble members [1,2,3]. We do provide some empirical evidence for this connection in [Figure 8](https://transfer.sh/91kwPXIxRw/fig8.png) where we see that the decision boundaries considered by SPTD are significantly more diverse than the boundaries derived by DE or SAT.  

**References**

[1] Kuncheva, Ludmila I., and Christopher J. Whitaker. "Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy." Machine learning 51 (2003): 181-207.

[2] Sollich, Peter, and Anders Krogh. "Learning with ensembles: How overfitting can be useful." NeurIPS 8 (1995).

[3] Morishita, Terufumi, et al. "Rethinking Fano’s Inequality in Ensemble Learning." ICML, 2022.

Reviewer comment (Experiments):
Does the tuning of the parameter k depend on test data?

Author response:
> Third, does the tuning of the parameter k depend on test data?

Although the parameter $k$ can be tuned on a per-dataset basis, we found that on the datasets we consider the particular choice of $k$ is robust in the interval $[1,3]$ (see [Figure 12](https://transfer.sh/03R3gwjBBR/fig12.png)). Ablating the parameter in this interval delivers comparable performance. We also consider even more choices for the weighting $v_t$ in [Figure 13](https://transfer.sh/u9s4eb2GXl/fig13.png) and find that concave and uniform weighting schemes do not outperform the convex choice proposed in the main section of the paper.

Reviewer comment (Experiments):
Part of the claim is that the proposed method can be applied on top of existing models, but is it always effective? Could we demonstrate such synergy between SPTD and other baselines? Additionally, how well would a simple probabilistic method with explicit prediction uncertainty, such as Bayesian Neural Net or Gaussian Process, fare against SPTD? This could be investigated either via experiments or by pointing out previous findings in the literature.

Author response:
> Fourth, part of the claim is the proposed method can be applied on top of existing models. But is it always effective? Could we demonstrate such synergy between SPTD and other baselines? [...] Furthermore, I wonder how well a simple probabilistic method with explicit prediction uncertainty [...]

Table 1 already provides evidence that employing SPTD on top of DE yields improved performance. We also experiment with using SPTD on top of SAT and find that SPTD also improves performance in combination with SAT (see [Figure 11](https://transfer.sh/3gC2n3yyut/fig11.png)). This result reinforces our claim that applying SPTD on top of existing approaches yields improved performance across various methods.

We are skeptical that additional results using BNNs or GPs would provide insightful alternate baselines. The reason for this is that Deep Ensembles, for which we provide results, have been found to dominate these methods [1,2]. Bayesian Neural Networks only allow for unimodal uncertainty quantification (approximate a single mode in the loss landscape) while Deep Ensembles allow for multimodal uncertainty estimation (approximate multiple modes in the loss landscape). Moreover, to the best of our knowledge, Gaussian Processes are rarely used as an uncertainty quantification method for high-dimensional image classification with large amounts of data. Inverting the Gram matrix is typically intractable in these use cases and approximations of the Gramm Matrix and its inversion reduce the quality of the provided uncertainty.

**References**

[1] Fort, Stanislav, Huiyi Hu, and Balaji Lakshminarayanan. "Deep ensembles: A loss landscape perspective." arXiv preprint arXiv:1912.02757 (2019).

[2] Ovadia, Yaniv, et al. "Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift." NeurIPS 32 (2019).

Reviewer comment (Experiments):
Will SPTD still be robust regarding checkpoint resolution if the model complexity increases? ResNet18 is probably not a SOTA model for a hard classification task such as CIFAR-100 — how does SPTD work on a larger model, such as ViT?

Author response:
> Last, will SPTD still be robust regarding checkpoint resolution if the model complexity increases? [...]

We have previously experimented with VGG architectures and have found our results to be consistent independent of the chosen architecture. We remark that ViT models have not yet shown to consistently provide better utility compared to ResNets on the datasets we consider [1] and hence opted against ViT experimentation in our work. To still showcase our method with larger model complexity, we show results for CIFAR-100 on a Resnet-50 architecture in [Table 3](https://transfer.sh/L4SUbqq224/tab3.png). We observe that the effectiveness of SPTD carries over to this larger architecture.

**References**

[1] Zhu, Haoran, Boyuan Chen, and Carter Yang. "Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspective." arXiv preprint arXiv:2302.03751 (2023).

Reviewer comment (Experiments):
How do you set the threshold? In your experiment, was it set with respect to the training or test set?

Author response:
> How do you set the threshold? [...]

Across all experiments, $\tau$ is chosen to achieve a desired targeted coverage level. This is done by first computing the selection score $g$ across all data points, ranking the data points based on $g$ (effectively sorting them in ascending order), and then picking $\tau$ such that the first c% of test points are accepted. We don’t report these values for our experiments as they are method-dependent (different methods have different selection score distributions and therefore different thresholding values) and typically less interpretable than a specific targeted coverage level (for example, “accept 90% of data points” is often easier to understand than “accept all data points with selection score smaller than 0.5”). As described above and as is consistent with prior work, we compute the threshold on the test set directly.

Reviewer comment (Experiments):
The statement "checkpoint each model after processing 50 mini-batches of size 128. All models are trained over 200 epochs" is confusing. This seems to imply making 200 passes over the entire training set, and for CIFAR-10, looping over about 50K * 100 / 128 batches — meaning if you checkpoint every 50 batches, more than 1,000 checkpoints would need to be stored, which seems too extravagant.

Author response:
> I find this statement confusing: "checkpoint each model after processing 50 mini-batches of size 128. All models are trained over 200 epochs" [...]

The reviewer is right that we do indeed initially approximate the training dynamics on a very granular scale, leading to more than 1k checkpoints per dataset. As forward-propagating through > 1k checkpoints is evidently prohibitive, we show in the *Checkpoint Selection Strategy* paragraph on page 8 that subsampling 50, 25, or even 10 checkpoints still leads to strong selective classification. Especially for high targeted coverage (>50%), the performance across all checkpointing resolutions is indistinguishable from each other (Figure 5). This insight allows us to reduce the computational overhead of our method to the same cost as Deep Ensembles at test time while having a considerably less expensive training stage. Note that there is no need to compute the fine-grained trajectory and subsequently downsample to fewer checkpoints; this was merely done for our experiment to understand the limiting behavior of our approach. Practical implementations of our method can directly store checkpoints at a more coarse-grained resolution.

Reviewer comment (Experiments):
There is confusion here since later context clearly points out that the number of checkpoint models used in the experiment is between 25–50.

Author response:
> I find this statement confusing: "checkpoint each model after processing 50 mini-batches of size 128. All models are trained over 200 epochs" [...]

The reviewer is right that we do indeed initially approximate the training dynamics on a very granular scale, leading to more than 1k checkpoints per dataset. As forward-propagating through > 1k checkpoints is evidently prohibitive, we show in the *Checkpoint Selection Strategy* paragraph on page 8 that subsampling 50, 25, or even 10 checkpoints still leads to strong selective classification. Especially for high targeted coverage (>50%), the performance across all checkpointing resolutions is indistinguishable from each other (Figure 5). This insight allows us to reduce the computational overhead of our method to the same cost as Deep Ensembles at test time while having a considerably less expensive training stage. Note that there is no need to compute the fine-grained trajectory and subsequently downsample to fewer checkpoints; this was merely done for our experiment to understand the limiting behavior of our approach. Practical implementations of our method can directly store checkpoints at a more coarse-grained resolution.

Reviewer comment (Experiments):
In the synthetic experiment, would an explicit probabilistic method (such as BNN or GP) be over-confident? Can we do a quick check on this?

Author response:
> In addition, I wonder whether an explicit probabilistic method (such as BNN & GP) would be over-confident in the synthetic experiment. [...]

We provide an extended experiment as part of the updated PDF in [Figure 8](https://transfer.sh/91kwPXIxRw/fig8.png). We see that all methods reliably improve over the SR baseline. At the same time, we notice that SAT and DE still assign higher confidence away from the data due to limited use of decision boundary oscillations. SPTD addresses this limitation and assigns more uniform uncertainty over the full data space. We also provide a result using Bayesian linear regression in [Figure 9](https://transfer.sh/3OyOMS4bfV/fig9.png) which gives results comparable to deep ensembles.

Reviewer comment (Novelty):
The novelty of the method is limited, the ideas of re-using past checkpoints to form an ensemble can be found in e.g. [1]

Author response:
We thank the reviewer for alerting us to this relevant work! We remark that past work on ensembling (including [1]) considers averaging the predictions (and/or computing second moments) to get a better estimator of a model’s prediction (and/or uncertainty). Instead we investigate using checkpoints/ensembles as a means to check if a datapoint follows temporal patterns typical of datapoints we should accept from an example difficulty viewpoint, leading to novel aggregation schemes for ensembles. To state in other words, although [1] also uses training dynamics implicitly by constructing an average checkpoint ensemble, the goal in [1] is not selective prediction but boosting overall accuracy. Our approach focuses on selective prediction which requires us to derive a different aggregation scheme than [1]. We also demonstrate the applicability of our approach beyond classification and its composability with other selective classification approaches including composing on top of common ensembling techniques. Due to these key differences, we are confident that our work continues to be a valuable contribution to the selective prediction and uncertainty quantification communities. We have updated our related work section with a paragraph discussing this and other related training-dynamics specific works.

Reviewer comment (Theory):
The results for SPTD and Deep Ensemble (DE) are both relatively close to one another and it would be nice to derive conditions under which one method is expected to be better than the other.

Author response:
We understand the reviewer’s concern about the small improvements shown in Table 1. We also agree with the reviewer that deriving theoretical conditions for the effectiveness of SPTD would be interesting future work. At the same time, we would ask the reviewer to put the presented results into context: 
- Many past works from the selective classification domain [1,2,3,4] do not consider Deep Ensembles explicitly as a competing method. Since we mainly consider our method as an alternative to DE, these improvements can appear small. 
- Moreover, many past works [2,3] do not explicitly accuracy-align models at full coverage which can give the proposed methods an unfair head-start and as a result overestimate the method’s effectiveness. We make sure to compare all methods on an equal footing by disentangling selective prediction performance from gains in overall utility. We highlight the presence of accuracy alignment in the caption of Table 1.
- Our current intuition for the strong performance of SPTD (and DE+SPTD) is based on increased diversity. Note that while model ensembling can be achieved in various different ways, many past works have found that a key ingredient to well-performing ensembling is sufficient diversity between ensemble members [5,6,7]. We do provide some empirical evidence for this connection in [Figure 8](https://transfer.sh/91kwPXIxRw/fig8.png) where we see that the decision boundaries considered by SPTD are significantly more diverse than the boundaries derived by DE or SAT.
- Finally, we highlight that our method’s advantages extend beyond the SPTD results reported in Table 1. This includes (i) transparency w.r.t the training stage; (ii) retroactive applicability; as well as (iii) composability with existing SC approaches.

Reviewer comment (Experiments):
It is unclear how the performance of SPTD is tied to optimization noise. Especially, regression experiments use full-batch gradient descent, how would the results evolve when using smaller batches?

Author response:
As per the reviewer’s suggestion, we have run our regression experiments using mini-batches and find that the results were statistically indistinguishable from the results with full-batch gradient descent. Although the results are the same for the regression experiment, we do hypothesize that the randomness added by SGD is helpful in yielding more diverse intermediate models. As discussed above, diversity is a desirable property for ensemble models which in turn enables improved selective prediction performance.

Reviewer comment (Experiments):
The values for $v_t=(\frac{t}{T})^k$ seem a bit arbitrary.

Author response:
We note that the weighting $v_t$ is chosen to reflect sample difficulty patterns as described by past work [1,2,3]. As we already describe in Section 3.2, prior work has found that easy-to-optimize samples are learned early in training and converge faster. Our convex weighting is inspired by this insight as well as the convergence patterns derived in Figure 6. We have experimented with other weightings beside $v_t = (\frac{t}{T})^k$ with $k \in [0,\infty)$ but found that our choice of $v_t$ performs best across our experimental panel. This weighting encourages us to be particularly sensitive to late disagreements while at the same time incorporating diversity from models across the full training spectrum. We have updated the submission with a new experiment shown in [Figure 13](https://transfer.sh/u9s4eb2GXl/fig13.png) in which we also consider concave weightings $k \in (0,1]$ as well as a uniform weighting assigning the same weight to all checkpoints. It is evident from this experiment that our weighting choice performs better than other approaches.

Reviewer comment (Experiments):
How would your method compare to using conformal-based selective prediction [2, section 5.5]?

Author response:
The conformal-based selective prediction method in [2] is equivalent to softmax response (SR); they pick a threshold on $\hat{P}(x) = \max(\hat{f}(x))$. In particular, following typical conformal prediction methods, they use a calibration set to pick the threshold, while we (aligning with past selective prediction work) picked the threshold that gave a specific coverage/accuracy on the test set (as to compare this with other methods). In other words, conformal prediction in [2] deals with how to pick the threshold for softmax response, while our work shows there is a metric that gives a better signal for correctness than SR. Future work could try to apply conformal prediction to our SPTD metric as a means to pick the threshold.

Reviewer comment (Experiments):
Checkpoints are chosen based on a fixed schedule which can correspond to models of bad performance. A better approach is to follow the approach from [Huang et al. 2017] which constructs an ensemble by choosing points of good performance using a cyclic learning rate. Using good checkpoints removes the need for the complicated weighted aggregation of the disagreement functions as all the snapshots are good models.

Author response:
We thank the reviewer for alerting us to [Huang et al. 2017]! We remark that past work on ensembling (including [Huang et al. 2017]) considers averaging the predictions (and/or computing second moments) to get a better estimator of a model’s prediction (and/or uncertainty). Instead we investigate using checkpoints/ensembles as a means to check if a datapoint follows temporal patterns typical of datapoints we should accept from an example difficulty viewpoint, leading to novel aggregation schemes for ensembles. To state in other words, although [Huang et al. 2017] also uses training dynamics implicitly by constructing an average checkpoint ensemble, the goal in [Huang et al. 2017] is not selective prediction but boosting overall accuracy. Our approach focuses on selective prediction which requires us to derive a different aggregation scheme than [Huang et al. 2017]. We also demonstrate the applicability of our approach beyond classification and its composability with other selective classification approaches including composing on top of common ensembling techniques. Due to these key differences, we are confident that our work continues to be a valuable contribution to the selective prediction and uncertainty quantification communities. We have updated our related work section with a paragraph discussing this and other related training-dynamics specific works.

Reviewer comment (Experiments):
The proposed method has several hyperparameters including the number of checkpoints and the weights to calculate the selection function $g$ from the disagreement function $a$.

Author response:
We want to clarify that our method operates in the same number of hyper-parameters (checkpointing resolution, weighting parameter $k$) as other competing approaches. While SR is a hyper-parameter free method, Deep Gamblers (reward, pre-training duration), SAT (pre-training duration, SAT momentum for moving average), as well as Deep Ensembles (number of members, aggregation weighting) require tuning 2 hyperparameter, just like SPTD.

Reviewer comment (Experiments):
How was $\tau$ chosen?

Author response:
Across all experiments, $\tau$ is chosen to achieve a desired targeted coverage level. This is done by first computing the selection score $g$ across all data points, ranking the data points based on $g$ (effectively sorting them in ascending order), and then picking $\tau$ such that the first c% of points are accepted. We don’t report these values for our experiments as they are method-dependent (different methods have different selection score distributions and therefore different thresholding values) and typically less interpretable than a specific targeted coverage level (for example, “accept 90% of data points” is often easier to understand than “accept all data points with selection score smaller than 0.5”). Note that this procedure of reporting coverage instead of $\tau$ is the default way to evaluate selective prediction approaches [1,2,3,4].

**References**

[1] Feng, Leo, et al. "Towards Better Selective Classification." The Eleventh International Conference on Learning Representations. 2022.

[2] Huang, Lang, Chao Zhang, and Hongyang Zhang. "Self-adaptive training: beyond empirical risk minimization." Advances in neural information processing systems 33 (2020): 19365-19376.

[3] Liu, Ziyin, et al. "Deep gamblers: Learning to abstain with portfolio theory." Advances in Neural Information Processing Systems 32 (2019).

[4] Geifman, Yonatan, and Ran El-Yaniv. "Selectivenet: A deep neural network with an integrated reject option." International conference on machine learning. PMLR, 2019.

Reviewer comment (Reproducibility):
"Checkpoint each model after processing 50 mini-batches of size 128", how many checkpoints are chosen?

Author response:
Like $\tau$, the total number of checkpoints is also dataset-dependent and we report the exact values for the different datasets in Table 2 in the Appendix. Note that in the paper we first aim to derive a very detailed characterization of the training dynamics, leading to hundreds of checkpoints for each model, but then downsample from this full set of checkpoints to obtain a more tangible algorithm that is still performative. We discuss this selection in our *Checkpoint Selection Strategy* paragraph on page 8. As discussed there, subsampling 10-25 checkpoints is enough for performative selective prediction on the datasets we consider. Moreover, when particularly targeting the high coverage spectrum (>50%), we find that SPTD provides SOTA selective classification performance across a wide range of checkpointing resolutions.

Reviewer comment (Presentation):
In Table 1, the name of the baseline is SAT but in the text, it is mentioned that the baseline is SAT with Entropy Regularization (ER) and Softmax Response (SR) Selection from [Feng et al. 2023]. Which one is the baseline? If the latter, then please update the table as SAT and SAT+ER+SR are 2 different methods.

Author response:
We thank the reviewer for their attention to detail and have updated our draft with their suggestion.

Reviewer comment (Experiments):
Why not add SelectiveNet as a baseline for the regression task?

Author response:
We have not benchmarked SelectiveNet on neither the classification nor the regression task since ensemble-based methods (which we do include in our comparison) dominate SelectiveNet in our classification experiments. We have now also ran SelectiveNet (SN) on the regression task and observe in our updated [Figure 7](https://transfer.sh/9hOJrLkwTx/fig7.png) that SelectiveNet outperforms parametric outputs but does not reach selective utility levels obtained by DE or SPTD.

Reviewer comment (Experiments):
How were $g$ and $\tau$ chosen for the Deep Ensembles (DE)?

Author response:
Consistent with the Deep Ensembles paper [1], we choose $g(x) = \frac{1}{M} \sum_{m \in [M]} \max f_m(x)$. Note that $f_m(x)$ corresponds to the full $C$-class logit output and $\max f_m(x)$ extracts the maximum logit value. $\tau$ is chosen as described above.

**References**

[1] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural information processing systems 30 (2017).

Reviewer comment (Evaluation):
What is the intuition of SPTD performing better than DE for some coverages? It seems counter-intuitive as DE consists of high-performing models vs SPTD which has fixed checkpoints that do not have to be high-performing.

Author response:
Although we believe that future work should discuss this question in more detail, our current intuition for the strong performance of SPTD (and DE+SPTD) is based on increased diversity. Note that while model ensembling can be achieved in various different ways, many past works have found that a key ingredient to well-performing ensembling is sufficient diversity between ensemble members [1,2,3]. We do provide some empirical evidence for this connection in [Figure 8](https://transfer.sh/91kwPXIxRw/fig8.png) where we see that the decision boundaries considered by SPTD are significantly more diverse than the boundaries derived by DE or SAT. 

**References**

[1] Kuncheva, Ludmila I., and Christopher J. Whitaker. "Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy." Machine learning 51 (2003): 181-207.

[2] Sollich, Peter, and Anders Krogh. "Learning with ensembles: How overfitting can be useful." Advances in neural information processing systems 8 (1995).

[3] Morishita, Terufumi, et al. "Rethinking Fano’s Inequality in Ensemble Learning." International Conference on Machine Learning. PMLR, 2022.

Reviewer comment (Reproducibility):
How are the disagreement function, g, and $\tau$ chosen for DE+SPTD?

Author response:
For DE+SPTD, we first compute SPTD for multiple models $m \in [M]$ as follows: $g_{m}(x) = \sum_{t} v_t a_{m,t}(x)$. Note that $a_{m,t}(\cdot)$ now corresponds to the disagreement at time $t$ for model $m$. In a second step, we combine all SPTD scores into a single DE+SPTD score: $ \frac{1}{M} \sum_{m \in [M]} g_{m}(x)$. This procedure is informally described in the *Accuracy/Coverage Trade-off* paragraph on page 7. Once again, $\tau$ is chosen as described above.