Paper: PAGER: A Framework for Failure Analysis of Deep Regression Models (ICLR-2024)

Reviewer comment (Novelty):
The solution is based on anchoring predictive model. I realized the novelty majorly comes from the nature of the anchoring predictive model, which makes me concerning the contribution this particular paper (as it is more incremental now). In addition, how does people use this method to measure regression risk if they don't use anchoring model (which more likely happen in real world)?

Author response:
> **Novelty and Implications of using Anchoring**

While our solution is rooted in the established anchoring framework, this paper introduces several noteworthy contributions. 
+ We uncover a crucial insight that **uncertainties alone are insufficient** for a comprehensive characterization of failures in regression models. 
+ We advocate for incorporating **manifold non-conformity**, emphasizing adherence to the joint data distribution, as a valuable complement to uncertainty estimates. 
+ In an extension beyond the existing anchoring framework, we present a surprising finding that **non-conformity can be achieved through reverse anchoring** without the need for auxiliary models. 
+ Additionally, we propose a flexible analysis of model errors by introducing the concept of **risk regimes**, avoiding the necessity for a rigid definition of failure (such as an incorrect label) and the need for additional calibration data commonly seen in classification problems. 

Concerning the utilization of anchored models as the foundational framework, we believe that it is neither a weakness nor a limitation of PAGER. In this regard, we would like to offer two practical observations:

+ **Anchoring does not necessitate any adjustments to the optimizer, loss function, or training protocols**. The sole modification lies in the input layer, requiring additional dimensions for vector-valued data or channels for images, and a modest addition of parameters to the first layer. This approach is adaptable to any architecture, be it a Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), or Vision Transformer (ViT). Recent research indicates that **incorporating anchoring not only aligns with test performance but can even enhance generalization to out-of-distribution (OOD) regimes** [1,2].
+ Even in situations involving pre-trained models, it is feasible to **train solely an anchoring-based regression head attached to a conventionally trained feature extractor backbone**, effectively implementing PAGER. Optionally, one may choose to fine-tune the feature extractor concurrently with the anchored regression head in an end-to-end manner, adhering to standard practices.

As an illustration, in the experiment on CIFAR-10 rotation angle prediction under the Gaps setting with $\mathtt{Score1}$, we considered a variant, where we exclusively trained an anchored regression head while keeping the feature extractor frozen. Note, the feature extractor was obtained through standard training on the same dataset. Our findings reveal that even with this approach, the **performance is comparable to that of a fully anchored ResNet-34 model**. We have updated the paper with this result (Appendix A.6)

| Method                          | FN       | FP       | $C_{\text{low}}$ | $C_{\text{high}}$ |
|---------------------------------|----------|----------|------------------|-------------------|
| DEUP                            |   14.9   |   15.22  |       18.81      |       27.50       |
| Full Anchored Training          | **3.34** | **7.86** |     **3.28**     |      **5.34**     |
| Anchored Regression Head (only) |  _4.78_  |  _8.33_  |      _4.96_      |       _6.05_      |

Reviewer comment (Theory):
Multiple concepts introduced in this paper are hypothetical and may not be possible to know in practice. E.g. OOS / OOD are hard to distinguish.

Author response:
> **Definitions for OOD and OOS**

Thank you for your feedback. Our definitions of OOD (Out of Distribution) and OOS (Out of Support) are directly derived from existing literature [2], where the characterization is rooted in various regimes from which the test samples originate. Although an alternative variation, Out-of-Combination (OOC), has also been considered, in our framework, we categorize it within the broader umbrella of Out-of-Support data. Alongside these definitions, we have incorporated Figure 2 to offer readers a visual representation for enhanced clarity.

Reviewer comment (Evaluation):
The evaluation metric introduced in this paper is hard to be convincing in terms of the setting with 20 percentile or 90 percentile in the False Positive and $C_{low}$. Why? What makes the authors to choose those numbers. Is that the real reason of better performance compared with existing solutions?

Author response:
> **Clarification on evaluation metrics reported in the paper**

The defined thresholds in PAGER  **merely serve as guidelines for its usage, and the reported improvements do NOT hinge on them**. Addressing the inherent difficulty in characterizing failures in regression settings, where acceptable error tolerance can significantly vary across applications, it is a common practice to flag the most confident and error-prone samples. However, PAGER for the first time offers a nuanced characterization through multiple distinct risk regimes for ranking samples.

**Regarding Clow and Chigh**: The selection of the 90th percentile and 10th percentile to gauge the error ratio between moderate and high-risk bins (or equivalently ID and low-risk bins) is **intentionally more conservative and challenging**. Even when the thresholds are relaxed, the performance gap persists in PAGER, as the average error in the high-risk bin increases, and that of the moderate-risk bin decreases, resulting in a lower ratio. An illustrative example of Chigh on the CIFAR-10 rotation angle prediction task with various thresholds is provided.

|      Method     | 90th per. of Moderate Risk / 10th pct. of High Risk | 80th per. of Moderate Risk / 20th pct. of High Risk | 70th per. of Moderate Risk / 30th pct. of High Risk |
|:---------------:|:---------------------------------------------------:|:---------------------------------------------------:|:---------------------------------------------------:|
|       DEUP      |                        27.50                        |                        22.95                        |                        17.53                        |
| PAGER (Score 1) |                       **5.34**                      |                       **2.58**                      |                       **1.61**                      |

**False Positives (FP) and False Negatives (FN)**: The utilization of the 20th and 80th percentiles as thresholds for assessing FP and FN align with established protocols for evaluating failure detectors [3]. **While previous studies often compared the average Mean Squared Error (MSE) of samples in the bins, we perform a more stringent evaluation** by quantifying the rates of false positives and negatives.

In conclusion, although the selected thresholds are provided as guidelines and can be adjusted according to the specific requirements of an application, we advocate for a more rigorous evaluation and, in fact, present results for challenging scenarios.

> **References**

+ [1] Single model uncertainty estimation via stochastic data centering, NeurIPS 2022.
+ [2] Learning to extrapolate: A transductive approach, ICLR 2023.
+ [3] Data-SUITE: Data-centric identification of in-distribution incongruous examples, ICML 2022.

Reviewer comment (Evaluation):
I'd like to get more insight about the usefulness of $\text{score}_2$.

Author response:
We thank the reviewer for their positive assessment of our work.  If our responses have adequately addressed the concerns, we request the reviewer to increase their rating to help champion our paper.

> **Utility of $\mathtt{Score2}$ and Comparison to $\mathtt{Score1}$**

Conceptually, $\mathtt{Score2}$ employs a more rigorous optimization approach to **effectively address test scenarios in which both a test sample and its residual are individually out-of-distribution (OOD), not just their combination, with respect to $P(X)$ and $P(\Delta)$**. While the computationally efficient $\mathtt{Score1}$ tends to perform comparably (or even better) to $\mathtt{Score2}$ under challenging extrapolation conditions (such as gaps and tails in the target variable), $\mathtt{Score2}$ often demonstrates noteworthy enhancements in confusion scores (Clow and Chigh). This becomes particularly valuable in practical applications **where users cautiously identify failures (Top K%) and it is essential to prevent the misclassification of some high-risk samples as moderate risk**.

Another scenario where $\mathtt{Score2}$ can be very **useful is when the test samples are drawn from a different distribution compared to training (referred to as covariate shifts)**. To demonstrate this hypothesis, we repeated the CIFAR-10 rotation angle prediction experiment by applying *natural image corruptions (defocus blur and frost) at different severity levels*. Interestingly, we observed significant improvements in both false positive (FP) and false negative (FN) scores with $\mathtt{Score2}$ as the severity of corruption increased. In summary, while $\mathtt{Score1}$ excels in scalability and is well-suited for online evaluation, $\mathtt{Score2}$ proves advantageous in addressing challenging testing scenarios. We have updated the paper with these results (Appendix A.5).

|            	|                   	| **Defocus Blur** 	| **Defocus Blur** 	| **Defocus Blur** 	|  **Frost** 	|  **Frost** 	|  **Frost** 	|
|---------------|-----------------------|-------------------|-------------------|-------------------|---------------|---------------|---------------|
| **Metric** 	|     **Score**     	|    **Sev. 1**    	|    **Sev. 3**    	|    **Sev. 5**    	| **Sev. 1** 	| **Sev. 3** 	| **Sev. 5** 	|
|     FP     	| $\mathtt{Score1}$ 	|     **8.05**     	|       10.29      	|       18.54      	|  **7.81**  	|  **8.80**  	|    16.52   	|
|     FP     	| $\mathtt{Score2}$ 	|       9.28       	|     **9.95**     	|     **14.08**    	|    9.06    	|    9.58    	|  **11.76** 	|
|     FN     	| $\mathtt{Score1}$ 	|     **3.65**     	|       5.85       	|       19.81      	|    3.77    	|    6.12    	|    15.63   	|
|     FN     	| $\mathtt{Score2}$ 	|       3.92       	|     **5.05**     	|     **14.99**    	|    4.01    	|  **5.68**  	|  **11.14** 	|

Reviewer comment (Experiments):
Can the framework proposed in this paper for analyzing model failures be applied to multi-class classification tasks, and what are the potential differences between it and the regression task studied in the paper?

Author response:
Thanks for raising this important question. 

In classification tasks, incorrectly assigned labels are considered failures. Therefore, detecting failure is framed as **assessing the probability of an inaccurate label prediction**. State-of-the-art approaches typically employ auxiliary predictors, trained retrospectively, to correlate various sample-level scores (such as uncertainties, smoothness, and disagreement across multiple hypotheses) with the likelihood of an incorrect prediction. This approach inherently depends on extra labeled calibration data and has implications for the generalization of the learned failure detector in the face of shifts.

A key difficulty in characterizing failures in regression tasks stems from the **variability in the permissible tolerance levels on error estimates**, which can differ across use-cases. This is in contrast to the clear-cut definition of failure in classification tasks. Addressing this challenge, PAGER proposes to organize samples into various risk regimes rather than relying on a binary pass-fail (0-1) categorization. This approach not only allows for a nuanced assessment without the need for a predefined, rigid definition of failure, but also eliminates the necessity for labeled calibration data. It’s worth noting, however, that state-of-the-art methods like DEUP, which seek to emulate failure detectors from the classification literature by fitting an auxiliary predictor to estimate the loss, exhibit poor performance in practice.

Ultimately, while the risk regime characterization approach introduced by PAGER may not find direct applicability in the existing frameworks for failure detection in deep classifiers, it’s crucial to underscore that the proposed uncertainty and non-conformity scores can serve as alternatives to widely used scoring functions for training the auxiliary predictor. Nevertheless, unlike the implementation in PAGER, adopting these scores in current classifier failure detectors would necessitate access to additional calibration data. **We have added a discussion to the paper (Appendix A.7), emphasizing the importance of developing calibration-free failure detectors, even in the context of multi-class classification**.