Paper: Conformal Prediction via Regression-as-Classification (ICLR-2024)

Reviewer comment (Novelty):
To my opinion, the main contribution is proposing a new nonparametric method for modelling the conditional density. However, I don't see any contribution w.r.t. conformal prediction.

Author response:
Most papers on CP  focus on the choice of score function. Depending if one want to consider heteroscedasticity, bimodality, skewness etc. Our contribution single handedly covers these case contrarily to DCP, CQR, CHR.

Reviewer comment (Evaluation):
In the experiments, the authors only consider the standard split conformal prediction, which is arguably not the most useful type of evaluation for heteroscedastic regression problems. I would argue that conditional coverage guarantees need to be analyzed in such cases. This can be established via Mondrian conformal prediction or normalized nonconformity scores, such as normalized absolute residuals.

Author response:
We are afraid that the reviewer is confusing split conformal methods (which is an alternative to the highly expensive Full conformal methods) and the choice of the score function that account for heteroscedasticity. For example, CQR or DCP explicitly account for heteroscedasticity and comes with both Split and Full version, where the former is preferred for computational efficiency.

> *Conditional coverage guarantees need to be analyzed in such cases. This can be established via Mondrian conformal prediction*

There are several types of conditional guarantees Mondrian CP are designed to be conditional on the class of the input object. In this case, the p-values function itself is modified by including a specific taxonomy (e.g. provided by a classifier or clustering algorithm). These are available in both the Split CP version and the Full CP version (or even other cross-conformal alternatives). Being split or not is about computational efficiency. Heteroscedasticity or other properties are handled by the choice of the conformity score function. 

We would like to add that CQR provides a benchmark against variance-based normalization and achieved superior performance.

By conformalizing our density estimation leveraging a (Mondrian or Venn) taxonomy, our method trivially accounts for class conditionality since this is independent of the choice of score function (it is just about the choice of p-value function). For a review of these concepts, please refer to Chapter 4.6.1 of the book 'Algorithmic Learning in a Random World', second edition, by Vovk, Gammerman, and Shafer.

Moreover, most of the baselines in regression CP focus on split conformal prediction. It is a standard technique in the CP literature. We believe it is unfair to reject a paper for using a standard evaluation setup.

Reviewer comment (Experiments):
The authors consider a few other methods for estimating the conditional density, such as kernel density estimation and quantile regression. However, many other methods exist for estimating the conditional density. Since this is the key contribution of the paper, I would like to see a more thorough experimental and theoretical comparison with such methods.

Author response:
The goal of our work is to improve conformal prediction, rather than proposing a better method for conditional density estimation (which is a challenging problem in itself). This is why we use best performing CP methods as baselines. Showing that better conditional density leads to better CP method would be an interesting direction, but would lead to a different paper.

Reviewer comment (Novelty):
Examples of such methods are mean-variance neural networks, conditional transformation models and methods based on normalizing flows. The proposed method estimates the conditional density via a specific histogram. Methods based on nearest neighbors and regression trees also produce histogram-style conditional densities. Another approach to model multimodal distributions is via Gaussian mixtures. So, there is a lot out there already... Where should we situate the new method in this diverse landscape? I would classify the new method as yet another way of modelling the conditional density. Not better or worse than others, perhaps useful in specific situations, such as multimodal distributions, but more discussion is needed.

Author response:
These methods of Conditional density estimation have not been used in CP literature before. While they are reasonable ways to estimate conditional density, our method is very simple. There are many ways to do conditional density estimation like you mentioned (see Rothfuss et al.). However, these are not any simpler and are more involved than our technique. Moreover, the existing CP baselines do not compare with these techniques either. We do compare with methods like KDE or Histogram Regression which are both conditional density estimators and the latter uses neural networks. Relative to relevant papers such as CHR, we compare to the same amount or more baselines, so we believe our experimental setup is sufficient.

Reviewer comment (Experiments):
By modelling the conditional density as a histogram, one obtains a very non-smooth approximation of the density. So, that's why interpolation is needed to obtain good nonconformity scores. Some of the other methods that are described above don't have this limitation.

Author response:
If you look at Figure 4 in the appendix, we learn pretty smooth distributions out of the box. Interpolation is *not* used for this. It is only used to make a discrete density function into a continuous one. Our interpolation is a simple linear interpolation, which does no extra smoothing on top. In fact, the smoothness is a strength of our method. If you look at CHR, they learn very spiky distributions.

Reviewer comment (Experiments):
The number of bins looks like a very tricky hyperparameter to tune. The performance probably highly depends on that hyperparameter. What would be a good way to tune this hyperparameter? Another tricky hyperparameter is the level of entropy regularization. I would assume that the results are also very sensitive to that hyperparameter.

Author response:
No tuning was done at all. We hold $K=50$ bins and $\tau=0.2$ across all experiments which span 16 different datasets with widely varying distribution shapes. If tuning was done, we would expect even better results. We will add ablations in the final version of the paper to investigate how our results are dependent on these hyperparameters.

Reviewer comment (Evaluation):
I believe that the proposed method might work reasonably well for multimodal distributions that are not too complex, but I would not recommend this method for unimodal regression problems. For such problems, there are too many degrees of freedom.

Author response:
This is not correct. Many of the data such as MEPS are unimodal and we get the best results out of all methods.

Reviewer comment (Evaluation):
In Table 3 a lot of standard deviations are zero, or close to zero. How is that possible? What is randomized? With randomizations of the testset these numbers are unrealistically small I would say.

Author response:
We repeat the experiments over different seeds and report the average mean over all seeds. The value you are referring to in Table 3 is the standard error (we have clarified this in the uploaded version). Since we use a large proportion for the test set, this suggests that our method performs well across numerous seeds, and small standard errors are reasonable.

Reviewer comment (Evaluation):
Table 1 versus Table 3: normally one would see that the length of the intervals increases when the coverage increases. So, the two criteria highly depend on each other. Any comparison of two methods should always make the trade-off between those two criteria.

Author response:
We have fixed all experiments at coverage level of $90\%$. We have specified this in newly uploaded versions. This is standard in the CP literature.

Reviewer comment (Novelty):
What's novel in this paper compared to reference Stewart et al.?

Author response:
Stewart et. al focuses on learning conditional expectation while we focus on conditional density. Using their loss function naively performs worse than our smoothened version of the loss function according to the ablations. We will clarify better in the final version that Stewart et al’s loss is included in the ablations and did not work. Our loss function is tailored to our specific problem, for example with smoothing.

Reviewer comment (Writing):
The role of the discretization step may be explained better.

Author response:
Several works in the literature, including Stewart et al. mention that the classification problem tends to lead to stabler training than regression.  Stewart et al. empirically demonstrates that R2C estimates the conditional expectation better than regression alone. The reason that this is the case is an open theoretical problem. However, intuitively, the learning problem under R2C is an unconstrained learning problem with only $K$ outputs whereas methods like CHR have to have 1000 output nodes and average among them to achieve smoothness.

Reviewer comment (Novelty):
Is the smoothness-enforcing penalty new?

Author response:
Smoothing-enforcing penalty has existed in other contexts such as Variational Learning. However, in this context of learning conditional densities using Neural Networks, it is new to the best of our knowledge.

Reviewer comment (Experiments):
How were K and Tau chosen? Is part of the data used for training the conditional distribution?

Author response:
$K $, $p$, and $\tau$ were fixed at $50$, $.5$, and $.2$ for all experiments. We didn't change that or tune that as that would be unfair to other methods. Tuning these would likely improve the results of our method. 

We are not sure we understand the question. As usual, the dataset was split in two (training and calibration), and indeed, the training set was used to learn the conditional distribution.

Reviewer comment (Theory):
Why should the entropy regularization be expected to be good at learning bi-modal distributions?

Author response:
It’s true. It is possible that reducing entropy can sometimes help, but as a general tool, entropy regularization helps overall. Empirically, our structure still works well for bimodal distributions.

Reviewer comment (Evaluation):
The size of the prediction intervals is a good measure of efficiency. Another one is the correlation between the test errors and the corresponding intervals. It would be interesting to see a scatter plot of the absolute residual versus the prediction intervals in a couple of data sets where R2CCP is or is not the best algorithm.

Author response:
Thanks for your suggestion. We will try to illustrate this plot for a couple of benchmarks for the camera ready version.

Reviewer comment (Novelty):
Is the proposed approach equivalent to training a parameterized conformity function (see for example in [2] or [3])?

Author response:
We are very different from the Einbinder paper since their loss function tries to ensure its scores form a smooth CDF during training and is nondifferentiable. We do no such thing and use much simpler differentiable methods. The second Colombo paper proposes how to transform conformity scores to make it more suitable for APS.

Thanks for the additional reference, we will add [1] in our related work section.

Reviewer comment (Novelty):
The way authors frame the regression problem as a classification is very, very close to the standard ordinal regression problem, and I really missed some positioning with respect to this literature. There is a huge literature on ordinal regression (maybe look at a general paper, e.g., "Tutz, G. (2022). Ordinal regression: A review and a taxonomy of models. Wiley Interdisciplinary Reviews: Computational Statistics, 14(2), e1545."), but also a couple papers using ordinal conformal classification (see questions).

Author response:
Thanks for the suggestion. We completely agree with you that there are strong connections with ordinal regression. We noted on this connection with citations to Weigend & Srivastava (1995) and Diaz & Marathe (2019). More recently, in computer vision, also some investigation is done to understand the effectiveness of such methods (https://arxiv.org/pdf/2301.08915.pdf). We have also added several relevant citations to the ordinal regression literature in the related works in the newly uploaded paper.

Reviewer comment (Theory):
It seems to me that the conformal scores used in Algorithm 1 mostly rely on modal values. In ordinal regression, it is much more common to use the average rank (under $L_2$ loss) or the median (under $L_1$ loss) as predictions. Could you comment on such options?

Author response:
A key contribution of our paper is its versatility across different distributions via building a probability density function. Consider a bimodal label distribution; the median and mean would be very unlikely answers since they lie in the middle of two valleys. To account for such distributions, we found modal values to be a better fit.

Reviewer comment (Evaluation):
It would be nice to have an idea of how often the output prediction regions are not interval, to have an idea of how often we depart from classical methods. Would it be possible to have an idea (in the appendix or main paper)?

Author response:
We can do this in the final version of the paper.

Reviewer comment (Presentation):
Could you display full coverage curve (say, from 90% to 99%) rather than a table for a fixed value? (Side remark: there is a double ?? in appendix B).

Author response:
We have fixed the broken reference in the appendix in the newly uploaded version. Moreover, we can work on the full coverage curve for the camera-ready version.

Reviewer comment (Presentation):
Table 1 is very hard to understand because (i) the methods are not connected to the acronyms, (ii) it is not clear what is inside the brackets, and (iii) length and coverage are not defined in the paper.

Author response:
Thanks, we agree with your points. We have fixed these in the newly uploaded version.  We have specified that the brackets include the “Standard Error” over the experiments. We also provide definitions for length and coverage. We believed the graphs and terms were standard enough in the literature. To avoid confusion, we will make sure to be more specific in the final submission.

Reviewer comment (Evaluation):
Most results are not reported with error bars, making them less reliable. For instance, in Table 1, CHR does worse than CQR, which is unexpected since both methods are proposed by similar authors and CHR comes later.

Author response:
We run all of our experiments by averaging over the test set and over 5 different seeds. With every different seed, we retrain the model. The standard error of the results over the different seeds is shown in the brackets. Given that we are running experiments across significantly more datasets than the ones used in the CHR and CQR papers, it is reasonable that upon seeing new datasets, CHR does worse than CQR in some cases, and in some cases, CHR outperforms CQR. For both baselines, we use code written by the authors themselves and do not edit them at all. **We report the numbers given by their open-sourced code explicitly.**

Reviewer comment (Experiments):
There is no baseline where a regressor is learned directly using the same neural network architecture, treated as \bar{q}, and then the same conformalization applied. This would be similar to the "optimal prediction sets" in Appendix F of https://arxiv.org/pdf/1910.10562.pdf.

Author response:
From what we are aware of, learning conditional density for regression problem using neural network require more involved methods with specific prior such as Gaussian mixture parametrization or Bayesian network (see Rothfuss et al.). Here, we are interested in computing a confidence set instead of the harder problem of getting accurate density estimation. As such, we propose to bypass those difficulties by converting the problem into a multiclassification problem where the parametrization is simpler. As you stated above, we rely on  “a traditional neural network with softmax output, which everyone understands”. Also, we note that CHR uses a histogram powered by a neural network to estimate the conditional density in a regressor setting, as you suggest. On top of that, we use 6 baseline effective conformal methods in our benchmark. Of course, we can always add more, but we believe that our numerical experiments are revealing enough.

Reviewer comment (Novelty):
It is unclear which aspects of the R2C (regression-as-classification) method are novel and which are derived from prior work. There is a concern about whether significant changes have been made to previous R2C methods to adapt them to the conformal problem. The key relevant technique appears to be smoothing; it is unclear whether this has been proposed before.

Author response:
As is presented, this method of R2C for conformal prediction has not been done before. Smoothing in different contexts has been proposed, such as in ordinal regression, but not in R2C literature. However, Stewart et al. is the standard R2C loss function, and using that directly does not work (see our ablation studies). We will make it clearer that the loss function from Stewart et al. is the one studied in the ablations. They focus on conditional expectation, whereas our loss focuses on conditional density. Our smoothened loss function works better according to the ablations. This use of R2C to learn conditional density alongside smoothing is novel to this paper.

Reviewer comment (Writing):
It is not clear what linear interpolation is used to go from q_\theta to \bar{q}_\theta; further elaboration is needed.

Author response:
We have added this in the newly uploaded version. We clarify it as below (note here the $\hat{y_k}$ and $\hat{y_{k+1}}$ are chosen such that  $\hat{y_k} < y < \hat{y_{k+1}}$: 


$\bar{q}(y \mid x) = $ $\frac{(\hat{y_{k + 1}} - y) q(\hat{y_k} \mid x)}{\hat{y_{k + 1}} - \hat{y_k}}  $ $+ \frac{(y - \hat{y_{k}}) q(\hat{y_{k+1}} \mid x)}{\hat{y_{k + 1}} - \hat{y_k}} $

Reviewer comment (Theory):
The justification for regression-as-classification is unconvincing. It is questioned why one would first destroy the ranking structure of real numbers and then reinstate it using entropic regularization, instead of performing usual regression.

Author response:
Several techniques exist to use regression techniques to learn conditional densities, as shown in our baselines. However, these methods can suffer from poor stability during training. For example, see Figure 4, and you will see that CHR’s technique using regressors is not smooth. Our loss function results in smoother and more stable densities. In fact, the Stewart et al. paper demonstrates empirically where R2C outperforms regression techniques for estimating the conditional expectation. Why this is the case theoretically is still an open research problem, as far as we know.

Reviewer comment (Novelty):
Which aspects of the R2C method are novel and which directly derive from previous work, and whether significant changes were made to adapt them to the conformal problem, remain unclear. The role of smoothing and whether it has been proposed before should be clarified.

Author response:
As is presented, this method of R2C for conformal prediction has not been done before. Smoothing in different contexts has been proposed, such as in ordinal regression, but not in R2C literature. However, Stewart et al. is the standard R2C loss function, and using that directly does not work (see our ablation studies). We will make it clearer that the loss function from Stewart et al. is the one studied in the ablations. They focus on conditional expectation, whereas our loss focuses on conditional density. Our smoothened loss function works better according to the ablations. This use of R2C to learn conditional density alongside smoothing is novel to this paper.