Paper: Understanding Pathologies of Deep Heteroskedastic Regression (ICLR-2024)

Reviewer comment (Theory):
(main) Is there a sharp phase transition? This paper lacks quantitative measurement and results. It would be interesting to calculate, at least under certain simple generative models, the free energy and check if there is a first-order/second-order phase transition.

Author response:
We agree that the phase transitions are worthy of further investigation. Unfortunately, as seen in the cross sections through $\rho-\gamma$ space, though these phase transitions are continuous, they are not smooth in the order parameter, and we are unable to study this behavior analytically.

Reviewer comment (Novelty):
(main) Technically speaking, not sure which part of the paper is innovative---for example, Eqns 7--10, are they new or semi-new (i.e., similar derivations are obtained in a different context)? I would be skeptical that Eqns 7--10 are entirely new. Also, the proposed regularization is well-studied in the literature.

Author response:
Yes, weight decay is a well-studied method for regularization, but we believe that our parameterization is unique, and splitting the regularization across two separate networks has not yet been studied in depth. [2] also looks at fitting heteroskedastic regression models with different levels of regularization applied to the two networks, and we will add this reference to the camera-ready version.

Reviewer comment (Writing):
I don’t understand the sentence below equation (9): ‘Interestingly, both resulting relationships include a regularization coefficient divided by the density of $x$.’ Does the word ‘regularization coefficient’ refer to the term with Laplace operator, which originates from the regularization term?

Author response:
> I don’t understand the sentence below equation (9): ‘Interestingly, both resulting relationships include a regularization coefficient divided by the density of p(x).’ Does the word ‘regularization coefficient’ refer to the term with Laplace operator, which originates from the regularization term?  

Yes, that is correct.

Reviewer comment (Writing):
Typo: on top of equation (1), (i.e., $y_i \sim \mathcal{N}(\mu_i, \Lambda_i)$), $\Lambda_i$ or $\Lambda_i^{\frac{1}{2}}$?

Author response:
> Typo: on top of equation (1)  

$\Lambda$ represents the precision (inverse-variance), so $\Lambda^{-1/2}$ is the standard deviation.

Reviewer comment (Theory):
The challenge of conditional variance estimation arises from overfitting, yet the main result is stated for a *population objective* without any account for sample size.

Author response:
> Furthermore, the challenge of conditional variance estimation arises from overfitting, yet the main result is stated for a population objective without any account for sample size.

In our formulation, we can account for sample size through the $\rho$ parameter that scales the importance of the loss between the regularizers and the likelihood.

Reviewer comment (Theory):
The references to statistical physics appear completely unnecessary. Calling Eq. (7) a "nonparametric free energy" does not provide any new insight. The proof of the main result also makes no use of techniques or ideas from statistical physics.

Author response:
> Additionally, the references to statistical physics appear completely unnecessary. 

Looking at the phase diagrams was an idea inspired by statistical physics. The resulting phase diagram reveals a complex interaction between the regularization strength of the respective functions that cannot be easily summarized by a ratio or linear dependence. This new perspective over combinations of regularization strengths allowed us to notice that an efficient 1-dimensional search along the off-diagonal could be sufficient to find a well-calibrated fit.

Reviewer comment (Theory):
The authors are encouraged to study the proposed learning objective in a relevant and non-trivial regime, and possibly to familiarize themselves with notions in learning theory and nonparametric statistics.

Author response:
> The authors are encouraged to study the proposed learning objective in a relevant and non-trivial regime, and possibly to familiarize themselves with notions in learning theory and nonparametric statistics

We acknowledge that there are other avenues for approaching this problem, but do not consider that a reason to avoid other methods. Learning theory and nonparametric statistics do not relate to phase transitions to our knowledge.