Paper: Unnormalized Density Estimation with Root Sobolev Norm Regularization (ICLR-2024)

Reviewer comment (Novelty):
The proposed estimator is rather straightforward and has less impact the machine learning community.

Author response:
>The proposed estimator is rather straightforward and has less impact the machine learning community. 

We will make clarifications in the text, re-emphsizing and incorporating many of the points in the comments above. In view of this, would the reviewer consider modifying the above statement?

We further note, that as the reviewer pointed out, 
a *two-dimensional* case was published only three years ago in JRSS, while here we treat high dimensions, and also obtain very competitive results on a  mainstream AD benchmark.  We believe this may put the impact of the paper in a different perspective. 

In connection with straightforwardness of the methods, we believe this would ultimately always remain somewhat hard to argue about. However, as we mention above, there are many components beyond the formulation itself that are required to make the approach practical, and we feel that it is not obvious how that could be done or even whether it could be done at all.

Reviewer comment (Reproducibility):
Once the estimator for the unnormalized model is estimated, how can the estimator be utilized?

Author response:
>Once the estimator for the unnormalized model is estimated, how can the estimator be utilized? 

We are not sure we understand this question. We have included an application 
of the method to a state of the art anomaly detection problem, and showed that the method performs very well, outperforming many dedicated anomaly detection methods on many datasets.  In addition, in the future we hope to develop  Markov Chain Monte Carlo methods of sampling the density, thus turning the model into a generative model. Metropolis sampling algorithms do not require the density to be normalised, and thus are well suited for such situations. 

Does this resolve the question of utilisation?

Reviewer comment (Evaluation):
Section 6.3 reports the ratio of negative values for the estimated function f. I'm not sure why the ratio of negative values is important to assess the computational properties.

Author response:
> Section 6.3 reports the ratio of negative values for the estimated function f. I'm not sure why the ratio of negative values is important to assess the computational properties.

This point is important and we now clarify it. 
As we discuss in the paragraph following Eq. (5), the main optimisation objective (5) is not convex as a function of $f$. Moreover, if one nevertheless tries to optimise (5) in $f$ (i.e., in $\alpha$, using $f_{\alpha}$) directly, one would obtain very poor results. The gradient descent will get stuck in a local optimum with poor values. 
We can provide an experimental demonstration of this if that would be of interest. 


However, we observe that (5) is convex on the non-negative cone, 
i.e., $f$ s.t. $f>0$ everywhere. (It is also sufficient to require  only $f(x_i)>0$ on data points $x_i$).    
Thus, if one restricts to such cone, one can obtain the optimal solution on that cone. We observe that such solutions are empirically much better. 
Next, we note that solving a constrained optimisation problem, requiring the solutions to be restricted to the cone is computationally difficult.  Moreover, since the cone is not closed (topologically), this is also not quite clear theoretically, since projection methods can not be applied. 


To resolve this, we observe that  *natural gradient* preserves the cone. 
This means that one can in fact do *unconstrained* optimisation. 
This is true for non-negative kernels, such as the Matern kernels (discussion following Equation (5) and Proposition 10), and is approximately true for the SDO kernel.  

In the experiment in Section 6.3 we show that empirically, for the SDO kernel, natural gradient indeed tends to preserve positivity, while the regular gradient does not. This demonstrates that natural gradient is indeed better for optimisation purposes. Note that, theoretically, this is one of the very few settings were we know *why* the natural gradient is better (i.e. due to positivity). 

Moreover, we can also directly show that the results in Fig 2, our main empirical results, would be worse if regular gradient descent is used instead of natural. 

Does this answer make sense? We would be glad to provide additional clarifications.

Reviewer comment (Theory):
Sobolev space is closely related to the RKHS with Matern kernel; see [1] below. Some supplementary comments on that relationship would be helpful for researchers interested in theoretical analysis of the learning algorithms.

Author response:
>Sobolev space is closely related to the RKHS with Matern kernel; see [1] below. Some supplementary comments on that relationship would be helpful for researchers interested in theoretical analysis of the learning algorithms.

Thank you for including the reference [1]. It contains a clear relevant results on the Matern kernel in context of Sobolev space and we will include it in the paper.   
Please see the discussion in the global "General Reply" comment on the relation between the SDO kernel and Matern kernels.

Reviewer comment (Novelty):
The estimator is similar to the one proposed in Ferraccioli's JRSS paper. Please clarify the main difference between them.

Author response:
>The estimator is similar to the one proposed in Ferraccioli's JRSS paper. Please clarify the main difference between them. 

We discuss the relation with the paper 

*Nonparametric density estimation over complicated domains, Ferraccioli et al,  JRSS 2020*

In the above paper, the authors also use a reparametrisation to ensure non-negativity of the density, and regularise using  a version of a Sobolev space. However, as we discuss in our introduction and literature review sections, 
that general principle is known since the 1980s for one dimension. The challenge is how one can implement the principle in $d>1$.  


The above mentioned paper *is solely restricted to two dimensions, d=2*. (both for applications and theory). In particular, to obtain their solutions, they discretize (triangulate) the domain of interest in $R^2$. 
This is clearly infeasible in any higher dimension. 


In contrast, the approach we develop, including the natural gradient optimisation and the use of Fisher divergence for parameter tuning, allow us to obtain good anomaly detection performance on datasets with $d>100$. 
Moreover, we consider the fact that such methods *are* useful in high dimensional situations is a significant contribution by itself, since such methods were not considered in recent literature in high dimensions. We will emphasize this more in the discussion in the paper. 
Would the reviewer agree that this is indeed a contribution of interest?

Reviewer comment (Evaluation):
Section 6.3 reports the ratio of negative values for the estimated function f. I'm not sure the reason why the ratio of negative values is important to assess the computational properties.

Author response:
> Section 6.3 reports the ratio of negative values for the estimated function f. I'm not sure the reason why the ratio of negative values is important to assess the computational properties.

This point is important and we now clarify it. 
As we discuss in the paragraph following Eq. (5), the main optimisation objective (5) is not convex as a function of $f$. Moreover, if one nevertheless tries to optimise (5) in $f$ (i.e., in $\alpha$, using $f_{\alpha}$) directly, one would obtain very poor results. The gradient descent will get stuck in a local optimum with poor values. 
We can provide an experimental demonstration of this if that would be of interest. 


However, we observe that (5) is convex on the non-negative cone, 
i.e., $f$ s.t. $f>0$ everywhere. (It is also sufficient to require  only $f(x_i)>0$ on data points $x_i$).    
Thus, if one restricts to such cone, one can obtain the optimal solution on that cone. We observe that such solutions are empirically much better. 
Next, we note that solving a constrained optimisation problem, requiring the solutions to be restricted to the cone is computationally difficult.  Moreover, since the cone is not closed (topologically), this is also not quite clear theoretically, since projection methods can not be applied. 


To resolve this, we observe that  *natural gradient* preserves the cone. 
This means that one can in fact do *unconstrained* optimisation. 
This is true for non-negative kernels, such as the Matern kernels (discussion following Equation (5) and Proposition 10), and is approximately true for the SDO kernel.  

In the experiment in Section 6.3 we show that empirically, for the SDO kernel, natural gradient indeed tends to preserve positivity, while the regular gradient does not. This demonstrates that natural gradient is indeed better for optimisation purposes. Note that, theoretically, this is one of the very few settings were we know *why* the natural gradient is better (i.e. due to positivity). 

Moreover, we can also directly show that the results in Fig 2, our main empirical results, would be worse if regular gradient descent is used instead of natural. 

Does this answer make sense? We would be glad to provide additional clarifications.

Reviewer comment (Writing):
The paper mentions on the top section of p. 2 that regularizing over \( |f^*|_{L_2} \) is a desirable property, but it is unclear what this means.

Author response:
> (ii) the paper mentions on the top section of p. 2 that regularizing over  $\|f\|_{L_2}^2$ is a desirable property: what does it mean?

Our intention here was to say by using the Sobolev type kernel, we automatically deal with functions that satisfy $\inf f^2(x)dx < \infty$. Such functions are densities up to a constant. Non integrable functions on the other hand can not represent densities (for the purposes of sampling from them, for instance). This is not a trivial condition, as there is no reason why an "arbitrary" function should be integrable.

Reviewer comment (Presentation):
The paper mentions the consistency of the estimator in the main manuscript, but delays all discussions about consistency to the supplementary materials. It would be helpful to bring up the main theorems for consistency in the main text.

Author response:
>The paper mentions the consistency of the estimator in the main manuscript, but delays all discussions about consistency to the supplementary materials - I believe that it would be helpful to bring up the main theorems for consistency in the main text.

We will definitely consider this. The challenge here is that there seems to be no other part of the paper which is non crucial enough to be moved to the supplementary instead.

Reviewer comment (Theory):
The paper proposes an RKHS norm that involves the L2 norm of the function, and also its derivatives at a certain degree \(m\), leading to a kernel with no analytical expression but a tractable spectral density (up to a normalization constant). The type of Sobolev norm \( \sum_{|\kappa|_1=m} ||D^{\kappa} f||^2 \) used is unusual, given that the original Sobolev norm involves derivatives up to degree \(m\): \( \sum_{|\kappa|_1\leq m} ||D^{\kappa} f||^2 \), and Sobolev spaces are norm-equivalent to the RKHS of a Matérn kernel. The authors should explain why their choice of Sobolev norm is useful, and what results would be obtained if one used the Matérn kernel instead of the SDO kernel.

Author response:
We comment on an additional point on equivalence of Sobolev spaces, which was mentioned in the review.  It is true for instance that the spaces corresponding to the Laplacian and SDO kernels are equivalent for all values of the Laplacian parameter $\sigma$ and SDO parameter $a$ (see the discussion above). However, this only means that the spaces *contain* the same set of functions. This in turn implies that the norms are equivalent, but the equivalence constants diverge with the dimension. Since for regularization we are interested in an actual value of the norm, the resulting algorithms will be different between such spaces. Consider for instance that $\|\cdot \|_1$ and $\|\cdot\|_2$ norms on $R^d$ are equivalent norms, but with drastically different regularisation properties. Thus just norm equivalence is too coarse a notion to distinguish (or unify) between different regularisation algorithms.

Reviewer comment (Writing):
The paper's main method is marked as "INER" in Figures 1 and 2, but the text makes no mention of what this stands for, raising the question of whether this is a mistake.

Author response:
>The paper's main method is marked as "INER" in Figures 1 and 2, but the text makes no mention of what it stands for: is this a mistake?

Yes, it should be RSR. Will be fixed.

Reviewer comment (Theory):
I'm not convinced by the motivation in Page 1, Paragraph 1. First, many nonparametric methods have been described for data of arbitrary dimension (including in some of the cited papers). However, standard minimax lower bounds show that high-dimensional non-parametric density estimation is statistically intractable, in terms of many performance metrics. Thus, I'm not convinced that a new method will perform well in high dimensions unless it makes some more explicit assumptions on the density being estimated. Second, while recent neural network based density estimation methods are technically parametric, the complexity of these models is so large that they often behave more like non-parametric methods; i.e., they can fit quite complex densities. So it's not clear that a new nonparametric method should outperform these models in practical applications.

Author response:
**Regarding point 1:**

>I'm not convinced by this motivation, for a two reasons...

>Thus, I'm not convinced that a new method will perform well in high dimensions unless it makes some more explicit assumptions on the density being estimated.

Please note that density estimation from a finite sample is inherently about bias of the estimator, i.e., about assumptions. Without any assumptions, there is no way of going from a finite set of points to a density.   
Thus all methods have a certain bias, and there no "best" method. This is well known in Anomaly Detection, for instance; see the ADBench paper. 
In some methods the bias is clear, while in some, like neural networks, little is understood about it. We specifically were interested in a method with as simple bias as possible: the norm of a derivatives of one fixed order. 

We emphasize that empirically, at least, for Anomaly Detection, the estimator is very useful.

Reviewer comment (Theory):
Section 3.1, just after Eq. (5): Although it seems reasonable in practice, the solution proposed here to avoid restricting the optimization to the positive cone \(\mathcal{C}\) is *ad hoc*, and it's not clear how this relates to the consistency guarantee (Theorem 11). Ultimately, in Appendix K, the paper assumes "positivity on the data points \(x_i\)", and it's not clear whether this condition is likely to hold as \(n \to \infty\). While the estimate might approach the true (non-negative) density, the number of points \(x_i\) is increasing. This appears to be a big hole in the consistency guarantee. The paper should analyze whether the "positivity on the data points \(x_i\)" condition holds as \(n \to \infty\), and whether any additional conditions are necessary to ensure this (e.g., it might suffice if the true density is bounded away from \(0\)?).

Author response:
First, we address the "positivity on the data points" question in the consistency result. This was raised in point 4 of the review and in the related Question 2. 

**positivity on the data points**:

The reason we only look for solutions in the cone (104) with strictly positive $f(x_i)>0$ is that the gradient of the loss is not defined otherwise (Eq. (6) or (106)).  Note that the functions in the cone satisfy 
$f(x_i)>0$ simply because we defined them to satisfy this, thus, there is no contradiction here.   
In addition, we do want to evaluate the gradient of the *true* density $v$ at the sample points. We believe this was the source of the issue in the review, when $n\rightarrow \infty$.  However, this also does not constitute a problem. 
Indeed, note that we do not require a lower bound on $f(x_i)$, we just need to know the gradient is well defined, i.e., $f(x_i)\neq 0$.  To this end, 
for any *continuous*  density $v^2$, we have the following: 

Let $\{x_i\}$ be an infinite sequence of iid samples from $v^2$. Then $v(x_i)>0$ for all $i$ with probability 1. (since choosing $x$ with $f(x) =0$ is a 0 probability event). 

As an example, let $v^2$ be either a Gaussain, or a uniform density on a sphere. In both cases samples will have $v^2(x_i)>0$. The infimum of such values will  certainly be 0 for the Gaussian but there is no contradiction in that. Such points, approaching 0,  will not be optimal, and again, we only need the gradient to be well defined.   

Does this resolve the issue? We would be glad to discuss it further.

Reviewer comment (Evaluation):
I didn't really understand the purpose of Section 5. The two curves in Figure 2(b) seem almost identical, up to a constant scaling factor. Since the \(y\)-axis is log likelihood, this just looks like INER = (KDE)\(^2\), up to a constant factor. Related to this, I didn't understand why "the gap between the values on the first and second cluster [being] larger for the RSR model" explains why RSR is better for anomaly detection -- what matters for anomaly detection is probably the *ratio of the between-cluster variation to the within-cluster variation*, but the within-cluster variation is also much bigger for RSR (INER) than for KDE.

Author response:
**Purpose of Section 5 (point 6):**


Our goal in Section 5 is to show that KDE and RSR can produce arbitrarily different results on the same data. Note that we do not claim here that either one is better for Anomaly Detection on the basis of this. As we mention above, it is unlikely that one can be better than the other universally. 
However, we believe it is still important to understand why RSR is truly a new estimator, not equivalent to KDE. Proposition 3 and Figure 1  show this. 
To show the estimators describe a different distributions, we consider the 
ratio $f(x)/f(x')$ where $x,x'$ are points in first and second cluster. 
If this ratio is larger, this means the estimator can change its value from one cluster to another faster.   (Here $f$ are the densities themselves, not the roots).

In Fig 1b we have shown the log values of KDE and RSR estimators on both clusters. We have normalised them so that they have similar value on the first. We then see that log values of RSR are lower, meaning larger ratio, as expected from Proposition 3.

Reviewer comment (Novelty):
A central idea of this paper is to estimate a square root of the target density function, in order to ensure non-negativity of the estimated density (after squaring). This is closely related to the proposal of [MFBR20], who propose a general framework for estimating non-negative quantities built on this idea. In the particular case of nonparametric density estimation, I think their proposal is very similar to the present paper's (namely, maximum likelihood with Sobolev norm regularization). They also propose various mechanisms to enforce the constraint that the density estimate integrates to 1. How exactly does the present paper's proposal differ from that of [MFBR20], and what, if any, are the advantages?

Author response:
Next we address the other major question, i.e., the relation to work [MFBR20]. 

**relation to [MFBR20]**:


At a very high level, the approach of [MFBR20] is indeed similar to ours, since they consider quadratic functions and regularisation with a version of an RKHS norm. 
The differences are both in scope and in details of the construction that enable computation.  From a computational viewpoint, they optimise in the space on non-negative matrices. This makes the problem convex without any need for positivity, but it requires $N^2$ parameters to encode such a matrix (their Theorem 1), and $O(N^3)$ computational cost per step (their top of page 6). Further, the optimisation is a constrained one, which is also significantly more difficult. It is unlikely their methods can work on something like ADBench benchmark, and indeed, they only test it on a 1 dim Gaussian with N=50. 

Second, while [MFBR20] is concerned with other applications as well, we 
investigate the density estimation problem much more closely. 
We prove consistency, which is completely orthogonal to all their results, we prove the method is different from KDE, and we do provide evaluation on a state of the art benchmark of datasets. 


Finally, they do treat the normalisation (their Prop 4, and paragraph below it). 
However, this is simply by a straight forward opening of the brackets. 
That is (in our case), if $f(x) = \sum_{i} \alpha_i k(x_i,x)$ then 
$\int f^2 (x)dx = \sum_{i,j} \alpha_i \alpha_j k(x_i,x)k(x_j,x)$. 

Thus, if one knows $\int k(x_i,x)k(x_j,x) dx$ for every $i,j$ one can compute  theoretically the normalisation constant. The issues with this are as follows: (a) This would involve summing $N^2$ floating point numbers, which would be inherently unstable for large N. (b) Computing 
$\int k(x_i,x)k(x_j,x) dx$ analytically is easy *only* for the Gaussian kernel. Even for exponential (Laplacian) kernel in $d>1$, we do not see any way to evaluate this analytically. Of course, for the SDO kernel, this also does not seem to be computable. Due to this reason, we have not pursued normalisation. 

It is worth noting however, that while the work of [MFBR20] advocates for using the computation above as a constraint, if one uses the Sobolev norms as we do, 
the integral is guaranteed to be finite, and it is thus sufficient to normalise only once, when the optimisation is finished.

Reviewer comment (Theory):
Is there a motivation for explicitly enforcing the \(L_2\) penalty \(\|f\|_2\), as opposed to simply enforcing the Sobolev pseudo-norm penalty \(\|D f\|_2\)?

Author response:
**Minor Questions:**

1) There are two reasons why having the term $\|f\|\_{L_2}^2$ in a norm such as (8) is desirable. First $\|Df\|\_{L_2}^2$ alone is only a semi-norm, 
(it zeros out all constant functions), and thus it doesn't create a proper RKHS, causing issues with the representer theorem which is essential to obtain computable models. Second, in our setting this also conveniently guarantees that the function $f$ has a finite integral, so it can be renormalised to be a density.

Reviewer comment (Theory):
Equation (9): I think there is an extra square in the definition of this inner product (i.e., \(\langle D^\kappa f, D^\kappa g \rangle^2\) should be \(\langle D^\kappa f, D^\kappa g \rangle\)).

Author response:
**Minor Questions:**

2) yes, the square is a typo.

Reviewer comment (Presentation):
Figure 1: What is "INER"? Should this be "RSR"?

Author response:
**Minor Questions:**

3) yes, its should be RSR.

Reviewer comment (Presentation):
Figure 2: What do the error bars indicate? Quantiles?

Author response:
**Minor Questions:**

4) yes, quantiles. The orange dot is the mean, the line in the middle is median. the large bar is 25%-75% quantiles.