A Probabilistic Deep Image Prior over Image Space
Abstract
The deep image prior regularises under-specified image reconstruction problems by reparametrising the target image as the output of a CNN.
We induce a prior over images by scoring
CNN outputs using a classical image reconstruction regulariser.
We translate this functional prior into weight space using a change of variables and propose an efficient linearised Laplace inference algorithm.
Hyperparameters are optimised with Type-II MAP.
We obtain pixelwise uncertainty estimates, which we show to be calibrated to the reconstruction error
1. Introduction
Inverse problems in imaging centre around the recovery of an unknown image x ∈ R d x from the corrupted measurement
[Equation]
where y δ ∈ R
d y is the noisy measurement data, A ∈ R d y ×
d x a linear forward operator, and η an i.i.d.
noise (e.g., Gaussian noise η ∼ N (0 , σ 2 y I ) ).
Many tomographic reconstruction problems take this form (e.g., computed tomography (CT), magnetic resonance imaging and magnetic particle imaging).
Due to the inherent ill-posedness of the reconstruction task (e.g., in the under-determined case d y (cid:28) d x ), suitable regularisation is crucial and is key for a successful recovery of x (Tikhonov and Arsenin, 1977; Engl et al., 1996;
Ito and Jin, 2014).
A successful approach is the deep image prior (DIP), introduced by Ulyanov et al.
(2018), which regularises the reconstruction by reparametrising x as the output of a deep convolutional neural network (CNN).
Liu et al.
and Baguer et al.
(2020) combine
DIP with the total variation (TV) regulariser, one of the most popular and well-established penalties for image reconstruction (Rudin et al., 1992).
In this work, we re-cast the deep image prior as a probabilistic prior over images. Within the probabilistic framework, a natural approach to regularise the ill-posed inverse imaging problem is to place a prior over the reconstruction.
Our prior scores the outputs of a neural network according to their TV seminorm.
We then translate this functional prior into weight space using the predictive complexity prior (PredCP) framework of Nalisnick et al.
We provide a linearised Laplace inference algorithm with computational [Figure] cost scaling in O ( d x d 2 y ) , and select hyperparameters via a Type-II MAP objective with a complexity O ( d 3 y ) .
Our method provides pixel-wise uncertainty estimates, which we show to be predictive of the reconstruction error.
We show that our TV-PredCP prior results in increased calibration, cf. fig.
A distinct feature of the proposed approach is that it keeps the mean reconstruction obtained using the traditional regularised DIP formulation, which allows us to leverage state of the art DIP training methods (Baguer et al.
; Barbano et al.
2. Deep Image Prior and Total Variation Regulariser
DIP (Ulyanov et al., 2018, 2020) aims at finding the minimiser of the fidelity (cid:107)
A x − y δ (cid:107) 2 , by assuming that the unknown x is the output of a CNN, x = f
( A † y δ , θ ) . As input it takes an approximate reconstruction A † y δ of x
(Barbano et al., 2021), which is constant and we can thus abbreviate our notation f ( A † y δ , θ ) = f ( θ ) .
A † is an approximate inverse, e.g., filter back-projection (FBP) in CT (Dudgeon and Mersereau, 1984).
θ ∈ R d θ denotes the network’s parameters to be learned.
The standard choice of the network architecture f is U-Net (Ronneberger et al., 2015).
Thus, DIP solves
[Equation]
and presents f ( θ ∗ ) as the recovered image.
Ulyanov et al.
(2018) use early stopping to avoid overfitting to noise in y δ .
However, the need for early stopping can be avoided by properly regularising the TV of the network output, leading to an objective
[Equation]
[Figure]
where λ > 0 balances the two terms.
TV( f ( θ )) denotes the anisotropic TV seminorm of the network output f ( θ )
[Equation]
where X ∈ R h × w denotes the image vector x reshaped into a matrix of height h by width w .
TV has been widely used in image processing (Chambolle et al., 2010), and more recently also found to improve performance of DIP methods (Liu et al., 2019; Baguer et al., 2020).
See Appendix C for a more in-depth discussion on TV as a regulariser.
3. Transforming the DIP into a Bayesian Prior over Images
We can interpret the loss in eq.
(3) as a maximum a posteriori (MAP) objective, where we have imposed a probabilistic prior over neural functions f that favour smoothness in a TV sense
[Equation]
with
[Equation]
We combine this prior with sum of squares projection loss between f ( θ ) and y δ .
The latter is obtained from a Gaussian conditional likelihood: p ( y δ | f )
=
N ( y δ ; A f , σ 2 y I ) with σ 2 y =1 .
Despite the space of functions providing a convenient canvas in which to express our prior beliefs, it significantly complicates subsequent probabilistic reasoning with NNs (Sun et al., 2019;
Burt et al., 2020).
Instead, we consider the following weight-space hierarchical model
[Equation]
in which, building upon Fortuin et al.
(2021), the spatial covariance Σ θ ( (cid:96) , σ 2 θ ) among parameters is specified as
[Equation]
where i indexes a convolutional filter and j the spatial location of a specific parameter within a filter.
∆( · , · ) is the Euclidean distance between filter pixels.
(cid:96) d acts as a characteristic length-scale and σ 2 d is the marginal prior variance.
These two parameters are defined per convolutional block d ∈ { 1 , 2 , . . .
, D } in the CNN such that (cid:96) = [ (cid:96) 1 , (cid:96) 2 , . . .
, (cid:96) D ] and σ 2 θ =
[ σ 21 , σ 22 , . . . ,
σ 2 D ] .
We do not place a prior over the latter, instead treating it as a hyperparameter.
A diagram showing the correspondences between blocks 1 , . . . , D and U-Net architectural components is included in appendix A.
By choosing a larger (cid:96) , we constrain the parameters in the convolutional filters and thus effectively enforce smoothness in the output.
Thus, a prior placed over the filter length-scale (cid:96) can act as a surrogate for the TV prior in eq.
To make this connection explicit, we construct a predictive complexity prior (PredCP) (Nalisnick et al., 2021) over (cid:96) : p
[Equation]
(cid:96) )
= p
[Equation]
[Equation] d (cid:12)(cid:12)(cid:12)(cid:12) , with (7)
[Equation]
Here κ d is the expected smoothness (in a TV sense) of the network output f ( θ ) induced by the prior over parameters of block d , when the parameters of all other blocks are set to some reference value θ ∗ .
The random variable κ d is modelled by an exponential distribution π ( κ d ) =
Exp ( κ d ; 1) and is related to the filter lengthscale (cid:96) d by means of the change of variable formula.
The separable factorisation of p ( (cid:96) ) expresses our belief that blocks placed at different positions within the network will have different sized contributions to
TV( f ( θ )) .
The definition of κ d ensures that we can measure the contribution of every block independently of the rest, ensuring the dimensionality match formally needed in the change of variables.
We discuss the bijectivity of the mapping between (cid:96) d and κ d in Appendix E.
In summary, by using the predictive complexity prior, we translate the image space smoothness prior in eq.
(4) into a prior on the spatial smoothness of our convolutional filter parameters.
4. Linearised Laplace Inference in the Low-Dimensional Dual Space
Even when working in parameter space, inference in the model from eq.
(5) is computationally intractable.
We resort to locally linearising the U-Net around θ ∗
[Equation]
with
[Equation]
where J = ∇ θ f ( θ ) | θ = θ ∗ ∈ R d x × d θ denotes the Jacobian matrix of the network with respect to θ .
The covariance structure of this model matches that of a generalised linear model with
J as a design matrix.
It can be written over image space as Σ f = J Σ θ ( (cid:96) , σ 2 θ ) J (cid:62) ∈ R d x × d x .
The Jacobian term enforces the structure of CNN while the weight prior covariance enforces TV smoothness in eq.
We display draws from this model in fig.
For brevity, we hereon omit the dependence on kernel parameters ( (cid:96) , σ 2 θ ) .
4.1. Efficient Posterior Predictive Computation
The linearised posterior over the image space is obtained in closed form using linearGaussian conjugacy p ( f | y δ ) =
N ( f ; µ f | y δ , Σ f | y δ )
∝ N ( y δ ; A f , σ 2 y I ) N ( f ; h ( 0 ) , Σ f ) .
Its mean is µ f | y δ =
f ( θ ∗ ) , leaving the DIP’s output unchanged.
Its covariance is Σ f | y δ
=
(Σ − 1 f + σ − 2 y A (cid:62) A ) − 1 .
Using the Woodbury matrix identity, we adopt the dual form
[Equation]
and obtain an expression in terms of the inverse of the observation space posterior covariance Σ y
=
A Σ f A (cid:62) + σ 2 y I ∈
R
d y × d y .
The computational complexity of eq.
(10) scales as O ( d x d 2 y ) as opposed to O ( d 3 x ) or O ( d 3 θ ) for the image space or parameter space views, respectively.
[Figure]
4.2. Type-II MAP Learning of Hyperparamters with the TV-PredCP
Once a mode θ ∗ of the posterior p ( θ | y δ ) has been found with the MAP criterion eq.
(3), we select hyperparameters ( σ 2 y , (cid:96) , σ 2 θ ) that will provide well calibrated error-bars by maximising the volume of the posterior mode.
This is done with the linearised Type-II MAP objective:
[Equation]
where C is independent of our hyperparameters.
See Appendix D.2 for a derivation of (11).
We again employ the dual formulation to keep our update cost O ( d 3 y ) .
As shown in fig.
3, the PredCP acts not only on the convolutional filters’ lengthscale (cid:96) , but also informs their prior marginal variances σ 2 θ which parametrise the bijection in eq.
Additionally, as described in Appendix D, the linearised model allows us to compute κ d in closed form.
[Table]
5. Experimental Evaluation
We reconstruct CT measurement data y δ simulated from the Kuzushiji-MNIST (KMNIST) dataset consisting of 28 × 28 grayscale images of Hiragana characters (Clanuwat et al., 2018).
The linear forward map A is given by the discrete Radon transform.
For each simulated measurement y δ , we train f ( θ ) via (3), fixing the input to a coarse FBP reconstruction, for 25 k iterations.
We then optimise the hyperparameters ( σ 2 y , (cid:96) , σ 2 θ ) with Type-II MAP using (11) for 1.25k iterations.
We also consider omitting the PredCP in (11), reverting to the standard marginal likelihood (MLL).
Qualitative Results
Figure 1 shows an exemplary character which we recover with both
FBP and DIP.
The reconstruction problem is highly ill-posed (10 directions), thus FBP reconstructions exhibit strong artefacts.
As expected, the DIP reconstruction obtains 12dB higher peak noise-to-signal ratio (psnr).
The pixel-wise standard deviation provided by our method correlates strongly with the DIP reconstruction error.
However, the hyperparameters found via MLL lead to excessively large uncertainty.
TV-PredCP regularisation leads to hyperparameters that produce better-calibrated uncertainty.
In fig.
3 we show how the TV-PredCP drives (cid:96) to larger values and σ 2 θ to be smaller.
The use of the PredCP restricts our prior, and thus posterior, to functions that are smooth in a TV sense, resulting in smaller error-bars.
This can also be seen in fig.
2, where samples from the TV-PredCP model are smoother and present less artefacts.
See Appendix A for additional experimental figures.
Quantitative Results
We consider 20 and 10 directions taken uniformly from 0 ◦ to 180 ◦ .
We add 5% (low-noise) and 10% (high-noise)
Gaussian noise to A x .
We conduct each experiment using the same 10 randomly chosen KMNIST test images.
We find some images to include spurious large valued pixels far away from the region of interest, violating the modelling assumption that x is noiseless eq.
DIP treats these pixels as noise, does not reconstruct them and the hyperparameter optimisation eq.
(11) converges to larger values of σ 2 y .
We thus add the correction term σ 2 y ( A (cid:62) A ) † to our predictive variance and provide further discussion in appendix D.1.
We study the standalone effect of this correction by treating it as a baseline, labelled “MLL σ 2 y ”. We also consider an isotropic unit variance noise model.
Table 1 show test log-density, obtained by method (predictive posterior given in section 4) and baselines, for all experimental settings.
All methods share the same network parameters and thus the same mean reconstruction.
Consequently, higher values in logdensity indicate better uncertainty calibration. In all settings, capturing model uncertainty significantly improves performance.
There is a clear but smaller benefit in employing the TV-PredCP prior which is most notable in the low-noise setting.
6. Limitations and Conclusion
We have demonstrated how a probabilistic formulation of the DIP can yield well-calibrated uncertainty in a computational tomography setting.
Our results suggest that the TV, apart from acting as a regulariser, can yield a performant probabilistic prior over images for the inverse problem.
Open questions are whether performance can be improved by giving a fully probabilistic treatment to (cid:96) and whether the strength of the TV can be chosen with an MLL objective.
The KMNIST setting under consideration is quite small.
We will investigate how to scale our method to larger output spaces, where Jacobian computation might be intractable and the observation dimension d y may still be large. Acknowledgments
The authors would like to thank Marine Schimel, Alexander Terenin, Eric Nalisnick, Erik
Daxberger and James Allingham for the fruitful discussions.
We also thank Johannes
Leuschner for his help in the preparation of the manuscript. R.B. acknowledges support from the i4health PhD studentship (UK EPSRC EP/S021930/1), and from The Alan Turing Institute (UK EPSRC EP/N510129/1). The work of BJ is partially supported by UK EPSRC grant EP/T000864/1.
JA acknowledges support from Microsoft Research, through its PhD Scholarship Programme, and from the EPSRC.
This work has been performed using resources provided by the Cambridge Tier-2 system operated by the University of
Cambridge Research Computing Service (http://www.hpc.cam.ac.uk) funded by EPSRC
Tier-2 capital grant.
Appendix A. Additional Experimental Details
In our experiments we use a down-sized version of U-Net.
The reduced output dimension d x , and the simplicity of the reconstruction problem allow us to adopt a shallow architecture without compromising on reconstruction quality.
Figure 4 shows the network architecture used for this work.
We adopt the U-Net architecture proposed by Barbano et al.
(2021), with the only difference being that we remove group-normalisation layers.
Each light-blue box corresponds to a multi-channel feature map.
The number of channels is set to 32 at every scale.
The arrows denote the different operations.
We give a Bayesian treatment to all parameters in convolutional layers (approx.
78k parameters).
We denote with ∗ the operations that do not receive a Bayesian treatment (1x1 convolution mixing layers).
We identify 4 blocks: Down 0 ( d = 0 ), Down 1 ( d = 1 ), Up 0 ( d = 2 ) and Up 1 ( d = 3 ).
The Down blocks consists of a 3 × 3 convolution followed by a 3 × 3 convolution operation; the Up blocks instead consists of two successive 3 × 3 convolutional operations.
[Figure]
A.1. Additional Experimental Figures
[Figure]
m a x : 2 .e - 01 , m i n : 3 .e -| x − x ∗ | m a x : 2 .e - 01 , m i n :
[Figure]
[Figure]
Figure 5 and fig.
6 show additional reconstructions of two exemplary characters taken from the KMNIST dataset.
We also include additional samples from both the linearised prior N ( 0 , Σ f ) and posterior N ( f ( θ ∗ ) , Σ f | y δ ) for the image from fig.
1 in fig.
7 and fig.
8, respectively.
A careful visual inspection suggests that the sample obtained using the hyperparameters optimised with Type-II MAP presents higher degree of smoothness, along with less artefacts, as one may expect from the construction of the PredCP prior.
Appendix B. Discussion on Previous Work on Bayesian DIP
Cheng et al.
(2019) perform stochastic gradient Langevin dynamics (SGLD) inference with an isotropic Gaussian prior over the network parameters.
Laves et al.
Tölle et al.
(2021) use both Monte Carlo (MC) dropout inference, and then Gaussian mean-field variational inference.
These works resorted to a Bayesian treatment of DIP as a way to alleviate the need for early-stopping.
However, Baguer et al.
(2020) show that the TV regulariser is a simpler yet more effective approach.
Our proposed probabilistic method aims at producing prior and predictive distributions in the image space, which is in its goal essentially different from existing probabilistic treatment of DIP (Cheng et al., 2019; Tölle et al., 2021).
Indeed, Dropout and SGLD have been found to not be very robust methods for uncertainty estimation (Snoek et al., 2019;
Antorán et al., 2020; Daxberger et al., 2021).
Furthermore, the previous work is sampling based, and does not place an explicit likelihood function over the image space.
To the best of our knowledge there is little existing work on the application of predictive uncertainty to deep tomographic reconstructions, with the most notable being (Seeger and Nickisch, 2011). However, there is a rapidly growing body of literature on the utility of uncertainty for human computer interaction (Antorán et al., 2021; Bhatt et al., 2021). With our work we hope to take steps towards building a foundation for more research in this area.
Appendix C. Discussion on the TV Regulariser as a Prior
We may question whether TV is the ideal regulariser for the KMIST dataset.
The TV regulariser enforces sparsity in the overall image gradients, which in turn induces smoothness in the image.
A TV regulariser is highly recommended when we observe sparsity in the edges present in an image, especially when the edges are a small fraction of the overall image pixels.
That is often the case in high-resolution medical images.
Intuitively, the higher the resolution of the image is, the higher the sparsity level of the edges is.
However, in the KMIST dataset, due to the low resolution of the images, the edges constitute a considerable fraction of the total pixels.
Therefore, a TV regulariser could be sub-optimal.
In KMNIST, it is difficult to clearly distinguish (in TV sense)
what is part of the image structure, what is part of the background.
The stroke is only a few pixels wide, and ground-truth pixel values are generated through interpolation (Clanuwat et al., 2018).
In future work, we plan to compare different regularisers, such as an L 1 norm.
It is tempting to think that we do not need the PredCP machinery to translate the
TV regulariser into the weight space.
Indeed, the Laplace approximation simply involves a quadratic approximation around a mode of the log posterior, without placing any requirements on the prior used to induce said posterior.
Along this line of reasoning, we can decompose the Hessian of the log posterior log p ( θ | y δ ) into the contributions from the likelihood and the prior as
[Equation]
and quickly realise that the log of the anisotropic TV prior chosen to be p ( f ) ∝ exp( − λ TV( f )) as in eq.
(4) is only once differentiable.
Ignoring the the origin (where the absolute value function is non-differentiable), we obtain:
[Equation]
Thus, a naive application of the Laplace approximation would eliminate the effect of the prior, leaving the posterior ill defined.
Appendix D. Derivation of Method Proposed in SectionD.1. Posterior Predictive Covariance
We provide an alternative derivation for the posterior predictive covariance, where the probabilistic reasoning is performed in the weight space.
We start from the model introduced in eq.
[Equation]
and write down the linearized Laplace approximate posterior distribution over weights:
[Equation]
with
[Equation]
This expression is obtained by substituting the NN f ( θ ) with the linearised surrogate h and computing the product of Gaussian prior and posterior.
We refer to (Mackay, 1992; Immer et al., 2021) for a derivation.
Then we rewrite the above expression using the Woodbury matrix inversion identity:
[Equation]
The predictive distribution over images can be built by marginalising the NN parameters in the conditional likelihood p ( x | y
[Equation]
As mentioned in section 5, KMNIST presents spurious high valued pixels away from the region containing the handwritten character. This presents a problem due to our modelling assumptions in eq.
(1), assuming x is noiseless, and thus our likelihood function from eq.
(5) being defined over the space of observations y δ .
We translate the uncertainty induced by the observation noise to the space of images by computing the conditional log-likelihood Hessian with respect to f : −∇ 2 f log p ( y δ | f ) =
σ − 2 y A (cid:62) A ∈ R d x × d x .
This matrix is of rank at most d y , which can be much smaller than d x due to the ill-conditioning of the reconstruction problem, and therefore cannot act as a proper Gaussian precision matrix on its own.
However, we incorporate the noise uncertainty from the observation subspace into the image space x by adding the pseudoinverse σ 2 y ( A (cid:62) A ) † to the predictive covariance.
We move forward with weight marginalisation when using the linear model to recover the predictive distribution:
[Equation]
[Figure]
D.2. Laplace Marginal Likelihood and Type-II MAP eq. (11)
The Laplace method provides an estimate of the model evidence, also known as marginal likelihood (MLL) or Type-II maximum likelihood, based on a quadratic approximation to the volume of a mode θ ∗ of the log joint log p
( y δ , θ ) (Mackay, 1992):
[Equation]
where the Hessian determinant of the joint log-density with respect to the model parameters
H = −∇
2 θ log p ( y δ , θ ) θ
= θ ∗ captures the width of the mode θ ∗ in question.
Expanding the above expression we have:
[Equation]
[Equation]
Note that the observation conditional log-density log p ( y δ | θ ∗ ) and the weight-mode log prior density log p ( θ ∗ ) are given respectively by
[Equation]
Accordingly, the Hessian term H can be decomposed into
[Equation]
Under the linearised approximation (i.e., substituting f ( θ ) with its first-order Taylor expansion h = f ( θ ∗ ) + J ( θ − θ ∗ ) ), we obtain:
[Equation]
Trivially
[Equation]
we have −∇ 2 θ log p ( θ ∗ ) =
Σ − 1 θ .
By the matrix determinant lemma, the determinant of H is given by
[Equation]
Finally, recall from eq.
(6) that Σ θ depends on hyperparameters ( σ 2 θ , (cid:96) ) [Equation] which is explicitly indicated for the purpose of derivation.
Thus, the expanded linearised Laplace model evidence is given by
[Equation]
where C captures all terms constant with respect to ( σ 2 y , σ 2 θ , (cid:96) ) .
Recall that Σ y ( σ 2 θ , (cid:96) ) =
AJ Σ θ ( σ 2 θ , (cid:96) ) J (cid:62) A (cid:62) .
Next we turn to the Pred-CP TV prior over (cid:96) : log p [Equation] [Equation] =
[Equation]
Combining this with eq.
(17), we obtain a Type-II maximum a posteriori (MAP) objective from eq.
[Equation]
d (cid:12)(cid:12)(cid:12) (cid:12) + C.
Figure 9 shows the evolution of standard marginal likelihood (MLL) and Type-II MAP during optimisation along with their individual components.
The TV-PredCP introduces additional constraints into the model by encouraging the prior to contract (stronger parameter correlations and smaller marginal variances as shown in fig. 3).
This prior concentrates a higher density on θ ∗ .
In turn, this results in a more contracted posterior which we observe as a larger Hessian determinant.
The conditional data density is unaffected by the
TV-PredCP.
Appendix E. Monotonicity of the TV in the Prior Lengthscales
In order to apply the change of variables formula in eq.
(7), we require bijectivity in the relationship between (cid:96) d and κ d := E p ( θ d | (cid:96) d ) (cid:81) Di
=1 ,i (cid:54) =
d δ ( θ i − θ ∗ i )
[TV( f ( θ ))] .
In the current setting, both variables are one-dimensional, making this constraint easier to satisfy.
In fact, it suffices to show monotonicity between the two.
We estimate the relationship between this variable pair empirically for every convolutional block in the U-net using Monte Carlo.
The results are displayed in fig.
In practise, we work with the linearised model in eq.
(9) for the purposes of inference.
In fig.
11, we show very compelling numerical evidence for the monotonicity.
We observe that κ increases in (cid:96) since large values for (cid:96) lead to an increased marginal variance σ 2 θ over images.
Just as expected, after fixing the marginal variance to 1, we observe that the lengthscales have a monotonically decreasing relationship with the expected TV.[-Authors 0. κ × 10 
 Down    κ × 10 3 Down0.-]
[-κ × 
 Up2. κ × 
 Up10 −10‘ 1.-][Figure]
[-κ × 10 3 Down10 −10‘ 1. κ × 10 3 Down10 −10‘ 1. κ × 
 Up10 −10‘ 1.-](cid:96)
[-κ × 
 Up-][Figure]
However, analytically can be as matches
[Equation]
assuming that the output is a one-dimensional signal so there is only one derivative to simplify the discussion.
First we interpret the distribution of h ( θ ) i − h
( θ ) i +1 .
Note that h ( θ ) can be written as h ( θ )
=
C + J ( θ − θ ∗ ) , by slightly abusing the notation C to denote the terms constant with respect to (cid:96) d and i indexes an entry of ( J θ ) ∈
R d x .
Thus, we can rewrite this expression as an inner product between two vectors:
[Equation]
where J i ∈ R 1 × d θd denotes our NN’s Jacobian for a single output pixel i (i.e., the i th row of the Jacoian matrix J, corresponding to the block parameters θ d , which has a length d θ d ) and v i
=
J i − J i +1 ∈ R 1 × d θd , i = 1 , . . .
, d x − 1 .
Now, recall that the block parameters θ d is distributed as
[Equation]
in the expectation in (18), whereas the remaining parameters are fixed at the mode θ ∗ j , j (cid:54) =
d , i.e., (cid:81) Dj =1 ,j (cid:54) = d δ ( θ j − θ ∗ j ) .
Let V d ∈ R ( d x − 1) ×
d θd correspond to the stacking of the vectors v i ∈ R 1 × d θd , i.e., the Jacobian of the network output with respect to the weights in convolutional group d .
Since the affine transformation of a Gaussian distribution remains
Gaussian, V d θ d is distributed according to
[Equation]
Note that the matrix V d Σ d V (cid:62) d is not necessarily invertible, and if not, as usual, the inverse covariance should be interpreted in the sense of pseudo-inverse.
Let a
=: V d θ d ∈ R d x − 1 .
Then we can rewrite our quantity of interest as
[Equation]
The distribution of | a i | can be expressed in terms of the truncated normal distributions.
It is known that the expectation increases monotonically with the variance v i Σ d v (cid:62) i .
It remains to examine the monotonicity of v i Σ
d v (cid:62) i in (cid:96) d .
Indeed, by the definition of Σ d , we have
[Equation]
Direct computation gives
[Equation]
and thus
[Equation]
Then it follows that if the vectors v i were arbitrarily, the monotonicity issue essentially rests on the positive definiteness of the associated derive kernel. For example, for the standard Gaussian kernel e [Equation]
[Equation]
(cid:96)d (i.e., ∆ is the squared Euclidean distance), we need to verify the positive definiteness of the kernel k ( x, y )
= ( x − y ) 2 e − ( x − y )2 (cid:96)d .
This issue seems generally challenging to verify directly, since ( x − y ) 2 is not a positive semidefinite kernel by itself on R , even though the Gaussian kernel e − ( x − y )2 (cid:96)d is indeed positive semidefinite.
Thus, one cannot use the standard Schur product theorem to conclude the positive definiteness.
Alternatively, one can also compute the Fourier transform of the kernel k ( x ) = x 2 e − xdirectly, which is given by
[Equation]
Clearly, the Fourier transform is not positive over the whole real line R .
By Bochner’s theorem, this kernel is actually not positive.
The fact that the kernel is no longer positive definite makes the analytical study challenging.
This indicates the risk for a potential nonmonotonicity in (cid:96) .
Nonetheless, we emphasise that this condition is only sufficient, but not necessary.
We leave a full investigation of the monotonicity to a future work, given the the compelling empirical evidence for monotonicity in both the NN and linearised settings.