Variational Determinant Estimation with Spherical Normalizing Flows
Abstract
This paper introduces the Variational Determinant Estimator (VDE), a variational extension of the recently proposed determinant estimator discovered by Sohl-Dickstein (2020).
Our estimator signiﬁcantly reduces the variance even for low sample sizes by combining (importance-weighted) variational inference and a family of normalizing ﬂows which allow density estimation on hyperspheres.
In the ideal case of a tight variational bound, the
VDE becomes a zero variance estimator, and a single sample is suﬃcient for an exact (log) determinant estimate.
1. Introduction
The computation of the (log) absolute determinant of matrices is a problem that is encountered in machine learning in areas such as normalizing ﬂows (Rezende and Mohamed, 2015;
Dinh et al., 2016).
Sohl-Dickstein (2020) connects the inverse absolute determinant of A with an expectation over matrix-vector products:
[Equation]
where samples are drawn from a uniform distribution on the n − 1 dimensional hypersphere.
For better readibility we will write shorthand U ( s ) for the uniform spherical distribution in the following.
A common unbiased estimator for Equation 1 is then given via Monte Carlo (MC) integration:
[Equation]
with
[Equation]
Sohl-Dickstein (2020) has empirically shown that the na¨ıve MC approach of Equationcan have problematic high variance, meaning that we need around 10 6 samples to correctly estimate even a 10 × 10 matrix.
This paper extends the MC determinant estimator using Spherical Normalizing Flows to introduce the Variational Determinant Estimator.
This new estimator achieves lower variance and, as a result, requires fewer samples for accurate estimates.
For an example see Figure 1.
1.1. Spherical Normalizing Flows
Normalizing ﬂows (Rezende and Mohamed, 2015; Dinh et al., 2016; Kingma and Dhariwal, 2018) are generative models which transform a base distribution
π ( z ) on a space Z into a more complex distribution q ( s ) on another space S via a diﬀeomorphism f : Z → S .
The relationship between those is given by the change of variables formula:
[Equation]
where J f is the Jacobian.
Most work has been done when Z and S are Euclidean spaces with ﬂat geometry.
Flows for hyperspherical geometries were introduced in Rezende et al.
(2020), that is f : S n → S n is a diﬀeomophism.
For these ﬂows we usually choose a uniform base distribution π ( z )
=
U ( z ) since the underlying spaces are compact.
As a consequence, q ( s ) is a distribution on the hypersphere parametrized by complicated invertible functions that we can straightforwardly sample from and compute the likelihood.
2. The Variational Determinant Estimator
To estimate the determinant and log determinant more eﬃciently with less variance, we introduce the Variational Determinant Estimator:
[Equation]
which by Owen (2013) has the least variance when q ( s ) ∝ U ( s ) (cid:107) As (cid:107) − n .
An example to achieve this proportionality is to minimize the divergence
[Equation]
where U ( s ) (cid:107) As (cid:107) − n is treated as an (unnormalized) probability distribution and Z is an unknown normalization constant which does not inﬂuence the gradient.
To avoid clutter with unnecessary constants, we drop Z in the following and note that the resulting KL divergence is an abuse of notation because the well-known properties such as non-negativity do not necessarily hold anymore for KL( q ( s ) ; U ( s ) (cid:107) As (cid:107) − n ).
Using this divergence has the additionally desired eﬀect that:
[Equation]
As [Equation] , and thus the negative KL gives a lower bound on the log absolute determinant of A −due to Jensen’s inequality.
Consequently, this gives a direct method to estimate the log absolute determinant of A using the upper bound KL( q ( s ) ; U ( s ) (cid:107) As (cid:107) − n ) which is tight when q ( s ) ∝ U ( s ) (cid:107) As (cid:107) − n .
In this ideal case, the VDE becomes a zero variance estimator, see Goli´nski et al.
(2019) Section 2.1.
The proposal distribution q is modeled by a ﬂow introduced in Section 1.
If we substitute q in KL using the change of variables formula in Equation 3 with a normalizing ﬂow f , and we choose a uniform base distribution π ( s )
=
U ( s ), the objective simpliﬁes:
[Equation]
Equation 4 is optimized via naive Monte Carlo integration.
The result of such a learned q in the case of a 3 × 3 matrix A is illustrated in Figure 1 and the optimal proposal distribution is visualized in Figure 3 of the Appendix A.1.
3. Related Work
Importance sampling has a long history as a study object.
Hesterberg (1988) introduced extensions for the importance weights such as regression or non-linear exponential estimates to allow the method to be eﬀectively applied in a wider range of settings such as multivariate outputs.
Kingma and Welling (2014); Rezende et al.
(2014) have introduced deep learning-based variational inference and Burda et al.
(2015) have shown tighter bounds with importance-weighted variational inference.
Although these works were originally aimed at estimating log probabilities, they can be more generally be applied to marginalize a probabilistic latent variable.
M¨uller et al.
(2019) utilize ﬂows to learn a proposal distribution for importance sampling in a Euclidean space.
Goli´nski et al.
(2019) introduce amortized Monte Carlo integration, which combines diﬀerent proposal distributions for better estimates.
Normalizing Flows (Tabak and Turner, 2013; Rezende and Mohamed, 2015) are an attractive generative model to learn distributions because they admit exact likelihood evaluation, and they are fast to sample from as opposed to autoregressive models.
There have been many advances for ﬂows on Euclidean manifolds (Dinh et al., 2016; Kingma et al., 2016; Chen et al., 2019;
Perugachi-Diaz et al., 2020).
Recently, Gemici et al.
; Rezende et al.
(2020) have introduced normalizing ﬂows for hyperspheres.
As a result, it is now also possible to learn expressive distributions on hyperspheres with exact likelihood estimates and eﬃcient sampling.
4. Results
In this section, we demonstrate the performance of our method in determinant estimation.
We consider two cases: estimating the determinant of randomly sampled 10 × 10 dense matrices and estimating the determinant of a convolutional layer.
4.1. Dense Matrix
The determinant is estimated for ﬁve 10 × 10 matrices where the entries are sampled from unit Gaussians, see the Appendix A.2 for the speciﬁc matrices.
The spherical ﬂow utilizes a Moebius transformation for the circle part with N C = 12 number of centers and Neural
[Figure]
[Table]
[Figure]
Spline ﬂows (Durkan et al., 2019) with N B = 16 number of bins for the interval part, see
Rezende et al.
(2020) for details of the architecture and the parameters.
Furthermore, we stacked N F = 8 ﬂows on top, used coupling layers, and trained the models for 10 k iterations with a batch size of 1024.
Flows based on autoregressive masking are also possible.
The results can be seen in Table 1, where we present the mean of the relative absolute diﬀerences of the estimated determinant in comparison to the true absolute determinant.
The variational determinant estimator achieves even for a low sample size of 10 2 low relative diﬀerences, whereas in contrast, the na¨ıve Monte Carlo estimate still has high variance throughout all sample sizes.
4.2. Convolutional Layer
In this experiment, the (log) determinant of a convolutional layer is estimated.
The reason for this experiment is that these types of sparse linear transformations often occur in deep learning, and they typically have cheap matrix-vector products.
We manually reconstructed the 16 × 16 equivalent matrix W of a convolution of a 3 × 3 ﬁlter with an 4 × 4 image, see Appendix A.3 for the ﬁlter.
The determinant of W equals the determinant of the convolution operation.
We chose the same architecture as in our previous experiment but ran the experiment this time for 40 k iterations with the same batch size.
Note that the
[Table]
parametrization of our spherical ﬂow is fully connected, and better optimization behavior is expected when the parametrization would be convolutional.
Figure 2 and Table 2 show the results of the experiment in terms of absolute and log abs.
determinant estimates.
We observe the same behavior as in the previous experiment:
The variational determinant estimator achieves low errors with already low sample sizes, whereas the MC determinant estimator is not able to capture the determinant correctly with even high sample sizes.
5. Conclusion
In this paper [Table] we introduced the Variational Determinant Estimator, which achieves with low sample sizes high accuracy in estimating a determinant of a linear operator.
Interestingly, the estimator in its original variant and the VDE allows estimation if only matrixvector products are available.
In our experiments, we considered an oﬄine setting where a large number of samples are required ﬁrst to optimize the model.
However, in future work, VDE could also be applied in an online, moving target settings.
In this case, small updates to the matrix A would only require small updates to the density model q ( s ).
A perpendicular direction for VDE would be to estimate the Jacobian determinant of a function f .
The Jacobian would depend on the input x , and the density model q ( s | x ) can be amortized.
Additionally, not the entire
Jacobian but only Jacobian-vector products would be required to estimate the determinant.
Code to reproduce the results and to enable further research concerning the Variational Determinant Estimator and spherical normalizing ﬂows will be made public in the git repository 1 of one of the authors at a later time point.
Appendix A. Experimental Details A.1. Cover Density
The cover image of this paper illustrates the learned proposal distribution q ( s ) corresponding to the matrix
A =   − 0 .0 .
6741 − 0 .0 .1 .0 .−
1 . 2754 − 0 .1 .
 , which is created with torch.randn(3,3) and torch manual seed 15.
The optimal proposal distribution q ∗ ∝ (cid:107) As (cid:107) − n is illustrated in Figure 3.
We trained the model for 10k iterations and in contrast to the architecture in Section 4, we used N F = 6 ﬂows with autoregressive masking and Neural Spline ﬂows for both the spherical part and the interval part of S 1 × [ − 1 , 1] with N B = 32 bins, see again Rezende et al.
(2020) for details.
A.2. Dense 10 × 10 Matrices
In this section we publish the absolute determinants in Table 3 and the 10 × 10 matrices of our experiment in Section 4.1.
Numbers are rounded to two decimals.
[Table]
A 1 =   − 1 .
[Table]
[Table] [Table] [Table] [Table]
 A 2 =    
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table]
02 − 2 .2 .1 .0 .
14 − 1 .0 .0 .1 .
33 − 0 .
88 − 0 .
73 − 0 .2 .− 0 .
11 − 0 .0 .
68 − 1 .0 .0 .0 .0 .0 .
41 − 0 .− 0 .0 .0 .
7 − 1 .1 .
77 − 0 .
79 − 0 .1 .0 .0 .1 .
24 − 0 .0 .
41 − 1 .
02 − 0 .
64 − 0 .
21 − 1 .0 .0 .
6 − 0 .0 .
71 − 0 .
91 − 0 .0 .1 .
13 − 0 .1 .
85 − 0 .0 .
29 − 1 .0 .
52 − 1 .0 .
48 − 2 .1 .
52 − 0 .1 .
26 − 1 .
42 − 0 .
02 − 1 .− 1 .
01 − 1 .0 .
42 − 0 .1 .
− 0 .
04 − 0 .0 .
52 − 1 .
91 − 1 .0 .− 0 .
1 − 0 .
39 − 0 .0 .
21 − 0 .0 .0 .0 .
77 − 0 .
  
A 3 =   − 0 .
15 − 1 .
65 − 0 .0 .1 .− 0 .0 .0 .0 .
23 − 1 .0 .
61 − 1 .
81 − 0 .0 .0 .2 .
36 − 0 .0 .0 .0 .1 .
91 − 1 .
44 − 0 .0 .
96 − 2 .
56 − 0 .
01 − 1 .0 .−
2 .0 .0 .−
1 . 3 − 0 .−
9 − 0 .0 .0 .1 .
28 − 0 .
57 − 0 .0 .
55 − 1 .
34 − 1 .
37 − 1 .
29 − 0 .
28 − 0 .0 .
77 − 0 .0 .
85 − 2 .− 0 .1 .
95 − 1 .0 .0 .0 .0 .
29 − 1 .2 .0 .0 .0 .
52 − 0 .
71 − 0 .
59 − 1 .
57 − 1 .0 .
46 − 0 .0 .2 .0 .0 .
01 − 0 .
06 − 0 .1 .1 .
05 − 1 .
35 − 0 .
04 − 0 .
62 − 0 .− 2 .− 0 .1 .1 .
77 − 0 .− 0 .1 .
17 − 1 .
06 − 0 .0 .−
44 − 0 .1 .2 .0 .
81 − 0 .
36 − 0 .1 .
24 − 0 .

A 4 =    − 0 .
31 − 0 .
68 − 0 .
22 − 0 .1 .
64 − 0 .
41 − 0 .
66 − 0 .1 .0 .−
29 − 0 .1 .
89 − 1 .
85 − 0 .1 .0 .0 .
88 − 1 .0 .
69 − 1 .
75 − 0 .0 .0 .
08 − 0 .
11 − 0 .1 .0 .−
4 − 1 .0 .0 .
57 − 0 .1 .
64 − 1 .0 .
91 − 2 .1 .
1 − 1 .0 .
47 − 0 .0 .0 .0 .
03 − 0 .1 .0 .0 .
43 − 0 .
02 − 0 .1 .0 .
2 − 1 .1 .1 .
23 − 0 .2 .− 0 .0 .
48 − 1 .0 .
6 − 0 .
19 − 0 .
18 − 0 .
88 − 1 .
53 − 0 .
66 − 0 .1 .
32 − 1 .0 .1 .1 .
02 − 0 .1 .
51 − 0 .
51 − 0 .
73 − 0 .0 .
21 − 2 .0 .0 .1 .
41 − 1 .
93 − 2 .0 .
23 − 0 .0 .−
1 .0 .0 .1 .− 1 . 1 − 0 .0 .0 .
09 − 0 .2 .
 
A 5 =    0 .
36 − 0 .0 .0 .−
0 .− 0 .0 .
58 − 0 .0 .0 .0 .
11 − 0 .
02 − 0 .
66 − 0 .
98 − 0 .
28 − 1 . 61 − 0 .1 .1 .0 .−
.0 .−
1 . 5 − 0 .
33 − 0 .− 0 .
6 − 0 .
84 − 0 .
43 − 0 .1 .− 2 .
33 − 0 .0 .0 .
88 − 0 .
85 − 0 .0 .0 .0 .1 .0 .1 .
61 − 0 .1 .
15 − 1 .2 .
28 − 0 .
32 − 0 .−
2 .0 .−
68 − 0 .
54 − 0 .1 .0 .
7 − 1 .0 .
15 − 0 .
27 − 0 .1 .− 0 .
83 − 0 .
52 − 0 .
83 − 1 .1 .
79 − 0 .0 .1 .0 .1 .−
1 . 34 − 0 .0 .
12 − 0 .
95 − 0 .2 .
15 − 0 .
67 − 0 .1 .1 .0 .
54 − 0 .
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
  
A.3. Filter of Convolutional Layer
Here we present the 3 × 3 ﬁlter k which was used to create the 16 × 16 matrix W of our experiment in section 4.2.
The matrix W is the equivalent matrix of the convolution k (cid:63) x where x is an arbitrary 4 × 4 image.
[Table]