C ONVOLUTIONAL N ORMALIZING F LOWS
Anonymous authors
Paper under double-blind review
A BSTRACT
Bayesian posterior inference is prevalent in various machine learning problems.
Variational inference provides one way to approximate the posterior distribution, however its expressive power is limited and so is the accuracy of resulting approximation.
Recently, there has a trend of using neural networks to approximate the variational posterior distribution due to the ﬂexibility of neural network architecture.
One way to construct ﬂexible variational distribution is to warp a simple density into a complex by normalizing ﬂows, where the resulting density can be analytically evaluated.
However, there is a trade-off between the ﬂexibility of normalizing ﬂow and computation cost for efﬁcient transformation.
In this paper, we propose a simple yet effective architecture of normalizing ﬂows, ConvFlow , based on convolution over the dimensions of random input vector.
Experiments on synthetic and real world posterior inference problems demonstrate the effectiveness and efﬁciency of the proposed method.
I NTRODUCTION
Posterior inference is the key to Bayesian modeling, where we are interested to see how our belief over the variables of interest change after observing a set of data points.
Predictions can also beneﬁt from Bayesian modeling as every prediction will be equipped with conﬁdence intervals representing how sure the prediction is.
Compared to the maximum a posterior estimator of the model parameters, which is a point estimator, the posterior distribution provide richer information about the model parameter hence enabling more justiﬁed prediction.
Among the various inference algorithms for posterior estimation, variational inference (VI) and
Monte Carlo markov chain (MCMC) are the most two wisely used ones.
It is well known that MCMC suffers from slow mixing time though asymptotically the samples from the chain will be distributed from the true posterior.
VI, on the other hand, facilitates faster inference, since it is optimizing an explicit objective function and convergence can be measured and controlled, and it’s been widely used in many Bayesian models, such as Latent Dirichlet Allocation (Blei et al., 2003), etc.
However, one drawback of VI is that it makes strong assumption about the shape of the posterior such as the posterior can be decomposed into multiple independent factors.
Though faster convergence can be achieved by parameter learning, the approximating accuracy is largely limited.
The above drawbacks stimulates the interest for richer function families to approximate posteriors while maintaining acceptable learning speed.
Speciﬁcally, neural network is one among such models which has large modeling capacity and endows efﬁcient learning.
(Rezende & Mohamed, 2015) proposed normalization ﬂow, where the neural network is set up to learn an invertible transformation from one known distribution, which is easy to sample from, to the true posterior.
Model learning is achieved by minimizing the KL divergence between the empirical distribution of the generated samples and the true posterior.
After properly trained, the model will generate samples which are close to the true posterior, so that Bayesian predictions are made possible.
Other methods based on modeling random variable transformation, but based on different formulations are also explored, including NICE (Dinh et al., 2014), the Inverse Autoregressive Flow (Kingma et al., 2016), and Real
NVP (Dinh et al., 2016).
One key component for normalizing ﬂow to work is to compute the determinant of the Jacobian of the transformation, and in order to maintain fast Jacobian computation, either very simple function is used as the transformation, such as the planar ﬂow in (Rezende & Mohamed, 2015), or complex tweaking of the transformation layer is required.
Alternatively, in this paper we propose a simple and yet effective architecture of normalizing ﬂows, based on convolution on the random input vector.
Due to the nature of convolution, bi-jective mapping between the input and output vectors can be easily established; meanwhile, efﬁcient computation of the determinant of the convolution Jacobian is achieved linearly.
We further propose to incorporate dilated convolution (Yu & Koltun, 2015;
Oord et al., 2016a) to model long range interactions among the input dimensions.
The resulting convolutional normalizing ﬂow, which we term as Convolutional Flow (ConvFlow) , is simple and yet effective in warping simple densities to match complex ones.
The remainder of this paper is organized as follows: We brieﬂy review the principles for normalizing ﬂows in Section 2, and then present our proposed normalizing ﬂow architecture based on convolution in Section 3.
Empirical evaluations and analysis on both synthetic and real world data sets are carried out in Section 4, and we conclude this paper in Section 5.
P RELIMINARIES 2. T RANSFORMATION OF RANDOM VARIABLES
Given a random variable z ∈ R d with density p ( z ) , consider a smooth and invertible function f : R d → R d operated on z .
Let z (cid:48)
=
f ( z ) be the resulting random variable, the density of z (cid:48) can be evaluated as
[Equation] [Equation]
) thus
[Equation]
2. N ORMALIZING FLOWS
Normalizing ﬂows considers successively transforming z 0 with a series of transformations { f 1 , f 2 , ..., f K } to construct arbitrarily complex densities for z K =
f K ◦ f K − 1 ◦ ... ◦
f 1 ( z 0 ) as
[Equation]
Hence the complexity lies in computing the determinant of the Jacobian matrix.
Without further assumption about f , the general complexity for that is O ( d 3 ) where d is the dimension of z .
In order to accelerate this, (Rezende & Mohamed, 2015) proposed the following family of transformations that they termed as planar ﬂow :
[Equation]
where w ∈ R d , u ∈ R d , b ∈ R are parameters and h ( · ) is a univariate non-linear function with derivative h (cid:48) ( · ) .
For this family of transformations, the determinant of the Jacobian matrix can be computed as
[Equation]
where ψ ( z ) = h (cid:48) ( w (cid:62) z
+ b ) w .
The computation cost of the determinant is hence reduced from
O ( d 3 ) to O ( d ) .
Applying f to z can be viewed as feeding the input variable z to a neural network with only one single hidden unit followed by a linear output layer which has the same dimension with the input layer.
Obviously, because of the bottleneck caused by the single hidden unit, the capacity of the family of transformed density is hence limited.
A NEW TRANSFORMATION UNIT
In this section, we ﬁrst propose a general extension to the above mentioned planar normalizing ﬂow, and then propose a restricted version of that, which actually turns out to be convolution over the dimensions of the input random vector.
3. N ORMALIZING FLOW WITH d HIDDEN UNITS
Instead of having a single hidden unit as suggested in planar ﬂow, consider d hidden units in the process.
We denote the weights associated with the edges from the input layer to the output layer as
W ∈ R d × d and the vector to adjust the magnitude of each dimension of the hidden layer activation as u , and the transformation is deﬁned as
[Equation]
where (cid:12) denotes the point-wise multiplication.
The Jacobian matrix of this transformation is
[Equation]
As det( diag ( u (cid:12) h (cid:48) ( W z + b ))) is linear, the complexity of computing the above transformation lies in computing det( W ) .
Essentially the planar ﬂow is restricting W to be a vector of length d instead of matrices, however we can relax that assumption while still maintaining linear complexity of the determinant computation based on a very simple fact that the determinant of a triangle matrix is also just the product of the elements on the diagonal.
3. C ONVOLUTIONAL F LOW
Since normalizing ﬂow with a fully connected layer may not be bijective and generally requires
O ( d 3 ) computations for the determinant of the Jacobian even it is, we propose to use 1-d convolution to transform random vectors.
3 5 6 71 24 5 6 7 8 w1 w2 w w1 w2 w w1 w2 w w1 w2 w w1 w2 w w1 w2 w w1 w2 w w1 w2 w (a) 3                3        3     0 dilation= dilation= dilation=
[Figure]
Figure 1(a) illustrates how 1-d convolution is performed over an input vector and outputs another vector.
We propose to perform a 1-d convolution on an input random vector z , followed by a non-linearity and necessary post operation after activation to generate an output vector.
Speciﬁcally,
[Equation]
where w ∈ R k is the parameter of the 1-d convolution ﬁlter ( k is the convolution kernel width), conv ( z , w ) is the 1d convolution operation as shown in Figure 1(a), h ( · ) is a monotonic nonlinear activation function 1 , (cid:12) denotes point-wise multiplication, and u ∈ R d is a vector adjusting the magnitude of each dimension of the activation from h ( · ) .
We term this normalizing ﬂow as
Convolutional Flow (ConvFlow) .
ConvFlow enjoys the following properties
• Bi-jectivity can be easily achieved if proper padding and a monotonic activation function are adopted; • Due to local connectivity, the Jacobian determinant of ConvFlow only takes O ( d ) computation independent from convolution kernel width k since
[Equation]
where w 1 denotes the ﬁrst element of w .
For example for the illustration in Figure 1(a), the Jacobian matrix of the 1d convolution conv ( z , w ) is
[Equation]
which is a triangular matrix whose determinant can be easily computed; • ConvFlow is much simpler than previously proposed variants of normalizing ﬂows.
The total number of parameters of one ConvFlow layer is only d + k where generally k < d , particularly efﬁcient for high dimensional cases.
Notice that the number of parameters in the planar ﬂow in (Rezende & Mohamed, 2015) is 2 d and one layer of Inverse Autoregressive Flow (IAF) (Kingma et al., 2016) and Real NVP (Dinh et al., 2016) require even more parameters. In Section
, we discuss the key differences of ConvFlow from IAF in detail.
A series of K ConvFlows can be stacked to generate complex output densities.
Further, since convolutions are only visible to inputs from adjacent dimensions, we propose to incorporate dilated convolution to the ﬂow to accommodate interactions among dimensions with long distance apart.
Figure 1(b) presents a block of 3 ConvFlows stacked, with different dilations for each layer.
Larger receptive ﬁeld is achieved without increasing the number of parameters.
We term this as a ConvBlock.
From the block of ConvFlow layers presented in Figure 1(b), it is easy to verify that dimension i (1 ≤ i ≤ d ) of the output vector only depends on succeeding dimensions, but not preceding ones.
In other words, dimensions with larger indices tend to end up getting little warping compared to the ones with smaller indices.
Fortunately, this can be easily resolved by a Revert Layer , which simply outputs a reversed version of its input vector.
Speciﬁcally, a Revert Layer g operates as
[Equation]
It’s easy to verify a Revert Layer is bijective and that the Jacobian of g is a d × d matrix with 1s on its anti-diagonal and 0 otherwise, thus log (cid:12)(cid:12)(cid:12) det ∂g∂ z (cid:12)(cid:12)(cid:12) is 0.
Therefore, we can append a Revert Layer after each ConvBlock to accommodate warping for dimensions with larger indices without additional computation cost for the Jacobian as follows
[Equation]
[Table]
+
[Table]
[Equation]
Inspired by the idea of constructing complex tractable densities from simpler ones with bijective transformations, different variants of the original normalizing ﬂow (IAF) (Rezende & Mohamed, 2015) have been proposed. Perhaps the one most related to ConvFlow is Inverse Autogressive Flow (Kingma et al., 2016), which employs autoregressive transformations over the input dimensions to construct output densities.
Speciﬁcally, one layer of IAF works as follows
[Equation]
where [
[Equation]
[Equation]
←
[Equation]
) are outputs from an autoregressive neural network over the dimensions of z .
There are two drawbacks of IAF compared to the proposed ConvFlow: • The autoregressive neueral network over input dimensions in IAF is represented by a Masked Autoencoder (Germain et al., 2015), which generally requires O ( d 2 ) parameters per layer, where d is the input dimension, while each layer of ConvFlow is much more parameter efﬁcient, only needing k + d parameters ( k is the kernel size of 1d convolution and k < d ). • More importantly, due to the coupling of σ ( z ) and z in the IAF transformation, in order to make the computation of the overall Jacobian determinant det ∂f∂ z linear in d , the Jacobian of the autoregressive NN transformation is assumed to be strictly triangular (Equivalently, the Jacobian determinants of µ and σ w.r.t z are both always 0. This is achieved by letting the i th dimension of µ and σ depend only on dimensions 1 , 2 , ..., i − 1 of z ).
In other words, the mappings from z onto µ ( z ) and σ ( z ) via the autogressive NN are always singular, no matter how their parameters are updated, and because of this, µ and σ will only be able to cover a subspace of the input space z belongs to , which is obviously less desirable for a normalizing ﬂow.
2 Though these sigularity transforms in the autoregressive NN are somewhat mitigated by their ﬁnal coupling with the input z , IAF still performs slightly worse in empirical evaluations than ConvFlow as no singular transform is involved in ConvFlow.
E XPERIMENTS
We test performance the proposed ConvFlow on two settings, one on synthetic data to infer unnormalized target density and the other on density estimation for hand written digits and characters.
4. S YNTHETIC DATA
We conduct experiments on using the proposed ConvFlow to approximate an unnormalized target density of z with dimension 2 such that p ( z ) ∝ exp( − U ( z ))
where U ( z ) =
12 (cid:104) z
2 − w 1 ( z ) 0 .(cid:105) 2 and w 1 ( z ) = sin (cid:0) π
z 1 2 (cid:1) .
The target density of z are plotted as the left most column in Figure 2, and we test to see if the proposed ConvFlow can transform a two dimensional standard Gaussian to the target density by minimizing the KL divergence
[Equation]
where all expectations are evaluated with samples taken from q 0 ( z 0 ) .
We use a 2-d standard Gaussian as q 0 ( z 0 ) and we test different number of ConvBlocks stacked together in this task.
Each ConvBlock in this case consists a ConvFlow layer with kernel size 2, dilation 1 and followed by another ConvFlow layer with kernel size 2, dilation 2.
Revert Layer is appended after each ConvBlock, and leaky ReLU with a negative slope of 0 .
01 is adopted in ConvFlow.
Experimental results are shown in Figure 2 for different layers of ConvBlock to be stacked to compose f .
It can be seen that even with 4 layers of ConvBlocks, it’s already approximating the target density despite the underestimate about the density around the boundaries.
With 8 layers of ConvFlow, the transformation from a standard Gaussian noise vector to the desired target unnormalized density can be accurately learned.
Notice that with 8 layers, we are only using 40 parameters ( (4 + 1) ∗ 8 with bias terms of convolution counted).
[Figure] [Figure] 4.2. S ETUPS
To test the proposed ConvFlow for variational inference we use standard benchmark datasets MNISTand OMNIGLOT 4 (Lake et al., 2013).
Our method is general and can be applied to any formulation of the generative model p θ ( x, z ) ; For simplicity and fair comparison, in this paper, we focus on densities deﬁned by stochastic neural networks, i.e., a broad family of ﬂexible probabilistic generative models with its parameters deﬁned by neural networks.
Speciﬁcally, we consider the following two family of generative models
[Equation]
where p ( z ) and p ( z 1 ) are the priors deﬁned over z and z 1 for G 1 and G 2 , respectively.
All other conditional densities are speciﬁed with their parameters θ deﬁned by neural networks, therefore ending up with two stochastic neural networks.
This network could have any number of layers, however in this paper, we focus on the ones which only have one and two stochastic layers, i.e., Gand G 2 , to conduct a fair comparison with previous methods on similar network architectures, such as VAE, IWAE and Normalizing Flows.
We use the same network architectures for both G 1 and G 2 as in (Burda et al., 2015), speciﬁcally shown as follows
G 1 : A single Gaussian stochastic layer z with 50 units.
In between the latent variable z and observation x there are two deterministic layers, each with 200 units;
where a Gaussian stochastic layer consists of two fully connected linear layers, with one outputting the mean and the other outputting the logarithm of diagonal covariance.
All other deterministic layers are fully connected with tanh nonlinearity.
Bernoulli observation models are assumed for both
MNIST and OMNIGLOT.
For MNIST, we employ the static binarization strategy as in (Larochelle &
Murray, 2011) while dynamic binarization is employed for OMNIGLOT.
The inference networks q ( z | x ) for G 1 and G 2 have similar architectures to the generative models, with details in (Burda et al., 2015).
ConvFlow is hence used to warp the output of the inference network q ( z | x ) , assumed be to Gaussian conditioned on the input x , to match complex true posteriors.
Our baseline models include VAE (Kingma & Welling, 2013), IWAE (Burda et al., 2015) and Normalizing
Flows (Rezende & Mohamed, 2015).
Since our propose method involves adding more layers to the inference network, we also include another enhanced version of VAE with more deterministic layers added to its inference network, which we term as VAE+.
With the same VAE architectures, we also test the abilities of constructing complex variational posteriors with IAF and ConvFlow, respectively.
All models are implemented in PyTorch.
Parameters of both the variational distribution and the generative distribution of all models are optimized with Adam (Kingma & Ba, 2014) for 2000 epochs, with a ﬁxed learning rate of 0.0005, exponential decay rates for the 1st and 2nd moments at 0.9 and 0.999, respectively.
Batch normalization (Ioffe & Szegedy, 2015) is also used, as it has been shown to improve learning for neural stochastic models (Sønderby et al., 2016).
For inference models with latent variable z of 50 dimensions, a ConvBlock consists of following
ConvFlow layers
[Equation]
and for inference models with latent variable z of 100 dimensions, a ConvBlock consists of following
ConvFlow layers
[Equation]
A Revert layer is appended after each ConvBlock and leaky ReLU with a negative slope of 0 .
01 is used as the activation function in ConvFlow.
For IAF, the autoregressive neural network is implemented as a two layer masked fully connected neural network.
4.2. G ENERATIVE D ENSITY E STIMATION
For MNIST, models are trained and tuned on the 60,000 training and validation images, and estimated log-likelihood on the test set with 5000 importance weighted samples are reported.
Table 1 presents the performance of all models, when the generative model is assumed to be from both G 1 and G 2 .
Firstly, VAE+ achieves higher log-likelihood estimates than vanilla VAE due to the added more layers in the inference network, implying that a better posterior approximation is learned (which is still assumed to be a Gaussian).
Second, we observe that VAE with ConvFlow achieves much better density estimates than VAE+, which conﬁrms our expectation that warping the variational distribution with convolutional ﬂows enforces the resulting variational posterior to match the true complex posterior.
Also, adding more blocks of convolutional ﬂows to the network makes the variational posterior further close to the true posterior.
We also observe that VAE with Inver Autoregressive Flows (VAE+IAF) does not always improve likelihood estimates, and even if they do, the improvements are not as signiﬁcant as ConvFlow.
This also conﬁrms our analysis on the singular transformation and
[Table] subspace issue in IAF. Lastly, combining convolutional normalizing ﬂows with multiple importance weighted samples, as shown in last row of Table 1, further improvement on the test set log-likelihood is achieved.
Overall, the method combining ConvFlow and importance weighted samples achieves best NLL on both settings, outperforming IWAE signiﬁcantly by about 6.3 nats on G 1 and 5.5 nats on G 2 .
Notice that, ConvFlow combined with IWAE achieves an NLL of 78.51, slightly better than the best published result of 79.10, achieved by PixelRNN (Oord et al., 2016b). Also it’s 1 nat better than the best IAF result of 79.88 reported in (Kingma et al., 2016), which demonstrates the representative power of ConvFlow compared to IAF 6 .
Results on OMNIGLOT are presented in Table 2 where similar trends can be observed as on MNIST.
One observation different from MNIST is that, the gain from IWAE+ConvFlow over IWAE is not as large as it is on MNIST, which could be explained by the fact that OMNIGLOT is a more difﬁcult set compared to MNIST, as there are 1600 different types of symbols in the dataset. Again on OMNIGLOT we observe IAF with VAE doesn’t perform as well as ConvFlow.
4.2. G ENERATED S AMPLES
After the models are trained, generative samples can be obtained by feeding z ∼ N (0 , I ) to the learned generative model G 1 (or z 2 ∼ N (0 , I ) to G 2 ).
Since higher log-likelihood estimates are obtained on G 2 , Figure 3 shows the random generative samples from our proposed method trained with G 2 on both MNIST and Ominiglot, compared to real samples from the training sets.
We observe the generated samples are visually consistent with the training data.
C ONCLUSIONS
This paper presents a simple and yet effective architecture to compose normalizing ﬂows based on convolution on the input vectors.
ConvFlow takes advantage of the effective computation of convolution, as well as maintaining as few parameters as possible.
To further accommodate long range interactions among the dimensions, dilated convolution is incorporated to the framework without OMNIGLOT [Table] [Table]
= 50)
[Table] VAE+ ( K = 1) [Table] VAE+ ( K = 4)
[Table]
VAE+ ( K = 8)
[Table]
VAE+IAF ( K = 1)
[Table]
VAE+IAF ( K = 2)
[Table]
VAE+IAF ( K = 4)
[Table]
VAE+IAF ( K = 8)
[Table]
VAE+ConvFlow ( K = 1)
[Table]
VAE+ConvFlow ( K = 2)
[Table]
VAE+ConvFlow ( K = 4)
[Table]
VAE+ConvFlow ( K = 8)
[Table]
IWAE+ConvFlow ( K = 8 , IW =
[Table]
[Table]
[Figure]
increasing model parameters.
A Revert Layer is used to maximize the opportunity that all dimensions get as much warping as possible.
Experimental results on inferring target complex density and density estimation on generative modeling on real world handwritten digits data demonstrates the strong performance of ConvFlow.
Particularly, density estimates on MNIST show signiﬁcant improvements over state-of-the-art methods, validating the power of ConvFlow in warping multivariate densities.
It remains an interesting question as to how many layers of ConvFlows are best to exploit its full performance.
We hope to address the theoretical properties of ConvFlow in future work.