CInC Flow: Characterizable Invertible 3 × 3 Convolution
Abstract
Normalizing ﬂows are an essential alternative to
GANs for generative modelling, which can be optimized directly on the maximum likelihood of the dataset.
They also allow computation of the exact latent vector corresponding to an image since they are composed of invertible transformations.
However, the requirement of invertibility of the transformation prevents standard and expressive neural network models such as CNNs from being directly used.
Emergent convolutions were proposed to construct an invertible 3 × 3 CNN layer using a pair of masked CNN layers, making them inefﬁcient.
We study conditions such that 3 × 3 CNNs are invertible, allowing them to construct expressive normalizing ﬂows.
We derive necessary and sufﬁcient conditions on a padded CNN for it to be invertible.
Our conditions for invertibility are simple, can easily be maintained during the training process.
Since we require only a single CNN layer for every effective invertible CNN layer, our approach is more efﬁcient than emerging convolutions.
We also proposed a new coupling layer for more ﬂexibility and expressiveness, Quad-coupling.
We benchmark our approach and show similar performance results to emergent convolutions while improving the model’s efﬁciency.
Code available on GitHub a .
a https://github.com/Naagar/
INTRODUCTION
The availability of large datasets has resulted in improved machine learning solutions for more complex problems.
However, supervised datasets are expensive to create.
Hence unsupervised methods like generative models are increasingly worked on.
Generative models that have been proposed can be broadly categorized under Likelihood-based methods and Generative Adversarial Methods.
For example, an optimization algorithm could directly minimize the former’s negative log-likelihood of the unsupervised examples.
At the same time, in the latter, the loss function itself is modelled as a discriminator network that is trained alternatively.
Hence likelihood-based methods directly optimize the probability of examples.
In contrast, in GANs, the function being optimized is implicit and hard to reason about.
An essential type of Likelihood-based Generative models is normalizing ﬂow-based models.
Normalizing ﬂow-based models transform a latent vector usually sampled from a continuous distribution like the Gaussian by a sequence of invertible functions to produce the sample.
Hence even though the latent vector distribution is simple, the sample distribution could be highly complex, provided we are using an expressive set of invertible transformations.
Also, the invertibility of the model implies that one can ﬁnd the exact latent vector corresponding to an example from a dataset.
All other approaches to generative modelling can compute the latent vector, for example, only approximately.
The ability of a normalizing ﬂow based model to express complex real-world data distributions depends on the expressive power of the invertible transformations used.
In supervised models in vision tasks, complex, multilayered
CNNs with different window sizes are used.
CNN’s with larger window size helps in spatial mixing of information about the images, resulting in expressive features.
Glow used invertible 1 × 1 convolutions to build normalizing ﬂow models Kingma and Dhariwal [2018].
For a 1 × 1 convolution (if it is invertible), the inverse is also a 1 × 1 convolution.
We show that this approach does not generalize to larger window sizes.
In particular, the inverse of an invertible 3 ×convolution necessarily depends on all the feature vector dimensions, unlike CNNs, which only require local features.
Emerging convolutions proposed a way of inverting convolutions with large window sizes Hoogeboom et al.
The inverse is not a convolution and is computed by a linear equa- tion system that can efﬁciently be solved using back substitution.
However, for obtaining an invertible convolution, they required 2 CNN ﬁlters to be applied.
Hence for every effective invertible convolution, they are required to do two convolutions back to back.
We propose a simple approach using padding of obtaining invertible convolutions, which only uses a single convolutional ﬁlter.
Furthermore, we are able to give a characterization (necessary and sufﬁcient conditions) for the convolutions to be invertible.
This allows us to optimize over the space of invertible convolutions during training directly.
We have compared our method with Emerging convolution (Hoogeboom et al.
[2019]) and Autoregressive convolution (Germain et al.
[2015]) in Appendix-C
Main Contributions.
• We give necessary and sufﬁcient conditions for a 3 ×convolution to be invertible by making some modiﬁcations to the padding (see Section 3.1).
• We also propose a more expressive coupling mechanism called Quad-coupling (see Section 3.2).
• We use our characterization and Quad-coupling to train ﬂow-based models that give samples of similar quality as previous works while improving upon the run-time compared to the other invertible 3 × 3 convolutions proposed (see Section 4).
RELATED WORKS
Normalizing ﬂows.
A normalizing ﬂow aims to model an unknown data distribution (Kingma and Dhariwal [2018],
Durkan et al.
[2019a,b]), that is, to be able to sample from this distribution and estimate the likelihood of an element for this distribution.
To model the probability density of a random variable x , a normalizing ﬂow apply an invertible change of variable x
= g θ
( z ) where z is a random variable following a known distribution for instance z ∼ N (0 , I d ) .
Then we can get the probability of x by applying the change of variable formula
[Equation]
(cid:18)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:19) where f θ denotes the inverse of g θ and (cid:12)(cid:12)(cid:12)
∂f θ ( x ) ∂x T (cid:12)(cid:12)(cid:12) its Jacobian.
The parameters θ are learned by maximizing the actual likelihood of the dataset.
At the same time, the model is designed so that the function g θ can be inverted and have its
Jacobian computed in a reasonable amount of time.
Glow.
RealNVP deﬁnes a normalizing ﬂow composed of a succession of invertible steps (Dinh et al.
Each of these steps can be decomposed into layers steps.
Improvements for some of these layers where proposed in later articles (Kingma and Dhariwal [2018], Hoogeboom et al.
Actnorm : The actnorm layer performs an afﬁne transformation similar to batch normalization.
First, its weights are initialized so that the output of the actnorm layer has zero mean and unit variance.
Then its parameters are learned without any constraint.
Permutation : RealNVP proposed to use a ﬁxed permutation to shufﬂe the channels as the coupling layer only acts on half of the channels.
Later, Kingma and Dhariwal [2018] replaced this permutation with a 1 × 1 convolution in Glow.
These can easily be inverted by inverting the kernel.
Finally,
Hoogeboom et al.
[2019] replaced this 1x1 convolution with the so-called emerging convolution.
These have the same receptive convolution with a kernel of arbitrary size.
However, they are computed by convolving with two successive convolutions whose kernel is masked to help the inversion operation.
Coupling layer : The coupling layer is used to provide ﬂexibility to the architecture.
The Feistel scheme (Hoang and
Rogaway [2010]) inspires its design.
They are used to build an invertible layer out of any given function f .
Here f is learn as a convolutional neural network.
[Equation]
Where we get x 1 and x 2 by splitting the input x along the channel axis.
Invertible Convolutional Networks.
Complementary to normalizing ﬂows, there has been some work done designing more ﬂexible invertible networks.
For example, Gomez et al.
[2017] proposed reversible residual networks (RevNet) to limit the memory overhead of backpropagation, while (Jacobsen et al.
[2019]) built modiﬁcations to allow an explicit form of the inverse, also studying the ill-conditioning of the local inverse.
Ho et al.
[2019] proposed a ﬂow-based model that is the non-autoregressive model for unconditional density estimation on standard image benchmarks
Invertible 1 × 1 Convolution: Kingma and Dhariwal [2018] proposed the invertible 1 × 1 convolution replacing ﬁxed permutation (Dinh et al.
[2017]) that reverses the ordering of the channels.
Hoogeboom et al.
[2019] proposed normalizing ﬂow method to do the inversion of 1 × 1 convolution with doing some padding on the kernel and two distinct autoregressive convolutions, which also provide a stable and ﬂexible parameterization for invertible 1 × 1 convolutions.
Invertible n × n Convolution: Reformulating n × n convolution using invertible shift function proposed by Truong et al.
[2019] to decrease the number of parameters and remove the additional computational cost while keeping the range of the receptive ﬁelds.
In our proposed method, there is no need for the reformulation of standard convolutions.
Hoogeboom et al.
[2019] proposed two different methods to produce the invertible convolutions : (1) Emerging Convolution and (2)
Invertible Periodic Convolutions.
Emerging requires two autoregressive convolutions to do a standard convolution, but our method requires only one convolution as compare to the method proposed by Hoogeboom et al.
[2019] and increase the ﬂexibility of the invertible n × n convolution.
OUR APPROACH
We propose a novel approach for constructing invertible 3 × 3 convolutions and coupling layers for normalizing ﬂows.
We propose two modiﬁcations to the existing layers used in previous normalizing ﬂow models:
• convolution layer: instead of using 1 × 1 convolutions or emerging convolutions, we propose to use standard convolutions with a kernel of any size with a speciﬁc padding.
• coupling layer: we propose to use a modiﬁed version of the coupling layer designed to have a bigger receptive ﬁeld.
We also show how invertibility can be used to manipulate images semantically.
3.1 INVERTIBLE 3 × 3 CONVOLUTION
We give necessary and sufﬁcient conditions for an arbitrary convolution with some simple modiﬁcations on the padding to be invertible.
Moreover, the inverse can also be computed by an efﬁcient back substitution algorithm.
Deﬁnition 1 (Convolution) .
The convolution of an input
X with shape H × W × C with a kernel K with shape k × k × C × C is Y =
X ∗ K of shape ( H − k + 1) ×
( W − k + 1) × C which is equal to Y i,
[Equation]
(cid:88) [Equation] [Equation] In this setting, the output Y has a smaller size than the input to prevent this input is padded before applying the convolution.
Deﬁnition 2 (Padding) .
Given an image I with shape H ×
W × C , the ( t, b, l, r ) padding of I is the image
ˆ I of shape ( H + t + b ) ×
( W + l + r ) × C deﬁned as ˆ I i,j,
c = (cid:40)
I i − t,j − l,c if i − t <
H and j − l < W otherwise (2)
As zero padding does not add any bias to the input, the convolution between a padded input ˆ
I and a kernel K is still a linear map between the input and the output.
As such, it can be described as matrix multiplication.
An image I of shape ( H, W, C ) can be seen as an vector (cid:126)I of R H × W × C .
In the rest of this paper we will always use the following basis I i,j,c
= (cid:126)I c + Cj + CHi .
For any index i ≤ HWC , let ( i y , i x , i c ) denote the indexes that satisfy (cid:126)I i
= I
i y ,i x ,i c .
Note that i < j iff ( i y , i x , i c ) ≺ ( j y , j x , j c ) where ≺ denotes the lexicographical order over R 3 .
If C =this means that the pixel ( j y , j x ) is on the right or below the pixel ( i y , i x ) .
Deﬁnition 3 (Matrix of a convolution.) .
Let K be a kernel of shape k × k × C × C .
The matrix of a convolution of kernel K with input X of size H × W × C with padding ( t, b, l, r ) is a matrix describing the linear map
X
(cid:55)→ ˆ X ∗ K .
Characterization of invertible convolutions: We consider convolution with top and left padding only.
For such convolutions, we give necessary and sufﬁcient conditions for it to be invertible.
Let K be the kernel of the convolution with shape 3 × 3 × N × N where 3 × 3 is the window size, and N is the number of channels.
Note that number of input channels should be equal to the number of output channels for it to be invertible.
Lemma 1.
Let y =
M ˆ x ,
M is a lower triangular matrix with all diagonal entries
[Equation]
Where the matrix M is which produces the equivalent result when multiplied with a vectorized input ( ˆ x ).
Proof.
Consider any entry in the upper right half of M .
That is ( i, j ) such that i < j according to the ordering given in the deﬁnition of M .
M i,j is nothing but the scalar weight that needs to be multiplied to the j th pixel of input when computing i th pixel of the output.
The linear equation relating these two variable is as follows:
[Equation]
From this equations follows that if j x > i x or j y > i y then the i th pixel of the output does not depend on the j th pixel of the input and thus M i,j = 0 .
This also justiﬁes that all diagonal coefﬁcients of M are equal to K 3 ,
We ﬁrst describe our conditions for the case when N = 1 .
We prove the following theorem.
Theorem 1 (Characterization for N = 1 ) .
[Equation]
Note that the equivalent matrix M is triangular .
Proof.
The proof of the theorem uses Lemma 1.
Since M is lower triangular, the determinant is nothing but the product of diagonal entries, which is =
K h ∗ w 3 , 3 where h, w is the dimensions of the input/output image.
At its core, the convolution layer is a linear operation.
However, we have no guarantees regarding it as invertibility.
The result z of the convolution of input x with kernel k can be expressed as the product as x with a matrix M .
When zeropadding is used around the input so that x and z have the same shape, the matrix M is not easily invertible because the determinant of M can be zero (see matrix M in Figure 1(a)).
However, when padding only on two sides (left and top), the corresponding M is blocked triangular (see Figure 1(b)).
To further ensure invertibility and speed up the inversion process, we also mask part of the kernel so that the matrix corresponding to the convolution is triangular, see in Figure 1(c) and for more details which K ( n, n ) we need to mask see Appendix-B.
In this conﬁguration, the Jacobian of the convolution can also be easily computed.
For further details of the padding the input, see Appendix-A.
3.2 QUAD-COUPLING
The coupling layer is used to have some ﬂexibility as its functions can be of any form.
However, it only combines the effects of half channels.
To overcome this issue we designed a new coupling layer inspired from generalized
Feistel (Hoang and Rogaway [2010]) schemes.
Instead of dividing the input x into two blocks we divide it into four x = [ x 1 , x 2 , x 3 , x 4 ] along the feature axis.
Then we keep x 1 unchanged and use it to modify the other blocks in an autoregressive manner (see Figure 2):
[Equation]
where ( f i ) i ≤ 3 and ( g i ) i ≤ 3 are learned.
The output of the layer is obtained by concatenating the ( y i )
i ≤ 4 .
EXPERIMENTAL RESULTS
The architecture is based on Hoogeboom et al.
We modiﬁed the emerging convolution layer to use our standard convolution.
We also introduced the Quad-coupling layer in place of the afﬁne coupling layer.
Finally, we evaluate the model on a variety of models and provide images sampled from the model.
For detailed overview of the architecture see Figure 3.
Training setting: To train the model on Cifar10, we used the 3 level (L) and depth (D) of 32 and lr 0 .
001 for theepochs.
To train on ImageNet32, L = 3 , D = 48 , lr 0 .for
the 600 epochs and for ImageNet64, L = 4 , D = 48 , lr 0 .
001 for the 1000 epochs.
See Figure 3 for the model architecture.
x and denote input and output.
The model has levels, and modules per
Quantitative results: The Comparison of the performance of 3 × 3 invertible convolution with the emerging convolution (Hoogeboom et al.
[2019]) for the cifar10 dataset in Table 1.
The performance of our layers was tested on CIFAR10 (Krizhevsky et al.
[2009]), ImageNet (Russakovsky et al.
[2015]) as well as on the galaxy dataset (Ackermann et al.
[2018]) see Table 2.
We also tested our architecture on networks with a smaller depth ( D = 4 or D = 8 ) see
Table 3 which could be used when computational resources are limited as their sampling time is much lower.
In this case, using standard convolution and Quad-coupling offers a more considerable performance improvement than with bigger models (see Table 3). [Table]
[Table]
Galaxy —
[Table]
Emerging results were obtained by using the code provided in Hoogeboom et al.
[2019], 3 × 3 is replacing the emerging convolutions by our 3 × 3 invertible convolutions and Quad uses Quad-coupling on top of this.
Sampling Times: We compared our method’s sampling time (Table 4) against Glow (Kingma and Dhariwal [2018]) and Emerging Convolutions (Hoogeboom et al.
Our convolution still requires solving a sequential problem to be inverted and, as such, need to be inverted on CPU, unlike Glow that can be inverted while performing all the computation on GPU.
This explains the gap between the sampling time of our model compared to Glow.
However, it is still roughly two times faster than emerging convolutions; this comes from the need to solve two inversion problems to invert one emerging convolution layer.
The Quad-coupling layer does not affect sampling time too much.
[Table] [Table]
Interpretability results: To show the interpretability of our invertible network, we used the Celeba dataset (Liu et al.
[2015]) which provides images of faces and attributes corresponding to these faces.
In Figure 4 are the randomly generated fake sample images for the cifar10 dataset.
The covariance matrix between the attributes of images in the dataset and their latent representation indicates how to modify the latent representation of an image to add or remove features.
Examples of such modiﬁcations can be seen in
Figures 5(a, b, c) and 5(d).
CONCLUSION
In this paper
[Table]
we propose a new method for Invertible n × n
Convolution.
Coupling layers solve two problems for normalizing ﬂows: they have a tractable Jacobian determinant and can be inverted in a single pass.
We propose a new type of coupling method, Quad-coupling.
Our method shows consistent improvement over the Emerging convolutions method, and we only need a single CNN layer for every effective invertible convolution.
This paper shows that we can invert a convolution with only one effective convolution, and additionally, the inference time and sampling time improved notably.
We show the inversion of 3 × 3 convolution and the generalization of the inversion for the n × n kernel.
Furthermore, we demonstrate improved quantitative performance in terms of log-likelihood on standard image modelling benchmarks.
A INPUT PADDING
To make sure the matrix ( M ) block triangular, padding of ( k + 1) / 2 on top and left of input ( x ) is applied where k is the size of kernels.
B MASKING OF KERNELS
Considering the assumption that the numbers of input channels is equals to the output channels ( N ) then masking of the kernels depends only on the numbers of channels ( N ), let the kernel size is k × k and the K a,bk,k are the entries corresponding to block ( N × N ) diagonals entire in the convolution matrix ( M ), where a and b are a, b ∈ 1 , 2 , ..., N .
K a,bk,k is a square matrix ( C ) of size N × N ) and the diagonal entries of the block of matrix M are the diagonal of C and to ensure the invariability of the M , we have to set all the entries C a,b to zero when a > b and one when a = b .
C
COMPARISON OF OUR METHOD WITH THE EXISTING INVERTIBLE NORMALIZING FLOW METHODS
See ﬁgure 6.