Invertible Monotone Operators for Normalizing Flows
Abstract
Normalizing flows model probability distributions by learning invertible transformations that transfer a simple distribution into complex distributions.
Since the architecture of ResNet-based normalizing flows is more flexible than that of coupling-based models, ResNet-based normalizing flows have been widely studied in recent years.
Despite their architectural flexibility, it is wellknown that the current ResNet-based models suffer from constrained Lipschitz constants.
In this paper, we propose the monotone formulation to overcome the issue of the Lipschitz constants using monotone operators and provide an in-depth theoretical analysis.
Furthermore, we construct an activation function called Concatenated Pila (CPila) to improve gradient flow.
The resulting model, Monotone Flows , exhibits an excellent performance on multiple density estimation benchmarks (MNIST, CIFAR-10, ImageNet32, ImageNet64). Code is available at https://github.com/mlvlab/MonotoneFlows .
Introduction
Normalizing flows [1, 2] are a method for constructing complex distributions by transforming a probability density through a series of invertible transformations.
Normalizing flows are trained using a plain log-likelihood function and they are capable of exact density evaluation and efficient sampling.
Applications include image generation [2, 3, 4, 5, 6, 7, 8], image super-resolution [9, 10, 11], image noise modelling [12, 13], audio synthesis [14, 15], anomaly detection [16, 17] and computational physics [18, 19, 20, 21], highlighting the importance of developing expressive normalizing flows.
However, a major hurdle to designing normalizing flow architectures is that not only should each transformation be invertible but also its Jacobian determinant needs to be tractable [22, 23].
Despite multiple efforts to satisfy the aforementioned constraints [2, 3, 4, 24, 25], the expressive power of the normalizing flow transformations stays limited due to the adoption of specialized architectures [5].
In this regard, ResNet-based normalizing flows [5, 6, 7] are a compelling alternative since they do not impose any structural restrictions.
However, they keep the Lipschitz constant of the residual branch less than 1 in order to ensure invertibility.
As a result, the Lipschitz constant of each residual block is less than 2 .
This causes a serious issue on the expressive power since a large Lipschitz constant is often required in converting between distributions.
A recent work [26] tackled this problem, but as we show in this paper, their method is equivalent to a simple rescaling of ResNet-based normalizing flows, which has a similar expressive power.
In this work, we propose the approach of monotone operators to alleviate the Lipschitz constraint.
More precisely, in order to parameterize monotone operators, we use the well-known fact from monotone operator theory: the Cayley operator of maximally monotone operators (MMOs) is a 1Lipschitz function .
The residual branch of ResNet-based normalizing flows can be used to parameterize the Cayley operators. [Figure]
Rotate +45°
Rotate x y
In summary, our contributions are as follows:
• We propose Monotone Flows , which greatly loosens the Lipschitz constraint while retaining invertibility and architectural flexibility. • We derive the monotone formulation , a monotone operator-based normalizing flow by parametrizing the Cayley operator, and provide efficient training and inference schemes. • We propose a new activation function called Pila and its variant Concatenated Pila (CPila) to alleviate the saturated gradient problem.
• We theoretically analyze the expressive power of the monotone formulation and related models [6, 26], and prove our formulation outperforms the other models in practice.
• Monotone Flows consistently outperform comparable baseline normalizing flows on multiple image density estimation benchmarks as well as on 2D toy datasets.
In addition, ablation studies demonstrate the effectiveness of the proposed methods.
Preliminaries
The monotone formulation is a normalizing flow that parametrizes monotone operators in terms of Cayley operators.
The parametrization, which must be 1-Lipschitz, and the methods for log determinant computation, are borrowed from ResNet-based normalizing flows.
Also, the resulting model is an implicit neural network involving inverse functions.
Here, we briefly review these topics.
Normalizing flows [1] explicitly model probability distributions by transforming a tractable source distribution, e.g ., Gaussian, into a target distribution with a differentiable invertible transform F : R n → R n .
Suppose F maps a sample space X to a latent space Z .
Given a source distribution p Z ( z ) , the change-of-variables theorem yields the log-likelihood of any x ∈ X :
[Equation]
where z = F ( x ) and J F = ∂z/∂x is the Jacobian.
Normalizing flows are beneficial since the evaluation of (1) yields the exact density p X ( x ) .
Furthermore, target data x ∼ p X ( x ) can be efficiently sampled by first sampling z ∼ p
Z ( z ) and then mapping z to X by x = F − 1 ( z ) .
When F consists of multiple transformations applied in succession, each transformation is called a flow .
ResNet-based normalizing flows have been initially proposed in i-ResNets [5], which are built with a variant of the residual blocks R ( x ) = x + G ( x ) .
By requiring G ( x ) to be a contraction map, they ensured the invertibility of R ( x ) with the Banach fixed point theorem.
The contractiveness of G ( x ) is implemented by spectral normalization [27] and 1-Lipschitz activation functions.
However, they use a biased truncation of the log Taylor series for computing the log determinant.
Residual Flows [6] address this issue with an unbiased estimator based on randomized truncation.
They also introduce a new activation function LipSwish to avoid the gradient saturation problem.
i-DenseNets [7] improve
Residual Flows with a DenseNet variant accompanying learnable concatenation coefficients.
They also introduce the Concatenated LipSwish activation function to further improve the gradient flow.
Implicit neural networks map an input x to an output y through an implicit equation F ( x, y )
= 0 , in contrast with explicit models of the form y = F ( x ) .
The main advantage of implicit mappings is their ability to express a richer family of functions under the same parameter budget [28].
Implicit Flows [26] propose to use invertible implicit neural networks in normalizing flows and study an example where residual blocks and the inverse of residual blocks are stacked in an alternating fashion.
Monotone operator F : R n → R n is an operator that satisfies ⟨ u − v, x − y ⟩ ≥ 0 for all x, y ∈ R n and all u ∈ F ( x ) , v ∈ F ( y ) .
This definition generalizes monotonic functions in R , to R n .
The resolvent of a monotone operator F is defined as R F =
(Id + F ) − 1 , where Id is the identity operator.
The Cayley operator of F is defined as C F = 2 R F − Id .
It is well-known that the Cayley operators of monotone operators are exactly 1-Lipschitz operators.
The Cayley operator can be interpreted as a 45-degrees rotation (up to constant scaling) of the corresponding monotone operator in R n × R n ; see
Figure 1 for a 1D illustration.
Minty surjectivity theorem ([29] and [30, Theorem 21.1]) shows that the Cayley operators of maximally monotone operators are exactly 1-Lipschitz functions , which are defined at every point.
We formally state this correspondence in Theorem 2.
See [31] for a survey and [30] for a comprehensive treatment on monotone operators.
Monotone Flows
In this section, we propose a novel approach, the monotone formulation , and describe the training and inference procedure.
Moreover, we introduce a new activation function Concatenated Pila (CPila).
3.1 Motivation from monotone operator theory
ResNet-based normalizing flows consist of blocks of the form R ( x ) =
x + G ( x ) , where G ( x ) is a parametrized neural network.
If G ( x ) is a contraction (the Lipschitz constant Lip( G ) < 1 ), the
Banach fixed point theorem shows that R ( x ) is invertible.
In this case, Lip( R ) is strictly less than 2 , resulting in a severe limitation on the expressive power of the neural network R since a large Lipschitz constant is required in general to transform the source distribution into the target distribution.
However, as shown in the next example, G ( x ) need not be a contraction for R ( x ) to be invertible.
Example 1.
Consider G 1 ( x ) = 5 x .
Then, R 1 ( x )
= x + G 1 ( x ) = 6 x is invertible even though G 1 ( x ) is not a contraction
Hence, when considering a relaxed condition and parametrization, it is possible to enhance the expressive power to include a larger class of functions as in Example 1.
For this, we introduce monotone operators which have been widely studied in the fields of functional analysis and partial differential equations.
We first recall various definitions of the monotonicity of operators in R n :
Definition 1.
Consider an operator F : R n → R n .
(i) F is monotone if for all x, y ∈ R n , ⟨ F ( x ) − F ( y ) , x − y ⟩ ≥ 0 .
(ii) F is strictly monotone if for all x, y ∈ R n ( x ̸ = y ) , ⟨ F ( x ) − F ( y ) , x − y ⟩ > 0 .
(iii) F is η -strongly monotone ( η > 0) if for all x, y ∈ R n , ⟨ F ( x ) − F ( y ) , x − y ⟩ ≥ η
∥ x − y ∥ 22 .
(iv) F is maximally monotone if F is monotone and not a proper subset of any monotone operator.
It is noteworthy that monotonicity does not guarantee injectivity nor surjectivity, and strict monotonicity guarantees injectivity but not surjectivity.
However, we have the following theorem:
Theorem 1.
An η -strongly monotone continuous function F : R n → R n is invertible for any η > 0 .
Proof.
See Corollary 20.28 and Proposition 22.11 in [30].
Theorem 1 easily shows that R 1 ( x ) in Example 1 is invertible.
Moreover, if G ( x ) is L -Lipschitz
( L < 1) , R ( x ) is (1 − L )
-strongly monotone (see Appendix A.1) and thus invertible by Theorem 1.
As in Example 1, the ResNet-based formulation of normalizing flows does not include all possible invertible and monotone functions.
To address this issue, we directly parameterize monotone operators in terms of their Cayley operators.
We first formally state the rationale behind our construction.
Theorem 2.
Let F : R n → R n be an operator and C F = 2(Id + F ) − 1 − Id be its Cayley operator.
Then the followings hold.
(i) F is monotone ⇔ C F is 1-Lipschitz.
(ii) F is maximally monotone ⇔ C F is 1-Lipschitz and dom C F = R n .
Proof.
See Proposition 23.8 and Proposition 4.4 in [30]
Theorem 2 shows that each 1-Lipschitz function characterizes a unique MMO since the Lipschitz continuity of C F implies C F is single-valued.
Hence, the Lipschitz continuous parametrizations of the residual branch in ResNet-based models fit naturally for specifying C F .
We now revisit Example 1 and apply Theorem 2.
The Cayley operator of R 1 ( x ) = 6 x is given by
C R 1 ( x ) = 2(Id + R 1 ) − 1 ( x ) − x = − (5 / 7) x , which is indeed 1-Lipschitz in accord with Theorem 2.
In general, however, MMOs can be multivalued or undefined at some points.
However, when the
Cayley operator C F is an L -Lipschitz function with L < 1 , then F is (1 − L ) /
(1 + L )
-strongly monotone and (1 + L ) / (1 − L ) -Lipschitz continuous; see Appendix A.8 for a complete derivation.
Theorem 1 then implies F is invertible.
Hence, we use L -Lipschitz functions for Cayley operators.
3.2 Description of the monotone formulation
Definition 2.
Let G : R n → R n be a Lipschitz-continuous function with Lipschitz constant L < 1 .
The monotone formulation of G is defined as the following function F : R n → R n :
[Equation]
where Id denotes the identity function.
Here, F is well-defined since (Id + G ) / 2 has a unique inverse by the Banach fixed point theorem [5].
The definition of F is a direct application of Theorem 2.
On the other hand, from the Cayley operator identity C F + C F − 1 = 0 , we deduce that F − 1 has the Cayley operator − G .
Hence, if y = F ( x ) ,
[Equation]
which is well-defined by an analogous argument.
Thus we arrive at the following theorem:
Theorem 3.
For each G with Lip( G ) < 1 , there is a unique invertible function F as in (2) .
Note that by rearranging (2) or (3), we obtain the implicit equation connecting x and y :
[Equation]
To parametrize G , we use the architecture proposed in i-DenseNets [7], but replace the activation function with a new function called Concatenated Pila (CPila), which we introduce later in the paper.
Remark.
While we parametrize each layer as a monotone operator, the resulting functions need not be monotone.
For example, consider the π/ 3 counterclockwise rotation operator R π/ 3 in R 2 .
Although
R π/ 3 is monotone, the composition of R π/ 3 with itself is not monotone.
This illustrates that monotone functions are not closed under composition, hinting neural networks generated by the composition of invertible monotone operators can represent a much broader class of invertible functions than just monotone operators.
Indeed, functions of the monotone formulation are sup -universal diffeomorphism approximators since the monotone formulation includes the “nearly-Id” functions ( R L in section 4) whose composition suffices for sup -universal diffeomorphism approximation [32].
3.3 Training and inference
Training and inference largely follow previous approaches developed in [6, 26] except for minor modifications to adapt them to our monotone formulation.
For the function F of Definition 2, implicit differentiation 3 gives
[Equation]
(5) where J G = ∂G ( w ) /∂w at w = (cid:0) Id+ G 2 (cid:1) − 1 ( x ) ; see Appendix A.3 for derivation.
Now, we construct an unbiased estimator of the log determinant using the Hutchinson trace estimator and the unbiased
Russian roulette estimator, following the approach of [6].
Combining (1) and (5),
[Equation]
− [Figure] 5 (a) x y − 5 − 4 − 3 − 2 −     0 .0 . 4 (b) x y where p N ( n ) is a distribution with support over the positive integers.
Note that the derivative of the log-likelihood (6) can be evaluated by the memory-efficient Neumann gradient estimator as in [6].
Training consists of forward and backward propagations using (2) and (6).
For forward propagation,
F can be evaluated with fixed-point iterations since G is a contraction mapping while it suffices to backpropagate through (Id + G ) − 1 since F ( x ) = (cid:0) Id+ G 2 (cid:1) − 1 ( x ) − x
= (Id + G ) − 1 (2 x ) − x .
For this, we follow the approach of [26].
Let G be parametrized by θ .
Denote the training objective (6) as ℓ and write w =
(Id + G ) − 1 ( u ) (where u = 2 x ).
Implicit differentiation yields
[Equation]
where G and J G are evaluated at w = (Id + G ) − 1 ( u ) ; see Appendix A.5 for derivation.
Since the linear map J G : R n → R n is a contraction mapping, the vector-matrix inverse product in (7) can be evaluated by a fixed-point iteration.
For this, we adopt the secant method, but when convergence is not improved for 10 consecutive iterations, the Krasnoselskii-Mann iteration is used instead.
Inference consists of forward propagation through (3), which is almost the same as the forward propagation in training except that the sign of G is the opposite.
3.4 Concatenated positive identity 1-Lipschitz activation function
The LipSwish activation function proposed in [6] is based on the premise that having a non-vanishing second-order gradient would improve training.
However, LipSwish still suffers from the vanishing gradient problem since its first-order derivative stays around 0.5 in the neighborhood of x = 0 where most of the preactivations would reside.
Concatenated LipSwish (CLipSwish) [7] improves upon
LipSwish by concatenating LipSwish ( x ) and LipSwish ( − x ) , but a similar issue still occurs as shown in Figure 2 (b).
In this regard, we propose Pila (positive identity 1-Lipschitz activation) function, which makes the gradient flow equal to the identity function on x ≥ 0 during backpropagation.
When x < 0 , Pila converges to zero as x → −∞ while matching the identity function at x = 0 up to the third derivative.
The Pila function is described as follows:
[Equation]
Here, k > 0 is treated either as a fixed hyperparameter or a learnable parameter.
In all of our experiments, we set k = 5 and do not learn k .
Note that Lip ( Pila ) =
1 for any k > 0 .
As in [7], we introduce a concatenated version of Pila, which we call Concatenated Pila (CPila):
[Equation]
(9) where α 1 = 1 / 1 . 06 and α 2 = 0 . 2 .
By rescaling and translating with α 1 and α 2 , we fabricate the
CPila function to be 1-Lipschitz.
Figure 2 (b) describes the speed of the curve 4 , which shows CPila has a speed closer to 1 around x = 0 compared to the speed of CLipSwish.
[Figure]
1 1+ L 1 + L 1 − L 1 + L
[Figure]
−
[Figure]
1 −
[Figure]
In this section, we analyze the expressive power of the monotone formulation and related models. We prove that the monotone formulation is superior to the residual formulation [6] and the implicit formulation of Lu et al.
[26] when the Lipschitz constant L of the G -network is less than unity.
In practice, since L should not be close to unity, the monotone formulation is a better choice than the residual formulation or the implicit formulation of Lu et al.
We start with a formal definition of the function classes of each model.
Definition 3.
For 0 ≤ L < 1 , we define the following function classes:
L -Lipschitz functions
[Equation]
Residual formulation
[Equation]
Inverse residual formulation
[Equation]
Monotone formulation
[Equation]
Here, C 2 ( R n , R n ) denotes the set of twice continuously differentiable functions.
Note that the implicit formulation of Lu et al.
[26] defines one block as the composition of two functions F − 1◦ F 1 for any F 1 , F 2 ∈ R L .
We perform our analysis only on the second layers, which corresponds to the comparison between R L and I L .
For convenience, we use the following notation.
Definition 4.
Let A be one of G L , R L , I L , or M L , and α > 0 . Then,
[Equation]
We now examine the relations between the function spaces.
Theorem 4.
For 0 ≤ L < 1 , the following holds.
[Equation]
Proof.
See Appendix A.6 for a complete proof.
Figure 3 shows the ranges of q = F ( x 1 ) − F ( x 2 ) with respect to p = x 1 − x 2 for R L , I L , and M L , clearly demonstrating the relations (i)-(iv) of Theorem 4.
Theorem 4 shows the inverse residual formulation I L can only express functions that are a constant multiple of the functions expressed by the residual formulation R L .
Hence, the inverse residual formulation has no advantage over the residual formulation in this respect.
On the other hand, the monotone formulation M L can express functions that are not expressible by any constant multiple of the functions in R L , because 2 L/ (1 + L 2 ) > L whenever L < 1 .
When L can approach unity arbitrarily close, the monotone formulation has the same expressive power as the residual formulation or the inverse residual formulation.
However, this is not possible in practice because the variance of the log determinant estimator (6) diverges to infinity when L is greater than a specific threshold less than unity that depends on the distribution p N ( n ) [33].
Since the log determinant estimator (6) is a linear combination of the log determinant estimator of the same G -network’s residual formulation, (6) has a finite variance whenever the corresponding residual formulation’s estimator possesses a finite variance.
Suppose the constraints, in practice, bound the
Lipschitz constant of the G -networks to be at most L max < 1 .
Then, the residual formulation gives rise to R + R L max whereas the monotone formulation provides R + M L max
= R + R 2 L max 1+ L 2max ⊋ R + R L max .
An analogous argument holds for R + I L max .
Hence, the analysis shows that the monotone formulation outperforms the other formulations in terms of the expressive power.
Related work
Normalizing flows.
Normalizing flows require each layer to be invertible and the Jacobian determinant of each layer to be tractable, which have spurred multiple approaches for architecture design.
Restricted Jacobian models introduce specific structures to the Jacobian to make the Jacobian determinants tractable.
They include determinant identity-based models [1, 25] which use the Weinstein–Aronszajn identity with reduced intermediate layer dimensions, and coupling-based models [2, 3, 4, 34, 35] which split the input into two parts and transform one part conditioned on another.
However, the restrictions on the Jacobian limit their expressivity.
In contrast, free-form Jacobian models are not subject to the same constraints and include ResNet-based models (discussed in section 2) and Neural ODEs.
Neural ODEs [36] parametrize the rate of change in ODEs with neural networks.
They ensure invertibility through the existence and uniqueness theorem of ODEs.
However, they have relatively high time complexity since they require solving ODEs numerically on every iteration.
So far, the normalizing flows we have discussed are non-autoregressive models.
In contrast, autoregressive models [37, 38, 39] decompose probability distributions as conditional distributions depending only on the previous values.
While autoregressive models typically achieve higher loglikelihoods, they are orders of magnitudes slower to train or sample [4].
Therefore, we focus on non-autoregressive models in our paper.
Monotone operators in neural networks.
[40] pioneers monotone operator neural networks to assure the convergence of fixed-point iterations in implicit depth networks.
They build linear monotone operators by parametrizing the symmetric and antisymmetric parts separately.
Positive defininteness is enforced only on the symmetric part.
One shortcoming is that the construction is limited to linear monotone operators, which our work addresses using nonlinear Cayley operators.
[41] proposes the first neural network model of nonlinear MMOs by parametrizing the Cayley operators.
One key difference is that the 1-Lipschitz constraint on Cayley operators is enforced through a penalty loss, which may fail to control the Lipschitz constant globally [5].
Our method systematically ensures 1Lipschitzness by the use of spectral normalization [27] and 1-Lipschitz activations, following [5, 6, 7].
Also, their work does not consider normalizing flows.
To the best of our knowledge, our work proposes the first normalizing flow based on monotone operators.
Experiments
We start by validating the modeling capacity of our model with a 1D toy function.
We then evaluate
Monotone Flows on density estimation of 2D toy and image datasets and conduct ablation studies.
In all experiments, we parameterize the G -network in Definition 2 using i-DenseNets [7] with our activation function CPila (except for the 1D toy experiments that use CReLU and for part of the ablation study where we ablate CPila).
The G -network consists of fully-connected layers for 1D and 2D toy tasks and convolutional layers for image tasks.
We use the Poisson distribution for p N ( n ) whenever the estimator (6) is used.
We use Adam [42] for all experiments.
6.1 Toy 1D experiments
Figure 4 illustrates the expressive power of the monotone formulation with a 1D toy example.
We fit a 1D function that consists of four consecutive step-like shapes designed to exhibit a high Lipschitz constant.
We consider four different 2-layer networks, as shown in Figure 4.
Experimental details
[Figure]
We display the average of the test loss for the last
[Table] are discussed in Appendix C.1.
We denote residual block, inverse residual block, and monotone block by RB, IRB, and MB.
Note that the learnable scaling (LS) is used only for Figure 4 (b)-(d).
The G -networks in the experiments (a)-(d) have the same structure, and hence the same number of parameters are used.
Consequently, differences in the result solely come from the differences in the formulations.
Figure 4 (a) and (b) demonstrate that the expressive power is improved when using R +
R L instead of R L .
Figure 4 (b) and (c) reveal that the expressive power of R + R L and R + I L are similar, in agreement with Theorem 4.
Figure 4 (d) shows that our monotone formulation R + M L exceedingly outperforms the other networks.
6.2 Density estimation on 2D toy data
We use the 2D toy datasets provided with the official source code of i-DenseNets [7].
Following [7], we use ten flow blocks (see Appendix C.2 for details).
Exact Jacobian evaluation is used instead of the stochastic estimation (6) as it is inexpensive for 2D data.
We train the models for 50K epochs with the learnable concatenation of i-DenseNets enabled after 25K epochs.
We use a batch size of 500.
Qualitative and quantitative results in Figure 5 and Table 1 show that Monotone Flows outperform i-DenseNets in almost all settings, especially on challenging datasets.
One limitation of normalizing flows manifests in the qualitative results.
For instance, in the dataset ‘rings’, the circles cannot be closed because normalizing flows are required to preserve topology.
Despite the inherent restriction, our model finds an excellent approximation.
6.3 Density estimation on images
We evaluate our method on MNIST [43], CIFAR-10 [44], and the downscaled version of ImageNet in 32 × 32 and 64 × 64 [45].
6 We use (6) for estimating the Jacobian determinant.
We use uniform
[Table]
ImageNet
[Table]
[Figure]
dequantization for both training and testing.
For MNIST and CIFAR-10, the learnable concatenation is enabled after 25 epochs; for ImageNet32 and ImageNet64, it is enabled from the start.
We train for 100, 1,000, 20, 20 epochs with batch sizes 64, 64, 256, 256 with learning rates 0.001, 0.001, 0.004, 0.004 for MNIST, CIFAR-10, ImageNet32, ImageNet64, respectively.
We report single-seed results following [4, 5, 6, 7].
Details are in Appendix C.3.
Samples from the trained models are displayed in
Figure 6.
More samples can be found in Appendix E.
Table 2 shows our model outperforms baselines on all datasets considered, demonstrating the effectiveness of our proposal.
In concrete numbers, our model reduces 0.042 bpd in MNIST, 0. bpd in CIFAR-10, 0.019 bpd in ImageNet32, and 0.023 bpd in ImageNet64 compared to baseline normalizing flow models. Remark on comparison.
Flow++ [34] introduced variational dequantization which improves the log-likelihood, reporting 3.08 and 3.86 bits/dim on CIFAR-10 and ImageNet32, respectively.
With variational dequantization, Monotone Flows achieve 3.062 and 3.901 bits/dim on the same benchmarks.
Also, there is a line of research that achieves better likelihood values by introducing latent variables in extra dimensions, at the cost of losing exact likelihood computation [46, 47, 48]. On the other hand, ScoreFlow [49] achieves better likelihood values by training with a weighted score-matching objective that lower bounds the log likelihood. We leave it for future research to adapt these methods for Monotone Flows. 6.4 Ablation study
[Table]
Our model has two new components: the monotone formulation and CPila activation function.
We quantify their contributions by ablating either of the two components while leaving the other intact.
When ablating the monotone formulation (row #2), we used the residual connections of [7]; when ablating CPila (row #3), we used CLipSwish instead.
Table 3 compares the performance of i-DenseNets (row #1) and the ablated models on the CIFARdataset under the same experimental setup as in section 6.3.
Details are in Appendix C.5.
Results confirm that both components are beneficial for achieving a low bits-per-dimension (bpd).
The monotone formulation provides a higher performance gain (0.028 bpd) than CPila (0.014 bpd).
Conclusion
We presented Monotone Flows, a novel parametrization of normalizing flows based on monotone operators (the monotone formulation) combined with a new activation function called Concatenated
Pila (CPila).
Our theoretical analysis elucidated why the monotone formulation is more expressive than the residual or inverse residual formulations.
Experimental results demonstrated the effectiveness of the proposed method on various density estimation benchmarks.
We believe our contribution is a solid step towards improving the expressivity of normalizing flows.
Acknowledgments and Disclosure of Funding
Our work is supported by ICT Creative Consilience program (IITP-2022-2020-0-01819) supervised by the IITP, and Kakao Brain Corporation.
The work of Y. Hong was supported by Basic
Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2021R1A2C1093579) and the Korea government (MSIT) (No.
2022R1A4A3033571).