L EARNING C ONTINUOUS N ORMALIZING F LOWS F OR F ASTER C ONVERGENCE T O T ARGET D ISTRIBUTION VIA A SCENT R EGULARIZATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Normalizing ﬂows (NFs) have been shown to be advantageous in modeling complex distributions and improving sampling efﬁciency for unbiased sampling.
In this work, we propose a new class of continuous NFs, ascent continuous normalizing ﬂows (ACNFs), that makes a base distribution converge faster to a target distribution.
As solving such a ﬂow is non-trivial and barely possible, we propose a practical implementation to learn ﬂexibly parametric ACNFs via ascent regularization and apply in two learning cases: maximum likelihood learning for density estimation and minimizing reverse KL divergence for unbiased sampling and variational inference.
The learned ACNFs demonstrate faster convergence towards the target distributions, therefore, achieving better density estimations, unbiased sampling and variational approximation at lower computational cost.
Furthermore, the ﬂows show to stabilize themselves to mitigate performance deterioration and are less sensitive to the choice of training ﬂow length T .
I NTRODUCTION
Normalizing ﬂows (NFs) provide a ﬂexible way to deﬁne an expressive but tractable distribution which only requires a base distribution and a chain of bijective transformations (Papamakarios et al., 2021).
Neural ODE (Chen et al., 2018) extends discrete normalizing ﬂows (Dinh et al., 2014; 2016;
Papamakarios et al., 2017; Ho et al., 2019) to a new continuous-time analogue by deﬁning the transformation via a differential equation, substantially expanding model ﬂexibility in comparison to the discrete alternatives.
(Grathwohl et al., 2018; Chen and Duvenaud, 2019) propose a computationally cheaper way to estimate the trace of Jacobian to accelerate training, while other methods focus on increasing ﬂow expressiveness by e.g. augmenting with additional states (Dupont et al., 2019; Massaroli et al., 2020), or adding stochastic layers between discrete NFs to alleviate the topological constraint (Wu et al., 2020).
Recent diffusion models like (Hodgkinson et al., 2020; Ho et al., 2020; Song et al., 2020; Zhang and Chen, 2021) extend the scope of continuous normalizing ﬂows (CNFs) with stochastic differential equations (SDEs).
Although these diffusion models signiﬁcantly improve the quality of the generated images, introduced diffusion comes with some costs: some models do not allow for tractable density estimation; or the practical implementations of these models rely on a long chain of discretizations, thus needing relatively more computations than tractable CNF methods, which can be critical for some use cases such as online inference.
(Finlay et al., 2020; Onken et al., 2021; Yang and Karniadakis, 2020) introduce several regularizations to learn simpler dynamics using optimal transport theory, which decrease the number of discretization steps in integration and thus reduce training time.
(Kelly et al., 2020) extends the L 2 transport cost to regularize any arbitrary order of dynamics.
Although these regularizations are beneﬁcial for decreasing computational costs, it does not improve the slow convergence of density to the target distributions like vanilla CNF models shown in Figure 1.
To accelerate the ﬂow convergence, STEER (Ghosh et al., 2020) and TO-FLOW (Du et al., 2022) propose to optimize ﬂow length T in two different approaches: STEER randomly samples the length during training while TO-FLOW establishes a subproblem for T during training.
To understand the effectiveness of these methods, we train multiple Neural ODE models with different ﬂow length T n for a 2-moon distribution.
Figure 2 shows the estimated log-likelihoods of all learned ﬂows.
Although optimizing T dynamically performs a model selection during training and makes models reach higher estimates at shorter ﬂows, it cannot prevent the divergence of ﬂows after T n .
Furthermore, ﬂows with shorter length are more limited in expressiveness and sensitive to ﬂow length.
In this work, we present a new family of CNFs, ascent continuous normalizing ﬂows (ACNFs), to address the aforementioned problems.
ACNF concerns a ﬂow that makes a base distribution monotonically converge to a target distribution, and the dynamics is imposed to follow the steepest
ACNF.
However, solving such a steepest ﬂow is non-trivial and barely possible.
We propose a practical implementation to learn parametric ACNFs via ascent regularization.
The learned ACNFs exhibit three main distinct behaviors: 1) faster convergence to target distribution with less computation; 2) self-stabilization to mitigate ﬂow deterioration; and 3) insensitivity to ﬂow training length T .
We demonstrate the beneﬁcial behaviors in three use cases: modeling data distributions; learning annealed samplers for unbiased sampling; and learning a tractable but more ﬂexible variational approximation.
C ONTINUOUS N ORMALIZING F LOWS
Considering a time- t transformation z ( t ) =
t ( x ) on the initial value x , i.e. z (0) =
x , the change of variable theorem reveals the relation between the transformed distribution p t ( z ( t )) and p ( x ) :
[Equation]
where J  is the Jacobian matrix of  .
As  t normalizes x towards some base distribution, p t ( z ( t )) is referred as normalized distribution, starting from the data distribution p ( x ) .
Continuous normalizing ﬂow is the inﬁnitesimal limit of the chain of discrete ﬂows and the inﬁnitesimal transformation is an ordinary differential equation (ODE):
[Equation]
The instantaneous change of variable theorem (Chen et al., 2018, theorem 1) shows the inﬁnitesimal changes of log p t ( z ( t [Equation]
[Equation]
Thus, the normalized distribution log p t ( z ( t )) at t can be obtained by integrating eq.(3) backwards with a common approximation to the base distribution µ , i.e. p T ⇡ µ :
[Equation]
where z ( t ) =
x +
R t 0 f ( z ( ⌧ ) , ⌧ )d ⌧ .
The accuracy of log p 0 ( x ) , obtained by the right hand side, depends on the approximation error of p T to µ and the error varies at different z ( T ) .
To avoid the problem and investigate how the ﬂow length affects on the distribution modeling, we introduce ˜ p t ( x ) to estimate the data distribution p ( x ) by a t -length ﬂow  t , via the change of variable theorem:
[Equation]
Shown by eq.(4) and Figure 3, ˜ p t initiates from the base distribution, i.e. ˜ p 0 ( x ) = µ (  0 ( x ))
=
µ ( x ) .Combining eq.(4) and eq.(1), the estimated likelihood of time- t ﬂow is related to the normalized distribution p t ( z ( t )) :
[Equation]
It shows that as p t !
µ , ˜ p t ( x ) !
p ( x ) .
When there exists a ﬂow, of which the normalized density p T is equal to the base distribution, i.e. p T = µ , then the estimated likelihood of time- T ﬂow is exact to the true data distribution, i.e. ˜ p T ( x ) =
p
( x ) .
Like the instantaneous change of variable theorem, we derive the inﬁnitesimal change of time- t estimated log-likelihood
[Equation]
Proposition 1 (Instantaneous Change of Log-likelihood Estimate) .
Let z ( t ) be a ﬁnite continuous random variable at time t as the solution of a differential equation d z ( t ) d t = f
( z ( t ) , t ) with initial value z (0) = x .
Assuming that ˜ p
= µ at t = 0 and f is uniformly Lipschitz continuous in z and t , then the change in estimated log-likelihood log ˜ p t ( x ) at t follows a differential equation:
[Equation]
Proof.
See Appendix A.1 for derivation and its relation to eq.(3).
Unlike the integral for p t ( z ( t )) that relies on the approximation and requires to solve the whole trajectory z ( ⌧ ) , ⌧ 2 [0 , T ] , the logarithm of density estimation, log ˜ p t ( x ) , at any or/and multiple t can be evaluated exactly in one single forward integration:
[Equation]
A SCENT C ONTINUOUS N ORMALIZING F LOWS
Using KL divergence as distance measure of distributions, we have the following duality:
[Equation]
so that maximum likelihood learning to ﬁt ˜
p t ( x ) for data from p ( x ) is equivalent to minimizing the reverse KL divergence in normalization direction.
Besides, we can measure the convergence rates of both KL divergences by their time derivative.
Therefore, we deﬁne ascent continuous normalizing ﬂows (ACNF) that decrease KL ( p t ( z ( t ))
|| µ ( z ( t ))) or equivalently increase the expectation of log ˜ p t ( x ) monotonically:
[Equation]
By applying total variation, we can ﬁnd the dynamics for the steepest descent of reverse KL divergence or the steepest ascent of the log-likelihood expectation:
Theorem 1 (Dynamics for Steepest Ascent Continuous Normalizing Flows) .
Let z ( t ) be a ﬁnite continuous random variable and the solution of a differential equation d z ( t ) d t = f
( z ( t ) , t ) with initial value z (0) =
x .
Its probability p t ( z ( t )) subjects to the continuity equation @
t p t + r · ( p t f ) = 0 .
The dynamics of the steepest ﬂow for decreasing KL ( p t ( z ( t ))
|| µ ( z ( t ))) is
[Equation]
Proof.
See Appendix A.2 for detailed derivation.
The steepest dynamics is deﬁned by the difference between two gradients: r log µ and r log p t w.r.t.
z ( t ) .
There are a few important implications of eq.(6): 1) the dynamics is time-variant as p t simultaneously evolves with the ﬂow; 2) at time t , it only depends on the current state z ( t ) , thus no history is needed; 3) the ﬂow is initiated by the difference between r log µ ( x ) and r log p ( x ) , and it gradually slows down and eventually stops when p t ( z ( t )) converges to µ ( z ( t )) .
The convergence rate of the steepest ﬂow can also be proven as the negative
Fisher divergence, @
KL ( p t k µ ) / @
t =
F ( p t || µ ) =
E p t kr log µ ( z )  log p t ( z ) k 22 , therefore this deterministic CNF is related to (overdamped) Langevin diffusion, see Appendix A.3 for the derivation of convergence rate and detailed discussion of their relation.
This optimal ﬂow can also be considered as a special instance of Wasserstein gradient ﬂow (Ambrosio et al., 2005) with KL divergence as the energy functional.
Previous works (Finlay et al., 2020; Yang and Karniadakis, 2020; Onken et al., 2021) apply the optimal transport theory to regularize ﬂow dynamics from Euclidean space perspective, while Wasserstein gradient ﬂow instead regularizes it in probability measure space.
We refer readers to (Ambrosio et al., 2005) for accessible introduction.
In some cases, the ﬂow can be solved indirectly by introducing a potential,
V ( z ( t ) , t ) =
p t ( z ( t )) /µ
( z ( t )) , and it has a partial differential equation (PDE):
[Equation]
with the initial condition V ( z (0) , 0)
=
p 0 ( z [Equation] µ ( z (0))
=
p ( x ) µ ( x ) .
See Appendix A.4 for its derivation.
However, solving this PDE for p t ( z ( t )) is non-trivial as the closed form solution is typically unknown.
JKO integration is commonly used in literature (Mokrov et al., 2021; Fan et al., 2021), which approximates the dynamics of density p t by its time discretization, but it requires to know the initial condition while p ( x ) is generally unknown and needs to be modeled from samples.
(Tabak and
Vanden-Eijnden, 2010) proposes to approximate p ( x ) by the spatial discretization of samples, which hardly can be scale up even for intermediate dimensions.
To tackle these difﬁculties and accelerate ﬂows for faster convergence, we propose ascent regularization to learn parametric ACNFs, inspired by previous works (Yang and Karniadakis, 2020; Onken et al., 2021; Finlay et al., 2020; Kelly et al., 2020; Ghosh et al., 2020) that enforce certain behaviors of ﬂows via regularization in training.
Ascent regularization penalizes the difference between the parametric dynamics and the steepest dynamics in eq.(6), by minimizing k f ✓  f ⇤ k 22 , which needs to evaluate score function r log p t ( z ( t )) .
We propose the instantaneous change of the score function:
Theorem 2 (Instantaneous Change of Score Function) .
Let z ( t ) be a ﬁnite continuous random variable with probability density p t ( z ( t )) at time t .
Let d z ( t ) d t = f
( z ( t ) , t ) be a differential equation describing a continuous-in-time transformation of z ( t ) .
Assuming that f is uniformly Lipschitz continuous in z and t , the inﬁnitesimal change in the gradient of log-density at t is
[Equation]
Proof.
See Appendix A.5 for detailed derivation.
r log p ( z ( t ) , t ) follows a linear matrix differential equation, where the linear coefﬁcient matrix is Jacobian and the bias term is the gradient of divergence.
To be noted, an alternative proof is derived in concurrent work (Lu et al., 2022, theorem D.1).
We demonstrate ACNF for two learning cases: maximum likelihood learning for data modeling and density estimation in Section 4; minimizing reverse KL divergence for learning annealed samplers for unbiased sampling in Section 5.
Require: Data samples X
= { x j } j =1 ,...,N , parameteric dynamics of ﬂow f ✓ , length of ﬂow T , ascent regularization coefﬁcient  , base distribution µ
Initialize ✓ while ✓ is not converged do
Sample a mini-batch of data x i ⇠ X
Integrate augmented states [ z i ( t ) , log ˜ p t ( x i )] forward with initial value [ x i , 0 ] from 0 to T
Integrate augmented states [ z i ( t ) ,
r log p t ( z i ( t ))] backwards with initial value [ z i ( T ) , r log µ ( z i ( T ))] from T to
Compute loss function L in eq.(9) and r ✓ L by adjoint sensitivity method Update ✓ by gradient descent algorithm end while
Algorithm for unbiased sampling
Require: target distribution ⇡ =  /Z , parameteric dynamics of ﬂow f ✓ , length of ﬂow
T , number of samples N , ascent regularization coefﬁcient  , base distribution µ
Initialize ✓ while ✓ is not converged do
Sample z i
⇠ p 0 = µ
Evaluate log µ ( z i 0 ) and r log µ ( z i 0 )
Integrate augmented states [ z i ( t ) , log p t ( z i ( t )) , r log p t ( z i ( t ))] with initial value
[ z i 0 , log µ ( z i 0 ) , r log µ ( z i 0 )] from 0 to T
Evaluate log w ( z i ( T ))
=
log  ( z i ( T ))
log p T (
z i ( T ))
Compute loss function L in eq.(10) and r ✓ L by adjoint sensitivity method
Update ✓ by gradient descent algorithm end while
M AXIMUM L IKELIHOOD L EARNING OF ACNF FOR DENSITY ESTIMATION VIA A SCENT R EGULARIZATION
For maximizing likelihood learning of ˜ p T to ﬁt data, the total objective with ascent regularization is: min
[Equation]
where  is the ascent regularization coefﬁcient to control the trade-off between maximizing likelihood and regularization on the ascent behavior of the learned dynamics.
When  = 0 , ACNF degrades to CNF.
The ﬁrst term in eq.(9) is obtained by integrating eq.(5) over [0 , T ] , simultaneously with z ( t ) , while the ascent regularization can be integrated backwards with augmented initial [ z ( T ) ,
r log p T ( z ( T ))] , with r log p T ( z ( T ))
⇡ r log µ ( z ( T )) .
We summarize the pseudo-code for maximum likelihood learning of ACNFs in Algorithm 1.
We show the interpretation of ascent regularization as score matching in Section A.6 in Appendix, thus Algorithm 1 can be implemented in more efﬁcient ways like (Lu et al., 2022; Song et al., 2021) for some cases.
L EARNING ACNF AS ANNEALED SAMPLER FOR UNBIASED SAMPLING
Except modeling data samples and performing density estimation, NF as a sampler shows to be more sample efﬁcient in Annealed Importance Sampling (AIS) (Neal, 2001) when comparing to classic
MCMC methods.
A typical AIS and its extension use a sequence of annealed targets { ⇡ k } k =
: K that bridges an easy-to-sample and tractable distribution ⇡ 0 = µ to the target ⇡ K := ⇡ =  ( · ) /Z .
SNF (Wu et al., 2020) and AFT (Arbel et al., 2021) propose to ﬁt K discrete NFs and each NF approximates the transport map between ⇡ k  1 and ⇡ k .
However, the sampling convergence rate largely depends on the pre-deﬁned annealed targets.
Besides, more steps are needed to decrease the variance of the estimator, which comes at an additional computational cost (Doucet et al., 2022).
As ACNF can also deﬁne a ﬂow from a base distribution to a target distribution, it can learn a continuous annealed target ﬂow replacing the pre-deﬁne ones and later generates samples.
Different to (Grosse et al., 2013), the annealed targets by ACNF do not require the speciﬁc form of distributions.
As the distribution by ACNF is enforced to converge faster to the target distribution, ACNF sampler potentially generates better samples than CNF or linear annealed scheduling especially at limited steps of sampling, thus estimates, e.g. on logarithm of normalization constant log Z , are more accurate.
Different to maximum likelihood in Section 4, training ACNF for an annealed sampler is to minimize the reverse KL divergence KL ( p T ( z ( T ))
k ⇡ ( z ( T ))) .
It can be evaluated up to a constant by the logarithm of importance weights, log w ( z ( T ))
= log  ( z ( T ))
log p T ( z ( T )) .
With ascent regularization, the total objective becomes:
[Equation]
where f ( z ( t ) , t ; ✓ ) is the annealed generation dynamics.
Unlike the previous section, as the base distribution is known here, log p t ( z ( t )) and r log p t ( z ) are integrated simultaneously with z ( t ) =
z (0) +
R t 0 f ( z ( ⌧ ) , ⌧ )d ⌧ with a sample z (0) ⇠ µ .
We summarize pseudo-code for learning ACNF annealed sampler in Algorithm 2.
Once ACNF sampler is learned, it can generate unbiased samples: generate one-shot samples from ACNF with ﬂow length t according to computation budget; correct samples by resampling according to importance weights like (Müller et al., 2019) or by Markov
Chain Monte Carlo methods with Metropolis-Hastings correction.
E XPERIMENTS 6. D ENSITY E STIMATION ON T OY 2D D ISTRIBUTIONS
Before we deploy ACNF for modeling complex distributions, we ﬁrst examine it on a 2-modal
Gaussian mixture and use a standard Gaussian as the base distribution in 2D. Figure 4 shows that the potential ﬁeld of learned ACNF is very similar to the PDE solutions in eq.(7) while CNF potential converges much slower than ACNF and then diverges after T .
See Appendix A.7 for experiment details and comparison on more  , T and other regularization methods.
We then train vanilla CNF, RNODE (Finlay et al., 2020) and ACNFs to model data from various 2D toy distributions and visualize the density estimation along ﬂows.
Figure 5 shows the densities at t 2 [0 , 2 T ] , T = 10 by learned CNF and ACNFs with various regularization coefﬁcients for 2-moon distribution.
The densities that are close to the target distribution are highlighted inside the red border.
We show that even slight regularization makes the learned ﬂows to 1) converge much faster towards the target; 2) maintain the best estimations for long time after T .
The quantitative evaluation on the log-likelihood estimates indicates the same conclusion, seen from the left of Figure 6.
More analysis on different T and experiment setups are given in Appendix A.8.
One may suspect that more complex dynamics explain the faster ascent of likelihood estimates.
To validate the actual improvements by ACNF, we evaluate the number of function evaluations (NFEs) by counting the times when an adaptive step size solver calls the dynamics function in the integration, with and without log-likelihood for all models.
The marks in the left of Figure 6 show NFEs along ﬂow while the middle one plots log-likelihood estimates versus NFEs.
ACNFs clearly demonstrate that they learn even less complex dynamics than CNF and RNODE, and log-likelihood gain per
NFE of ACNFs are much higher than the two baselines especially at early stage.
Among different ascent regularization, a larger coefﬁcient leads to more rapid gain on the log-likelihoods initially, however, too large regularization over-constrains models to reach a good maximum.
A moderate regularization beneﬁts on the balance of maximum likelihood and faster convergence.
Furthermore, we report NFEs at t/T = 1 for CNF, RNODE and ACNFs trained with various T = 0 .
5 , 1 , 5 ,and  = 0 .
on the right of Figure 6.
ACNFs have generally lower NFEs than CNF and RNODE and most models report lowest NFEs at T = 1 .
It indicates that optimizing T by TO-FLOW (Du et al., 2022) and STEER (Ghosh et al., 2020) can decrease computational cost, however, the gain is marginal as NFEs do not grow linearly to integration length.
More importantly, even an appropriate T cannot prevent slow convergence and density deterioration of CNF as shown in Figure 14 in Appendix A.8.
Figure 7 shows density evaluations on more multi-modal distributions.
Learned ACNFs show faster convergence than CNFs for all distributions and give a higher maximum density estimation on the challenging task, e.g. Olympics distribution.
6. D ENSITY E STIMATION ON R EAL D ATASETS
We demonstrate density estimations on real-world benchmarks datasets including POWER, GAS,
HEPMASS, MINIBOONE from the UCI machine learning data repository and BSDS300 natural image patches.
Like FFJORD, all tabular datasets and BSDS300 are pre-processed as in (Papamakarios et al., 2017).
Table 1 reports the averaged NLLs on test data for FFJORD, RNODE and ACNF trained with different  .
The detailed description of experiments and models refers to Appendix A.9.
Although FFJORD with multi-step ﬂows increases the ﬂexibility of ﬂows, it tends to have a worse performance than the base distribution initially and then improves NLL mainly at the late stage of ﬂows.
A larger ascent regularization of ACNFs contributes to more rapid initial increases on NLL that these ﬂows transform the base distribution faster towards the data distribution.
When training on
HEPMASS and BSDS300, a too large regularization coefﬁcient impedes model to converge.
6. ACNF AS A FASTER ANNEALING SAMPLING PROPOSAL FOR UNBIASED SAMPLING
Following Algorithm 2, we train CNF and ACNFs with regularization coefﬁcients  = 0 .
01 for learning annealed targets.
We evaluate the normalization constants of a Gaussian mixture with 8 components whose means are ﬁxed evenly in space and standard deviation as 0.
and the base distribution chooses a Gaussian N ( 0 , 3 2 I ) to have adequate support for the target distribu-
[Table]
tion.
Figure 8 compares the estimates of log Z along ﬂows and reports estimates versus NFEs.
Except
CNF and ACNFs, we also evaluate the linear annealed target log  k ( · ) =
k log  ( · )+(1   k ) log ⇡ 0 , where scheduling is  k = k/K = t k /T and K
= 20 , using {170, 25, 10}-step Metropolis sampler between each intermediate target.
As ACNFs converge faster towards the target distribution than CNF, the one-shot samples from ACNFs are less biased than those from CNF especially at the early stage of the ﬂows.
Besides, ACNFs are more computationally efﬁcient in terms of accuracy gain per NFEs.
ACNFs with coefﬁcients  = 0 . 01 , 0 .
001 shows less biased estimates earlier than the best tuned linear annealed target.
Besides, the linear annealed target requires at least 1 order more computations than CNF and ACNFs for comparable accuracy due to slow mixing of Metropolis sampler, and is sensitive to the number of MC steps.
Therefore, ACNFs prevent slow convergence of sampling by the suboptimal annealed target and have more accurate estimates with less computation.
Figure 18 in
Appendix A.10 shows generated samples by all methods in Figure 8.
Adding MC steps with learned
ACNFs can further accelerate sample convergence to target and increase the expressiveness of ACNF.
6. V ARIATIONAL I NFERENCE WITH ACNF S
In addition to density estimation and unbiased sampling, CNFs provides more ﬂexible variational approximations to improve variational inference (Rezende and Mohamed, 2015).
We follow the
[Table]
latent inference at the end of the ﬂow to the observation dimension.
To train a VAE-ACNF model, the log weight in eq.(10) is replaced by an ELBO estimate like (Kingma and Welling, 2014).
We compare VAE-ACNF to VAE-FFJORD and vanilla VAE without ﬂow on MNIST data.
To make a fair comparison, we ﬁx the learned encoder-decoder when training all three models.
A detailed description of model architecture and experimental setup can be found in Appendix A.11.
The averaged negative ELBO on test data of vanilla VAE is 86 .
50 and that of VAE-FFJORD and VAEACNFs with two regularization  = 1 e  4 , 1 e  3 along the ﬂows are reported in Table 2.
VAE-ACNFs show faster gains on ELBO at early stage of the ﬂows, compared to VAE-FFJORD, and a larger coefﬁcient accelerates the convergence of approximation to true posterior.
VAE-ACNFs circumvent the ELBO deterioration which happens to VAE-FFJORD, thanks to the self-stabilization behavior of
ACNF.
We also show some reconstruction examples from VAE-ACNF in Figure 9 and Figure 19 in
Appendix A.11.
These reconstructions tend to correct some defects from original images, add details to strengthen identities while remaining sharp as the original images.
S COPES AND L IMITATIONS
While we have demonstrated that ascent regularization is effective to learn ACNF that converges faster to a target distribution, there are still a number of limitations, which we would like to address in the future.
First, more efﬁcient implementations on score function evaluation by e.g. estimators or model design or via score matching learning (Song et al., 2021) can accelerate the model training for high-dimensional problems.
Second, Hypernet (Ha et al., 2016) is found very effective to illustrate faster convergence behavior of ACNF as time exerts a large impact on the dynamics, however, it is slower to train than other simpler network architectures.
A better architecture design for dynamics can improve the training speed while maintaining the faster convergence behavior of ﬂows.
Third, although the proposed ACNF and ascent regularization are discussed under the framework of CNF, the concept can be easily extended and explored for score-based models and other stochastic ﬂows.
Finally, the proposed ACNF and ascent regularization can be extended for a sequence of distributions, e.g. inference of sequential data, and computationally efﬁcient ACNF may beneﬁt online inference.
C ONCLUSION
We have proposed ACNFs, a new class of CNFs, that deﬁne ﬂows with monotonic convergence toward a target distribution.
We derive the dynamics for the steepest ACNF and propose a practical implementation to learn parametric ACNFs via ascent regularization.
We demonstrate ACNF in three use cases: modeling data and performing density estimation, learning an annealed sampler for unbiased sampling, and learning variational approximation for variational inference.
The learned
ACNFs illustrate three beneﬁcial behaviors: 1) faster convergence to the target distribution with less computation; 2) self-stabilization to mitigate performance deterioration; 3) insensitivity to ﬂow training length T .
Experiments on both toy distributions and real-world datasets demonstrate the effectiveness of ascent regularization on learning ACNFs for various purposes.
Ethics Statement As this work mainly concerns to propose a ﬂow-based model and practical implementation for learning, it does not involve human subjects, practices to data set releases, or security and privacy issue.
At this stage of study, we do not foresee the effects of potential system failures due to weaknesses in the proposed methods.
Reproducibility Statement All proposition (proposition 1) and theorems (theorem 1 and 2) proposed in this paper are proved with details in Appendix A.1, A.2 and A.5 as well as other minor derivations mentioned in main body of the paper.
The pseudo-code for both learning cases are provided in Algorithm 1 and Algorithm 2.
The datasets, models, experiment setups for each demonstration are described in details in Appendix A.8 ⇠ A.11.
Furthermore, we attach some source codes in supplementary material for further checkup.