Whitening Convergence Rate of Afﬁne Coupling Flows
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Afﬁne coupling ﬂows (RealNVPs) are a popular normalizing ﬂow architecturethat works surprisingly well in practice.
This calls for theoretical understanding.
Existing work shows that such ﬂows weakly converge to arbitrary data distributions[1].
However, they make no statement about the stricter convergence criterion usedin practice, the maximum likelihood loss.
For the ﬁrst time, we make a quantitativestatement about this kind of convergence: We prove that afﬁne coupling ﬂowsperform whitening of the data distribution (i e. diagonalize the covariance matrix)and derive corresponding convergence bounds that show a linear convergence ratein the depth of the ﬂow.
Numerical experiments demonstrate the implications ofour theory and point at open questions.
Introduction
Normalizing ﬂows [2, 3] are among the most promising approaches to generative machine learningand have already demonstrated convincing performance in a wide variety of practical applications,ranging from image analysis [4, 5, 6, 7, 8] to astrophysics [9], mechanical engineering [10], computational biology [11] and medicine [12].
As the name suggests, normalizing ﬂows represent complexdata distributions as bijective transformations (also known as ﬂows or push-forwards ) of standardnormal or other well-understood distributions.
In this paper, we focus on a theoretical underpinning of afﬁne coupling ﬂows (RealNVPs [5]), aparticularly effective realization of normalizing ﬂows in terms of invertible neural networks.
Allof the above mentioned applications are actually implemented using variants of RealNVP.
Theircentral building blocks are afﬁne coupling layers , which decompose the space into two subspacescalled active and passive subspace.
Only the active dimensions are transformed conditioned on thepassive dimensions, using afﬁne mappings that are computationally easy to invert.
In order to varythe assignment of dimensions to the active and passive subspaces, coupling layers are combined withpreceding orthonormal transformation layers into afﬁne coupling blocks .
These blocks are arrangedinto deep networks such that the orthonormal transformations are sampled uniformly at random fromthe orthogonal matrices and the afﬁne layers are trained with the maximum likelihood objective, see Equation (2).
Upon convergence of the training, the sequence of coupling blocks gradually transformsthe probability density that generated the given training data, into a standard normal distribution (ofthe same dimensionality) and vice versa.
Since the resulting normalizing ﬂows deviate signiﬁcantly from optimal transport ﬂows [13] andthe bulk of the mathematical literature is focusing on optimal transport, an analysis tailored tothe RealNVP architecture is lacking.
In a landmark paper, [1] proved that sufﬁciently large afﬁnecoupling ﬂows weakly converge to arbitrary data densities.
The notion of weak convergence iscritical here, as it does not imply convergence in maximum likelihood [14, Remark 3].
Maximumlikelihood (or, equivalently, the Kullback-Leibler divergence) is the loss that is actually used inpractice.
It can be used for gradient descent and it guarantees not only convergence in samples(“ x ⇠ q ( x ) ! x
⇠ p ( x ) ”) but also in density estimates (“ q ( x ) ! p ( x ) ”).
It is strong in the sense
[Figure]
We observe in this experiment that our bound is predictive for how reduces S . that the square root of the KL-divergence upper bounds (up to a factor 2) the total variation metric, andhence also the Wasserstein metric if the underlying space is bounded [15].
Moreover, convergenceunder the KL-divergence implies weak convergence which is fundamental for robust statistics [16].
We take a ﬁrst step towards showing that afﬁne coupling blocks also converge in terms of maximum likelihood.
To the best of our knowledge, our paper presents for the ﬁrst time a quantitativeconvergence analysis of normalizing ﬂows realized by RealNVPs based on this strong notion ofconvergence.
Speciﬁcally, we make the following contributions towards this goal:
• We utilize that the loss of a normalizing ﬂow can be decomposed into two parts (Figure 1):The divergence to the nearest Gaussian ( non-Gaussianity ) plus the divergence of thatGaussian to the standard normal ( non-Standardness ).•
The contribution of a single coupling layer on the non-Standardness is analyzed in terms ofmatrix operations (Schur complement and scaling).• Explicit bounds for the non-Standardness after a single coupling block in expectation overall orthonormal transformations are derived.
• We use these results to prove that a sequence of coupling blocks whitens the data covarianceand to derive linear convergence rates for this process.
We conﬁrm our theoretical ﬁndings experimentally and identify directions for further improvement.
Related work
Analyzing which distributions afﬁne coupling ﬂows can approximate is an active area of research.[1] showed it is possible to approximate arbitrary invertible functions using single-active dimensionafﬁne couplings. From this, they conclude that such afﬁne coupling ﬂows weakly converge to anyprobability density.
[14] require only three afﬁne coupling blocks to show convergence to arbitrarydensities in Wasserstein distance.
Both universality results, however, require that the couplingsbecome ill-conditioned (i.e. the learnt functions become increasingly discontinuous as the errordecreases, whereas in practice one observes that functions remain smooth).
Also, they consider only aﬁnite subspace of the data space.
Even more importantly, the convergence criterion employed in theirproofs (weak convergence resp.
convergence under Wasserstein metric) is critical:
[14] remarked thatthose criteria do not imply convergence in the loss that is employed in practice, the Kullback-Leibler divergence (equivalent to maximum likelihood loss).
An arbitrarily small distance in any of the abovemetrics can even result in an inﬁnite KL divergence.
In contrast to previous work on afﬁne couplingﬂows, we work directly on the KL divergence.
We decompose the KL in two contributions and showthe ﬂow’s convergence for one of the parts.
Regarding when ill-conditioned ﬂows need to arise to ﬁt a distribution, [18] showed that wellconditioned afﬁne couplings can approximate log-concave padded distributions, again in terms of
Wasserstein distance.
Lipschitz ﬂows on the other hand cannot model arbitrary tail behavior, but thiscan be ﬁxed by adapting the latent distribution [19].
For other kinds of normalizing ﬂows, similar universality results exist: [20] for weak convergenceof sum-of-squares polynomial coupling ﬂows, extended by [21] to the stronger total variationconvergence, which also does not imply convergence in KL divergence in general; [22] for zeropadded afﬁne coupling ﬂows in weak convergence; [23, 24] for Neural ODEs in invertible functionapproximation and weak convergence.
Closely related to our work, [14, Theorem 2] shows that 48 linear afﬁne coupling blocks are sufﬁcientto represent any invertible linear function
Ax + b with det( A ) > 0 .
This also allows mapping any
Gaussian distribution N ( m, ⌃ ) to the standard normal N (0 , I ) .
We put this statement into context interms of the KL divergence: The loss is exactly composed of the divergence to the nearest Gaussianand of that Gaussian to the standard normal.
We then make strong statements about the convergenceof the latter, concluding that for typical ﬂows a smaller number of layers is required for accurateapproximation than predicted by [14].
Afﬁne coupling ﬂow fundamentals
The idea of normalizing ﬂows is to learn an invertible function f ✓ ( x ) that maps data from someunknown distribution p ( x ) given by samples to latent variables z
= f ✓ ( x ) so that z follow a simpledistribution, typically the standard normal.
f ✓ then yields a model for the data distribution via thechange of variables formula (e.g. [5]):
[Equation]
where J = r f ✓ ( x ) is the Jacobian of f ✓ ( x ) .
We can train a normalizing ﬂow via the maximumlikelihood loss, which is equivalent to minimizing the Kullback-Leibler divergence between thedistribution of the latent code q ( z ) , as given by z = f ✓ ( x ) when x ⇠ p ( x ) , and the standard normal:
[Equation]
The invertible architecture that makes up f ✓ has to (1) be computationally easy to invert, (2) be ableto represent complex transformations, and (3) have a tractable Jacobian determinant [9].
Buildingsuch an architecture is an active area of research, see e.g. [2].
In this work, we focus on a typicalvariant of afﬁne coupling ﬂows, based on the simple and popular RealNVP architecture [5].
It isa deep architecture that consists of several blocks, each containing a rotation layer and an afﬁnecoupling layer:
[Equation]
The afﬁne coupling f cpl splits an incoming vector in two parts along the coordinate axis:
The ﬁrstpart, which we call passive , is transformed by a global afﬁne transformation.
The second part, whichwe call active , is modiﬁed as a function of the passive dimensions:
[Equation]
Here, s : R D/ 2 ! R D/ 2 + and t : R D/ 2 ! R D/ 2 are arbitrary functions, that are typically representedby neural networks, and r 2 R D/ 2 + and u 2 R D/ 2 are vector-valued parameters.
denotes theelement-wise product.
As the passive dimensions are transformed by f cpl in a simple manner, theabove function is trivial to invert: Call x 1 = ( p 1 ; a 1 ) the output of the layer, then solving forx
= ( p 0 ; a 0 ) {+is trivial as p 
+}
= ( p 1  u ) ↵ r ( ↵ is element-wise division).
Use p 0 to evaluate 109 s ( p 0 ) , t ( p 0 ) and {+recover a 
+}
= ( a 1  t ( p 0 )) ↵ s ( p 0 ) .
If we were to concatenate several RealNVP layers, the entire network would never change the passivedimensions apart from an element-wise afﬁne transformation.
Here, the rotation layers f rot ( x ) =
Qx come into play.
They multiply an orthogonal matrix Q to the data, changing which subspaces arepassive resp. active.
This matrix is typically ﬁxed at random at initialization and then left unchangedduring training.
A rotation layer is trivial to invert by transposing the matrix: f  1 rot ( x ) =
Q T x .
Note that the afﬁne transformation of the passive dimensions is a modiﬁcation from the originalformulation of RealNVP in [5].
However, it adds very little computational overhead while simplifyingour theoretical analysis greatly.
Also, it is used in practice like in the popular INN framework FrEIA .
Afﬁne coupling layers as whitening transformation
The central mathematical question we answer in this work is the following: How can a deep afﬁnecoupling ﬂow whiten the data?
As the latent distribution is a standard normal, whitening is a necessarycondition for the ﬂow to converge.
This is a direct property of the loss:
Proposition 1 (Pythagorean Identity, Proof in Appendix B.1) .
Given data with distribution p ( x ) withmean m and covariance ⌃ .
Then, the Kullback-Leibler divergence to a standard normal distributiondecomposes as follows:
[Equation]
and the non-Standardness again decomposes:
[Equation]
This splits the transport from the data distribution to the latent standard normal into three parts: (1)
From the data to the nearest Gaussian distribution N ( m, ⌃ ) , measured by G .
(2) From that nearest
Gaussian to the corresponding uncorrelated Gaussian N ( m, Diag( ⌃ )) , measured C .
(3) From theuncorrelated Gaussian to standard normal.
We do not make explicit use of the fact that the non-Standardness can again be decomposed,but we show it nevertheless to relate our result to the literature: The Pythagorean identity
D KL ( p ( x ) ||N ( m, Diag( ⌃ )))
=
G ( p ) + C ( p ) has been shown before by [17, Section 2.3].
Boththeir and our result are speciﬁc applications of the general [25, Theorem 3.8] from informationgeometry.
Our proof is given in Appendix B.1.
Proposition 1 is visualized in Figure 1.
In an experiment, we ﬁt a set of afﬁne coupling ﬂows ofincreasing depths to the EMNIST digit dataset [26] using maximum likelihood loss and measure thecapability of each ﬂow in decreasing G and S .
The form of the non-Standardness S is given by the 138 well-known the KL-divergence between the involved normal distributions, see Appendix B.1; it isinvariant under rotations Q :
[Equation]
By writing S ( m, ⌃ ) , we emphasize the sole dependence on the ﬁrst two moments.
The non-Standardness S will be our central measure on how far the covariance and mean haveapproached the target standard normal in the latent space.
We give explicit loss guarantees for S for asingle RealNVP block in Theorems 1 and 2 and imply a linear convergence rate for a deep networkin Theorem 3.
In practice, G converges similar to S (see Figure 1). While our results in the presentpaper provide quantitative bounds for S , we merely guarantee that G does not increase from block toblock and leave a quantitative analysis for future work.
Deep Normalizing Flows are typically trained end-to-end, i.e. all parameters are randomly initializedand the entire stack of blocks is trained at the same time.
In this work, our ansatz is to considerhow much an isolated coupling block can reduce the non-Standardness S ( m, ⌃ ) .
Then, we combinethe effect of many isolated blocks, disregarding potential further improvements to S due to joint,cooperative learning of all blocks.
This greatly simpliﬁes the theoretical analysis of the network, butstill results in strong statements.
It is not a restriction on the model: Any function that is achieved inblock-wise training could also be the solution of end-to-end training.
Our ﬁrst result shows which mean m 1 and covariance ⌃ 1 a single coupling layer f cpl produces tominimize S ( m 1 , ⌃ 1 ) given data with mean m and covariance ⌃ , rotated by Q :
Proposition 2 (Proof in Appendix B.2) .
Given D -dimensional data with mean m and covariance ⌃and a rotation matrix Q .
Split the covariance of the rotated data into four blocks, corresponding tothe passive and active dimensions of the coupling layer:
[Equation]
Then, the moments m 1 , ⌃ 1 minimizing Equation (7) that can be reached by f cpl are: [Equation]
[Equation]
At this minimum of S , G does not increase.
The function M takes a matrix A and rescales the diagonal to 1 as follows.
It is a well-knownoperation in numerics called Diagonal scaling or Jacobi preconditioning:
[Equation]
Proposition 2 shows that the main change the coupling block can enact on the covariance is makingthe passive and active dimensions uncorrelated.
This is achieved by the Schur complement ⌃ 0 ,aa ⌃ 0 ,ap ⌃  1 0 ,pp ⌃ 0 ,pa .
It coincides with the covariance of a Gaussian that is conditioned on the passivesubspace.
Afterwards, the diagonal is rescaled to one, matching the standard deviations of alldimensions with the desired latent code.
The result also ensures that any improvement in S is notborrowed from the loss by increasing G .
In practice, we observe that both S and G can decrease by 169 the action of a single layer (see Figure 1).
The proof is based on a more general result how a single layer maximally reduces the Maximum Likelihood Loss for arbitrary data [13]. Here, we apply their result to the non-Standardness S . Detailscan be found in Appendix B.2.
Figure 2 shows an experiment in which a single layer was trained to bring the covariance of EMNISTdigits [26] as close to the identity as possible.
The experimental result coincides with the predictionby Proposition 2.
Due to the ﬁnite batch-size, a small difference between theory and experimentremains.
Explicit convergence rate
In Section 4, we showed how a single afﬁne coupling layer acts on the ﬁrst two moments of a givendata distribution to whiten it.
We now explicitly demonstrate how much progress this means in termsof the non-Standardness S ( m 1 , ⌃ 1 ) , averaged over rotations Q , (Theorems 1 and 2) and show theconsequences for multiple blocks (Theorem 3).
5.1 Single afﬁne coupling block guarantees
Proposition 2 allows the computation of the minimum non-Standardness after a single coupling blockgiven its rotation Q , by evaluating S ( m 1 , ⌃ 1 ) .
In fact, if we were to choose Q such that the datais rotated so that principal components lie on the axes (i.e. obtain Q using PCA), a single couplingblock sufﬁces to reduce the covariance to the identity: ⌃ 0 = Q ⌃ Q T would be a diagonal matrixand {+⌃ 
+}
= I .
This is not the case in practice, where this optimal orientation has zero probability: Qis chosen uniformly at random before training from all orthogonal matrices.
One could argue thatone should whiten the data before passing it to the afﬁne coupling ﬂow, reducing S to zero from thestart.
However, any change in the architecture could possibly alter the performance of the networkwith regard to reducing the non-Gaussianity G .
Also, our work shows that afﬁne coupling ﬂows arealready well-equipped to bring the non-Standardness to zero without such modiﬁcations.
To properlydescribe the achievable non-Standardness S , we formulate all guarantees as expectations over therotation Q , corresponding to the loss averaged over training runs.
We make two mild assumptions on our data that are part of usual data-preprocessing, when the meanis subtracted from the data and all data points are divided by the scalar p tr ⌃ /D (not to be confusedwith diagonal preconditioning, which acts dimension-wise).
Assumption 1.
The data p ( x ) is centered: E x ⇠ p ( x ) [ x ] = 0 .
Assumption 2.
The covariance is normalized: tr ⌃ = D .
The assumptions simplify the non-Standardness in Equation (7), which now only depends on thedeterminant of ⌃ :
[Equation]
(11) for arbitrary rotation Q .
We aim to compute the average non-Standardness after a single block
E Q 2 p ( Q ) [ S ( ⌃ 1 ( Q ))] .
For any Q , S ( ⌃ 1 ) is again given by the determinant of the covariance ⌃ 1 ( Q )as Assumptions 1 and 2 remain fulﬁlled: By Proposition 2 m 1 = 0 and the diagonal preconditioning
M ensures that the trace of ⌃ 1 is D .
We write det( ⌃ 1 ) via M a and M p , the diagonal matrices thatmake up the diagonal preconditioning in Equation (10), and use the Schur determinantal formulafor the determinant of block matrices: det( ⌃ 0 ,pp ) det( ⌃ 0 ,aa  ⌃ 0 ,ap ⌃  1 0 ,pp ⌃ 0 ,pa ) = det( ⌃ 0 )
=det( ⌃ ) [27].
We thus get det( ⌃ 1 ) = det( M p ⌃ 0 ,pp M p ) det
( M a ( ⌃ 0 ,aa  ⌃ 0 ,ap ⌃  1 0 ,pp ⌃ 0 ,pa ) M a ) =det( M 2 p ) det( M 2 a ) det( ⌃ ) .
Inserting this into Equation (11), we ﬁnd:
[Equation]
The inequality S ( ⌃ 1 )  S ( ⌃ 0 ) holds because ⌃ 1 = ⌃ 0 is an admissible solution of the coupling 211 layer optimization, but ⌃ 1 as given by Proposition 2 is a minimizer of S ( ⌃ 1 ) .
We average this quantity over training runs, i.e. over rotations Q :
[Equation]
The main difﬁculty lies in the computation of E Q ⇠ p ( Q ) [log det M 2 a ] .
Here, we contribute the twostrong statements Theorems 1 and 2 below.
5.1.1 Precise guarantee
The ﬁrst result relies on projected orbital measures as developed by [28].
This theory describes theeigenvalues of submatrices of matrices in a random basis.
We require such a result for integratingover p ( Q ) in E Q ⇠ p ( Q ) [log det M 2 a ] .
In contrast to typical choices of p ( Q ) , the theory to this dateonly covers data rotated by unitary matrices.
1 To comply with [28], we make the following twoadditional assumptions:
Assumption 3.
The distribution of rotations is the Haar measure over unitary matrices
U ( D ) .
Assumption 4.
The eigenvalues of the covariance matrix ⌃ are distinct:  i 6 =  j for i 6 = j .
One could think that the step from orthogonal to unitary rotations takes us far away from the scenariowe want to consider.
We later observe empirically that the difference between averaging over unitary and orthogonal matrices is negligible.
Technically, the data remains real at the optimum if weapply the inverse of the rotation layer after the transformation (see Appendix B.3).
We will write
E Q ⇠ U ( D ) [ · ] to denote expectations over unitary matrices.
Assumption 4 is typically satisﬁed when 228 working with real data that are in ‘general position’.
We are now ready to compute the average training performance of a single afﬁne coupling block:
Theorem 1 (Proof in Appendix B.3) .
Given D -dimensional data with covariance ⌃ with eigenvalues 1 , . . .
D .
Assume that Assumptions 1 to 4 hold.
Then, after a single afﬁne coupling block, theexpected non-Standardness is bounded from above:
[Equation]
[Equation]
Here,  6 = i := {  1 , . . .
,  i  1 ,  i +1 , . . .
,  D } and R, e K are given by: [Equation] N
[Equation]
a  b i and e K
[Equation] { b i } [Equation] Inequality (14) sharply bounds the expected non-Standardness that can be achieved by a single afﬁneblock.
The only approximation made is an inequality which comes close to equality as the dimension D increases due to the concentration of the corresponding probability distribution.
Figure 3 shows an experiment conﬁrming Theorem 1.
We start with covariance matrices usingparametrized eigenvalue spectra.
On each, we ﬁrst apply a single afﬁne coupling block with random Q and train the coupling that maximally reduces S (Proposition 2).
Then we iteratively append32 additional blocks in the same manner, building a ﬂow of that depth.
We average the resultingempirical ratio S ( ⌃ 1 ) / S ( ⌃ ) over several orthogonal orientations Q of the rotation layer for eachinput covariance matrix.
Then, we compare this to (i) experimentally averaging over unitary rotationsand (ii) to the prediction by Theorem 1 and conﬁrm that it is a valid and close upper bound.
Detailsfor replication and more examples can be found in Appendix A.3.
The proof explicitly integrates E [ M 2 a ] using [28] (see Appendix B.3).
Numerically evaluating
Equation (14) can be hard even for small D as the summands scale as O (exp( D )) , but the overallsum scales as O ( D ) .
High values cancel due to R alternating in sign, and one requires arbitraryprecision ﬂoating point software to evaluate Equation (14).
5.1.2 Interpretable guarantee
The guarantee in Theorem 1 yields useful predictions, but it does not lend itself to further analysis:
How does the bound behave over several coupling blocks?
What is the behavior for varying dimension
D ?
Also, Assumption 3 restricts formal reasoning as we are interested in averaging over orthogonaland not unitary rotations.
Our second single-block guarantee depends only on simple metrics of thecovariance.
Moreover, we drop Assumptions 3 and 4, averaging over orthogonal, not unitary, Q :
Theorem 2 (Proof in Appendix B.4) .
Given D -dimensional data fulﬁlling Assumptions 1 and 2 withcovariance ⌃ with eigenvalues  1 , . . .
D .
Then, after a single afﬁne coupling block, the expectedloss can be bounded from above:
[Equation]
Here, g is the geometric mean of the eigenvalues: g =
Q D i =1  1 /Di =
exp(  2 S ( ⌃ ) /D ) <
1 whichis a bijection of S ( ⌃ ) .
These two new bounds on the average achievable non-Standardness S after a single block are alsodepicted in Figure 3.
They make useful predictions, but are less precise than Theorem 1.
The secondbound will be especially useful in what follows because it only depends on the non-Standardnessbefore the block S ( ⌃ ) .
The full proof is given in Appendix B.4.
It relies on the integration of monomials of entries of randomorthogonal matrices as described by [30] and the arithmetic mean-geometric mean inequality by [31].
The ﬁrst bound suggests an important property of the non-Standardness convergence of an afﬁnecoupling ﬂow in terms of dimension: The performance only marginally depends on the dimension.
Tosee this, divide Equation (16a) by D to obtain a statement about the non-Standardness per dimension
S /D .
Then take several data sets with different dimension but same spectrum characteristics(i.e.
same geometric mean, variance and maximum of covariance eigenvalues).
The guarantee is thenapproximately constant in D (it varies slightly with D 2 / ( D 2 + D  2) , which is always close to 1 ).
5.2 Deep network guarantee
The previous Section 5.1 was concerned with determining how much a single RealNVP block cantypically contribute towards reducing the S to zero.
Now, we extend this result to compute theexpected non-Standardness after a deep afﬁne coupling network as an explicit function of the numberof blocks.
We again treat the rotation layer of each block as a random variable, as it is randomlydetermined before training.
We ﬁnd that the convergence rate of the covariance to the identity is (at least) linear :
Theorem 3 (Proof in Appendix B.5) .
Given D -dimensional data fulﬁlling Assumptions 1 and 2 withcovariance ⌃ .
Then, after L afﬁne coupling blocks, the expected loss is smaller than:
[Equation]
where the convergence rate depends on the non-Standardness before training:
[Equation]
The non-Standardness decreases at least exponentially fast in the number of blocks.
The convergencerate that holds for a deep network is computed using the non-Standardness of the input data S ( ⌃ ) .
This rate comes from Equation (16b).
The proof uses that  ( S ) improves from block to block as
S decreases (see Appendix B.5).
Again, g ( S ) =
exp(  2 S /D ) < 1 is the geometric mean ofeigenvalues of ⌃ , which increases from block to block.
Figure 4 shows the convergence of the non-Standardness to zero in an experiment.
We build a toydataset of various covariances where we aim to capture a plethora of possible cases (see Appendix A.4).
We apply a single afﬁne coupling block with random Q and the coupling that maximally reduces Svia Proposition 2.
We iterate adding such blocks 32 times, building a ﬂow of that depth.
The resultingconvergence of S as a function of depth is averaged over 32 runs with different rotations.
Themeasured curve conﬁrms Theorem 3.
We ﬁnd that the rate  in Equation (18) is correct, but severalexperiments show even faster convergence in practice.
Indeed, the experiments suggest that dividingall upper bounds for E [ S ( ⌃ 1 )] in Theorems 1 and 2 by S ( ⌃ ) also bounds the non-Standardness ratiofor subsequent blocks.
Formally, we conjecture that E [ S ( ⌃ L )] / S ( ⌃ )  ( B/ S ( ⌃ ))
L where B is therhs.
of Equations (14) and (16a) (Theorem 3 shows exactly this for Equation (16b)).
We leave a proofor falsiﬁcation of this conjecture open to future work.
The experiment also suggests that all bounds agree after a few blocks, leaving a small gap to theexperiment.
We can explicitly compute this limit value of  ( S ) by taking S ! 0 :
[Equation]
[Equation] The two experimental observations together with this limit value suggest the heuristic that a singleadditional coupling block typically reduces the non-Standardness S by a factor of approximately50% if previous blocks are left unchanged, and possibly faster if cooperations between blocks aretaken into account.
Conclusion
To the best of our knowledge, this is the ﬁrst work on afﬁne coupling ﬂows that provides a quantitativeconvergence analysis in terms of the KL divergence.
Speciﬁcally, a minimal convergence rate isestablished at which afﬁne ﬂows whiten the covariance of the input data under this strong measureof discrepancy of probability distributions.
Splitting the loss into the non-Gaussianity G and thenon-Standardness S , we show that this whitening is a necessary condition for the ﬂow to converge andgive explicit guarantees.
Our derivations suggest the rule of thumb that S can typically be reduced byabout 50% per coupling block.
Our central idea was to separate out the contribution a single isolated block can make to reduce theloss, arguing that end-to-end training can only outperform the concatenation of isolated blocks.
Having separated the tasks an afﬁne coupling ﬂow has to solve, and having explained how thenon-Standardness S can be reduced to zero, we hope that explaining the convergence under nonGaussianity G comes within reach.
In practice, it converges similarly fast as S (see Figure 1).
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope
[Yes]
(b) Did you describe the limitations of your work
[Yes] see Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Generative
modeling, which this paper aims to improve, can be used in harmful ways to generate
Deepfakes for disinformation.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Full proofs are inAppendix B.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Code will be
made available upon publication.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix A.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Except for Figure 1 which shows a single training run perblock, but a different seed is used for each initialization.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[-= Σ − 
-]
(b) Did you describe any potential participant risks, with links to Institutional Review
[-=
-]
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]