Copula-Based Normalizing Flows
Abstract
Normalizing ﬂows, which learn a distribution by transforming the data to samples from a Gaussian base distribution, have proven powerful density approximators.
But their expressive power is limited by this choice of the base distribution.
We, therefore, propose to generalize the base distribution to a more elaborate copula distribution to capture the properties of the target distribution more accurately.
In a ﬁrst empirical analysis, we demonstrate that this replacement can dramatically improve the vanilla normalizing ﬂows in terms of ﬂexibility, stability, and effectivity for heavy-tailed data.
Our results suggest that the improvements are related to an increased local
Lipschitz-stability of the learned ﬂow.
1. Introduction
Normalizing Flows (NFs) are a recently developed class of density estimators, which aim to transform the distribution of interest P x to some tractable base distribution P z .
Using the change of variables formula, this allows for exact likelihood computation, which is in contrast to other deep generative models such as Variational Autoencoders (Kingma & Welling, 2014) or Generative Adversarial Networks (Goodfellow et al., 2014).
Impressive estimation results, especially in the ﬁeld of natural image generation, have lead to great popularity of these deep generative models.
Motivated by this success, much effort has been put into the development of new parametric classes of NFs to make them even more performant (Dinh et al., 2015; 2016;
Chen et al., 2019; Papamakarios et al., 2017; Grathwohl et al., 2019).
However, our theoretical understanding has not developed at the same speed, which, so we claim, slows down further progress in the development of powerful NF architectures.
Fortunately, very recent works have addressed theoretical limitations of these methods: One important limitation is the expressive power of NFs.
Because they are based on the change of variables formula, the learned transformations are required to be diffeomorphisms.
As a consequence, a NF with bounded Lipschitz constant is unable to map one distribution P x to a lighter-tailed distribution P z
(Wiese et al., 2019; Jaini et al., 2020).
Therefore, since vanilla NFs are implemented using an isotropic Gaussian base distribution, they are unable to learn heavy-tailed distributions, which are known to appear in natural image data (Zhu et al., 2014;
Horn & Perona, 2017; Zhang et al., 2017).
This is in conﬂict with observations recently made by Behrmann et al.
Even though NFs are typically designed to obey invertibility, this property is often violated in practice.
This is due to numerical inaccuracies, which are promoted by a large
Bi-Lipschitz constant.
Bounding the Bi-Lipschitz constant, however, conﬂicts with the previously mentioned theoretical requirements needed to avoid a limited expressiveness of the NF.
These ﬁndings emphasize the high importance of choosing an appropriate base distribution for NFs.
We therefore propose to generalize the isotropic Gaussian to a much broader class of distributions—the class of copula distributions.
Copulae are a well-known concept in statistical theory and are being used to model complex distributions in ﬁnance, actuarial science, and extreme-value theory (Genest et al., 2009; Joe, 2014;
Hofert et al., 2018).
Broadly speaking, a copula is some function that couples marginal distributions to a multivariate joint distribution.
Hence, copulae allow for ﬂexible multivariate modeling with marginals stemming from a huge range of suitable classes.
This allows for example to formulate NF base distributions that combine heavy-tailed marginals—as proposed by Wiese et al.
; Jaini et al.
; Alexanderson & Henter (2020)—with light-tailed marginals.
This paper presents a novel framework for choosing the base distribution of NFs building on the well-studied theory of copulae.
A ﬁrst empirical investigation demonstrates the beneﬁts brought by this approach.
Our experimental analysis on toy data reveals that using even the most simple copula model—the Independence Copula—we are able to outperform the vanilla NF approach, which uses an isotropic Gaussian base distribution.
The resulting NF converges faster, is more robust, and achieves an overall better test performance.
In addition, we show that the learned transformation has a better-behaved functional form in the sense of a more stable local Lipschitz continuity.
2. Background
In this section, we quickly review some background knowledge about NFs (Section 2.1), followed by an introduction to copula theory (Section 2.2).
2.1. Normalizing Flows
Density estimation via NFs revolve around learning a diffeomorphic transformation T θ that maps some unknown target distribution P x to a known and tractable base distribution
P z .
At the cornerstone of NFs is the change of variables formula
[Equation]
which relates the evaluation of the estimated density p θ of x ∼ P x to the evaluation of the base density p z , of T θ ( x ) , and of det J T θ ( x ) .
By composing simple diffeomorphic building blocks T θ := T θ,l ◦ · · · ◦ T θ, 1 , we are able to obtain expressive transformations, while presuming diffeomorphy and computational tractablity of the building blocks.
Due to the tractable PDF in (1), we are able to train the model via maximum likelihood estimation (MLE)
[Equation]
where p data is the PDF of the empirical distribution of x .
A comprehensive overview of NFs, including the exact parameterizations of certain ﬂow models T θ , computational aspects, and more, can be found in Kobyzev et al.
and
Papamakarios et al.
2.2. Copulae
A completely different approach of density estimation, which has mostly been left unrelated to NFs, is the idea of copulae.
Deﬁnition 2.1 (Copula) .
A copula is a multivariate distribution with CDF C : [0 , 1] D
→ [0 , 1] that has standard uniform marginals, i.e. the marginals C j of C satisfy
C j ∼ U [0 , 1] .
The fundamental idea behind copula theory is that we can associate every distribution with a uniquely deﬁned copula C .
Vice versa, given D marginal distributions, each copula C deﬁnes a multivariate distribution with the given marginals.
Formally, this is known as Sklar’s Theorem (Sklar, 1959; 1996).
Theorem 2.2 (Sklar’s Theorem) .
Taken from Hofert et al.
For any D -dimensional CDF F z with marginal CDFs
F z 1 , . .
, F z D , there exists a copula C such that
[Equation]
for all z ∈ R D .
The copula is uniquely deﬁned on
U :=
(cid:81)
Dj =
Im( F z j ) , where Im( F z j ) is the image of
F z j .
For all u ∈ U it is given by
[Equation]
where F ← z j are the right-inverses of F z j .
Conversely, given any D -dimensional copula C and marginal CDFs F z 1 , . . .
F z D , a function F z as deﬁned in (3) is a D -dimensional CDF with marginals
F z 1 , . .
, F z D .
Part 1 of Sklar’s Theorem ﬁnds much application in statistical dependency analysis (Joe, 2014).
In contrast to classical dependency measures, such as Pearson correlation, copulae are a more ﬂexible tool that allow the decoupling of the marginals and the dependency structure.
Part 2 of
Sklar’s Theorem is of relevance for statistical modeling, and more precisely, to deﬁne multivariate distributions.
Given marginal distributions, which are typically much easier to estimate than the full joint distribution, and a copula C we can “couple” the marginals and the dependency structure to a multivariate joint distribution.
This perspective ﬁnds various applications in the context of ﬁnance and related disciplines that need to take heavy tails and tail dependencies into account, see Genest et al.
(2009) for an overview.
In Section A of the Appendix we give some illustrative examples and further details about properties of copula distributions.
By differentiating (3), we obtain the PDF of z as
[Equation]
where c, p z 1 , . .
. , p z D are the PDFs of C, F z 1 , . . , F z D , respectively.
3. NFs With Copula-Base Distributions
In this paper, we propose to employ copulae to model a ﬂexible, yet appropriate base distribution with the goal of gaining a NF that is able to solve the limitations of NFs discussed in Section 1.
We expect to gain powerful and robust PDF approximators by combining different marginals and properties of theoretical sound copulae (see for instance Chapter 8 in Joe (2014)) with NFs, which allow for the estimation of complex densities.
3.1. A General Framework
We propose to replace the isotropic Gaussian base distribution in the vanilla NF framework by a more ﬂexible cop- ula distribution.
Importantly, we want to learn a base distribution that is able to represent the tail behavior of the distribution of x .
For training a NF with a copula base distribution we build on the fact that we can write the PDF of the latent variables as written in (5).
This requires two estimation steps: First, we need to estimate the marginal distributions F z 1 , . .
, F z D , which can further be used to calculate the marginal densities p z 1 , . . .
, p z D .
Secondly, we need to estimate the copula density c .
A popular approach for estimating the density in (5) is to employ the method of inference functions for margins (IFM)
(Joe & Xu, 1996), which sequentially estimates the marginals using MLE ﬁrst, and then employs these marginals to estimate the copula using MLE.
It is important to note that in contrast to standard applications of copula theory, we do not aim at estimating the full data generating distribution based on (5).
Instead, following the investigations by Jaini et al.
(2020), our goal is to capture the tail-behavior of x .
Hence, we propose to learn surrogate marginals z 1 , . . .
, z D that are able to represent the tailedness of the marginals x 1 , . . .
, x D .
By combining these marginals with some simple copula structure, such as the Gaussian Copula or the Independence Copula (see (6) below), we are able to create a joint distribution that represents the marginal tail behavior of x .
The proposed adjustment can be applied to any existing NF architecture as long as (5) remains tractable.
However, as the main goal of the base distribution is not to fully estimate the target but to represent the tail behavior of x , we can restrict ourselves to tractable parametric marginal distributions and copulae.
3.2. Experimental Analysis
In this section, we investigate the beneﬁts of the proposed approach by analyzing a toy problem.
In the following experiments, we employ the framework proposed in Section 3.1 using the most simple copula: the Independence
Copula, i.e. we consider a base distribution z with PDF p z (
[Equation]
supp
[Equation]
[Equation]
Note that by plugging Gaussian marginals in (6), we would obtain the vanilla NF.
We consider a training set generated from a 2-dimensional heavy-tailed distribution, 1 which has standardized t-distributed marginals x 1 , x 2 ∼ t 2 (0 , 1) with 2 degrees of freedom.
The corresponding copula is a Gumbel Copula with parameter ρ = 2 . 5 , i.e.
C
[Equation]
Appendix
[Figure]
As a proof of concept we compare the estimation of this heavy-tailed distribution using a NF 2 with an isotropic Gaussian base distribution, and with 3 different heavy-tailed base distributions constructed via (6).
We consider the following heavy-tailed marginals:
Laplace (0 , 4) and t 5 (0 , 2) .
We call this case heavierTails because one marginal is heavy-tailed; 2.
t 5 (0 , 1) and t 5 (0 , 1) .
We call this case correctFamily since both marginals stem from the same parametric class as the exact marginals; 3.
t 2 (0 , 1) and t 2 (0 , 1) .
We call this case exactMarginals .
Samples from the target distribution and from the different base distributions are visualized in Figure 6 and 7 in the Appendix.
Training and test loss In Figure 1, we plot the average training and test performance over 100 trails.
It is apparent that training using a base distribution with the correct type of tail behavior is beneﬁcial.
First of all, we observe a signiﬁcant gap between the test performance of the vanilla NF and the NFs with a heavy-tailed base distribution.
Notice that in Figure 1 we excluded all runs with a ﬁnal test loss of above 25 , which happened in 17 of the normal runs and not once in the other cases.
Furthermore, we clearly observe a much faster convergence and a more stable training procedure.
The ﬂuctuations and instabilities in the vanilla
NF are due to tail-samples that have a massive effect on the likelihood in (2), which can be reduced by choosing base distributions with slower decaying tails (Alexanderson &
Henter, 2020).
Learning the tails To illustrate the ability to model the tails, we compared the estimated empirical quantile func-
[Figure]
tions.
We did so for both marginal distributions (Figure 2) but also for the distribution of (cid:107) x (cid:107) 2 (Figure 9 in the Appendix).
In line with the ﬁndings by Jaini et al.
, we notice that the vanilla NF is not capable of modeling the quantiles of the target distribution.
More precisely, we observe that the corresponding quantile function is steeper around its center and has shorter tails.
This means that the distribution learned by the NF does not account for the heavy tails by directly modeling them, but instead covers samples from the tails of the data distribution by being more widespread.
In contrast, the base distributions that took the tailedness of x into account, could achieve a much better ﬁt to the quantiles, see Figure 8 in the Appendix for further results.
Invertibility and numerical stability As investigated by
Behrmann et al.
(2021), the Bi-Lipschitz constant plays a fundamental role in the practical invertibility and numerical stability of NFs.
To understand the learned transformation
T in terms of its Lipschitz continuity, we propose to study the Lipschitz surface of T .
Note that if T is differentiable and L -Lipschitz, we can follow the derivation by Behrmann et al.
(2021) (in equation (5) and (6) therein) to approximate
L =
[Equation]
where ε is some small constant.
This motivates to consider an estimate 3 of sup (cid:107)
[Equation] 2 =1 (cid:107) T ( x [Equation] − T ( x + εv ) (cid:107) 2 /ε for x ∈ supp( x ) as a local surrogate for the Lipschitzcontinuity of T .
Plotting these quantities for both, T and
T − 1 , and x ∈ [ − 10 , 10] 2 we obtain the Lipschitz surfaces, which are depicted in Figure 3.
We notice that the vanilla
NF has many ﬂuctuations in the local Lipschitz-continuity, while the proposed copula method leads to a well-behaved transformation.
The inverse transformation T − 1 θ in the vanilla NF has exploding local Lipschitz constants, while— again—the proposed method results in a stable inverse transformation.
[Figure]
4. Discussion
In this work, we paved the way toward a general extension of NF architectures using copulae.
Synthetic experiments revealed that the modeling performance of NFs can be improved substantially by replacing the vanilla Gaussian base distribution by a base distribution that reﬂects basic properties of the data distribution more accurately.
Of course, we have just scraped the surface of the underlying potential of the proposed approach: While we concentrate on the tail behavior of the marginals in this work, the general idea can potentially also be applied to incorporate other types of inductive bias, such as multimodality by choosing multimodal marginals, or symmetries and tail dependencies by selecting appropriate marginals and copulae.
Our experiments suggest that it is sufﬁcient to have only a broad estimate of the marginals.
As mentioned in Section 3.1, one could also employ IFM to learn these before training the NF.
However, the question of what is the best technique for choosing or estimating an appropriate marginal distributions and copula still requires further investigation.
Nevertheless, we think that this ﬂexibility brings additional improvement over the methods proposed by Jaini et al.
and Alexanderson & Henter (2020).
Of course, we have yet only gained preliminary results with our empirical study, which we plan to verify on real-world data and for different models in the future.
We believe that our analysis of the base functions can help to popularize NFs in a wide spectrum of domains.
One such application might be ﬁnancial risk analysis, where it is essential to model tail dependencies. Acknowledgement
This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under
Germany’s Excellence Strategy – EXC- 2092 CASA – 390781972.
We also thank the anonymous reviewers for their careful reading and their useful suggestions, which will help us extending this work to a full paper.
A. Examples of Copula Distributions
There is a wealth of different copula distributions, ranging from parametric distributions to semi-parametric and completely non-parametric copula models.
In this Section, we give some insights into the construction of one class of copula distributions and illustrate some basic properties.
Example A.1 (Construction using Sklar’s Theorem) .
There is a generic way to construct a copula according to Deﬁnition 2.1.
Consider any multivariate continuous and invertbile CDF Φ with marginal CDFs Φ 1 , . .
Φ D .
Then, following (4) in Sklar’s Theorem 2.2, we know that
[Equation]
deﬁnes a valid copula.
Setting Φ to be the multivariate
Gaussian distribution with correlation matrix R , we obtain the Gaussian Copula .
If R is the identity matrix we obtain the Independence Copula , which is simply the product of independent uniform distributions.
Both copulae are visualized in Figure 4.
Following the construction in (7) and replacing the CDF, we can obtain other copulae,such as the t-Copula.
Figure 5 visualizes one example for a distribution based on the Independence Copula and based on the
Gaussian Copula.
Example A.2 (Copulae that induce Tail-Dependency) .
A crucial property of models in ﬁnancial risk analysis is tail dependency .
Roughly said, a tail dependency casts marginals to be dependent in a tail event.
To give an example for a tail dependency, consider two essentially independent marginal distributions, such as x 1 = { debt status of your bank } and x 2 = { trust for your bank } .
Usually, our trust for the bank is not essentially determined by the amount of debts it has.
However, in a marginal tail event { x = bankrupt } , of course, our trust drops rapidly.
Mathematically, the upper tail dependency, i.e. the dependency in upper-tail events, for a random variable ( x 1 , x 2 ) is given by
[Equation]
Similarly, we can deﬁne the lower tail dependency λ L .
One such copula that accounts for upper tail dependency is the
Gumbel Copula , which is given by
[Equation]
8) for ρ ≥ 1 .
Figure 4 shows a visualization of the Gumbel copula.
One can show that λ U = 2 − 2 1 /θ (Joe, 2014).
Figure 5 visualizes the Gumbel copula distribution with
Gaussian and Gamma marginals.
We can observe a decent dependency—indicated by the peak pointing to the upper right—in the upper-tail events, which is mathematically described by λ U .
One could think of many more different properties that might be included using our copula approach, such as multimodality and symmetry.
However, it is still to be researched
[Figure]
[Figure]
[Figure]
[Figure]
whether properties, such as the tail-dependency, are being preserved by the NF.
Nonetheless, we think that ﬁxing speciﬁc known properties in the base distribution facilitates training in acting as a type of regularization towards these given properties.
B. Supplementary Experiments
In the following, we give some further empirical results that underpin our ﬁndings from the main paper.
Figure 6 shows the investigated base distributions and Figure 7 visualizes the target distribution.
Figure 8 and Figure 9 supplement the ﬁndings about the learned quantiles: Even the other heavy-tailed distributions— the cases heavierTails and correctFamily —are able to successfully approximate the quantiles.
This observation suggests that it is sufﬁcient to use a broad surrogate of the base distribution that captures the true tail behavior.
To address the invertibility and numerical stability of the learned transformation, we investigate the local Lipschitz constant as derived in Section 3.2 in the main text.
Fig-
[Figure]
[Figure]
[Figure]
ure 10 shows the supplementary Lipschitz surfaces of the cases heavierTails and correctFamily , which indicate a more regular transformation than the normal case.
Still, in the case correctFamily we ﬁnd large local Lipschitz constants in T − 1 θ , which, however, are structured in accordance to the base distribution.
In the vanilla NF this irregularity is less structured and, since these irregularities occur in highprobability areas of the base distribution, more relevant.
These results support our claim that choosing appropriately tailed base distribution can help in learning a numerically robust transformation.
C. Computational Details
In all of our experiments, we employed Masked Autoregressive Flows (Papamakarios et al., 2017) with 3 layers.
Each layer contains a reverse permutation, followed by an autoregressive transformation with 4 hidden features.
The code is based on the nﬂows package (Durkan et al., 2020).
We trained the NFs on a sample of size 10 000 .
Optimization was carried out using the Adam optimizer with the PyTorch default settings and a batch size of 128 .
Test losses are evaluated based on 10 000 test samples.
The reported training and test losses (Figure 1) have been averaged over 100 runs and the depicted conﬁdence intervals correspond to a conﬁdence of 95% .
To estimate the Lipschitz surface we rely on an estimation of sup
[Equation]
for some small constant ε .
We chose ε := 10 − 3 .
Further, we approximate (9) by
[Equation]
where v (1) , . . .
v (100) are i.i.d.
samples from N (0 , I ) , see
Behrmann et al.
(2021) for further details.
All code is provided with the submission.