S PREAD DIVERGENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
For distributions p and q with different supports, the divergence D ( p || q ) may not exist.
We deﬁne a spread divergence ˜ D ( p || q ) on modiﬁed p and q and describe sufﬁcient conditions for the existence of such a divergence.
We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread.
We also give examples of using a spread divergence to train and improve implicit generative models, including linear models (Independent
Components Analysis) and non-linear models (Deep Generative Networks).
I NTRODUCTION
A divergence D ( p || q ) (see, for example Dragomir (2005)) is a measure of the difference between two distributions p and q with the property
[Equation]
We are interested in situations in which the supports of the two distributions are different, supp ( p ) (cid:54)
=
supp ( q ) .
An important class is the f -divergence, deﬁned as
[Equation]
where f ( x ) is a convex function with f (1) = 0 .
A special case of an f -divergence is the well-known
Kullback-Leibler divergence KL ( p || q ) =
E p ( x ) (cid:104) log p ( x ) q ( x ) (cid:105) .
By setting p ( x ) to the empirical data distribution, maximum likelihood training of a model q ( x ) corresponds to minimising KL ( p || q ) .
However, this divergence may not be deﬁned since the ratio p ( x ) /q
( x ) can cause a division by zero.
This is a challenge since popular implicit generative models (Mohamed & Lakshminarayanan (2016)) of the form q ( x ) = (cid:82) δ
( x − g θ ( z )) p ( z ) dz only have limited support.
In this case, maximum likelihood to learn the model parameters θ is not available and alternative approaches to measure the difference between distributions such as Maximum Mean Discrepancy (Gretton et al.
(2012)) or
Wasserstein distance (Peyré et al.
(2019)) are required.
S PREAD D IVERGENCE
From q ( x ) and p ( x ) we deﬁne new distributions ˜ q ( y ) and ˜ p
( y ) that have the same support 1 .
Using the notation (cid:82) x to denote integration (cid:82) ( · ) dx for continuous x , and (cid:80) x ∈X for discrete x with domain
X , we deﬁne a random variable y with the same domain as x and distributions
[Equation]
where p ( y | x ) is ‘noise’ designed to ‘spread’ the mass of p and q such that ˜ p ( y ) and ˜ q ( y ) have the same support.
For example, if we use a Gaussian p ( y | x ) = N (cid:0) y x, σ 2 (cid:1) , then ˜ p and ˜ q both have support R .
We also impose an additional requirement on the noise p ( y | x ) , namely that D (˜ p || ˜ q ) = 0 ⇔ p = q .
As we show in section(2.1) this is guaranteed for certain ‘noise’ distributions.
Given these requirements, we can deﬁne the Spread Divergence ˜ D ( p || q ) ≡ D (˜ p || ˜ q ) .
This satisﬁes the divergence requirement ˜ D ( p || q ) ≥ 0 and ˜ D ( p || q ) = 0 ⇔ p = q .
For example, given two delta distributions p 0 ( x ) =
δ ( x − µ 0 ) , p 1 ( x ) = δ ( x − µ 1 ) , the KL divergence (or f -divergence) between them is not deﬁned.
However, the spread KL divergence (or f -divergence) is deﬁned.
Assume a Gaussian noise distribution p ( y | x ) =
N (cid:0) y x, σ 2 (cid:1) where σ 2 = 0 . 5 , the "spreaded" delta distributions have the form: ˜ p 0 ( y ) =
(cid:82) x δ ( x − µ 0 ) N (cid:0) y x, σ 2 (cid:1) =
N (cid:0) y µ 0 , σ 2 (cid:1) , ˜ p 1 ( y ) = (cid:82) x δ ( x − µ 1 ) N (cid:0) y x, σ 2 (cid:1) =
N (cid:0) y µ 1 , σ 2 (cid:1) .
Therefore, the spread KL divergence (with Gaussian noise) between two delta distributions is equivalent to the KL divergence between two Gaussian distributions with the same variance, which has closed form (see appendix(D) for a derivation):
[Equation]
It’s worth noticing that in the case of two delta distributions, the spread KL divergence is equal to the squared 2-Wasserstein distance (see Peyré et al.
; Gelbrich (1990)).
2. N OISE R EQUIREMENTS FOR A S PREAD D IVERGENCE
Our main interest is in using noise to deﬁne a new divergence in situations in which the original divergence D ( p || q ) is itself not deﬁned.
For discrete variables x ∈ { 1 , . . . , n } , y ∈ { 1 , . . . , n } , the noise P ij = p
( y = i | x = j ) must be a distribution (cid:80)
i P ij = 1 , P ij ≥ 0 and
[Equation]
which is equivalent to the requirement that the matrix P is invertible.
There is an additional requirement that the spread divergence exists.
In the case of f -divergences, the spread divergence exists provided that ˜ p and ˜ q have the same support.
This is guaranteed if
[Equation]
which is satisﬁed if P ij > 0 .
In general, therefore, there is a space of noise distributions p ( y | x ) that deﬁne a valid spread divergence.
For example, the ‘antifreeze’ method of Furmston & Barber (2009) is a special form of spread noise to deﬁne a valid KL divergence (see also Barber (2012)).
For continuous variables, in order for ˜ D
( p || q ) =
⇒ p =
q , the noise p ( y | x ) , with dim( Y ) =
dim( X ) must be a probability density and satisfy
[Equation]
In the following section we discuss the special case of stationary noise for continuous systems.
S TATIONARY S PREAD D IVERGENCES
Consider stationary noise p ( y | x )
=
K ( y − x ) where K ( x ) is a probability density function with
K ( x ) > 0 , x ∈ R .
In this case ˜ p and ˜ q are deﬁned as a convolution ˜
[Equation]
Since K> 0 , ˜ p and ˜ q are guaranteed to have the same support R .
A sufﬁcient condition for the existence of the Fourier Transform F { f } of a function f ( x ) for real x is that f is absolutely integrable.
Since all distributions p ( x ) are absolutely integrable, both F { p } and F { q } are guaranteed to exist.
Assuming F { K } exists, we can then use the convolution theorem to write
[Equation]
Let F{ K } (cid:54) = 0 or F{ K } = 0 on at most a countable set.
Then
F{ K }F{ p } = F{ K }F{ q } ⇒ F{ p } =
F{ q } .
The proof is given in appendix(A).
Using this we can write
[Equation] [Equation]
[Equation]
where we used the invertibility of the Fourier transform.
Hence, for stationary noise p ( y | x ) =
K ( y − x ) , we can deﬁne a valid spread divergence provided (i) K
( x ) is a probability density function and (ii) F{ K } (cid:54) = 0 or F{ K } =
on at most a countable set.
Interestingly, the sufﬁcient conditions for deﬁning a valid spread divergence such that D (˜ p ||
˜ q )
= 0 ⇔ p =
q are analogous to the characteristic condition on kernels such that the Maximum Mean Discrepancy MMD ( p, q ) = 0 ⇔ p =
q , see
Sriperumbudur et al.
; Gretton et al.
As an example of such a noise process, consider Gaussian noise,
[Equation]
Similarly, for Laplace noise
[Equation]
Since in both cases K> 0 and F { K } > 0 , Gaussian and Laplace noise can be used to deﬁne a valid spread divergence.
M AXIMISING D ISCRIMINATORY P OWER
From the data processing inequality (see appendix(B)), adding spread noise will always decrease the f -divergence D f
(˜ p ( y ) ||
˜ q ( y )) ≤ D f
( p ( x ) || q ( x )) .
Intuitively, spreading out distributions makes them more similar.
If we are to use a spread divergence to train a model using maximum likelihood (see section section(5)), there is the danger that adding too much noise may make the spreaded empirical distribution and spreaded model distribution so similar that it becomes difﬁcult to numerically distinguish them, impeding training.
It is useful therefore to deﬁne spread noise that maximally discerns the difference between the two distributions max ψ D (˜ p
( y ) ||
˜ q ( x )) for spread noise p ψ ( y | x ) parameterised by ψ .
In general we need to constrain the spread noise to ensure that the divergence remains bounded.
We discuss below two complementary approaches to adjust p ( y | x ) during training.
The ﬁrst approach adjusts the dimension-wise correlations (this corresponds to adjusting the covariance structure for
Gaussian p ( y | x ) ) and the second forms a mean transformation.
In principle, both approaches can be combined and easily generalized to other noise distributions, such as Laplace noise.
4. L EARNING C OVARIANCE STRUCTURE
Learning the covariance adjusts the shape of noise centered around the original model manifold.
When we maximize the divergence between two spreaded distributions max ψ D (˜ p
( y ) ||
˜ q ( x )) , the learned noise will discourage overlap between the two distributions.
Hence, if the data p and model q lie on the same manifold, the noise will be orthogonal to the manifold.
In learning the Gaussian spread distribution p ( y | x ) =
N ( y x, Σ) , the number of parameters in the covariance matrix Σ scales quadratically with the data dimension D .
We thus deﬁne Σ
= σ 2 I + LL T where σ 2 > 0 is ﬁxed (to ensure bounded spread divergence) and L is a learnable D × R matrix with R (cid:28) D .
Calculating the log likelihood and sampling can then be performed efﬁciently using standard Woodberry identities, see appendix(J).
4. L EARNING THE MEAN TRANSFORM
Consider p ( y | x )
=
K ( y − f ( x )) for injective 2 f and stationary K .
Then, we deﬁne
[Equation]
Note that this is a valid spread divergence since, using change of variables,
[Equation]
where J is the absolute Jacobian of f .
Hence, D (˜ p
y ||
˜ q y )
=
⇔ p z = q z ⇔ p x = q x .
Each injective f φ gives a different noise p ( y | x ) , we can thus search for the best noise implicitly by learning f φ .
In our experiments we use the invertible residual network Behrmann et al.
f ψ : R D → R D with f ψ = ( f 1 ψ ◦ . . .
◦
f Tψ ) denotes a ResNet with blocks f tψ = I ( · ) + g ψ t ( · ) .
Then f ψ is invertible if the Lipschitz-constants Lip ( g ψ t ) <
1 for all t ∈ { 1 , . . .
, T } .
Note that when using the spread divergence for training (see section(5.2.2))
we only need samples from ˜ p
( y ) which can be obtained from equation 16 by ﬁrst sampling x from p x ( x )
and then y from p ( y | x ) =
K ( y − f ( x )) ; this does not require computing the Jacobian or inverse f ψ − 1 .
S PREAD M AXIMUM L IKELIHOOD E STIMATION
Minimising the forward KL divergence between the empirical data distribution ˆ p ( x ) and a model p θ ( x ) is equivalent to Maximum Likelihood Estimation (MLE) of the parameters θ of the model.
Minimising instead the forward spread KL divergence, (cid:102) KL (ˆ p ( x ) || p θ ( x ))
=
− (cid:80) Nn =1 log p θ ( y n ) + const. , where y n are sampled i.i.d from ˜ p
( y ) =
(cid:82) x p ( y | x )ˆ p ( x ) , results in a new type of estimation, namely spread
MLE.
In what follows, we will discuss the statistical properties of spread MLE and demonstrate how it enables the training of models where maximum likelihood is not suited.
5. S TATISTICAL PROPERTIES
Maximum likelihood is a cherished criterion because it exhibits many favourable statistical properties, mainly consistency (convergence to the true parameters in the large data limit) and asymptotic efﬁciency (achieves the Cramér-Rao Lower Bound, which is a lower bound on the variance of any unbiased estimators) - see Casella & Berger (2002) for an introduction.
A key desideratum for spread
MLE is to analyse how these properties are affected.
In appendix(E) we demonstrate that spread
MLE (for a certain family of spread noise) needs weaker sufﬁcient conditions than MLE for both consistency and asymptotic efﬁciency.
Furthermore, a sufﬁcient condition for the existence of MLE is that the likelihood function is continuous over a compact parameter space Θ .
We provide an example in appendix(E.1) where the compactness requirement is violated, but spread MLE is still well deﬁned.
5. A PPLICATIONS
As an application to show the effectiveness of spread MLE, we use it to train implicit models
[Equation]
[Equation]
where θ are the parameters of the encoder g .
We show that, despite the likelihood not being deﬁned (see also section(K) for a simple linear model example), we can nevertheless successfully train such models using modiﬁed EM/variational algorithms (Barber (2012)).
5.2. T RAINING I MPLICIT LINEAR M ODELS : D ETERMINISTIC ICA
ICA (Independent Components Analysis) corresponds to the model p ( x, z )
= p ( x | z ) (cid:81)
i p ( z i ) , where the independent components z i follow a non-Gaussian distribution.
For Gaussian noise ICA an observation x is assumed to be generated by the process p ( x | z ) = (cid:81)
j N (cid:0) x
j g j ( z ) , γ 2 (cid:1) where g i ( z ) mixes the independent latent process z .
In linear ICA, g j ( z ) =
a T j z where a j is the j th column on the mixing matrix A .
For small observation noise γ 2 , it is well known that the maximum likelihood
EM algorithm to learn A from observed data is ineffective (Bermond & Cardoso, 1999; Winther &
Petersen, 2007).
To see this, consider X
= Z (where X and Z are the dimension of the data and latents respectively) and invertible A , x = Az .
At iteration k the EM algorithm has an estimate A k of the mixing matrix.
The M-step updates A k to
[Equation]
where, for zero observation noise ( γ = 0 ),
[Equation]
[Figure]
(a) Error versus observation noise γ .
[Figure]
and ˆ S ≡ 1 N (cid:80) n x n x T n is the moment matrix of the data.
Thus, A k +1 =
ˆ SA − T k (cid:16) A − 1 k ˆ SA − T k (cid:17) − 1 =
A k and the algorithm ‘freezes’.
Similarly, for low noise γ (cid:28) 1 progress critically slows down.
To deal with small noise and the limiting case of a deterministic model ( γ = 0 ), we consider Gaussian spread noise p ( y | x ) =
N [Equation] y x, σ 2 I X (cid:1) to give
[Equation]
Using spread noise, the empirical distribution is replaced by the spreaded empirical distribution ˆ p ( y )
= 1 N [Equation] y x n ,[-σ 
-]
[Equation]
I X (cid:1)
The M-step has the same form as equation 19 but with modiﬁed statistics
[Equation]
The E-step optimally sets
[Equation]
where Z q ( y ) is a normaliser and
[Equation]
[Equation]
Since the posterior p ( z | y ) peaks around N ( z µ ( y ) , Σ) , we rewrite equation 22 as
[Equation]
and similarly for E (cid:2) zz T (cid:3) .
Writing the expectations with respect to N ( z µ ( y ) , Σ) allows for a simple but effective importance sampling approximation focused on regions of high probability.
[Figure]
We implement this update by drawing S y samples from N (cid:0) y x n , σ
I X (cid:1) and, for each y sample, we draw S z samples from N ( z µ ( y ) , Σ) .
This scheme has the advantage over more standard variational approaches, see for example Winther & Petersen (2007), in that we obtain a consistent estimator of the M-step update for A .
We show results for a toy experiment in ﬁgure(1), learning the underlying mixing matrix in a deterministic non-square setting.
Note that standard algorithms such as FastICA (Hyvärinen, 1999) fail in this setting.
The spread noise is set to σ = max(0 .
5 ∗ sqrt ( mean ( AA T ))) .
This modiﬁed EM algorithm thus learns a good approximation of the underlying
A , with no critical slowing down.
Other applications of EM that suffer from this slow down phenomenon, such as MLE for Slow Feature Analysis (Turner & Sahani (2007)), or other probabilistic matrix factorization algorithms (Barber (2012)), can also beneﬁt from spread divergence.
5.2. T RAINING I MPLICIT N ON - LINEAR M ODELS : δ -VAE
A standard way to train a deep generative model p θ ( x ) =
(cid:82) p θ ( x | z ) p ( z ) dz is to use maximum likelihood (minimizing D (ˆ p ( x ) || p θ ( x )) ).
The likelihood equation 18 is in general intractable and it is common to use the variational lower bound (see (Kingma & Welling, 2013)).
However, for a deterministic observation model p θ ( x | z )=
δ ( x − g θ ( z )) and Z<X , this generative model describes only a low dimensional manifold in the data space and the divergence D (ˆ p ( x ) ||
p
θ ( x )) is not well deﬁned.
Additionally the above bound is not well deﬁned (due to log of a delta function) and the variational EM approach fails, as in the deterministic ICA setting.
To address this, we instead minimize the spread divergence KL (˜ p ( y ) ||
˜
p
θ ( y )) .
For Gaussian noise with ﬁxed diagonal noise p ( y | x )= N (cid:0) y x, σ
I X (cid:1) , we can write ˜ p
( y )
= 1 N (cid:80) Nn
=1 N (cid:0) y x n , σ 2 I X (cid:1) and ˜ p
θ ( y ) =
(cid:90) p ( y | x ) p θ ( x ) dx =
(cid:90) N (cid:0)
y g θ
( z ) , σ 2 I X (cid:1) p ( z ) dz = (cid:90) p θ ( y | z ) p ( z ) dz.
We then minimize the divergence
[Equation]
Typically, the integral over y is intractable, in which case we resort to a sampling estimation.
Neglecting constants, the divergence estimator is 1 NS (cid:80)
Nn =1 (cid:80)
Ss =1 log ˜ p
θ ( y ns ) , where y ns is a spread noise sample from p ( y n | x n ) ; for example y ns ∼ N (cid:0)
y ns x n , σ 2 I X (cid:1) .
For non-linear g , the distribution ˜ p
θ ( y ) is usually intractable
and we therefore use the variational lower bound
[Equation]
The approach is a straightforward extension of the standard variational autoencoder and in appendix(G) we also derive a lower variance objective and detail how to learn the spread noise (also see appendix(F)).
We dub this model and associated spread divergence training the ‘ δ -VAE’.
[Figure]
[Figure]
MNIST Experiment: We trained a δ -VAE on MNIST (LeCun et al. (2010)) with (i) ﬁxed Laplace spread noise, equation 15, (ii) ﬁxed Gaussian spread noise, equation 14 and (iii) Gaussian noise with learned covariance, section(4.1) with rank R = 20 ; see appendix(H) for details.
Figures 2(a,b,c) show samples from p θ ( x ) for these models; MNIST is sufﬁciently easy that it is hard to distinguish between the quality of the ﬁxed and learned noise samples.
However, qualitatively, the sharpness of the Laplace spread noise trained model is higher than for the Gaussian noise and motivates that the spread noise can affect the quality of the learned model.
We speculate that Laplace noise improves image sharpness since the noise focuses attention on discriminating between points close to the data manifold (since the Laplace distribution is leptokurtic and has a higher probability of generating points close to the data manifold than the Gaussian distribution).
Figure 2(d) visualizes the Gaussian learned covariance and shows that the learned noise is largely orthogonal to the data manifold.
CelebA Experiment: We trained a δ -VAE on the CelebA dataset (Liu et al., 2015) with (i) ﬁxed and (ii) learned spread with injective function, see appendix(I).
We compared to results from a standard
VAE with ﬁxed Gaussian noise p ( x | z ) =
N ( x g θ ( z ) , 0 .
5 I X )
Tolstikhin et al.
For (i) the ﬁxed spread divergence uses Gaussian noise N ( y x, 0 .
25 I X ) .
For (ii) we use Gaussian noise with learned injective function ResNet f ψ ( · ) =
I ( · ) + g ψ ( · ) ; see appendix(I) for more details.
Figureshows samples from δ -VAE trained using Gaussian spread divergence with both ﬁxed and learned spread noise (with g θ ( z ) initialised to the ﬁxed-noise setting).
It is notable how the ‘sharpness’ of the image samples substantially increases when learning the spread noise.
Table 1 shows FID (Heusel et al.
score comparisons between different algorithms 3 .
The δ -VAE signiﬁcantly improves on the standard VAE result; δ -VAE with injective function learning also improves on the ﬁxed-noise δ -VAE.
Indeed the injective δ -VAE results are comparable to popular GAN and WAE models (Gulrajani et al.
; Berthelot et al.
; Arjovsky et al.
; Kodali et al.
Mao et al.
(2017); Fedus et al.
; Tolstikhin et al.
Whilst the δ -VAE results are not fully state-of-the-art, we believe it is the ﬁrst time that implicit models have been trained using a principled maximum likelihood based approach.
Our expectation is that by increasing the complexity of the generative model g θ and injective function f ψ , or using different noise such as Laplace distribution, the results will become competitive with state-of-the-art GAN models.
R ELATED W ORK
MMD versus spread f -divergence:
In spite of the conditions required for deﬁning the spread divergence being closely related to the kernel requirement of MMD (Gretton et al., 2012), we also show that MMD and spread Total Variation distance 4 can be written as different norms ( L 2 , Lrespectively) of a common objective (see appendix(C)).
Instance noise: The instance noise trick to stabilize GAN training Roth et al.
Sønderby et al.
(2016) is a special case of spread divergence using ﬁxed Gaussian noise.
Whilst other similar tricks (for example Furmston & Barber (2009)) have been proposed previously, we believe that it is important to state the general utility of the spread noise approach.
δ -VAE versus WAE : The Wasserstein auto-encoder Tolstikhin et al.
(2017) is another implicit generative model that uses an encoder-decoder architecture.
The difference is that δ -VAE is based on
KL divergence which is corresponding to MLE but WAE uses the Wasserstein distance.
δ -VAE versus denoising VAE : The Denoising VAE Im et al.
(2017) uses a VAE with noise added to the data only.
In contrast, the δ -VAE adds noise to both the data and model.
Since the denoising VAE model only adds noise to the model, it cannot recover the true data distribution.
MMD GAN with kernel learning : The idea of learning a kernel to increase discrimination is also used in MMD GAN (Li et al.
Similar to ours, the kernel in MMD GAN is constructed by ˜ k
= k ◦ f ψ where k is a ﬁxed kernel and f ψ is a neural network.
To ensure M k ◦ f ψ ( p, q ) = 0 ⇔ p = q , this requires f ψ to be injective (Gretton et al.
However, in the MMD GAN framework, f ψ ( x ) usually maps x to a lower dimension.
This is crucial for MMD because the amount of data required to produce a reliable estimator grows with the data dimension (Ramdas et al.
and the computation cost of MMD scales quadratically with the amount of data.
Whilst using a lowerdimensional mapping makes MMD more practical it also makes it difﬁcult to construct an injective function f .
For this reason, heuristics such as the auto-encoder regularizer (Li et al.
(2017)) are considered.
In contrast, for the δ -VAE, the computational cost of estimating the divergence is linear in the number of datapoints.
For this reason there is no need for f ψ to be a lower-dimensional mapping; guaranteeing that f ψ is injective is therefore relatively straightforward for the δ -VAE.
Flow-based generative models : Invertible ﬂow-based functions (Rezende & Mohamed (2015)) have been used to boost the representation power of generative models.
Note our use of injective functions is quite distinct from the use of ﬂow-based functions to boost generative model capacity.
In our case, the injective function f does not change the model – it only changes the divergence.
For this reason, the spread divergence doesn’t require the log determinant of the Jacobian (which is required in
Rezende & Mohamed (2015); Behrmann et al.
meaning that more general invertible functions can be used to boost the discriminatory power of a spread divergence.
S UMMARY
We described how to deﬁne a divergence even when two distributions do not have the same support.
Previous approaches (Furmston & Barber, 2009; Sønderby et al., 2016) can be seen as special cases.
We showed that deﬁning divergences this way enables us to train deterministic generative models using standard likelihood based approaches.
In principle, we can learn the underlying true data generating process by the use of any valid spread divergence.
In practice, however, the quality of the learned model can depend strongly on the choice of spread noise.
We therefore investigated learning spread noise to maximally discriminate two distributions.
We found the resulting training approach stable and that it can signiﬁcantly improve the image generation results.
Whilst state-of-the-art image generation is not the focus of this work, we obtained promising results.
We also discussed the conditions under which spread MLE is consistent and asymptotically efﬁcient, some of which are weaker than the equivalent MLE conditions.
Perhaps the most appealing aspect of the spread noise is that is enables one to re-use standard machine learning approaches in statistics such as maximum likelihood to train models that would be otherwise unsuited to standard statistical training approaches.
A P ROOF OF T HEOREM A
Theorem 1.
Consider distributions p , q , and L 1 integrable function K and Fourier Transform F{ K } .
Let F{ K } (cid:54) = 0 or F{ K } = 0 on at most a countable set.
Then
[Equation]
Proof.
When F{ K } (cid:54) = 0 , F{ K }F{ p } =
F{ K }F{ q } ⇒ F{ p } =
F{ q } is trivial.
We ﬁrst show that the Fourier transform of an L 1 function is continuous on R d .
When F{ K } = 0 on at most a countable set, we then show that F{ q } and F{ p } cannot be different at a set of countable points.
Since any distribution q is in L 1 , we can write
[Equation]
[Equation]
So F{ q } is (uniformly) continuous.
The same argument applies to show that F{ p } is uniformly continuous.
Since F{ K } = 0 on at most a countable set C , we assume there is a point w 0 ∈ C where F{ q } ( w 0 ) (cid:54) =
F{ p } ( w 0 ) .
Without loss of generality, we assume F{ q } ( w 0 ) − F{ p } ( w 0 )
= θ > 0 .
For points w
+ h
that are not in C , F{ K } ( w 0 + h ) (cid:54)
=
and it follows therefore that F{ K }F{ p } = F{ K }F{ q } implies F{ q } ( w 0 + h ) −F{ p } ( w 0 + h )
= 0 .
By continuity of F{ p } and F{ q } , we have F{ q } ( w 0 + h ) − F{ p } ( x 0 + h ) → 0 when h → 0 , which leads to a contradiction ( θ cannot be zero).
B S PREAD NOISE MAKES DISTRIBUTIONS MORE SIMILAR
The data processing inequality for f -divergences (see for example Gerchinovitz et al.
(2018)) states that D f (˜ p ( y ) ||
˜ q ( y )) ≤ D f
( p ( x ) || q ( x )) .
For completeness, we provide here an elementary proof of this result.
We consider the following joint distributions
[Equation]
whose marginals are the spreaded distributions
[Equation]
The divergence between the two joint distributions is
[Equation]
The f -divergence between two marginal distributions is no larger than the f -divergence between the joint (see also Zhang et al.
To see this, consider
[Equation]
Hence,
[Equation]
Intuitively, spreading two distributions increases their overlap, reducing the divergence.
When p and q do not have the same support, D f ( q ( x ) || p ( x )) can be inﬁnite or not well-deﬁned.
C R ELATION TO MMD
Spread divergence can be generally constructed from f -divergence, we show how to build a connection to maximum mean discrepancy (Gretton et al.
by using the spread total variation distance:
For a translation invariant kernel k ( · , x ) , the MMD is (we use γ k to denote the MMD distance with kernel k )
[Equation]
H
Suppose both k
and it’s square root of Fourier transform (cid:112)
ˆ k
( ˆ represents the Fourier transform) are absolutely integratable, we can also rewrite the MMD distance the as following (see Sriperumbudur et al.
(2010) for a derivation):
[Equation]
where Φ :=
( (cid:112) ˆ k ) ∨
( ∨ represents the inverse Fourier transform).
We can further deﬁne ˜ Φ
= Z − 1 Φ , where Z = (cid:82) Φ( x ) dx .
Therefore, ˜ Φ is a probability density function and the MMD can be written as
[Equation]
Total variation distance between p ( x ) and q ( x ) is deﬁned as
[Equation]
In the spread total variation distance, we can deﬁne spread as the noise distribution k
= ˜Φ , thus
[Equation]
So MMD and spread Total Variation distance can be written as different norms of a common objective.
D S PREAD DIVERGENCE BETWEEN TWO DELTA DISTRIBUTIONS
Let p 0 ( x ) =
δ ( x − µ 0 ) , p q ( x ) =
δ ( x − µ 1 ) , assume Gaussian spread noise p ( y | x ) =
N (cid:0) y x, σ 2 (cid:1) and σ 2 = 0 . 5 , so
[Equation]
E S TATISTICAL P ROPERTIES OF M AXIMUM L IKELIHOOD E STIMATOR E. E XISTENCE OF S PREAD MLE
In some situations there may not exist a Maximum Likelihood Estimator (MLE) for p ( x | θ ) , but there is a MLE for the spread model p ( y | θ )
=
(cid:82) p ( y | x ) p ( x | θ ) dx .
For example, suppose that X ∼ N ( µ, σ 2 ) ( µ, 0 < σ 2 < ∞ ).
So θ = ( µ, σ 2 ) ∈ R × R
+ .
Assume we only have one data point x .
Then the log-likelihood function is L ( x ; θ ) ∝ − log σ − 12 σ 2 ( x − µ ) 2 .
Maximising with respect to µ , we have µ = x and the log-likelihood becomes unbounded as σ 2 → 0 .
In this sense, the MLE for ( µ, σ 2 ) does not exist.
On the other hand, we can check whether the MLE for p ( y | θ ) exists.
We assume Gaussian spread noise with ﬁxed variance σ 2 f .
Since we only have one data point x , the spread data distribution becomes p ( y | x ) =
N (cid:16) y x, σ 2 f (cid:17) , and the model is p ( y | θ ) =
N (cid:16) y µ, σ 2 + σ 2 f (cid:17) .
We can sample
N points from the spread model, so the spread log likelihood function is (neglecting constants)
L ( y 1 , . . .
, y N ; θ ) = − N 2 log( σ 2 + σ 2 f ) − 1 2( σ 2 + σ 2 f ) (cid:80) Ni
=
( y i − µ ) 2 .
The MLE solution for µ is µ = 1 N (cid:80) Ni =
y i ; the MLE solution for σ 2 is σ 2 = 1 N (cid:80) i
( y i − µ ) 2 − σ 2 f , which has bounded spread likelihood value.
Note that in the limit of a large number of spread samples N → ∞ , the MLE σ 2 = 1 N ( y i − µ ) 2 → σ 2 f tends to 0.
Throughout, however, the (scaled by N ) log likelihood remains bounded.
E. C ONSISTENCY
Consistency of an estimator is an important property that guarantees the validity of the resulting estimate at convergence as the number of data points tends to inﬁnity.
In what follows, we refer to sufﬁcient conditions for a consistent MLE estimator, before addressing the question of whether using spread MLE is also consistent and under what conditions.
E.2. C ONSISTENCY FOR MLE
Sufﬁcient conditions for the MLE being consistent and converging to the global maximum are given in Wald (1949).
However, they are usually difﬁcult to check even for some standard distributions.
The sufﬁcient conditions for MLE being consistent and converging to a local maxima are given in
Cramér (1999) and are more straight forward to check:
C1.
(Identiﬁable): p ( x | θ 1 ) =
p ( x | θ 2 ) → θ 1 = θ 2 .
C2.
The parameter space Θ is an open interval (¯ θ, ¯ θ ) , Θ : −∞ ≤ ¯ θ < θ < ¯ θ ≤ ∞ .
C3.
p ( x | θ ) is continuous in θ and differentiable with respect to θ for all x. C4.
The set A = { x : p θ ( x ) > 0 } is independent of θ .
Let X 1 , X 2 , . .
. be i.i.d with density p ( x | θ 0 ) ( θ ∈ Θ ) satisfying conditions C1–C4, then there exists a sequence ˆ θ n = ˆ θ n ( X 1 , ..., X n ) of local maxima of the likelihood function L ( θ 0 ) =
(cid:81) ni
=1 p ( x i | θ 0 ) which is consistent: ˆ θ p −→ θfor all θ ∈ Θ
The proof can be found in Lehmann (2004) or Cramér (1999).
E.2. C ONSISTENCY OF SPREAD MLE
We provide the necessary conditions for Spread MLE being consistent.
C1.
(Identiﬁable): p ( x | θ ) is identiﬁable.
From section(3)
it follows immediately that p ( y | θ 1 ) =
p ( y | θ 2 ) → p ( x | θ 1 ) = p ( x | θ 2 ) → θ 1 = θ 2 , where the ﬁnal implication follows from the assumption that p ( x | θ ) is identiﬁable.
Hence if p ( x | θ ) is identiﬁable, so is p ( y | θ ) .
C2.
The parameter space Θ is an open interval (¯ θ, ¯ θ ) , Θ : −∞ ≤ ¯ θ < θ < ¯ θ ≤ ∞ .
This condition is unchanged for p ( y | θ ) .
C3.
On p ( y | θ ) , we require the same condition on p ( x | θ ) as in MLE; p ( y | θ ) is continuous in θ and differentiable with respect to θ for all y . C4.
For spread noise p ( y | x ) who has full support on R d (for example Gaussian noise), p ( y | θ ) is greater than zero everywhere and hence the original condition C4 is automatically guaranteed
The conditions that guarantee consistency for spread MLE are weaker for the spread model p ( y | θ ) than for the standard model p ( x | θ ) , since C4 is automatically satisﬁed.
Ferguson (1982) gives an example for which MLE exists but is not consistent by violating condition C4, whereas spread MLE can be used to obtain a consistent estimator.
E. A SYMPTOTIC E FFICIENCY
A key desirable property of any estimator is that it is efﬁcient.
The Cramer-Rao bound places a lower bound on the variance of any unbiased estimator and an efﬁcient estimator much reach this minimal value in the limit of a large amount of data.
Under certain conditions (see below) the Maximum
Likelihood Estimator attains this minimal variance value meaning that there is no better estimator possible than maximum likelihood (in the limit of a large amount of data).
This is one of the reasons that the maximum likelihood is a cherished criterion.
E.3. A SYMPTOTIC E FFICIENCY FOR MLE
Building upon conditions C1-C4, additional conditions on p ( x | θ ) are required to show MLE is asymptotical efﬁcient:
C5.
For all x in its support, the density p θ ( x ) is three times differentiable with respect to θ and the third derivative is continuous.
C6.
The derivatives of the integral (cid:82) p θ ( x ) dx respect to θ can be obtained by differentiating under the integral sign, that is: ∇ θ (cid:82) p θ ( x ) dx = (cid:82) ∂ θ p θ ( x ) dx .
C7.
There exists a positive number c ( θ 0 ) and a function M θ 0 ( x ) such that
[Equation]
where A is the support set of x and E θ
[ M θ 0 ( x )] < ∞ .
Let X 1 , ..., X n be i.i.d with density p θ ( x ) and satisfy conditions C1-C7, then any consistent sequence ˆ θ
=
ˆ θ n ( X 1 , ..., X n ) of roots of the likelihood equation satisﬁes
[Equation]
where F − 1 ( θ 0 ) is the inverse of Fisher information matrix (also called Cramér-Rao Lower Bound, which is a lower bound on variance of any unbiased estimators ).
The conditions and proof can be found in Lehmann (2004).
E.3. A SYMPTOTIC E FFICIENCY FOR MLE
As with MLE above, we require further conditions on p ( y | θ ) for ensuring spread MLE is asymptotically efﬁcient:
C5.
On p ( y | θ ) , we require the same condition as applied to p ( x | θ ) in the MLE case; for all y in its support, the density p θ ( y ) is
three times differentiable with respect to θ
and the third derivative is continuous.
C6.
For spread noise p ( y | x ) , which has full support on R d (for example Gaussian noise), the support of y is independent of θ .
Leibniz’s rule 5 allows us to differentiate under the integral: ∇ θ (cid:82) p θ ( y ) dy
=
(cid:82) ∂
θ p θ ( y ) dy , so this condition is automatically satisﬁed.
C7.
On p ( y | θ ) , we require the same condition as applied to p ( x | θ ) in the MLE case; There exist positive number c ( θ 0 ) and a function M θ 0 ( y ) such that
[Equation]
all y
[Equation]
c
[Equation]
θ 0 ) where A is the support set of y and E θ
[ M θ 0 ( y )] < ∞ .
Thus the conditions that guarantee asymptotically efﬁcient for the spread model p ( y | θ ) are weaker than for the standard model p ( x | θ ) , since C4 and C6 are automatically satisﬁed.
F P ERTURBATION A PPROXIMATION OF G AUSSIAN S PREAD
Herein, in the ﬁxed noise setting, we derive a perturbation based approximation to the spread noise, which in princple can lead to a lower variance estimator.
We can write a function f with perturbed input and integrated over noise as
[Equation]
where p ( ξ ) =
N ( ξ 0 , Σ) .
Taylor expanding around ξ = 0 , we have
[Equation]
Where H is the Hessian matrix H i,j = ∂ 2 f ∂x
i ∂x j .
When x ’s dimension is small, we can use equationto explicitly calculate the trace.
When the dimension of x is large, we can form a Monte Carlo estimation of equation 35.
To do this we ﬁrst sample ξ ∼ p ( ξ ) and then calculate the Hessian-vector product H ξ .
This can be efﬁciently calculated by AutoDiff backward mode (Schraudolph (2002);
Pearlmutter (1994)), without storing the Hessian matrix in memory.
For example, in δ -VAE with ﬁxed Gaussian noise, according to the bound equation 56 (ignoring the constant):
[Equation]
[Equation] [Equation]
≈ H (Σ φ
[Equation]
x )
[Equation]
+
[Equation]
log p
[Equation]
x = g θ
[Equation]
µ φ
[Equation]
x
[Equation]
+ C φ
[Equation]
x
[Equation]
+
[Equation]
p
[Equation]
µ
[Equation]
x
[Equation]
φ
[Equation]
x
[Equation]
Where H is the Hessian matrix H i,j = ∂ 2 f ( x ) ∂x i ∂x j and C φ ( x ) is the Cholesky decomposition of Σ φ ( x ) .
G S PREAD D IVERGENCE FOR D ETERMINISTIC D EEP G ENERATIVE M ODELS
Instead of minimising the likelihood, we train an implicit generative model by minimising the spread divergence
[Equation]
[Equation]
For Gaussian noise with ﬁxed diagonal noise p ( y | x ) =
N ( y | x, σ 2 I X ) , we can write
[Equation]
(42) and
[Equation]
For the spread divergence with learned covariance Gaussian noise which is discussed in section(4.1), we can write
[Equation]
and spread divergence with learned injective function as discussed in section(4.2)
[Equation]
According to our general theory,
[Equation]
Here
[Equation]
Typically, the integral over y will be intractable and we resort to an unbiased sampled estimate (though see below for Gaussian q ).
Neglecting constants, the KL divergence estimator is
[Equation]
where y ns is a perturbed version of x n .
For example y ns ∼ N (cid:0) y ns x n , σ 2 I X (cid:1) for ﬁxed Gaussian noise case and other cases are similar.
In most cases of interest, with non-linear g , the distribution ˜ p θ ( y ) is intractable.
We therefore use the variational lower bound
[Equation]
Parameterising the variational distribution as a Gaussian,
[Equation]
then we can reparameterise and write
[Equation]
where H (Σ φ ( y )) is the entropy of a Gaussian with covariance Σ φ ( y ) .
For ﬁxed covariance Gaussian spread noise in D dimensions, this is
[Equation]
where C φ ( y ) is the Cholesky decomposition of Σ φ ( y )
[Equation]
We can integrate equation 52 over y to give the bound
[Equation]
) where
[Equation]
We notice that the second term is zero, so the ﬁnal bound for the ﬁxed Gaussian spread KL divergence is (ignoring the constant)
[Equation]
By analogy, for spread KL divergence with learned variance, the bound is (ignoring the constant)
[Equation]
Where S ψ is the cholesky decomposition of Σ ψ .
For speciﬁc covariance structure introduced in section section(4.1), efﬁcient methods for sampling, matrix inverting and log determinant calculation are available, see appendix(J).
For spread KL divergence with learned injective function, the bound is (ignoring the constant)
[Equation]
The overall procedure is therefore a straightforward modiﬁcation of the standard VAE method Kingma & Welling (2013) with additional learning the spread to maximize the divergence:
Choose a noise distribution p ( y | x ) 2.
Choose a tractable family for the variational distribution, for example q φ ( z | y )
=
N ( z µ φ ( y ) , Σ φ ( y )) and initialise φ . 3.
We then sample a y n for each datapoint (if we’re using S = 1 samples) 4.
If learning the spread noise: (a) Draw samples (cid:15) to estimate − log ˜
p θ ( y n ) according to the corresponding bound.
(b) Do a gradient ascent step in ψ . 5.
Draw samples (cid:15) to estimate log ˜
p
θ ( y n ) according to the corresponding bound.
Do a gradient ascent step in ( θ, φ ) .
Go to 3 and repeat until convergence.
H MNIST E XPERIMENT
We ﬁrst scaled the MNIST data to lie in [0 , 1] .
We use Laplace spread noise σ = 0 .
3 and Gaussian spread noise σ = 0 .
3 for the δ VAE.
Both encoder and decoder contains 3 feed-forward layers, each layer with 400 units and ReLu activation function.
The latent dimension is Z = 64 .
The variational inference network q φ ( z | y )
=
N (cid:16) z µ φ ( y ) , σ 2 φ I Z (cid:17) has a similar structure for the mean network µ φ ( y ) .
For ﬁxed spread δ -VAE , learning was done using the Adam Kingma & Ba (2014) optimizer with learning rate 5 e − 4 for 200 epochs.
For δ -VAE with learned spread (learned covariance), we additioanly train the covariance for 2 epochs using Adam optimizer with learning rate 5 e − 5 after everytime we train the model for 10 epochs.
I C ELEB A E XPERIMENT
We pre-processed CelebA images by ﬁrst taking 140x140 centre crops and then resizing to 64x64.
Pixel values were then rescaled to lie in [0 , 1] .
For the learned spread we use Gaussian noise with learned injective function ResNet f ψ ( · ) =
I ( · ) + g ψ ( · ) , where g ψ ( · ) is a one layer convolutional neural net with kernel size 3 × 3 and stride 1.
We use spectral normalization Miyato et al.
(2018) to satisfy the Lipschitz constraint: we replace the weight matrix w of the convolution kernel by w SN ( w ) :
= c × w/σ ( w ) where σ ( w ) is the spectral norm of w and c ∈ (0 , 1) .
This guarantees that f ψ is invertible – see Behrmann et al.
The encoder and decoder are 4-layer convolutional neural net with batch norm (Ioffe & Szegedy (2015)).
Both encoder and decoder used fully convolutional architectures with 5x5 convolutional ﬁlters and used vertical and horizontal strides 2 except the last deconvolution layer we used stride 1.
The injective function f is a 1 layer convolutional network with 3*3 kernl and stride 1.
Conv k stands for a convolution with k ﬁlters, DeConv k for a deconvolution with k ﬁlters, BN for the batch normalization Ioffe & Szegedy (2015),
Relu for the rectiﬁed linear units, and FC k for the fully connected layer mapping to R k .
[Equation]
We use batch size 100 and latent dimension z dim = 100 in all CelabA experiments.
For the δ -VAE with ﬁxed spread, we use the ﬁxed Gaussian noise with 0 mean and (0 . 5) 2 I covariance.
We train the model for 500 epochs using Adam optimizer with learning rate 1 e − 4 .
The learning rate decay with scaling factor 0 .
9 every 100000 iterations.
For the δ -VAE with ﬁxed spread We ﬁrst train a δ -VAE with ﬁxed f ( x ) = x and ﬁxed Gaussian noise with 0 mean and (0 . 5) 2 I diagonal covariance for 300 epochs, the learning rate decay with scaling factor 0 .
9 every 100000 .
Then we start iterative training by doing one step inner maximisation over the spread divergence’s parameter ψ using Adam optimizer with learning rate 1 e − 5 and one step minimization over the model parameter’s
( θ, φ ) using Adam optimizer for additional 200 epochs.
We can share the ﬁrst 300 epochs between two models.
When we sample form two models, we ﬁrst sample from a 100 dimensional standard Gaussian distribution z ∼ N (0 , I ) and use the same latent code z to get samples from both δ -VAE with ﬁxed and learned spread, so we can easily compare the sample quality between two models.
J W OODBERRY
When evaluating the log probability of this Gaussian, we use the Woodberry identity
[Equation]
so we only have to invert a R × R matrix.
A similar trick is applied to calculate the log determinant:
[Equation]
The parameter L is trained using reparameterization trick.
When sampling ξ ∼ N ( µ, LL T + Iσ 2 ) , we ﬁrst sample z ∈ R R from N ( z 0 , I ) and then sample noise (cid:15) ∼ N (0 , Iσ 2 ) , thus a sample from
N ( µ, LL T + Iσ 2 ) can be represented by ξ =
Lz + Iσ(cid:15) + µ .
K D ETERMINISTIC L INEAR L ATENT M ODEL
Our aim here is to show how the classical deterministic PCA algorithm can be derived through a maximum-likelihood approach, rather than the classical non-probabilistic least-squares derivation.
This is remarkable since the likelihood itself is not deﬁned for this model.
For isotropic Gaussian observation noise with variance γ 2 , the Probabilistic PCA model (Tipping &
Bishop, 1999) for X -dimensional observations and Z -dimensional latent is
[Equation]
When γ = 0 , the generative mapping from z to x is deterministic and the model p θ ( x ) has support only on a subset of R X and the data likelihood is in general not deﬁned for Z <
X .
In the following we consider general γ , later setting γ to zero throughout the calculation.
To ﬁt the model to iid data { x 1 , . .
, x N } using maximum likelihood, the only information required from the dataset is the data covariance ˆΣ .
For γ > 0 , the maximum likelihood solution for PPCA is
F = U Z (cid:0) Λ Z − γ
I Z (cid:1) 12 R , where Λ Z , U Z are the Z largest eigenvalues, eigenvectors of ˆΣ ; R is an arbitrary orthogonal matrix.
Using spread noise p ( y | x ) =
N (cid:0) y x, σ 2 I X (cid:1) , the spreaded distribution is a Gaussian ˜ p
θ ( y ) =
N (cid:0) y 0 , FF T + ( γ 2 + σ 2 ) I X (cid:1) .
Thus, ˜ p θ ( y ) is of the same form as PPCA, albeit with an inﬂated covariance matrix.
Adding Gaussian spread noise to the data also simply inﬂates the sample covariance to ˆΣ (cid:48)
=
ˆΣ + σ
I X .
Since the eigenvalues of ˆΣ (cid:48) ≡ ˆΣ + σ
I X are simply Λ (cid:48)
=
Λ + σ
I X , with unchanged eigenvectors, the optimal deterministic ( γ = 0 ) latent linear model has solution F = U Z (cid:0) Λ (cid:48)
Z − σ 2 I Z (cid:1) 12 R =
U Z Λ 12 Z R .
We have thus recovered the standard PCA solution; however, the derivation is non-standard since the likelihood of the deterministic latent linear model γ = 0 is not deﬁned.
Since classical deterministic
PCA cannot normally be described in terms of a likelihood, the usual derivation of PCA is to deﬁne it as the optimal least squares reconstruction solution based on a linear projection to a lower-dimensional subspace, see for example Barber (2012).
Nevertheless, using the spread divergence, we learn a sensible model and recover the true data generating process if the data were exactly generated according to the deterministic model.
[Figure] [Figure]