L EARNING WITH R EFLECTIVE L IKELIHOODS
Anonymous authors
Paper under double-blind review
A BSTRACT
Models parameterized by deep neural networks have achieved state-of-the-art results in many domains.
These models are usually trained using the maximum likelihood principle with a ﬁnite set of observations.
However, training deep probabilistic models with maximum likelihood can lead to the issue we refer to as input forgetting .
In deep generative latent-variable models, input forgetting corresponds to posterior collapse —a phenomenon in which the latent variables are driven independent from the observations.
However input forgetting can happen even in the absence of latent variables.
We attribute input forgetting in deep probabilistic models to the ﬁnite sample dilemma of maximum likelihood.
We formalize this problem and propose a learning criterion—termed reﬂective likelihood —that explicitly prevents input forgetting.
We empirically observe that the proposed criterion signiﬁcantly outperforms the maximum likelihood objective when used in classiﬁcation under a skewed class distribution.
Furthermore, the reﬂective likelihood objective prevents posterior collapse when used to train stochastic autoencoders with amortized inference.
For example in a neural topic modeling experiment, the reﬂective likelihood objective leads to better quantitative and qualitative results than the variational auto-encoder and the importance-weighted autoencoder.
I NTRODUCTION
Learning deep probabilistic models with maximum likelihood has led to many successes in density estimation, variational inference, and text and image generation (Dinh et al., 2016;
Kingma &
Dhariwal, 2018; Kingma & Welling, 2013; Rezende et al., 2014; Oord et al., 2016).
However, maximum likelihood learning of deep models often causes the problem of input forgetting —so called because it corresponds to ignoring the input.
We refer to an input here as any variable being conditioned upon—for example a covariate in supervised learning or a latent variable in deep generative latent variable models.
Input forgetting corresponds to latent variable collapse in the context of latent variable models and has been discussed in several works (Burda et al., 2015;
Bowman et al., 2015; Hoffman & Johnson, 2016; Chen et al., 2016;
Sønderby et al., 2016; Zhao et al., 2017;
Alemi et al., 2018; Dieng et al., 2018a).
It also occurs when learning with Restricted Boltzmann Machines ( RBM s ) where all the hidden units of the RBM collapse by learning to only capture the bias in the visible units (Cho et al., 2011).
Input forgetting makes posterior inference in deep generative models very difﬁcult.
In this paper we study input forgetting when the input is a latent variable and when the input is a covariate.
1. C ONTRIBUTIONS
In this paper, we propose a learning criterion that does not suffer from the input forgetting problem induced by maximum likelihood when ﬁtting deep probabilistic models.
This criterion naturally arises from what we call the ﬁnite sample dilemma of maximum likelihood.
The ﬁnite sample dilemma of maximum likelihood.
Consider a supervised learning setting where we have two random variables x and y .
Our goal is to learn the relationship between these two random variables by ﬁtting a conditional parametric model p θ ( y | x ) .
The learning problem corresponds to ﬁnding θ ∗ that minimizes the risk function,
[Equation]
where p ∗ denotes the population distribution and l ( · , · ) is a loss function that measures how different f θ ( x ) is from the true y .
Maximum likelihood corresponds to a log loss, l ( y , f θ ( x ))
= − log p
[Equation]
( y | x ) and R mle ( θ ) =
− E x ∼ p ∗
[Equation]
x )
[Equation]
E y ∼ p ∗
[Equation]
y
[Equation] [Equation]
p θ
[Equation]
y |
[Equation] [Equation]
In practice we approximate R mle ( θ ) using a ﬁnite dataset D = ( x n , y n ) Nn =1 that contains N realizations of the random variables x and y .
The sample-based maximum likelihood objective is
[Equation]
Now consider the risk of maximum likelihood under the same conditional model p θ ( y | x ) when x and y are independent,
[Equation]
The corresponding sample-based maximum likelihood objective is
[Equation]
The objectives ˜ L ( θ ; D ) and ˜ L (cid:48)
( θ ; D ) in Eq.
3 and Eq.
5 are related as follows
[Equation]
Although there is no relationship between the two population risks R mle ( θ ) and R (cid:48) mle ( θ ) , there is a relationship between their sample-based estimates.
When maximizing the likelihood under ﬁnite data—i.e. maximizing ˜ L ( θ ; D ) —the term ˜ L (cid:48)
( θ ; D ) is also being maximized.
However maximizing ˜ L (cid:48)
( θ ; D ) leads to input forgetting as the underlying data being used do not encode any relationship between x and y .
To see this consider maximum likelihood with the same data but where each label y is also paired with all possible inputs x .
This version of the data does not contain any relationship between x and y .
Maximum likelihood using this data where x and y are independent corresponds to maximizing ˜ L (cid:48)
( θ ; D ) .
A new learning criterion for supervised learning.
In light of the relationship in Eq.
6, we propose to explicitly discourage the independence behavior induced by maximizing ˜ L (cid:48)
( θ ; D ) .
We do so by maximizing the reﬂective likelihood ( RLL ).
Maximizing the RLL simultaneously maximizes the likelihood of the outputs under their true associated inputs and minimizes the average log-probability of these outputs when paired with all other possible inputs.
On a classiﬁcation under imbalance with the MNIST dataset, the classiﬁers trained with RLL outperform those trained with the usual maximum likelihood objective in all the instances of class imbalance.
The performance were similar when the class distribution is uniform.
A new family of stochastic auto-encoders for unsupervised learning.
We extend the proposed RLL to unsupervised learning with stochastic auto-encoders and amortized inference.
We call stochastic auto-encoders trained with the RLL objective Reﬂective AutoEncoders ( RAE s ).
RAE s do not suffer from the issue of posterior collapse—a phenomenon in which the estimated approximate posterior distribution of the latent variables reduces to the prior.
We evaluate RAE s on a neural topic modelling experiment and found they outperform models learned with the maximum likelihood objective both quantitatively and qualitatively.
1. R ELATED W ORK .
Our work is closely related to two lines of work: penalized maximum likelihood and posterior inference in deep generative models.
Traditional penalized maximum likelihood methods such as the Lasso and L 2 -norm regularization directly operate on the parameters of a model (Tibshirani, 1996; Hinton, 1987;
Louizos et al., 2017).
These regularizers penalize the magnitude of the parameters and correspond to speciﬁc prior distributions on the parameters from the Bayesian perspective.
In contrast several data-dependent regularizers have been proposed.
As early as 1995, Bishop (1995) proposed to add noise to the input when training a neural network with stochastic gradient descent for maximum likelihood learning and showed this corresponds to a form of Tikhonov regularization.
Several works have extended this to noise injection in the hidden units of a neural network (Srivastava et al., 2014; Maaten et al., 2013;
Gal & Ghahramani, 2016; Wager et al., 2013; Dieng et al., 2018b).
Our work relates to those datadependent regularizers that are explicit—in the sense that the corresponding objective function can be written as the sum of the initial objective function and an additional regularization term.
Our work also relates to the literature on posterior inference in deep generative models.
Latent variable models are ubiquitous in machine learning.
They often involve intractable integrals that are approximated with Markov chain Monte Carlo or variational inference.
Our work relates to variational methods for deep generative models (Kingma & Welling, 2013; Rezende et al., 2014).
The problem that often occurs in these settings is the latent variables are driven independent to the observations thus rendering posterior inference meaningless.
Several works have discussed this issue (Burda et al., 2015;
Bowman et al., 2015; Hoffman & Johnson, 2016; Chen et al., 2016;
Sønderby et al., 2016; Zhao et al., 2017;
Alemi et al., 2018; Dieng et al., 2018a).
In this paper we identify a possible cause to this problem and adopt a regularization approach to ﬁx it.
S UPERVISED L EARNING WITH R EFLECTIVE L IKELIHOODS
We consider the same setting as Section 1.1.
We have observations ( x n , y n ) Nn =1 and our goal is to ﬁt a parametric model p θ ( y | x ) to this data.
As stated in Section 1.1, the usual maximum likelihood objective might lead to parameters θ that do not capture the true relationship between x and y .
In this section we propose the RLL criterion that encourages parameter settings under which y is more likely when conditioning on its corresponding x than under the average of all the conditional probabilities of y given all possible inputs.
The RLL criterion has two forms.
Weak reﬂective likelihood.
The weaker form of RLL minimizes ˜ L (cid:48)
( θ ; D ) while maximizing the usual sample-based approximation ˜
L ( θ ; D ) ,
[Equation]
where α n ≥ 0 is a hyperparameter trading off these two terms.
We will discuss how to set α n at the end of this section.
The objective ˜ L weak RLL ( θ ; D ) provides us with a way to explicitly prefer
R mle ( θ ) in Eq.
(1) over R (cid:48) mle ( θ ) in Eq.
(4), over an entire training set.
The objective ˜ L weak RLL ( θ ; D ) , however, may capture the dependence between the input and the output for some examples but not necessarily for all examples.
This is because the second term in Eq.
7 can be driven to its minimal value by setting the term inside the log to zero for one example.
We tackle this issue by proposing a stronger version of RLL .
Strong reﬂective likelihood.
The stronger version of RLL favors parameters that capture the relationship between x and y by discouraging the independence behavior induced by maximization of ˜
L (cid:48)
( θ ; D ) ,
[Equation] [Equation]
The second term comes from an application of Jensen’s inequality,
[Equation]
and using the dataset D to approximate this upper bound.
The inner term of this upper bound is what we call a reﬂective probability ,
[Equation]
This reﬂective probability can be used as a diagnostic measure of whether a trained model has captured the dependence between the input and output for each pair.
The sample-based approximation of p reﬂ θ ( y ) is what appears inside the logarithm in the second term in Eq. 8.
Unlike ˜ L weak RLL , this stronger objective function ˜
L strong RLL explicitly decreases the reﬂective probability of each individual example ( x , y ) .
From here on out we focus on ˜ L strong RLL and refer to it as RLL .
Stochastic Approximation of RLL.
The objective in Eq. 8 is not amenable to large datasets.
First, when the sample size N is large computing ˜ L strong RLL ( θ ; D ) is intractable.
Second even if we use a small subset of data we are still left with approximating p reﬂ θ ( y ) which is also intractable for large
N .
To enable maximizing RLL with large datasets, we propose to maximize ˜ L strong RLL ( θ
[Equation]
M , M
[Equation]
) =
[Equation]
where M and M (cid:48) are two minibatch of data samples.
In practice we improve the efﬁciency by constructing the second minibatch M (cid:48) using the ﬁrst minibatch M .
For each pair ( x , y ) ∈ M , we uniformly select K examples from M at random to construct M (cid:48) .
Choice of penalty α n .
We investigate two strategies to set the cofﬁcient α n .
The ﬁrst strategy simply sets α n to a global ﬁxed coefﬁcient α n = α 0 , where α 0 is a hyperparameter.
The second strategy on the other hand dynamically adapts α n for each training example
[Equation]
x n , y n ) according to:
[Equation]
The second strategy is motivated by the intuition that the probability assigned to a correctly paired example ( x n , y n ) by a model p θ should be higher than the average of the probabilities assigned to incorrectly paired examples ( x , y n ) for all x ∼ p ∗ ( x ) .
It is however unclear how much higher the probability of a correctly paired example p θ ( y n | x n ) should be over the reﬂective probability p reﬂ θ ( y n ) , and we do not want to increase the difference if the former is already greater than the latter.
U NSUPERVISED L EARNING WITH R EFLECTIVE L IKELIHOODS
We extend RLL to unsupervised learning.
We do so using the auto-encoding framework (Hinton &
Salakhutdinov, 2006; Vincent et al., 2008; Kingma & Ba, 2014; Rezende et al., 2014).
We ﬁrst discuss stochastic auto-encoders and posterior collapse before proposing RAE s —a family of stochastic auto-encoders that do not suffer from the posterior collapse issue.
Stochastic auto-encoders and posterior collapse.
We position ourselves in the setting where there are global parameters θ and one latent variable z for every observation y .
An observation y is generated by ﬁrst drawing a latent variable z from some prior distribution p ( z ) —that we assume ﬁxed—and then sampling y from the conditional distribution of y given z .
This conditional distribution p θ ( y | z ) is parameterized by θ .
We are concerned with learning the parameters θ in the presence of the latent variables.
Maximum likelihood corresponds to maximizing
[Equation]
The likelihood p θ ( y | z ) is represented as a powerful deep neural network that takes z as input.
This objective Eq.
12 is intractable.
Stochastic auto-encoders posit a distribution q φ ( z | y ) over the latents and either maximize (Burda et al., 2015)
[Equation]
or its lower bound (Kingma & Ba
[Equation]
Rezende et al
[Equation] [Equation]
The distribution q φ ( z | y ) is parameterized by a deep neural network that takes the observation y as input.
As discussed in several works, the objective in Eq.
14 is prone to posterior collapse.
It can be rewritten as
[Equation]
What effectively happens is that the KL term quickly collapses to zero early on during optimization, leading to an approximate posterior that does not capture the data.
The importance-weighted auto-encoder (IWAE, Burda et al., 2015) was introduced to learn more expressive approximate posteriors by directly optimizing the objective in Eq.
(13) using K samples from the approximate posterior,
[Equation]
Burda et al.
(2015) show this leads to less posterior collapse.
However, the IWAE can still lead to posterior collapse due to the ﬁnite sample dilemma of maximum likelihood discussed in Section 1.
Reﬂective auto-encoders.
We propose a new class of stochastic auto-encoders that prevent posterior collapse by maximizing the RLL .
We reconsider the original population risk functions for maximum likelihood under both paired data,
[Equation]
and its upper bound under independent data,
[Equation]
The RLL in Eq.
8 is a sample-based approximation of
[Equation]
Note this is simply the difference between the original maximum likelihood objective and the log reﬂective probability.
In the case of unsupervised learning this corresponds to
[Equation]
where we deﬁne p θ ( y | y (cid:48) ) as
[Equation]
Using this deﬁnition we rewrite the objective as
[Equation] [Equation]
The expectations in Eq.
18 are intractable.
In this paper, we propose to approximate them using importance weighting (Burda et al., 2015).
For that we deﬁne a parametric proposal distribution q φ ( z | y ) —we make the conditioning on y explicit to account for recognition networks as proposal distributions.
We now write the different expectations as
[Equation]
where z k
[Equation]
q
[Equation]
z | y )
[Equation]
and E y
[Equation]
∼ p
[Equation]
E
[Equation] [Equation]
(cid:32)
[Equation]
where z k ∼ q φ ( z | y (cid:48) ) .
The expensive expectation over p ∗ ( y (cid:48) ) is dealt with as before by using a small random subset of training examples.
The importance weights ω ( y , z k ) and v ( y (cid:48) , z k ) are computed as
[Equation]
and
[Equation]
Maximizing the objective in Eq.
18 encourages the proposal q φ to output a distinct approximate posterior distribution for each input y .
We conjecture this helps avoid posterior collapse.
We verify this later in our empirical study.
C ONNECTIONS TO RANKING
Collobert et al.
(2011) proposed a pair-wise ranking loss for natural language processing.
This ranking loss is deﬁned as
[Equation]
(19) where f θ is a parametric scoring function, and s pos and s neg are positive and negative examples respectively.
s pos is often simply a training example such as a sentence, whereas s neg is a sample from a distribution conditioned on s pos .
The hyperparameter m is the margin.
Consider the RLL objective in Eq.
Consider the choice of α n in Eq.
Using stochastic approximation with a single sample ( x , y ) for the likelihood term and a single sample ( x (cid:48) , y ) for the reﬂective probability we can rewrite the objective ˜ L strong RLL ( θ ; D ) as
[Equation]
This reveals that RLL is a convex sum between the usual maximum likelihood objective and the ranking loss in Eq.
(19), modulated by the global coefﬁcient α 0 .
It directly tackles the ﬁnite sample dilemma discussed in Section 1 by maximizing the likelihood while ensuring that the correctly paired examples are better scored than incorrectly paired examples.
This in turn maximizes the dependence between the inputs and outputs.
A PPLICATIONS
In this section we apply our proposed method to two different problems: MNIST digit classiﬁcation under imbalance and neural topic modeling.
In classiﬁcation under imbalance it is hard to learn features of rare classes.
Because RLL promotes a stronger dependence between inputs and outputs we expect it to perform better than the maximum likelihood objective in the presence of imbalance.
We also consider neural topic models that use stochastic auto-encoders.
Posterior collapse is particularly severe in text modeling.
Neural topic models suffer from this problem (Srivastava & Sutton, 2017).
One manifestation of posterior collapse in neural topic modeling is that all the dimensions of the topic matrix collapse to the same topic which in turn contains words that are not necessarily related to each other (Miao et al., 2016; Srivastava & Sutton, 2017).
On both applications we found
RLL yields better performance both quantitatively and qualitatively.
It learns more useful features for rare classes in classiﬁcation under imbalance on MNIST as evidenced by higher F1 scores (See ﬁg. 1.)
Finally it learns more meaningful latent variables as evidenced by lower perplexity on document completion (See Table 2.)
We also found that the choice of α n is dependent on the application.
For classiﬁcation we found a ﬁxed schedule for alpha to perform best.
For neural topic modeling we found an adaptive schedule
[Figure]
[Table]
5. C LASSIFICATION UNDER C LASS I MBALANCE
We start with MNIST which has 5,000 training examples per class, totalling 50,000 training examples.
We refer to this original dataset as D 0 .
We build 9 additional training sets.
Each additional training set D k leaves only 5 training examples for each class j < k .
See Table 3 in the Appendix for all the class distributions.
The validation set in each case is constructed by collecting all the data points that are left from the training set.
We train a classiﬁer on each of these constructed training sets and test it on both the original test set , in which all the classes are uniformly represented
[Table]
and the matched test set , in which the class distribution matches that of the corresponding training set.
We measure both F1 score and accuracy for each setting
[Table]
As a classiﬁer
[Table]
we use a deep convolutional network (LeCun et al., 1995).
It has two convolutional layers and two fully connected layers.
Each convolutional layer consists of 5 × 5 convolution with stride 1 followed by 2 × 2 max pooling and a rectiﬁed linear unit (ReLU, Glorot et al., 2011).
The output from the ﬁnal convolutional layer is ﬂattened and processed by two fully-connected layers of 50 units.
We train the classiﬁer using the Adam optimizer with dropout regularization.
We ﬁnd the number of samples to estimate the second term in RLL by maximizing accuracy on the validation set.
We found that in most cases one sample sufﬁces.
We choose α 0 to maximize accuracy on the validation set.
We found α 0 = 0 .
5 to be the best.
In Table 1, we see that the classiﬁers trained with RLL always outperform those trained with the usual maximum likelihood criterion when the class distribution is imbalanced.
This is the case both in terms of F1 score and accuracy on both the original (balanced) test set as well as the matched test set.
[Table]
[Table]
N EURAL T OPIC M ODELLING
We use the 20NewsGroup (Joachims, 1996) for topic modelling.
20NewGroup consists of 11, training and 7,531 test articles.
The vocabulary size is 2,000.
We follow the standard preprocessing steps which include tokenization and the removal of non-UTF-8 characters and English stop words (Miao et al., 2016; Srivastava & Sutton, 2017; Card et al., 2017)
[Table]
We train a neural topic model, which is an extension of latent Dirichlet allocation (LDA, Blei et al., 2003) with amortized inference.
This is effectively a stochastic autoencoder with a bag-of-word input and output.
The inference network q φ of this auto-encoder is a two-layer feed-forward neural network with 100 tanh units each.
The output of this network is projected to a 15 -dimensional vector normalized by softmax to compute the topic proportion.
The likelihood p θ is a three-layer feedforward neural network that maps this topic proportion to a bag-of-words.
Our architecture choice is based on the earlier observation by Srivastava & Sutton (2017) that this model is particularly prone to posterior collapse.
We test both the VAE (see Eq.
and the IWAE (see Eq.
(16)) as baselines.
These are two auto-encoders trained with different forms of the maximum likelihood objective.
We use Adam (Kingma & Ba, 2014) and train each model for 200 epochs with a ﬁxed learning rate of 0 . 002 .
For RLL , we use 5 samples to estimate the reﬂective probability.
We choose an adaptive schedule for α n .
We measure both perplexity and the KL divergence between the estimated posterior distribution q φ and the prior distribution p ( z ) =
N ( z ; 0 , 1 2 ) .
The KL has been used as a metric for measuring posterior collapse.
For perplexity, we report both the perplexity on the test set (Full PPL) and the document completion perplexity (Completion PPL, Wallach et al., 2009).
Both types of perplexities are computed using importance sampling.
Document completion consists in holding out some words for each document in the test set to compute the topic proportions and then evaluating perplexity on the remaining words of each document using the topic proportions learned with the held out words.
It is a good way to assess the quality of the learned latent variables in topic modeling.
We held out the ﬁrst half of each document to compute the topic proportions and evaluated perplexity on the second half.
Similarly to the classiﬁcation experiment, we observe that RAE —the stochastic auto-encoder trained with RLL —always outperforms the neural topic models trained with a maximum likelihood objective (VAE and IWAE)
(see Table 2.)
In addition to having a lower perplexity, RLL yields slightly higher KL divergence scores—which suggests that RLL indeed improves upon VAE and IWAE in terms of posterior collapse.
We verify this further by looking at the learned topics; RLL the topics learned by RLL are as good as the topics learned by Latent Dirichlet Allocation ( LDA ).
This is not the case for the VAE.
IWAE also yields good topics but they are less sharp than the ones learned by
RAE .
These topics can be found in the appendix.
C ONCLUSION
In this paper we identiﬁed the ﬁnite sample dilemma when ﬁtting probabilistic models with maximum likelihood using a ﬁnite set of observations.
We conjectured this to be the cause of the input forgetting issue often encountered when ﬁtting deep probabilistic models using maximum likelihood.
We then proposed RLL —a new learning criterion that mitigates the input forgetting issue.
RLL encourages a strong dependence between observations and their conditioning variables.
This in turn results in better performance both in classiﬁcation under imbalance and neural topic modeling with stochastic auto-encoders
A PPENDIX
[Table] [Table]