S EMI -S UPERVISED L EARNING WITH N ORMALIZING
F LOWS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose Flow Gaussian Mixture Model (FlowGMM), a general-purpose method for semi-supervised learning based on a simple and principled probabilistic framework.
We approximate the joint distribution of the labeled and unlabeled data with a ﬂexible mixture model implemented as a Gaussian mixture transformed by a normalizing ﬂow.
We train the model by maximizing the exact joint likelihood of the labeled and unlabeled data.
We evaluate FlowGMM on a wide range of semi-supervised classiﬁcation problems across different data types: AG-News and Yahoo Answers text data, MNIST, SVHN and CIFAR-10 image classiﬁcation problems as well as tabular UCI datasets.
FlowGMM achieves promising results on image classiﬁcation problems and outperforms the competing methods on other types of data.
FlowGMM learns an interpretable latent representation space and allows hyper-parameter free feature visualization at real time rates.
Finally, we show that FlowGMM can be calibrated to produce meaningful uncertainty estimates for its predictions.
I NTRODUCTION
In many domains unlabeled data is plentiful, while labeled data may be scarce.
Semi-supervised learning framework leverages both labeled and unlabeled data reducing the need for expensive manual annotation.
Recently, consistency-based methods have shown outstanding performance in semisupervised image classiﬁcation (Laine & Aila, 2016;
Miyato et al., 2018; Tarvainen & Valpola, 2017;
Athiwaratkun et al., 2019; Verma et al., 2019; Berthelot et al., 2019) and are currently state-of-the-art on challenging datasets like CIFAR-10, CIFAR-100, and ImageNet.
However, these methods have not seen much application on domains other than images, where a suitable set of data perturbations to which the classiﬁer should be invariant is not known a priori.
In this paper, we employ a simple and principled probabilistic framework for semi-supervised learning.
We introduce a mixture distribution for modeling the data, where different components correspond to different classes.
We can maximize the joint likelihood of unlabeled (using the mixture) and labeled (using individual mixture components) data.
At test time, we classify the input to belong to the class corresponding to mixture component with the highest likelihood.
In order to apply this framework to complex data, we need to choose a sufﬁciently ﬂexible family of distributions in the mixture, preserving the ability to compute exact likelihoods.
We propose
Flow Gaussian Mixture Model (FlowGMM), a mixture model based on normalizing ﬂows (Dinh et al., 2014).
Each component of the mixture is modeled as a trainable invertible transformation (normalizing ﬂow) of the corresponding component of a Gaussian mixture in the latent space.
The transformation is shared among all the mixture components.
Due to invertibility, we can compute exact likelihood of the data using the change of variable formula.
We illustrate FlowGMM on a toy problem in Figure 1.
We are solving a binary semi-supervised classiﬁcation problem on the dataset shown in panel (a): the labeled data is shown with triangles colored according to their class, and unlabeled data is shown with blue circles.
We introduce a
Gaussian mixture with two components corresponding to each of the classes, shown in panel (c) in the latent space Z and an invertible transformation f .
The transformation f is then trained to map the data distribution in the data space X to the latent Gaussian mixture in the Z space, mapping the labeled data to the corresponding mixture component.
We visualize the learned transformation in panel (b), showing the positions of the images f ( x ) for all of the training data points.
The inverse
[Figure]
f − 1 of this mapping serves as a class-conditional generative model, that we visualize in panel (d).
To classify a data point x in the input space we compute its image f
( x ) in the latent space, and pick the class corresponding to the Gaussian that is closest to f ( x ) .
We visualize the decision boundary of the learned classiﬁer with a dashed line in panel (a).
Many of the previous attempts of using generative models for classiﬁcation (Salimans et al., 2016;
Nalisnick et al., 2019; Chen et al., 2019) have relied upon multitask learning, where a shared latent representation is learned for the generative model and the classiﬁer.
With the method of Chen et al.
(2019), hybrid modeling is observed to reduce performance for both tasks in the supervised case.
For GANs, Dai et al.
(2017) have pointed out that classiﬁcation performance and generative performance are in direct conﬂict: a perfect generator yields no beneﬁt to classiﬁcation performance.
FlowGMM takes a different approach, where the generative model is used directly as a Bayes classiﬁer, and in the limit of a perfect generative model the Bayes classiﬁer achieves provably optimal misclassiﬁcation rate (see e.g. Mohri et al., 2018).
FlowGMM naturally encodes the clustering principle : the decision boundary between classes must lie in the low-density region in the data space.
Indeed, in the latent space the decision boundary between two classes coincides with the hyperplane perpendicular to the line segment connecting means of the corresponding mixture components and passing through the midpoint of this line segment (assuming the components are normal distributions with identity covariance matrices); in panel (b) of Figure 1 we show the decision boundary in the latent space with a dashed line.
The density of the latent distribution near the decision boundary is low.
As the ﬂow is trained to represent data as a transformation of this latent distribution, the density near the decision boundary should also be low.
In panel (a) of Figure 1 the decision boundary indeed lies in the low-density region.
The contributions of this work are as follows:
• We propose FlowGMM, a new probabilistic classiﬁcation model based on normalizing ﬂows, that can be naturally applied to semi-supervised learning.
We evaluate FlowGMM on a range of semi-supervised classiﬁcation benchmarks including text, tabular and image data, and it works well.
FlowGMM outperforms alternative approaches on non-image data and shows promise for image data.
We propose modiﬁed consistency regularization for FlowGMM and empirically demonstrate that it substantially improves performance of the method on image classiﬁcation problems.
• We conduct a thorough empirical analysis of FlowGMM for supervised and semisupervised classiﬁcation.
One of the important features of FlowGMM is its interpretability.
To demonstrate it, we visualize the learned latent space representations for the proposed semi-supervised model and show that interpolations between data points from different classes pass through low-density regions.
We show how our classiﬁcation model can be used for optimization free feature visualization.
We also study the predictive uncertainties produced by the method and show that they can be calibrated by scaling the variances of mixture components.
B ACKGROUND : N ORMALIZING F LOWS
The normalizing ﬂow (Dinh et al., 2016) is an unsupervised model for density estimation deﬁned as an invertible mapping f : X → Z from the data space X to the latent space Z .
We can model the data distribution as a transformation f − 1 : Z → X applied to a random variable from the latent distribution z ∼ p
Z often chosen to be Gaussian.
The density of the transformed random variable x = f − 1 ( z ) is given by the change of variables formula
[Equation]
[Equation]
The mapping f is implemented as a sequence of invertible functions, parametrized by a neural network with architecture that is designed to ensure invertibility and efﬁcient computation of logdeterminants, and a set of parameters θ that can be optimized.
The model can be trained by maximizing the likelihood (equation 1) of the training data with respect to the parameters θ .
F LOW G AUSSIAN M IXTURE M ODEL (F LOW GMM)
In FlowGMM, we introduce a discrete latent variable y for the class label, y ∈ { 1 . . .
C} .
Our latent space distribution, conditioned on a given label k , is Gaussian with mean µ k and covariance Σ
k : p Z ( z |
[Equation]
|
[Equation]
k , Σ k ) .
The marginal distribution of z is then a Gaussian mixture.
When the classes are balanced, this distribution is
[Equation] [Equation]
Thus, the likelihood for labeled data is
[Equation]
and the likelihood for data with unknown label is p X ( x ) = (cid:80)
k
p X ( x | y = k ) p
( y = k ) .
If we have access to both a labeled dataset D (cid:96) and an unlabeled dataset D u , then we can train our model in a semi-supervised way to maximize the joint likelihood of the labeled and unlabeled data
[Equation]
over the parameters θ of the bijective function f , which learns a density model with a Bayes classiﬁer: given a test point x , the model predictive distribution is given by
[Equation]
We can then make predictions for a test point x with the Bayes decision rule
[Equation]
Alternatively to direct likelihood maximization, we can adapt Expectation Maximization algorithm for model training (discussed in Appendix A).
3. C ONSISTENCY R EGULARIZATION
Most of the existing state-of-the-art approaches to semi-supervised learning on image data are based on consistency regularization (Laine & Aila, 2016;
Miyato et al., 2018; Tarvainen & Valpola, 2017;
Athiwaratkun et al., 2019; Verma et al., 2019).
These methods penalize changes in network predictions with respect to input perturbations, such as random translations and horizontal ﬂips, with an additional loss term that can be computed on unlabeled data,
[Equation] [Equation]
where x (cid:48) , x (cid:48)(cid:48) are random perturbations of x , and g is the vector of probabilities over the classes.
Motivated by these methods, we introduce a simple consistency regularization term for FlowGMM.
Let y (cid:48)(cid:48) be the label predicted on image x (cid:48)(cid:48) by FlowGMM according to equation 5.
We then deﬁne the consistency loss term as the negative log likelihood of the input x (cid:48) given the label y
(cid:48)(cid:48) :
[Equation]
This loss term encourages the model to map small perturbations of the same unlabeled inputs to the same components of the Gaussian mixture distribution in the latent space.
Unlike the standard consistency loss of equation 6, the proposed loss in equation 7 takes values on the same scale as the data log likelihood (equation 4), and empirically we found it to perform better.
We refer to
FlowGMM with the consistency term as FlowGMM-cons.
The ﬁnal loss for FlowGMM-cons is then the weighted sum of the consistency loss (equation 7) and the negative log likelihood of both labeled and unlabeled data (equation 4).
R ELATED WORK
Generative models have been used extensively in semi-supervised learning.
In the work of Kingma et al.
(2014), it was shown how the likelihood model of Variational Autoencoder (Kingma & Welling, 2013) could be used for semi-supervised image classiﬁcation on datasets like MNIST and SVHN.
Xu et al.
(2017) later extended this framework to semi-supervised text classiﬁcation.
Generative
Adversarial Networks (GANs) have been employed for semi-supervised learning through multitask objective where the model learns to simultaneously discriminate generated images from real (labeled and unlabeled) images and classify labeled data.
Along with GANs and VAEs, Normalizing Flows (NF) (Dinh et al., 2014) present another major class of deep generative models.
Unlike GANs and VAEs, normalizing ﬂows can be trained using exact likelihood.
NFs admit controllable latent representations and can be sampled efﬁciently unlike auto-regressive models (Papamakarios et al., 2017; Oord et al., 2016).
Recent work (Dinh et al., 2016; Kingma & Dhariwal, 2018; Behrmann et al., 2018) demonstrated that normalizing ﬂows can produce high-ﬁdelity samples for natural image datasets.
Some normalizing ﬂow papers (such as
RealNVP (Dinh et al., 2016)) have used class-conditional sampling, where the transformation is conditioned on the class label.
To do so, they pass the class label as an input to coupling layers, conditioning the output of the ﬂow on the class.
Deep Invertible Generalized Linear Model (DIGLM, Nalisnick et al., 2019), most closely related to our work, trains a classiﬁer on the latent representation of a normalizing ﬂow to perform supervised or semi-supervised image classiﬁcation.
Our approach is principally different, as we use a mixture of Gaussians in the latent space Z and perform classiﬁcation based on class-conditional likelihoods (see equation 5), rather than training a separate classiﬁer.
One of the key advantages of our approach is the explicit encoding of clustering principle in the method and a more natural probabilistic interpretation.
E XPERIMENTS
In all experiments, we use the RealNVP normalizing ﬂow architecture.
Throughout training, Gaussian mixture parameters are ﬁxed: the means are initialized randomly from the standard normal distribution and the covariances are set to I .
See Appendix B for further discussion on GMM initialization and training.
5. S YNTHETIC DATA
We ﬁrst apply FlowGMM to a range of two-dimensional synthetic datasets, in order to get a better visual intuition for the method.
We use RealNVP architecture with 5 coupling layers, deﬁned by fully-connected shift and scale networks, each with 1 hidden layer of size 512.
In addition to the semi-supervised setting, we also trained the method only using the labeled data.
In Figure 2 we visualize the decision boundaries of the classiﬁer corresponding to FlowGMM for both of these
[Figure]
[Table]
settings on the two circles and pinwheel datasets.
On both datasets FlowGMM is able to leverage the unlabeled data to push the decision boundary to a low-density region, as expected.
On the two circles dataset the method is unable to ﬁt the data perfectly, as it is impossible to represent this dataset as an invertible continuous mapping of two Gaussians, because they are topologically distinct.
FlowGMM still produces a reasonable decision boundary and improves over the case when only labeled data is available.
We provide additional visualizations in Appendix C, Figure 4.
5. T EXT AND T ABULAR DATA
We believe that FlowGMM can be especially useful for semi-supervised learning on tabular data
Consistency-based semi-supervised methods have mostly been developed for image classiﬁcation, where the predictions of the method are regularized to be invariant to random ﬂips and translations of the image.
On tabular data, where the structure is less prominent, ﬁnding suitable transformations to apply for consistency-based methods is not-trivial.
Similarly, approaches based on GANs have mostly been developed for images.
We evaluate FlowGMM on the Hepmass and Miniboone UCI classiﬁcation datasets (previously used in Papamakarios et al.
(2017) for density estimation).
Along with standard tabular UCI datasets, we also consider text classiﬁcation on AG-News and
Yahoo Answers datasets.
Using the recent advances in transfer learning for NLP, we construct embeddings for input texts using the BERT transformer model (Devlin et al., 2018) trained on a corpus of Wikipedia articles, and then train FlowGMM and other baselines on the embeddings.
We compare FlowGMM to the graph transduction method from Zhou et al.
(2004), a Π -Model (Laine & Aila, 2016) that uses dropout perturbations, as well as supervised logistic regression and neural network that were trained on the labeled data only.
We report the results in Table 5.1, [Table] where FlowGMM beats the competing semi-supervised learning methods on each of the considered datasets. Implementation details for FlowGMM, the baselines
[Table]
and the dataset preprocessing details are listed in Appendix D.
5. I MAGE CLASSIFICATION
We next evaluate the proposed method on semi-supervised image classiﬁcation benchmarks on
CIFAR-10, MNIST and SVHN datasets.
For all the datasets, we use RealNVP (Dinh et al., 2016) architecture.
Exact implementation details are listed in the appendix
E. The supervised model is trained using the same loss (equation 4), where all the data points are labeled ( n u = 0 ).
We present the results for FlowGMM and FlowGMM-cons in Table 2.
We also report results from
DIGLM (Nalisnick et al., 2019) (which only report semi-supervised performance on MNIST and supervised performance on MNIST and SVHN) and the M1+M2 VAE model (Kingma et al., 2014).
FlowGMM outperforms M1+M2 model and performs better or on par with DIGLM.
Furthermore,
FlowGMM-cons improves over FlowGMM on all three datasets, suggesting that consistency regularization is very beneﬁcial for the proposed model when useful perturbations are available.
Following Oliver et al.
(2018), we evaluate FlowGMM-cons varying the number of labeled data points.
Speciﬁcally, we follow the setup of Kingma et al.
and train FlowGMM-cons on
MNIST with 100 , 600 , 1000 and 3000 labeled data points.
We present the results in Table 3.
FlowGMM-cons outperforms the M1+M2 model of Kingma et al.
(2014) in all the considered settings.
We note that the results presented in this section are not directly comparable with the state-of-theart methods using GANs or consistency regularization (see e.g. Laine & Aila, 2016;
Dai et al., 2017; Athiwaratkun et al., 2019; Berthelot et al., 2019), as the architecture we employ is much less powerful for classiﬁcation than the ConvNet and ResNet architectures that have been designed for classiﬁcation without the constraint of invertibility.
We believe that invertible architectures with better inductive biases for classiﬁcation (possibly like iResNet (Behrmann et al., 2018)) may help bridge this gap.
The space of images is challenging to model and while there has been substantial progress using normalizing ﬂows, there is still much ground to cover. [Table] [Table]
M ODEL A NALYSIS 6. U NCERTAINTY AND C ALIBRATION
In many machine learning applications [Table] it is crucial to understand how conﬁdent a model is in its predictions.
In classiﬁcation problems, well-calibrated models are expected to output meaningful probabilities of belonging to a particular class.
In Guo et al.
(2017), it was observed that modern deep learning models are highly overconﬁdent, however, simple temperature scaling can substantially improve model’s calibration.
In this section, we analyze the predictive uncertainties produced by the FlowGMM.
In Appendix Section F, we additionally study the ability of FlowGMM to detect out-of-domain data.
When using FlowGMM for classiﬁcation
[Table]
the class predictive probabilities are p ( y = k | x ) =
N ( x | µ k , Σ k ) / (cid:80) m N ( x | µ m , Σ m ) .
Since we initialize Gaussian mixture means randomly from the standard normal distribution and do not train them along with the ﬂow parameters (see section B), FlowGMM predictions become inherently overconﬁdent due to the curse of dimensionality.
Indeed, consider two Gaussians with means sampled independently from the standard normal µ 1 , µ 2 ∼ N (0 , I ) in D -dimensional space.
If s 1 ∼ N ( µ 1 , I ) is a sample from the ﬁrst Gaussian, then its expected squared distances to both mixture means are E (cid:2) (cid:107) s 1 − µ 1 (cid:107) 2 (cid:3)
=
D and
E (cid:2) (cid:107) s 1 − µ 2 (cid:107) 2 (cid:3)
= 3 D (for a detailed derivation see Appendix Section G.).
In high dimensional spaces, such logits would lead to hard label assignment in FlowGMM ( p ( y | x ) = 1 for exactly one class).
In fact, in the experiments we observe that FlowGMM is overconﬁdent and performs hard label assignment: predicted class probabilities are all close to either 1 or 0 .
We address this problem by learning a single scalar parameter σ 2 for all components in the Gaussian mixture (the component k will be N ( µ k , σ 2 I ) ) by minimizing the negative log likelihood on a validation set.
This way we can re-calibrate the variance of the latent GMM in a natural way.
This procedure is also equivalent to applying temperature scaling (Guo et al., 2017) to logits log N ( x | µ k , Σ k ) .
We test FlowGMM calibration on MNIST and CIFAR datasets in the supervised setting.
On MNIST we restricted the training set size to 1000 objects, since on the full dataset the model makes too few mistakes which makes evaluating calibration harder.
In Table 4, we report negative log likelihood and expected calibration error (ECE, see Guo et al.
(2017) for a description of this metric).
We can see that re-calibrating variances of the Gaussians in the mixture signiﬁcantly improves both metrics and mitigates overconﬁdence.
The effectiveness of this simple rescaling procedure suggests that the latent space distances learned by the ﬂow model are correlated with the probabilities of belonging to a particular class: the closer a datapoint is to the mean of a Gaussian in the latent space, the more likely it belongs to the corresponding class. x x ( ) vs ( a [ c , :, :] c )/ c [Figure]
6. L EARNED L ATENT R EPRESENTATIONS
We next analyze the latent representation space learned by FlowGMM.
We examine latent interpolations between members of the same class in Figure 3 (a) and between different classes in Figure 3 (b) for our MNIST FlowGMM-cons model trained with n (cid:96) = 1 k labels.
As expected, inter-class interpolations pass through regions of low-density, leading to low quality samples but intra-class interpolations do not.
These observations suggest that, as expected, the model learns to put the decision boundary in the low-density region of the data space.
In Appendix section H we present images corresponding to the means of the Gaussian mixture and class-conditional samples from FlowGMM.
Distance to Decision Boundary To explicitly test this conclusion, we compute the distribution of distances from unlabeled data to the decision boundary for FlowGMM-cons and FlowGMM
Sup trained on labeled data only.
In order to compute this distance exactly for an image x , we ﬁnd the two closest means µ (cid:48) , µ (cid:48)(cid:48) to the corresponding latent variable z = f ( x ) , and evaluate the expression d ( x [Equation] =
[Equation]
We visualize the distributions of the distances for the supervised and semi-supervised method in Figure 3 (c).
While most of the unlabeled data are far from the decision boundary for both methods, the supervised method puts a substantially larger fraction of data close to the decision boundary.
For example, the distance to the decision boundary is smaller than 5 for 1089 unlabeled data points with supervised model, but only 143 data points with FlowGMM-cons.
This increased separation suggests that FlowGMM-cons indeed pushes the decision boundary away from the data distribution in agreement with the clustering principle.
6. F EATURE V ISUALIZATION
Feature visualization has become an important tool for increasing the interpretability of neural networks.
The majority of methods rely on maximizing the activations of a given neuron, channel, or layer over a parametrization of an input image with different kinds of image regularization (Szegedy et al., 2013; Olah et al., 2017; Mahendran & Vedaldi, 2015).
These methods, while effective, require optimization and regularization hyper-parameters and iterative optimization too costly for real time interactive exploration.
Since our classiﬁcation model uses a ﬂow which is a sequence of invertible transformations f ( x ) = f : L ( x ) :
=
f L ◦ f L − 1 ◦ . . .
f 1 ( x ) , intermediate activations can be inverted directly.
This means that we can combine the methods of feature inversion and feature maximization directly by feeding in a set of input images, modifying intermediate activations
Given a set of activations in the (cid:96) th layer a (cid:96) [ c, i, j ] = f : (cid:96) ( x ) cij with channels c and spatial extent i, j , we may perturb a single neuron with
[Equation]
where δ c is a one hot vector at channel c ; and σ c is the standard deviation of the activations in channel c over the the training set and spatial locations.
This can be performed at real time rates to explore the activation parametrized by α and the location cij without any optimization or hyper-parameters.
The feature visualization of intermediate layers on CIFAR10 test images are shown in panel (d) of
Figure 3.
The given channel being visualized appears to activate on the zeroed pixels from random translations as well as the green channel, giving us insight into the workings of the model.
D ISCUSSION
We proposed FlowGMM, a natural and interpretable model for semi-supervised learning with normalizing ﬂows.
FlowGMM outperforms graph-based and consistency-based baselines on tabular data including semi-supervised text classiﬁcation with BERT embeddings.
On image classiﬁcation,
FlowGMM is not yet competitive with the state-of-the-art approaches (Athiwaratkun et al., 2019;
Verma et al., 2019), but we expect that the results could be greatly improved by using a better ﬂow architecture that has stronger inductive biases for classiﬁcation.
FlowGMM is greatly extensible and can be easily adapted to different settings.
For example, consider an extreme case of semi-supervised learning where some classes may be completely unlabeled, and the model has to identify those extra classes and cluster them separately.
An effective model that can operate in this setting could, for example, be used for automated scientiﬁc discovery: when applied to classify certain experimental data, it can identify new separate classes that were not considered by experts.
FlowGMM can be applied in this scenario: instead of the standard Gaussian mixture model (GMM) in the latent space we can use the Chinese Restaurant Process GMM (CRPGMM), which can infer the number of mixture components from data automatically (Rasmussen, 2000).
Further, we can easily adapt FlowGMM to few-shot learning setting where the model has to generalize to classes not seen at train time by only using a few examples of that new class.
In particular, we can ﬁx the parameters of the ﬂow model and add a new Gaussian to the mixture whenever we add a new class.
We can estimate the parameters of this Gaussian from the few available examples.
We view interpretability as another strong advantage of FlowGMM.
The access to latent space representations and feature visualization technique discussed in Section 6 as well as the ability to sample from the model can be used to get insights into the performance of the model in practical applications.
A E XPECTATION M AXIMIZATION
As an alternative to direct optimization of the likelihood (equation 4), we consider ExpectationMaximization algorithm (EM).
EM is a popular approach for ﬁnding maximum likelihood estimates in mixture models.
Suppose X
= { x i } ni
=1 is the observed dataset, T = { t i } ni =1 are corresponding unobserved latent variables (often denoting the component in mixture model) and θ is a vector of model parameters.
EM algorithm consists of the two alternating steps: on E-step, we compute posterior probabilities of latent variables for each data point q ( t i | x i ) =
P ( t i | x i , θ ) ; and on M-step, we ﬁx q and maximize the expected log likelihood of the data and latent variables with respect to θ : E q log P ( X, T | θ ) → max θ .
The algorithm can be easily adapted to the semi-supervised setting where a subset of data is labeled with { y li } n l i =1 : then, on E-step we have hard assignment to the true mixture component q
( t i | x i ) =
I [ t i = y li ] for labeled data points.
EM algorithm is applicable in our setting which is ﬁtting the transformed mixture of Gaussians.
We can perform the exact E-step for unlabeled data in the model since
[Equation]
k ) which coincides with the E-step of EM algorithm on Gaussian mixture model.
On M-step, the objective has the following form:
[Equation]
Since the exact solution is not tractable due to complexity of the ﬂow model, we perform a stochastic gradient step to optimize the expected log likelihood with respect to ﬂow parameters θ .
Note that unlike regular EM algorithm for mixture models, we have Gaussian mixture parameters { ( µ k , Σ k ) } C k
=1 ﬁxed in our experiments, and on M-step the update of θ induces the change of z i =
f θ ( x i ) latent space representations.
Using EM algorithm for optimization in the semi-supervised setting on MNIST dataset with 1000 labeled images, we obtain 98.97% accuracy which is comparable to the result for FlowGMM with regular SGD training.
However, in our experiments, we observed that on E-step, hard label assignment happens for unlabeled points ( q ( t | x ) ≈ 1 for one of the classes) because of the high dimensionality of the problem (see section 6.1) which affects the M-step objective and hinders training.
B L ATENT D ISTRIBUTION M EAN AND C OVARIANCE C HOICES
Initialization In our experiments, we draw the mean vectors µ i of Gaussian mixture model randomly from the standard normal distribution µ i ∼ N (0 , I ) , and set the covariance matrices to identity Σ
i =
I for all classes; we ﬁxed GMM parameters throughout training.
However, one could potentially beneﬁt from data-dependent placing of means in the latent space.
We experimented with different initialization methods, in particular, initializing means using the mean point of latent representations of labeled data in each class: µ i = (1 /n il ) (cid:80) n il m =1 f ( x im ) where x im represents labeled data points from class i and n il is the total number of labeled points in that class.
In addition, we can scale all means by a scalar value ˆ µ i =
rµ i to increase or decrease distances between them.
We observed that such initialization leads to much faster convergence of FlowGMM on semi-supervised classiﬁcation on MNIST dataset, however, the ﬁnal performance of the model was worse compared to the one with random mean placing.
We hypothesize that it becomes easier for the ﬂow model to warm up faster with data-dependent initialization because Gaussian means are closer to the initial latent representations, but afterwards the model gets stuck in a suboptimal solution.
GMM training FlowGMM would become even more ﬂexible and expressive if we could learn
Gaussian mixture parameters in a principled way.
In the current setup where means are sampled from the standard normal distribution, the distances between mixture components are about √ 2 D where
D is the dimensionality of the data (see Appendix G).
Thus, classes are quite far apart from each
T w o
C i r c l e s 8 G a u ss i a n s
P i n w h ee l f f f f −f −f −
X , Data
Z , Latent Z , Latent
X , Data
[Figure]
other in the latent space, which, as observed in Section 6.1, leads to model miscalibration.
Training
GMM parameters can further increase interpretability of the learned latent space representations: we can imagine a scenario in which some of the classes are very similar or even intersecting, and it would be useful to represent it in the latent space.
We could train GMM by directly optimizing likelihood (equation 4), or using expectation maximization (see Section A), either jointly with the ﬂow parameters or iteratively switching between training ﬂow parameters with the ﬁxed GMM and training GMM with the ﬁxed ﬂow.
In our initial experiments on semi-supervised classiﬁcation on MNIST, training GMM jointly with the ﬂow parameters did not improve performance or lead to substantial change of the latent representations.
Further improvements require careful hyperparameter choice which we leave for future work.
C S YNTHETIC E XPERIMENTS
In Figure 4 we visualize the classiﬁcation decision boundaries of FlowGMM as well as the learned mapping to the latent space and generated samples for three different synthetic datasets.
D T ABULAR DATA PREPARATION AND HYPERPARAMETERS
The AG-News and Yahoo Answers were constructed by applying BERT embeddings to the text input, yielding a 768 dimensional vector for each data point.
AG-News has 4 classes while Ya-
[Table]
hoo Answers has 10 .
The UCI datasets Hepmass and Miniboone were constructed using the data preprocessing from Papamakarios et al.
(2017), but with the inclusion of the removed background process class so that the two problems can be used for binary classiﬁcation.
We then subsample the fraction of background class examples so that the dataset is balanced.
For each of the datasets, a separate validation set of size 5 k was used to tune hyperparameters.
All neural network models use the ADAM optimizer (Kingma & Ba, 2014).
k-Nearest Neighbors
: We tested both using both L2 distance and L2 with inputs normalized to unit norm, ( sin 2 distance), and the latter performed the best.
The value k chosen in the method was found sweeping over 1 − 20 , and the optimal values for each of the datasets are shown in 5.
3 Layer NN + Dropout : The 3 -Layer NN + Dropout baseline network has three fully connected hidden layers with inner dimension k = 512 , ReLU nonlinearities, and dropout with p = 0 .
We use the learning rate 3 e − 4 for training the supervised baseline across all datasets.
Π -Model : The Π -Model uses the same network architecture, and dropout for the perturbations.
The additional consistency loss per unlabeled data point is computed as L Unlab =
|| g
( x (cid:48)(cid:48) ) − g ( x (cid:48) )
|| 2 , where g is are the output probabilities after the softmax layer of the neural network and the consistency weight λ = 30 which worked the best across the datasets.
The model was trained forepochs with labeled and unlabeled batch size n (cid:96) for AG-News and Yahoo Answers, and labeled and unlabeled batch sizes n (cid:96) and 2000 for Hepmass and Miniboone.
Label Spreading : We use the local and global consistency method from Zhou et al.
(2004), Y ∗ =
( I − αS ) − 1 Y where in our case Y is the matrix of labels for the labeled, unlabeled, and test data but ﬁlled with zeros for unlabeled and test.
S
=
D − 1 / 2 WD − 1 / 2 computed from the afﬁnity matrix W ij = exp ( − γ sin 2 ( x i , x j )) where sin 2 ( x i , x j ) :
= 1 − (cid:104) x i ,x j (cid:105) (cid:107) x i (cid:107)(cid:107) x j (cid:107) .
This is equivalent to L2 distance on the inputs normalized to unit magnitude.
Because the algorithm scales poorly with number of unlabeled points for dense afﬁnity matrices, O ( n 3 u ) , we we subsampled the number of unlabeled data points to 10 k and test data points to 5 k for this graph method.
However, we also evaluate the label spreading algorithm with a sparse kNN afﬁnity matrix on using a larger subset 20 k of unlabeled data.
The two hyperparameters for label spreading ( γ / k and α ) were tuned by separate grid search for each of the datasets. In both cases, we use the inductive variant of the algorithm where the test data is not included in the unlabeled data.
FlowGMM : We train our FlowGMM model with a RealNVP normalizing ﬂow, similar to the architectures used in Papamakarios et al.
Speciﬁcally, the model uses 7 coupling layers, with 1 hidden layer each and 256 hidden units for the UCI datasets but 512 for text classiﬁcation.
UCI models were trained for 50 epochs of unlabeled data and the text datasets were trained for 30 epochs of unlabeled data.
The labeled and unlabeled batch sizes are the same as in the Π -Model.
The tuned learning rates for each of the models that we used for these experiments are shown in table 5.
E I MAGE DATA PREPARATION AND HYPERPARAMETERS
We use the CIFAR-10 multi-scale architecture with 2 scales, each containing 3 coupling layers deﬁned by 8 residual blocks with 64 feature maps.
We use Adam optimizer (Kingma & Ba, 2014) with learning rate 10 − 3 for CIFAR-10 and SVHN and 10 − 4 for MNIST.
We train the supervised model for 100 epochs, and semi-supervised models for 1000 passes through the labeled data for CIFAR-10 and SVHN and 3000 passes for MNIST.
We use a batch size of 64 and sample 32 labeled and 32 unlabeled data points in each mini-batch.
For the consistency loss term (equation 7), we linearly increase the weight from 0 to 1 for the ﬁrst 100 epochs following Athiwaratkun et al.
For FlowGMM and FlowGMM-cons, we re-weight the loss on labeled data by λ = 3 (value tuned on validation (Kingma et al., 2014) on CIFAR-10), as otherwise, we observed that the method underﬁts the labeled data.
F O UT - OF - DOMAIN DATA DETECTION
Density models have held promise for being able to detect out-of-domain data, an especially important task for robust machine learning systems (Nalisnick et al., 2019).
Recently, it has been shown that existing ﬂow and autoregressive density models are not as apt at this task as previously thought, yielding high likelihood on images coming from other (simpler) distributions.
The conclusion put forward is that datasets like SVHN are encompassed by, or have roughly the same mean but lower variance than, more complex datasets like CIFAR10 (Nalisnick et al., 2018).
We examine this hypothesis in the context of our ﬂow model which has a multi-modal latent space distribution unlike methods considered in Nalisnick et al.
[Figure]
Using a fully supervised model trained on MNIST, we evaluate the log likelihood for data points coming from the NotMNIST dataset, consisting of letters instead of digits, and the FashionMNIST dataset.
We then train a supervised model on the more complex dataset FashionMNIST and evaluate on MNIST and NotMNIST.
The distribution of the log likelihood log p X ( · ) = log p
Z ( f ( · ))
+ log (cid:12)(cid:12)(cid:12) det (cid:16) ∂f∂x (cid:17)(cid:12)(cid:12)(cid:12) on these datasets is shown in Figure 5.
For the model trained on
MNIST we see that the data from Fashion MNIST and NotMNIST is assigned lower likelihood, as expected.
However, the model trained on FashionMNIST predicts higher likelihoods for MNIST images.
The majority ( ≈ 75% ) of the MNIST data points get mapped into the mode of the FashionMNIST model corresponding to sandals, which is the class with the largest fraction of pixels that are zero.
Similarly, for the model trained on MNIST the image of all zeros has very high likelihood and gets mapped to the mode corresponding to the digit 1 which has the largest fraction of empty space.
G E XPECTED D ISTANCES BETWEEN G AUSSIAN S AMPLES
Consider two Gaussians with means sampled independently from the standard normal µ 1 , µ 2 ∼ N (0 , I ) in D -dimensional space.
If s 1 ∼ N ( µ 1 , I ) is a sample from the ﬁrst Gaussian, then its expected squared distances to both mixture means are: [Equation]
[Equation]
For high-dimensional Gaussians the random variables (cid:107) s 1 − µ 1 (cid:107) 2 and (cid:107) s 1 − µ 2 (cid:107) 2 will be concentrated around their expectations.
Since the function exp( − x ) decreases rapidly to zero for positive x , the probability of s 1 belonging to the ﬁrst Gaussian exp( −(cid:107) s 1 − µ 1 (cid:107) 2 ) / (cid:0) exp( −(cid:107) s [Equation] + exp( −(cid:107)
s [Equation] (cid:107) 2 ) (cid:1) ≈ exp( − D ) /
(exp( − D ) + exp( − 3 D ))
= 1 / (1 + exp( − 2 D )) saturates at 1 with the growth of dimensionality D .
[Figure]
In Figure 6a we show the images f − 1 ( µ i ) corresponding to the means of the Gaussians representing each class.
We see that the ﬂow correctly learns to map the means to samples from the corresponding classes.
Next, in Figure 6b we show class-conditional samples from the model.
To produce a sample from class i , we ﬁrst generate z ∼ N ( µ i , TI ) , where T is a temperature parameter that controls trade-off between sample quality and diversity; we then compute the samples as f − 1 ( z ) .
We set
T = 0 .
25 2 to produce samples in Figure 6b.
As we can see, FlowGMM can produce reasonable class-conditional samples simultaneously with achieving a high classiﬁcation accuracy ( 99 . 63% ) on the MNIST dataset.