Original: Learning Consistent Deep Generative Models from Sparsely Labeled Data
Revised: Learning Consistent Deep Generative Models from Sparsely Labeled Data

Original: Anonymous Authors

Original: Anonymous Institution

Original: 1. Introduction
Revised: 1. Introduction

Original: We develop broadly applicable methods for learning ﬂexible models of high-dimensional data, like images, that are paired with (discrete or continuous) labels.
Revised: We develop broadly applicable methods for learning ﬂexible models of high-dimensional data, like images, that are paired with (discrete or continuous) labels.

Original: We are particularly interested in semi-supervised learning (SSL) (Zhu, 2005; Oliver et al., 2018) from data that is sparsely labeled, a common situation in practice due to the cost or privacy concerns associated with data annotation.
Revised: We are particularly interested in semi-supervised learning (SSL) (Zhu, 2005; Oliver et al., 2018) from data that is sparsely labeled, a common situation in practice due to the cost or privacy concerns associated with data annotation.

Original: Given a large and sparsely labeled dataset, we seek a single probabilistic model that simultaneously makes good predictions of labels and provides a high-quality generative model of the high-dimensional input data.
Revised: Given a large and sparsely labeled dataset, we seek a single probabilistic model that simultaneously makes good predictions of labels and provides a high-quality generative model of the high-dimensional input data.

Original: Prior approaches for semi-supervised learning of deep generative models include variational autoencoders (VAEs) (Kingma et al.
Revised: Prior approaches for semi-supervised learning of deep generative models include variational autoencoders (VAEs) (Kingma et al.

Original: (2014)), generative adversarial networks (GANs) (Dumoulin et al., 2017; Kumar et al., 2017), normalizing ﬂows (Nalisnick et al., 2019; Izmailov et al., 2020) and hybrids of these (Larsen et al., 2016; de Bem et al., 2018; Zhang et al., 2019).
Revised: (2014)), generative adversarial networks (GANs) (Dumoulin et al., 2017; Kumar et al., 2017), normalizing ﬂows (Nalisnick et al., 2019; Izmailov et al., 2020) and hybrids of these (Larsen et al., 2016; de Bem et al., 2018; Zhang et al., 2019).

Original: We favor VAEs due to their ability to learn low-dimensional representations with high predictive accuracy and their ability to evaluate a learned probability density function.
Revised: We favor VAEs due to their ability to learn low-dimensional representations with high predictive accuracy and their ability to evaluate a learned probability density function.

Original: We have three key contributions.
Revised: We have three key contributions.

Original: First, we expose the conceptual and practical deﬁciencies of SSL VAEs built on the M2 approach of Kingma et al.
Revised: First, we expose the conceptual and practical deﬁciencies of SSL VAEs built on the M2 approach of Kingma et al.

Original: Second, to address these limitations we provide a new downstream SSL framework – prediction constrained variational autoencoders (PC-VAEs) – which learns high-quality generative models while simultaneously enforcing accurate predictions.
Revised: Second, to address these limitations we provide a new downstream SSL framework – prediction constrained variational autoencoders (PC-VAEs) – which learns high-quality generative models while simultaneously enforcing accurate predictions.

Original: Third, we show that the generative model structure leads to a natural consistency constraint vital for eﬀective semi-supervised learning from very sparse labels.
Revised: Third, we show that the generative model structure leads to a natural consistency constraint vital for eﬀective semi-supervised learning from very sparse labels.

Original: Our experiments demonstrate that consistent prediction-constrained (CPC) VAE training leads to accuracy competitive with state-of-the-art SSL methods and integrates well with generative modelling improvements such as “very-deep” VAEs (Child, 2021).
Revised: Our experiments demonstrate that consistent prediction-constrained (CPC) VAE training leads to accuracy competitive with state-of-the-art SSL methods and integrates well with generative modelling improvements such as “very-deep” VAEs (Child, 2021).

Original: 2. Background: Semi-supervised VAEs
Revised: 2. Background: Semi-supervised VAEs

Original: We now describe VAEs as deep generative models and review previous methods for SSL of
Revised: We now describe VAEs as deep generative models and review previous methods for SSL of

Original: VAEs.
Revised: VAEs.

Original: SSL tasks provide two training datasets: an unsupervised (unlabeled) dataset D U of N feature vectors x and a supervised (labeled) dataset D S containing M pairs ( x, y ) of features x and label y ∈ Y .
Revised: SSL tasks provide two training datasets: an unsupervised (unlabeled) dataset D U of N feature vectors x and a supervised (labeled) dataset D S containing M pairs ( x, y ) of features x and label y ∈ Y .

Original: Labels are often very sparse ( M (cid:28) N ).
Revised: Labels are often very sparse ( M (cid:28) N ).

Original: The variational autoencoder (Kingma and Welling, 2014) is an unsupervised learning framework with two components: a generative model and an inference model.
Revised: The variational autoencoder (Kingma and Welling, 2014) is an unsupervised learning framework with two components: a generative model and an inference model.

Original: The generative model deﬁnes for each example a joint distribution p θ ( x, z ) over “features” (observed vector x ∈ R D ) and “encodings” (latent vector z ∈ R C ).
Revised: The generative model deﬁnes for each example a joint distribution p θ ( x, z ) over “features” (observed vector x ∈ R D ) and “encodings” (latent vector z ∈ R C ).

Original: The VAE “inference model” deﬁnes an approximate posterior q φ ( z | x ) , which is trained to be close to the true posterior ( q φ ( z | x )
Revised: The VAE “inference model” deﬁnes an approximate posterior q φ ( z | x ) , which is trained to be close to the true posterior ( q φ ( z | x )

Original: ≈ p
Revised: ≈ p

Original: θ ( z | x ) )
Revised: θ ( z | x ) )

Original: but easier to evaluate:
Revised: but easier to evaluate:

Original: [Equation]
Revised: [Equation]

Original: The likelihood F is often multivariate normal, where (deterministic) functions µ θ and σ θ deﬁne the mean and covariance viatrainable parameters θ .
Revised: The likelihood F is often multivariate normal, where (deterministic) functions µ θ and σ θ deﬁne the mean and covariance via parameters θ .

Original: Given x , the posterior of z is approximated as normal with mean µ φ and (diagonal) covariance σ φ .
Revised: Given x , the posterior of z is approximated as normal with mean µ φ and (diagonal) covariance σ φ .

Original: Here, φ may parameterize multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), or other (deep) neural nets.
Revised: Here, φ may parameterize multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), or other (deep) neural nets.

Original: We would ideally learn generative parameters θ by maximizing the marginal likelihood p θ ( x ) , integrating over z .
Revised: We would ideally learn generative parameters θ by maximizing the marginal likelihood p θ ( x ) , integrating over z .

Original: As this is intractable, we instead maximize a variational bound: max θ,φ [Equation] x [Equation]
Revised: As this is intractable, we instead maximize a variational bound: max θ,φ (cid:80) x ∈D L VAE ( x ; θ, φ ) , where,

Original: ( x ; θ, φ ) , where,

Original: [Equation]
Revised: [Equation]

Original: This expectation can be evaluated via Monte Carlo samples from the inference model q φ ( z | x ) .
Revised: This expectation can be evaluated via Monte Carlo samples from the inference model q φ ( z | x ) .

Original: Gradients with respect to θ, φ can be similarly estimated by the reparameterization “trick” of representing q φ ( z | x ) as a linear transformation of standard normal variables (Kingma and Welling, 2014; Rezende et al., 2014).
Revised: Gradients with respect to θ, φ can be similarly estimated by the reparameterization “trick” of representing q φ ( z | x ) as a linear transformation of standard normal variables (Kingma and Welling, 2014; Rezende et al., 2014).

Original: 2.1. Two-Stage SSL: VAE then Predict
Revised: 2.1. Two-Stage SSL: VAE then Predict

Original: VAEs may be used for SSL via a two-stage “VAE + GLM”.
Revised: VAEs may be used for SSL via a two-stage “VAE + GLM”.

Original: First, train a VAE to maximize the unsupervised likelihood (2) of all features x (both labeled D S and unlabeled D U ).
Revised: First, train a VAE to maximize the unsupervised likelihood (2) of all features x (both labeled D S and unlabeled D U ).

Original: Second,  learn a label-from-code predictor ˆ y w ( z ) that maps latent codes z to prediction scores.
Revised: Second, ﬁxing φ and using only D S , learn a label-from-code predictor ˆ y w ( z ) that maps latent codes z to prediction scores.

Original: Our experiments use a generalized linear model (GLM) with weights w trained to minimize the cross-entropy loss (cid:80) x,y ∈D S E q φ ( z | x ) [ (cid:96) S ( y, ˆ y w ( z ))] .
Revised: Our experiments use a generalized linear model (GLM) with weights w trained to minimize the cross-entropy loss (cid:80) x,y ∈D S E q φ ( z | x ) [ (cid:96) S ( y, ˆ y w ( z ))] .

Original: This second stage ﬁxes φ and uses only the small labeled dataset.

Original: 2.2. SSL of VAEs via Joint Likelihoods
Revised: 2.2. SSL of VAEs via Joint Likelihoods

Original: Motivated by limitations of two-stage SSL, Kingma et al.
Revised: Motivated by limitations of two-stage SSL, Kingma et al.

Original: (2014) proposed a VAE-inspiredmodel called “M2”  for joint generative modeling of labels y and data x .The M2generative model ﬁrst samples labels y with frequencies π , and then features x : p θ ( x, y, z ) =
Revised: (2014) proposed a VAE-inspired “M2” model for joint generative modeling of labels y and data x . M2 ﬁrst generates labels y with frequencies π , and then features x : p θ ( x, y, z ) =

Original: N ( z | 0 , I C ) ·
Revised: N ( z | 0 , I C ) ·

Original: Cat ( y | π ) · F ( x | µ θ ( y, z ) , σ θ ( y, z )) .
Revised: Cat ( y | π ) · F ( x | µ θ ( y, z ) , σ θ ( y, z )) .

Original: M2 inference sets q φ ( y, z | x ) =
Revised: M2 inference sets q φ ( y, z | x ) =

Original: q φ y | x ( y | x ) q φ z | x,y ( z | x, y ) .
Revised: q φ y | x ( y | x ) q φ z | x,y ( z | x, y ) .

Original: To train M2, Kingma et al.
Revised: To train M2, Kingma et al.

Original: (2014) maximize the likelihood of all observations:
Revised: (2014) maximize the likelihood of all observations:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Like unsupervised VAEs, Eq.
Revised: Like unsupervised VAEs, Eq.

Original: (3) and its gradients may be approximated via samples from the variational posterior.
Revised: (3) and its gradients may be approximated via samples from the variational posterior.

Original: The ﬁrst, “supervised” term bounds the feature-and-label joint likelihood: log p θ ( x, y ) ≥ L S ,
Revised: The ﬁrst, “supervised” term bounds the feature-and-label joint likelihood: log p θ ( x, y ) ≥ L S ,

Original: [Equation]
Revised: [Equation]

Original: | x,y ( z | x,y ) (cid:21) .

Original: VAE + GLM
Revised: VAE + GLM

Original: PC-VAE (2) CPC-VAE (2)
Revised: PC-VAE (2) CPC-VAE (2)

Original: M2 (C=2)
Revised: M2 (C=2)

Original: M2 (14)
Revised: M2 (14)

Original: CPC-VAE (14) 4 3 2 1 0 1 2 3 41177.9% 4 3 2 1 0 1 2 3 41178.1% 4 3 2 1 0 1 2 3 41198.4% 4 3 2 1 0 1 2 3 41198.1% 4 3 2 1 0 1 2 3 41180.6% 4 3 2 1 0 1 2 3 41198.5%
Revised: CPC-VAE (14) 4 3 2 1 0 1 2 3 41177.9% 4 3 2 1 0 1 2 3 41178.1% 4 3 2 1 0 1 2 3 41198.4% 4 3 2 1 0 1 2 3 41198.1% 4 3 2 1 0 1 2 3 41180.6% 4 3 2 1 0 1 2 3 41198.5%

Original: The second, “unsupervised” term is a variational bound for the features-only likelihood log p θ ( x ) ≥ L U , where L U =
Revised: The second, “unsupervised” term is a variational bound for the features-only likelihood log p θ ( x ) ≥ L U , where L U =

Original: E q φ ( y,z | x ) (cid:104) log p θ ( x,y,z ) q φ ( y,z | x ) (cid:105) in terms of L S is:
Revised: E q φ ( y,z | x ) (cid:104) log p θ ( x,y,z ) q φ ( y,z | x ) (cid:105) in terms of L S is:

Original: [Equation]
Revised: [Equation]

Original: M2’s prediction dilemma and heuristic ﬁx.
Revised: M2’s prediction dilemma and heuristic ﬁx.

Original: After training parameters θ, φ , we need to predict labels y given test data x . M2’s structure assumes we make predictions via the inference model’s discriminator q φ y | x ( y | x ) .
Revised: After training parameters θ, φ , we need to predict labels y given test data x . M2’s structure assumes we make predictions via the inference model’s discriminator q φ y | x ( y | x ) .

Original: However, the discriminator’s parameter φ y | x is only informed by the unlabeled data via the objective L U of (4); it is not used to compute
Revised: However, the discriminator’s parameter φ y | x is only informed by the unlabeled data via the objective L U of (4); it is not used to compute

Original: L S .
Revised: L S .

Original: We cannot expect good predictions from a parameter that does not touch labels.
Revised: We cannot expect good predictions from a parameter that does not touch labels.

Original: To partially overcome this issue, Kingma et al.
Revised: To partially overcome this issue, Kingma et al.

Original: (2014) use a weighted objective: 
Revised: (2014) use a weighted objective: max

Original: [Equation]
Revised: [Equation]

Original: This objective pushes the inference model’s discriminator q φ y | x ( y | x ) to do well on the labeled set via an extra loss term (weighted by hyperparameter α > 0 ).
Revised: This objective pushes the inference model’s discriminator q φ y | x ( y | x ) to do well on the labeled set via an extra loss term (weighted by hyperparameter α > 0 ).

Original: Weight λ > 0 further balances supervised and unsupervised terms.
Revised: Weight λ > 0 further balances supervised and unsupervised terms.

Original: 3. CPC Variational Autoencoders 3.1. Prediction Constrained VAEs
Revised: 3. CPC Variational Autoencoders 3.1. Prediction Constrained VAEs

Original: We develop a framework for jointly learning a generative model of features x and making label-given-feature predictions ˆ y ( x ) of uncompromised quality, by requiring predictions to meet a user-speciﬁed quality threshold.
Revised: We develop a framework for jointly learning a generative model of features x and making label-given-feature predictions ˆ y ( x ) of uncompromised quality, by requiring predictions to meet a user-speciﬁed quality threshold.

Original: Our prediction constrained training objective enables end-to-end estimation of all parameters while incorporating the task-speciﬁc costs relevant in evaluation (“test”) scenarios, via user-chosen loss functions.
Revised: Our prediction constrained training objective enables end-to-end estimation of all parameters while incorporating the task-speciﬁc costs relevant in evaluation (“test”) scenarios, via user-chosen loss functions.

Original: Generative model.
Revised: Generative model.

Original: Our generative model does not include labels y , only features x and encodings z .
Revised: Our generative model does not include labels y , only features x and encodings z .

Original: The joint distribution p θ ( x, z ) and inference model q φ ( z | x ) factorize as in the unsupervised VAE of Eq.
Revised: The joint distribution p θ ( x, z ) and inference model q φ ( z | x ) factorize as in the unsupervised VAE of Eq.

Original: While M2 included labels in its generative model (Kingma et al., 2014), our goals are diﬀerent: we seek to predict labels from features, but do not need other conditionals.
Revised: While M2 included labels in its generative model (Kingma et al., 2014), our goals are diﬀerent: we seek to predict labels from features, but do not need other conditionals.

Original: Label-
Revised: Label-

Original: from-feature prediction
Revised: from-feature prediction

Original: To predict labels y from features x , we sample encoding z ∼ q φ ( z | x ) from the inference model and then predict a label ˆ y w ( z ) as in the two-stage method of Sec. 2.1.
Revised: To predict labels y from features x , we sample encoding z ∼ q φ ( z | x ) from the inference model and then predict a label ˆ y w ( z ) as in the two-stage method of Sec. 2.1.

Original: By sharing latent code z , the generative model is involved in predictions.
Revised: By sharing latent code z , the generative model is involved in predictions.

Original: Li et al.
Revised: Li et al.

Original: (2019) promote the robustness properties of this downstream model structure, which corresponds to their “GBZ” architecture, but do not explore SSL.
Revised: (2019) promote the robustness properties of this downstream model structure, which corresponds to their “GBZ” architecture, but do not explore SSL.

Original: Constrained PC objective.
Revised: Constrained PC objective.

Original: Unlike two-stage models, our approach does not predict post-hoc with a previously learned generative model.
Revised: Unlike two-stage models, our approach does not predict post-hoc with a previously learned generative model.

Original: Instead, we train the predictor simultaneously with the generative model via a novel prediction-constrained (PC) objective:
Revised: Instead, we train the predictor simultaneously with the generative model via a novel prediction-constrained (PC) objective:

Original: [Equation]
Revised: [Equation]

Original: , subj.
Revised: , subj.

Original: to:
Revised: to:

Original: [Equation]
Revised: [Equation]

Original: The constraint requires that any feasible solution achieve average prediction loss less than (cid:15) > 0 on the labeled training set.
Revised: The constraint requires that any feasible solution achieve average prediction loss less than (cid:15) > 0 on the labeled training set.

Original: The loss function (cid:96) S may be ﬂexibly speciﬁed based on task-speciﬁc needs.
Revised: The loss function (cid:96) S may be ﬂexibly speciﬁed based on task-speciﬁc needs.

Original: Unconstrained PC objective.
Revised: Unconstrained PC objective.

Original: Using the KKT conditions, we deﬁne an equivalent unconstrained objective that maximizes the likelihood but penalizes inaccurate predictions:
Revised: Using the KKT conditions, we deﬁne an equivalent unconstrained objective that maximizes the likelihood but penalizes inaccurate predictions:

Original: [Equation]
Revised: [Equation]

Original: Here λ > 0 is a Lagrange multiplier chosen to ensure that the target prediction constraint is achieved; smaller loss tolerances (cid:15) require larger values of λ .
Revised: Here λ > 0 is a Lagrange multiplier chosen to ensure that the target prediction constraint is achieved; smaller loss tolerances (cid:15) require larger values of λ .

Original: This PC objective, and gradients for parameters θ, φ, w , can be estimated via Monte Carlo samples from q φ ( z | x ) .
Revised: This PC objective, and gradients for parameters θ, φ, w , can be estimated via Monte Carlo samples from q φ ( z | x ) .

Original: Justiﬁcation.
Revised: Justiﬁcation.

Original: While the PC objective of Eq.
Revised: While the PC objective of Eq.

Original: (7) may look superﬁcially similar to Eq.
Revised: (7) may look superﬁcially similar to Eq.

Original: (5), we emphasize two key diﬀerences.
Revised: (5), we emphasize two key diﬀerences.

Original: First, it couples a generative likelihood and a prediction loss via shared variational parameters φ z | x .
Revised: First, it couples a generative likelihood and a prediction loss via shared variational parameters φ z | x .

Original: This makes both generative and discriminative performance depend on the same learned encoding z .
Revised: This makes both generative and discriminative performance depend on the same learned encoding z .

Original: In contrast, the M2 objective uses a label-given-features predictor that does not share any parameters φ y | x with the supervised likelihood L S .
Revised: In contrast, the M2 objective uses a label-given-features predictor that does not share any parameters φ y | x with the supervised likelihood L S .

Original: Our approach may coherently use any amount of labels (from very sparse to fully-labeled) to inform the generative model, while fully-labeled M2 decouples the generative and discriminative models.
Revised: Our approach may coherently use any amount of labels (from very sparse to fully-labeled) to inform the generative model, while fully-labeled M2 decouples the generative and discriminative models.

Original: Second, our objective is more aﬀordable: no term requires an expensive marginalization over labels (or lossy approximation to avoid this marginalization), easing applications to big unlabeled datasets and enabling learning from continuous labels.
Revised: Second, our objective is more aﬀordable: no term requires an expensive marginalization over labels (or lossy approximation to avoid this marginalization), easing applications to big unlabeled datasets and enabling learning from continuous labels.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: The major hyperparameter inﬂuencing PC training is the constraint multiplier λ ≥ 0 .
Revised: The major hyperparameter inﬂuencing PC training is the constraint multiplier λ ≥ 0 .

Original: Setting λ = 0 leads to unsupervised maximum likelihood training (or MAP training, given priors on θ ) of a classic VAE.
Revised: Setting λ = 0 leads to unsupervised maximum likelihood training (or MAP training, given priors on θ ) of a classic VAE.

Original: Setting λ = 1 and choosing a probabilistic loss − log p ( y | z ) produces a “supervised VAE” that maximizes the joint likelihood p θ ( x, y ) .
Revised: Setting λ = 1 and choosing a probabilistic loss − log p ( y | z ) produces a “supervised VAE” that maximizes the joint likelihood p θ ( x, y ) .

Original: In practice we use validation data to select the best of several candidate λ values.
Revised: In practice we use validation data to select the best of several candidate λ values.

Original: 3.2. Improved Predictions via Consistency
Revised: 3.2. Improved Predictions via Consistency

Original: While the PC objective is eﬀective given suﬃcient labeled data (see supplement), it may generalize poorly when labels are very sparse (see Fig. 1).
Revised: While the PC objective is eﬀective given suﬃcient labeled data (see supplement), it may generalize poorly when labels are very sparse (see Fig. 1).

Original: This fundamental problem arises because in the PC objective of Eq.
Revised: This fundamental problem arises because in the PC objective of Eq.

Original: (9), the parameters w of the predictor ˆ y w
Revised: (9), the parameters w of the predictor ˆ y w

Original: ( z ) are only informed by the small labeled training dataset.
Revised: ( z ) are only informed by the small labeled training dataset.

Original: Let x ∼ p θ ( · | z (cid:48) ) and ¯ x ∼ p θ ( · | z (cid:48) ) be two observations sampled from the same latent code z (cid:48) .
Revised: Let x ∼ p θ ( · | z (cid:48) ) and ¯ x ∼ p θ ( · | z (cid:48) ) be two observations sampled from the same latent code z (cid:48) .

Original: Even if the true label y of x is uncertain, we know that for this model to be useful for predictive tasks, ¯ x must have the same label as x .
Revised: Even if the true label y of x is uncertain, we know that for this model to be useful for predictive tasks, ¯ x must have the same label as x .

Original: We formalize this (and dramatically boost performance) via a consistency constraint requiring label predictions for common-code data pairs ( x, ¯ x ) to approximately match .
Revised: We formalize this (and dramatically boost performance) via a consistency constraint requiring label predictions for common-code data pairs ( x, ¯ x ) to approximately match .

Revised: [Table]

Original: Given x , we predict labels ˆ y w ( z ) via codes z ∼ q φ ( z | x ) .
Revised: Given x , we predict labels ˆ y w ( z ) via codes z ∼ q φ ( z | x ) .

Original: Alternatively, given x we can simulate alternative features ¯ x with matching code z by sampling from the inference and generative models, and then predict the label for ¯ x .
Revised: Alternatively, given x we can simulate alternative features ¯ x with matching code z by sampling from the inference and generative models, and then predict the label for ¯ x .

Original: We force the label predictions y for x , and ¯ y for ¯ x , to be similar via a (cross-entropy) consistency penalty (cid:96) C ( y, ¯ y ) .
Revised: We force the label predictions y for x , and ¯ y for ¯ x , to be similar via a (cross-entropy) consistency penalty (cid:96) C [Equation] y, ¯ y ) .

Original: We constrain the maximum consistency violations on unlabeled and labeled data:
Revised: We constrain the maximum consistency violations on unlabeled and labeled data:

Original: [Equation]
Revised: [Equation]

Original: Consistent PC: Unconstrained objective.
Revised: Consistent PC: Unconstrained objective.

Original: To train parameters, we use multiplier γ >to enforce consistency constraints for both unlabeled and labeled features , yielding the objective:
Revised: To train parameters, we use multiplier γ >to enforce consistency constraints for both unlabeled and labeled features , yielding the objective:

Original: [Equation]
Revised: [Equation]

Original: where L is the unsupervised likelihood bound, P is the predictor loss, and C U , C S are the consistency costs.
Revised: where L is the unsupervised likelihood bound, P is the predictor loss, and C U , C S are the consistency costs.

Original: Aggregate label consistency.
Revised: Aggregate label consistency.

Original: We further regularize predictions with an aggregate label consistency constraint, which forces the distribution of label predictions to be close to a target distribution π (typically, the empirical distribution of D S ).
Revised: We further regularize predictions with an aggregate label consistency constraint, which forces the distribution of label predictions to be close to a target distribution π (typically, the empirical distribution of D S ).

Original: This discourages predictions on unlabeled examples from collapsing to a single value.
Revised: This discourages predictions on unlabeled examples from collapsing to a single value.

Original: We deﬁne the aggregate consistency loss as (cid:96) A ( π, E x ∼D U ,z ∼ q ( z | x ) [ˆ y w ( z )]) , and again use a cross-entropy penalty.
Revised: We deﬁne the aggregate consistency loss as (cid:96) A ( π, E x ∼D U ,z ∼ q ( z | x ) [ˆ y w ( z )]) , and again use a cross-entropy penalty.

Original: Generative Model Innovations By design, our CPC-VAE may boost SSL performance by incorporating advances in generative models.
Revised: Generative Model Innovations By design, our CPC-VAE may boost SSL performance by incorporating advances in generative models.

Original: Our experiments explore three improvements to the basic VAE: noise-robust pixel likelihoods, aﬃne transformations for poorly-aligned data, and “very deep” VAEs with many stochastic layers (Child, 2021).
Revised: Our experiments explore three improvements to the basic VAE: noise-robust pixel likelihoods, aﬃne transformations for poorly-aligned data, and “very deep” VAEs with many stochastic layers (Child, 2021).

Original: More details about all these innovations are in the supplement.
Revised: More details about all these innovations are in the supplement.

Original: 4. Experiments
Revised: 4. Experiments

Original: We compare our consistent prediction-constrained (CPC) VAE to baselines on two goals: useful generative modeling of images x and classiﬁcation accuracy of y given x . 
Revised: We compare our consistent prediction-constrained (CPC) VAE to baselines on two goals: useful generative modeling of images x and classiﬁcation accuracy of y given x . For our meth- [Figure] [Figure]

Revised: D i s c

Original: For our methods, we use encoder/decoder networks based on the WRN-28-2 architecture (Zagoruyko and
Revised: r i m i n a t i v e n e t w o r k a cc u r a c y M2CCVAESHOT-VAE [Figure]

Revised: images

Revised: [Figure]

Revised: GLM

Revised: [Figure]

Revised: ods, we use encoder/decoder networks based on the WRN-28-2 architecture (Zagoruyko and

Original: Komodakis, 2016) and train with Adam (Kingma and Ba, 2014) using balanced minibatches of 50% labeled and 50% unlabeled data.
Revised: Komodakis, 2016) and train with Adam (Kingma and Ba, 2014) using balanced minibatches of 50% labeled and 50% unlabeled data.

Original: Hyperparameter search used Optuna (Akiba et al., 2019) to maximize validation accuracy.
Revised: Hyperparameter search used Optuna (Akiba et al., 2019) to maximize validation accuracy.

Original: Reproducible details are in the supplement.
Revised: Reproducible details are in the supplement.

Original: Datasets.
Revised: Datasets.

Original: For the CelebA dataset (Liu et al., 2015) with 1000 labeled and 159,770 unlabeled images, we predict 4 classes that combine gender (woman/man) and facial expression (neutral/smiling).
Revised: For the CelebA dataset (Liu et al., 2015) with 1000 labeled and 159,770 unlabeled images, we predict 4 classes that combine gender (woman/man) and facial expression (neutral/smiling).

Original: We also test SVHN (Netzer et al., 2011) and MNIST (LeCun et al., 2010)
Revised: We also test SVHN (Netzer et al., 2011) and MNIST (LeCun et al., 2010)

Original: CPC-VAEs improve SSL classiﬁcation accuracy.
Revised: CPC-VAEs improve SSL classiﬁcation accuracy.

Original: In Tab. 1, CPC achieves the top accuracy (86.22%) among all methods (6 VAEs, 2 discrim.)
Revised: In Tab. 1, CPC achieves the top accuracy (86.22%) among all methods (6 VAEs, 2 discrim.)

Original: on CelebA.
Revised: on CelebA.

Original: On SVHN, CPC also tops the SSL VAEs while matching the non-generative VAT.
Revised: On SVHN, CPC also tops the SSL VAEs while matching the non-generative VAT.

Original: CPC also is in the top 3 of all methods on MNIST, less than [Table] [Table] [Figure]
Revised: CPC also is in the top 3 of all methods on MNIST, less than 0.4% from the leader.

Original: [Figure]

Original: [Figure]

Original: D i s c

Original: r

Original: i m i n a t i v e n e t w o r k a cc u r a c y

Original: M2CCVAESHOT

Original: [Figure]

Original: VAE

Original: [Figure]

Original: images

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: CPC-VAEs generate better images.
Revised: CPC-VAEs generate better images.

Original: Because predictive likelihoods may be an ineﬀective measure of generative model quality (Theis et al., 2016), we use a reclassiﬁcation paradigm (like Joy et al.
Revised: Because predictive likelihoods may be an ineﬀective measure of generative model quality (Theis et al., 2016), we use a reclassiﬁcation paradigm (like Joy et al.

Original: for assessment of label-informed generative models
Revised: for assessment of label-informed generative models.

Original: [Table]

Original: Fig.
Revised: Fig.

Original: 2shows that the CPC-VAE, and especially the (very) Deep CPC-VAE, generate images that are better recognized (by an independent classiﬁer) as examples of the intended class.
Revised: 2shows that the CPC-VAE, and especially the (very) Deep CPC-VAE, generate images that are better recognized (by an independent classiﬁer) as examples of the intended class.

Original: Samples and reconstructions from our Deep CPC-VAE are visually plausible (Fig. 2a).
Revised: Samples and reconstructions from our Deep CPC-VAE are visually plausible (Fig. 2a).

Original: See the supplement for visuals for other models and datasets.
Revised: See the supplement for visuals for other models and datasets.

Original: Appendix A. Code Availability
Revised: Appendix A. Code Availability

Original: Code for our methods is available for download here: https://www.dropbox.com/s/b43xcsnhs5kevue/PC-VAE-REVIEW-RELEASE.zip?dl=1 .
Revised: Code for our methods is available for download here: https://www.dropbox.com/s/b43xcsnhs5kevue/PC-VAE-REVIEW-RELEASE.zip?dl=1 .

Original: It is available for browsing here: https://anonymous.4open.science/repository/0fa4ef53-5e92-40df-ad0a-bcdab28f6df0/ pcvae/ .
Revised: It is available for browsing here: https://anonymous.4open.science/repository/0fa4ef53-5e92-40df-ad0a-bcdab28f6df0/ pcvae/ .

Original: Code for our very-deep VAE model is forthcoming.
Revised: Code for our very-deep VAE model is forthcoming.

Original: Appendix B. Methods: Very Deep CPC-VAE
Revised: Appendix B. Methods: Very Deep CPC-VAE

Original: For large-scale experiments on complex datasets such as Celeb-A, we employ a variant of the very-deep VAE proposed by (Child, 2021).
Revised: For large-scale experiments on complex datasets such as Celeb-A, we employ a variant of the very-deep VAE proposed by (Child, 2021).

Original: This model uses a ladder-VAE (Sønderby et al., 2016) structure that operates at progressively ﬁner-scales in the generative process.
Revised: This model uses a ladder-VAE (Sønderby et al., 2016) structure that operates at progressively ﬁner-scales in the generative process.

Original: Please refer to these works for the speciﬁcs of the model, here we focus on our modiﬁcations to this architecture.
Revised: Please refer to these works for the speciﬁcs of the model, here we focus on our modiﬁcations to this architecture.

Original: In the original very-deep VAE, the initial “bottom-up” encoder produces parameters for the variational posterior of the topmost stochastic layer q φ ( z 0 | x ) , while intermediate outputs from this network are used as ladder connections inﬂuencing corresponding variational distributions for intermediate stochastic layers.
Revised: In the original very-deep VAE, the initial “bottom-up” encoder produces parameters for the variational posterior of the topmost stochastic layer q φ ( z 0 | x ) , while intermediate outputs from this network are used as ladder connections inﬂuencing corresponding variational distributions for intermediate stochastic layers.

Original: Figure 3(left) illustrates this structure, where d K , . . .
Revised: Figure 3(left) illustrates this structure, where d K , . . .

Original: , d 1 , d 0 are (deterministic) layer outputs from the bottom up encoder network.
Revised: , d 1 , d 0 are (deterministic) layer outputs from the bottom up encoder network.

Original: We let
Revised: We let

Original: K equal the number of intermediate stochastic layers (2 for the simpliﬁed model in Fig.
Revised: K equal the number of intermediate stochastic layers (2 for the simpliﬁed model in Fig.

Original: 3, many more in our experiments).
Revised: 3, many more in our experiments).

Original: For our very-deep CPC-VAE we modify this structure by splitting the bottom-up encoder into two separate networks.
Revised: For our very-deep CPC-VAE we modify this structure by splitting the bottom-up encoder into two separate networks.

Original: We retain the ladder-structured bottom up network (parameters ξ ) to inﬂuence the approximate posteriors for intermediate layers ( q ξ ( z 1 | x ) , q ξ ( z 2 | x ) , . . . ).
Revised: We retain the ladder-structured bottom up network (parameters ξ ) to inﬂuence the approximate posteriors for intermediate layers ( q ξ ( z 1 | x ) , q ξ ( z 2 | x ) , . . . ).

Original: We do not modify the architecture of this network, but we do not use its ﬁnal output q ξ ( z 0 | x ) .
Revised: We do not modify the architecture of this network, but we do not use its ﬁnal output q ξ ( z 0 | x ) .

Original: x zzzzzzddx x zzzzzzddx dy
Revised: x zzzzzzddx x zzzzzzddx dy

Original: [Figure]
Revised: [Figure]

Original: Instead we introduce a separate encoder network for q φ ( z 0 | x ) , with parameters denoted φ .
Revised: Instead we introduce a separate encoder network for q φ ( z 0 | x ) , with parameters denoted φ .

Original: For this network we use the same WRN architecture employed in our single stochastic layer
Revised: For this network we use the same WRN architecture employed in our single stochastic layer

Original: VAE experiments.
Revised: VAE experiments.

Original: Figure 3 (right) illustrates this modiﬁed encoder structure.
Revised: Figure 3 (right) illustrates this modiﬁed encoder structure.

Original: The generative model remains unchanged, as it is not inﬂuenced by ladder connections.
Revised: The generative model remains unchanged, as it is not inﬂuenced by ladder connections.

Original: We ﬁnd that this model architecture helps encourage consistency that aﬀects the entire generative hierarchy, leading to higher test accuracy.
Revised: We ﬁnd that this model architecture helps encourage consistency that aﬀects the entire generative hierarchy, leading to higher test accuracy.

Original: Note that this change also does not aﬀect the bottom-up factorization of the variational distribution.
Revised: Note that this change also does not aﬀect the bottom-up factorization of the variational distribution.

Original: As in prior work, p ( z ) and q ( z | x ) are factorized as follows:
Revised: As in prior work, p ( z ) and q ( z | x ) are factorized as follows:

Original: [Equation]
Revised: [Equation]

Original: PC-VAE Architecture:
Revised: PC-VAE Architecture:

Original: For the PC-VAE and CPC-VAE the prediction constraint is applied only to a subset of the latent variables, speciﬁcally those of the topmost stochastic layer ( z 0 ), so that the constraint only aﬀects a small number of global latent variables.
Revised: For the PC-VAE and CPC-VAE the prediction constraint is applied only to a subset of the latent variables, speciﬁcally those of the topmost stochastic layer ( z 0 ), so that the constraint only aﬀects a small number of global latent variables.

Original: With this structure implicit label information is accessible at every scale in the generative process
Revised: With this structure implicit label information is accessible at every scale in the generative process

Original: [Equation]
Revised: [Equation]

Original: Our (constrained) PC-VAE objective becomes:
Revised: Our (constrained) PC-VAE objective becomes:

Original: [Equation]
Revised: [Equation]

Original: This design simpliﬁes the classiﬁcation structure and limits over-ﬁtting.
Revised: This design simpliﬁes the classiﬁcation structure and limits over-ﬁtting.

Original: Due to the multi-scale structure of the very-deep VAE, latent variables lower in the hierarchy are highly localized, making them less suitable as features for predicting global image classes.
Revised: Due to the multi-scale structure of the very-deep VAE, latent variables lower in the hierarchy are highly localized, making them less suitable as features for predicting global image classes.

Original: CPC-VAE Architecture: When applying our consistency constraint, we follow our assumption that z 0 should fully determine the class of an image and thus we condition our consistency reconstruction only on the topmost stochastic layer, z 0 .
Revised: CPC-VAE Architecture: When applying our consistency constraint, we follow our assumption that z 0 should fully determine the class of an image and thus we condition our consistency reconstruction only on the topmost stochastic layer, z 0 .

Original: In practice, this preserves the global representation relevant to the class label, while allowing consistency reconstructions to exhibit signiﬁcant local variations.
Revised: In practice, this preserves the global representation relevant to the class label, while allowing consistency reconstructions to exhibit signiﬁcant local variations.

Original: The distribution for "neighboring" images ¯ x assumed to have consistent class labels is deﬁned as:
Revised: The distribution for "neighboring" images ¯ x assumed to have consistent class labels is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Our corresponding supervised and unsupervised consistency losses are then deﬁned as:
Revised: Our corresponding supervised and unsupervised consistency losses are then deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Appendix C. Methods: Robust Likelihoods with Spatial Transformations C.1. Noise-Normal Likelihood
Revised: Appendix C. Methods: Robust Likelihoods with Spatial Transformations C.1. Noise-Normal Likelihood

Original: As discussed in Sec. 3.4, we use a “Noise-Normal” distribution as the pixel likelihood for many of our experiments.
Revised: As discussed in Sec. 3.4, we use a “Noise-Normal” distribution as the pixel likelihood for many of our experiments.

Original: We deﬁne this distribution to be a parameterized two-component mixture of a truncated-normal distribution and a uniform distribution.
Revised: We deﬁne this distribution to be a parameterized two-component mixture of a truncated-normal distribution and a uniform distribution.

Original: We will use ρ to denote the mixture probability of the Normal component, and µ and σ to denote the mean and standard deviation of the truncated-normal, respectively.
Revised: We will use ρ to denote the mixture probability of the Normal component, and µ and σ to denote the mean and standard deviation of the truncated-normal, respectively.

Original: The generative model (or decoder) predicts a distinct outlier probability (1 − ρ ) for each pixel.
Revised: The generative model (or decoder) predicts a distinct outlier probability (1 − ρ ) for each pixel.

Original: We assume that pixel intensities are deﬁned on the domain [ − 1 , 1] and rescale our datasets to match.
Revised: We assume that pixel intensities are deﬁned on the domain [ − 1 , 1] and rescale our datasets to match.

Original: We can write the probability density function of the Noise-Normal distribution via the standard normal
Revised: We can write the probability density function of the Noise-Normal distribution via the standard normal

Original: PDF φ ( · ) , and standard normal CDF Φ( · ) , as follows:
Revised: PDF φ ( · ) , and standard normal CDF Φ( · ) , as follows:

Original: [Equation]
Revised: [Equation]

Original: We can similarly express the cumulative distribution function of the Noise-Normal distribution as:
Revised: We can similarly express the cumulative distribution function of the Noise-Normal distribution as:

Original: [Equation]
Revised: [Equation]

Original: In order to propagate gradients through the sampling process of the noise-normal distribution, we use the implicit reparameterization gradients approach of (Figurnov et al., 2018).
Revised: In order to propagate gradients through the sampling process of the noise-normal distribution, we use the implicit reparameterization gradients approach of (Figurnov et al., 2018).

Original: Given a sample x drawn from this distribution, we compute the gradient with respect to the parameters ρ , µ , and σ as:
Revised: Given a sample x drawn from this distribution, we compute the gradient with respect to the parameters ρ , µ , and σ as:

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: When ﬁtting the parameters of this distribution using gradient descent, we enforce the constraints that ρ ∈ [0 , 1] , µ ∈ [ − 1 , 1] , and σ > 0 .
Revised: When ﬁtting the parameters of this distribution using gradient descent, we enforce the constraints that ρ ∈ [0 , 1] , µ ∈ [ − 1 , 1] , and σ > 0 .

Original: To do this, we optimize unconstrained parameters ρ ∗ , µ ∗ , σ ∗ , and then deﬁne ρ = sigmoid ( ρ ∗ ) , µ = tanh ( µ ∗ ) , and σ = softplus ( σ ∗ ) .[Figure]
Revised: To do this, we optimize unconstrained parameters ρ ∗ , µ ∗ , σ ∗ , and then deﬁne ρ = sigmoid ( ρ ∗ ) , µ = tanh ( µ ∗ ) , and σ = softplus ( σ ∗ ) .

Original: C.2. Spatial Transformer VAE
Revised: C.2. Spatial Transformer VAE

Original: Our spatial transformer VAE retains the structure of a standard VAE, but reinterprets the latent code z as two components.
Revised: Our spatial transformer VAE retains the structure of a standard VAE, but reinterprets the latent code z as two components.

Original: As described in Sec. 4.3, the ﬁrst 6 latent dimensions z t are associated with aﬃne transformation parameters capturing image translation, rotation, scaling, and shear:
Revised: As described in Sec. 4.3, the ﬁrst 6 latent dimensions z t are associated with aﬃne transformation parameters capturing image translation, rotation, scaling, and shear:

Original: • z (1) t → horizontal translation , • z (2) t → vertical translation , • z (3) t → rotation , • z (4) t → shear , • z (5) t → horizontal scale , • z (6) t → vertical scale .
Revised: • z (1) t → horizontal translation , • z (2) t → vertical translation , • z (3) t → rotation , • z (4) t → shear , • z (5) t → horizontal scale , • z (6) t → vertical scale .

Original: The remainder of the latent code, z ∗ , generates parameters for independent per-pixel likelihoods.
Revised: The remainder of the latent code, z ∗ , generates parameters for independent per-pixel likelihoods.

Original: To constrain our transformations to a ﬁxed range of plausible values, we construct
Revised: To constrain our transformations to a ﬁxed range of plausible values, we construct

Original: M t using parameters ¯ z ( i ) t = tanh ( z ( i ) t ) that are ﬁrst mapped to the interval [ − 1 , +1] , and then linearly rescaled to an appropriate range via hyperparameters α (1) , . .
Revised: M t using parameters ¯ z ( i ) t = tanh ( z ( i ) t ) that are ﬁrst mapped to the interval [ − 1 , +1] , and then linearly rescaled to an appropriate range via hyperparameters α (1) , . .

Original: , α (6) .
Revised: , α (6) .

Original: Figureillustrates that the induced prior for ¯ z ( i ) t is heaviest for extreme values, encouraging aggressive augmentation when sampling from the prior.
Revised: Figureillustrates that the induced prior for ¯ z ( i ) t is heaviest for extreme values, encouraging aggressive augmentation when sampling from the prior.

Original: The mapping function could be changed to modify this distribution for other applications.
Revised: The mapping function could be changed to modify this distribution for other applications.

Original: Given these latent transformation parameters, we deﬁne an aﬃne transformation matrix
Revised: Given these latent transformation parameters, we deﬁne an aﬃne transformation matrix

Original: M t :
Revised: M t :

Original: [Equation]
Revised: [Equation]

Revised: ) t

Original: [Equation]
Revised: [Equation]

Revised: 0 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: ·   [Equation] sin [Equation]

Original:   
Revised: )      ·   [Equation] ( α [Equation]

Original: [Equation]
Revised: [Equation]

Revised: ) ¯ z

Original:  To determine the parameters of the likelihood function for the pixel at coordinate ( i, j ) , we use the generative model (or decoder) output at the pixel ( i (cid:48) , j (cid:48) ) for which  
Revised: [Equation] [Equation] t      (17) To determine the parameters of the likelihood function for the pixel at coordinate ( i, j ) , we use the generative model (or decoder) output at the pixel ( i (cid:48) , j (cid:48) ) for which  

Original: i j  [Equation]
Revised: i j   =

Original:    i (cid:48) j (cid:48)   .
Revised: M t   i (cid:48) j (cid:48)   .

Original: This corresponds to applying horizontal and vertical scaling, followed by rotation and shear, followed by translation.
Revised: This corresponds to applying horizontal and vertical scaling, followed by rotation and shear, followed by translation.

Original: As ( i (cid:48) , j (cid:48) ) may not correspond to integer coordinates, a spatial transformer layer (Jaderberg et al., 2015) uses bilinear interpolation of the non-transformed likelihood parameters, appropriately padding the boundaries of the decoder output.
Revised: As ( i (cid:48) , j (cid:48) ) may not correspond to integer coordinates, a spatial transformer layer (Jaderberg et al., 2015) uses bilinear interpolation of the non-transformed likelihood parameters, appropriately padding the boundaries of the decoder output.

Original: For the
Revised: For the

Original: Noise-Normal distribution we independently interpolate the ρ , µ , and σ 2 parameters.
Revised: Noise-Normal distribution we independently interpolate the ρ , µ , and σ 2 parameters.

Original: Appendix D. Related work D.1. Related work on SSL of VAEs since M 
Revised: Appendix D. Related work D.1. Related work on SSL of VAEs since M

Original: Gordon and Hernández-Lobato (2020) explore SSL for VAEs by indirectly coupling discriminative and generative parameters via a joint prior.
Revised: Gordon and Hernández-Lobato (2020) explore SSL for VAEs by indirectly coupling discriminative and generative parameters via a joint prior.

Original: Such “parameter coupling” (Lasserre et al., 2006) still requires expensive sums over labels when computing likelihoods.
Revised: Such “parameter coupling” (Lasserre et al., 2006) still requires expensive sums over labels when computing likelihoods.

Original: We show that directly integrating generative parameters in predictions leads to superior performance (see Sec. 4).
Revised: We show that directly integrating generative parameters in predictions leads to superior performance (see Sec. 4).

Original: Joy et al.
Revised: Joy et al.

Original: (2021) propose the Characteristic-Capturing VAE (CCVAE) which, like M2, optimizes a lower bound on the joint probability of x and y via an upstream generative structure: p ( x, y, z )
Revised: (2021) propose the Characteristic-Capturing VAE (CCVAE) which, like M2, optimizes a lower bound on the joint probability of x and y via an upstream generative structure: p ( x, y, z )

Original: =
Revised: =

Original: p ( y ) p ( z | y ) p ( x | z ) .
Revised: p ( y ) p ( z | y ) p ( x | z ) .

Original: The CCVAE has improved class-speciﬁc generative performance, but Joy et al.
Revised: The CCVAE has improved class-speciﬁc generative performance, but Joy et al.

Original: acknowledge that it does not signiﬁcantly improve SSL accuracy.
Revised: acknowledge that it does not signiﬁcantly improve SSL accuracy.

Original: Feng et al.
Revised: Feng et al.

Original: (2021) introduce SHOT-VAE, a variant of M2 that seeks a better justiﬁcation for the questionable log q ( y | x ) term.
Revised: (2021) introduce SHOT-VAE, a variant of M2 that seeks a better justiﬁcation for the questionable log q ( y | x ) term.

Original: They apply label smoothing (Szegedy et al., 2016) to M2, producing a “smoothed” variational objective that contains a KL-divergence term incorporating log q ( y | x ) .
Revised: They apply label smoothing (Szegedy et al., 2016) to M2, producing a “smoothed” variational objective that contains a KL-divergence term incorporating log q ( y | x ) .

Original: The experiments of Feng et al.
Revised: The experiments of Feng et al.

Original: suggest that the SSL accuracy gains of SHOT-VAE are partially due to their additional inclusion of a variant of Mixup data augmentation (Zhang et al., 2017).
Revised: suggest that the SSL accuracy gains of SHOT-VAE are partially due to their additional inclusion of a variant of Mixup data augmentation (Zhang et al., 2017).

Original: The SeGMA model (Smieja et al., 2021) maps classes to components of a Gaussian mixture in the latent code space, and adapts Wasserstein autoencoders to avoid explicit representation of discrete mixture assignments.
Revised: The SeGMA model (Smieja et al., 2021) maps classes to components of a Gaussian mixture in the latent code space, and adapts Wasserstein autoencoders to avoid explicit representation of discrete mixture assignments.

Original: SeGMA may be used for classiﬁcation, but their results emphasize its ability to interpolate observations.
Revised: SeGMA may be used for classiﬁcation, but their results emphasize its ability to interpolate observations.

Original: D.2. Related Work on Constrained Learning
Revised: D.2. Related Work on Constrained Learning

Original: Imposing constraints to supervise probabilistic models is a widespread idea (Jaakkola et al., 1999; Chang et al., 2007; Mann and McCallum, 2010).
Revised: Imposing constraints to supervise probabilistic models is a widespread idea (Jaakkola et al., 1999; Chang et al., 2007; Mann and McCallum, 2010).

Original: Our PC objective can be seen as an instance of the broad family of posterior regularization (PR) techniques (Ganchev et al., 2010) for latent variable models.
Revised: Our PC objective can be seen as an instance of the broad family of posterior regularization (PR) techniques (Ganchev et al., 2010) for latent variable models.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2014) present a regularized Bayesian formulation for fully-supervised nonparametric latent variable models.
Revised: (2014) present a regularized Bayesian formulation for fully-supervised nonparametric latent variable models.

Original: We emphasize that PR is an extremely broad family of approaches; our novel constraints are crucial to the success of the
Revised: We emphasize that PR is an extremely broad family of approaches; our novel constraints are crucial to the success of the

Original: CPC-VAE, and are not similar to prior PR methods.
Revised: CPC-VAE, and are not similar to prior PR methods.

Original: Li et al.
Revised: Li et al.

Original: (2018) propose an objective related to Eq.
Revised: (2018) propose an objective related to Eq.

Original: (9) for training VAEs from fullysupervised data with a max-margin loss.
Revised: (9) for training VAEs from fullysupervised data with a max-margin loss.

Original: They also claim that an M2-like model is more eﬀective in semi-supervised tasks when labels are rare, yet lack empirical evidence for this claim (no direct comparisons).
Revised: They also claim that an M2-like model is more eﬀective in semi-supervised tasks when labels are rare, yet lack empirical evidence for this claim (no direct comparisons).

Original: In contrast, we demonstrate that with consistency constraints,
Revised: In contrast, we demonstrate that with consistency constraints,

Original: Eq.
Revised: Eq.

Original: (9) is in fact a superior semi-supervised objective for VAEs in terms of accuracy, reliability, and training speed.
Revised: (9) is in fact a superior semi-supervised objective for VAEs in terms of accuracy, reliability, and training speed.

Original: Hughes et al.
Revised: Hughes et al.

Original: (2018) also suggest a constrained objective like Eq.
Revised: (2018) also suggest a constrained objective like Eq.

Original: (9), but specialized to semi-supervised topic models .
Revised: (9), but specialized to semi-supervised topic models .

Original: Our unconstrained PC objective (7) has connections to the multi-conditional objective of McCallum et al.
Revised: Our unconstrained PC objective (7) has connections to the multi-conditional objective of McCallum et al.

Original: (2006), which was extended to deep generative models (both explicit
Revised: (2006), which was extended to deep generative models (both explicit

Original: VAEs and implicit GANs) by Kuleshov and Ermon (2017).
Revised: VAEs and implicit GANs) by Kuleshov and Ermon (2017).

Original: This prior work does not use consistency , does not present our constraint-based view of SSL, and exclusively uses implicit
Revised: This prior work does not use consistency , does not present our constraint-based view of SSL, and exclusively uses implicit

Original: GANs in SSL experiments.
Revised: GANs in SSL experiments.

Original: The DIVA model (Ilse et al., 2020) uses a similar prediction penalty for domain adaptation, but does not impose consistency constraints.
Revised: The DIVA model (Ilse et al., 2020) uses a similar prediction penalty for domain adaptation, but does not impose consistency constraints.

Original: Related work on consistency.
Revised: Related work on consistency.

Original: Recent non-generative image classiﬁers have used loss functions that encourage both accuracy and a notion of consistency on unlabeled data, such as consistency under adversarial perturbations (Miyato et al., 2019) or feature interpolations (Berthelot et al., 2019).
Revised: Recent non-generative image classiﬁers have used loss functions that encourage both accuracy and a notion of consistency on unlabeled data, such as consistency under adversarial perturbations (Miyato et al., 2019) or feature interpolations (Berthelot et al., 2019).

Original: Unsupervised Data Augmentation (UDA) (Xie et al., 2020) achieves state-of-the-art vision and text SSL classiﬁcation by enforcing label consistency on augmented perturbations of unlabeled features, but requires highly-engineered augmentation routines (e.g., image processing libraries).
Revised: Unsupervised Data Augmentation (UDA) (Xie et al., 2020) achieves state-of-the-art vision and text SSL classiﬁcation by enforcing label consistency on augmented perturbations of unlabeled features, but requires highly-engineered augmentation routines (e.g., image processing libraries).

Original: In contrast, we learn a generative model that samples features whose label predictions need to be consistent.
Revised: In contrast, we learn a generative model that samples features whose label predictions need to be consistent.

Original: Our CPC-VAE applies to new domains where augmentation routines are unavailable; the learned generator provides augmentations.
Revised: Our CPC-VAE applies to new domains where augmentation routines are unavailable; the learned generator provides augmentations.

Original: More broadly, “cycle-consistency” has improved generative adversarial learning for images (Zhu et al., 2017; Zhou et al., 2016) and biomedical data (McDermott et al., 2018).
Revised: More broadly, “cycle-consistency” has improved generative adversarial learning for images (Zhu et al., 2017; Zhou et al., 2016) and biomedical data (McDermott et al., 2018).

Original: Others have developed cycle-consistent objectives for VAEs (Jha et al., 2018) that make the encodings z consistent.
Revised: Others have developed cycle-consistent objectives for VAEs (Jha et al., 2018) that make the encodings z consistent.

Original: Miller et al.
Revised: Miller et al.

Original: (2019) consider feature-to-label prediction in VAEs and enforce consistency with reconstructed predictions on fully-labeled data.
Revised: (2019) consider feature-to-label prediction in VAEs and enforce consistency with reconstructed predictions on fully-labeled data.

Original: In contrast, our work focuses on SSL and enforces consistency in code-to-label prediction.
Revised: In contrast, our work focuses on SSL and enforces consistency in code-to-label prediction.

Original: Appendix E. Ablation study
Revised: Appendix E. Ablation study

Original: We compare variants of CPC, M2, and MMVA (Li et al., 2018) for SSL training on MNIST using a common architecture.
Revised: We compare variants of CPC, M2, and MMVA (Li et al., 2018) for SSL training on MNIST using a common architecture.

Original: We test variants of CPC without consistency, without spatial transformations, and without aggregate label loss.
Revised: We test variants of CPC without consistency, without spatial transformations, and without aggregate label loss.

Original: Ablations of M2 and MMVA try diﬀerent model capacities (number of layers) and α penalties.
Revised: Ablations of M2 and MMVA try diﬀerent model capacities (number of layers) and α penalties.

Original: [Table]
Revised: [Table]

Original: VAE + GLM Supervised VAE
Revised: VAE + GLM Supervised VAE

Original: PC-VAE
Revised: PC-VAE

Original: CPC-VAE
Revised: CPC-VAE

Original: M2 (“style” z ) 3 2 1 0 1 2 3 4 5    54.9% 4 3 2 1 0 1 2 311 66.2% 4 2 0 2 440 74.1% 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 84 22 4  81.1%    4    69.1%
Revised: M2 (“style” z ) 3 2 1 0 1 2 3 4 5    54.9% 4 3 2 1 0 1 2 311 66.2% 4 2 0 2 440 74.1% 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 84 22 4  81.1%    4    69.1%

Original: Appendix F. Visualization of latent space
Revised: Appendix F. Visualization of latent space

Original: To provide intuition for diﬀerences among model variants, Fig.
Revised: To provide intuition for diﬀerences among model variants, Fig.

Original: 5 shows encodings for VAE models of MNIST digits with latent dimension C = 2 , given only 10 labeled examples per class.
Revised: 5 shows encodings for VAE models of MNIST digits with latent dimension C = 2 , given only 10 labeled examples per class.

Original: Appendix G. CPC-VAEs as a decision network
Revised: Appendix G. CPC-VAEs as a decision network

Original: Fig.
Revised: Fig.

Original: 5 provides further intution for the CPC-VAE method by formalizing it as a decision network.
Revised: 5 provides further intution for the CPC-VAE method by formalizing it as a decision network.

Original: Appendix H. Results: Visualizations of Sampled Images from Generative Models Spatial Transformations and Consistency Constraints
Revised: Appendix H. Results: Visualizations of Sampled Images from Generative Models Spatial Transformations and Consistency Constraints

Original: To illustrate the simulated data that plays a key role in our consistency constraints, Figuresand 8 show images sampled from CPC-VAE models trained on MNIST and SVHN.
Revised: To illustrate the simulated data that plays a key role in our consistency constraints, Figuresand 8 show images sampled from CPC-VAE models trained on MNIST and SVHN.

Original: Given some real image x , we ﬁrst sample z ∼ q φ ( · | x ) , and then generate ¯ x ∼ p θ ( · | z ) .
Revised: Given some real image x , we ﬁrst sample z ∼ q φ ( · | x ) , and then generate ¯ x ∼ p θ ( · | z ) .

Original: We further show how images change as aﬃne parameters for our spatial transformer VAE are varied.
Revised: We further show how images change as aﬃne parameters for our spatial transformer VAE are varied.

Original: These parameters are not ﬁxed when applying consistency constraints, encouraging models to learn how to align images.
Revised: These parameters are not ﬁxed when applying consistency constraints, encouraging models to learn how to align images.

Original: Class-conditional sampling
Revised: Class-conditional sampling

Original: A standard VAE generates data by sampling z ∼ N (0 , I ) , and then sampling x ∼ N ( µ θ ( z ) , σ θ ( z )) , or an alternative like the Noise-Normal likelihood.
Revised: A standard VAE generates data by sampling z ∼ N (0 , I ) , and then sampling x ∼ N ( µ θ ( z ) , σ θ ( z )) , or an alternative like the Noise-Normal likelihood.

Original: For the PC-VAE or CPC-VAE, we can further sample images conditioned on a particular class label.
Revised: For the PC-VAE or CPC-VAE, we can further sample images conditioned on a particular class label.

Original: As labels are not explicitly part of the generative model, we accomplish this by sampling images that would be conﬁdently predicted as the target class.
Revised: As labels are not explicitly part of the generative model, we accomplish this by sampling images that would be conﬁdently predicted as the target class.

Original: We use a rejection sampler, repeatedly sampling z ∼ N (0 , I ) until a sample meets the criteria: p w ( y | z ) > 1 − (cid:15) , for some target threshold (cid:15) .
Revised: We use a rejection sampler, repeatedly sampling z ∼ N (0 , I ) until a sample meets the criteria: p w ( y | z ) > 1 − (cid:15) , for some target threshold (cid:15) .

Original: We typically use (cid:15) = 0 . 05 in our experiments.
Revised: We typically use (cid:15) = 0 . 05 in our experiments.

Original: Alternatively, as shown in ﬁgure 5, we can reject a fraction of the samples of each class.
Revised: Alternatively, as shown in ﬁgure 5, we can reject a fraction of the samples of each class.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 2 in the main text shows 2-dimensional latent space encodings of the MNIST dataset using several diﬀerent models.
Revised: 2 in the main text shows 2-dimensional latent space encodings of the MNIST dataset using several diﬀerent models.

Original: We provide a complementary visualization of generative models in Fig.
Revised: We provide a complementary visualization of generative models in Fig.

Original: 12, where we compare class-conditional samples for three of these models.
Revised: 12, where we compare class-conditional samples for three of these models.

Original: The unsupervised VAE’s encodings of some classes (e.g., 2’s and 4’s and 8’s and 9’s) are not separated, and samples thus frequently appear to be the wrong class.
Revised: The unsupervised VAE’s encodings of some classes (e.g., 2’s and 4’s and 8’s and 9’s) are not separated, and samples thus frequently appear to be the wrong class.

Original: Model M2 (Kingma et al., 2014) explicitly encodes the class label as a latent variable, but nevertheless many sampled images do not visually match the conditioned class.
Revised: Model M2 (Kingma et al., 2014) explicitly encodes the class label as a latent variable, but nevertheless many sampled images do not visually match the conditioned class.

Original: In contrast, for our CPC-VAE model almost all samples are easily recognized as the target class.
Revised: In contrast, for our CPC-VAE model almost all samples are easily recognized as the target class.

Original: We illustrate class-conditional samples for our CPC-VAE model of SVHN in Fig. 13, and for our CPC-VAE models of Celeb-A in Fig.
Revised: We illustrate class-conditional samples for our CPC-VAE model of SVHN in Fig. 13, and for our CPC-VAE models of Celeb-A in Fig.

Original: The SVHN samples show rich variability while clearly retaining the digit identity.
Revised: The SVHN samples show rich variability while clearly retaining the digit identity.

Original: For both the standard and very-deep CPC-VAE models of
Revised: For both the standard and very-deep CPC-VAE models of

Original: Celeb-A, the corresponding class can be easily determined from the sampled images.
Revised: Celeb-A, the corresponding class can be easily determined from the sampled images.

Original: We also see the clearly superior image detail and realism that the state-of-the-art very-deep CPC-VAE architecture provides, which leads to substantially improved reclassiﬁcation accuracy in Fig.
Revised: We also see the clearly superior image detail and realism that the state-of-the-art very-deep CPC-VAE architecture provides, which leads to substantially improved reclassiﬁcation accuracy in Fig.

Original: Appendix I. Results: Training time comparison
Revised: Appendix I. Results: Training time comparison

Original: Figure 17 below provides an empirical comparison of the average training time cost per step using the MNIST models summarized in our main paper’s
Revised: Figure 17 below provides an empirical comparison of the average training time cost per step using the MNIST models summarized in our main paper’s

Original: Table 2.
Revised: Table 2.

Original: Our CPC-VAE implementation runs both the encoder and decoder networks twice to compute the objective (once for the standard VAE loss and an additional time to compute the consistency reconstruction and prediction), thus the runtime is approximately twice that of the PC-VAE: the PC-VAE requires 38 milliseconds per training step, while the CPC-VAE requires 80.7 milliseconds.
Revised: Our CPC-VAE implementation runs both the encoder and decoder networks twice to compute the objective (once for the standard VAE loss and an additional time to compute the consistency reconstruction and prediction), thus the runtime is approximately twice that of the PC-VAE: the PC-VAE requires 38 milliseconds per training step, while the CPC-VAE requires 80.7 milliseconds.

Original: Furthermore, our empirical ﬁndings show that training M2 is more expensive than our proposed CPC-VAE in practice, which we expect given the runtime analysis in Sec. 2.
Revised: Furthermore, our empirical ﬁndings show that training M2 is more expensive than our proposed CPC-VAE in practice, which we expect given the runtime analysis in Sec. 2.

Original: The
Revised: The

Original: M2 model must run the encoder and decoder networks once per class in order to compute the loss, due to the marginalization of the labels required for the unsupervised loss in Eq.
Revised: M2 model must run the encoder and decoder networks once per class in order to compute the loss, due to the marginalization of the labels required for the unsupervised loss in Eq.

Original: This increases the runtime by a factor equivalent to the number of classes.
Revised: This increases the runtime by a factor equivalent to the number of classes.

Original: In our empirical test, we see that the training time per step is 6.7x that of the PC-VAE model, close to the 10x slowdown we would expect for the 10 digit classes of MNIST.
Revised: In our empirical test, we see that the training time per step is 6.7x that of the PC-VAE model, close to the 10x slowdown we would expect for the 10 digit classes of MNIST.

Original: In our experiments, we did not ﬁnd substantial diﬀerences in the size of networks or number of training steps needed to train each of these models eﬀectively.
Revised: In our experiments, we did not ﬁnd substantial diﬀerences in the size of networks or number of training steps needed to train each of these models eﬀectively.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Appendix J. Results: Additional classiﬁcation results
Revised: Appendix J. Results: Additional classiﬁcation results

Original: Figure 18 shows extended results on the toy half-moons dataset, including results usinglabels.
Revised: Figure 18 shows extended results on the toy half-moons dataset, including results usinglabels.

Original: These extended results demonstrate that the PC-VAE model works well when the fraction of labeled data is large enough, signiﬁcantly outperforming the 2-stage VAE+GLM model.
Revised: These extended results demonstrate that the PC-VAE model works well when the fraction of labeled data is large enough, signiﬁcantly outperforming the 2-stage VAE+GLM model.

Original: When the labeled fraction becomes extremely small, as in the 6-label case, the consistency constraint becomes necessary for good performance.
Revised: When the labeled fraction becomes extremely small, as in the 6-label case, the consistency constraint becomes necessary for good performance.

Original: We also performed experiments on the NORB dataset (LeCun et al., 2004), to compare to results in prior semi-supervised VAE works.
Revised: We also performed experiments on the NORB dataset (LeCun et al., 2004), to compare to results in prior semi-supervised VAE works.

Original: Results are shown in table 3.
Revised: Results are shown in table 3.

Original: These results
Revised: These results

Original: [Table]
Revised: [Table]

Original: further reinforce our claims that the CPC-VAE outperforms previous SSL VAEs on image tasks.
Revised: further reinforce our claims that the CPC-VAE outperforms previous SSL VAEs on image tasks.

Original: Appendix K. Results: Sensitivity to constraint multiplier hyperparameters
Revised: Appendix K. Results: Sensitivity to constraint multiplier hyperparameters

Original: We compare the test accuracy for our consistency-constrained model for MNIST over a range of values for both λ (the prediction constraint multiplier) and γ (the consistency constraint multiplier) in Figure 21.
Revised: We compare the test accuracy for our consistency-constrained model for MNIST over a range of values for both λ (the prediction constraint multiplier) and γ (the consistency constraint multiplier) in Figure 21.

Original: All runs used our best consistency-constrained model for MNIST using dense networks.
Revised: All runs used our best consistency-constrained model for MNIST using dense networks.

Original: We kept all hyperparameters identical to the previous results (see section L), changing only the value of interest for each run.
Revised: We kept all hyperparameters identical to the previous results (see section L), changing only the value of interest for each run.

Original: We see that the resulting test accuracy smoothly varies across several orders of magnitude, with the optimal result being at or near the values we chose for our experiments.
Revised: We see that the resulting test accuracy smoothly varies across several orders of magnitude, with the optimal result being at or near the values we chose for our experiments.

Original: Performance is superior to the M2 baseline model for a wide range of hyperparameter values.
Revised: Performance is superior to the M2 baseline model for a wide range of hyperparameter values.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Appendix L. Experimental Protocol
Revised: Appendix L. Experimental Protocol

Original: Here we provide details about models and experiments which did not ﬁt into the primary paper.
Revised: Here we provide details about models and experiments which did not ﬁt into the primary paper.

Original: L.1. Hyperparameter optimization
Revised: L.1. Hyperparameter optimization

Original: The hyperparameter search for all models, including the CPC-VAE and various baselines, used Optuna (Akiba et al., 2019) to achieve the best accuracy on a labeled validation set.
Revised: The hyperparameter search for all models, including the CPC-VAE and various baselines, used Optuna (Akiba et al., 2019) to achieve the best accuracy on a labeled validation set.

Original: For our 2-layer and 4-layer M2 experiments, we used our own implementation (available in our code release) and followed the hyperparameters used by the original authors.
Revised: For our 2-layer and 4-layer M2 experiments, we used our own implementation (available in our code release) and followed the hyperparameters used by the original authors.

Original: For the 4-layer variant, we tested 10 diﬀerent settings of α , ranging from 0 .
Revised: For the 4-layer variant, we tested 10 diﬀerent settings of α , ranging from 0 .

Original: 05 to 50 , reporting both the result using the original suggested value of α = 0 .
Revised: 05 to 50 , reporting both the result using the original suggested value of α = 0 .

Original: 1 and the best value from our search ( α = 10 ).
Revised: 1 and the best value from our search ( α = 10 ).

Original: For M2, we also dynamically reduced the learning rate when the validation loss plateaued.
Revised: For M2, we also dynamically reduced the learning rate when the validation loss plateaued.

Original: L.2. Network architectures
Revised: L.2. Network architectures

Original: For our PC-VAE and CPC-VAE models of the MNIST data, we use fully-connected encoder and decoder networks with two hidden layers, 1000 hidden units per layer, and softplus activation functions.
Revised: For our PC-VAE and CPC-VAE models of the MNIST data, we use fully-connected encoder and decoder networks with two hidden layers, 1000 hidden units per layer, and softplus activation functions.

Original: Like the M2 model (Kingma et al., 2014), we use a 50-dimensional latent space.
Revised: Like the M2 model (Kingma et al., 2014), we use a 50-dimensional latent space.

Original: The original M2 experiments used networks with a single hidden layer of 500 units.
Revised: The original M2 experiments used networks with a single hidden layer of 500 units.

Original: We compare this to replications with networks matching ours, as well as 4-layer networks.
Revised: We compare this to replications with networks matching ours, as well as 4-layer networks.

Original: For the SVHN and NORB datsets, we adapt the wide-residual network architecture (WRN28-2) (Zagoruyko and Komodakis, 2016) that was proposed as a standard for semi-supervised deep learning research in (Oliver et al., 2018).
Revised: For the SVHN and NORB datsets, we adapt the wide-residual network architecture (WRN28-2) (Zagoruyko and Komodakis, 2016) that was proposed as a standard for semi-supervised deep learning research in (Oliver et al., 2018).

Original: In particular, we use this architecture for our encoder with two notable changes: We replace the ﬁnal global average pooling layer with a ﬁnal dense layer that outputs means and variances for the latent space.
Revised: In particular, we use this architecture for our encoder with two notable changes: We replace the ﬁnal global average pooling layer with a ﬁnal dense layer that outputs means and variances for the latent space.

Original: We ﬁnd that this structure provides the capacity needed for accomplishing both generative and discriminative tasks with a single network.
Revised: We ﬁnd that this structure provides the capacity needed for accomplishing both generative and discriminative tasks with a single network.

Original: For the decoder network we use a "mirrored" version of this architecture, reversing the order of layer sizes used, replacing convolutions with transposed convolutions, and removing pooling layers.
Revised: For the decoder network we use a "mirrored" version of this architecture, reversing the order of layer sizes used, replacing convolutions with transposed convolutions, and removing pooling layers.

Original: We maintain the residual structure
Revised: We maintain the residual structure

Original: [Figure]
Revised: [Figure]

Original: VAE + GLM
Revised: VAE + GLM

Original: PC-VAE (2) CPC-VAE (2)
Revised: PC-VAE (2) CPC-VAE (2)

Original: M2 (C=2)
Revised: M2 (C=2)

Original: M2 (14)
Revised: M2 (14)

Original: CPC-VAE (14) 4 3 2 1 0 1 2 3 41177.9% 4 3 2 1 0 1 2 3 41178.1% 4 3 2 1 0 1 2 3 41198.4% 4 3 2 1 0 1 2 3 41198.1% 4 3 2 1 0 1 2 3 41180.6% 4 3 2 1 0 1 2 3 41198.5% 4 3 2 1 0 1 22077.9% 4 3 2 1 0 1 22098.2% 4 3 2 1 0 1 22098.4% 4 3 2 1 0 1 22098.1% 4 3 2 1 0 1 22096.4% 4 3 2 1 0 1 22098.1%
Revised: CPC-VAE (14) 4 3 2 1 0 1 2 3 41177.9% 4 3 2 1 0 1 2 3 41178.1% 4 3 2 1 0 1 2 3 41198.4% 4 3 2 1 0 1 2 3 41198.1% 4 3 2 1 0 1 2 3 41180.6% 4 3 2 1 0 1 2 3 41198.5% 4 3 2 1 0 1 22077.9% 4 3 2 1 0 1 22098.2% 4 3 2 1 0 1 22098.4% 4 3 2 1 0 1 22098.1% 4 3 2 1 0 1 22096.4% 4 3 2 1 0 1 22098.1%

Original: ( a ) t
Revised: ( a ) t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the network.
Revised: the network.

Original: Our best classiﬁcation results with this architecture were achieved with a latent space dimension of 200 for SVHN and NORB
Revised: Our best classiﬁcation results with this architecture were achieved with a latent space dimension of 200 for SVHN and NORB

Original: [Figure]
Revised: [Figure]

Original: and a dimension of 50 for MNIST and CelebA.
Revised: and a dimension of 50 for MNIST and CelebA.

Original: For CelebA we used a similar WRN structure, but with 2 fewer residual blocks per level (WRN-20-2).
Revised: For CelebA we used a similar WRN structure, but with 2 fewer residual blocks per level (WRN-20-2).

Original: This allowed for faster training on the higher resolution images, while providing good validation accuracy
Revised: This allowed for faster training on the higher resolution images, while providing good validation accuracy

Original: [Figure]
Revised: [Figure]

Original: For the "very-deep" VAE model we used the larger (WRN-28-2) wide resnet for the zencoder.
Revised: For the "very-deep" VAE model we used the larger (WRN-28-2) wide resnet for the zencoder.

Original: For the ladder encoder and decoder we retained the bottleneck block structure from (Child, 2021), but used only 30 stochastic layers.
Revised: For the ladder encoder and decoder we retained the bottleneck block structure from (Child, 2021), but used only 30 stochastic layers.

Original: Each latent representation used 16 channels.
Revised: Each latent representation used 16 channels.

Original: Each bottleneck block used an input/output width of 128 channels with a bottleneck width of 64 channels.
Revised: Each bottleneck block used an input/output width of 128 channels with a bottleneck width of 64 channels.

Original: All other details were retained from (Child, 2021).
Revised: All other details were retained from (Child, 2021).

Original: L.3. Additional regularization
Revised: L.3. Additional regularization

Original: In developing and evaluating our CPC model we explored several common regularizers taken from deep semi-supervised learning and VAE literature.
Revised: In developing and evaluating our CPC model we explored several common regularizers taken from deep semi-supervised learning and VAE literature.

Original: The SVHN results in Table 1 in the main text included these terms as part of the training loss (with the weighting speciﬁed in the hyperparameter summary below).
Revised: The SVHN results in Table 1 in the main text included these terms as part of the training loss (with the weighting speciﬁed in the hyperparameter summary below).

Original: Our experiments on CelebA omitted these terms, and more recent (unpublished) testing on SVHN suggests that these additions are unnecessary for achieving good semi-supervised performance.
Revised: Our experiments on CelebA omitted these terms, and more recent (unpublished) testing on SVHN suggests that these additions are unnecessary for achieving good semi-supervised performance.

Original: However, we lacked the time and resources to repeat our full set of replicated experiments without these regularizers, so we discuss them here for completeness.
Revised: However, we lacked the time and resources to repeat our full set of replicated experiments without these regularizers, so we discuss them here for completeness.

Original: Beta-VAE.
Revised: Beta-VAE.

Original: As an additional form of regularization for our model, we allow our hyperparameter optimization to adjust a weight on the KL-divergence term in the variational lower bound, which we call β as in previous work (Higgins et al., 2017):
Revised: As an additional form of regularization for our model, we allow our hyperparameter optimization to adjust a weight on the KL-divergence term in the variational lower bound, which we call β as in previous work (Higgins et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: This allows us to encourage q φ ( z | x ) to more closely conform to the prior, which may be necessary to balance the scale of the objective, depending on the likelihoods used and the dimensionality of the dataset.
Revised: This allows us to encourage q φ ( z | x ) to more closely conform to the prior, which may be necessary to balance the scale of the objective, depending on the likelihoods used and the dimensionality of the dataset.

Original: Prediction model regularization.
Revised: Prediction model regularization.

Original: We add two standard regularization terms to the prediction model used in our constraint, ˆ y w ( y | z ) .
Revised: We add two standard regularization terms to the prediction model used in our constraint, ˆ y w ( y | z ) .

Original: The ﬁrst is an (cid:96) 2 regularizer on the regression weights, || w || 22 , to help reduce overﬁtting.
Revised: The ﬁrst is an (cid:96) 2 regularizer on the regression weights, || w || 22 , to help reduce overﬁtting.

Original: The second is an entropy penalty.
Revised: The second is an entropy penalty.

Original: As ˆ y w ( z ) deﬁnes a categorical distribution over labels, we compute this as: − E ˆ y w ( y | z ) [log ˆ y w ( y | z )] , which has been shown to be helpful for semi-supervised learning in (Grandvalet and
Revised: As ˆ y w ( z ) deﬁnes a categorical distribution over labels, we compute this as: − E ˆ y w ( y | z ) [log ˆ y w ( y | z )] , which has been shown to be helpful for semi-supervised learning in (Grandvalet and

Original: Bengio, 2004) and was used as part of the standardized training framework of (Oliver et al., 2018).
Revised: Bengio, 2004) and was used as part of the standardized training framework of (Oliver et al., 2018).

Original: We allowed our hyperparameter optimization approach to select appropriate weights for both terms.
Revised: We allowed our hyperparameter optimization approach to select appropriate weights for both terms.

Original: L.4. Image pre-processing
Revised: L.4. Image pre-processing

Original: For all of our image datasets, we rescale the inputs to the range [-1, 1].
Revised: For all of our image datasets, we rescale the inputs to the range [-1, 1].

Original: For our NORB classiﬁcation results, we downsample each image to 48x48 pixels.
Revised: For our NORB classiﬁcation results, we downsample each image to 48x48 pixels.

Original: For our SVHN classiﬁcation results, we convert images to greyscale to reduce the representational load on our generative model.
Revised: For our SVHN classiﬁcation results, we convert images to greyscale to reduce the representational load on our generative model.

Original: Before the grayscale conversion, we apply contrast normalization to better disambiguate the colors within each image.
Revised: Before the grayscale conversion, we apply contrast normalization to better disambiguate the colors within each image.

Original: For the SVHN and NORB results, we follow the recommendation of a recent survey of semi-supervised learning methods (Oliver et al., 2018) and apply a single data augmentation technique: random translations by up to 2-pixels in each direction.
Revised: For the SVHN and NORB results, we follow the recommendation of a recent survey of semi-supervised learning methods (Oliver et al., 2018) and apply a single data augmentation technique: random translations by up to 2-pixels in each direction.

Original: For generative results, we retained the original color images and trained with full labels.
Revised: For generative results, we retained the original color images and trained with full labels.

Original: For the CelebA dataset we use the aligned images, ﬁrst cropping images to be square, then rescaling to 96x96 and ﬁnally taking a center crop of 64x64 pixels to remove most of the background.
Revised: For the CelebA dataset we use the aligned images, ﬁrst cropping images to be square, then rescaling to 96x96 and ﬁnally taking a center crop of 64x64 pixels to remove most of the background.

Original: We retained color, but applied contrast normalization as with SVHN.
Revised: We retained color, but applied contrast normalization as with SVHN.

Original: No data augmentation was used for this dataset.
Revised: No data augmentation was used for this dataset.

Original: L.5. Likelihoods
Revised: L.5. Likelihoods

Original: For all experiments on the half-moon toy data, we used a normal likelihood.
Revised: For all experiments on the half-moon toy data, we used a normal likelihood.

Original: For all of our image datasets except CelebA, we use the Noise-Normal likelihood for our CPC methods.
Revised: For all of our image datasets except CelebA, we use the Noise-Normal likelihood for our CPC methods.

Original: For our implementation of M2 for extensive experiments on MNIST we retained the
Revised: For our implementation of M2 for extensive experiments on MNIST we retained the

Original: Bernoulli likelihood used by the original authors (Kingma et al., 2014).
Revised: Bernoulli likelihood used by the original authors (Kingma et al., 2014).

Original: That is, we rescaling each pixel’s numerical intensity value to the unit interval [0,1], and then sampled binary values from a Bernoulli with probability equal to the intensity.
Revised: That is, we rescaling each pixel’s numerical intensity value to the unit interval [0,1], and then sampled binary values from a Bernoulli with probability equal to the intensity.

Original: For our experiments on CelebA, we found that the Noise-Normal likelihood was unnecessary.
Revised: For our experiments on CelebA, we found that the Noise-Normal likelihood was unnecessary.

Original: Instead we employed a Normal likelihood, using the formulation suggested by (Rybkin et al., 2020), which trains a single, global variance parameter, σ ∗ θ , that is shared across all images/pixels: x ∼ N ( µ θ ( z ) , σ ∗ θ ) .
Revised: Instead we employed a Normal likelihood, using the formulation suggested by (Rybkin et al., 2020), which trains a single, global variance parameter, σ ∗ θ , that is shared across all images/pixels: x ∼ N ( µ θ ( z ) , σ ∗ θ ) .

Original: This likelihood was used for both the standard and very-deep VAEs trained on Celeb-A.
Revised: This likelihood was used for both the standard and very-deep VAEs trained on Celeb-A.

Original: L.6. Optimization with Minibatches that Balance Labeled and Unlabeled Sets
Revised: L.6. Optimization with Minibatches that Balance Labeled and Unlabeled Sets

Original: All models were trained using minibatch stochastic gradient descent via the ADAM (Kingma and Ba, 2014) optimizer.
Revised: All models were trained using minibatch stochastic gradient descent via the ADAM (Kingma and Ba, 2014) optimizer.

Original: Batch sizes were 100 for MNIST, 64 for SVHN/NORB and 32 for
Revised: Batch sizes were 100 for MNIST, 64 for SVHN/NORB and 32 for

Original: CelebA.
Revised: CelebA.

Original: We decayed the learning rate according the cosine decay (Loshchilov and Hutter, 2016) strategy (without restarts).
Revised: We decayed the learning rate according the cosine decay (Loshchilov and Hutter, 2016) strategy (without restarts).

Original: Importantly, each minibatch was engineered to contain exactly 50% labeled and 50% unlabeled samples.
Revised: Importantly, each minibatch was engineered to contain exactly 50% labeled and 50% unlabeled samples.

Original: Several prior works in semi-supervised deep learning (Kingma et al., 2014;
Revised: Several prior works in semi-supervised deep learning (Kingma et al., 2014;

Original: Oliver et al., 2018) have employed a balanced stochastic gradient optimization approach where training batches are selected to have equal numbers of labeled and unlabeled examples.
Revised: Oliver et al., 2018) have employed a balanced stochastic gradient optimization approach where training batches are selected to have equal numbers of labeled and unlabeled examples.

Original: This prevents instances where batches have no labeled examples and reduces the variance of stochastic training.
Revised: This prevents instances where batches have no labeled examples and reduces the variance of stochastic training.

Original: We can implement a similar scheme for the consistency-constrained VAE without changing the expectation of the objective
Revised: We can implement a similar scheme for the consistency-constrained VAE without changing the expectation of the objective.

Original: [Equation]

Original: Recall the (unconstrained) CPC-VAE objective from section 3:
Revised: Recall the (unconstrained) CPC-VAE objective from section 3:

Original: [Equation]
Revised: [Equation]

Original: We can rewrite an equivalent objective, separating out the supervised and unsupervised terms:
Revised: We can rewrite an equivalent objective, separating out the supervised and unsupervised terms:

Original: [Equation]
Revised: [Equation]

Original: Rewriting in terms of expectations over the labeled and unlabeled datasets, leads to a natural approach for optimizing via balanaced batches:
Revised: Rewriting in terms of expectations over the labeled and unlabeled datasets, leads to a natural approach for optimizing via balanaced batches:

Original: [Equation]
Revised: [Equation]

Original: We see that in our stochastic optimization, we can get an unbiased estimate of the objective and its gradient by sampling labeled and unlabeled batches separately.
Revised: We see that in our stochastic optimization, we can get an unbiased estimate of the objective and its gradient by sampling labeled and unlabeled batches separately.

Original: With batch size B for both labeled and unlabeled batches our estimated objective becomes:
Revised: With batch size B for both labeled and unlabeled batches our estimated objective becomes:

Original: [Equation]
Revised: [Equation]

Original: We may also normalize by an additional scale factor of 1 |D S | + |D U | to approximately remove dependence on the dataset size.
Revised: We may also normalize by an additional scale factor of [Equation] to approximately remove dependence on the dataset size.

Original: In our experiments we remove the terms |D S | and |D U | , essentially treating the labeled and unlabeled datasets as being of equal size.
Revised: In our experiments we remove the terms |D S | and |D U | , essentially treating the labeled and unlabeled datasets as being of equal size.

Original: We found this approach simpler, though we have not performed a thorough set of experiments to evaluate the practical diﬀerences.
Revised: We found this approach simpler, though we have not performed a thorough set of experiments to evaluate the practical diﬀerences.

Original: L.7. Hardware
Revised: L.7. Hardware

Original: Experiments were run on a variety of hardware systems.
Revised: Experiments were run on a variety of hardware systems.

Original: Models for SVHN and Norb were trained primarily on an Nvidia DGX-2 system with V100 GPUs.
Revised: Models for SVHN and Norb were trained primarily on an Nvidia DGX-2 system with V100 GPUs.

Original: Experiments on MNIST and Celeb-A were run on a workstation using a Titan RTX and 2080TI GPUs.
Revised: Experiments on MNIST and Celeb-A were run on a workstation using a Titan RTX and 2080TI GPUs.

Original: Additional
Revised: Additional

Original: MNIST and Celeb-A experiments were run using 2080TI GPUs rented through vast.ai.
Revised: MNIST and Celeb-A experiments were run using 2080TI GPUs rented through vast.ai.

Original: L.8. Summary of hyperparameter settings for ﬁnal results
Revised: L.8. Summary of hyperparameter settings for ﬁnal results

Original: Table 1 provides all hyperparameter settings used in our experiments.
Revised: Table 1 provides all hyperparameter settings used in our experiments.

Original: Appendix M. Dataset Details
Revised: Appendix M. Dataset Details

Original: For each dataset considered in our paper, we provide a more detailed overview of its contents and properties.
Revised: For each dataset considered in our paper, we provide a more detailed overview of its contents and properties.

Original: We comply with the stated terms of use for all listed datasets and assert that our work is for non-commercial research purposes.
Revised: We comply with the stated terms of use for all listed datasets and assert that our work is for non-commercial research purposes.

Original: M.1. MNIST
Revised: M.1. MNIST

Original: Overview.
Revised: Overview.

Original: We consider a 10-way exclusive categorization task for MNIST digits.
Revised: We consider a 10-way exclusive categorization task for MNIST digits.

Original: We use 28-by-28 pixel grayscale images.
Revised: We use 28-by-28 pixel grayscale images.

Original: Public availability.
Revised: Public availability.

Revised: We will make code to extract our version available after publication.

Revised: Data statistics

Revised: Statistics for MNIST are shown in Table 5.

Revised: M.2. SVHN

Revised: Overview.

Revised: We consider a 10-way exclusive categorization task for SVHN digits.

Revised: We use 32x32 pixel

Revised: grayscale

Revised: images.

Revised: Public availability.

Original: We will make code to extract our version available after publication.
Revised: We will make code to extract our version available after publication.

Original: Data statistics
Revised: Data statistics

Original: Statistics for MNIST are shown in Table 5.
Revised: Statistics for SVHN are shown in Table 6.

Original: Hyperparameter
Revised: Hyperparameter

Original: MNIST (100)SVHN (1000)NORB (1000)CelebA (1000)
Revised: MNIST (100)SVHN (1000)NORB (1000)CelebA (1000)

Original: Encoder/decoder 2 FC layers
Revised: Encoder/decoder 2 FC layers

Original: WRN-28WRN-28WRN-20-2 (very-deep: see L.2)
Revised: WRN-28WRN-28WRN-20-2 (very-deep: see L.2)

Original: Dense layer size 1000 units - - -
Revised: Dense layer size 1000 units - - -

Original: Network activations
Revised: Network activations

Original: Softplus
Revised: Softplus

Original: Leaky ReLU Leaky ReLU
Revised: Leaky ReLU Leaky ReLU

Original: Leaky ReLU / GeLU (very-deep)
Revised: Leaky ReLU / GeLU (very-deep)

Original: Latent dimension   50 / 16 (very-deep)
Revised: Latent dimension   50 / 16 (very-deep)

Original: Pixel likelihood
Revised: Pixel likelihood

Original: Noise-Normal Noise-Normal
Revised: Noise-Normal Noise-Normal

Original: Noise-Normal ( σ ) Normal (Rybkin et al., 2020)
Revised: Noise-Normal ( σ ) Normal (Rybkin et al., 2020)

Original: Prediction multiplier λ  150 / 25 (very-deep)
Revised: Prediction multiplier λ  150 / 25 (very-deep)

Original: Consistency multiplier γ 4.25 λ 1.25 λ 4 λ 2 λ
Revised: Consistency multiplier γ 4.25 λ 1.25 λ 4 λ 2 λ

Original: Aggregate consistency 0.1 λ 0.2 λ 0.2 λ 0.5 λ β -VAE weight 1.
Revised: Aggregate consistency 0.1 λ 0.2 λ 0.2 λ 0.5 λ β -VAE weight 1.

Original: Predictor reg.
Revised: Predictor reg.

Original: ( || w || 22 )
Revised: ( || w || 22 )

Original: Entropy reg.
Revised: Entropy reg.

Original: 0.5 λ 0.5 λ 0.5 λ ( E p w ( y | z ) [log p w ( y | z )] ) -
Revised: 0.5 λ 0.5 λ 0.5 λ ( E p w ( y | z ) [log p w ( y | z )] ) -

Original: Translation range 0.2 × W 0.2 × W 0.2 × W - ( α (1) =
Revised: Translation range 0.2 × W 0.2 × W 0.2 × W - ( α (1) =

Original: α (2) )
Revised: α (2) )

Original: Rotation range ( α (3) )
Revised: Rotation range ( α (3) )

Original: 0.4 rad 0.5 rad 0.4 radShear range ( α (4) )
Revised: 0.4 rad 0.5 rad 0.4 radShear range ( α (4) )

Original: 0.2 rad 0.2 rad 0.2 radScale range 1.
Revised: 0.2 rad 0.2 rad 0.2 radScale range 1.

Original: - ( α (5) =
Revised: - ( α (5) =

Original: α (6) )
Revised: α (6) )

Original: Optimizer
Revised: Optimizer

Original: [Table]
Revised: [Table]

Original: Learning rate
Revised: Learning rate

Original: [Table] [Table]
Revised: [Table] [Table]

Original: M.2. SVHN
Revised: M.3. NORB Overview.

Original: Overview.

Original: We consider a 10-way exclusive categorization task for SVHN digits.

Original: We use 32x32 pixel grayscale images.
Revised: We use 48x48 pixel grayscale images.

Original: Public availability.
Revised: Public availability.

Original: We will make code to extract our version available after publication.
Revised: We will make code to extract our version available after publication.

Original: Data statistics.

Original: Statistics for SVHN are shown in Table 6.

Original: [Table]
Revised: [Table]

Original: M.3. NORB Overview.

Original: We use 48x48 pixel grayscale images. Public availability. We will make code to extract our version available after publication. Data statistics.
Revised: [Table]

Original: Statistics for NORB are shown in Table 7.
Revised: Statistics for NORB are shown in Table 7.

Original: [Table]
Revised: [Table]

Original: M.4. CelebA Overview.
Revised: M.4. CelebA Overview.

Original: Labels for our verision of the CelebA dataset were generated from the provided attributes.
Revised: Labels for our verision of the CelebA dataset were generated from the provided attributes.

Original: Our dataset used 4 classes: woman/neutral face, man/neutral face, woman/smiling, man/smiling.
Revised: Our dataset used 4 classes: woman/neutral face, man/neutral face, woman/smiling, man/smiling.

Original: Public availability.
Revised: Public availability.

Original: We will make code to extract our version available after publication.
Revised: We will make code to extract our version available after publication.

Original: Data statistics.
Revised: Data statistics.

Original: Statistics for CelebA are shown in Table 8. 
Revised: Statistics for CelebA are shown in Table 8. [Table]

Original: Appendix N. Broader Impacts
Revised: Appendix N. Broader Impacts

Original: Like many models used to make predictions, ours has the potential for both beneﬁcial and not-so-beneﬁcial impact on human society.
Revised: Like many models used to make predictions, ours has the potential for both beneﬁcial and not-so-beneﬁcial impact on human society.

Original: For any work that uses our models to make real-world predictions, we recommend that users give thought about how even simple natural image datasets may be biased towards certain regions and cultures (DeVries et al., 2019;[Table]
Revised: For any work that uses our models to make real-world predictions, we recommend that users give thought about how even simple natural image datasets may be biased towards certain regions and cultures (DeVries et al., 2019;

Original: Merler et al., 2019; Shankar et al., 2017).
Revised: Merler et al., 2019; Shankar et al., 2017).

Original: We further suggest application developers should invest in developing diagnostics to measure and report how predictions impact diﬀerent subpopulations (Mitchell et al., 2019).
Revised: We further suggest application developers should invest in developing diagnostics to measure and report how predictions impact diﬀerent subpopulations (Mitchell et al., 2019).

Original: Future work could integrate constraints that enforce various deﬁnitions of group-level fairness (Mitchell et al., 2018) into our approach, as have been developed for other deep latent variable models (Madras et al., 2019).
Revised: Future work could integrate constraints that enforce various deﬁnitions of group-level fairness (Mitchell et al., 2018) into our approach, as have been developed for other deep latent variable models (Madras et al., 2019).

Original: Speciﬁcally for the CelebA dataset of celebrity faces, we acknowledge that some of its recorded attributes may be used problematically (e.g. “big nose” or identiﬁcation of speciﬁc racial groups).
Revised: Speciﬁcally for the CelebA dataset of celebrity faces, we acknowledge that some of its recorded attributes may be used problematically (e.g. “big nose” or identiﬁcation of speciﬁc racial groups).

Original: We deliberately did not focus on such attributes but instead focused on two attributes thought to be more benign: facial expression (neutral vs. smiling) and gender (male vs. female).
Revised: We deliberately did not focus on such attributes but instead focused on two attributes thought to be more benign: facial expression (neutral vs. smiling) and gender (male vs. female).

Original: We acknowledge and aﬃrm that human gender is not binary; our classiﬁers learn the stereotypical patterns of celebrity photos from a speciﬁc human culture and time period.
Revised: We acknowledge and aﬃrm that human gender is not binary; our classiﬁers learn the stereotypical patterns of celebrity photos from a speciﬁc human culture and time period.