Paper:
Latent representation learned from multi-layered neural networks via hierarchical feature abstraction enables recent success of deep learning. Under the deep learning framework, generalization performance highly depends on the learned latent representation. In this work, we propose a novel latent space modeling method to learn better latent representation. We designed a neural network model based on the assumption that good base representation for supervised tasks can be attained by maximizing the sum of hierarchical mutual informations between the input, latent, and output variables. From this base model, we introduce a semantic noise modeling method which enables semantic perturbation on the latent space to enhance the representational power of learned latent feature. During training, latent vector representation can be stochastically perturbed by a modeled additive noise while preserving its original semantics. It implicitly brings the effect of semantic augmentation on the latent space. The proposed model can be easily learned by back-propagation with common gradient-based optimization algorithms. Experimental results show that the proposed method helps to achieve performance benefits against various previous approaches. We also provide the empirical analyses for the proposed latent space modeling method including t-SNE visualization.
1 INTRODUCTION
Enhancing the generalization performance against unseen data given some sample data is the main objective in machine learning. Under that point of view, deep learning has been achieved many breakthroughs in several domains such as computer vision (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016), natural language processing (Collobert & Weston, 2008; Bahdanau et al., 2015), and speech recognition (Hinton et al., 2012; Graves et al., 2013). Deep learning is basically realized on deep layered neural network architecture, and it learns appropriate taskspecific latent representation based on given training data. Better latent representation learned from training data results in better generalization over the future unseen data. Representation learning or latent space modeling becomes one of the key research topics in deep learning. During the past decade, researchers focused on unsupervised representation learning and achieved several remarkable landmarks on deep learning history (Vincent et al., 2010; Hinton et al., 2006; Salakhutdinov & Hinton, 2009). In terms of utilizing good base features for supervised learning, the base representation learned from unsupervised learning can be a good solution for supervised tasks (Bengio et al., 2007; Masci et al., 2011).
The definition of ‘good’ representation is, however, different according to target tasks. In unsupervised learning, a model is learned from unlabelled examples. Its main objective is to build a model
∗Corresponding author
to estimate true data distribution given examples available for training, so the learned latent representation normally includes broadly-informative components of the raw input data (e.g., mutual information between the input and the latent variable can be maximized for this objective). In supervised learning, however, a model is learned from labelled examples. In the case of classification, a supervised model learns to discriminate input data in terms of the target task using corresponding labels. Latent representation is therefore obtained to maximize the performance on the target supervised tasks.
Since the meaning of good representations vary according to target tasks (unsupervised or supervised), pre-trained features from the unsupervised model are not be guaranteed to be useful for subsequent supervised tasks. Instead of the two stage learning strategy (unsupervised pre-training followed by supervised fine-tuning), several works focused on a joint learning model which optimizes unsupervised and supervised objectives concurrently, resulting in better generalization performance (Goodfellow et al., 2013; Larochelle & Bengio, 2008a; Rasmus et al., 2015; Zhao et al., 2015; Zhang et al., 2016; Cho & Chen, 2014).
In this work, we propose a novel latent space modeling method for supervised learning as an extension of the joint learning approach. We define a good latent representation of standard feed-forward neural networks under the basis of information theory. Then, we introduce a semantic noise modeling method in order to enhance the generalization performance. The proposed method stochastically perturbs the latent representation of a training example by injecting a modeled semantic additive noise. Since the additive noise is randomly sampled from a pre-defined probability distribution every training iteration, different latent vectors from a single training example can be fully utilized during training. The multiple different latent vectors produced from a single training example are semantically similar under the proposed latent space modeling method, so we can expect semantic augmentation effect on the latent space.
Experiments are performed on two datasets; MNIST and CIFAR-10. The proposed model results in better classification performance compared to previous approaches through notable generalization effect (stochastically perturbed training examples well cover the distribution of unseen data).
2 METHODOLOGY
The proposed method starts from the existing joint learning viewpoint. This section first explains the process of obtaining a good base representation for supervised learning which is the basis of the proposed latent space modeling method. And then, we will describe how the proposed semantic noise modeling method perturbs the latent space while maintaining the original semantics.
2.1 BASE JOINT LEARNING MODEL
In a traditional feed-forward neural network model (Figure 1(a)), output Y of input data X is compared with its true label, and the error is propagated backward from top to bottom, which implicitly learns a task-specific latent representation Z of the input X . As an extension of a joint learning approach, an objective to be optimized can be described in general as below (Larochelle & Bengio, 2008b):
min θ λLunsup + Lsup (1)
where Lunsup and Lsup are respectively an unsupervised loss and a supervised loss, and θ and λ are model parameters to be optimized during training and a loss weighting coefficient, respectively. In terms of modeling Lunsup in Eq. (1), we assume that good latent representation Z is attained by maximizing the sum of hierarchical mutual informations between the input, latent, and output variables; i.e. the sum of the mutual information between the input X and the Z and the mutual information between the Z and the output Y . Each mutual information is decomposed into an entropy and a conditional entropy terms, so the sum of hierarchical mutual informations is expressed as follows:
I(X;Z) + I(Z;Y ) = H(X)−H(X|Z) + H(Z)−H(Z|Y ) (2)
where I(·; ·) is the mutual information between random variables, and H(·) and H(·|·) are the entropy and the conditional entropy of random variables, respectively. Note that the sum of those mutual informations becomes equivalent to the total correlation ofX , Z, and Y under the graphical structure of the general feed-forward model described in Figure 1(a); P (X,Z, Y ) = P (Y |Z)P (Z|X)P (X). The total correlation is equal to the sum of all pairwise mutual informations (Watanabe, 1960).
Our objective is to find the model parameters which maximize I(X;Z)+ I(Z;Y ). Since H(X) and H(Z) are non-negative, and H(X) is constant in this case, the lower bound on I(X;Z) + I(Z;Y ) can be reduced to 1:
I(X;Z) + I(Z;Y ) ≥ −H(X|Z)−H(Z|Y ) . (3)
It is known that maximizing −H(X|Z) can be formulated as minimizing the reconstruction error between the input x(i) (i-th example sampled from X) and its reconstruction x(i)R under the general audo-encoder framework (Vincent et al., 2010). Since H(X|Z) + H(Z|Y ) is proportional to the sum of reconstruction errors of x(i) (with its reconstruction x(i)R ) and z
(i) (with its reconstruction z (i) R ), the target objective can be expressed as follows (refer to Appendix (A1) for the details of mathematical derivations):
min θ ∑ i Lrec(x (i), x (i) R ) + Lrec(z (i), z (i) R ) (4)
where Lrec is a reconstruction loss.
Figure 1(b) shows the target model obtained from the assumption that good latent representation Z can be obtained by maximizing the sum of hierarchical mutual informations. Given an input sample x, feed-forward vectors and their reconstructions are attained deterministically by:
z = fθ1(x)
y = fθ2(fθ1(x))
xR = gθ′1(z) = gθ′1(fθ1(x))
zR = gθ′2(y) = gθ′2(fθ2(fθ1(x)) .
(5)
1Although H(Z) is an upper bound of H(Z|Y ), H(Z) is anyway affected by the process of H(Z|Y ) being minimized in Eq. (3). In Section 4, we experimentally show that we can obtain good base model even from the relatively loose lower bound defined in Eq. (3).
Given a set of training pairs (x(i), t(i)) where x(i) and t(i) are the i-th input example and its label, target objective in Eq. (1) under the model described in Figure 1(b) can be organized as below (with real-valued input samples, L2 loss LL2 is a proper choice for the reconstruction loss Lrec):
min θ:{θ1,θ′1,θ2,θ′2} ∑ i λ ( LL2(x (i), x (i) R ) + LL2(z (i), z (i) R ) ) + LNLL(y (i), t(i)) (6)
where LNLL is a negative log-likelihood loss for the target supervised task. Note that Eq. (6) represents the ‘proposed-base’ in our experiment (see Section 4.3).
2.2 SEMANTIC NOISE MODELING
Based on the architecture shown in Figure 1(b) with the target objective in Eq. (6), we conjecture that stochastic perturbation on the latent space during training helps to achieve better generalization performance for supervised tasks. Figure 1(c) shows this strategy which integrates the stochastic perturbation process during training. Suppose that ZP is a perturbed version of Z, and YP is an output which is feed-forwarded from ZP . Given a latent vector z = fθ1(x) from an input sample x,
z′ = z + ze and ŷ = fθ2(z ′) (7)
where z′ and ŷ are a perturbed latent vector and its output respectively, and ze is an additive noise used in the perturbation process of z. Based on the architecture shown in Figure 1(c), target objective can be modified as:
min θ:{θ1,θ′1,θ2,θ′2} ∑ i λ1 ( LL2(x (i), x (i) R ) + LL2(z (i), z (i) R ) ) + λ2LNLL(y (i), t(i)) + LNLL(ŷ (i), t(i)) .
(8)
Using random additive noise directly on ze is the most intuitive approach (‘proposed-perturb (random)’ in Section 4.3). However, preserving the semantics of the original latent representation z cannot be guaranteed under the direct random perturbation on the latent space. While the latent space is not directly interpretable in general, the output logit y of the latent representation z is interpretable, because the output logit is tightly coupled to the prediction of the target label. In order to preserve the semantics of the original latent representation after perturbation, we indirectly model a semantic noise on the latent space by adding small random noise directly on the output space.
Based on the output (pre-softmax) logit y, the semantic-preserving variation of y (i.e. y′) can be modeled by y′ = y + ye, where ye is a random noise vector stochastically sampled from a zeromean Gaussian with small standard deviation σ; N (0, σ2I). Now, the semantic perturbation z′ can be reconstructed from the random perturbation y′ through the decoding path gθ′2 in Figure 1(c). From the original output logit y and the randomly perturbed output logit y′, semantic additive noise ze on the latent space can be approximately modeled as below:
zR = gθ′2(y) z′R = gθ′2(y ′) = gθ′2(y + ye)
ze ' z′R − zR = gθ′2(y + ye)− gθ′2(y) (9)
By using the modeled semantic additive noise ze and the original latent representation z, we can obtain the semantic perturbation z′ as well as its output ŷ via Eq. (7) for our target objective Eq. (8).
From the described semantic noise modeling process (‘proposed-perturb (semantic)’ in Section 4.3), we expect to achieve better representation on the latent space. The effect of the proposed model in terms of learned latent representation will be explained in more detail in Section 4.4.
3 RELATED WORKS
Previous works on deep neural networks for supervised learning can be categorized into two types as shown in Figure 2; (a) a general feed-forward neural network model (LeCun et al., 1998; Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016), and (b) a joint learning model which optimizes unsupervised and supervised objectives at the same time (Zhao et al., 2015; Zhang et al., 2016; Cho & Chen, 2014). Here are the corresponding objective functions:
min θ:{θ1,θ2} ∑ i LNLL(y (i), t(i)) (10)
min θ:{θ1,θ′1,θ2} ∑ i λLL2(x (i), x (i) R ) + LNLL(y (i), t(i)) (11)
where λ is a loss weighting coefficient between unsupervised and supervised losses.
Since the feed-forward neural network model is normally implemented with multiple layers in a deep learning framework, the joint learning model can be sub-classified into two types according to the type of reconstruction; reconstruction only with the input data x (Eq. (11)) and reconstruction with all the intermediate features including the input data x as follows:
min θ ∑ i λ0LL2(x(i), x(i)R ) +∑ j λjLL2(h (i) j , h (i) jR ) + LNLL(y (i), t(i))  . (12) where h(i)j and h (i) jR
are the j-th hidden representation of the i-th training example and its reconstruction.
Another type of the joint learning model, a ladder network (Figure 3), was introduced for semisupervised learning (Rasmus et al., 2015). The key concept of the ladder network is to obtain robust features by learning de-noising functions (gθ′) of the representations at every layer of the model via reconstruction losses, and the supervised loss is combined with the reconstruction losses in order to build the semi-supervised model. The ladder network achieved the best performance in semi-supervised tasks, but it is not appropriate for supervised tasks with small-scale training set (experimental analysis for supervised learning on permutation-invariant MNIST is briefly summarized
in Appendix (A2)). The proposed model in this work can be extended to semi-supervised learning, but our main focus is to enhance the representational power on latent space given labelled data for supervised learning. We leave the study for semi-supervised learning scenario based on the proposed methodology as our future research.
4 EXPERIMENTS
For quantitative analysis, we compare the proposed methodology with previous approaches described in Section 3; a traditional feed-forward supervised learning model and a joint learning model with two different types of reconstruction losses (reconstruction only with the first layer or with all the intermediate layers including the first layer). The proposed methodology includes a baseline model in Figure 1(b) as well as a stochastic perturbation model in Figure 1(c). Especially in the stochastic perturbation model, we compare the random and semantic perturbations and present some qualitative analysis on the meaning of the proposed perturbation methodology.
4.1 DATASETS
We experiment with two public datasets; MNIST (including a permutation-invariant MNIST case) and CIFAR-10. MNIST (10 classes) consists of 50k, 10k, and 10k 28×28 gray-scale images for training, validation, and test datasets, respectively. CIFAR-10 (10 classes) consists of 50k and 10k 32×32 3-channel images for training and test sets, respectively. We split the 50k CIFAR-10 training images into 40k and 10k for training and validation. Experiments are performed with different sizes of training set (from 10 examples per class to the entire training set) in order to verify the effectiveness of the proposed model in terms of generalization performance under varying sizes of training set.
4.2 IMPLEMENTATION
Figure 4 shows the architecture of the neural network model used in this experiment. W ’s are convolution or fully-connected weights (biases are excluded for visual brevity). Three convolution (3×3 (2) 32, 3×3 (2) 64, 3×3 (2) 96, where each item means the filter kernel size and (stride) with the number of filters) and two fully-connected (the numbers of output nodes are 128 and 10, respectively) layers are used for MNIST. For the permutation-invariant MNIST setting, 784-512- 256-256-128-10 nodes of fully-connected layers are used. Four convolution (5×5 (1) 64, 3×3 (2) 64, 3×3 (2) 64, and 3×3 (2) 96) and three fully-connected (128, 128, and 10 nodes) layers are used for CIFAR-10. Weights on the decoding (reconstruction) path are tied with corresponding weights on the encoding path as shown in Figure 4 (transposed convolution for the tied convolution layer and transposed matrix multiplication for the tied fully-connected layer).
In Figure 4, z′ is perturbed directly from z by adding Gaussian random noise for random perturbation. For semantic perturbation, z′ is indirectly generated from y′ which is perturbed by adding Gaussian random noise on y based on Eq. (9). For perturbation, base activation vector (z is the base
vector for the random perturbation and y is the base vector for the semantic perturbation) is scaled to [0.0, 1.0], and the zero-mean Gaussian noise with 0.2 of standard deviation is added (via elementwise addition) on the normalized base activation. This perturbed scaled activation is de-scaled with the original min and max activations of the base vector.
Initial learning rates are 0.005 and 0.001 for MNIST and permutation-invariant MNIST, and 0.002 for CIFAR-10, respectively. The learning rates are decayed by a factor of 5 every 40 epochs until the 120-th epoch. For both datasets, the minibatch size is set to 100, and the target objective is optimized using Adam optimizer (Kingma & Ba, 2015) with a momentum 0.9. All the λ’s for reconstruction losses in Eq. (11) and Eq. (12) are 0.03 and 0.01 for MNIST and CIFAR-10, respectively. The same weighting factors for reconstruction losses (0.03 for MNIST and 0.01 for CIFAR-10) are used for λ1 in Eq (8), and 1.0 is used for λ2.
Input data is first scaled to [0.0, 1.0] and then whitened by the average across all the training examples. In CIFAR-10, random cropping (24×24 image is randomly cropped from the original 32×32 image) and random horizontal flipping (mirroring) are used for data augmentation. We selected the network that performed best on the validation dataset for evaluation on the test dataset. All the experiments are performed with TensorFlow (Abadi et al., 2015).
4.3 QUANTITATIVE ANALYSIS
Three previous approaches (a traditional feed-forward model, a joint learning model with the input reconstruction loss, and a joint learning model with reconstruction losses of all the intermediate layers including the input layer) are compared with the proposed methods (the baseline model in Figure 1(b), and the stochastic perturbation model in Figure 1(c) with two different perturbation methods; random and semantic). We measure the classification performance according to varying sizes of training set (examples randomly chosen from the original training dataset). Performance is averaged over three different random trials.
Table 1 summarizes the classification performance for MNIST and CIFAR-10. As we expected, the base model obtained by maximizing the sum of mutual informations (proposed-base) mostly performs better than previous approaches, and the model with the semantic perturbation (proposedperturb (semantic)) performs best among all the comparison targets. Especially in MNIST, the error rate of ‘proposed-perturb (semantic)’ with 2k per-class training examples is less than the error rate of all types of previous works with the entire training set (approximately 5k per-class examples).
We further verify the proposed method on the permutation-invariant MNIST task with a standard feed-forward neural network. Classification performance is measured against three different sizes of training set (1k, 2k, and 5k per-class training examples). ‘Proposed-perturb (semantic)’ achieves the best performance among all the configurations; 2.57%, 1.82%, and 1.28% error rates for 1k, 2k, and 5k per-class training examples, respectively. The joint learning model with the input reconstruction loss performs best among three previous approaches; 2.72%, 1.97%, and 1.38% error rates for 1k, 2k, and 5k per-class training examples, respectively.
4.4 QUALITATIVE ANALYSIS
As mentioned before, random perturbation by adding unstructured noise directly to the latent representation cannot guarantee preserving the semantics of the original representation. We compared two different perturbation methods (random and semantic) by visualizing the examples reconstructed from the perturbed latent vectors (Figure 5). Top row is the original examples selected from training set (among 2k per-class training examples), and the rest are the reconstructions of their perturbed latent representations. Based on the architecture described in Figure 1(b), we generated five different perturbed latent representations according to the type of perturbation, and reconstructed the perturbed latent vectors through decoding path for reconstruction.
Figure 5(a) and (b) show the examples reconstructed from the random and semantic perturbations, respectively. For both cases, zero-mean Gaussian random noise (0.2 standard deviation) is used for perturbation. As shown in Figure 5(a), random perturbation partially destroys the original semantics; for example, semantics of ‘1’ is mostly destroyed under random perturbation, and some examples of ‘3’ are reconstructed as being similar to ‘8’ rather than its original content ‘3’. Figure 5(b) shows the examples reconstructed from the semantic perturbation. The reconstructed examples show subtle semantic variations while preserving the original semantic contents; for example, thickness difference in ‘3’ (example on the third row) or writing style difference in ‘8’ (openness of the top left corner).
Figure 6 shows the overall effect of the perturbation. In this analysis, 100 per-class MNIST examples are used for training. From the trained model based on the architecture described in Figure 1(b), latent representations z of all the 50k examples (among 50k examples, only 1k examples were used for training) are visualized by using t-SNE (Maaten & Hinton, 2008). Only the training examples of three classes (0, 1, and 9) among ten classes are depicted as black circles for visual discrimination in
Figure 6(a). The rest of the examples which were not used for training (approximately 4.9k examples per class) are depicted as a background with different colors. We treat the colored background examples (not used for training) as a true distribution of unseen data in order to estimate the generalization level of learned representation according to the type of perturbation. Figure 6(b) and (c) show the training examples (100 examples per class with yellow circles) and their perturbed ones (3× sampled from each example with blue crosses) through random and semantic perturbations, respectively.
In Figure 6(b), perturbed samples are distributed near the original training examples, but some samples outside the true distribution cannot be identified easily with appropriate classes. This can be explained with Figure 5(a), since some perturbed samples are ambiguous semantically. In Figure 6(c), however, most of the perturbed samples evenly cover the true distribution. As mentioned before, stochastic perturbation with the semantic additive noise during training implicitly incurs the effect of augmentation on the latent space while resulting in better generalization. Per-class t-SNE results are summarized in Appendix (A4.2).
5 DISCUSSION
We introduced a novel latent space modeling method for supervised tasks based on the standard feed-forward neural network architecture. The presented model simultaneously optimizes both supervised and unsupervised losses based on the assumption that the better latent representation can be obtained by maximizing the sum of hierarchical mutual informations. Especially the stochastic perturbation process which is achieved by modeling the semantic additive noise during training enhances the representational power of the latent space. From the proposed semantic noise modeling process, we can expect improvement of generalization performance in supervised learning with implicit semantic augmentation effect on the latent space.
The presented model architecture can be intuitively extended to semi-supervised learning because it is implemented as the joint optimization of supervised and unsupervised objectives. For semisupervised learning, however, logical link between features learned from labelled and unlabelled data needs to be considered additionally. We leave the extension of the presented approach to semisupervised learning for the future.

Review:
This paper introduces a maximum total correlation procedure, adds a target and then adds noise perturbations.
Technical issues:
The move from (1) to (2) is problematic. Yes it is a lower bound, but by igoring H(Z), equation (2) ignores the fact that H(Z) will potentially vary more significantly that H(Z|Y). As a result of removing H(Z), the objective (2) encourages Z that are low entropy as the H(Z) term is ignored, doubly so as low entropy Z results in low entropy Z|Y. Yes the -H(X|Z) mitigates against a complete entropy collapse for H(Z), but it still neglects critical terms. In fact one might wonder if this is the reason that semantic noise addition needs to be done anyway, just to push up the entropy of Z to stop it reducing too much.
In (3) arbitrary balancing paramters lamda_1 and lambda_2 are introduced ex-nihilo - they were not there in (2). This is not ever justified.
Then in (5), a further choice is made by simply adding L_{NLL} to the objective. But in the supervised case, the targets are known and so turn up in H(Z|Y). Hence now H(Z|Y) should be conditioned on the targets. However instead another objective is added again without justification, and the conditional entropy of Z is left disconnected from the data it is to be conditioned on. One might argue the C(X,Y,Z) simply acts as a prior on the networks (and hence implicitly on the weights) that we consider, which is then combined with a likelihood term, but this case is not made. In fact there is no explicit probabilistic or information theoretic motivation for the chosen objective.
Given these issues, it is then not too surprising that some further things need to be done, such as semantic noise addition to actually get things working properly. It may be the form of noise addition is a good idea, but given the troublesome objective being used in the first place, it is very hard to draw conclusions.
In summary, substantially better theoretical justification of the chosen model is needed, before any reasonable conclusion on the semantic noise modelling can be made.