D ISENTANGLED A CTIVATIONS IN D EEP N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
One of the main reasons for the success of deep neural networks is their capability to automatically learn representations of data in different levels of abstraction.
An important property of a useful representation is disentanglement.
In this paper we propose a novel regularization method that directly penalize covariance between dimensions of the hidden layers in a network.
This makes the network learn nonlinear representations that are linearly uncorrelated, yet allows the model to obtain good results on a number of tasks, as demonstrated by our experimental evaluation.
The proposed technique can be used to ﬁnd a suitable dimensionality of a layer, because it effectively disables dimensions that aren’t needed.
Our approach is simple and computationally cheap, as it can be applied as a regularizer to any gradient-based learning model.
I NTRODUCTION
A good data representation should ultimately uncover underlying factors in the raw data while being useful for a model to solve some task.
Deep neural networks learn representations that are increasingly abstract in deeper layers, disentangling the causes of variation in the underlying data (Bengio et al., 2013).
Formal deﬁnitions of disentanglement are lacking, although Ver Steeg & Galstyan (2015); Achille & Soatto (2017) both use the total correlation as a measure of disentanglement.
Inspired by this, we consider a simpler objective: a representation disentangles the data well when its components do not correlate, and we explore the effects of penalizing this linear dependence betweeen different dimensions in the representation.
Ensuring independence in the representation space results in a distribution that is factorizable and thus easy to model (Kingma & Welling, 2014;
Rezende et al., 2014).
We propose a novel regularization scheme that penalizes the cross-correlation between the dimensions of the learned representations, and helps artiﬁcial neural networks learn disentangled representations.
The approach is very versatile and can be applied to any gradient-based machine learning model that learns its own distributed vector representations.
A large body of literature have been published about techniques for learning non-linear independent representations (Lappalainen & Honkela, 2000; Honkela & Valpola, 2005;
Dinh et al., 2015), but in comparison our approach is simpler, and does not impose restrictions on the model used.
The proposed technique penalizes representations with correlated activations.
It strongly encourages the model to ﬁnd the dimensionality of the data, and thus to disable superﬂuous dimensions in the resulting representations.
The experimental evaluation on synthetic data veriﬁes this: the model is able to learn all useful dimensions in the data, and after convergence, these are the only ones that are active.
This can be of great utility when pruning a network, or to decide when a network needs a larger capacity.
The disabling of activations in the internal representation can be viewed as (and used for) dimensionality reduction.
The proposed approach allows for interpretability of the activations computed in the model, such as isolating speciﬁc underlying factors.
The solution is computationally cheap, and can be applied without modiﬁcation to many gradient-based machine learning models that learns distributed representations.
Moreover, we present an extensive experimental evaluation on a range of tasks on different data modalities, which shows that the proposed approach disentangles the data well; we do get uncorrelated components in the resulting internal representations, while retaining the performance of the models on their respective task.
The main contributions of this work include: L Σ regularization, a novel approach penalizing the covariance between dimensions in a representation (see Section 2).
The regularizer encourages a model to use the minimal number of dimensions needed in the representation.
The approach is computationally cheap and can be applied without any restrictions on the model.
The experimental evaluation shows how different models can beneﬁt from using L Σ regularization.
From autoencoders on synthetic data to deep convolutional autoencoders trained on CIFAR-10, we show that L Σ helps us learn uncorrelated and disentangled representations (see Section 3).
D ISENTANGLEMENT THROUGH PENALIZING CROSS - CORRELATIONS
We present a novel regularizer based on the covariance of the activations in a neural network layer over a batch of examples.
The aim of the regularizer is to penalize the covariance between dimensions in the layer to decrease linear correlation.
2. D EFINITION
The covariance regularization term ( L Σ ) for a layer, henceforth referred to as the coding layer, is computed as
[Equation]
where p is the dimensionality of the coding layer,
[Equation]
is the element wise L1 matrix norm of C , and C ∈ R
p × p is the sample covariance of the activations in the coding layer over N examples
[Equation]
Further, H =
[ h 1 ; ... ; h N ] is a matrix of all activations in the batch, 1 N is an N -dimensional column vector of ones, and ¯ h is the mean activation.
2. U SAGE
As L Σ has the structure of a regularizer, it can be applied to most gradient based models without changing the underlying architecture.
In particular, L Σ is simply computed based on select layers and added to the error function, e.g. Loss = Error + λL Σ
E XPERIMENTS
This section describes the experimental evaluation performed using L Σ regularization on different models in various settings, from simple multi-layer perceptron-based models using synthetic data (see Section 3.2 and 3.3) to convolutional autoencoders on real data (see Section 3.4).
However, before describing the experiments in detail we deﬁne the metrics that will be used to quantify the results.
3. E VALUATION METRICS
A number of different metrics are employed in the experiments to measure different aspects of the results.
Mean Absolute Pearson Correlation (MAPC) Pearson correlation report the normalized linear correlation between variables ∈ [ − 1 , 1] where 0 indicates no correlation.
To get the total linear correlation between all dimensions in the coding layer the absolute value of each contribution is averaged.
[Equation]
Covariance/Variance Ratio (CVR)
Though mean absolute Pearson correlation measure the quantity we are interested in it becomes ill deﬁned when the variance of one (or both) of the variables approaches zero.
To avoid this problem we deﬁne a related measure where all variances are summed for each term.
Hence, as long as some dimension has activity the measure remains well deﬁned.
More precise, the CVR score is computed as:
[Equation]
where ||C|| 1 is deﬁned as in Equation 2.
The intuition behind CVR is simply to measure the fraction of all information that is captured in a linear uncorrelated fashion within the coding layer.
Top d-dimension Variance/total variance (TdV) TdV measure to what degree the total variance is captured inside the variance of the top d dimensions.
When d is equal to the actual dimension of the underlying data this measure is bounded in [0,1].
Utilized Dimensions (UD) UD is the number of dimensions that needs to be kept to retain a set percentage, e.g. 90% in the case of UD 90% , of the total variance.
This measure has the advantage that the dimension of the underlying data does not need to be known a priori.
3. D IMENSIONALITY REDUCTION
The purpose of this experiment is to investigate if it is possible to disentangle independent data that has been projected to a higher dimension using a random projection, i.e. we would like to ﬁnd the principal components of the original data.
The model we employ in this experiment is an auto encoder with a linear coding layer.
The model is trained using the proposed covariance regularization L Σ on the coding layer.
The data is generated by sampling a d dimensional vector of independent features z ∼ N (0 , Σ) , where Σ ∈ R d × d is constrained to be non-degenerate and diagonal.
However, before the data is feed to the autoencoder it is pushed through a random linear transformation x = g ( z ) s .
The goal of the model is to reconstruct properties of z in the coding layer while only having access to x .
The model is trained on 10000 iid random samples for 10000 epochs using 9 different regularization constants.
First point on each curve is λ = 0 , i.e. no regularization, followed by 8 points logarithmically spaced between 0.001 and 1.
The experiment is repeated 10 times using a different random projection g and the average is reported.
The result of the experiment is reported using all four metrics deﬁned in Section 3.1.
The result in terms of MAPC and CVR is reported in Figure 2.
The ﬁrst thing to notice is that L Σ consistently lead to lower correlation while incurring less MSE penalty compared to L 1 .
Further, looking at the MAPC it is interesting to notice that it is optimal for a very small values of L Σ .
This is because higher amounts of L Σ leads to lowering of the dimensionality of the data, see Figure 3, which in turn yields unpredictable Pearson correlation scores between these inactivated neurons.
However, this effect is compensated for in CVR for which L Σ quickly converges towards the optimal value of one, which in turn indicates no presence of linear correlation.
Turning the attention to dimensionality reduction
Figure 3 shows that L Σ consistently outperform
L 1 .
Further, looking closer at the TdV score, L Σ is able to compress the data almost perfectly, i e.
TdV=1, at a very small MSE cost while L 1 struggle even when accepting a much higher MSE cost.
Further, the UD 90% scores again show that L Σ achieves a higher compression at lower MSE cost.
In this instance the underlying data was of 4 dimensions which L Σ quickly achieves.
At higher amounts of L Σ the dimensionality even locationally fall to 3, however, this is because the threshold is set to 90%.
3. D EEP NETWORK OF UNCORRELATED FEATURES
In Section 3.2 we showed that we can learn a minimal orthogonal representation of data that is generated to ensure that each dimension is independent.
However, in reality it is not always possible to encode the necessary information, to solve the problem at hand, in an uncorrelated coding layer.
However, using a deep network it should be possible to learn a nonlinear transformation that enables uncorrelated features in higher layers.
To test this in practice on a problem that has this property but still is small enough to easily understand we turn to the XOR problem.
It is well known that the XOR problem can be solved by a neural network of one hidden layer consisting of a minimum of two units.
However, instead of providing this minimal structure we would like the network to discover it by itself during training.
Hence, the model used is intentionally overspeciﬁed consisting of two hidden layers of four logistic units each followed by a one dimensional logistic output layer.
The model was trained on XOR examples, e.g. [1,0]=1, in a random order until convergence with
L Σ applied to both hidden layers and added to the cost function after scaling it with λ = 0 . 2 .
As can be seen in Figure 4 the model was able to learn the optimal structure of exactly 2 dimensions in the ﬁrst layer and one dimension in the second.
Further, as expected, the ﬁrst layer do encode a negative covariance between the two active units while the second layer is completely free from covariance.
Note that, even though the second hidden layer is not the output of the model it does encode the result in that one active neuron.
For comparison, see Figure 5 for the same model trained without L Σ .
3. N ON - LINEAR UNCORRELATED CONVOLUTIONAL FEATURES
Convolutional autoencoders have been used to learn features for visual input and for layer-wise pretraining for image classiﬁcation tasks.
Here, we will see that it is possible to train a deep convolutional autoencoder on real-world data and learn representations that have low covariance, while retaining the reconstruction quality.
To keep it simple, the encoder part of the model used two convolutional layers and two fully connected layers, with a total of roughly 500.000 parameters in the whole model.
The regularization was applied to the coding layer which has 84 dimensions, giving a bottleneck effect.
The model
[Figure]
was trained and evaluated on the CIFAR-10 dataset (Krizhevsky & Hinton, 2009), containing 32x pixel colour images tagged with 10 different classes.
The model was trained on 45,000 images, while 5,000 were set aside for validation, and 10,000 make out the test set.
We compare the results from using L Σ regularization with L1 regularization and with no regularization at all.
The autoencoder was trained with a batch size of 100, using the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of 0.001.
Training was run until the MSE score on the validation set stopped improving 1 .
The regularization parameter λ was chosen to be 0.08, for a reasonable trade-off between performance and covariance/variance ratio.
The reported scores in Table 1 and
Figure 6 are averages from training the model ﬁve times with different initialization.
The results (see Table 1) show that the high-level features become more disentangled and has a lower
CVR (6.56) using L Σ regularization.
Without regularization, the score is 20.00, and with L1 regu- larization the score is 4.03.
The model with L Σ regularization obtains a reconstruction error (MSE) of 0.0398, roughly the same as without regularization (0.0365), both of which are much better than using L1 regularization, with an MSE of 0.0569.
Figure 6 shows the CVR score plotted against the
MSE, illustrating that the L Σ technique leads to more disentangled representations while retaining a better MSE score.
As you increase the regularization factor both L Σ regularization pushes down the
CVR quickly, while retaining an MSE error that is almost constant.
L1 regularization also pushes the model towards learning representation with lower CVR, although slower, and while worsening the MSE error.
The UD 90% results show that L Σ encourages representations that concentrate the variation, and the model constantly learns representations with lower UD 90% score than using L1.
With λ > 0 .
08 , the MSE, the CV R , and the UD 90% all becomes much worse when using L1 regularization, while the L Σ seems to continue smoothly to improve CV R and UD 90% , as the MSE starts to grow.
[Table]
R ELATED WORK
Disentanglement is important in learned representations.
Different notions of independence have been proposed as useful criteria to learn disentangled representations, and a large body of work has been dedicated to methods that learn such representations.
Principal component analysis (PCA; Pearson, 1901) is a technique that ﬁts a transformation of the (possibly correlated) input into a space of lower dimensionality of linearly uncorrelated variables.
Nonlinear extensions of PCA include neural autoencoder models (Kramer, 1991), using a network layout with three hidden layers and with a bottleneck in the middle coding layer, forcing the network to learn a lower-dimensional representation.
Self-organizing maps (Kohonen, 1982) and kernelbased models (Sch ¨ olkopf et al., 1998) have also been proposed for nonlinear PCA.
Independent component analysis (ICA; Hyv ¨ arinen et al., 2004) is a set of techniques to learn additive components of the data with a somewhat stronger requirement of statistical independence.
A number of approaches have been made on non-linear independent components analysis, (Lappalainen & Honkela, 2000; Honkela & Valpola, 2005).
While ICA has a somewhat stronger criterion on the resulting representations, the approaches are generally more involved.
Dinh et al.
, (2015; 2017) proposed a method to train a neural network to transform data into a space with independent components.
Using the substitution rule of differentiation as a motivation, they learn bijective transformations, letting them use the neural transformation both to compute the transformed hidden state, to sample from the distribution over the hidden variables, and get a sample in the original data space.
The authors used a ﬁxed factorial distribution as prior distribution (i.e. a distribution with independent dimensions), encouraging the model to learn independent representations.
The model is demonstrated as a generative model for images, and for inpainting (sampling a part of the image, when the rest of it is given).
Achille & Soatto (2017) connected the properties of disentanglement and invariance in neural networks to information theoretic properties.
They argue that having invariance to nuisance factors in a network requires that its learned representations to carry minimal information.
They propose using the information bottleneck Lagrangian as a regularizer for the weights.
Our approach is more ﬂexible and portable, as it can be applied as a regularization to learn uncorrelated components in any gradient-based model that learns internal representations.
Brakel & Bengio (2017) showed that it is possible to adversarial training to make a generative network learn a factorized, independent distribution p ( z ) .
The independence criterion (mutual information) makes use of the Kullback-Leibler divergence between the joint distribtion p ( z ) (represented by the generator network) and the product of the marginals (which is not explicitly modelled).
In this paper, the authors propose to resample from the joint distribution, each time picking only the value for one of the components
z i , and let that be the sample from the marginal for that component, p ( z i ) .
A discriminator (the adversary) is simultaneously trained to distinguish the joint from the product of the marginals.
One loss function is applied to the output of the discriminator, and one measures the reconstruction error from a decoder reconstructing the input from the joint.
Thomas et al.
(2017) considers a reinforcement learning setting where there is an environment with which one can interact during training.
The authors trained one policy π i ( a | s ) for each dimension i of the representation, such that the policy can interact with the environment and learn how to modify the input in a way that modiﬁes the representation only at dimension i , without changing any other dimensions.
The approach is interesting because it is a setting similar to humans learning by interaction, and this may be an important learning setting for agents in the future, but it is also limited to the setting where you do have the interactive environment, and cannot be applied to other settings discussed above, whereas our approach can.
C ONCLUSIONS
In this paper, we have presented L Σ regularization, a novel regularization scheme based on penalizing the covariance between dimensions of the internal representation learned in a hierarchical model.
The proposed regularization scheme helps models learn linearly uncorrelated variables in a non-linear space.
While techniques for learning independent components follow criteria that are more strict, our solution is ﬂexible and portable, and can be applied to any feature-learning model that is trained with gradient descent.
Our method has no penalty on the performance on tasks evaluated in the experiments, while it does disentangle the data.
We saw that our approach performs well applied to a standard deep convolutional autoencoder on the CIFAR-10 dataset (Krizhevsky & Hinton, 2009); the resulting model performs comparable to the model without L Σ regularization, while we can also see that the covariances between dimensions in the internal representation decrease drastically.