O N THE RELATIONSHIP BETWEEN DISENTANGLEMENT AND MULTI - TASK LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
One of the main arguments behind studying disentangled representations is the assumption that they can be easily reused in different tasks.
At the same time ﬁnding a joint, adaptable representation of data is one of the key challenges in the multi-task learning setting.
In this paper, we take a closer look at the relationship between disentanglement and multi-task learning based on hard parameter sharing.
We perform a thorough empirical study of the representations obtained by neural networks trained on automatically generated supervised tasks.
Using a set of standard metrics we show that disentanglement appears naturally during the process of multi-task neural network training.
I NTRODUCTION
Disentangled representations have recently become an important topic in the deep learning community (Eastwood & Williams, 2018;
Locatello et al., 2019a; Ma et al., 2019; Sanchez et al., 2019; Do & Tran, 2020).
The main assumption in this problem is that the data encountered in the real world is generated by few independent and explanatory factors of variation.
It is commonly accepted that such representations are not only more interpretable and robust but also perform better in tasks related to transfer learning and one-shot learning (Bengio, 2013; Lake et al., 2017;
Sch ¨ olkopf et al., 2012; Locatello et al., 2019c).
Intuitively, a disentangled representation encompasses all the factors of variation and as such can be used for various tasks based on the same input space.
On the other hand, non-disentangled representations, such as those learned by vanilla neural networks, might focus only on one or a few factors of variations that are relevant for the current task, while discarding the rest.
Such a representation may fail when encountering different tasks that rely on distant aspects of variation which have not been captured.
Exploiting prevalent features and differences across tasks is also the paradigm of multi-task learning.
In a standard formulation of a multi-task setting, a model is given one input and has to return predictions for multiple tasks at once.
The neural network might be therefore implicitly regularized to capture more factors of variation than a network that learns only a single task.
Based on this intuition, we hypothesize that disentanglement is likely to occur in the latent representations in this type of problem.
This paper aims to test this hypothesis empirically.
We investigate whether the use of disentangled representations improves the performance of a multi-task neural network and whether disentanglement itself is achieved naturally during the training process in such a setting.
Our key contributions are:
• Construction of synthetic datasets that allow studying the relationship between multi-task and disentanglement learning.
• Study of the effect of multi-task learning with hard parameter sharing on the level of disentanglement obtained in the latent representation of the model.
• Analysis of the informativeness of the latent representation obtained in the single- and multi-task training.
• Inspection of the effect of disentangled representations on the performance of a multi-task model.
We verify our hypotheses by training multiple models in single- and multi-task settings and investigating the level of disentanglement achieved in their latent representations.
In our experiments, we ﬁnd that in a hard-parameter sharing scenario multi-task learning indeed seems to encourage disentanglement.
However, it is inconclusive whether disentangled representations have a clear positive impact on the models performance, as the obtained by us results in this matter vary for different datasets.
R ELATED W ORK 2. D ISENTANGLEMENT
Over the recent years, many methods that directly encourage disentanglement have been proposed.
This includes algorithms based on variational and Wasserstein auto-encoders (Kim & Mnih, 2018;
Higgins et al., 2017; Kumar et al., 2017; Brakel & Bengio, 2017; Spurek et al., 2020), ﬂow networks (Dinh et al., 2014; Sorrenson et al., 2020) or generative adversarial networks (Chen et al., 2016).
The main interest behind disentanglement learning lays in the assumption that such transformation unravels the semantically meaningful factors of variation present in the observations and thus it is desired in training deep learning models.
In particular, disentanglement is believed to allow for informative compression of the data that results in a structural, interpretable representation, which is easily adaptable for new tasks (Bengio, 2013; Lake et al., 2017; Schmidhuber, 1992; Lipton, 2018).
Several of these properties have been experimentally proven in applications in many domains, including video processing tasks (Hsieh et al., 2018), recommendation systems (Ma et al., 2019) or abstract reasoning (Van Steenkiste et al., 2019; Steenbrugge et al., 2018).
Moreover, recent research in reinforcement learning concludes that disentangling embeddings of skills allows for faster retraining and better generalization (Petangoda et al., 2019).
Finally, disentanglement seems also to be positively correlated with fairness when sensitive variables are not observed (Locatello et al., 2019a).
On the other hand, some empirical studies suggest that one should be cautious while interpreting the properties of disentangled representations.
For instance, the latest studies in the unsupervised learning domain point that increased disentanglement does not lead to a decreased sample complexity in downstream tasks (Locatello et al., 2019b).
Another key challenge in studying disentangled representations is the fact that measuring the quality of the disentanglement is a nontrivial task (Do & Tran, 2020; Eastwood & Williams, 2018; Kim & Mnih, 2018), especially in a unsupervised setting (Locatello et al., 2019b).
This motivates the research on practical advantages of disentanglement representations and their impact on the studied problem in possible future applications, which is the main focus of our work in the case of multi-task learning.
2. M ULTI - TASK L EARNING
Multi-task learning aims at simultaneously solving multiple tasks by exploiting common information (Ruder, 2017).
The approaches used predominantly to this problem are soft (Duong et al., 2015) and hard (Caruana, 1993) parameter sharing.
In hard parameter sharing the weights of the model are divided into those shared by all tasks, and task-speciﬁc.
In deep learning, this idea is typically implemented by sharing consecutive layers of the network, which are responsible for learning a joint data representation.
In soft parameter sharing each task is given a set of separate parameters.
The limitations are then imposed by information-sharing or regularizing the distance between the parameters by adding an applicable loss to the optimization objective.
Multi-task learning is widely used in the Deep Learning community, for instance in applications related to natural language processing (Liu et al., 2019), computer vision (Misra et al., 2016) or molecular property prediction modeled by graph neural networks (Capela et al., 2019).
One may observe that the premises of multi-task and disentanglement learning are related to each other and thus it is interesting to investigate whether the joint data representation obtained in a multi-task problem exhibits some disentanglement-related properties.
M ETHODS
In this section, we describe the methods and datasets used for conducting the experiments.
3. D ATASET C REATION
In order to investigate the relationship between multi-task learning and disentanglement, we require a dataset that fulﬁlls two conditions:
It provides access to the true (disentangled) generative factors z from which the observations x are created.
It proposes multiple tasks for a supervised learner by providing labels y i which non-linearly depend on the true factors z .
[Figure]
The ﬁrst condition is required in order to measure how well the learned representations approximate the true latent factors z .
Access to the true factors allows for full control over the experimental settings and permits a fair comparison through the use of supervised disentanglement metrics.
Note that even though unsupervised metrics have been proposed in the literature as well, they typically yield less reliable results, as we further discuss in section 3.3.
The second condition is needed to train a network on multiple nontrivial tasks to approximate the real-world setting of multi-task learning.
To our best knowledge, no nontrivial datasets exist that would abide by both those requirements.
Most of the available disentanglement datasets, such as dSprites, Shapes3D, and MPI3D do fulﬁll the ﬁrst condition, as they provide pairs ( x, z ) of observations and their true generative factors.
However, those datasets do not offer any type of challenging task on which our model could be trained.
On the other hand, many datasets used for supervised multi-task learning fulﬁll the second condition by providing pairs ( x, y ) , but do not equip the researcher with the latent factors z (ground truth), failing the ﬁrst condition.
Thus, we aim to create our own datasets which fulﬁll both conditions by incorporating nontrivial tasks into standard disentanglement datasets.
Since in multi-task approaches one often tries to solve tens of tasks at once, designing them by hand is infeasible and as such we decide to generate them automatically in a principled way.
In particular, since supervised learning tasks might be formalized as ﬁnding a good approximation to an unknown function h ( x ) given a set of points ( x, h ( x )) , we generate random functions h ( z ) which are then used to obtain targets for our dataset (see Figure 1).
We require h ( z ) to be both nontrivial (i.e. non-linear and non-convex) and sufﬁciently smooth to approximate the nature of real-life tasks.
In order to ﬁnd a family of functions that fulﬁlls those conditions, we take inspiration from the ﬁeld of extreme learning, which ﬁnds that features obtained from randomly initialized neural networks are useful for training linear models on various realworld problems (Huang et al., 2011).
As such, randomly initialized networks should be able to approximate these tasks up to a linear operation.
In particular, in order to generate the dataset, we deﬁne a neural network architecture h
( z, θ ) .
For this purpose, we used an MLP with four hidden layers with 300 units, tanh activations, and an output layer which returns a single number.
Then we sample n weight initializations of this network from the Gaussian distribution θ i ∼ N (0 , 1) , where i ∈ { 1 , . . . , n } .
Each of the networks h ( z, θ i ) obtained by random initialization deﬁnes a single task in our approach.
Thus, for a given dataset
D =
( x, z ) containing observations and their true generative factors, we obtain a dataset for multitask supervised learning by applying:
[Equation]
where h ( z ) is a vector of stacked target values for each task, whose element i is given by h ( z ) i
= h ( z, θ i ) .
We use this data as a regression task, i.e. for a given neural network f φ parameterized by φ the goal is to ﬁnd: arg min φ (cid:88) ( x,
[Figure]
D (cid:107) f
φ ( x ) − y (cid:107) 22 .
We use this process to create multi-task supervised versions of dSprites, Shapes3D, and MPI3D, with 10 tasks for each dataset.
3. M ODELS 3.2. M ULTI - TASK MODEL
We investigate the relation between disentanglement and multi-task learning based on a hard parameter sharing approach.
In this setting, several consecutive hidden layers of the model are shared across all tasks in order to produce a joint data representation.
This representation is then propagated to separate task-speciﬁc layers which are responsible for computing the ﬁnal predictions.
[Figure]
In particular, we use a network consisting of a shared convolutional encoder and separate fully-connected heads for each of the tasks.
The encoder learns the joint representation by transforming the inputs into a d -dimensional latent space.
The heads are implemented by 4-layer
MLPs with ReLU activations, in order to match the capacity of the networks used for task generating functions h i ( x ) .
This overview of the model is illustrated in Figure 2.
3.2. A UTO - ENCODER MODEL
In the second part of our experiments we want to understand if disentangled representation provides some beneﬁts for the multitask problem.
In order to produce disentangled representations, we decided to use three different representation-learning algorithms: a vanilla auto-encoder, the (beta)-variational autoencoder (Kingma & Welling, 2013; Higgins et al., 2017) and FactorVAE (Kim & Mnih, 2018).
All these variants of the auto-encoder architecture encompass a similar framework.
An auto-encoder imposes a bottleneck in the network which forces a compressed knowledge representation of the original input.
In some variants of those models, we additionally try to constrain the latent variables to be highly informative and independent which further correlates to disentanglement, e.g. in models like β -VAE and FactorVAE.
We use latent representations from these models to train task-speciﬁc heads and evaluate if disentanglement helped to decrease an error for that task.
The vanilla auto-encoder is also used in Section 4.2, where we add a decoder with transposed convolutions to pre-trained encoders from Section 4.1.
This treatment is aimed to decode information for particular encoders in the most efﬁcient way.
As such, we ﬁnd auto-encoders to be a useful tool for investigating disentanglement.
3. D ISENTANGLEMENT M ETRICS
Measuring the qualitative and quantitative properties of the disentanglement representation discovered by the model is a nontrivial task.
Due to the fact that the true generating factors of a given dataset are usually unknown, one may assume that decomposition can be obtained only to some extend.
Commonly used unsupervised metrics are based on correlation coefﬁcients which measure the intrinsic dependencies between the latent components.
Such measures are widely used in the independent component analysis (Hyvarinen & Morioka, 2016; 2017; Hirayama et al., 2017; Brakel &
Bengio, 2017; Spurek et al., 2020; Bedychaj et al., 2020).
However, uncorrelatedness does not imply stochastical independence.
Furthermore, metrics based on linear correlations may not be able to capture higher-order dependencies and are often ineffective in large dimensional or in over-determined spaces.
All this makes the use of such unsupervised metrics questionable.
An alternative solution would be to use supervised metrics, which usually are more reliable (Locatello et al., 2019b).
This is of course only possible after assuming access to the true generative factors.
Such an assumption is rarely valid for real-world datasets, however, it is satisﬁed for synthetic datasets.
Synthetic datasets present therefore a reasonable baseline for benchmarking disentanglement algorithms.
Frequently used metrics which use supervision are mutual information gap (MIG)
(Chen et al., 2018), the FactorVAE metric (Kim & Mnih, 2018), Separated Attribute Predictability (SAP) score (Kumar et al., 2018) and disenanglement-completness-informativeness (DCI) (Eastwood &
Williams, 2018).
In order to comprehensively assess the level of disentanglement in our experiments, we have decided to use all of the above-mentioned metrics to validate our results.
A more detailed description of those metrics is available in Appendix B .
R ESULTS AND D ISCUSSION
In this section, we describe the performed experiments and discuss the obtained results.
For more details on the training regime and experimental setup please refer to Appendix C .
4. D OES H ARD P ARAMETER S HARING E NCOURAGE D ISENTANGLEMENT ?
One of the most common approaches to multi-task learning is hard parameter sharing.
The key challenge in this method is to learn a joint representation of the data which is at the same time informative about the input and can be easily processed in more than one task.
It is therefore tempting to verify whether disentanglement arises in those representations implicitly, as a consequence of hard parameter sharing.
In order to investigate this problem we build a simple multi-task model described in Section 3.2 and evaluate it on the three datasets discussed in Section 3.1: dSprites, Shapes3D, and MPI3D, each with 10 artiﬁcial tasks.
After the training is complete, we calculate each of the disentanglement metrics described in Section 3.3 on the latent representation of the input data 2 .
We compare the obtained results with the same metrics computed for an untrained (randomly initialized) network and for single-task models.
In all the cases we use the same architecture and training regime.
Note that in the single-model scenario we train a separate model for each of the 10 tasks, which is implemented by utilizing only one, dedicated head in the optimization process.
We train all models three times, using a different random seed in the parameters initialization procedure.
We report the mean results and standard deviations in Figure 3.
We observe that disentanglement metrics computed for the representations obtained in the multi-task setting are typically signiﬁcantly better than the values obtained for single-task or random representations.
Note that even the maximum mean result over all ten single-task models is in almost every case further than one standard deviation from the multitask mean.
Moreover, this is true for all the tested datasets.
Let us also point out that instead of using separate heads for each of the tasks in the multi-task model one could simply use one head with the output dimension equal to the number of tasks and perform standard multivariate regression (with no parameter sharing).
As presented in Figure 4, the latent representations emerging in such a scenario are less disentangled (in terms of the considered metrics) than the representations obtained when utilizing hard parameter sharing.
However, the achieved values are still better than in single-task models.
This suggests that even though the increase in the metrics may be partially caused by simply training the network on higher-dimensional targets, the positive inﬂuence of hard parameter sharing cannot be ignored.
This advocates in favor of the hypothesis that multi-task representations are indeed more disentangled than the ones arising in single-task learning.
4. W HAT A RE THE P ROPERTIES OF THE L EARNED R EPRESENTATIONS ?
The previous section discussed the obtained representations by analyzing quantitative disentanglement metrics.
Here, we provide more insights into the characteristics of latent encodings.
4.2. UMAP EMBEDDINGS
In order to gain intuition behind the differences between the representations obtained in the previous experiment we compute a 2D-embedding of the latent encodings using the UMAP algorithm (McInnes et al., 2018).
The results are presented in Figure 5.
The embeddings obtained for the multi-task representations are much more semantically meaningful, with easily distinguishable separate clusters.
Moreover, the position and internal structure of the clusters correspond to different values of the true factors.
This cannot be observed for the untrained or single-task representations, suggesting that the multi-task representations are indeed more successful in encompassing the information about the real values of the generative sources of the data.
[Figure]
original input images
The of the reconstruction for the random and single-task representation is very poor.
Contrary, the multi-task encoder provided a latent space that can be successfully decoded into images that closely resemble the corresponding examples from the input.
Thus, we conclude that the multi-task representations are more informative about the data and provide better compression.
See Appendix D.2 for reconstructions for other datasets.
4.2. L ATENT SPACE TRAVERSAL
Providing qualitative results of the retrieved factors is a common practice in disentanglement learning (Locatello et al., 2019c; Kumar et al., 2017; Sanchez et al., 2019; Sorrenson et al., 2020;
Locatello et al., 2019b).
In particular, visual presentation of the interpolations over the latent space allows assessing — from a human perspective — the informativeness and decomposition of the ob- tained representations.
Note that such analysis is possible only after adding and training a suitable decoder network, which maps the retrieved factors back to the image space.
-   input recon factor 1 factorfactor 3 factorfactor 5 factorfactor 7 factor
[Figure]
-   input recon factor 1 factorfactor 3 factorfactor 5 factorfactor 7 factor
[Figure]
-   input recon factor 1 factorfactor 3 factorfactor 5 factorfactor 7 factor
[Figure]
In our setting, the decoder mirrors the architecture of the encoder (the convolutions are replaced by transposed convolutions of the same size — see Appendix A ).
Given the latent representations as an input, the decoder optimizes the reconstruction error (as measured by MSE) between its outputs and the original images.
We train three separate decoders corresponding to the different encoders from the previous section — a randomly initialized encoder, an encoder produced by one of the single-task models, and a multi-task encoder.
First, let us discuss the reconstruction quality achieved by each of the tested decoders.
Results of this experiment are presented in Figure 6 3 .
Reconstructions produced for the multitask encodings are clearly superior to the ones obtained for the single-task encodings.
In the ﬁrst case, the resulting images are sharp and contain almost no noise.
In contrast, the single task reconstructions are blurry and similar to the ones produced for the randomly initialized encoder.
We would like to emphasise that all the decoders used the same architecture and that during their optimization the parameters of the corresponding encoders were kept ﬁxed.
Therefore the quality of the reconstruction is an important property of a latent representation, as it allows us to assess the compression capacity of the representation.
From this perspective, the compression obtained in the multi-task scenario is much more informative about the input than in the single-task scenario.
Another approach to the visualisation of the latent variables is to perform interpolations (traversals) in the latent space.
We start by selecting a random sample from the dataset and compute its encoding ˜ z ∈ R d .
By modifying one of the components of vector ˜ z from − 1 towith 0 .
1 step and leaving the d − 1 unchanged, we produce a traversal along that particular factor.
We repeat this procedure for all the factors in order to capture their impact on the decoded example.
Results of such traverses for the dSprites dataset are shown in Figure 7.
Note that since the models were not trained directly for disentanglement but only to solve a supervision task, it is not surprising that the representations are not as clearly factorized as in specialized methods such as FactorVAE.
However, for the multi-task model, certain latent dimensions still appear to be disentangled and one can easily spot the difference in quality between the single and multi-task representations.
In the multi-task traversals, we can notice components that are responsible for the position and scale of a given ﬁgure (in Figure 7c, consider the 5th and 7th factors, respectively).
In contrast, the results for single task representations demonstrate that even a slight change in any of the single latent dimensions leads to a degradation of the reconstructed examples.
Appendix D
As expected, this effect is even more evident for the random (untrained) representations, where the corruption over latent factor is even more prevalent than in the case of a single-task traversal.
4. D OES D ISENTANGLEMENT H ELP IN T RAINING M ULTI - TASK M ODELS ?
In the previous sections, we studied whether multi-task learning encourages disentanglement.
Here we consider an inverse problem by asking whether using disentangled representation helps in multitask learning.
To investigate this issue, we train an auto-encoder-based model devised speciﬁcally to produce disentangled latent representations without access to the true latent factors.
Next, we freeze its parameters and use the encoder function to transform the inputs.
The obtained latent encodings are then passed directly to the heads of a multi-task network which minimizes the average regression loss given the target values of the artiﬁcial tasks.
We consider three different auto-encoder-based algorithms described in Section 3.2.2: a vanilla autoencoder (AE), a variational auto-encoder (VAE), and the FactorVAE.
The vanilla auto-encoder does not directly enforce latent disentanglement during the training.
In the VAE model, the prior normal distribution with identity covariance matrix implies some disentanglement.
Finally, FactorVAE introduces a new module to the VAE architecture that explicitly induces informative decomposition.
Therefore, the representations obtained for each subsequent model should be also naturally ordered by the level of the achieved disentanglement.
For the exact values of the calculated metrics please refer to Appendix F .
In addition, we also study a scenario in which we explicitly provide the true source factors.
We trained all regression models three times, using a different random seed in the parameters initialization procedure.
[Figure]
[Table]
Table 1 summarizes the performance of the multi-task model trained on the representations obtained for the abovediscussed methods.
Although the representations obtained from FactorVAE are better (see, for instance, MIG or
DCI measures in Appendix F ) than those from VAE and AE, the encodings produced by the vanilla AE are the best among the tested, exceeding the others on Shapes3D and MPI3D and being second on dSprites.
Note that these results coincide with observations presented in the literature.
For example, (Locatello et al., 2019b) compared different models that enforce disentanglement during the training and showed that even a high value of that property within the factors do not constitute a better model performance.
However, in two out of three datasets, the use of the ground true factors seems to signiﬁcantly improve the obtained results.
This may suggest that the representations produced by the considered disentanglement methods are not fully factorized.
It is therefore inconclusive whether the discrepancy between the obtained results is due to the shortcomings of the used methods or a manifestation of the impracticality of disentanglement.
C ONCLUSIONS
In this paper, we studied the relationship between multi-task and disentanglement representation learning.
A fair evaluation of our hypothesis is impossible on real-world datasets, without provided ground truth factors.
To evaluate our results we had to introduce synthetic datasets that contain all necessary properties to be seen as a benchmark in this ﬁeld.
Next, we studied the effects of multi-task learning with hard parameter sharing on representation learning.
We found that nontrivial disentanglement appears in the representations learned in a multi-task setting.
Obtained factors have intuitive interpretations and correspond to the actual ground truth components.
Finally, we inverted the question and investigated the hypothesis that disentangled representation is needed for multitask learning, the results however are not conclusive.
We found out that multi-task models beneﬁt from disentanglement only on speciﬁc datasets.
However, we cannot name an indicator of when this unambiguously applies.
A S UMMARY OF THE ARCHITECTURE OF THE MULTI - TASK MODEL
The architecture of the convolutional encoder E ( x ) is provided in Table 2, together with the architecture of the corresponding decoder, which was used in experiments in Section 4.2.
For the fully-connected heads, we used the same architecture as the one utilized during dataset creation, which is presented in Table
[Table] [Table]
[Table]
B D ISENTANGLEMENT METRICS
In our experiments
[Table]
we decided to use four measures of disentanglement to comprehensively validate our results.
For the convenience of the reader, in this part of the appendix
[Table]
we shortly describe the used measures (for wider context we encourage the reader to refer to the original papers).
B. M UTUAL I NFORMATION G AP (MIG)
MIG computes the mutual information between each of the ground truth components
z i and the disentangled factor ˜ z
j .
The mutual information between z i and ˜
z j is denoted by I ( z i , ˜ z j ) .
Next, the latent dimension with maximum mutual information score is identiﬁed for each of the retrieved factor (denoted by I ( z i ,
˜ z max 1 ) ), along with the second-best result of the same score (denoted by
I ( z i , ˜ z max 2 ) ).
The difference between those values gives a gap, which ﬁnally is normalized with respect to the total mutual information associated with the studied factor:
[Equation]
Where m is the dimension of ground truth components space.
To report one score we average the
MIG scores of all factors.
B. F ACTOR VAE METRIC
We start by normalizing retrieved factors by their respective standard deviation computed over the dataset.
For a subset of the dataset, a ground truth component is then randomly selected and ﬁxed at a random value.
Variance is then computed over normalized retrieved factors in this subset.
Next, the lowest variance factor — the one that should mostly resemble the ﬁxed ground truth component — is associated with that ground truth component.
This procedure with selecting the subsets and ﬁxing one of its ground truth components is then repeated multiple times (in our experiments 10000 times).
As a result, the associations between disentangled factor and ground truth component are used as inputs in a majority vote classiﬁer.
FactorVAE
metric is the mean accuracy of the classiﬁer.
B. S EPARATED A TTRIBUTE P REDICTABILITY (SAP)
SAP attributes a score S ij to all pairs of ground truth components
z i and disentangled factors ˜
z j .
For continuous components, linear regression predicts the disentangled factors, and S ij is the coefﬁcient of determination ( R 2 ) of the regression.
In the case of categorical features, SAP ﬁts a decision tree on ground truth components and reports the balanced classiﬁcation accuracy.
The ﬁnal SAP score is achieved by computing the difference between the two highest S ij values for all factors:
[Equation]
where n is the dimension of ground truth components space, S i max 1 is the highest score for component
z i and S
i max 2 is the second highest score for the same component.
B. D ISENTANGLEMENT , C OMPLETENESS , AND I NFORMATIVENESS (DCI)
Unlike previous measures, DCI is a complete framework that allows verifying several properties of the achieved representation.
Disentanglement and completeness are estimated by inspecting the regressor’s parameters to derive predictive importance weights R ij for each pair ( z i , ˜ z j ) of ground truth z i and retrieved ˜ z
j components.
The completeness for ground truth component
z i is given by
[Equation]
where m stands for ground truth dimension and p ij is the probability that disentangled factor ˜
z j is important to predict z i .
These probabilities are obtained by dividing each importance weight by the sum of all importance weights related to a given component:
[Equation]
The ﬁnal compactness score is an average of compactness scores over all components.
Disentanglement for retrieved factor ˜ z
j is given by
[Equation]
where d is the dimension of the latent space and p ij is the probability that the latent factor ˜
z j is important to predict only the component z i .
Analogously to completeness, those probabilities are normalized with respect to potentially disentangled factors:
[Equation]
The ﬁnal disentanglement score is a weighted average of the individual disentanglement scores:
[Equation]
If a disentangled variable ˜
z
i is irrelevant for predicting z j , then its µ i (and thus contribution to the overall disentanglement) will be near zero.
Finally, the prediction error of the regressor measures the informativeness of the representation.
Normalized inputs and outputs allow to compute the estimation error for a completely random mapping and use it to normalize the score between 0 and 1 .
C T RAINING REGIME AND EXPERIMENTAL SETUP C. T HE MULTI - TASK MODEL — EXPERIMENT 4.
We train the multi-task model to minimize the sum of the task errors.
The training is performed for 200 epochs with learning rate 0 .
001 and batch size 256 , by using the AdaM optimizer (Kingma & Ba, 2014) with β 1 = 0 .
9 and β 2 = 0 .
We repeat this procedure three times, changing the random seed initialization, and report the mean and average values of the disentanglement metrics.
C. L ATENT VISUALISATIONS — EXPERIMENT 4.
The encoder architecture was taken from the experiments in Section 4.3.
The multi-task model for each experiment was randomly selected from one of the seeds from the 10 tasks setting.
Additionally, one of the single-task encoders was selected out of the trained ones for the same seed.
The random encoder was initialized by the default initialization used by the pytorch library.
The decoder architecture was optimized by minimizing the mean square error between the decoded and input image.
The training was performed over 500 epochs.
We used mini-batches of 64 images and gradually reduced learning rate starting from 0 . 0002 , with a reduction of 50% every 100 epochs.
C. C LASSIFICATION BASED ON LATENT FACTORS — EXPERIMENT 4.
We used the same auto-encoder and multi-task architectures like the one used in previous experiments (and deﬁned in Section A), however with non-linearity given by tanh function.
We trained all auto-encoders for 100 epochs, using batch size 64, learning rate 0.0001, Adam optimizer (Kingma & Ba, 2014) and latent dimension equal to 8.
Other hyperparameters settings were adapted from (Abdi et al., 2019).
Multi-task networks were trained for 30 epochs, using batch size 64, learning rate 0.0001, and adam optimizer.
In order to average the scores over different runs, we repeated the multi-task network training 3 times.
D V ISUALISATIONS OF DECODED REPRESENTATIONS D. UMAP E MBEDDINGS
In order to visualize the latent representations obtained for the random (untrained), single-task, and multi-task models we embed them into a two-dimensional space by using the UMAP algorithm.
The results are shown in Figure 8.
It may be observed that the embeddings obtained for the multi-task representations are much more semantically meaningful.
This is especially evident for the dSprites and Shapes3D datasets.
The MPI3D dataset is a signiﬁcantly more difﬁcult problem, and although the multi-task embeddings seem to be correlated to some of the true factors, the difference is not as visible in this case.
D. R ECONSTRUCTIONS
As described in Section 4.2, we trained decoders over various latent spaces produced by the encoders in the experiment from Section 4.1.
We provide the numerical values of the reconstruction error in
Table 4 and qualitative images of the reconstructed examples in Figure 9.
It may be observed that the latent representations produced by random and single task encoders do not allow the decoder to successfully restore the input examples.
Moreover, the decoder trained on single-task latent is even worse (in the case of reconstruction) than the random one.
[Table] [Table]
D. T RAVERSALS IN LATENT SPACE
In parallel to the study of the quality of the reconstructions, we have also explored the traversals in latent spaces.
Given a latent representation ˜ z of an arbitrary image x we compute the traverse along each one of the components of ˜ z , as described in Section 4.2.
This traversal represents how the image changes if only one component is slightly modiﬁed.
This procedure provides a visually qualitative way of assessing the level of disentangled in the obtained representations.
In order to complement the discussion conducted in Section 4.2 we present here also the traversals for the Shapes3D and MPI3D datasets (in Figures 11 and 12, respectively).
One may observe that the results align with the quantitative studies of disentangled metrics from Figure 3 — where we showed that the most disentangled representation is obtained in the multi-task scenario.
Note that the most informative changes of a particular feature for a given object may be observed in multitask traversals.
One may spot that object factors — although not totally disentangled — change independently from each other.
The same is not true for single-task traversals.
In the example from Shapes3D dataset (Figure 11), we observe that the single-task traversals capture only the color change of the background wall.
It is also not surprising that the least informative traversal comes from the randomly initialized encoder.
E D ISENTANGLEMENT AND H ARD P ARAMETER S HARING
In Section 4.1 we discuss the inﬂuence of hard parameter sharing on disentanglement learning.
Here we present the computed metrics for all models (including regression) in a tabulated manner in Table 5.
In addition, we also present the average MSE loss on the test dataset in Figure 13.
F D ISENTANGLED REPRESENTATION AS BASES FOR MULTI - TASK TRAINING
In Section 4.3 we took the opportunity to discuss how disentanglement inﬂuences multi-task training.
In this section, we present numerical results of all computed disentanglement metrics across trained encoders.
It is not surprising that FactorVAE representations are most disentangled in the
[Table]
predominant number of cases.
What can be read as a surprise is that FactorVAE representations are never the best in terms of the root mean square error metric of the model that was trained on them.
Apart from the tested in the main paper scenario with 10 tasks, we also conducted experiments with varying number of tasks n from the list of [5 , 10 , 20 , 30 , 40 , 50] . The results are presented in Figure 14. It is impossible to draw any clear conclusions from this results, as the results vary a lot.
It may be observed that in some cases increasing the number of tasks up to 30 leads to higher values of selected metrics, but at the same time having a negative impact on the others (for instance, Shapes3D and Factor VAE versus DCI disentanglement or DCI completeness).
These discrepancies are also not consistent between datasets (consider the top row for the dSprites dataset versus the Shapes3D).
H V ARYING THE NUMBER OF USED GENERATING FACTORS
Apart from the presented in Section 4.3 approach which uses all of the factors to generate a task, we also considered a scenario in which a random subset of the factors is sampled for each task, and a scenario in which the tasks are generated from disjoint subsets (every odd task depends only on the ﬁrst half of the factors and every even task on the other half).
We compare these approaches in Figure 15.
The computed disentanglement measures vary and the precise subset of incorporated factors in the task generating procedure does not have any conclusive impact on the ﬁnal quality of the learned representation.
I N UMBER OF RETRIEVED COMPONENTS
In addition to the results presented in Section 4.3 we also compute the number of retrieved components and the mean correlation values between the retrieved components and the ground truth factors in Table 7.
The results are computed for the representations obtained on the test splits for each datasets used in the UMAP embedding experiment in Section 4.3.
To get the number of retrieved components for each of the component of the representation we compute the spearman correlation with each of the ground true factor and choose the one for which the correlation is the largest and statistically signiﬁcant.
We next return the number of unique components matched in this way.
[Table]
J P ERFORMANCE ON SINGLE TASKS
In this section we provide the test losses on all tasks in Tables 8,
[Table]
and 10 for the dSprites, Shapes3D, and MPI3D datasets, respectively.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Figure]
dSprites
Shapes3D
MPI3D i npu t r a ndo m
s i
ng l e - t
a s k m u lti - t a s k
[Figure]
[Figure] [Figure]