An Empirical Study on Disentanglement of Negative-free Contrastive Learning
Abstract
Negative-free contrastive learning has attracted a lot of attention with simplicityand impressive performances for large-scale pretraining.
But its disentanglementproperty remains unexplored.
In this paper, we take different negative-free contrastive learning methods to study the disentanglement property empirically.
Wefind the existing disentanglement metrics fail to make meaningful measurements forthe high-dimensional representation model
so we propose a new disentanglementmetric based on Mutual Information between representation and data factors.
Withthe proposed metric, we benchmark the disentanglement property of negative-freecontrastive learning for the first time, on both popular synthetic datasets and areal-world dataset CelebA. Our study shows that the investigated methods canlearn a well-disentangled subset of representation.
We extend the study of thedisentangled representation learning to high-dimensional representation space andnegative-free contrastive learning for the first time.
Introduction
Learning a disentangled representation is a long-desired goal in the deep learning community (5; 39;13 ; 4; 44; 30; 49).
A disentangled representation matches how humans understand the world andallows us to use much fewer labels to learn challenging downstream tasks (51).
Previous disentangledlearning research is usually limited to using generative models (26; 16; 25; 12; 9; 33), on simplesynthetic datasets (38; 6) and with low representation dimensions, e g., no more than 20-d.
In contrast to generative methods, contrastive learning is a class of discriminative methods.
It istrained in a self-supervised manner by pulling the representation of two augmentations of the sameimage close.
Recent contrastive methods have achieved advanced performance on image pretrainingtasks and require high representation dimensions, e.g., usually at least 1000-d.
Given the success of contrastive learning, it remains unknown whether it can lead to disentangledrepresentations.
Recently, some works (52; 56) reveal that contrastive learning can approximatelyinvert the data generation process and this makes the learned representation have identifiability,which is related to the disentanglement property.
However, all these advances necessarily rely on thecontrast provided by negative samples.
The disentanglement property of contrastive learning withoutnegatives, or “non-contrastive self-supervised learning”, remains unexplored.
And no contrastivelearning methods have been evaluated in the standard disentanglement benchmarks yet.
Therefore, in this work, we study the disentanglement property of contrastive learning withoutusing negatives.
We study this empirically by introducing it into disentangled representation learningbenchmarks.
Given the fact that self-supervised contrastive learning methods require a high dimensionof latent representation.
This continues to bring a derivative study of quantitative disentanglementmeasurement in a high-dimensional representation space, which remains blank as well.
We findthough multiple disentanglement metrics (16; 25; 7; 11; 29) are demonstrated well aligned in lowdimensional space (36), they do not agree anymore in high-dimensional space.
Moreover, thedesign limitation of them makes the measurement even not meaningful anymore.
We thus propose a new disentanglement evaluation metric that is robust to representation dimension change andthus adaptable to a high-dimensional representation model.
The new metric is named as “ M utualinformation based E ntroy D isentanglement” and MED for short.
With the proposed metric, we find that negative-free contrastive learning methods can achieve gooddisentanglement in a subset of latent dimensions.
Given that the recent disentanglement study islimited to simple synthetic datasets but contrastive learning is especially powerful for complicated andlarge-scale datasets, we also extend the quantitative benchmarking to a real-world dataset CelebA (35).
On CelebA, existing low-dimensional generative disentangled learning methods can not learn goodrepresentation anymore, suggesting the gap between current disentangled learning research and thereal-world data complexity.
To summarize, our contributions in the work are three-folded:1.
We find that existing disentanglement metrics fail to extend to high-dimensional representationspace and propose a new metric MED to extend this area to high-dimensional space.2.
We extend the study of disentangled representation learning to real-world complicated datasets andhigh-dimensional representation space, revealing the gap between current disentangled representationlearning and real-world data complexity.3.
We empirically study the disentanglement property of contrastive learning without negatives.
Wefind it can learn a well-disentangled subspace of latent representation.
Related Works
Disentangled Representation Learning.
Disentangled representation aims to represent a humaninterpretable pattern (5; 9; 28), to make the downstream tasks learned more easily (51) and generalizebetter (1).
There are two lines of work related to this problem, i.e., the Independent Component
Analysis (ICA) and the Disentangled Deep Representation Learning.
ICA (19) usually assumes thatthe pattern of noise (18; 23) or some additional auxiliary variables (20; 24) can be observed.
Onthe other hand, deep representation learning makes no explicit assumption on the noise distributionor representation prior and usually emphasizes unsupervised learning.
Under this setting, deeprepresentation learning is usually based on deep generative models such as VAE-based methods (16;25; 7; 29) and Generative Adversarial Networks (GAN) (12; 9).
These two lines of study focus ondifferent but relevant (27) aspects of a learned representation.
In this paper, we focus on empiricallystudying the disentanglement property of negative-free contrastive learning methods to introducethem into the scope of disentangled representation learning.
Disentanglement Metrics.
The measurement also shows the difference between ICA and Disentangled Deep Representation Learning.
ICA aims to achieve good identifiability and uses Mean
Correlation Coefficient (MCC) as the common metric.
On the other hand, the metrics used in thedeep disentangled representation learning community are very diverse.
DisLib (36) (ICML 2019 bestpaper) makes an intensive empirical study of a stack of deep disentangled representation learningmethods and summarizes six popular disentanglement metrics, i.e., DCI (11), SAP (29), MIG (7),
BetaVAE score (16), FactorVAE score (25) and Modularity (43).
DisLib finds that the five metricsexcept for Modularity have good agreement in evaluating disentanglement quality.
Till now, nowidely accepted definition of “disentanglement” is available.
So the empirical agreement of metricsand their shared high-level intention of disentanglement, i.e. the degree to which a representationdimension only responds to a single factor, make the basis of quantitatively evaluating disentanglement.
However, all of the metrics are evaluated on low-dimensional latent space, which is around10 dimensions.
We find severe problems when applying those metrics to high-dimensional latentspaces where they show significant disagreement.
Our proposed metric, i.e., Mutual Informationbased Entropy Disentanglement, is designed to be more robust to high dimensional latent spaces andthus is a proper tool to evaluate the disentanglement of contrastive learning methods.
Contrastive Learning.
Contrastive learning (CL) creates “views” by augmentations over images.
Views of the same image serve as positives, and views of other images as negatives.
This setup isalso known as examplar classification (10), or instance discrimination (53).
Recently, some works tryto understand contrastive learning either theoretically (52; 2; 48; 47; 32) or empirically (45; 55; 40).
Zimmermann et al.
suggests that the contrastive method inverts a data generation process, whichis related to the disentanglement property.
This conclusion is based on Wang and Isola whereinfinite negative samples are required.
Moreover, empirical evidence supporting these theorems isinadequate and fails in real-world practice where some assumptions are violated (e.g., augmented data distribution).
There is still no work connecting CL methods with standard disentanglementbenchmarks.
On the other hand, negative-free CL (42; 8; 54) has its own advantages, such as avoidingmodel collapse (46; 17; 22), but its disentanglement property remains unexplored, either empiricallyor theoretically.
Thus, we aim to empirically study whether negative-free CL can learn disentangledrepresentations in this paper.
Method and Metric
In Section 3.1, we review the methods of negative-free contrastive learning.
Then we introduce ourproposed MI-based Entropy Disentanglement score (MED) in Section 3.2 and its variant to evaluatedisentanglement of a subspace of representation in Section 3.3.
3.1 Negative-free Contrastive Learning Methods
There are multiple negative-free contrastive learning methods proposed.
They share critical characteristics that they do not use the contrast between positive and negative samples to encouragediscriminative representations.
They generate positive “views” of data by applying different augmentations to the same input data and a pair of views is respectively forward into two steams of networks.
These two networks have a latent encoder of the same architecture.
However, recent works on thisline, such as BYOL (42), SimSiam (8) and Barlow Twins (54), have their own unique designs.
Thedevelopment of CL still heavily relies on empirical observation to learn non-trivial representations.
As the first and the most classic method in this line, BYOL (42) discovers that self-supervised learningcan avoid trivial solutions, i.e., “model collapse”, even without using negative samples to providecontrast.
The key of BYOL is to add a predictor layer following the commonly adopted “encoderprojector” network of contrastive learning methods (15).
This provides additional asymmetry.
Besidesthis, BYOL also proposes the stop-gradient strategy in the stream of the target network but its weightsare still updated by momentum from the online network.
As a follow-up, SimSiam (8) continuesto remove the momentum update from BYOL and proves that “predictor+stop-gradient” is enoughfor self-supervised learning to learn non-trivial representation.
More recently, Barlow Twins (54)provides a new perspective where even the predictor or the trick of stop-gradient is not necessaryanymore.
It moves the target of learning from samples to the embedding space.
The key idea isto make the empirical cross-correlation matrix among different views close to the identity.
Thisencourages the embedding vectors of views, which are distorted versions of the same sample, to besimilar while minimizing the redundancy between these vectors.
Positive-negative contrast in self-supervised learning is considered useful to encourage learnedrepresentation to have uniformity on a hyper-sphere (52).
When negative views are not availableanymore, the property of the learned representation remains a mystery, especially from the perspectiveof disentanglement we focus on in this work.
Therefore, we aim to take the mentioned negative-freecontrastive learning methods into an empirical study to reveal this property of interest.
3.2 Mutual Information based Entropy Disentanglement
Typical contrastive learning methods need a high dimensional representation space, or “latent space”,to train well.
However, the previous study of disentangled representation learning only deals withlow-dimensional representation space.
For example, in Locatello et al., the latent space dimension isset to 10 .
So existing disentanglement metrics (see Appendix C in the supplementary materials fordetails) are designed for low-dimensional representation model and have intrinsic flaws in extendingdisentanglement evaluation to high-dimensional space.
To be precise, we have observations as below:
• Metrics based on learnable classifiers, such as BetaVAE score and FactorVAE score , allow unfairadvantages to high-dimensional model whose redundant parameters can trick the classifier moreeasily.
For example, a randomly initialized 1000-d model could reach a FactorVAE score of 61.4on dSprites, close to many well-trained 10-d VAE-based models’ scores (see Table 4 in Appendix).• Metrics taking only one or two dimensions into score calculation, such as SAP and MIG , are
biased to representations of different dimensions.
Because a higher dimension makes it harder for
an informative dimension to stand out and enjoy a large informativeness gap over other dimensions.
• DCI Disentanglement score uses a learnable regressor to score the importance of each latent
dimension to each data factor.
The learnable regressor, such as Gradient Boosting Tree (GBT),
encourages sparsity in the output importance matrix (see Figure 6 in Appendix).
So it also gives
an advantage to high-dimensional models, making it unfair to compare models of different latent
dimensions.
Moreover, the construction of regressors is time-intensive in high-dimension space.
For example, it usually takes hours to evaluate a 1000-d representation model by DCI using GBT.
These flaws will be demonstrated by our experiments.
More importantly, when the dimension ofthe representation model is high, these existing metrics do not agree with each other anymore asin DisLib.
But such an empirical agreement is the basis of this community to make quantitativeevaluations without a uniformly shared definition of “disentanglement”.
So they are no longer eligibleto make meaningful disentanglement measurements in the high-dimensional representation space.
Given (1) the bias and limitations of existing disentanglement metrics, (2) the nature of contrastivelearning to require high-dimensional representation, and (3) our objective to evaluate its disentanglement, we propose a new disentanglement metric for high dimensional latent spaces, which we nameas “ M utual Information based E ntropy D isentanglement", or MED in short.
The calculation of MEDis based on mutual information (MI) between latent dimensions and dataset ground truth factors as MIis a widely accepted tool to measure correlation of variables and is not biased to models of differentdimensions.
Given a dataset generated by K ground truth factors v ∈ R K and a representation vectorc ∈ R D , we construct an importance matrix
R ∈ R D × K defined by
[Equation]
where I ( c i , v j ) denotes the mutual information between the i th latent dimension c i and the j thground truth factor v j .
Here, each row denotes a representation dimension and each column representsa ground truth factor.
We normalize the mutual information by columns, such that an entry in thematrix indicates the relative importance of one dimension over all dimensions.
This normalization isnecessary before computing the disentanglement level among factors, since different dimensions mayhave different overall informativeness to all factors.
This operation fixes the gap between models ofdifferent dimensions by estimating the relative importance of a single dimension.
After normalizing over the columns, we evaluate the contribution of a dimension to different factors,which is described by a row of R .
If one dimension is informative to only one ground truth factor,then this dimension is perfectly disentangled.
This matches the mechanism of entropy.
So we usethe entropy to describe the disentanglement level of a dimension.
We treat each row as a discretedistribution over factor index by normalization: P ij = R ij / (cid:80) K − 1 k =
R ik , where higher probability P ijindicates that the dimension c i encodes more information of the factor v j .
Then the disentanglementscore S i for a latent dimension
c i is calculated as
[Equation]
where H K ( P i · ) = − (cid:80) K − 1 k =
P ik log P ik is the entropy.
S i will be higher if c i exhibits moreinformativeness to one factor while less relevance to other factors.
Finally, to summarize the overalldisentanglement of a representation model, MED score is the weighted average of the disentanglementscores for all dimensions as
[Equation]
where ρ i = (cid:80)
j R ij / (cid:80) ij R ij is the relative importance of each dimension.
Our proposed MED does not use any learnable classifier or regressor.
Also, it inherits a DCI-stylenormalized importance matrix by taking all dimensions into calculation instead of using only oneor two dimensions.
These characteristics allow MED more robust to the latent dimensionality andeligible for a meaningful disentanglement measurement in high-dimensional space.
Besides theadvantages MED is designed to have, it is also very computationally efficient: for a 1000-d BYOLrepresentation model on Cars3D, the evaluation with DCI with Gradient Boost Tree by DisLib takesmore than 14 hours, while MED only takes less than 20 seconds on the same machine.
3.3 Partial Disentanglement Evaluation Metric
It is challenging to learn a fully disentangled representation by high-dimensional models withoutexplicitly encouraging disentanglement, especially when there are fewer independent data factors thanthe number of latent dimensions.
However, if a high-dimensional model has a subset of dimensionsthat disentangle
well, it is still worth studying.
This motivates us to design a version of MED toevaluate the partial disentanglement, which we name it as “Top-k MED".
[Figure]
S h a p e
S ca l e
P o s iti on_ X
P o s iti on_ Y
F ac t
o r
[Figure]
l e v a ti on
A z i m u t h
O b j ec t t yp e F ac t o r
[Figure]
The main difference is that we pick the most disentangled k dimensions for each factor andcompute MED on this subset of representations.
For each ground truth factor v j , G j ={ i | arg max m R im = j } is the set of latent dimensions emphasizing this factor.
Then we pickthe top k latent dimensions with the highest disentanglement score in each G j to construct a subset
P j = (cid:8) i | S
i ≥ S k , i ∈
G j (cid:9) .
Here, S i is the disentanglement score of the latent dimension
c
i definedin Equation 2.
S k is the k th highest disentanglement score in G j .
Finally we obtain the subset
P = ∪ j P j .
We take the sub-vector after selection ˜ c
= { c i } i ∈P as the representation to evaluate
MED.
The top-k MED is defined as MED (˜ c ) .
Understanding the learned representation
In this part, we qualitatively study the disentanglement of the learned representation by negative-freecontrastive learning with BYOL as an example.
Since contrastive methods are not generative models,it is hard to directly do factor-controlled pixel-wise reconstruction for visualization.
Instead, wemeasure the mutual information between the learned representations and ground truth factors, whichis also the foundation of our proposed MED metric.
More qualitative study is available in Appendix B.
4.1 Correspondence of Representation and Factor
Shape
Scale Position_X Position_Y Factor
S h a p e
S ca l e
P o s iti on_ X
P o s iti on_ Y
F ac t
o r
Co-occurrence of Mutual Information 0.
To understand the disentanglement of a representation model, a basicquestion is how representation dimensions correspond to factors ofdata.
After encoding an input image to a representation vector, wecompute the normalized mutual information (MI) between eachground truth factor and each representation dimension, i.e. R in
MED defined as in Equation 1, to measure the correspondence.
The mutual information between dimensions and the factors is included in Figure 1a and Figure 1b.
The heatmap is actually thetranspose of the importance matrix R ⊤ .
As shown by the brightnessof the entries, the informativeness of the latent dimensions variesgreatly.
In fact, we can identify three types of columns: columnswith single , multiple and no bright elements, corresponding to threetypes of latent dimensions: disentangled dimensions, entangled dimensions and uninformative dimensions.
The disentangled dimensions only capture information of one factor while the entangleddimensions encode multiple factors.
In contrast, the uninformative dimensions can not represent factors independently.
Note that the entangled and uninformative dimensions are non-negligible, hencethe representations are not fully disentangled.
However, by extracting the subset of disentangleddimensions we can derive a well-disentangled subspace.
First, we analyze the model on dSprites dataset.
dSprites has five factors (shape, scale, orientation,position_x, and position_y).
But the orientation is ill-defined with ambiguity.
For example, it is impossible to distinguish if a square rotates 0 degrees or 180 degrees.
As shown in Figure 1a, thedisentangled dimensions occupy a significant proportion, indicating an evident partially disentangledpattern.
We note that the degree of disentanglement varies on different datasets.
An example on
Cars3D is shown in Figure 1b.
Cars3D is a dataset with 183 different car objects rendered from 4elevations and 24 azimuths and the ground truth factors are not fully independent of Cars3D (see
Appendix B).
It is extremely hard to represent its azimuth and object id with few dimensions.
Thusits object-type row and azimuth row in Figure 1b are more spread out among multiple latent dimensions.
This also shows the difficulty of understanding the disentanglement of the high-dimensionalrepresentation model on complicated datasets.
Therefore, we will continue to conduct a quantitativeevaluation with our proposed MED metric in Section 5.
Such quantitative study on more datasets isavailable in Appendix B in the supplementary materials.
4.2 Uniqueness of Factor-Representation Correspondence
In the ideal pattern of disentanglement, a representation dimension should uniquely correspond to onlyone factor.
Now we show to what extent multiple factors are responded to by a single representationdimension in a well-disentangled subspace.
Given the mutual information between the representationand the i th factor, noted as I i , i.e., the i th row in the top-k version of Figure 1a, a good indicator ofthe uniqueness of factor-representation correspondence is the normalized co-occurrence of mutualinformation between the i th 1 factor and the i th 2 factor, which is defined as
[Equation]
where ˜ c is the representation after the selection process in Section 3.3.
We visualize the normalizedco-occurrence of mutual information among the four factors by the learned representation in Figure 2.
It agrees that a dimension usually encodes only one factor.
Moreover, it indicates that the learnedrepresentation tends to encode shape and scale together, which also agrees with the intuitive analysisof the independence of factor pairs.
For example, the shape and scale of dSprites objects are notdisentangled and independent because objects with the same scale value but in different shapes havedifferent pixel area.
[Figure]
4.3 Influence by Manipulating Factors
Another intuitive perspective to studying the relationship between representation and factors is theinfluence on representation when manipulating the factors.
Given that the original representationvector dimension is much higher than the number of factors, we first make the representation morecompact to have a more concise illustration.
Here, we reduce the representation dimension by theselection process of top-k MED described in Section 3.3.
Figure 3 shows the result of representation vector variation when changing only one factor at once.
We take BYOL on dSprites as a representative.
Here we set k = 2 for top-k MED, i.e., we pickthe 2 most disentangled dimensions for each factor and derive a 10-dim representation.
Givenimages with one factor traversing through all its support values while fixing the other factors, wegenerate the 10-dim representation vectors from them.
Then, we compute the variance of each ofthe 10 dimensions across the images, leading to 10 scalars.
The larger the variance is, the morethat dimension responds to the factor.
Figure 3(a), (b) and (c) show how the reduced representationvector changes when manipulating position_x , position_y , scale factor respectively.
Note that we set k = 2 , therefore it shows good disentanglement that exactly two representation dimensions havehigh variation.
However, in Figure 3(d) we show a failure mode of the ill-defined factor orientationthat change of factor causes multiple dimensions of reduced representation to have large variations,indicating that this factor is represented in an entangled way.
From the results, we observe thatmanipulating different well-defined independent factors causes evident variance on disjoint sets ofdimensions.
And it demonstrates the existence of a well-disentangled subset of latent dimensions.
We also conduct a similar qualitative study where the representation dimension is reduced by theunsupervised PCA technique and that shows the similar pattern observed here.
The details areprovided in Section B.4 of the Appendix in the supplementary materials.
Quantitative Evaluation
In this section, we conduct quantitative studies on the disentanglement property of contrastive learningmethods.
We first introduce the experiment setup in Section 5.1.
Then we show quantitative resultsunder both MED and existing disentanglement metrics in Section 5.2 which shows the disagreementof existing metrics to support the necessity of proposing MED.
Finally, we make a full quantitativebenchmark of methods of interest with MED in Section 5.3 and ablation study about the dimensionof the representation model in Section 5.4.
More quantitative study is available in Appendix C.
5.1 Experiments Setup
The details for reproducibility are introduced in Appendix Section A included in the supplementarymaterials.
Here we provide a brief description of the setup of the experiments.
Datasets.
Representation disentanglement is usually evaluated on synthetic datasets, such asdSprites (38), Cars3D (41), Shapes3D (6), and SmallNORB (31).
Besides those datasets, wealso include a real-world dataset CelebA (34).
CelebA contains human face images with 40 binaryattributes.
The attributes include fine-grained properties of the human face, such as whether wearingglasses or having wavy hair.
We include the details of dataset factors in the supplementary materials.
Evaluation Protocol.
We conduct experiments with both MED and the existing metrics to revealthe disagreement between existing metrics.
Then we use MED as the main metric to study thedisentanglement of contrastive learning methods.
The implementation of evaluation metrics isadapted from the protocol provided by DisLib (36).
All results are calculated with three randomseeds and we report both the average score and the standard deviation.
More details are introduced in
Appendix Section A.2 in the supplementary materials.
Reference Methods.
We investigate most of the popular disentangled representation learning methodsas studied in the standard benchmark of DisLib (36).
Besides, we also compare with a recentlyproposed ICA method called ICE-BeeM (24).
Since we do not assume the ground truth factors areknown during the training, we use its unconditional version.
We term it EBM (energy-based model).
For the contrastive learning methods, we take not just negative-free methods, such as BYOL (42),
Barlow Twins (54) and SimSiam (8), into the benchmark, but also those using negative samples, suchas MoCo and MoCov2 (15).
Model Implementation.
All methods use a shared architecture of encoder network as explainedin the appendix in supplementary materials.
The latent dimension of contrastive learning methodsis set to 1000 since they require a high-dimensional latent space to work.
For other methods, thelatent dimension is set to be 10 on synthetic datasets as in DisLib and 128 on CelebA dataset whenevaluating with MED.
For the evaluation with Top-k MED, the dimension of all methods is set to be1000-d for fairness.
On dSprites, Cars3D and SmallNORB, we acquire checkpoints from DisLib ifthey are provided.
We train our checkpoints on CelebA and Shapes3D.
5.2 Disagreement of Existing Metrics
As there is not a uniform definition of the “disentanglement” concept yet, existing disentanglementmetrics are motivated by different desired properties of a disentangled representation.
We relyon them before because they show good agreement as proven by the large-scale experiments in
DisLib (36).
However, when we extend the disentanglement study beyond low-dimensional scenarios,this agreement does not stand anymore.
We select representative low-dimensional VAE-basedmethods and a high-dimensional BYOL model to evaluate on a representative dataset, the SmallNORBdataset.
The results are in Figure 4, which show a significant disagreement among metrics on BYOL,while they agree on the low-dimensional VAE methods.
Aligned with our analysis in Section 3.2,
[Figure]
BetaVAE score, FactorVAE score, and DCI overestimate the disentanglement degree of the highdimensional model while MIG and SAP underestimate.
Since previous metrics fail to evaluatehigh-dimensional models, we opt to use MED as the main evaluation metric in the following sections.
Please refer to Appendix C and G for more evidence of the disagreement of existing metrics.
[Table]
5.3 Disentanglement Benchmark with Contrastive Learning Methods
For the benchmarking of disentanglement, we use both MED and the partial version of MED, i.e.
Top-k MED.
For Top-k MED, we set the MED partial evaluation hyperparameter k = 2 for dSprites,
Shape3D, and SmallNORB, and k = 3 for Cars3D and CelebA. The values of k are chosen suchthat the selected dimensions are roughly close to the latent space dimension of the low-dimensionalreference methods.
And we extend the latent dimension of all methods to 1000 when evaluating
Top-k MED for fairness.
We note that the hyperparameters we tried fail to train good EBM weightson Cars3D and CelebA, so we keep that empty.
The results are in Table 1.
We also encourage to readthe results together with existing metrics in Table 4 in Appendix C.
In the upper part of Table 1, we show the MED score for previous disentangled methods as well ascontrastive methods.
We find that contrastive methods achieve lower disentanglement scores on 3of the 5 datasets (Shapes3D, Cars3D, SmallNORB).
Contrastive methods achieve slightly higher disentanglement scores on CelebA. On dSprites, some negative-free contrastive methods (SimSiam,
BYOL) achieve scores close to SOTA, but the other contrastive methods’ score is much lower.
Insummary, in most cases, contrastive methods achieve inferior disentanglement property than the bestmethods; in few settings, it achieves scores comparable to SOTA scores.
[Figure]
This result seems disappointing, but not surprising, sincecontrastive methods are not explicitly designed to maximize the feature disentanglement.
Further, since the underlying number of factors is usually quite small, on theorder of 10, the 1000-d feature space will likely have dimensions that are not related to the ground truth factors, ora combination of the ground truth factors.
This result doesnot contradict with Zimmermann et al.
(56) since (1) theirconclusion is the learned feature is a linear transformationof the ground truth factors, which doesn’t necessarily disentangle (2) they use augmentations on factors that cannot be done in practice.
The lower part of Table 1 shows top-k MED measurements on various methods.
We find that contrastive learning methods (especially the negative-free ones) in generalshow a better disentanglement in a selected subspace andthe disentanglement is stronger than the reference methods.
This shows that there exists a subspace in the learnedrepresentation that is well disentangled.
Moreover, whenwe compare the subspace in contrastive methods (gray partin the lower section of Table 5.3) to the traditional approach that directly trains a low-dimensionallatent space (non-gray part in the upper section of Table 5.3), we find that the disentanglement of theformer is usually better than or on par with that of the latter.
This means that we probably shouldnot constraint the dimension of the latent space and require it to be fully disentangled, but rathershould encourage to use high dimensional latent spaces and only require it to have a subset with gooddisentangle properties.
To conclude, we find that contrastive methods, especially the negative-free ones, do not have a fullydisentangled feature space, but there exists a subspace that is well disentangled, sometimes muchbetter than the previous SOTA approaches, which may serve as a proxy between CL, which requireshigh dimension to train, and a more compact low-dimensional model from its subspace.
5.4 Influence of Dimension
As representation dimension is found as a core variable in disentangled representation learning, weevaluate the influence of dimension over disentanglement by top-2 MED on dSprites with BYOL asan example versus VAE-based methods.
The results are shown in Figure 5.
We also study its influenceon the full space regarding MED in Figure 12 in Appendix C. We find that the BYOL’s top-2 MEDscore increases as the latent dimension increases and plateaus at 512 dimensions.
This is consistentwith previous literature on the difficulty to train contrastive models with low latent dimensions (14)which shows lower latent dimension leads to inferior linear accuracy.
We further show that a lowerlatent dimension leads to a less disentangled subspace as well.
On the contrary, we also note thatthe VAE methods fail to scale to higher latent dimensions.
We have spent a reasonable effort to tune
VAEs’ hyperparameters, but we still find a large gap between higher-dimensional VAEs and their10-dim version.
This suggests the gap between existing disentangled representation methods and thereal-world data complexity, which can not be represented in a limited dimension anymore.
Conclusion
In this paper, we provide an empirical study of the disentanglement property of contrastive learningwithout negatives for the first time.
In the high-dimensional space, we find the difficulty of adoptingthe existing disentanglement metrics.
Therefore, we propose a new metric MED and top-k MEDto evaluate disentanglement based on mutual information.
The evaluation shows that even withoutnegative samples, contrastive learning can learn a well-disentangled subset of representation.
Recently,the study of contrastive learning, or general self-supervised learning, is still motivated by empiricalobservations.
We hope our work can reveal some clues to motivate future theoretical justifications.
A Reproducibility
In this section, we provide the information required to reproduce our results reported in the maintext.
And we commit to making the code implementation and evaluating checkpoints public.
Ourexperiments are run on a machine with AMD Ryzen Threadripper 3970X 32-Core Processor and
GeForce RTX 3090 GPU.
Contrastive Learning implementation For the implementation details of contrastive learning,please refer to Appendix A.1.
The model architecture, training setups, and dataset preprocessing areall explained in detail.
Our implementations are based some public and official implementations of
MoCo/MoCov2 2 , BYOL/ SimSiam 3 and Barlow Twins 4 .
VAE methods implementation For evaluation on synthetic datasets, i.e., dSprites, Cars3D, SmallNORB, and Shapes3D, the disentanglement score is from the original logs of DisLib Locatello et al.
In the released logs, each method has different training configurations, and our reported result isfrom the configuration with the highest average performance overall provided random seeds.
Forevaluation on CelebA dataset, we follow an open-sourced implementation in Pytorch 6 and align theencoder architecture of all methods to be the same as described in Appendix A.1.
For the results on
Shapes3D, because DisLib does not release the pretrained checkpoints, we use the same open-sourcedimplementation to reproduce with the configuration indicated by DisLib.
Parameters are kept asthe default well-tuned version in the provided implementation.
When the latent dimension is 1000,training of BetaTC VAE will collapse with the default hyperparameters, we have to decrease the β to3.0 to work it around.
GAN methods implementation Limited by the text length, we do not include the performanceof GAN methods in the main text, but we will report some in the following appendix content.
It ishard to include GAN methods’ performance in the benchmark as the training is not always stable andthe discriminator weights are usually not provided in many public codebases.
When evaluating onsynthetic datasets, the FactorVAE scores of InforGAN, IB-GAN, and InfoGAN-CR are provided inthe paper of Lin et al..
But the evaluation of other metrics in Lin et al.
uses a not aligned settingswith Locatello et al.
, so we check its officially release 7 to reevaluate the provided implementationand model weights under the unified evaluation setup.
We perform the same evaluation process forresults on the CelebA dataset.
Energy-based Model (EBM)
We refer to the implementation of ICE-BeeM (24) for this method.
We use the officially released codebase for it 8 .
The encoder implementation has been aligned withour default already.
The only modification we make is to use the unconditional version instead of itsdefault conditional version in loss computation to satisfy the fully unsupervised settings.
Please referto the runners/real_data_runner.py file of the codebase for details.
Evaluation Protocol For MED, we first compute MI following the implementation of MIG by
DisLib (36).
Then we calculate the entropy disentanglement score in the same way as the DCI Disentanglement score in DisLib.
For other disentanglement metrics evaluation, we use the implementationof DisLib.
The settings of some important parameters rather than our proposed MED are provided in
Appendix A.2.
A.1 Implementation of contrastive learning model
Architecture To make a fair comparison with previous methods, we follow the encoder architecturein Factor VAE (25).
The pipeline details are shown in Table 2.
After each convolutional layer inthe figure, there is a ReLU activation layer and a group normalization (group number = 4) layerfor BYOL.
So, the encoder is a stack of (Conv-ReLU-GN) blocks.
For other contrastive learningmethods, we keep the default batch normalization to replace GN.
By default, the final output channel number is 1000, i.e, D = 1000 .
For other details of contrastive learning methods, we follow theconvention in their official implementations.
Besides the representation network (encoder), BYOL also has a projector network and a predictornetwork.
Both of them consist of a pipeline “Linear −→ BN −→ ReLU −→ Linear”.
The projectiondimension is 256, and the hidden dimension of the projector is 4096.
The predictor keeps a 256dimensional feature vector in its pipeline.
[Table]
Training settings We make minor modifications to the training setting of default BYOL to apply tocontrastive learning methods without negative samples.
For training on all datasets, the images areresized to 64x64.
For data preprocessing, we copy 1-channel images of dSprites and SmallNORB to3-channel.
During the training stage, we use such a pipeline of augmentation (in PyTorch -style):
RandomApply(transforms.
ColorJitter(0.8, 0.8, 0.8, 0.2), p=0.3)2.
RandomHorizontalFlip()
RandomApply(transforms.
GaussianBlur((3,3), (1.0, 2.0)), p=0.2)
RandomResizeCrop(size=(64, 64), scale=(0.6,1.0))5.
normalization
For the normalization, the pixel value of images from dSprites and SmallNORB is uniformly normalized from [0,255] to [0,1.0].
For Cars3D, Shapes3D, and CelebA, we adopt the commonly used
Imagenet-statistic normalization for preprocessing the image values.
During training, we use Adam optimizer by default, whose learning rate is 3 e − 4 without weightdecay.
The batch size is set to be 512 without exceptional notation.
For evaluation on dSprites,
Shapes3D, and CelebA, we select the weights after training for 15 epochs for evaluation.
We selectthe weights after training for 140 epochs for evaluation on Cars3D and the weights of the 200th epochon SmallNORB considering the small scale of these two datasets.
To decrease the influence of randomness, we train each model configuration multiple times withdifferent random seeds (seed=0, 1, 2).
We report the average and standard deviation.
To be precise,as our implementation is based on Pytorch, we initialize the libraries of numpy , torch , torch.cuda ,and random with the same random seeds.
A.2 Evaluation Metrics
In the main text, we compare the evaluation metrics provided in the DisLib protocol with our proposed
MED metric.
Here we provide more details about them.
Moreover, we would conduct evaluationsunder all of them in the next section.
BetaVAE Metrics Introduced in Higgins et al.
(16), BetaVAE score assumes each dimensioncorresponds to one category in a linear classifier.
Representations are obtained after the generatedsamples with only one factor fixed.
Calculating the summation of the divergence between differentrepresentations and putting this result into a linear classifier, we train a model that possibly outputsthe corresponding k.
The accuracy of this linear model is the value of BetaVAE metric.
[Table]
FactorVAE Metrics Kim and Mnih (25) argues the BetaVAE score has the tendency to fail intoa spurious disentanglement and proposes a new metric based on a majority vote classifier.
Representations are obtained after the generated samples with only factor k fixed.
Normalizing eachdimension in representations in terms of standard deviation.
Index of dimension with lowest variancesof normalized representation and the factor index k is the input/output of the linear classifier.
Theaccuracy of the classification is the FactorVAE score.
Mutual Information Gap Chen et al.
(7) assumes the disentanglement model has the property thatmost information of one specific factor is contained in one dimension or a group of certain dimensions.
The mutual information gap is the summation of the difference between the highest and secondhighest normalized mutual information between a fixed factor and dimensions in representation.
Theformula can be illustrated as below:
[Equation]
Where K is the overall number of ground truth factors.
v is the latent representation and z k is thefactors of latent variables and
j k = arg max j I ( v j , z k ) .
DCI disentanglement As Eastwood and Williams (11) suggests, the disentanglement is measuredby the entropy of relative importance for each dimension in predicting factors.
First, we haveto know the importance of each dimension of the representation for predicting each factor.
Theimportance is determined by a regressing model such as Lasso or Random Forest in the original
DCI implementation (11) or Gradient Boosting Tree in DisLib implementation (36).
We note theimportance matrix R where R ij is the importance of the i-th dimension in prediction the j-th factor.
Then disentanglement score for the i-th dimension is defined as D i =
(1 − H K ( P i )) where H K ( P i ) =
− (cid:80) K − 1 k =
P ik log K P ik denotes the entropy and P ij =
R ij / (cid:80) K − 1 k =
R ik denotes the normalizedimportance of i-th dimension in prediction the j-th factor.
Finally the overall disentanglement score iscalculated as D = (cid:80) i ρ i D i where ρ i = (cid:80) j R ij / (cid:80) ij R ij is the weighting of the each dimension’sinformativeness in representing factors.
SAP (29) proposes the Separated Attribute Predictability (SAP) score.
A score metrics is computedwith classification score of predicting j th factors on i th dimension as the ij th entry.
SAP is the meanof the difference between the highest and second-highest scores for each column.
We follow the implementation provided by DisLib (36) for the evaluation protocol.
Despite exceptions,the evaluation batch size is 64, the prune_dims.threshold is 0.06.
If a classifier is required to be trainedduring evaluation, num_train is 10000, and num_eval is 5000.
For Mutual information computation,the discretizer function is the histogram discretizer, and the number of bins in the discretization is20.
For the evaluation of MIG and SAP on dSprites, SmallNORB, Cars3D, and Shapes3D, BYOLrepresentation vectors are reduced to 10 dimensions by PCA to be aligned with other methods.
Forthe evaluation of MIG and SAP on CelebA, to have a fair comparison, the representation vectorsof all methods are reduced to 40 dimensions.
For the implementation of our proposed MED, thebasic logic is the same as DCI Disentanglement, but we replace the classifier output with the mutualinformation based scores.
[Figure]
S h a p e
S ca l e
P o s iti on_ X
P o s iti on_ Y
F ac t
o r
[Figure] [Figure]
[Figure]
B More Qualitative Study
Limited by the main text length limitation, we provide more qualitative studies about the disentanglement property shown by the contrastive learning here.
We still use BYOL as an example of thenegative-free contrastive learning methods.
B.1 Importance Distribution by DCI
[Figure]
In the main text, we concisely talked aboutthe potential variables introduced by thelearnable model under some metrics.
Herewe show an example for the widely used
DCI Disentanglement metric.
We follow DisLib to use Gradient Boosting Treeto produce the Importance estimation between each factor and each latent dimension.
All parameters are set the same asits default protocol.
The visualization isshown in Figure 6.
Compared with themutual information distribution shown in
Figure 1a, the importance distribution isobviously much more sparse.
Sparsity isencouraged during constructing the GBTregressor.
However, the observation canlead to the misunderstanding that the correlation between factors and latent dimensions is sparse which is not true.
By using the pure measurement without involving additionaladaptive models, such problem will not be raised in the proposed MED metric.
B.2 Mutual Information Heatmaps
We compute MI between each latent dimension and each generative factor and visualize them byheatmaps, which offer us an intuitive picture of the learned representation space.
For completeness,we show the MI heatmaps of SmallNORB, Shapes3D, and dSprites with all factors in Figure 7a,
Figure 7b and Figure 7c respectively.
We can see that the disentangled pattern described in the maintext still emerges.
There is a group of columns brighter than others in each row, and these groups donot overlap for most rows.
However, we find that some latent dimensions may emphasize more thanone factor.
We provide a more detailed analysis from the perspective of factor co-occurrence on thisphenomenon in sectionB.3 below.
B.3 Co-occurrence of Factors
To understand to what extent one dimension of the learned representation would respond to morethan one factor, we make the co-occurrence of mutual information to factors on more datasets here.
The visualizations are shown in Figure ,8b, Figure 8a and Figure 8c on SmallNORB, Cars3D, and
Shapes3D respectively.
SmallNORB Though most non-diagonal entries have very low co-occurrence of mutual information,two pairs of factors show slightly higher co-occurrence.
They are “azimuth-elevation” and “instancecategory-lighting”.
After investigating the dataset, we find the two pairs of factors are not fullyindependent.
Figure 9 show some samples with corresponding factors manipulated.
We could seethat the elevation and azimuth are not fully independent.
And the correlation between the instancecategory and the lighting factor is even more obvious because the lighting condition is sensibly relatedto the shadow around the object, whose distribution and shape are highly determined by the instancecategory.
Cars3D Only one pair of factors show some co-occurrence, i.e. “elevation-object type”.
Werandomly selected samples from Cars3D by different object types and elevations, as shown in
Figure 10.
It shows that with the same value of elevation, samples of different object types havedifferent visual elevations.
So these two factors are not fully independent.
This might explain theslightly higher co-occurrence of mutual information between this pair of factors.
Shapes3D
The result shows relatively bad disentanglement.
To be precise, some factor pairs showlow mutual information co-occurrence as expected, such as the color factors of floor, wall, and objectand the pair of “object color - azimuth”.
But the MI co-occurrence of “wall color - object size” and“object color - object size;” are higher than we expected as we did not recognize their high dependence.
This result might relate to our model’s relatively poor performance on Shapes3D.
[Table]
dSprites We omit the factor orientation in the visualization at main text for clarity.
Here we showthe results including all factors.
For dSprites dataset, the value of orientation is highly correlatedwith shape .
This is because that different shape has different symmetry properties for rotation.
Asa result, dimensions that encode information of shape tend to capture information of orientationsimultaneously.
This explains that, in Figure 7c, some dimensions have two brighter entries at “shape”and “orientation” rows.
Thus, it is not surprising that the “shape - orientation” cell is brighter thanother off-diagonal elements in Figure 8d.
B.4 Manipulating Factors
[Figure]
In the main paper, we studied the influence to representation by manipulating the factors, wherethe representation is reduced by selecting dimensions as in calculating Top-k MED.
Here, we dothe qualitative study of the influence on representation by manipulating factors in another way butstill on dSprites.
To make the original high-dimensional representation space more compact, weuse the unsupervised dimension reduction by PCA instead, which is more general when the factorpattern is unknown.
Here, we reduce the representation dimension by PCA to 10.
Note that sincethe PCA operation mixes the original latent space with a linear combination, it might destroy theexisting disentanglement property in the high dimensional space, or enhance the disentanglementif the original high dimensional space is a linear combination of the ground truth factors.
But suchinfluence is usually considered secondary to the disentanglement learned by a model.
No matter which case, if the dimension-reduced representation shows disentangled properties, the original spaceat least captures linearly transformed ground truth factors, and the dimension reduction techniquessuch as PCA can make the representation more compact in a qualitative study.
Figure 11 shows the result of representation vector variation when changing only one factor at once.
Given three images with only one factor’s value being different, we generate the 10-dim representationvectors from them.
Then, we compute the variance across the three vectors, leading to 10 scalars.
The larger the variance is, the more that dimension responds to the factor change.
Figure 11(a) and(b) show how reduced representation vector changes when manipulating position_x and position_yfactor respectively.
It shows good disentanglement that only one representation dimension has highvariation.
However, in Figure 11(c) we show a failure mode of the ill-defined factor orientation thatchange of factor causes both the 6th and the 9th dimensions of reduced representation to have largevariations.
From the results, we observe that manipulating one well-defined independent factor causesevident variance in only one dimension.
And it shows that we could make the learned representationvector more compact by unsupervised dimension reduction.
C More Quantitative Results
[Figure]
We present the complete disentanglement benchmark results in the next section because of length
[Figure]
restrictions.
Moreover, we plot the MED version of Figure 5 to reveal the significance of the latent
[Figure]
dimension for disentangled representation learning.
[Figure]
C.1 Full Disentanglement Benchmark
[Figure]
In the main text, we evaluate the disentanglement under
[Figure]
our proposed MED metric for full space and top-k subspace on multiple datasets.
In this section, to provide amore complete understanding of the disentanglement property of contrastive learning without negatives, we reportthe disentanglement scores with other metrics, such as
FactorVAE score, BetaVAE score, MIG, SAP, and DCI
Disentanglement, here.
For the results of VAE-based methods, as the large-scalebenchmark of Locatello et al.
(36) provides the originallogs on dSprites, Cars3D, and SmallNORB datasets, wesimply report the performance of the best configuration.
The original logs on Shapes3D are not available, so wetrain and evaluate on Shapes3D by ourselves for the MEDscores.
For scores under other metrics, we report themedian disentanglement scores.
Some results are from Locatello et al.
(37) but the std error is not available.
Themedian performance of SlowVAE is from its original paper (27).
For the results of CelebA, the result of InfoGANCR is from its officially released checkpoint without availability to the std error.
For other methods, we report themean value of our trained weights over three random seedsas default.
Because the evaluation of DCI is extremely time-consuming, around 14 hours for a 1000-dmodel, we only take BYOL as an example here for negative-free contrastive learning methods.
Allresults are combined and shown in Table 4.
We also include traditional metrics results on theirselected top-k subspace for contrastive learning methods.
Same as the analysis we provide in the main text, the results show significant disagreement amongthe existing metrics.
To be precise, for those metrics (BetaVAE score, FactorVAE score, DCI
Disentanglement) using a learnable model such as a regressor or classifier, the high-dimensional
BYOL model achieves a significant advantage.
However, for the metrics relying on only one ortwo dimensions to reveal the connection between a latent dimension and a factor (MIG and SAP),
BYOL’s performance is not that impressive anymore.
Finally, the result on CelebA shows the great robustness of BYOL’s learned representations to bedisentangled on real-world datasets.
Yet, the large gap between the score of CelebA and those onsynthetic datasets emphasizes the difficulty of learning disentangled factors on real-world images.
It
[Table]
is hard to empirically study whether it is the high dimension that gives BYOL advantages on somemetrics because the nature of BYOL makes it hard to be trained with a small latent dimension tomake a comparison.
In the last rows for each dataset, we show previous disentanglement metrics scores on the top-ksubspaces of BYOL.
We find that BYOL (top-k) has performed better than or on par with unsupervisedmethods with low latent dimensions.
Note that Ada-GVAE, SlowVAE, and EBM are (weakly)supervised methods.
The SOTA performance justifies our claim of the well-disentangled subspaces.
MIG and SAP require a large gap between the two most vital representation dimension-factorresponses, measured by mutual information and classification accuracy.
But out of 1000 dimensions,we choose the k most relative ones.
Hence the gap ought to be minimal.
Consequently, BYOL(top-k) receives lower MIG and SAP scores.
Except for dSprites and CelebA, BYOL (top-k) havelower DCI scores.
The primary cause of this is the top-k process’s massive information loss, whichcompletely obliterates the original encoding pattern.
As a result, the gradient boosting tree (GBT)developed throughout the DCI process tends to “borrow" information from latent dimensions thatemphasize additional relative factors to classify some complex factors.
The “information-lending"dimensions then become less disentangled since they are also important for predicting those complexfactors.
Taking the Cars3D dataset as an example, we set k = 3 for top-k MED.
To predict thefactor object type , the GBT needs to classify 183 types of cars from only 9 dimensions with only3 emphasizing object type .
This forces GBT to utilize information from elevation since these twoattributes are correlated (see Appendix B.3).
However, we keep all they have learned from trainingfor other low-dimensional methods.
Therefore, the information loss problem does not apply to them.
In summary, we obtain the following conclusions regarding metrics and top-k subspace with thecomplete benchmarks.
(1) The scores of the whole space of BYOL empirically verify the disagreementof previous metrics.
(2) Traditional metric scores on the top-k subspace support the well-disentangleproperty of the subspace.
C.2 Infuence of Dimension on MED
In the main text, we show the influence of dimension on top-k MED.
To make the conclusion morerigorous, we report the correspondence analysis for MED in Figure 12.
Combining with Figure 5,we cover disentanglement performance for BYOL with lower dimensions and VAEs with higherdimensions.
We find that as the dimension grows, the MED scores evolve in the same pattern as top-k
MED scores.
This suggests that the dimension of representation influences both overall and partialdisentangle properties.
Thus representation dimension should be considered a core parameter fordisentangled representation learning.
D Ablation Study
Limited by the main content page length, we put some additional ablation studies here to helpbetter understand the influence of important inductive bias of BYOL when studying representationdisentanglement.
[Table]
We experiment with five normalization layers configuration in the encoder network on the dSpritesdataset.
The results are shown in Table 5.
For group normalization, we set the group number to 4.
OndSprites, we find the commonly used BN decreases the disentanglement performance.
By keeping thebatch norm in the projector and the predictor, removing the batch norm in the encoder will not causethe model to collapse, which agrees with the observation in previous works (42).
On the contrary,replacing batch norm in encoder with group norm or layer norm will increase the representationdisentanglement while achieving similar accuracy in downstream factor prediction.
We notice that asimilar phenomenon has been discovered before in supervised representation disentanglement.
Forexample, Bau et al.
(3) discovered that a network trained with batch normalization layers has lessinterpretable (disentangled) neurons.
On the other hand, instance norm (50) completely breaks the contrastive learning process.
We still do not fully understand this behavior, but we hypothesize that itmay be caused by the shared batch statistics that make it hard for a feature to be aligned to the groundtruth factor.
E Limitations
Our work still has some limitations, especially considering the design of contrastive learning methodsstill depends on heavy empirical practice.
For the fair comparison in the benchmark, we use ashared encoder architecture for all methods but they may still have other inductive bias potentiallyinfluencing the results such as the hyperparameters in VAE-based methods and contrastive learningmethods.
We select the normalization in BYOL as an example in the ablation study above showingthat such inductive bias can make influence over the evaluation results.
But we can not do theablation study on all possible inductive biases.
We basically inherit the available best settings from DisLib (36) if possible and the settings from the public official implementations of other methods.
All hyperparameters and details we customize have been indicated in the Section of Reproducibility.
F MED Details
The main text briefly introduces MED’s motivation and evaluation process.
In the following section,we will first explain why we can trust MED to measure disentanglement properties.
Then we discussthe drawbacks of previous metrics in high dimensional space in detail.
Finally, we provide moreempirical results on the properties of the top-k subspace.
F.1 Intuition and Rationale
MED is designed to measure the disentanglement property following the DCI framework (11), i.e.,the degree to which a representation dimension only responds to a single generative factor.
Thenotion of disentanglement implies two requirements.
The metric should (1) accurately measure thelatent dimensions’ “response” to factors, and (2) demonstrate the singleton correspondence to factorsof latent dimensions.
To satisfy the above two properties, we combine two widely-accepted tools, mutual information(MI), and entropy.
By definition, MI is a standard measurement of information shared by variables.
Therefore the importance matrix calculated by MI (see Eq. 1) indeed indicates the correlationbetween representation dimensions and factors.
As for the second requirement, we show thatthe disentanglement pattern matches the mechanism of entropy in 3.2 because it indicates theinformativeness a dimension has for factors.
And by normalizing entropy across factors, we measurethe relative importance of one dimension to different factors, which fits the intention of estimatingdisentanglement.
Here we present some examples to verify.
According to the row distribution defined as P ij =
R ij / (cid:80) K − 1 k =
R ik , a perfectly disentangled dimension induces that P ik = 1 , P ij ̸
= k = 0 for a certaink , yielding S ( i ) =
1 (see Eq. 2) .
On the contrary, a fully entangled dimension captures combinationinformation of all factors, corresponding to a uniform P i ′ · , yielding the minimum S ( i ′ )
= 1 − log K ,where K is the number of ground truth factors.
In summary, the two requirements naturally followthe definition of MI and entropy.
Therefore MED is indeed a metric of disentanglement property.
F.2 Limitations of Previous Metrics in High-Dimensional Space
In 3.2 we report our observations of the flaws of existing metrics.
Here we will discuss their limitationsin detail following the two requirements described in F.1.
In addition, we will show how MED canresolve the issues of previous metrics.
Metrics based on learnable classifiers, namely the BetaVAE score and FactorVAE score , fail tosatisfy both of the requirements.
As the main text describes, a higher dimension with redundantinformation makes it easier to overfit the classifier.
As in evaluating disentanglement, standardprotocol draws training and testing sample points from the same distribution.
Such overfit canmake a huge advantage in final scores.
Therefore the classifier developed during evaluation tends tooverestimate the “response”.
Thus their mechanisms are not aligned well with the disentanglement inhigh-dimensional cases.
[Figure]
S h a p e
S ca l e
O r i e n t a ti on
P o s iti on_ X
P o s iti on_ Y
F ac t
o r
[Figure]
[Table]
Metrics like MIG and SAP are based on an extended intention of explaining disentanglement : notonly should a dimension correlate to only a single factor but a factor should be responded by onlya single dimension.
In fact, they indicate the degree to which a factor is only captured by a singlerepresentation dimension, which is the completeness property in the DCI framework (11).
Forcontrastive learning methods, the dimension of representations (about 1000) is much larger than thenumber of factors (less than 10).
This increases the conflict of completeness and disentanglement .
Inmore general cases where the factor number is unknown, these metrics would give a bonus to themodels that happen to have dimensions close to the underlying factor numbers.
For DCI Disentanglement Score , the first prerequisite is also not met.
The learnable regressor suchas GBT (see Appendix A.2) tends to underestimate information captured by some dimensions since itencourages sparsity.
The learnable module in DCI also makes unfair comparisons when the modelrepresentation dimension varies.
MED avoids the problems above by the following properties.
Firstly, MED calculates MI betweenfactors and representation dimensions one by one.
Thus it is robust to dimension growth and MIis fair for different model dimensions.
Moreover, MED focuses on the entropy of the rows of R ,decoupling disentanglement and completeness .
F.3 Properties of the Top-k Subspace
We propose the top- k selection process to study the partial disentangle properties.
Here we providemore details.
We first visualize the top- k selection process based on the heatmap of importance matrix.
Then we will discuss the effects of k and the downstream performance comparison of thetop- k subspace and full space.
Visualization of the top- k Process We describe the top- k process in the Section 3.3.
Herewe colored the picked dimensions yellow in the Figure 13.
It shows that most picked dimensions only have one entry much brighter than the others, indicating their disentangledproperties.
Since orientation is ill-defined in dSprites, BYOL tends to encode it with shape .
Hence the most disentangled dimensions for shape also capture information of orientation .
[Figure]
Ablation on k
The parameter k is designed as an evaluation choice.
For analytical purposes, we choose thevalue of k to match the dimension of VAE methods inthe main text.
To investigate the influence of k , we setk = 1 , 2 , 10 , 50 , 100 and evaluate the top- k MED scoresof 1000-d VAEs and BYOL on dSprites.
The results areshown in Figure 14.
We find that with different valuesof k , the ranks of top- k MED scores remain unchanged.
Since the top- k process is a supervised greedy selection,as k grows the scores decrease.
In conclusion, top- k MEDcan measure the subspace disentangle properties with awide range of k .
Therefore we can choose k according toactual needs.
Downstream performance of top- k subspace We study the disentangle properties of the fullspace and top- k subspace in 5.3.
Here we analyze the downstream performance of these spaces.
We train a Logistic Regressor to predict each factor from the full representation or the top- k pickedrepresentation.
Then we take the mean prediction accuracy of all factors as the measure of downstreamperformance.
We compare the downstream performance of full space of VAEs and BYOL, and top- ksubspace of BYOL on multiple datasets.
Here we set k = 2 for dSprites, Shape3D, and SmallNORB,and k = 3 for Cars3D and CelebA to match the dimension of VAEs.
The results are in Table 6.
We note that, except on CelebA, performance drops non-negligibly when completing downstreamtasks with the top- k representations.
This suggests that the top- k subspace is well-disentangled butother dimensions also contain factor-related information.
It is unsurprising because we cut downthe dimension drastically (from 1000 to 10 on synthetic datasets and from 1000 to 100 on CelebA) by subscribing.
However, we find that the top- k spaces learned by contrastive methods achievebetter or compatible performance compared with the full spaces of VAEs on 3 out of 5 datasets(dSprites, Shapes3D, and CelebA).
It is reasonable that a high-dimensional model can be better fordownstream tasks as the downstream model can be trained to leverage each little piece of informationfrom the input representations.
But we get a message from the results that by drawing a subspacefrom the powerful CL-trained model, its performance on downstream tasks can be comparable to orsuperior to the low-dimensional models.
And the selected low-dimensional model can have muchbetter compactness and interpretability.
G Disagreement of existing Metrics
Limited by the page limit, in the main paper content, we can only briefly explain the disagreement ofexisting disentanglement metrics in Section 5.2.
Figure 4 shows the disagreement of FactorVAE score,
BetaVAE score, MIG, SAP, and DCI Disentanglement when the high-dimensional representationmodel joins the comparison.
In the background that there is no uniformly recognized definition of“disentanglement”, the agreement of existing metrics and their “basically shared” definition or expectation of “disentanglement” is one of the foundations of quantitatively evaluating disentanglementin this area since DisLib (36).
However, the results in Figure 4 break the belief.
Given the fact thatself-supervised representation learning always requires a high dimension to be trained, we think theproposal of MED is necessary if we want to extend the study of disentangled representation learningto complicated real-world datasets and self-supervised representation learning methods.
[Figure]
[Figure]
Here, we redraw that figure with the value of multiple metrics on the same model in a subfigure in
Figure 15.
Moreover, we note by including multiple metrics of different meanings of scale into onesubfigure, Figure 15 may cause confusion.
So we make a visualization to show the disagreement ofexisting metrics by ranking different methods with them.
The results are shown in Figure 16.
If allmetrics agree perfectly, there should be no relative ranking switch.
However, the frequent switch,especially with BYOL being taken into comparison, strongly suggests their disagreement.
Results in
Table 4 further show that different metrics may make completely different rankings with large valuegaps.
All these results in this section show the obvious failure of using existing metrics to evaluatehigh-dimensional representation models.
Together with the logistic analysis in Appendix F and thequantitative results in Appendix C, these results suggest the necessity to propose a new metric like
MED to make it possible to introduce CL into the disentanglement benchmarks. [Table] [Table]
H Latent Traverse on Shapes3D
Because CL methods are not generative models and have high dimensions, they are hard to be adoptedfor visualization with latent traversal.
We thus adopt VAE-based generative models here to performlatent traverse to show that MED can provide evaluation results aligned with human intuition ofdisentanglement degrees.
As discussed in Appendix B.3, among the four synthetic datasets we have,dSprites, Cars3D and SmallNORB have significant co-occurrence of different factors.
So we followthe practice in DisLib (36) to use Shapes3D as the dataset for the latent traverse.
For fairness andreproducibility, we trained VAE models on Shapes3D with the published configurations providedby Van Steenkiste et al.
because DisLib (36) does not provide checkpoints or training parameters on
Shapes3D.
The checkpoints provided by Van Steenkiste et al.
have various level of disentanglementperformance on Shape3D so they are proper for the visualization purpose.
The results are shownin Figure 17.
For each column in each subfigure, only the value of one dimension of the latentcode is manipulated.
The manipulation is performed the same way as the default setup for traversevisualization in DisLib.
Given the six factors on Shapes3D with index, we indicate under eachsubfigure if (1) red arrow: a dimension is entangled to more than one factor or (2) green arrow: adimension is disentangled and responsive to only one factor.
Through the visualizations and thecorresponding MED scores, we can clearly see that MED can well represent the disentanglementdegree.
To make the visualization, we re-train multiple VAE-based models to have divergent MEDscores on Shapes3D. We could observe a clear pattern that if a model achieves a higher MED score,it has more dimensions disentangled and responsive to only one factor while fewer dimensionsentangled to multiple factors.
This suggests that the results of MED can well align with the intuitionof humans.
I The Superiority of MED
Here, we provide both experimental observations and theoretical analysis in synthetic scenarios toshow the superiority of MED: in what cases other metrics outputs meaningless or even oppositeresults but MED can still perform a meaningful evaluation.
I.1 Experimental Observations
As we discuss throughout the paper, the five existing metrics we investigate, i.e. BetaVAE score,
FactorVAE score, SAP, MIG, and DCI Disentanglement, are unfair to models of different dimensions.
So we extend the results mentioned in Section 3.2 to show the superiority of MED by confirmingits stability under different dimensions.
For a good disentanglement metric, we obviously do notexpect it to output a high score on randomly initialized models whatever its dimension is.
Followinghuman intuition, we try to estimate the randomly initialized weights of low-dimensional and highdimensional models by different metrics and find their malfunctions.
• MED vs MIG/SAP : First of all, we emphasize that SAP/MIG is designed not just fordisentanglement but also completeness .
So they encourage a strict one-to-one mappingbetween latent dimensions and factors.
So they tend to reward models whose dimension isclose to the factor number.
On the popular synthetic datasets with only a limited numberof factors, low-dimensional models have an unfair advantage with MIG/SAP metrics.
Wecompare the well-trained 1000-d BYOL model and a 10-d randomly initialized encoder in
Factors:
[Figure]
[Figure]
• MED vs BetaVAE/FactorVAE score : BetaVAE and FactorVAE will reward highdimensional models as they have more parameters to trick the learnable classifier.
InTable 7, we find even randomly initialized high-dimensional (1000-d) model can achieve
higher BetaVAE/FactorVAE score than a well-trained 10-d model (DIP-VAE-II).
But MED
keeps a reasonable comparison between them.
[Figure]
• MED vs DCI : Similar to BetaVAE/FactorVAE scores, DCI also overestimate highdimensional models as they have a higher chance to get lottery dimensions, especiallywith the sparsity encouraged by GBT.
Also, another huge drawback of DCI is the compu-
tation efficiency: for a 1000-d model, DCI takes more than 14 hours for evaluation while
MED only takes less than 20 seconds.
The computation of MED is more than 2000 timesfaster than DCI.
Through the experimental observations, we find the abnormals from existing metrics when comparingmodels of different dimensions.
Compared to them, MED maintains reasonable outcomes alignedwith human high-level sense.
I.2 Theoretical Analysis
In this part, we construct synthetic scenarios to investigate the validity of MED and existing metrics.
We will show that in such situations, existing metrics will fail to reflect the real disentanglementquality but MED can show superiority to maintain reasonable evaluations.
For the following cases,we assume two factors F 0 , F 1 on the data.
We note their value as v 0 , v 1 ∈ { 0 , 1 } .
And they aresampled independently and uniformly, i.e., p ( v j = 1) = p ( v j = 0)
= 0 .
Then we construct linearcases to simplify the analysis that the value of latent dimensions is the combination of factor values.
I.2.1 MED vs MIG/SAP.
As mentioned, MIG and SAP want not just disentanglement but also completeness .
So, for a 1000-dmodel whose latent code value is c i
=
v i mod 2 , 1 ≤ i ≤ D, D = 1000 .
I.e. half of the latent code isthe true first factor, and the other half latent code is the second factor.
It is clear that each dimensionperfectly correlates to a single factor.
But its MIG/SAP scores turn out to be 0, which is significantlyopposite to our intuition of the disentanglement measurement.
But in this case, for MED we have
[Equation]
So we have MED ( c )
= 1 , indicating that every dimension is perfectly disentangled to a singlefactor and fully responsive to it.
Also, this conclusion applies to different values of D , showing therobustness of MED with models of different dimensions.
I.2.2 MED vs DCI.
With a constructed latent code as
[Equation]
To simplify, we assume the importance matrix output by GBT in DCI metric is the absolute offirst-order derivative, namely d ji =
| ∂c
i ∂v j | of the dimension i with respect to the factor F j .
GBTencourages sparsity in the output importance matrix.
To reflect this, we assume the desired sparsityof the GBT is 2.
Thus, we have the dimensions beyond the sparsity limit are given 0 importance.
Then the importance matrix value R ij is
[Equation]
otherwise .
(8) d j is a randomly selected dimension between [2 , D − 1] .
For the two factors F 0 and F 1 , d j is selectedindependently.
So that chance that d 0 = d 1 is p ( d 0 = d 1 ) =
[Equation]
(1) If d 0 = d 1 =
k , we have
[Equation]
(2) If d 0 ̸ = d 1 , we have
[Equation]
And DCI Disentanglement score is DCI ( c )
{+= (cid:80) D − 1 i =
+}
ρ i S
i
We
have
DCI ( c ) = 1 − log 2 = 0 . 31 in
situation (1) and DCI ( c ) = 1 in situation (2).
So, in such a simplified version, the expectation of DCI
Disentanglement score is
[Equation]
On the other hand, similarly we can derive the MED score is
[Equation]
[Equation]
which is quite consistent along the change of D .
Note that the drastic difference between MED and
DCI is caused by the GBT used in DCI leads to a sparse importance matrix M , while the mutualinformation in MED won’t.
This difference doesn’t make much difference in low-dimension latentcodes, but DCI will be high even if the latent code doesn’t disentangle at all in the high-dimensionalcase.
We plot the curves of MED score and the expectation of DCI score in Figure 18.
It shows that whenthe latent dimension D increases, the expectation of the DCI score increases and approaches 1.
Forinstance, if D = 3 , we have E [ DCI ( c )] = 30 .
7 ; if D = 1000 , we have E [ DCI ( c )] = 99 .
Such anabnormal from DCI Disentanglement alerts us that it can not give a meaningful and fair comparisonbetween methods of different dimensions.
However, MED is much more robust with the changeof model dimension.
For instance, if D = 3 , we have MED ( c ) =
9 and if D = 1000 , we have
MED ( c ) = 30 . 8 .
I.2.3 MED vs BetaVAE/FactorVAE score.
Situation 1 We construct a situation where the first two dimensions are a weighted combination ofthe factors and the value of all following dimensions are the average of the factor values.
Thus wehave the latent code as
[Equation]
i =12 ( v 0 + v 1 ) , i >.
We can justify it as a very entangled representation.
But the latent code value change with respect tothe factor value is what BetaVAE and FactorVAE scores care about.
FactorVAE calculates the variance of latent code, i.e. V ar ( c ) , when changing the factors to representthe factor-dimension response degree.
During calculating FactorVAE score, one factor is alwaysfixed.
So we have
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
 
[Equation]
i
[Equation]
[Equation]
[Equation]
, (14) so for each factor, we have a single dimension that gives the minimum variance.
In this case,
FactorVAE score uses a majority vote to determine the dimension with the smallest variance withrespect to the value change of a factor as the “most-disentangled” dimension to it.
And the voteaccuracy is regarded as the FactorVAE score.
So, it is clear that because of the existence of thelucky dimension j = i , FactorVAE score can be expected to approach 100, indicating “perfectlydisentangled”!
With the increase of model dimension, the chance of having such a lucky dimensionincreases as well in real cases.
The bias is that FactorVAE only needs one dimension which has a lowresponse to all other factors except for one factor to get high scores.
On the other hand, BetaVAE score calculates the latent code value change to represent the responsebetween factors and dimensions.
This is represented by the absolute value of the first-order derivative.
So for the i -th dimension with respect to the j -th factor as R ij , we have
[Equation]
[Equation]
BetaVAE score uses a learnable linear classifier to determine the dimension.
In this case, predictionj from R · j can achieves 100% accuracy.
So similarly, it also requires one “lucky” dimension tomake the prediction, offering an advantage to high-dimensional models.
And in fact, because of thepowerful capacity of a learnable linear classifier, the BetaVAE score is close to 100 for many cases asin Table 4.
It thus has another flaw of low distinguishability of methods’ disentanglement property.
On the other hand, we could now calculate the MED score for this situation.
The results turns outto be MED ( c )
= 1 − log 2 = 30 .
7 for any value of D .
This is a relatively low score and we think itmakes better sense in such entangled cases compared to the falsely high score from BetaVAE and
FactorVAE scores.
Situation 2 Let us build another case as in the discussion of DCI above
[Equation]
FactorVAE and BetaVAE scores can still give a “perfect” scoring over such a representation.
However,when the dimension D is very large, only a small subset of the representation dimensions is disentangled while most are highly entangled.
From this inspection, maintaining the perfect scoring withincreasing dimension is not aligned with our intuition and expectation of a robust metric especiallyconsidering that a higher-dimensional model has a better chance to get a lucky dimension.
From thissense, Top-K MED has its special use.
When D = 1000 , we know MED ( c ) = 30 . 8 , but we can stillget a subset of the 1000 dimensions to get a high Top-K MED score.
Such a subset of representationsis more compact and has better explainability to the factor variance.
With all the situations constructed and analyzed above, we could notice the failure of existingmetrics to achieve evaluation results (1) meaningful, (2) fair, and (3) aligned with the human senseto disentanglement in some scenarios.
On the other hand, our proposed MED always keeps theevaluation and comparison reasonable and aligned with the expectation from the human high-level institution.