Abstract
Scope of Reproducibility
In this study, we evaluate the paper ’Identifying Through Flows for Recovering
Latent Representations’.
Speciﬁcally, we evaluate the papers’ claimed practical advantages and effectiveness of their proposed method iFlow over previous methods,namely iVAE.
Methodology
First, we reproduce the obtained MCC scores for both iFlow and iVAE usingthe original code-base.
To place these results into context, we also evaluate twobaseline Flow models.
Furthermore, we discuss the proposed method’s usability,and apply it on a different Flow model, which is trained on the Half Moon datasetto analyse the learned latent representation.
With this, we assess the beneﬁt ofusing the proposed method over regular Flow.
It takes around 20 minutes for aniFlow model, and 75 seconds for an iVAE model to train and evaluate conformto the original implementation and dataset on an RTX 2080 Ti GPU with 11GBof VRAM.
Additionally, the iFlow network with planar-ﬂow Flow model takesaround 7 minutes to train on the same hardware.
Finally, the two Flow models onthe half moon datasets are trained on an AMD 3900X CPU with 32GB of DDR
RAM, each taking roughly 3 minutes.
Results
Our results are within 2.5% of the values presented by the paper, verifying theauthors’ claim of iFlow’s theoretical advancements over iVAE.
However, whencompared to the baseline Flow models, iFlow only shows up to 10% improvementin MCC scores, compared to a 45% improvement over iVAE.
Furthermore, whenanalysing the learned latent representation for the Half Moon dataset iFlow doeslearn a more robust latent representation compared to Flow, and unlike Flow, issometimes able to reach principled disentanglement, partly verifying the paper’sclaim of iFlow’s practical advantages and effectiveness.
What was easy
The original code implementation was not difﬁcult to setup and run speciﬁcallyfor the iFlow model.
The code provided the proper run script for training and
evaluating iFlow.
Furthermore, implementing the proposed identiﬁability methodto different ﬂow models is not difﬁcult - the authors provide a clear derivationof the objective function.
Finally, in the code, a different use of the activationfor the natural parameters is suggested, which we found to be straightforward toimplement.
What was difﬁcult
The code-base lacked documentation, thus besides running the default iFlow setup,running different models such as iVAE was quite challenging.
In general, understanding the code itself, particularly the code used to generate the dataset, was notstraightforward.
No code was offered to save the results and construct the ﬁguresfrom the paper.
Finally, despite the supposed support for using planar-ﬂow insteadof the default cubic spline-ﬂow in the code base, training iFlow with planar-ﬂowwas not trivial.
This was due to both an incorrect initialization of the planar-ﬂowmodel, where it called the wrong class, and an incorrect return statement.
Communication with original authors
There has been no communication with the authors.
[Re] Identifying Through Flows for Recovering Latent Representations
Anonymous Author(s)
AfﬁliationAddress email
Introduction
One of the most fundamental goals of unsupervised representation learning is recovering the truejoint distribution over the observed data and latent variables from which the data is generated.
Ifwe are able to learn this true distribution, we would also recover the true distribution over the latentvariables.
Since by deﬁnition latent variables are not observed, this is generally considered to be anextremely difﬁcult task.
However, it is shown that for a broad family of deep-latent-variable modelsrecovering the true latent representation is possible [5].
Speciﬁcally, it is possible to recover the truelatent representation if the model is identiﬁable .
In broad terms, a model is identiﬁable if and only ifit has a unique solution for a given set of parameters.
By recovering the true latent representations,models are able to achieve principled disentanglement.
To build a deep latent-variable model which allows for recovery of true latent representations theauthors of [7].
propose unifying identiﬁability with normalizing ﬂows.
Their proposed method iFlowis mainly built upon the foundation of the previous identiﬁable Variational Auto Encoder (iVAE)introduced by [5].
The authors of [7]. claim that ﬂow-based models are particularly suitable foridentiﬁable models, as the objective directly maximises the density of the estimation model.
Incontrast, VAEs only optimise a lower bound, which leads to a less identiﬁable model.
In this report, we perform an extensive study on both the reproducibility of the results presented in thepaper as well as the supposed beneﬁts of using iFlow over existing representation learning methods asclaimed by the authors [7].
The contribution of this work is therefore two-fold: We reproduce originalresults presented in paper using the existing implementation, add some baseline experiments to betterplace the obtained results into context, and discuss the metric used for evaluation.
Next, we analysethe claimed practicality of the proposed method and perform experiments to gauge the beneﬁts thatidentiﬁability brings to learning stronger, and more importantly, explainable latent representations.
We provide the source code 1 to run the experiments performed in this report.
1.1 Target Questions
To verify the results presented in the paper and to asses the claimed superiority of iFlow over previousdeep generative models on representation learning tasks, this report mainly focuses on answering thefollowing four questions:
• Can the claimed superiority of iFlow over iVAE with respect to MCC-scores be reproducedusing only the methods described in the paper and provided source code?• How well does iFlow perform compared to its non-identiﬁable counterparts, consideringmultiple ﬂow-based models such as Spline-Flow and Planar-Flow?• To what extent does iFlow offer practical advantages over existing deep representationlearning frameworks such as ICA, iVAE and Flow?
• To what extent does identiﬁability improve upon normalising ﬂows?
Reproducibility of Mean Correlation Coefﬁcients
Recently, [5] proposed an identiﬁable Variational Auto Encoder (iVAE) that reaches identiﬁability byconditioning the latent variables on an auxiliary variable.
However, the authors of [7] claim that iVAEleads to sub-optimal identiﬁability, both in theory and practice.
This is due to the intractable KLdivergence between the approximate and true posterior and iVAE only maximising the lower boundlog-likelihood evidence.
To overcome these limitations, the authors propose identiﬁable generativemodel through ﬂows (iFlow).
According to the authors, its ability to directly maximise the likelihoodyields stronger identiﬁable latent representations, achieving better principled disentanglement.
In this section, we reproduce the original results as presented by the authors.
We will evaluateusing the implementation provided by the authors.
Speciﬁcally, we reproduce the Mean Correlation
Coefﬁcient (MCC) scores, which is a standard measure used in independent component literature.
As the MCC metric can be sensitive to synthesised data for different seeds, the authors run theexperiment for 100 different data seeds.
Due to time constraints, we run our experiments for 20 dataseeds, which is ample to verify the original results.
We use the same Gaussian time-series dataset asused in the paper, as described in [3] (see Supplemental Material Section B).
For the ﬂow function,the authors use RQ-NSF(AR) cubic-spline ﬂow [2].
All the relevant hyper parameters and hardwareconﬁgurations can be found in Supplementary Materials Section A.1.
It must be noted that there is some discrepancy between the paper and the provided implementation.
In the paper, it is mentioned that the softplus non-linearity is applied to the ﬁnal layer of the auxiliarymapping function λ .
To ensure ﬁniteness, a negative activation is then taken over the second orderparameters.
In the implementation, however, it is suggested that the softplus non-linearity should onlybe applied to the second order parameters.
Therefore, we evaluate both methods.
The different modelsused throughout the experiments are implemented using the PyTorch Deep Learning Framework [8].
Figure 1 shows the MCC scores achieved by iFlow and iVAE for both the original implementationas described in the paper and the alternative implementation suggested in the code-base.
Whencomparing our results to those presented in the original paper, we observe merely small differences.
For the original implementation, our experiment yields slightly lower results for iFlow for the mean
MCC and standard deviation with a difference of 0.011 and 0.005 respectively.
For iVAE, our mean
MCC scores 0.0263 below the result of the paper, and the standard deviation is more or less equal.
For the alternative implementation, we reach an mean MCC score higher than both our previousresult and those presented in the paper, with a difference of 0.0258 for the latter.
We also observe aslightly higher standard deviation.
Our results do not make clear which of the two implementation iscorrect, as the differences between the results are minor.
Both our results and those presented in the paper show a relatively high standard deviation overdifferent data seeds, while net network seed remains ﬁxed.
Therefore, we conduct a further experimentwhere we compare a ﬁxed network and variable data seed with a variable network and ﬁxed data seed for both iFlow and iVAE to gauge model stability.
Again, we evaluate on 20 different seedsfor each experiment.
For iFlow, Figure 2 shows very comparable mean MCC scores and standarddeviations when ﬁxing either the data or network seed.
This suggests the seed chosen to run theexperiments in the paper was not cherry-picked.
For iVAE, we observe very similar mean MCCscores, but the standard deviation is halved when the data seed is ﬁxed.
This may be attributed tohaving a particularly well suited data seed for this speciﬁc case.
In conclusion, our results seemsto be in line with those presented in the paper, also when considering variable network seeds.
Weobserve that iFlow indeed performs signiﬁcantly better than iVAE, verifying the authors’ claim.
Comparing iFlow to Flow
As is shown in Section 2, our ﬁndings are in line with those that are reported in the original paper [7].
The results show that iFlow reaches higher MCC scores compared to iVAE.
However, these resultsare not placed into context: Two inherently different models are being compared.
Therefore, weconduct an experiment where we compare the identiﬁable ﬂow models with their non-identiﬁablecounterparts.
It is important to note the difference in parameter complexity between iFlow and iVAE.
The authors use the Q-NSF(AR) cubic-spline ﬂow [2], making their iFlow implementation contain2.944.980 compared to 18.170 parameters for iVAE.
Furthermore, the original code base also includes
Planar-Flow model containing 2.170 parameters.
We evaluate MCC-scores for both ﬂow models.
The result are displayed in Figure 3 and include iVAE performance for reference.
We immediately observe that both non-identiﬁable ﬂow models reach signiﬁcantly higher MCCscores compared to iVAE.
Clearly, both ﬂow-based models allow for learning a richer representation.
We also observe that the spline-based model outperforms the planar-ﬂow model, likely due to itsincreased parameter count.
Interestingly, we do notice slightly higher mean MCC-scores for theidentiﬁable ﬂow models.
However, since performance is measured using MCC, these difference are slim, and seem to suggest a slight improvement at best, especially for the planar-ﬂow model.
This issomething the authors fail to mention, but we feel it is an important aspect to gain a better perspectiveon the actual beneﬁt of using iFlow.
We also want to raise two important observations regarding the MCC metric used to evaluate theperformance of the identiﬁable models.
First of all, it must be noted that the discussed identiﬁablemodels are described as being identiﬁable up to some equivalence class.
Speciﬁcally, they areidentiﬁable up to some invertible afﬁne transformation over the sufﬁcient statistics.
This does notseem to be accounted for when the MCC is calculated between the original sources and learned latentrepresentation.
Moreover, we ran a small experiment where we replaced the ﬂow mapping function to an identityfunction.
With this model, we measured a signiﬁcant correlation between the data samples andsources from which these samples are generated.
For the data used in the experiments described in
Section 2, we found an MCC score of 0.58 between data and sources.
This means that the modeloften learns a latent representation that is less correlated with the original sources compared to thedata itself.
Note that in general the MCC metric varies greatly between both different data seeds anddifferent network seeds because the MCC metric can be quite susceptible to small deviations in thelearned latent representations.
Because of these reasons, we question the effectiveness of the MCCmetric to evaluate the model’s ability for learning a true latent representation.
Assessing iFlow’s Usability
In this section, we clarify to what extent iFlow offers practical advantages over previous representationlearning methods.
In the paper [7], the authors speciﬁcally mention three existing representationlearning frameworks: Nonlinear independent component analysis (ICA) [4], identiﬁable Variational
Auto-Encoders (iVAEs) [5], and Flow-based models [6].
The authors explicitly ascribe all threeframeworks speciﬁc shortcomings and claim that by adding identiﬁability to ﬂow-based models,iFlow provides practical advantages over existing methods.
However, besides expanding on iFlow’stheoretical ability to recover the true latent space, and showcasing that iFlow achieves higher MCCscores (Section 2) than iVAE on synthesized Gaussian time-series data [3], the authors do notexplicitly make clear to what extent iFlow overcomes or preserves other mentioned shortcomings of
ICA, iVAE and Flow.
In this section, we elaborate on iFlow’s practical usability as compared to theseprevious methods by investigating how readily iFlow can be used on various datasets.
We explored iFlow’s practical usability by attempting to apply it to two datasets: the MNIST datasetand the Gaussian Blob dataset.
We chose these two datasets because MNIST is one of the mostfrequently used datasets in machine learning research in general, and because the Gaussian Blobdataset is frequently used particularly in representation learning research.
For a more elaborateexplanation of the datasets, see Figure S3 and Figure S4 in the supplementary materials.
For bothdatasets, we took the source code for iFlow and tried to re-implement it such that iFlow should beable to learn the dataset’s latent distribution.
Our ﬁndings are summarized in Table 1.
[Table]
for [Table] For MNIST, we found that the provided labels (i.e. numbers) could function as auxiliary variables.
Nevertheless, both in the paper as well as in the provided source code, the dimension of the observed datapoints and the latent space must be of the same size for iFlow to work.
However, in MNISTdatapoints are of size 28 by 28, and it is very unlikely that that the underlying latent space is of size784:
That would imply that each pixel is an explanatory source in itself, while we reason that thereshould be no more explanatory sources for recognizing single numbers than that there are singlenumbers to begin with.
iVAE does not have this limitation: There actually is an implementation iVAEfor MNIST available.
Furthermore, in contrast to the synthesizes Gaussian data used in the paper, wedo not know beforehand what the latent space of MNIST should look like.
It is therefore difﬁcult toevaluate how well iFlow recovers this latent space because we cannot compare it to the original latentspace; making it impossible, for instance, to compute MCC-scores.
In contrast to MNIST, for the Gaussian Blob dataset the latent space is known beforehand, potentiallymaking it a candidate for evaluating iFlow’s performance.
However, the constraints iFlow puts on thedimensions of the observed and latent space are again problematic: because the datapoints are againimages, iFlow’s assumption that the latent space is of the same size as the observed datapoints seemsonce again to result in a latent space with much more dimensions that we should reasonably expect.
Furthermore, it is not straightforward to determine what should be used as the auxiliary variablefor this dataset.
We could use the quadrant indices indicating in which quadrant the blob appearsas auxiliary variable, but this is only possible because this dataset, just like synthesized Gaussian
Time-series data, is generated artiﬁcially and we therefore know the underlying distribution.
In conclusion, our exploration of attempting to apply iFlow to different datasets shows that iFlowpractical usability has more caveats than warranted by the authors.
First of all, we assume thedimensionality of latent space to be (much) smaller than dimensionality of the observed datapoints.
Nevertheless, iFlow’s current implementation requires the latent space to be of the same size asthe observed datapoints, leading to unreasonably high-dimensional latent spaces when workingwith pixel-valued image data.
Furthermore, using datasets where the underlying distribution isnot known beforehand (which can be reasonably be expected to be the most common case forpractical applications) makes it non-trivial to evaluate iFlow’s performance and to come up withthe required auxiliary variable.
While this requirement for an auxiliary variable is inherent to allidentiﬁable methods, this does put severe limits on the applicability of these models.
Consequently,non-identiﬁable models such as Flow can be more readily applied to a much wider range of datasets.
Why then, one would consider using iFlow at all, we discuss in Section 5.
Evaluating iFlow’s Disentangled Identiﬁability
In addition to the supposed practical advantages iFlow has over other methods, which we havenuanced in Section 4, the authors mainly emphasize iFlow’s theoretical guarantee for learning thetrue latent representation.
Furthermore, while the authors do provide visualisations of learned latentrepresentations, these visualisations do not lend themselves well for identifying whether the modelhas learned a principled disentangled representation.
In this section, we aim to further analyseiFlow’s recovered latent representation, and how it improves upon the representation recovered by itsnon-identiﬁable counterpart Flow.
First, we provide a brief overview of the notion of entanglement.
Disentangled representations differ from entangled ones in at least one signiﬁcant way.
Both entangledas well as disentangled representations help leverage a signiﬁcant bottleneck conventionally present inmachine learning work: instead of having to labor-intensively use human ingenuity to feature-engineerthe representations that support effective machine-learning, representation learning frameworksuse available data to learn these representations in an unsupervised-manner [1].
Disentangledrepresentations not only learn representations that lead to discriminative information for the task athand, but they also learn to successfully separate the various explanatory sources underlying the data
Theoretically, we should be able to circumvent this limitation of iFlow by engineering the latent space of the underlying ﬂow-based model in a very task-speciﬁc way, but we found no code or suggestions in the paper to do so.
For example, when we look at a glass on a table, in our mind we can very readily separate the glass from the table, and even from the shadows it casts on that table.
However, the stimuli of all three sources reach our retina in the same way.
Apparently, something in our brain learns to separate these various sources that explain the stimuli we receive.
A disentangled representation learning framework aims to achieve the same for machine learning models that receive input data.
Principled disentangled representations go a step further: not only are these explanatory sourcessuccessfully separated, the components modeling these sources are independent.
To investigate to what extent iFlow leads to a more principle disentangled learned representation than
Flow, we used the provided source code to implement an identiﬁable RealNVP model.
From hereon, we refer to RealNVP as Flow, and the identiﬁable RealNVP as iFlow.
We evaluated both modelson the Half Moon dataset and compared their respective learned representations.
The reasons forchoosing this dataset are twofold: First of all, both Flow and iFlow can be readily applied to thisdataset because the dataset has a well-deﬁned auxiliary variable and has none of the dimensionalityproblems as described in Section 4.
Secondly and more importantly, the dataset has a latent spacethat is relatively easy to comprehend and therefore allows us to manually evaluate to what extent bothframeworks entangle or disentangle the learned latent representation.
For more details on the dataset,see Figure S2 in the supplementary materials.
The paper and provided source-code do not offer a method to generate new datapoints using iFlow.
We found that coming up with a own sampling technique is not straightforward, because nowhereduring the forward pass of the model is the latent space saved for later use.
Nevertheless, it seemstheoretically possible to design and implement an additional deep model that learns this encodingstep.
However, due to time-constraints, we choose to model the data as a simple two-dimensionalmultivariate Gaussian (tuning µ and σ on the data).
For Flow, the data is generated using the givenprior and is classiﬁed using the nearest neighbour algorithm and previously observed data.
The resultsare shown in Figure 4.
The results seem to show that iFlow indeed leads to slightly better disentanglement than Flow:
iFlow more distinctly separates both group of datapoints into independent clusters.
Consequently,generated datapoints using iFlow also seem to more accurately map back onto the originally observeddistribution.
However, for principled disentanglement we should also check whether moving alongone axis within the recovered latent clusters only results in variation along one underlying explanatorysource while keeping other sources relatively constant.
To investigate this, we ran one ﬁnal experimentfor both trained models.
Ideally, we want to observe the mapped latent space in a continuous and global way, where each axiscorresponds to an independent property of observed data.
This would suggest the model has learned aprincipled disentangled latent representation.
Figure 5 shows the mapping of the well-deﬁned regionof the learned latent representation to the original data space.
Here, we segmented the latent spaceusing different icons to distinguish the gradient of the space in the horizontal direction and coloursfor the gradient in the vertical direction.
In Figure 5 we can indeed observe the desired properties for the identiﬁable model.
The meaning ofthe latent space is globally deﬁned.
When we move across the horizontal dimension while ﬁxing thevertical dimension, a clear gradient across the original half moons is present.
The vertical positionindicates which half moon the latent space is mapped to.
For the non-identiﬁable model, this is lessdeﬁned.
Here, the horizontal dimension encodes both the position on each half moon and to whichhalf moon a point is mapped to.
The vertical dimension only encodes the vertical position a on
However, it must be noted that the results presented in Figure 5 are not guaranteed.
First, we observethat for both Flow and iFlow the meaning of the latent space is not always globally deﬁned, althoughlocal meaning may still be present, as in Figure S7c.
Also, the identiﬁable model often does notreach the claimed principle disentanglement, showing a learned latent representation that is similar to
Flow which can be observed in Figure S7a.
One key difference that is consistently observed between
Flow and iFlow, the the ability to learn a continuous latent representation.
It seems that for iFlow, therelative structures in the learned latent representation remain intact, whereas for regular Flow, thisdoes not have to be the case.
This is shown in Figure S8b.
These preliminary ﬁndings indicate thatiFlow can indeed be effective in learning a stronger latent representation compared to Flow.
Conclusion
In this work, we investigated the reproducibility of the results presented in [7], comparing iFlow’sperformance in recovering a true latent representation to the previously proposed iVAE.
Next, weelaborated on the fairness of the metrics used in the paper.
Finally, we assessed the usability andpractical advantages the authors claim iFlow has over existing representation learning methods.
We found that we could reproduce the reported MCC-scores of iFlow and iVAE to acceptableprecision, having us conclude that the results in the paper showing that iFlow outperforms iVAEwith respect to these MCC-scores are valid.
However, we found that the paper does not put theseresults into context.
Therefore, we investigated iFlow’s performance when replacing its underlyingcubic-spline ﬂow model with a less parameter-intensive planar-ﬂow model.
We found that whileplanar-based iFlow still outperforms iVAE with respect to MCC-scores, it does so by a signiﬁcantlesser degree.
Furthermore, we show that their non-identiﬁable counterparts outperform iVAE aswell, and the difference in MCC-scores between the identiﬁable and non-identiﬁable models is lessprofound when compared to the difference between iFlow and iVAE.
Next, we found a signiﬁcant correlation between the data and the underlying distributions from whichthis data is generated.
This raises the question whether MCC-scores alone are sufﬁcient and validin determining whether the true latent representation is found.
We also attempted to apply iFlow torecover the latent representations of two datasets not presented in the paper:
The MNIST datasetand the Gaussian Blob dataset.
We found that mainly iFlow’s requirement for auxiliary variablesand speciﬁcally sized input data puts signiﬁcant constraints on datasets it can be used for, having usconclude that the authors claim for iFlow’s practical advantages over other existing methods seemstoo simplistic.
Finally, we evaluated how well iFlow delivers on its promise for principled disentanglement ascompared to its more readily applicable, but non-identiﬁable, counterpart Flow.
By visually exploringthe learned representations of both models on the half-moon dataset, we found that iFlow is capableof recovering principled disentangled representations, where Flow is not.
However, often this state ofrepresentation is not reached, and the obtained representation is similar to that of Flow.
Nevertheless,we do observe a consistent stronger recovered latent representation when utilising iFlow.
In conclusion, iFlow shows us a promising next step in the area of recovering true latent representations and reaching principled entanglement.
Future research in identiﬁable models and possiblemetrics to further analyse their performance is deﬁnitely warranted, as this could solve many of theissues illustrated in this reputability report
A Hyperparameters and Hardware for Conducted ExperimentsA.1 Original Paper Experiments
[Table]
[Table]
MLP (2,256,256,2) leaky ReLU
[Figure]
Datasets Explanation
[Figure]
B.
[Figure]
B.3 MNIST Dataset
[Figure]
C Encoding, Decoding and Sampling for iFlow and Flow
D Latent Space Exploration for iFlow and Flow
D.1 iFlow
[Figure] [Figure]
(c)
[Figure] [Figure]
[Figure]
) (c)
[Figure]