I DENTIFYING C OARSE - GRAINED I NDEPENDENT C AUSAL M ECHANISMS WITH S ELF - SUPERVISION
Anonymous authors
Paper under double-blind review
A BSTRACT
Current approaches for learning disentangled representations assume that independent latent variables generate the data through a single data generation process.
In contrast, this manuscript considers independent causal mechanisms (ICM), which, unlike disentangled representations, directly model multiple data generation processes (mechanisms) in a coarse granularity.
In this work, we aim to learn a model that disentangles each mechanism and approximates the groundtruth mechanisms from observational data.
We outline sufﬁcient conditions under which the mechanisms can be learned using a single self-supervised generative model with an unconventional mixture prior, simplifying previous methods.
Moreover, we prove the identiﬁability of our model w.r.t. the mechanisms in the self-supervised learning scenario.
We compare our approach to disentangled representations on various downstream tasks, showing that our approach is more robust to intervention, covariant shift, and noise due to the disentanglement between the data generation processes.
I NTRODUCTION
The past decade witnessed the great success of machine learning (ML) algorithms, which achieve record-breaking performance in various tasks.
However, most of the successes are based on discovering statistical regularities that are encoded in the data, instead of causal structure.
As a consequence, standard ML model performance may decrease signiﬁcantly under minor changes to the data, such as color changes that are irrelevant for the task, but which affect the statistical associations.
On the other hand, human intelligence is more robust against such changes (Szegedy et al., 2013).
For example, if a baby learns to recognize a digit, the baby can recognize the digit regardless of color, brightness, or even some style changes.
Arguably, it is because human intelligence relies on causal mechanisms (Sch ¨ olkopf et al., 2012; Peters et al., 2017) which make sense beyond a particular entailed data distribution (Parascandolo et al., 2018).
The independent causal mechanisms (ICM) principle (Sch ¨ olkopf et al., 2012; Peters et al., 2017) assumes that the data generating process is composed of independent and autonomous modules that do not inform or inﬂuence each other.
The promising capability of causal mechanisms grows an activate subﬁeld (Parascandolo et al., 2018;
Locatello et al., 2018a;b; Bengio et al., 2019).
Recent works deﬁne the mechanisms to be: 1) functions that generate a variable from the cause (Bengio et al., 2019), 2) functions that transform the data (e.g. rotation) (Parascandolo et al., 2018), and 3) a disentangled mixture of independent generative models that generate data from distinct causes (Locatello et al., 2018a;b).
Throughout this paper, we refer to type 2) mechanisms as shared mechanisms and type 3) mechanisms as generative mechanisms. Despite the recent progress, unsupervised learning of the generative and shared mechanisms from complex observational data (e.g. images) remains a difﬁcult and unsolved task.
In particular, previous approaches (Locatello et al., 2018a;b) for disentangling the generative mechanisms rely on competitive training, which does not directly enforce the disentanglement between generative mechanisms.
The empirical results show entanglement.
Additionally, Parascandolo et al.
(2018) proposed a mixture-of-experts-based method to learn the shared mechanisms using a canonical distribution and a reference distribution, which contains the transformed data from the canonical distribution.
Such a reference distribution is generally unavailable in real-world datasets.
To create a reference distribution, we need to use the shared mechanisms that we aim the learn. This causes a chickenegg problem.
Besides, the unsupervised learning of the deep generative model is proved to be unidentiﬁable (Locatello et al., 2019; Khemakhem et al., 2020). Recent methods (Locatello et al., 2020; Khemakhem et al., 2020) leverage weak-supervision or auxiliary variables to identify the right deep generative model. However, such weak-supervision or auxiliary variables still do not exist in conventional datasets (e.g. MNIST). Lacking identiﬁability makes it impossible to learn the right disentangled model (Locatello et al., 2019).
We, therefore, seek a practical algorithm with identiﬁability result that disentangles the mechanisms from i.i.d data without manual supervision.
To this end, we propose a single self-supervised generative model with an unconventional mixture prior.
In the following sections, we refer to our model as the ICM model. Using a single self-supervised generative model would allow us to leverage the recent progress in deep generative clustering (Mukherjee et al., 2019), which would enforce the disentanglement between the generative mechanisms. We use the following example to illustrate the relationship between the generative model and the mechanisms.
Let us assume we have a generative model G : Z −→ X , two generative mechanisms M 0 : Z M 0 −→ X M 0 , M 1 :
Z M 1 −→ X M 1 , and one shared mechanism M S : X M , Z S −→ X where Z = [ Z M 0 , Z M 1 , Z S ] and X M = X M 0 ∪ X M 1 . We have G ([ z M 0 , 0 , z S ])
=
M S ( M 0 ( z M 0 ) , z S ) and G ([ 0 , z M 1 , z S ]) =
M S ( M 1 ( z M 1 ) , z S ) .
Our mixture prior is unconventional because the mixture components are { [ N ( 0 , I ) , 0 , N ( 0 , I )] , [ 0 , N ( 0 , I ) , N ( 0 , I )] } instead of {N ( µ 0 , σ 20 ) , N ( µ 1 , σ 21 ) } .
We disentangle the generative mechanisms by disentangling the type of variations (causes) carried by each z M k , ∀ k ∈ { 1 , 2 , ..., N } where N is the number of generative mechanisms.
The disentanglement between the generative mechanisms and the shared mechanisms will be guaranteed by the prior itself.
Furthermore, we theoretically prove that the ICM model is identiﬁable w.r.t. the mechanisms without accessing any label.
The key contributions of this paper are:
• We propose a simpler method to learn the mechanisms with only self-supervision.
• We design an unconventional mixture prior that enforce disentanglement. • We prove the ﬁrst identiﬁability result w.r.t. the mechanisms in the self-supervised scenario. • We develop a novel method to quantitatively evaluate the robustness of ML models under covariant shift using the covariant that is naturally encoded in the data.
• We conduct extensive experiments to show that our ICM model is more robust against intervention, covariant shift, and noise compared to disentangled representations.
R ELATED W ORK
Functional Causal Model In functional causal model (FCM), the relationships between variables are expressed through deterministic, functional equations: x i =
f i ( pa i , u i ) , i = 1 , ..., N .
The uncertainty in FCM is introduced via the assumption that variables u i , i = 1 , ..., N , are not observed (Pearl et al., 2000).
If each function in FCM represents an autonomous mechanism, such FCM is called a structural model.
Moreover, if each mechanism determines the value of one and only one variable, then the model is called a structural causal model (SCM).
Taking the view from the SCM’s perspective, we want to learn a mixture of causal models whose inputs are pure latent variables and whose output is a single high-dimensional variable that describes complex data such as images.
Different from other SCM approaches, where the unobserved variables only introduce uncertainty to the model, the latent variables in our model carries distinct variations in the dataset.
Independent Component Analysis Discovering independent components of the data generating process has been studied intensively (Hyv¨arinen & Oja, 2000; Hyvarinen et al., 2019). A recent work (Khemakhem et al., 2020) bridges the gap between the nonlinear independent component analysis (ICA) and the deep generative model. The nonlinear ICA with auxiliary variables brings parameter-space identiﬁability to variational auto-encode. The nonlinear ICA tackles the parameterspace identiﬁability of deep generative models.
However, the parameter-space identiﬁability does not guarantee the disentanglement between causes.
We will discuss the difference in section 4.
Disentangled Representations Disentangled representations assume that the data is generated using a set of independent latent explanatory factors (Bengio et al., 2013).
Previous works (Higgins et al., 2017; Kumar et al., 2017) proposed various unsupervised methods to disentangle the latent explanatory factors.
Later, unsupervised disentanglement is proved to be impossible (Locatello x z (a)
D k
M S k
M k z M k (b) x z Mz M...
z M N z S z C (c) ...
D
D N
D
M S ◦ M... ◦ ... M S ◦ M N z M...
z M N z S (d) et al., 2019).
Recent works (Shu et al., 2019; Locatello et al., 2020) leverage weak-supervision (e.g. pair-wise label) to identify the right disentangled model.
Compared to disentangled represtations, our ICM model takes a different assumption.
We assume there are groups of latent explanatory factors that generate clusters in the data distribution, respectively.
G ENERATIVE M ODEL FOR D ISENTANGLED M IXTURE OF M ECHANISMS 3. P RELIMINARIES
Many generative models (Kingma & Welling, 2013; Goodfellow et al., 2014; Locatello et al., 2020) assume that the data x is generated through a two-step procedure: (1) A sample z is drawn from an unobserved continuous prior distribution p ( z ) , which is usually assumed to be N ( 0 , I ) .
An observational data sample x is drawn from an unknown conditional distribution p ( x | z ) or is generated by a function f : Z −→ X .
Figure 1(a) and 1(b) further visualize this procedure from a causal mechanisms’ perspective.
Generative Adversarial Network Generative Adversarial Network (GAN) learns the data generation process via a two-player minmax game between a generator G : Z −→ X and a discriminator D : X −→ Y : min G max D V ( G, D ) =
E x ∼ p ( x ) [log D ( x )]
+ E z ∼ p
( z )
[log(1 − D ( G ( z )))] .
The goal is to minimize the divergence between the generated data and the real data.
Recent research has shown that the Wasserstein distance (Arjovsky et al., 2017) is a good choice of divergence in practice.
Thus, we use Wasserstein GAN with gradient penalty (WGAN-GP)
(Gulrajani et al., 2017) as our generative model.
GAN are less popular in unsupervised learning of disentangled representations as is difﬁcult to approximate the posterior p ( z | x ) .
However, recent work (Mukherjee et al., 2019) showed its advantages in clustering, which is very relevant to disentangling the data generation process.
3. T HE U NCONVENTIONAL M IXTURE P RIOR AND S TRUCTURAL L ATENT S PACE
Unlike most generative models which assume p ( z ) = (cid:81) di =1 p ( z i ) , we use an unconventional mixture prior: p ( z ) = (cid:88) z C p ( z S ) N (cid:89) k
= p ( z M k | z C ) =
p ( z S )
N (cid:88) k = p k ( z M ) p ( z M k | z C ) = (cid:26) N ( 0 , I ) , z C = k , otherwise .
(1) where p ( z S ) = (cid:81) d S i =1 p ( z S i ) , p ( z M k ) =
(cid:81)
d Mk i =1 p ( z M ki ) , and z M =
[ z M 1 , z M 2 , ..., z M N ] .
z S represents the causes for the shared mechanisms and z M k represents the causes for the k th generative mechanisms.
We convert the sum of the conditional joint distribution to our unconventional mixture distribution by deﬁning p 1 ( z M ) =
[ N ( 0 , I ) , 0 , ..., 0 ] , p 2 ( z M ) =
[ 0 , N ( 0 , I ) , ..., 0 ] and so on.
Figure 1(c) and 1(d) further visualize the graphical model and the assumed data generation process with the unconventional mixture prior.
Our prior disentangles the mixture components by encoding the components in orthogonal latent subspaces. The mixture components of Gaussian mixture prior, however, are shown to entangle in the latent space (Mukherjee et al., 2019) because it is perfectly ﬁne for mixture components N ( µ 0 , σ 20 ) and N ( µ 1 , σ 21 ) to have overlap. Such an overlap entangles the causes of different mechanisms.
For the shared mechanisms, we encode all causes, such as rotation and brightness, in z S and do not further disentangle them because such disentanglement is proved to be impossible without supervision (Locatello et al., 2019).
Although we do not disentangle each shared cause, it usually does not hurt the performance of downstream tasks because most of the predictive tasks are associated with the generative mechanisms.
Thus, we only need to disentangle the shared mechanisms from the generative mechanisms.
[-In addition, we pose an ”isolation” constraint for the ICM-conditioned mechanisms such that z M j = 0 if z M i (cid:54) = 
-]
In section 4, we show such disentanglement is guaranteed, provably.
Besides the Gaussian mixture prior, we also discuss the advantage of our unconventional mixture prior over the existing disentangled prior (Higgins et al., 2017; Locatello et al., 2020).
To see this, consider that by deﬁnition, each z i in disentangled prior z , which admits a density p ( z )
= (cid:81) di =1 p ( z i ) , represents one type of variations in disentangled representations.
With the disentangled prior, we can generate data through the generative model by setting each z i to an arbitrary value. If we use each z i [-(cid:54) = 
-]to represent each cause, the prior would allow the generative mechanism to generate data by arbitrarily combining the causes.
This contradicts our assumption that each cluster in the data distribution has its distinct cause and none of the data is generated by the combined causes.
To resolve this conﬂict, we need to either create a new dataset or adopt the unconventional mixture prior.
3. T HE S ELF - SUPERVISED M ODEL
Our model is composed of a generator network G : Z −→ X , a discriminator network D : X −→ Y , and an auxiliary encoder network E : X −→ Z .
The auxiliary encoder is a basic deterministic encoder that tries to invert the generator.
The self-supervision means predicting which generative mechanism does the generated data come from using the auxiliary encoder network E .
Such selfsupervision would encourage the data samples from the same generative mechanism to be similar to each other and vice versa.
Thus, we can disentangle generated data from different mechanisms in the sense that they are separable and their sources are predictable.
Once the data samples are disentangled into clusters, the generative mechanisms are disentangled as well.
The loss function of our method is:
[Equation]
where P g denotes the distribution covered by the generator network G , P r denotes the real data distribution, W ( P g
|| P r ) denotes the Wasserstein distance between two distributions,
z = [ z C , z M , z S ] , z C is a categorical variable represents the index of the generative mechanisms, z M and z S are the same as is deﬁned in section 3.2, and H denotes the cross-entropy loss.
We have tried to replace the Euclidean distance with Cosine distance in the loss function, but both methods yield similar results.
As the readers may have noticed, we add z C to z in our implementation. The purpose is to eliminate the ambiguity when z M ≈ 0 and the generator does not know which generative mechanism to use (Antoran & Miguel, 2019).
As we are only ﬁxing a corner case, we leave it as an implementation detail and keep the unconventional mixture prior deﬁnition unchanged.
The loss function above can be decomposed into four parts: 1) W ( P g || P r ) is the WGAN loss. 2) E z ∼ P ( z ) H
( z C , E ( G ( z )) C ) is the self-supervision loss that enforces the disentanglement between generated data. 3) E x ∼ p ( x ) || x − G ( E ( x ))
||
22 is the forward cycle-consistent loss, which enforces x ≈ G ( E ( x )) . The encoder takes the generated data from the generator as input.
However, the generated data distribution may differ from the real data distribution. Thus, certain data in the real distribution may become out-of-distribution (OOD) for the encoder.
Such distribution divergence hurts the performance of downstream tasks.
The forward cycle-consistency mitigates the distribution divergence by encouraging the ICM model to cover the whole real distribution. 4) E z ∼ p ( z ) || z M − E ( G ( z ))
{+M || 
 + β S E z+}
∼ p ( z ) || z S − E ( G ( z )) S || 22 is the backward cycle-consistent loss, which enforces z ≈ E ( G ( z )) .
The backward cycle-consistency prevents the generator from discarding certain causes in the generation process.
Identiﬁability is crucial for disentanglement. We can construct inﬁnitely many generative models that have the same marginal distribution (Locatello et al., 2019; Khemakhem et al., 2020). Without identiﬁability, any one of these models could be the true causal generative model for the data, and the right model cannot be identiﬁed given only the observational data (Peters et al., 2017).
Prior work (Locatello et al., 2019) shows that we can not identify the disentangled generative model using the marginal distribution. On learning the causal mechanisms, we are interested in identifying the right generative model that disentangles the causes in the latent space. Speciﬁcally, each z M k and z S should represent distinct causes when z is paired with the right generative model g .
Before beginning the proof, we introduce the impossible result in unsupervised learning of disentangled representations, where they aim to let each latent explanatory factor z i ∈ z carries a distinct type of variations.
Theorem 1.
(Locatello et al., 2019)
For d > 1 , let z ∼ P denote any distribution which admits a density p ( z )
= (cid:81) di =1 p ( z i ) .
Then, there exists an inﬁnite family of bijective functions h : supp( z ) −→ supp( z ) such that ∂h i ( u ) ∂u j (cid:54) =
almost everywhere for all i and j (i.e., z and h ( z ) are completely entangled) and P ( z ≤ u )
= P ( h ( z ) ≤ u ) for all u ∈ supp( z ) (i.e., they have the same marginal distribution).
Theorem 1 indicates we can not identify the disentangled model because we can not tell whether the latent explanatory factors are entangled by h or not from the marginal distribution. Compared to the parameter-space identiﬁability in nonlinear ICA (Khemakhem et al., 2020), identifying a disentangled model is more challenging because even if we can identify the right distribution, the function h may still entangle the latent explanatory factors. In fact, recent generative models in disentangled representations always learn the ground-truth distribution, which is an isotropic Gaussian, via regularization (Kumar et al., 2017; Locatello et al., 2020). Therefore, we formulate our identiﬁability in the function space:
Deﬁnition 1. Let ∼ h be an equivalence relation on g : Z −→ X as follows: [Equation] where h : Z −→ Z is a smooth invertable function.
Deﬁnition 2.
We say that g : Z −→ X is identiﬁable up to ∼ h (or ∼ h -identiﬁable) if: [Equation] Under the deﬁnitions 1 and 2, the following lemma shows that the generative model g is ∼ hidentiﬁable: [Figure]
Lemma 2.
Let G be the space of smooth invertable functions with smooth inverse (i.e., a diffeomorphism) that map Z to X , and h : Z −→ Z is a smooth invertable function.
Then, any function g ∈ G can be represented as g = g ∗ ◦ h , where g ∗ : Z −→ X is the assumed ground-truth disentangled generative model in the function space G .
Formally, we have g ∼ h g ∗ , ∀ g ∈ G and the model g is ∼ h -identiﬁable. The proof is in Appendix A. The following theorem further shows that the function h in ∼ h disentangles the groups of causes for different mechanisms with the unconventional mixture prior.
Theorem 3.
Let p ( z ) denote the unconventional mixture prior. We use M to denote the manifold where the ground-truth latent variable z lies on.
Let k ∈ { 1 , 2 , ..., N } be the index of generative mechanisms M . We assume each z M k lies on M M k and z S lies on M S .
We let ˆ z
= h ( z ) which lies on ˆ M . g ∗ is the ground-truth disentangled model.
Then, if there exists an smooth invertable function h : Z −→ Z such that g = g ∗ ◦ h maps Z to X , we have h maps each M M k to disjoint sub-manifold ˆ M M k and maps M S to ˆ M S , which is disjoint from all ˆ M M k .
We leave the proof of Theorem 3 in Appendix [-B and provide two examples for ICM-conditioned mechanisms in this paragraph. Assume we have two ICM-conditioned mechanisms M i and M j and h : Z −→ Z is an transformation that entangles M i and M j . We further assume z M i ∈ R 2 and z M j ∈ R 2 . Then, we can construct a z where z M i (cid:54) = 0 and z M j = 
-]B.
[-but h ( z ) M i (cid:54) = 
-]
By a slight abuse of notation, we deﬁne the disentangled h as h D .
Then, the disentangled ICM model is ∼ h D -identiﬁable.
E XPERIMENT
For various downstream tasks, we use the encoder network E to extract data representations.
In this section, we ﬁrst examine the robustness of our model against interventions.
As we only apply interventions on the root node, such interventions are equivalent to conditional generation. We visualize the result by latent space traversal.
We then evaluate the robustness of our model under the covariant shift by measuring the downstream ML model performance.
The covariate shift is the change in the distribution of the covariates, that is, the independent variables.
More speciﬁcally, we consider disentangled single covariant shifts (e.g. rotation) and entangled multiple covariant shifts (e.g. brightness and width).
Unlike previous works (Arjovsky et al., 2019; Locatello et al., 2020), which rely on manual annotation, we propose a new method to extract the latent covariants from the datasets and use them to create experiments.
Finally, we measure the robustness of our model against uniform noise and Gaussian noise by measuring the downstream ML model performance.
Through our experiments, we use: 1) Two datasets, which are MNIST and FashionMNIST. 2)
Three competitors, which are VAE (Kingma & Welling, 2013), β -VAE (Higgins et al., 2017), and AdaGVAE (Locatello et al., 2020).
3) Four covariant shifts, which are implicitly carried by the datasets.
5. I NTERVENTIONAL R OBUSTNESS AND L ATENT S PACE T RAVERSAL
We show the latent space traversal of the causal mechanisms on MNIST in Figure 2.
In this experiment, each generative mechanism M k has an one dimensional z M k . The shared mechanisms has a four dimensional z S .
For the k th generative mechanism subspace traversal, we set the z C to k and manually change z M k .
During this process, all the irrelevant dimensions are set to 0 .
As Figure
[Table] [Table] [Table] 2(a) shows, each row represents a traversal of an generative mechanism subspace.
For the shared mechanism, we ﬁrst let z C = 0 .
Then, we do a traversal for a dimension in the shared mechanism subspace.
After this traversal, we increase z C by 1 and repeat the procedure.
In Figures 2(b) and 2(c), each ﬁgure represents a traversal for a dimension in the shared mechanism subspace and each row in the same ﬁgure represents a traversal of a ﬁxed dimension with different z C .
Due to the limited space, we report more visualization results in the Appendix.
We say our ICM model is robust against intervention in the sense that no matter how we change z M k and z S while keeping z C and z \ M k ﬁxed, the generator does not generate data that does not belong to mechanism M k
(e.g. digit 0 does not change to other digits during the traversal).
Furthermore, we show that the traversal of each generative mechanism yields a distinct, mechanisms-speciﬁc type of variation. The traversal of the shared mechanism yields the same type of variation.
Compared with conditional generative models (Kingma et al., 2014; Klys et al., 2018), our work do not pose any requirement on the label during training.
Finally, we note that the number of generative mechanisms is chosen to be larger than the number of classes, which means one ground-truth mechanism may be split into two estimated mechanisms.
Previous works (Parascandolo et al., 2018; Locatello et al., 2018b) also adopt the same setting and we do not ﬁnd it triggers any issue in the following experiment.
5. R OBUSTNESS UNDER COVARIANT S HIFT
To quantitatively measure the robustness under the covariant shift, we ﬁrst partition the dataset using the learned representations as the covariant.
For convince, we use the encoder of the NVAE model (Antoran & Miguel, 2019) to extract the representations.
Using the shared mechanism in our ICM model would yield the same result.
More speciﬁcally, we partition the dataset into subsets { x | x ∈ X , ˆ z i =
E ( x ) i ∈
[ C lower , C upper ) } using the predicted value in dimension i .
For the training set, we set [ C lower , C upper ) to [0 , ∞ ) .
For each test set, we set [ C lower , C upper ) to [ − 0 .
1 , 0) , [ − 0 .
2 , − 0 .
1) , ..., [ − 3 . 0 , − 2 . 9) .
We use the C trainlower − C testupper to represent the strength of the covariant shift, which is also called the shift distance in Table 3.
In this experiment, we consider [Figure] [Table] stroke thickness (T), width and rotation (W&R), rotation (R) and, darkness and width (D&W) as covariants.
Figure 4 in the Appendix further visualizes these covariants
[Table] Then [Table] we use a gradient boosting classiﬁer (GBT) from Scikit-learn with default parameters as the downstream ML model, which is the same as previous works (Locatello et al., 2019; 2020).
We train the classiﬁer with 1000 samples, which is sufﬁcient for producing good accuracy, and test its accuracy on a sequence of test sets.
For each experiment, we evaluate the classiﬁer 10 times and collect the average accuracy as well as the accuracy variations.
Tables 1 and 2 show the average accuracy and the normalized variation of the classiﬁer using each model’s representation as input under different covariant shift, respectively.
Our ICM model achieves the best average accuracy across all the experiments except for MNIST (R).
For the normalized accuracy variations, which is the standard deviation over the accuracy, our ICM model always achieves the lowest variation.
It means that the ICM model is less sensitive to the choice of training samples as the GBT classiﬁer does not contribute many variations.
We further investigate the issue of our model under MNIST (R) covariant shift.
Figure 3 shows the accuracy changes as the shift strength increases.
In both experiments, our method shows more robustness when the shift strength is low as its accuracy decreases slower and its advantage grows bigger.
However, after a threshold, our method begins to lose its advantage.
There are two possible reasons: 1) The test data shifts too far from the training data and the base generative model can not generalize to the test sets.
After the test set shifts too far away, none of the methods perform well.
It’s hard to conclude that a model with ∼ 40% accuracy is better than a model with ∼ 35% accuracy.
The test set there contains too few samples, which is just around tens or hundreds, and makes the evaluation inaccurate.
As we can see from Figure 3, the larger the shift, the bigger the accuracy variations.
To eliminate this interference, we instead measure how much distance the covariant shift needs to go to decrease the accuracy by a percentage relatively.
Such evaluation will put more weight on the test sets which have more samples and yield reasonable accuracy.
Table 3 and Tables 9, 10 in Appendix D.2 show our method can tolerate more covariant shift before the accuracy relatively drops by 10%, 20%, and 40%.
5. R OBUSTNESS AGAINST N OISE
We evaluate the robustness of our ICM model under noise using a similar setting as the previous section.
In this experiment, we use the train the GBT with 1000 samples and test the GBT accuracy on whole train set.
Table 5.3 shows our ICM model is generally more robust against Gaussian noise [Table] N ( 0 , I ) .
Table 5.3 further shows that although there are methods that perform slightly better than our method on MNIST (W&R), our method still performs better before the noise becomes too large and decrease the accuracy by a high percentage.
We further show the results under uniform noise Uniform( 0 , I ) in Appendix D.3.
C ONCLUSION
In this paper, we present a self-supervised method to learn the independent causal mechanisms.
We show the presents of generative mechanisms and shared mechanisms.
Our model can learn these two types of mechanisms through an unconventional mixture prior.
Furthermore, we outline the sufﬁcient conditions for theoretically identifying the mechanisms from observational data.
Experiments show that our ICM model is generally more robust against interventions, covariant shifts, and noise.