Do-Operation Guided Causal Representation Learning with Reduced Supervision Strength
Abstract
Causal representation learning has been proposed to encode relationships between factors presented in the high dimensional data.
However, existing methods suffer from merely using a large amount of labeled data and ignore the fact that samples generated by the same causal mechanism follow the same causal relationships.
In this paper, we seek to explore such information by leveraging do-operation for reducing supervision strength.
We propose a framework which implements do-operation by swapping latent cause and effect factors encoded from a pair of inputs.
Moreover, we also identify the inadequacy of existing causal representation metrics empirically and theoretically and introduce new metrics for better evaluation.
Experiments conducted on both synthetic and real datasets demonstrate the superiorities of our method compared with state-of-the-art methods
Introduction
[Figure]
Causal representation learning [14] has been proposed to extract causal relations from high dimension observations.
To this end, CausalVAE [16] contains a causal layer and a mask layer as parts of deep neural network (DNN) architecture, and uses labels of generative factors to learn the causal relationship between different latent factors.
However, training CausalVAE requires labels of all generative factors, which may still pose a strong assumption.
For instance, all semantic causal factors need to be carefully annotated, which is either costly or hard to be identified in the first place.
Further, since it relies on full supervision, CausalVAE limits the dimensionality of the latent representation to be the same as the number of generative factors and leaves no space for other unknown confounding factors which can be entangled with semantically meaningful latent factors and harm the performance.
Moreover,
CausalVAE incorporates ground-truth generative factors so that causal layer of CausalVAE can be trained separately from VAE.
Thus, rather than extracting causal relations from high dimension observations, causal representation in CausalVAE is merely obtained through ground-truth generative factors, which is also used as a part of inputs during training.
To avoid the constraint of using fully supervised training, we utilize the do-operation , illustrated in Figure 1, to learn causal representation with reduced supervision.
Do-operation [13] defines an
[Figure] [Figure]
[Figure] [Figure]
Causal- [Figure] [Figure]
[Figure] [Figure]
̂'
[Figure]
̂' ! )
[Figure]
operationmodule
[Figure]
[Figure]
̂' !
̂'
[Figure]
[Figure] [Figure]
[Figure]
[Figure]
intervention that remove certain relationships in the causal graph and replace a factor with a constant.
According to Pearl et al.
[13] the causal effects can only propagate from cause factors to effect factors and not inversely.
Thus, when do-operation is applied to cause factors, a new and factual sample will be generated.
Conversely, when do-operation is applied to effect factors, the cause factors should be unaffected.
Further, since do-operation changes the values of effect factors to constants, the newly generated sample can be counterfactual.
When training the model, since the supervision strength is reduced to limited or even no labels, we use two latent representations encoded from a pair of inputs and apply do-operation via exchanging their latent factors with each other.
By comparing the new reconstructions after do-operation with the original inputs, a supervision signal can be introduced.
CausalVAE [16] uses MIC and TIC [8] to evaluate the performance of causal representation learning.
However, MIC and TIC only calculate mutual information between the latent representation and its corresponding ground truth generative factors.
We argue, therefore, that MIC and TIC can only reflect the correctness of the marginal distribution of each factor itself, whereas no causal relationship between factors can be measured.
Therefore, we propose new metrics for better evaluation.
Related work: Disentangled representation learning aims at attaining mutual independent latent factors [1] and Variational Autoencoder (VAE) [7] is the basic framework of most disentanglement methods, where the loss function is L V AE ( x, z ) =
− E [Equation] ϕ ( z | x ) [ logp θ ( x
[Equation]
] + D KL ( q ϕ ( [Equation] || p [Equation]
Other unsupervised VAEs including β -VAE [5] AnnealedVAE [2], LadderVAE [9] and β -TCVAE [3] are proposed by modifying L V AE .
Causal representation learning is the extension of disentangled representation learning.
To achieve causal representation, CausalVAE [16], built upon iVAE [6], removes the requirement of prior knowledge of true causal graph by introducing the causal layer and mask layer into the model.
However, all generative factor labels are required to train CausalVAE.
Method
Model architecture:
We propose a new architecture, shown in Figure 2, as well as a training algorithm that greatly reduces the supervision strength via a do-operation module.
We use x denotes an input image, ε =
[ ε c , ε u ] denotes exogenous latent factors which is further split into causal and unknown nuisance exogenous factors, z denotes latent causal factors, ˆ z denotes latent causal factors after causal discovery and ˆ x denotes reconstructed images.
In contrast with CausalVAE, our framework uses ε u to encode unknown nuisance factors.
Meanwhile, similar to CausalVAE, the exogenous factors ε c are first transformed to endogenous latent factors z , and a causal discovery layer (CDL) propagates causal effects from parent factors to their child factors.
We use a graph autoencoder (GAE) [12] as CDL, which learns nonlinear causal relationships and thus generalizes over NOTEARS [17] used in CausalVAE.
The unknown nuisance latent factors ε u are concatenated with the latent causal factors ˆ z as the input of a decoder.
As discussed in [10], unsupervised learning can not identify expected latent representations so that supervision is necessary.
To reduce supervision strength in CausalVAE and inspired by [11], we use a pair of inputs and implement do-operation during training to utilize a weak supervision signal.
The CDL, that applies causal effect from parent factors to child factors, is the key to implement do-operation , descried in
Section 2, in order to decrease supervision strength.
By using this new training strategy, we show in
Section 4 that no label is needed during training on synthetic datasets and only a small amount of labels is needed on real datasets.
[Figure]
Do-operation illustrates causal relationship:
Dooperation [13] defines an intervention that deletes a specific relationship in the causal graph and replaces factors with constants.
As shown in Figure 1, if do-operation is applied to cause factors, since the original causal graph stays unchanged, the effect factors will be affected according to their parent factors.
Conversely, when do-operation is applied to effect factors, cause factors will not affect the value of effect factors.
This process can be shown in Equations (1) and (2).
[Equation]
where f is the causal relationship function, and generative factors z are split into cause factors z c and effect factors z e .
By assigning previous cause factors z ( l ) c with new value z ( m ) c , effect factors z e will change accordingly.
Oppositely, if do-operation is applied to effect factors z ( l ) e whose value is replaced by z ( m ) e , cause factors z c should stay unchanged.
Besides, The output of causal function f can be counterfactual since the original causal relationship has changed.
Do-operation on cause factors (Do-Cause): As shown in the Equation (1), if we apply do-operation to cause factors z c , since z c have no parent factors, the causal graph is unchanged and the value of effect factors z e should change accordingly.
To train our model, since no label or limited labels of generative factors are available, we use pairs of images as a weak supervision signal to encourage the model to learn causal representation.
As illustrated in Figure 3a, except the regular propagation of inputs, after two endogenous latent factors z 1 and z 2 are encoded from a pair of inputs x 1 and x 2 , we exchange the cause factors of two latent representations with each other to create two new latent representation z ′ 1 =
[ do ( z 1 c ) , z 1 e ] and z ′ 2 =
[ do ( z 2 c ) , z 2 e ] , where a latent factor is cause or effect is determined by the learnable causal matrix A in CDL.
As shown in Equation (3), two new representations z ′ 1 and z ′ 2 are fed into the CDL and then concatenated with their corresponding unknown nuisance factors ε u 1 and ε u 2 as inputs of the decoder.
[Equation]
Recall that from Equation (1), the new outputs of CDL should be same with the original outputs of CDL, where ˆ z ′
=
ˆ z 2 and ˆ z ′
=
ˆ z 1 , since do-operation on cause factors does not change causal graph, and the unchanged causal graph propagates causal relationships from cause factors to effect factors.
Since the new latent causal representation ˆ z ′ 2 and ˆ z ′ 1 should be same with the original latent causal representation z 1 and z 2 , their corresponding reconstructions ˆ x ′ 1 and ˆ x ′ 2 after the decoder should also be same with the the original inputs x 2 , x 1 .
As shown in Equation (4), by comparing new reconstructions with the original inputs, the model is encouraged to learn the correct causal relationships, where d is distance function, such as binary cross entropy or mean square error.
[Equation]
Do-operation on effect factors (Do-Effect): Compared with do-operation on the cause factors, since the causal graph will change when applying do-operation to the effect factors, the latent effect factors should be exchanged after the CDL in order to remove the effect of cause factors.
The whole process of do-operation on the effect factors can be shown in Equation (5).
[Equation]
Since do-effect changes the existing causal graph, the new latent representations ˆ z ′′ 1 and ˆ z ′′ 2 are not consistent with their corresponding latent representations z 1 and z 2 .
Thus, after decoder, the new reconstructions ˆ x ′′ 1 and ˆ x ′′ 2 will be different from their original inputs x 1 and x 2 .
Further, the new reconstructions are are actually counterfactual images as illustrated in Figure 1b.
In practice, using
MSE or BCE may lead to degenerated solution where ˆ x ′′ are random noise.
To solve this issue, we use a classifier C w to distinguish factual images, including x i , ˆ x i and ˆ x ′ i , with counterfactual images ˆ x ′′ i , where classifier and VAE are trained alternatively.
The losses of training classifier and do-operation on the effect factors are shown in Equations (6) and (7) respectively.
[Equation]
Training model with reduced supervision strength: As discussed in Section 1 and empirically proven in Section 4, our method only requires a small amount of supervision to train.
For synthetic datasets, where actually no label are needed, the loss function is shown in Equation (8), where L V AE is same with L V AE shown in Section 1.
[Equation]
where α , β and γ are hyperparameters for regularizations.
||
ˆ z i − z
i || 22 is added to the loss since the outputs of CDL should align with their inputs.
h ( A ) is an acyclicity constraint for the causal graph A .
In our implementation, we use h ( A )
=
tr ( e A ⊙ A ) − d as proposed in [17].
If some labels of generative factors are available, similar to CausalVAE [16], we utilize them by adding label constrains to Equation (8) which leads to Equation (9), where f is CDL.
[Equation]
Evaluation Metrics For Causal Representation Learning
Maximum Information Coefficient (MIC) and Total Information Coefficient (TIC) [8] have originally been proposed as general purpose metrics to measure correlation between two random variables.
Both metrics range from 0 to 1 and the higher value indicates better performance.
[Figure]
CausalVAE [16] suggested using MIC and TIC for evaluating causal representation learning, despite the following inadequacy.
In CausalVAE, MIC and TIC first calculate the information relevance between every ground truth labels and their corresponding learned latent factors.
Then, the means of MIC and TIC for every factors are used as the final metrics values.
However, MIC and TIC only measure correlations between a latent factor and its corresponding generative factor, and can not evaluate the correctness of relationships between cause and effect factors.
Therefore, we argue that MIC and TIC are not suitable for evaluating causal representation learning where the goal is to learn the correct causal relationships between cause and effect factors.
An intuitive example for illustrating the deficiency of MIC and TIC can be found in Appendix.
To address this issue, we propose four new metrics: PosMIC, PosTIC, NegMIC and NegTIC.
PosMIC and PosTIC are used to evaluate the causal relation correctness between latent factors, where higher value are expected.
NegMIC and NegTIC are used to evaluate the falseness of causal relation discovery among latent factors, where lower value are expected.
Additionally, to fully characterize the performance of causal representation learning using a single metric, we propose using the harmonic mean of the new metrics, i.e. F MICand F TIC.
We will first describe how the proposed new metrics are calculated and then discuss their adequacy over the metrics used in CausalVAE.
Calculating PosMIC, PosTIC, NegMIC and NegTIC:
As illustrated in Figure 4, to calculate
PosMIC and PosTIC, given ground truth causal graph G , we first set the latent effect factors ( zand z 4 in Figure 4) to 0 .
If the causal layer learns the correct relationship between the latent cause factors and the latent effect factors, z 3 and z 4 values are determined by the cause factors z 1 and z 2 .
Then, we separately calculate the MIC/TIC values of the latent effect factors and their corresponding
[Table]
generative factors. Finally, the means of the MIC/TIC of all latent effect factors values are taken to be the PosMIC and PosTIC values. NegMIC and NegTIC are calculated in the opposite way, where the latent cause factors are set to 0 , and the final MIC/TIC values are calculated between the latent cause factors after the causal layer and their corresponding generative factors.
Ideally, the causal relationship should unidirectionally propagates from cause to effect, not in the opposite direction.
Thus, the lower NegMIC and NegTIC indicate better performance of causal representation learning.
To better compare different models and fully characterize the performance of causal representation learning, we consider Pos and Neg metrics together by calculating the harmonic mean: F MIC
[Equation]
[Figure]
of PosTIC
and NegTIC is calculated similarly.
[Figure]
[Figure]
Adequacy of proposed metrics:
By conducting experiments on the Pendulum dataset, introduced in Section 4, we empirically show the advantage of the new metrics by proving that MIC and TIC fail to distinguish between models with correct and wrong causal graphs.
We initialize causal graphs A of the CausalVAEs with different causal graphs and stop the gradient of elements if they are initialized with zero, such that CausalVAEs are created with various correctness levels of the causal graphs.
If a causal graph A is initialized identical to the correct causal graph, the performance of that CausalVAE is expected to be optimal since the correct causal relationship is obtained by initialization.
Conversely, the performance of wrong causal graph initialized CausalVAE is expected to be poor.
After training, we calculate correlations among metrics used for causal representation learning and rubrics used in the causal discovery research area: True Positive Rate (TPR), False Discovery Rate (FDR), and Structural Hamming Distance (SHD).
TPR and FDR calculate the rate of discovering correct and wrong causal relations, respectively.
SHD is the minimum number of modifications to correct a causal graph.
As shown in Figure 5, MIC and
TIC have a low correlation with TPR, FDR, and SHD.
In contrast, our proposed new metrics PosMIC, PosTIC, NegMIC, and NegTIC have significant higher correlation with three rubrics used in causal inference.
PosMIC and PosTIC are more positively correlated with TPR, and NegMIC and NegTIC are positively correlated with FDR and SHD.
Experimental Evaluation
Datasets: Following [16], we use two synthetic datasets and two real world datasets.
Pendulum focuses on pendulum angle, light angle, shadow location and shadow length, and Flow focuses on ball size, water height, hole and water flow.
CelebA(SMILE) focuses on gender, smile, eyes open and mouth open, and CelebA(BEARD) focuses on age, gender and beardedness and baldness.
We refer readers to [16] and Appendix for more details.
Besides using MIC and TIC as evaluation metrics, we also use our new metrics for better evaluating causal representation learning.
beard
CausalVAE Ours
[Figure] [Figure]
Synthetic datasets: Our method achieves comparable results on MIC and TIC compared with the fully supervised learning methods CausalVAE [16] and ConditionVAE [15], and outperform other unsupervised learning methods. As shown in Table 1, comparing to CausalVAE and ConditionVAE, our method can achieve slightly better performance on PosMIC, PosTIC, NegMIC and NegTIC.
Unsupervised methods achieve low value on NegMIC and NegTIC due to barely learning semantic information.
Further, the result of using a few labels to train our method is included in Appendix.
Real datasets: Compared with synthetic datasets, there are 40 generative factors in CelebA dataset.
If no label is available during training CelebA datasets, the search space for the model becomes intractable as there are 2 40 different binary causal graphs for 40 factors.
To decrease the difficulty, label information is needed to control the semantic factors which are encoded in each dimension of latent space [4].
For comprehensive comparison, our model and baselines are trained with { 20% , 40% , 60% , 80% , 100% } of labeled data, and the remaining samples are unlabelled.
As shown in Figures 6 and 7, our method consistently and significantly outperforms CausalVAE.
Furthermore, with fewer labels, our method outperforms CausalVAE more appreciably.
Conclusion
In this work, we propose a novel architecture for causal representation learning with reduced supervision strength, exploiting the do-operation .
We use a pair of images and apply do-operation to both latent cause and effect factors for new reconstructions.
By comparing the new reconstructions after do-operation and the original inputs, the supervision strength is reduced.
Furthermore, to better evaluate causal representation learning, we propose new metrics to address adequacy of existing metrics.
We empirically demonstrate the advantages of our method on both synthetic and real datasets.
Acknowledgement: This material is based on research sponsored by Air Force Research Laboratory under agreement number FA8750-19-1-1000.
The U
S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.
A Appendix A.1 Importance of the do-operation
To prove the importance of the different do-operation modules used in our method [Table] we evaluate our model by removing different do-operation modules in the architecture.
As shown in Table A1, by removing Do-Cause, the model loses the ability of finding causal relationship.
Removing Do-effect will lead to performance decrease on NegMIC and NegTIC.
Therefore, the performance on MIC, TIC, PosMIC and PosTIC degrades similar to unsupervised
CausalVAE.
By removing Do-effect and keeping Do-Cause, the performance on MIC, TIC, PosMIC and PosTIC significantly improves, while the performance on NegMIC and NegTIC is worse than full model where both cause and effect do-operation modules are used.
[Table]
A.2 GAE comparison with NOTEARS
As we mentioned in Section [Table] our method incorporate a graph autoencoder (GAE)
[17] as causal discovery layer.
GAE can learn nonlinear structural causal relationships thus generalizing over
NOTEARS [29] which can only learn linear mapping.
As shown in Table A2, if we replace GAE with NOTEARS for causal discovery layer, the performance of our model will be harmed since the causal relationships between latent factors can be nonlinear in many cases.
[Table]
A.3 Synthetic datasets experiments of using a few labels
In Section 4, we demonstrate our method can outperform other methods which do not use the label, and our method can achieve comparable performance compared with CausalVAE evaluated by
PosMIC, PosTIC, NegMIC, and NegTIC.
To test our method more comprehensively on synthetic datasets, we conduct experiments of our method, CausalVAE and ConditionVAE using only 10% of labels.
As shown in Table A3, trained under only 10% of labeled data, CausalVAE and ConditionVAE are difficult to learn either good semantic meaning latent factors which is reflected by MIC and TIC, or attain true causal relationship between cause factors and effect factors, which is shown by PosMIC and PosTIC.
As we discussed in Section 4, since CausalVAE fails to encode useful enough semantic factors information, it achieves a low value on NegMIC and NegTIC.
ConditionVAE achieves low
NegMIC and NegTIC because it aims at learning disentangled latent representation, where each latent factor is enforced to be independent of each other.
Thus no causal relationship, correct or wrong, will be learned.
A.4 Experiments detail
The true causal graph of each datasets are shown in Figure A1.
We use one NVIDIA
Ti GPU as our training and inference device.
Following CausalVAE [26] architecture, we show the VAE architecture of synthetic datasets in Table A4 and VAE architecture
[Table]
of CelebA dataset in Table A5.
For latent representation, we also follow the setting of CausalVAE where latent space z is extanded to matrix z ∈ R n × k and n is the number of concept and k is latent dimension of each concept.
k is set to 4 for VAE used in synthetic datasets and k is set to 32 for VAE used in CelebA dataset [Table]
As described in Section [Table] our loss function for no label training is shown in Equation 10 and the loss for label training is shown in Equation 11.
The hyperparameters ( α, β, γ ) are grid search among { 1 e − 3 , 1 e − 2 , 1 e − 1 , 1 . 0 } .
For training with label, the hyperparameter of l u is always set to 1 .
[Table] [Table]
A.5 Do-operation implementation detail
As we described in section 3, we apply do-operation to both latent cause and effect factors.
To better show the implementation of do-operation in our work, we describe the process in Figure A2.
As illustrated in Figure A2, the cause and effect factors in the latent space are decided by learned causal matrix A which is identical to causal matrix used in causal discovery layer.
After deciding the cause and effect factors, we separately apply do-operation on cause and effect factors.
Applying do-operation to cause factors is straightforward since cause factors have no parent factors and the causal graph stays unchanged.
Oppositely, applying do-operation to effect factors will both fix the value of effect factors and remove affects from cause factors.
Thus, if we swap the effect factors before causal discovery layer, the original causal relationships from cause factors to effect factors still hold.
To eliminate the original causal relationships, the swapping operation on effect factors should be applied after causal discovery layer.
According to [18], do-operation replace factors with constants and remove all causal relationships towards the factors.
If the label information is available, the do-operation is straightforward since the latent factors value can be easily fixed with the label value.
However, if the label information is missing, even though the latent factor value can be replaced by some random values, such random values do not guarantee to be meaningful.
To obtain the proper constants which replace latent factors, another sample is needed since the reconstruction task force the latent representation encoded from the input are meaningful and can be used as source for do-operation .
A.6 Counter example to prove the weakness of MIC and TIC
Assuming we have four independent gaussian variables A , B , C and D , where A ∼ N ( µ a , σ 2 a ) ,
B ∼ N ( µ b , σ 2 b ) , C ∼ N ( µ c , σ 2 c ) and D ∼ N ( µ d , σ 2 d ) .
We can create other four gaussian variables
A ′ , B ′ , C ′ and D ′ where A ′ ∼ N ( µ a , σ 2 a ) , B ′ = µ b µ a · A ′ + ( σ b − µ b µ a σ a ) · N (0 , 1) , C ′ =
µ c µ a · A ′ + ( σ c − µ c µ a σ a ) · N (0 , 1) and D ′ =
µ d µ a · A ′ + ( σ d − µ d µ a σ a ) · N (0 , 1) .
By creating new variables like this, it is easy to see that A ′ has same distribution with A , B ′ has same distribution with B , C ′ has the same distribution with C and D ′ has the same distribution with D .
Since MIC and TIC only evaluate the marginal distribution of each variable separately, they can not distinguish A from A ′ , B from B ′ , C from C ′ and D from D ′ .
However, ( A, B, C, D ) have totally different joint distribution from ( A ′ , B ′ , C ′ , D ′ ) .
A.7 Metrics implement details
The superiorities of the proposed new metrics and a simple example has been discussed in Section 4.
More details about those new metrics will be discussed in this section.
For fully supervised learning or semi-supervised learning method, the metrics calculation are straightforward since every latent elements is controlled by their corresponding label information [5].
For unsupervised methods and our reduced supervision method without using label, we have to first determine the correspondence between each latent factor and each label.
We use MIC to choose which latent element represent the label information.
As we described in Section 4, MIC can be used to measure the information relevance between a latent factor and a generative label.
For each generative factor label, we choose the latent element which achieve maximum MIC value evaluated with that generative factor.
After choosing the correspondence between each latent factor with all generative factors label, we can apply Pos/Neg metrics according to the true causal graph provided by the datasets.
A.8 Reconstruction results
We include the image reconstruction results in this section.
Shown in figs.
A3 to A6, when changing the cause factors, the effect factors shown in reconstructions are changed corresponding.
On the contrary, when changing the effect factors, the reconstructions can be counterfactual images and the cause factors stay unchanged.