Original: Mitigating input-causing confounding in multimodal learning via the backdoor adjustment
Revised: Mitigating input-causing confounding in multimodal learning via the backdoor adjustment

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We adopt a causal perspective to address why multimodal learning often performsworse than unimodal learning.
Revised: We adopt a causal perspective to address why multimodal learning often performs worse than unimodal learning.

Original: We put forth a structural causal model (SCM) forwhich multimodal learning is preferable over unimodal learning.
Revised: We put forth a structural causal model (SCM) for which multimodal learning is preferable over unimodal learning.

Original: In this SCM,which we call the multimodal SCM, a latent variable causes the inputs, and theinputs cause the target.
Revised: In this SCM, which we call the multimodal SCM, a latent variable causes the inputs, and the inputs cause the target.

Original: We refer to this latent variable as the input-causing confounder.
Revised: We refer to this latent variable as an input-causing confounder.

Original: By conditioning on all inputs, multimodal learning d -separates the inputcausing confounder and the target, resulting in a causal model that is more robustthan the statistical model learned by unimodal learning.
Revised: By conditioning on all inputs, multimodal learning d -separates the input-causing confounder and the target, resulting in a causal model that is more robust than the statistical model learned by unimodal learning.

Original: We argue that multimodallearning fails in practice because our finite datasets appear to come from an alternative SCM, which we call the spurious SCM.
Revised: We argue that multimodal learning fails in practice because our finite datasets appear to come from an alternative SCM, which we call the spurious SCM.

Original: In the spurious SCM, the input-causingconfounder and target are conditionally dependent given the inputs.
Revised: In the spurious SCM, the input-causing confounder and target are conditionally dependent given the inputs.

Original: This meansthat multimodal learning no longer d -separates the input-causing confounder andthe target, and fails to estimate a causal model.
Revised: This means that multimodal learning no longer d -separates the input-causing confounder and the target, and fails to estimate a causal model.

Original: We use a latent variable model tomodel the input-causing confounder, and test whether its undesirable dependencewith the target is present in the data.
Revised: We use a latent variable model to model the input-causing confounder, and test whether its undesirable dependence with the target is present in the data.

Original: We then use the same model to removethis dependence and estimate a causal model, which corresponds to the backdooradjustment.
Revised: We then use the same model to remove this dependence and estimate a causal model, which corresponds to the backdoor adjustment.

Original: We use synthetic data experiments to validate our claims.
Revised: We use synthetic data experiments to validate our claims.

Original:  Introduction
Revised:  Introduction

Original: Multimodal learning refers to the joint modeling of data from different modalities such as images,speech, and text [Baltrusaitis et al., 2019].
Revised: Multimodal learning refers to the joint modeling of data from different modalities such as images, speech and text [Baltrusaitis et al., 2019].

Original: It achieves promising results in areas such as joint visionand language modeling.
Revised: It achieves promising results in areas such as joint vision and language modeling.

Original: For example, models trained to contrast images and their textual descriptionsachieve strong zero-shot image classification performance on ImageNet [Deng et al., 2009, Radfordet al., 2021, Jia et al., 2021, Pham et al., 2021].
Revised: For example, models trained to contrast images and their textual descriptions achieve strong zero-shot image classification performance on ImageNet [Deng et al., 2009, Radford et al., 2021, Jia et al., 2021, Pham et al., 2021].

Original: Moreover, these models are highly robust to datadistribution shifts that cripple the performance of their unimodal counterparts trained solely on images.
Revised: Moreover, these models are highly robust to data distribution shifts that cripple the performance of their unimodal counterparts trained solely on images.

Original: These robustness results are significantly better than what has been achieved using only images, andalso exhibits promising scaling behavior [Pham et al., 2021].
Revised: These robustness results are significantly better than what has been achieved using only images, and also exhibits promising scaling behavior [Pham et al., 2021].

Original: The models can also be fine-tuned inorder to achieve state-of-the-art in-distribution classification performance while maintaining theirrobustness [Wortsman et al., 2021].
Revised: The models can also be fine-tuned in order to achieve state-of-the-art in-distribution classification performance while maintaining their robustness [Wortsman et al., 2021].

Original: These results suggest that multimodal learning maybe a part ofthe solution to the longstanding problem of out-of-distribution generalization in deep learning.
Revised: These results suggest that multimodal learning maybe a part of the solution to the longstanding problem of out-of-distribution generalization in deep learning.

Original: Despite these successes, multimodal learning struggles in other areas such as visual question answering.
Revised: Despite these successes, multimodal learning struggles in other areas such as visual question answering.

Original: In this setting, models are given an image and a question as inputs, and are asked to answerthe question.
Revised: In this setting, models are given an image and a question as inputs, and are asked to answer the question.

Original: Progress in this area is stalled by a degenerate behavior where models often answer aquestion while ignoring the image
Revised: Progress in this area is stalled by a degenerate behavior where models often answer a question while ignoring the image

Original: [Agrawal et al., 2016, 2018, Sheng et al., 2021].
Revised: [Agrawal et al., 2016, 2018, Sheng et al., 2021].

Original: More generally, itis frequently observed that multimodal learning performs worse than unimodal learning across a widerange of problems [Cadène et al., 2019, Gat et al., 2020, Wang et al., 2020, Wu et al., 2020].
Revised: More generally, it is frequently observed that multimodal learning performs worse than unimodal learning across a wide range of problems [Cadène et al., 2019, Gat et al., 2020, Wang et al., 2020, Wu et al., 2020].

Original: Given these mixed results, there is an active research direction which aims to make multimodal learningmore effective.
Revised: Given these mixed results, there is an active research direction which aims to make multimodal learning more effective.

Original: In this work, we offer a causal perspective to complement the existing literature.
Revised: In this work, we offer a causal perspective to complement the existing literature.

Original:  When is multimodal learning preferable over unimodal learning?
Revised:  When is multimodal learning preferable over unimodal learning?

Original: [Figure]
Revised: [Figure]

Original: We begin by specifying a structural causal model (SCM) under which multimodal learning has a clearadvantage over unimodal learning.
Revised: We begin by specifying a structural causal model (SCM) under which multimodal learning has a clear advantage over unimodal learning.

Original: It is given by
Revised: It is given by

Original: [Equation]
Revised: [Equation]

Original: where the ϵ ’s are noise variables.
Revised: where the ϵ ’s are noise variables.

Original: We refer to this SCM as the multimodal SCM , and its correspondinggraph is shown in Fig.
Revised: We refer to this SCM as the multimodal SCM , and its corresponding graph is shown in Fig.

Original: 1a.
Revised: 1a.

Original: Since U causes the inputs, we refer to it as the input-causing confounder.
Revised: Since u causes the inputs, we refer to it as the input-causing confounder.

Original: This SCM exhibits unobserved confounding that unimodal learning is vulnerable to, and multimodallearning is invariant to.
Revised: This SCM exhibits unobserved confounding that unimodal learning is vulnerable to, and multimodal learning is invariant to.

Original: Suppose we conduct unimodal learning by only modeling a single modality X along with the target Y .
Revised: Suppose we conduct unimodal learning by only modeling a single modality x along with the target y .

Original: Since the other modality X ′ is unobserved, this corresponds to the graph in Fig.
Revised: Since the other modality x ′ is unobserved, this corresponds to the graph in Fig.

Original: 1b.
Revised: 1b.

Original: This opensthe backdoor path X ← U → X ′ → Y , making X and Y spuriously dependent.
Revised: This opens the backdoor path x ← u → x ′ → y , making x and y spuriously dependent.

Original: A unimodal modelwould learn this spurious dependency, in addition to the causal relation X → Y .
Revised: A unimodal model would learn this spurious dependency, in addition to the causal relation x → y .

Original: In other words, wehave
Revised: In other words, we have

Original: [Equation]
Revised: [Equation]

Original: P ( Y | do ( X )) is unidentifiable from observations alone, since neither U nor Y are observed.
Revised: p ( y | do ( x )) is unidentifiable from observations alone, since neither u nor y are observed.

Original: This isa clear weakness for unimodal learning, since it can fail to generalize under test-time shifts in P ( U ) .
Revised: This is a clear weakness for unimodal learning, since it can fail to generalize under test-time shifts in p ( u ) .

Original: In contrast, a multimodal model, that considers both modalities X and X ′ , d -separates U and Y ,rendering it invariant to shifts in P ( U ) .
Revised: In contrast, a multimodal model, that considers both modalities x and x ′ , d -separates u and y , rendering it invariant to shifts in p ( u ) .

Original: In other words, we have
Revised: In other words, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, under the multimodal SCM, multimodal learning is preferable over unimodal learning,since the former yields a causal model that is more robust to shifts in P ( U ) .
Revised: Therefore, under the multimodal SCM, multimodal learning is preferable over unimodal learning, since the former yields a causal model that is more robust to shifts in p ( u ) .

Original:  Explaining the failures of multimodal learning
Revised:  Explaining the failures of multimodal learning

Original: We have just discussed the multimodal SCM, an SCM for which multimodal learning is more robustthan unimodal learning to shifts in P ( U ) .
Revised: We have just discussed the multimodal SCM, an SCM for which multimodal learning is more robust than unimodal learning to shifts in p ( u ) .

Original: How then, do we explain the frequent failures of multimodallearning?
Revised: How then, do we explain the frequent failures of multimodal learning?

Original: Our argument is based on the idea that a finite dataset sampled from an SCM can appearas though it came from an alternative SCM.
Revised: Our argument is based on the idea that a finite dataset sampled from an SCM can appear as though it came from an alternative SCM.

Original: In our case, although U ⊥⊥ Y | X, X ′ holds in theoriginal multimodal SCM, a finite dataset can appear as though there is a residual dependency,
Revised: In our case, although u ⊥⊥ y | x , x ′ holds in the original multimodal SCM, a finite dataset can appear as though there is a residual dependency,

Original: i.e.
Revised: i.e.

Original: [Figure]
Revised: [Figure]

Original: U ̸⊥⊥ Y | X, X ′ .
Revised: u ̸⊥⊥ y | x , x ′ .

Original: This can make the data appear as if it came from an alternative SCM, where allvariables are defined the same as in Eq.
Revised: This can make the data appear as if it came from an alternative SCM, where all variables are defined the same as in Eq.

Original: (1) except for Y , which is given by
Revised: (1) except for y , which is given by

Original: [Equation]
Revised: [Equation]

Original: This changes the graph by introducing an additional edge U → Y , as seen in Fig.
Revised: This changes the graph by introducing an additional edge u → y , as seen in Fig.

Original: 2a.
Revised: 2a.

Original: We refer to thisalternative SCM as the spurious SCM .
Revised: We refer to this alternative SCM as the spurious SCM .

Original: The edge U → Y poses a problem for multimodal learning, since conditioning on both inputs nolonger d -separates U and Y .
Revised: The edge u → y poses a problem for multimodal learning, since conditioning on both inputs no longer d -separates u and y .

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: Under this setting, multimodal learning may not be more robust than unimodal learning to shifts in P ( U ) .
Revised: Under this setting, multimodal learning may not be more robust than unimodal learning to shifts in p ( u ) .

Original: The influence of the edge U → Y depends on the degree of dependence between f ( U ) and Y giventhe inputs.
Revised: The influence of the edge u → y depends on the degree of dependence between f ( u ) and y given the inputs.

Original: For the sake of explanation, let us rewrite Eq.
Revised: For the sake of explanation, let us rewrite Eq.

Original: 3 as
Revised: 3 as

Original: [Equation]
Revised: [Equation]

Original: where f is a function that modulates how much information about U is passed to Y .
Revised: where f is a function that modulates how much information about u is passed to y .

Original: On one extreme,if f is a constant function, then we recover the original multimodal SCM, and there are no issues withmultimodal learning.
Revised: On one extreme, if f is a constant function, then we recover the original multimodal SCM, and there are no issues with multimodal learning.

Original: On the other extreme, if f ( U ) =
Revised: On the other extreme, if f ( u ) =

Original: Y , then U collapses into Y .
Revised: y , then u collapses into y .

Original: Now, each edgebetween Y and the inputs are bidirectional, turning the graph into a Markov random field (MRF).
Revised: Now, each edge between y and the inputs are bidirectional, turning the graph into a Markov random field (MRF).

Original: This is shown in Fig. 2b, and the joint distribution of the observed variables is given by
Revised: This is shown in Fig. 2b, and the joint distribution of the observed variables is given by

Original: [Equation]
Revised: [Equation]

Original: where the ϕ ’s are potential functions.
Revised: where the ϕ ’s are potential functions.

Original: Prediction is given by
Revised: Prediction is given by

Original: [Equation]
Revised: [Equation]

Original: If we assume that ϕ Y is constant, then this is equivalent to prediction with an ensemble of unimodalmodels.
Revised: If we assume that ϕ y is constant, then this is equivalent to prediction with an ensemble of unimodal models.

Original: This explains why multimodal learning frequently fails to perform better than unimodal learning.
Revised: This explains why multimodal learning frequently fails to perform better than unimodal learning.

Original: Even though U ⊥⊥ Y | X, X ′ holds in the multimodal SCM, a finite dataset may easily appear tohave come from the spurious SCM, where U ̸⊥⊥ Y | X, X ′ holds instead.
Revised: Even though u ⊥⊥ y | x , x ′ holds in the multimodal SCM, a finite dataset may easily appear to have come from the spurious SCM, where u ̸⊥⊥ y | x , x ′ holds instead.

Original: The extent to which this isproblematic depends on the strength of the relation U ̸⊥⊥ Y | X, X ′ .
Revised: The extent to which this is problematic depends on the strength of the relation u ̸⊥⊥ y | x , x ′ .

Original: In the most extreme case wheref ( U ) =
Revised: In the most extreme case where f ( u ) =

Original: Y , a multimodal model becomes equivalent to an ensemble of unimodal models.
Revised: y , a multimodal model becomes equivalent to an ensemble of unimodal models.

Original:  Testing whether U → Y can be observed in our data
Revised:  Testing whether u → y can be observed in our data

Original: Given a finite multimodal dataset D , we should ideally be able to test whether multimodal learning isworth pursuing over unimodal learning.
Revised: Given a finite multimodal dataset D , we should ideally be able to test whether multimodal learning is worth pursuing over unimodal learning.

Original: That is, we want to know whether D is more consistent withthe multimodal SCM, or the spurious SCM.
Revised: That is, we want to know whether D is more consistent with the multimodal SCM, or the spurious SCM.

Original: This amounts to testing whether U ⊥⊥ Y | X, X ′ holds.
Revised: This amounts to testing whether u ⊥⊥ y | x , x ′ holds.

Original: This cannot be done using traditional independence testing, since U is unobserved.
Revised: This cannot be done using traditional independence testing, since u is unobserved.

Original: Here, we presenta way to test this condition using latent variable modeling.
Revised: Here, we present a way to test this condition using latent variable modeling.

Original: We define a latent variable Z to capture anything about Y that cannot be explained using X and X ′ .
Revised: We define a latent variable z to capture anything about y that cannot be explained using x and x ′ .

Original: This information corresponds to the dependence between U and Y given the inputs.
Revised: This information corresponds to the dependence between u and y given the inputs.

Original: We then traina conditional variational autoencoder (VAE) [Kingma and Welling, 2014] to maximize the lowerbound of log P ( Y | X, X ′ ) .
Revised: We then train a conditional variational autoencoder (VAE) [Kingma and Welling, 2014] to maximize the lower bound of log p ( y | x , x ′ ) .

Original: This model consists of an encoder Q ( Z | X, X ′ , Y ) and a decoder P ( Y | X, X ′ , Z ) , both parameterized by neural networks.
Revised: This model consists of an encoder q ( z | x , x ′ , y ) and a decoder p ( y | x , x ′ , z ) , both parameterized by neural networks.

Original: The VAE is conditional because X and X ′ are always observed, and can be considered to be inverted, since a conditional VAE is typicallyconditioned on Y .
Revised: The VAE is conditional because x and x ′ are always observed, and can be considered to be inverted, since a conditional VAE is typically conditioned on y .

Original: The evidence lower bound (ELBO) to be maximized is the right-hand side oflog P ( Y | X, X ′ ) ≥ E Q ( Z | X,X ′ ,Y )
Revised: The evidence lower bound (ELBO) to be maximized is the right-hand side of

Original: [log P ( Y | X, X ′ , Z )] − D KL ( Q ( Z | X, X ′ , Y ) ∥ P ( Z )) .
Revised: [Equation]

Original: Maximizing the ELBO can be thought of as using the encoder Q ( Z | X, X ′ , Y ) to encode information Z that is useful for the decoder P ( Y | X, X ′ , Z ) to later reconstruct Y .
Revised: Maximizing the ELBO can be thought of as using the encoder q ( z | x , x ′ , y ) to encode information z that is useful for the decoder p ( y | x , x ′ , z ) to later reconstruct y .

Original: The KL divergence term inthe ELBO measures how useful Z is in modeling Y given X and X ′ .
Revised: The KL divergence term in the ELBO measures how useful z is in modeling y given x and x ′ .

Original: If there is no useful informationavailable, this means that there is nothing for Z to encode.
Revised: If there is no useful information available, this means that there is nothing for z to encode.

Original: Maximizing the ELBO would then pushthe KL divergence term to zero.
Revised: Maximizing the ELBO would then push the KL divergence term to zero.

Original: This mechanism is the basis of our test for whether multimodallearning is preferable over unimodal learning.
Revised: This mechanism is the basis of our test for whether multimodal learning is preferable over unimodal learning.

Original: If the KL divergence term is close to zero, then it islikely that Z ⊥⊥ Y | X, X ′ , and we can conclude that D is consistent with the multimodal SCM.
Revised: If the KL divergence term is close to zero, then it is likely that z ⊥⊥ y | x , x ′ , and we can conclude that D is consistent with the multimodal SCM.

Original: Therefore, we would expect multimodal learning to be more robust than unimodal learning to shiftsin P ( U ) .
Revised: Therefore, we would expect multimodal learning to be more robust than unimodal learning to shifts in p ( u ) .

Original:  Mitigating the influence of U → Y in our data
Revised:  Mitigating the influence of u → y in our data

Original: [Figure]
Revised: [Figure]

Original: X and X ′ in the original graph in (a), we remove all edges coming into X and X ′ , resulting in the mutilated graph in (b).
Revised: coming into x and x ′ , resulting in the mutilated graph in (b).

Original: We can then compute P M ( Y | X, X ′ ) in the mutilated graph to estimate P ( Y | do ( X ) , do ( X ′ ) under the original graph.
Revised: We can then compute p M ( y | x , x ′ ) in the mutilated graph to estimate p ( y | do ( x ) , do ( x ′ ) under the original graph.

Original: In order to mitigate the influence of U → Y , we must use P ( Y | do ( X ) , do ( X ′ )) for predictioninstead of P ( Y | X, X ′ ) .
Revised: In order to mitigate the influence of u → y , we must use p ( y | do ( x ) , do ( x ′ )) for prediction instead of p ( y | x , x ′ ) .

Original: To do so, we notice that Z satisfies the backdoor criterion relative to( X, Y ) , as well as to ( X ′ , Y )
Revised: To do so, we notice that z satisfies the backdoor criterion relative to ( x , y ) , as well as to ( x ′ , y )

Original: [Pearl, 2009].
Revised: [Pearl, 2009].

Original: Therefore, P ( Y | do ( X ) , do ( X ′ )) is identifiable via thebackdoor adjustment.
Revised: Therefore, p ( y | do ( x ) , do ( x ′ )) is identifiable via the backdoor adjustment.

Original: To do so, we remove the edges going into X and X ′ in the original graph in 
Revised: To do so, we remove the edges going into x and x ′ in the original graph in Fig.

Original: Fig. 3a, which results in the mutilated graph in Fig.
Revised: 3a, which results in the mutilated graph in Fig.

Original: 3b.
Revised: 3b.

Original: P ( Y | do ( X ) , do ( X ′ )) under the distribution P of the original graph is equivalent to P M ( Y | X, X ′ ) under the distribution P M of the mutilatedgraph.
Revised: p ( y | do ( x ) , do ( x ′ )) under the distribution P of the original graph is equivalent to p M ( y | x , x ′ ) under the distribution p M of the mutilated graph.

Original: We therefore have
Revised: We therefore have

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that P ( Z ) and P ( Y | X, X ′ , Z ) are the same across the original and mutilatedgraphs.
Revised: where we used the fact that p ( z ) and p ( y | x , x ′ , z ) are the same across the original and mutilated graphs.

Original: The expectation on the right-hand side of Eq.
Revised: The expectation on the right-hand side of Eq.

Original: 4 may have high variance if we were to sample from P ( Z ) .
Revised: 4 may have high variance if we were to sample from p ( z ) .

Original: We therefore use importance sampling and take the expectation w.r.t.
Revised: We therefore use importance sampling and take the expectation w.r.t.

Original: P ( Z | X, X ′ ) , whichwe can estimate using our approximate posterior Q ( Z | X, X ′ , Y ) from our test in the previoussection.
Revised: p ( z | x , x ′ ) , which we can estimate using our approximate posterior q ( z | x , x ′ , y ) from our test in the previous section.

Original: We use our training set to minimize D KL ( Q ( Z | X, X ′ , Y )
Revised: We use our training set to minimize D KL ( q ( z | x , x ′ , y )

Original: ∥ Q ( Z | X, X ′ )) , and then compute 
Revised: ∥ q ( z | x , x ′ )) , and then compute p ( y | do ( x ) , do ( x ′ )) ≈ 1 M

Original: [Equation]
Revised: M (cid:88) m = p ( z ( m ) ) q ( z ( m ) | x , x ′ ) p ( y | x , x ′ , z ( m ) ) , z ( m ) ∼ q ( z | x , x ′ ) .

Original:  Experimental setup 
Revised:  Experimental setup 6.1 Dataset

Original: 6.1 Dataset

Original: We validate our claims using a synthetic data experiment, where the SCM is
Revised: We validate our claims using a synthetic data experiment, where the SCM is

Original: [Equation]
Revised: [Equation]

Original: Data sampled from this SCM appears to come from the spurious SCM, where β ∈ [0 , 1] defines thedegree of the strength of the dependence of U and Y given the inputs.
Revised: Data sampled from this SCM appears to come from the spurious SCM, where β ∈ [0 , 1] defines the degree of the strength of the dependence of U and Y given the inputs.

Original: Unless otherwise noted, wesample a dataset of one hundred thousand examples, and use 90% for the training set, 10% for thevalidation set, and 10% for the test set.
Revised: Unless otherwise noted, we sample a dataset of one hundred thousand examples, and use 90% for the training set, 10% for the validation set, and 10% for the test set.

Original: 6.2 Model
Revised: 6.2 Model

Original: We experiment with a VAE with encoder Q ( Z | X, X ′ , Y ) and decoder P ( Y | X, X ′ , Z ) .
Revised: We experiment with a VAE with encoder q ( z | x, x ′ , y ) and decoder p ( y | x, x ′ , z ) .

Original: Theencoder consists of two multi-layer perceptrons (MLPs), which turn ( X, X ′ , Y ) into the mean andcovariance of Z .
Revised: The encoder consists of two multi-layer perceptrons (MLPs), which turn ( X, X ′ , Y ) into the mean and covariance of z .

Original: The decoder also consists of two MLPs, which turn ( X, X ′ , Z ) into the mean andcovariance of Y .
Revised: The decoder also consists of two MLPs, which turn ( X, X ′ , z ) into the mean and covariance of Y .

Original: The latent variable Z is a vector with length 100.
Revised: The latent variable z is a vector with length 100.

Original: 6.3 Training methodology
Revised: 6.3 Training methodology

Original: We optimize our models using AdamW [Loshchilov and Hutter, 2019] with a learning rate of 1 × 10 − 3and a weight decay of 1 × 10 − 5 .
Revised: We optimize our models using AdamW [Loshchilov and Hutter, 2019] with a learning rate of 1 × 10 −and a weight decay of 1 × 10 − 5 .

Original: Training is done for 200 epochs with a batch size of 100 , and weearly stop once the validation loss fails to improve for 20 epochs.
Revised: Training is done for 200 epochs with a batch size of 100 , and we early stop once the validation loss fails to improve for 20 epochs.

Original: We use NumPy [Harris et al., 2020],
Revised: We use NumPy [Harris et al., 2020],

Original: PyTorch [Paszke et al., 2019], and PyTorch Lightning [Falcon et al., 2019], where the licensinginformation can be found in the respective papers.
Revised: PyTorch [Paszke et al., 2019], and PyTorch Lightning [Falcon et al., 2019], where the licensing information can be found in the respective papers.

Original: We ran our experiments on a single NVIDIA
Revised: We ran our experiments on a single NVIDIA

Original: RTX8000 GPU on our high performance computing system.
Revised: RTX8000 GPU on our high performance computing system.

Original:  Results
Revised:  Results

Original: First, we demonstrate that our test from Section 4 can detect the influence of the edge U → Y .
Revised: First, we demonstrate that our test from Section 4 can detect the influence of the edge U → Y .

Original: Theresults are shown in Fig.
Revised: The results are shown in Fig.

Original: Fig.
Revised: Fig.

Original: 4a shows that if we vary β in Eq. 5 between one and zero, the KL[Figure] divergence term in the ELBO is significantly larger than zero for β = 1 , and gradually decreasesto zero.
Revised: 4a shows that if we vary β in Eq. 5 between one and zero, the KL divergence term in the ELBO is significantly larger than zero for β = 1 , and gradually decreases to zero.

Original: This is in line with our expectations.
Revised: This is in line with our expectations.

Original: As we decrease β , the influence of the edge U → Ylessens, and X and X ′ are sufficient for modeling Y .
Revised: As we decrease β , the influence of the edge U → Y lessens, and X and X ′ are sufficient for modeling Y .

Original: In Fig.
Revised: In Fig.

Original: 4b, we demonstrate that a similar effectcan be observed when we set β = 0 , and vary the dataset size.
Revised: 4b, we demonstrate that a similar effect can be observed when we set β = 0 , and vary the dataset size.

Original: This confirms our suspicions - eventhough we generated data from the multimodal SCM where the edge U → Y is not present, our finitedatasets can appear to come from the spurious SCM where the edge does exist. 
Revised: This confirms our suspicions - even though we generated data from the multimodal SCM where the edge U → Y is not present, our finite datasets can appear to come from the spurious SCM where the edge does exist. [Figure]

Original: In our next set of experiments, we apply our method from Section 5 to mitigate the influence of
Revised: In our next set of experiments, we apply our method from Section 5 to mitigate the influence of

Original: U → Y .
Revised: U → Y .

Original: For this, we generate a training set from the spurious SCM by setting β = 1 , and comparelog P ( Y | do ( X ) , do ( X ′ )) to log P ( Y | X, X ′ ) across a range of test sets, where we vary β from oneto zero.
Revised: For this, we generate a training set from the spurious SCM by setting β = 1 , and compare log p ( y | do ( x ) , do ( x ′ )) to log p ( y | x, x ′ ) across a range of test sets, where we vary β from one to zero.

Original: This simulates a test-time shift in P ( U ) .
Revised: This simulates a test-time shift in p ( u ) .

Original: Our results in Fig.
Revised: Our results in Fig.

Original: 5 show that P ( Y | do ( X ) , do ( X ′ )is more robust than P ( Y | X, X ′ ) to the shift in P ( U ) , which validates our method.
Revised: 5 show that p ( y | do ( x ) , do ( x ′ ) is more robust than p ( y | x, x ′ ) to the shift in p ( u ) , which validates our method.

Original: [Figure]
Revised: [Figure]

Original:  Related work
Revised:  Related work

Original: Our work is strongly influenced by Wang and Blei [2019], who used a latent variable model toestimate a surrogate confounder, and perform the backdoor adjustment to estimate a causal model.
Revised: Our work is strongly influenced by Wang and Blei [2019], who used a latent variable model to estimate a surrogate confounder, and perform the backdoor adjustment to estimate a causal model.

Original: Their approach, called the deconfounder, assumes the same causal graph as the spurious SCM in ourwork.
Revised: Their approach, called the deconfounder, assumes the same causal graph as the spurious SCM in our work.

Original: The difference between our work is that they use a latent variable Z ∼ P ( Z | X, X ′ ) to modelthe dependence between the inputs.
Revised: The difference between our work is that they use a latent variable z ∼ p ( z | x , x ′ ) to model the dependence between the inputs.

Original: In contrast, our latent variable Z ∼ Q ( Z | X, X ′ , Y ) models thedependence of U and Y given the inputs.
Revised: In contrast, our latent variable z ∼ q ( z | x , x ′ , y ) models the dependence of u and y given the inputs.

Original: Our work also belongs to a recent line of work which estimates intervention distributions to improveout-of-distribution generalization in the setting of high-dimensional unstructured data [Makinoet al., 2022].
Revised: Our work also belongs to a recent line of work which estimates intervention distributions to improve out-of-distribution generalization in the setting of high-dimensional unstructured data [Makino et al., 2022].

Original: While we consider multimodal learning as opposed to multitask learning, our work isconceptually similar to Makino et al.
Revised: While we consider multimodal learning as opposed to multitask learning, our work is conceptually similar to Makino et al.

Original: [2022], since we also use the notion of unobserved confoundingto explain weaknesses in machine learning, and use ideas from causality to improve them.
Revised: [2022], since we also use the notion of unobserved confounding to explain weaknesses in machine learning, and use ideas from causality to improve them.

Original:  Conclusion
Revised:  Conclusion

Original: We addressed the issue of multimodal learning failing to perform better than unimodal learning.
Revised: We addressed the issue of multimodal learning failing to perform better than unimodal learning.

Original: Weput forth the multimodal SCM, under which multimodal learning estimates a causal model, and ismore robust than unimodal learning to shifts in P ( U ) .
Revised: We put forth the multimodal SCM, under which multimodal learning estimates a causal model, and is more robust than unimodal learning to shifts in p ( u ) .

Original: We argue that multimodal learning can fail inpractice because our finite datasets can appear consistent with an alternative SCM, which we call thespurious SCM.
Revised: We argue that multimodal learning can fail in practice because our finite datasets can appear consistent with an alternative SCM, which we call the spurious SCM.

Original: The spurious SCM contains the edge U → Y , which makes U and Y conditionallydependent given the inputs.
Revised: The spurious SCM contains the edge u → y , which makes u and y conditionally dependent given the inputs.

Original: Under the spurious SCM, multimodal learning no longer estimates acausal model.
Revised: Under the spurious SCM, multimodal learning no longer estimates a causal model.

Original: Since the problem is with the edge U → Y , we devise a test that uses latent variablemodeling to detect the influence of U → Y in a given dataset.
Revised: Since the problem is with the edge u → y , we devise a test that uses latent variable modeling to detect the influence of u → y in a given dataset.

Original: The same latent variable model canthen be used to perform the backdoor adjustment and estimate a causal model.
Revised: The same latent variable model can then be used to perform the backdoor adjustment and estimate a causal model.

Original: A current limitation ofour work is that we used synthetic data experiments to validate our claims.
Revised: A current limitation of our work is that we used synthetic data experiments to validate our claims.

Original: We plan to move to amore realistic setting and experiment with more complex multimodal datasets.
Revised: We plan to move to a more realistic setting and experiment with more complex multimodal datasets.

Original: Looking forward, weare excited by the opportunities to use causality to explain and resolve difficult issues in machinelearning.
Revised: Looking forward, we are excited by the opportunities to use causality to explain and resolve difficult issues in machine learning.