Generative multitask learning mitigates target-causing confounding
Anonymous Author(s)
AffiliationAddress email
Abstract
We propose a simple and scalable approach to causal representation learning formultitask learning.
Our approach requires minimal modification to existing MLsystems, and improves robustness to target shift.
The improvement comes frommitigating unobserved confounders that cause the targets, but not the input.
Werefer to them as target-causing confounders.
These confounders induce spuriousdependencies between the input and targets.
This poses a problem for the conventional approach to multitask learning, due to its assumption that the targetsare conditionally independent given the input.
Our proposed approach takes intoaccount the dependencies between the targets in order to alleviate target-causingconfounding.
All that is required in addition to usual practice is to estimate thejoint distribution of the targets to switch from discriminative to generative classification, and to predict all targets jointly.
Our results on the Attributes of Peopleand Taskonomy datasets reflect the conceptual improvement in robustness to targetshift.
Introduction
Deep neural networks (DNNs) excel at extracting patterns from unstructured data, but these patternscan fail to generalize outside of the training distribution.
These failures are often attributed to DNNslearning spurious dependencies rather than causal relations [Ribeiro et al., 2016, Jo and Bengio, 2017,
Geirhos et al., 2020].
Therefore, there is a concerted effort to make patterns learned by DNNs satisfycertain properties of causal relations, such as invariance and modularity [Schölkopf, 2019, Schölkopfet al., 2021].
This research direction is called causal representation learning.
Existing approaches to causal representation learning often require additional information in theform of labeled environments [Arjovsky et al., 2019, Lu et al., 2021] or labeled confounders [Puliet al., 2021].
These requirements are restrictive, and it is therefore beneficial to identify more naturalsettings where existing information can be used for causal representation learning.
We consider the setting of multitask learning (MTL)
[Caruana, 1997, Ruder, 2017, Zhang and Yang,2017], where there is an input X and multiple targets.
Without loss of generality, we consider twotargets, where Y is the main target and Y ′ is the auxiliary target.
Both targets are used for training,but only the main task is evaluated.
The conventional approach to MTL, which we call discriminativemultitask learning (DMTL), assumes the targets are conditionally independent given the input.
Thisleads the inference objective to factorize, i.e.
[Equation]
Since we only evaluate the main target Y , the inference objective is
[Equation]
[Figure]
propose GMTL where the inference objective is argmax
y y log p
| y y between
DMTL does not make causal assumptions explicitly, but its conditional independence assumption isconsistent with the causal graph in Fig.
1a.
We argue that DMTL is flawed under an alternative set ofassumptions.
First, we assume the targets cause the input, which implies we are predicting the cause from the effect.
This is called anticausal learning, and is considered to describe many common problems such asimage classification [Schölkopf et al., 2012].
For example, in object recognition, the object categorycauses the pixel representation of that object.
Second, we assume there exists a variable U that causesthe targets, but not the input.
We call U a target-causing confounder .
The problem with DMTL is that the auxiliary target Y ′ is unobserved in Eq.
This is illustrated in
Fig.
1b.
This is problematic because it opens up the backdoor path Y ← U
→ Y ′ → X , and makes
X and Y spuriously dependent.
Since the backdoor path passes through U , DMTL is sensitive tointervention on U , which is a shift in p ( u ) .
When p ( u ) shifts, the relationship between X and Ychanges, and DMTL fails to generalize.
We refer to this issue as target-causing confounding.
We propose generative multitask learning (GMTL) as a way to mitigate target-causing confounding.
GMTL is based on the following idea.
If we write the causal factorization of the joint distribution ofthe observed variables as
[Equation]
this shows that p ( x | y, y ′ ) is invariant to intervention on U .
Therefore, if we use
[Equation]
for the inference objective, we become invariant to intervention on U .
Importantly, unlike Eq. 1, thisinference objective no longer factorizes over the targets.
This means that even if we only care aboutevaluating the main target Y , we also need to consider Y ′ .
This is the main idea behind GMTLconditioning on all targets during inference closes the backdoor paths between the inputs and targets.
This d -separates X and U , and removes the spurious dependencies between the input and targets.
Eq. 3 also shows that p ( y, y ′ ) changes in response to intervention on U .
This means that beinginvariant to intervention on U is equivalent to being robust to shifts in p ( y, y ′ ) .
A shift in p ( y, y ′ )is commonly referred to as target shift [Zhang et al., 2013].
While our goal is to be invariant tointervention on U , in the remainder of this work we will speak in terms of robustness to target shift.
This is because shifts in p ( y, y ′ ) are measurable, unlike shifts in the unobservable p ( u ) .
In order to validate GMTL, we perform experiments on two datasets: Attributes of People [Bourdevet al., 2011] and Taskonomy [Zamir et al., 2018].
Across various pairs of tasks on these datasets, our results show that GMTL improves robustness to target shift.
We attribute this improvement to
GMTL’s ability to mitigate target-causing confounding.
This empirical success is evidence that causalrepresentation learning shows promise for MTL.
Background2.1 Out-of-distribution generalization
Out-of-distribution (OOD) generalization refers to the capability of ML systems to generalize on dataoutside of the training distribution.
Bengio et al.
[2021] describe it as one of the greatest challengesin ML, and one that cannot be solved solely by scaling up datasets and computation.
The failure of
ML systems to generalize OOD is a general problem that has been reported across a wide range ofproblems spanning computer vision and natural language processing [D’Amour et al., 2020].
Thecore issue is this - when ML systems are trained to optimize an objective function, it is often easierfor them to learn patterns that are dataset-specific, rather than those that hold universally.
Thesedataset-specific patterns have many names, such as surface statistical regularities [Jo and Bengio,2017], non-robust features [Tsipras et al., 2019], and shortcuts [Geirhos et al., 2020].
We take a causalperspective and adopt the term spurious dependencies , which are defined as statistical dependenciesnot supported by causal links [Pearl, 2009].
2.2 Dataset biases induce spurious dependencies
The following example shows that spurious dependencies can arise from design choices in datasetcreation.
Recht et al.
[2019] demonstrated that state-of-the-art image classifiers trained on ImageNet [Deng et al., 2009] failed to generalize to a replicated test set designed to mirror the originaltest set distribution.
Engstrom et al.
[2020] offered an explanation for this phenomenon.
The authorsreported that the dataset replication procedure relied on estimating a human-in-the-loop metric calledselection frequency, and that bias in estimating this statistic led to undesirable variation across datasets.
Selection frequency is defined for an input-target pair, and measures the proportion of crowdsourcedannotators who deem the pair correctly labeled.
Recht et al.
[2019] performed statistic matchingon this metric, which corresponds to conditioning on it.
Since selection frequency is caused by theinput and target, this corresponds to conditioning on a collider, which induces a spurious dependencybetween the input and target.
Such a dependency would fail to generalize OOD, which may explainwhy predictive performance degraded on the replicated test set.
In this work, we address a different source of dataset bias that we call target-causing confounding.
A variable is a target-causing confounder if it is the common cause of the targets, but not the input.
For a simple example, suppose we have images of people in indoor scenes, and the targets areclothing-related attributes such as hat and scarf.
The season causes the targets, since people pairvarious types of clothing depending on the season.
The season does not cause the image, sincethe images are of indoor scenes.
Therefore, the season is a target-causing confounder that inducesspurious dependencies between the input and targets.
Relying on such dependencies for predictioncan be problematic if the training and test sets represent different seasons.
Although this is a contrivedexample for the purpose of illustration, dataset biases can occur in real-life settings like medicine,and have serious societal impacts [Mehrabi et al., 2021].
Therefore, mitigating such biases is animportant goal for fair and equitable ML.
2.3 Causal representation learning
In Section 2.1, we made the distinction between patterns that are dataset-specific and those thathold universally.
Causal inference formalizes the notion of patterns that hold universally, callingthem causal relations [Pearl, 2009].
Causal representation learning aims to use ideas from causalityto improve machine learning methods, particularly in the area of out-of-distribution generalization [Zhang et al., 2013, Schölkopf, 2019, Schölkopf et al., 2021, Arjovsky et al., 2019, Puli et al.
Our work falls under this category, since GMTL improves robustness to target shift by usingan intervention distribution for prediction.
This idea is also shared by Subbaswamy et al.
[2019],who propose a general framework for estimating intervention distributions for robust prediction.
GMTL is a practical method for achieving this in MTL on large-scale, high-dimensional datasets.
Another causality-motivated paper for robust prediction is Makar et al.
[2022], who use importance reweighting to learn distributions that are invariant to intervention on an auxiliary target.
Both
Subbaswamy et al.
[2019] and Makar et al.
[2022] share the same causal graph as GMTL.
2.4 Multitask learning
MTL can reduce computational cost through parameter sharing, and improve predictive performanceover training on each task individually
[Standley et al., 2020].
Its use is widespread, spanningdomains such as scene understanding [Chowdhuri et al., 2019, Zamir et al., 2018, 2020], medicaldiagnosis [Kyono et al., 2021], natural language processing [McCann et al., 2018, Wang et al., 2019,
Radford et al., 2019, Liu et al., 2019, Worsham and Kalita, 2020, Aghajanyan et al., 2021], recommender systems [Covington et al., 2016, Ma et al., 2018], and reinforcement learning [Kalashnikovet al., 2021].
Distinguishing between main and auxiliary targets is common [Ruder, 2017, Liebel and Körner, 2018,
Vafaeikia et al., 2020], but not universal, and is primarily important for concerns such as which lossto use for early stopping [Caruana, 1997].
We assume this distinction exists throughout this paper,denoting the main task as Y and the auxiliary task as Y ′ .
However, our method still applies if wespecify an arbitrary target to be the main target, and the others as being auxiliary.
Generative multitask learning
In GMTL, we condition on all targets in order to d -separate X
and U , and remove spurious dependencies between the input and targets.
During inference, we rely solely on p ( x | y, y ′ ) , a distribution thatis invariant to intervention on U .
Approaching this naively by estimating p ( x | y, y ′ ) directly can bedifficult, especially when X is high-dimensional.
We can get around this by using Bayes rule to write
[Equation]
This shows that the argmax over the targets results in the difficult-to-estimate p ( x ) term to drop out,leaving us with p ( y, y ′
| x ) and p ( y, y ′ ) .
In other words, to implement GMTL, we must estimatep ( y, y ′
| x ) as we have been doing with DMTL, and additionally estimate p ( y, y ′ ) .
3.1 Interpolating between DMTL and GMTL
In practice, we replace the inference objective of GMTL in Eq.
5 with the following approximation:
[Equation]
This introduces two changes.
The first is that we allow p ( y, y ′
| x ) to factorize, as was the case in
DMTL.
This is purely out of convenience, and lets us keep the training of multitask networks thesame as DMTL.
This is beneficial since multitask networks are notoriously difficult to train [Standleyet al., 2020].
We can allow this factorization as long as we capture the dependency between thetargets when estimating p ( y, y ′ ) .
This prevents Eq.
6 from factorizing over the targets, and satisfiesour requirement of conditioning on all targets in order to remove spurious dependencies between theinput and targets.
The second change is the parameter α ∈ [0 , 1] , which enables us to interpolate between DMTL( α = 0) and the most extreme case of GMTL ( α = 1) .
α controls the degree to which we removespurious dependencies between the input and targets.
The ability to adjust this effect is importantbecause if the training and test distributions are identical, then we should be willing to use anydependencies available, regardless of whether they are spurious or causal.
However, it is morerealistic to assume that training and test distributions will be different, and under these circumstances,setting α > 0 can improve robustness to target shift.
3.2 Interpreting α
The parameter α represents our uncertainty about the test target distribution.
In the case of completecertainty, that is, if we know that the test target distribution is identical to the empirical target distribution, we set α = 0 to use the empirical distribution.
In contrast, in the case of completeuncertainty, we set α = 1 to completely remove the influence of the empirical distribution.
This isequivalent to assuming the test target distribution is uniform.
Generally, it can be shown that GMTLcorresponds to replacing the empirical target distribution p ( y, y ′ ) with a distribution proportional top ( y, y ′ )
1 − α .
The derivation is in the supplementary materials.
We perform this replacement duringinference in order to keep training the same as DMTL.
In principle, we could achieve the sameinvariance by reweighting examples by 1 /p ( y, y ′ ) α during training [Zhang et al., 2013, Makar et al.,2022].
This raises an important point - α represents an assumption, and is not a hyperparameter that can betuned in the conventional sense.
Instead, tuning α is analogous to model selection w.r.t.
an unknowndistribution, which is a difficult open problem [Gulrajani and Lopez-Paz, 2021].
Progress in thisdirection is likely to benefit GMTL, as well as other methods with similar parameters [Wortsmanet al., 2022].
In the remainder of this work, we assume oracle access to the optimal α for a given
OOD test distribution.
That is, our main focus is not on the difficulty of tuning α , which is a separateresearch question.
Instead, we study the efficacy of GMTL assuming we have correctly assessed howsimilar the training and test target distributions are, i.e. knowing the optimal α .
Having said that, we provide a simple and effective heuristic for setting α even for an unknowntest distribution, which makes GMTL highly practical.
This is based on the fact that α represents atrade-off.
Increasing α improves target shift robustness at the expense of IID predictive performance.
We show empirically in Section 6 that this relationship is very strong, and holds across two datasets,multiple pairs of tasks, and three MTL methods.
Therefore, our heuristic is to set α to the maximumvalue within an acceptable loss in IID predictive performance, where the latter is measured usingthe validation set.
This ensures that we are as robust as possible to target shift, while keeping IIDpredictive performance at an acceptable level.
These results are in the supplementary materials.
If we do have access to the test target distribution q ( y, y ′ ) , then we can do away with α and simplyreplace p ( y, y ′ ) with q ( y, y ′ ) , which results in the following inference objective:
[Equation]
Alternatively, we can use q ( y, y ′ ) /p
( y, y ′ ) to correct for target shift during training [Zhang et al.
3.3 Estimating p ( y, y ′ )
GMTL requires estimating p ( y, y ′ ) , which is trivial for categorical targets.
The maximum likelihoodestimate of p ( y, y ′ ) is obtained by counting the number of occurrences of the pair ( y, y ′ ) , and dividingby the total across all pairs.
However, there is an important precaution to be made, since it is possiblethat certain pairs of ( y, y ′ ) are never observed.
This is problematic, since p ( y, y ′ )
=
0 results intaking a logarithm of zero in Eq.
The issue of sparsity in the target distribution is the primarychallenge in scaling GMTL across tasks.
This can be addressed with additive smoothing, where apseudocount ϵ > 0 is added to the count for each pair prior to normalization.
An illustrative example
We construct a synthetic example where a target-causing confounder U induces a spurious dependencybetween X and Y .
We illustrate how DMTL is sensitive to this dependency, while GMTL is invariantto it.
Suppose there are two targets Y, Y ′ ∈ { 0 , 1 } that cause the input X ∈ R .
Let us assume thatp ( y, y ′ ) is bivariate Bernoulli, where P ( Y = 1)
= θ , P ( Y ′ = 1)
= θ ′ , and Cov[ Y, Y ′ ]
=
σ Y,Y ′ .
Xis generated by a mixture of Gaussians with latent variable Z = 2 Y + Y ′ , i.e. X | z ∼ N ( x ; z, σ 2 z ) .
The value of σ Y,Y ′ is set by an unobserved variable U .
U is a target-causing confounder, since itcauses Y and Y ′ , but not X .
We fix θ = θ ′ =
5 , σ 2 Z =
= σ 2 Z =1 = 0 .
4 , and
σ 2 Z =2 =
σ 2 Z =3 = 0 . 6 ,and interpolate σ Y,Y ′ from -0.2 to 0.2.
In other words, the only property of the data generatingprocess that changes is the covariance between Y and Y ′ .
Fig.
2 shows how the decision boundaries of DMTL and GMTL with α = 1 are affected by changes inσ Y,Y ′ .
The decision boundary of DMTL is the value of x for which the solution to argmax y p ( y | x )changes, where Y ′ has been marginalized out.
In contrast, the decision boundary of GMTL is the value of x where the Y component of the solution to argmax y,y ′ p ( x | y, y ′ ) changes.
The decisionboundary for DMTL changes in response to σ Y,Y ′ , while it remains constant for GMTL.
This isbecause GMTL only uses the Gaussian densities for prediction, which have no connection to σ Y,Y ′ .
In this example, we have shown that GMTL and DMTL arrive at different solutions by changing onlythe dependency between Y and Y ′ induced by the target-causing confounder U .
The key differencebetween the two methods is that DMTL uses the spurious dependency between X and Y , while
GMTL does not.
Which of the two methods achieves better predictive performance depends on theparticular test distribution, since the spurious dependency will be predictive in IID settings, and notin OOD settings.
[Figure]
Experimental setup
We perform experiments to demonstrate that GMTL improves robustness to target shift.
We evaluateacross a wide range of target shifts ranging from mild to severe, and analyze how the optimal αchanges w.r.t.
the severity of target shift.
Recall from Section 3.2 that we assume oracle access tothe optimal α , since we are focusing on validating GMTL, not on whether we can perform modelselection w.r.t.
unknown distributions, which is a separate open problem.
The purpose of theseexperiments is to show that if we correctly assess how similar the training and test distributions are,we can benefit from using GMTL.
In order to draw robust conclusions, we perform our experimentsusing two datasets, multiple pairs of classification tasks, and three MTL methods.
Here we describeour experimental setup.
5.1 Datasets and tasks
Attributes of People The Attributes of People dataset [Bourdev et al., 2011] consists of 8,035images of people.
There are 4,013 examples in the training set, and 4,022 in the test set.
Since theauthors did not specify a validation set, we use 20% of the training set as the validation set.
Eachimage comes labeled with up to nine binary attributes: male, long hair, glasses, hat, t-shirt, longsleeves, shorts, jeans, and long pants.
We use each attribute as a binary classification task and predictwhether the attribute appears in the image.
We drop the gender attribute, since it has the potential fornegative downstream applications
[Wang and Kosinski, 2018].
We experiment with various pairs oftasks, taking turns specifying one as the main task, and the other as the auxiliary task.
For each pairof tasks, we report the classification accuracy for the main task.
Since not all attributes are labeled foreach example, we consider the three pairs of tasks with the highest number of co-occurrences.
Theseare: hat and long sleeves, long hair and hat, and glasses and hat.
Taskonomy The Taskonomy dataset [Zamir et al., 2018] is of a much larger scale than the firstdataset, containing approximately four million images of indoor scenes.
We use a subset of thedataset that is provided by the authors for faster experimentation, with 548,785 examples in thetraining set, 121,974 in the validation set, and 90,658 in the test set.
Each image comes paired withtwenty six labels relevant to scene understanding, but since we are focusing on classification, we onlyuse the object and scene annotations as targets.
We use the object annotations as a 100-way objectclassification task, and the scene annotations as a 64-way scene classification task.
Between objectand scene classification, we take turns specifying one as the main task, and the other as the auxiliarytask.
We report the top-1 accuracy for the main task.
5.2 MTL methods
No Parameter Sharing (NPS)
The first MTL method we use is a trivial combination of thetwo trained STL networks.
We call this No Parameter Sharing (NPS).
This is equivalent to STLnetworks during training, but becomes different during prediction if α > 0 , since the STL networksinfluence one another through the − α log p ( y, y ′ ) term.
While NPS is primarily meant to be a simplesetting for comparing GMTL and DMTL, it offers some practical utility as well.
MTL methodsare notoriously difficult to train, and it is often challenging to get them to perform better than the
STL baseline [Alonso and Plank, 2017].
NPS combined with GMTL offers a way to combine two
STL networks to reap the benefits of MTL, just by estimating p ( y, y ′ )
and predicting jointly over thetargets.
Shared Trunk Networks (STN)
The second MTL method is a hard parameter sharing method thatwe call Shared Trunk Networks (STN), where all weights except for the final classification layer areshared.
This is not an effective architecture for many of the tasks, since the predictive performanceis worse than the single-task baseline.
This is common for STN [Alonso and Plank, 2017], but weinclude it for completeness because it is the canonical MTL architecture.
The results comparing STLand MTL methods on the IID test set are in the supplementary material.
Cross-stitch Networks (CSN)
For the third MTL method, we use a soft parameter sharing methodcalled Cross-stitch Networks (CSN)
[Misra et al., 2016].
CSN takes two trained STL networks, andtakes a linear combination of the activations at each layer during the forward pass.
Training CSNinvolves learning the coefficients of the linear combinations, as well as fine-tuning the STL networkweights.
This network performs better than the STL baseline across most datasets and tasks.
Full Parameter Sharing (FPS)
For our experiments on Attributes of People, we model the jointtarget space directly as a four-class classification.
Since this is a single network with all weightsshared across both tasks, we call this Full Parameter Sharing (FPS).
We omit FPS on Taskonomy dueto the complications of a large and sparse joint target space, and leave this to future work.
5.3 Methodology for training and prediction
For all datasets and tasks, we use ResNet-50 [He et al., 2016] pretrained on ImageNet [Deng et al.
,2009] for the STL networks.
We train using Adam [Kingma and Ba, 2015] with L 2 regularization.
We tune the learning rate and regularization multiplier.
For data augmentation, during training weresize the images to 256 × 256 , randomly crop them to 224 × 224 and randomly horizontally flip them.
During validation and testing, we resize the images to 256 × 256 , and center crop them to 224 × 224 .
For all experiments, we train STL networks from five random initializations.
Details regarding otherhyperparameters are included in the supplementary materials.
5.4 Simulating target shift with importance sampling
In these experiments, we wish to evaluate across a wide range of target shifts.
Since it is impracticalto collect many different test sets, we simulate target shifts using importance sampling.
Suppose thatthe original test distribution is p ( x, y, y ′ )
=
p ( x | y, y ′ )
p ( y, y ′ ) , and the simulated test distributionis q ( x, y, y ′ ) =
p ( x | y, y ′ )
q ( y, y ′ ) .
p ( x | y, y ′ ) is invariant due to our assumption that it is a causalrelation.
The expected accuracy l ( x, y, y ′ ) under the simulated test distribution q ( x, y, y ′ ) is given by [Equation] q ( y ( n ) [Equation]
y ′
[Equation]
p [Equation] y ′ [Equation]
) ≈
N
[Equation]
p [Equation] Now that we can evaluate across a wide range of OOD q ( y, y ′ ) ’s, we need a way to quantify theseverity of the target shift between p ( y, y ′ ) and q ( y, y ′ ) . 5.5 Measuring the severity of target shift between p ( y, y ′ ) and q ( y, y ′ )
Our metric to measure the severity of target shift between p ( y, y ′ ) and q ( y, y ′ ) must be such thatpredictive performance degrades w.r.t.
the severity of shift.
Not all metrics satisfy this criteria.
Tobuild intuition, consider a single task binary classification problem where P ( Y = 1)
= θ .
If we usea norm-based metric such as total variation, this treats a change in θ from 0.6 to 0.8 and a changefrom 0.6 to 0.4 as being the same.
However, from the point of view of prediction, the latter should bemore detrimental, since it reverses the ranking of the classes.
Similarly, Kullback-Keibler (KL) divergence is a common choice for measuring the distance betweendistributions, but it suffers from a related problem.
Consider θ changing from 0.9 to 0.9999, and from0.9 to 0.4.
Due to taking the logarithm of a small number, KL divergence considers the first to be amore severe shift.
This is undesirable, because the change from 0.9 to 0.4 represents a reversal in theranking of the classes.
These examples point to a need for a ranking-based metric, since a significant change to the rankingof classes is detrimental to predictive performance.
Weighted rank correlation satisfies this.
If theweighted rank correlation between p ( y, y ′ ) and q ( y, y ′ ) is positive with large magnitude, it meansthat there are no significant changes in the ranking of classes, and that the target shift is not severe.
Incontrast, if the weighted rank correlation is negative with large magnitude, it implies the rankingsof classes has changed significantly.
This constitutes a more severe target shift.
We therefore useweighted Kendall’s τ
[Shieh, 1998], which we henceforth refer to as τ , to measure the severityof target shift between p ( y, y ′ ) and q ( y, y ′ ) .
We provide the definition of τ in the supplementarymaterials.
It relies on a weighting function, for which we use 1 / ( r + 1) , where r is the ranking.5.
Sampling q
( y, y ′ )
Our choice of distance metric informs how to sample q ( y, y ′ ) , since our goal is to evaluate across awide range of target shift severities.
We designed a method which shuffles log p ( y, y ′ ) and perturbsit with noise.
To be precise, let f ( y,
y ′ ) ∈ { 1 , 2 , . .
|Y × Y ′ |} such that each ( y, y ′ ) is assigned aunique integer index.
We use p f ( y,y ′ )
=
p ( y, y ′ ) Let i = α · |Y × Y ′
| , where α ∼ U (0 , 0 . 5) .
Werandomly shuffle these indices with π : { 1 , 2 , . . .
, i } → { 1 , 2 , . . .
, α × i } .
then, we define q ( y, y ′ )by
[Equation]
f ( y
[Equation]
y
[Equation]
where ϵ ∼ N (0 , σ 2 ) and σ ∼ U (10 − 12 , 5) .
Results
We compute the OOD test set accuracy across target shifts ranging from mild to severe, where p ( y, y ′ )is the original test set distribution, and q ( y, y ′ )
a randomly sampled OOD distribution.
First, wesample an OOD q ( y, y ′ ) , and compute τ relative to p ( y, y ′ ) .
τ represents the severity of target shiftbetween q ( y, y ′ ) and the original test distribution p ( y, y ′ ) .
We then use importance sampling tocompute the OOD accuracy under each q ( y, y ′ ) across a range of α ∈ [0 , 1] .
The results are split into five groups that are equally spaced in terms of τ .
Each group represents theaccuracies for a range of α under a particular severity of target shift.
For each group, we computethe optimal value of α that attains the highest accuracy.
We also compute the difference in accuracybetween GMTL and DMTL at the optimal value of α for each group.
We observe strikingly similar results for both datasets and all pairs of tasks.
As the target shiftbecomes more severe, the optimal α increases.
A representative example for each dataset is shown inthe left subfigures in Fig.
3, and the rest are in the supplementary materials.
This observation matchesour understanding of α from Section 3.2 - since α is our degree of belief on how similar p ( y, y ′ )and q ( y, y ′ ) are, the optimal α tends to be higher when the target shift is more severe.
Increasing αremoves more of the spurious dependencies induced by target-causing confounding, which is morebeneficial for severe target shifts.
Also, as the severity of target shift increases, the difference in accuracy between GMTL and DMTLfor the optimal α increases as well.
This can be seen in the right subfigures in Fig.
3, and is also
[Figure]
The reason is that the spurious dependencies induced by target-causing confounding become less predictive as the shift severity increases, and therefore removing them yields larger improvements in accuracy.
in-line with our expectations.
As the target shift becomes more severe, the spurious dependenciesinduced by target-causing confounding become less predictive, and removing them yields larger gainsin predictive performance.
Both patterns hold very similarly for all MTL methods.
This suggests that it is our formulation of
GMTL that is significant, rather than the particular parameterization used to learn p ( y, y ′ | x ) .
Boththe optimal α , as well as the corresponding gain in accuracy using GMTL, increase monotonicallyw.r.t.
the severity of target shift.
This is a strong relation that holds across datasets, pairs of tasks, and
MTL methods.
To quantify the generality of this result, we aggregate all of our results, and computethe correlation between τ and the optimal α .
Since τ is used in the context of an interval, we takethe midpoint of each interval.
This results in a correlation of -0.847.
This indicates a strong positivecorrelation between the severity of target shift and the optimal α .
The generality of this relation isstrong evidence that GMTL improves robustness to target-causing confounding, and also that ourinterpretation of α is correct.
Our experimental results using the α -selection heuristic described in
Section 3.2, as well as other results showing the accuracy for a range of α , not just the optimal one,are in the supplementary materials.
Conclusion
We presented generative multitask learning (GMTL), which is an approach for causal representationlearning for MTL that requires minimal changes to existing ML systems.
Our approach mitigatesthe effect of target-causing confounders, which are variables that cause the targets, but not the input.
This removes spurious dependencies between the input and targets, and improves robustness to targetshift.
Looking forward, we are excited by the new perspectives that causal representation learning canbring to existing ML paradigms.
Our work demonstrates the potential for this in the context of MTL.
We plan to investigate what other areas of ML can be improved by adopting a causal perspective.