Robustness to certain distribution shifts is a key requirement in many ML applications. Often, relevant distribution shifts can be formulated in terms of interventions on the process that generates the input data . Here, we consider the problem of learning a predictor whose risk across such shifts is invariant. A key challenge to learning such risk-invariant predictors is shortcut learning, or the tendency for models to rely on spurious correlations in practice, even when a predictor based on shift-invariant features could achieve optimal i.i.d generalization in principle . We propose a flexible, causally-motivated approach to address this challenge. Specifically, we propose a regularization scheme that makes use of auxiliary labelsfor potential shortcut features, which are often available at training time . Drawing on the causal structure of the problem, we enforce a conditional independence between the representation used to predict the main label and the auxiliary labels. We show both theoretically and empirically that this causally-motivated regularization scheme yields robust predictors that generalize well both in-distribution and under distribution shifts, and does so with better sample efficiency than standard regularization or weighting approaches .
Robustness to certain forms of distribution shift is a key concern in many ML applications. Often, robustness can be formulated as enforcing invariances to particular interventions on the data generating process . Here, we study a flexible, causally-motivated approach to enforcing such invariances, paying special attention to shortcut learning, where a robust predictor can achieve optimal i.i.d generalization in principle , but instead it relies on spurious correlations or shortcuts in practice. Our approach uses auxiliary labels, typically available at training time , to enforce conditional independences between the latent factors that determine these labels. We show both theoretically and empirically that causally-motivated regularization schemes (a) lead to more robust estimators that generalize well under distribution shift, and (b) have better finite sample efficiency compared to usual regularization schemes, even in the absence of distribution shifts. Our analysis highlights important theoretical properties of training techniques commonly used in causal inference, fairness, and disentanglement literature .
R
distribution shifts
forms of distribution shift
R
requirement
concern
R
relevant distribution shifts
robustness
R
in terms of
as enforcing invariances to particular
R
process that generates the input data
data generating process
R
consider the problem of learning a predictor whose risk across such shifts is invariant. A key challenge to learning such risk-invariant predictors is
study a flexible, causally-motivated approach to enforcing such invariances, paying special attention to
R
or the tendency for models to rely on spurious correlations in practice, even when a predictor based on shift-invariant features could
where a robust predictor can
R
. We propose a flexible, causally-motivated approach to address this challenge. Specifically, we propose a regularization scheme that makes use of auxiliary labelsfor potential shortcut features, which are often
, but instead it relies on spurious correlations or shortcuts in practice. Our approach uses auxiliary labels, typically
R
. Drawing on the causal structure of the problem, we enforce a conditional independence between the representation used to predict the main label and the auxiliary
, to enforce conditional independences between the latent factors that determine these
D
this
R
scheme yields robust predictors
schemes (a) lead to more robust estimators
R
both in-distribution and under distribution shifts, and does so with better sample efficiency than standard regularization or weighting approaches
under distribution shift, and (b) have better finite sample efficiency compared to usual regularization schemes, even in the absence of distribution shifts. Our analysis highlights important theoretical properties of training techniques commonly used in causal inference, fairness, and disentanglement literature