  Due to spurious correlations, machine learning systems often fail to
generalize to environments whose distributions differ from the ones used at
training time. Prior work addressing this, either explicitly or implicitly,
attempted to find a data representation that has an invariant causal
relationship with the target. This is done by leveraging a diverse set of
training environments to reduce the effect of spurious features and build an
invariant predictor. However, these methods have generalization guarantees only
when both data representation and classifiers come from a linear model class.
We propose Invariant Causal Representation Learning (ICRL), a learning paradigm
that enables out-of-distribution (OOD) generalization in the nonlinear setting
(i.e., nonlinear representations and nonlinear classifiers). It builds upon a
practical and general assumption: the prior over the data representation
factorizes when conditioning on the target and the environment. Based on this,
we show identifiability of the data representation up to very simple
transformations. We also prove that all direct causes of the target can be
fully discovered, which further enables us to obtain generalization guarantees
in the nonlinear setting. Extensive experiments on both synthetic and
real-world datasets show that our approach significantly outperforms a variety
of baseline methods. Finally, in the concluding discussion, we further explore
the aforementioned assumption and propose a general view, called the Agnostic
Hypothesis: there exist a set of hidden causal factors affecting both inputs
and outcomes. The Agnostic Hypothesis can provide a unifying view of machine
learning in terms of representation learning. More importantly, it can inspire
a new direction to explore the general theory for identifying hidden causal
factors, which is key to enabling the OOD generalization guarantees in machine
learning.
