W HAT W OULD THE E XPERT do ( · ) ?:
C AUSAL I MITATION
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We develop algorithms for imitation learning from policy data that was corrupted by unobserved confounders.
Sources of such confounding include (a) persistent perturbations to actions or (b) the expert responding to a part of the state that the learner does not have access to.
When a confounder affects multiple timesteps of recorded data, it can manifest as spurious correlations between states and actions that a learner might latch on to, leading to poor policy performance.
To break up these spurious correlations, we apply modern variants of the classical instrumental variable regression (IVR) technique, enabling us to recover the causally correct underlying policy without requiring access to an interactive expert.
In particular, we present two techniques, one of a generative-modeling ﬂavor ( DoubIL ) that can utilize access to a simulator and one of a game-theoretic ﬂavor ( ResiduIL ) that can be run entirely ofﬂine.
We discuss, from the perspective of performance, the types of confounding under which it is better to use an IVR-based technique instead of behavioral cloning and vice versa.
We ﬁnd both of our algorithms compare favorably to behavioral cloning on a simulated rocket landing task.
I NTRODUCTION
Much of the theory of imitation learning (IL) indicates that with enough demonstrations, we should be able to accurately recover the expert’s policy.
When we apply IL algorithms in practice however, we sometimes see them produce manifestly incorrect estimates of the expert’s policy (Muller et al., 2006; Codevilla et al., 2019; de Haan et al., 2019; Bansal et al., 2018; Kueﬂer et al., 2017).
One possible reason for this phenomenon is that empirically, we only have access to noisy recordings of what the expert did.
This critical detail has been thus far neglected by most prior theoretical work in imitation learning.
We focus in this paper on how best to learn from two kinds of noisy data:
• Exogenous noise : When we observe expert actions corrupted by a persistent noise (e.g. a faulty joystick that persistently perturbs actions before they are executed in the game).
• Endogenous noise : When we do not observe the full state an expert used to pick an action (e.g. the learner not knowing there’s an enemy behind a door).
The net effect of either kind of persistent noise (more formally, an unobserved confounder ) is to introduce temporal correlations in the recorded actions that do not have their true cause in the recorded state.
For example, consider recordings of an expert driver slowing down at a stop sign.
If all we present the learner with as state input is whether the expert was slowing down at the last timestep, they will likely learn to simply repeat the expert’s past action.
Thus, once the car begins to slow down, it continues to slow down, regardless of whether there is a stop sign present .
At a more abstract level, these sorts of inertia problems can result from temporal correlations between pairs of actions (e.g. the effect of the stop sign) being reﬂected in the state (e.g. the past action variable), leading to spurious correlations between state and action that the learner might unfortunately latch onto (e.g. repeating the past action).
What should we hope to learn then in these confounded settings?
Given we do not have access to the unobserved confounder, a reasonable choice is to ensure that we match the behavior of an expert that has access to the same information we do.
That is, if we could query the expert for an action with only the information we have available, we should strive to produce an action that matches
[Figure]
this queried action. While applying an interactive imitation learning algorithm (Ross et al., 2011) would allow us to collect a dataset uncorrupted by confounding, a queryable expert is not a realistic assumption for many domains.
We therefore focus on approaches for the off-policy setting.
We base our algorithms on a technique from econometrics for dealing with confounding in recorded data known an instrumental variable regression (IVR) (Angrist et al., 1996).
The high-level idea of
IVR is to leverage an instrument , a source of random variation independent of the confounder, to deconfound inputs to a learning procedure via conditioning on the instrument.
In dynamical systems, history can act as this source of variation, as it is unaffected by future confounding (Hefny et al., 2015). Our key insight is that we can leverage past states as instruments to break the spurious correlation between states and actions caused by an unobserved confounder .
Our work provides the following contributions:
We formalize confounding in imitation learning.
We provide a structural causal model that captures inertia effects that result from temporally correlated exogenous or endogenous noise.
We also derive a test to detect whether this sort of confounding is present in a dataset. 2.
We present a uniﬁed derivation of modern instrumental variable regression techniques.
We show how two recent extensions of the classical IVR technique share a common structure.
We also extend the theoretical analysis of previous work by deriving accuracy bounds.
We provide two novel algorithms to deal with confounding in imitation learning.
We derive two novel IVR-based algorithms: • DoubIL is a generative modeling approach that and can utilize access to a simulator for reduced sample complexity.
• ResiduIL is a simulator-free, game-theoretic approach.
We derive performance bounds for policies produced by these algorithms under exogenous noise.
We empirically investigate the effect of the persistence of the confounder on this bound.
We also compare the performance of these approaches to behavioral cloning under endogenous noise.
R ELATED W ORK
Imitation Learning.
Broadly speaking, imitation learning approaches can be grouped into three classes: ofﬂine, online, and interactive.
Our work is most similar to ofﬂine imitation learning algorithms (e.g. Behavioral Cloning (Pomerleau, 1989), ValueDice (Kostrikov et al., 2019), AdVIL (Swamy et al., 2021)) that operate purely on collected data.
Unlike previous work however, we consider the effect of unobserved confounding.
Our work shares the goal of interactive imitation learning algorithms (e.g. DAgger (Ross et al., 2011), AggreVaTe (Ross & Bagnell, 2014)), in that we seek to match the output of a query to the expert.
However, we focus on matching the output of a query on recorded data, rather than on learner rollouts, as is standard for interactive approaches.
This is because the confounder decouples the actions recorded in the data and queried actions.
Zhang et al.
(2020) consider imitation learning through the lens of causal inference but focus on the one-step setting, while we consider multiple timesteps.
Kumor et al.
(2021) contemporaneously consider the
P ROBLEM
C ORRELATED S TATE
A CTION
C ONFOUNDER
[Table]
multi-step setting and come to similar conclusions as us about the challenges of endogenous noise. They derive a necessary and sufﬁcient structural condition for successful imitation learning, while we focus on practical algorithms with performance guarantees for a particular graphical model under exogenous noise.
In contrast to imitation learning methods that seeks to match moments of the expert’s behavior (Swamy et al., 2021), we focus only on matching average expert actions. We leave matching arbitrary moments to future work.
Inertia Effects in Imitation Learning.
Several authors have empirically observed a latching effect in policies trained via imitation learning: (Muller et al., 2006; Codevilla et al., 2019; de Haan et al., 2019; Bansal et al., 2018; Kueﬂer et al., 2017), where learned policies tend to inappropriately repeat the same action.
We seek to provide a plausible explanation and correction for the phenomenon reported in these works, and list several examples in Table 1.
We note that when attempting to explain inertia effects, de Haan et al.
(2019) propose causal confounding as the root cause of the error.
However, as previously pointed out by Spencer et al.
(2021), there is no actual confound in the theoretical or empirical examples in the work of de Haan et al.
This is because the learner observes all of the variables the expert was using to make decisions.
Instrumental Variable Regression.
The classical approach to instrumental variable regression (Wright, 1928) is a two-stage least squares procedure (e.g. in Angrist et al.
(1996)’s textbook).
We focus on the more general nonlinear setting and instead base our approaches on the more recent
D EEP IV (Hartford et al., 2017) and AGMM (Dikkala et al., 2020).
We present extensions to the work in these papers, including a uniﬁed derivation of both methods and error analysis for D EEP IV.
A B RIEF R EVIEW OF I NSTRUMENTS IN C AUSAL M ODELING
Z X Y
U V g h
- sider We are interested in ﬁnding h , the causal relationship from X to Y , even though there is an unobserved confounder, U . To do so, we leverage the effect of Z , which provides independent randomness from U . We begin by discussing the concept of an instrument before deriving our algorithmic approaches in a simpliﬁed, non-sequential setting.
Let X , Y , and Z be random variables on (potentially inﬁnite) sample spaces X , Y , and Z .
Assume that X , Y , and Z have the causal, rather than statistical, dependency structure in Fig.
Given a dataset of ( x, y, z ) tuples, we are interested in determining the causal relationship between X and Y , E
[ Y | do ( x )] , where do ( · ) is the interventional operator of Pearl et al.
Intuitively, E [ Y | do ( x )] is the expected value of Y when we intervene and set X = x , rather than observe such an X .
In the SCM to the right, h ( x ) =
E [ Y | do ( x )] .
Because of the presence of an unobserved confounder, U , that affects both X and Y , standard regression (e.g. Ordinary Least Squares or OLS) generically produces inconsistent estimates.
Coarsely, this occurs because OLS will over-estimate the inﬂuence of the parts of X that are affected by the confounder.
If we only have observational data and are unable to perform randomized control trials, a canonical technique to recover h is IVR (Winship & Morgan, 1999).
Formally, an instrument Z must satisfy three structural conditions:
Unconfounded Instrument : Z ⊥⊥ U – i.e. independent randomization from confounder.
Exclusion :
Z ⊥⊥ Y | X, U – i.e. no extraneous paths.
Relevance : Z (cid:54)⊥⊥ X – i
e. conditioning has an effect.
Z satisﬁes these three conditions in the SCM of Fig.
1 Without loss of generality, we assume that E [ U ] = 0 .
This allows us to concisely derive a set of conditional moment restrictions (CMR):
[Equation]
In words, these constraints are saying that a necessary condition for recovery of h ( x ) is that for all values of the instrument, the actual and predicted expected values of Y | Z are equal.
We further assume that noise U enters additively to Y , 2 and write out the following equations:
[Equation]
We now derive an appropriate loss function for ﬁnding an (cid:98) h that approximately satisﬁes the CMR.
If we only have ﬁnite samples and can therefore only estimate conditional expectations up to some tolerance, it is natural to relax the CMR to
[Equation]
where the δ z are slack variables.
Then, the Lagrangian (with the natural P ( z ) -weighted inner product that captures how often each we expect each z to occur) is
[Equation]
where λ is the vector of Lagrange multipliers.
By the stationarity component of the KKT conditions,
[Equation]
implying that δ z = λ z .
Plugging this back into the Lagrangian, we can simplify our function to
[Equation]
We refer to (7) as the Regularized Lagrangian or ReLa for short.
Now, solving for the optimal
Lagrange multipliers via stationarity, we arrive at
[Equation]
which implies the optimal λ z is equal to E [ Y − (cid:98)
h
( X ) | z ] .
Plugging this back into (7) recovers the loss function,
[Equation]
This expression is the square of the Projected Root Mean Squared Error (PRMSE) of Chen & Pouzo (2012).
To recap, by minimizing Eq.
9, we are attempting to ﬁnd an (cid:98) h that approximately satisﬁes the CMR.
Minimizing PRMSE is a necessary condition for recovering E [ Y | do ( X )] .
For it to be a sufﬁcient condition, one needs the natural identiﬁability assumptions – we refer interested readers to Chen & Pouzo (2012) for a more thorough discussion.
3. G ENERATIVE M ODELING A PPROACH
How should we minimize the PRMSE then?
One option is learning the distribution P ( X | z )
=
g ( z ) , passing samples from it to a candidate (cid:98) h , and trying to match E [ Y | z ] .
This is a generalization of the standard Two-Stage Least Squares (2SLS) (Angrist et al., 1996) procedure to nonlinear functions.
The nonlinearity of the second stage means that one cannot simply compute the ﬁrst moment of the P ( X | z ) distribution, which is recovered by linearly regressing from X to Z in the 2SLS procedure.
This sort of approach was ﬁrst proposed for the IVR setting by Hartford et al.
and amounts to ﬁrst learning a g ( z ) (e.g. via maximum likelihood estimation) and then solving
[Equation]
[Equation]
The work of Hartford et al.
(2017) did not have theoretical analysis regarding the effect of errors in g ( z ) upon attempts to learn h ( x ) .
We prove the following in Appendix A:
Theorem 1.
Assume we learn a g ( z ) s.t.
[Equation]
Then, optimizing (10) to value (cid:15) corresponds to recovering a (cid:98) h ( x ) s.t. PRMSE ( (cid:98) h ) ≤ √ δ + √ (cid:15) .
3. G AME -T HEORETIC A PPROACH
One can also proceed by instead solving the two-player zero-sum game with the ReLa (7) as the payoff.
Denoting by f ∈ F = {Z → R } the function that maps z ’s to their Lagrange multipliers, we can write this game as
[Equation]
This game is the core objective of the AGMM method of Dikkala et al.
Importantly, one does not need to learn a generative model of P ( X | z ) for these sorts of game-theoretic approaches.
We prove the following theorem in Appendix A:
Theorem 2.
Assume that H and F are bounded, closed under negation, convex, compact, and that h ∈ H and ∀ (cid:98) h ∈ H , f ( z ) =
E
[ Y − (cid:98) h
( X ) | z ] ∈ F .
Then, an (cid:15) -approximate
Nash equilibrium of (11) corresponds to recovering a (cid:98) h (
x ) s.t. PRMSE ( (cid:98) h ) ≤ √ (cid:15) .
One can ﬁnd such an equilibrium via a standard reduction to no-regret online learning (Freund & Schapire, 1997).
In summary, one can frame nonlinear IVR as a generative modeling or game-theoretic problem, leading to different error characteristics.
We now turn our attention to applying these methods to imitation learning with unobserved confounders.
C AUSAL C ONFOUNDING IN I MITATION L EARNING
the effect of exogenous noise, unlike IV. We begin with a brief, intuitive sketch to illustrate the challenges of confounding for IL: consider an expert trying to ﬂy a quadcopter straight but their actions being perturbed by wind (i.e. a form of exogenous noise).
Because it attempts to reproduce expert actions, behavioral cloning would reproduce these deviations, producing trajectories that deviate even further from a straight path in a windy environment.
In contrast, by ﬁltering out the effects of the confounder, a policy trained via IVR would only be affected by the wind present at test time and therefore produce trajectories similar to those of the expert.
We now formalize this sort of confounding and how one can use IVR to mitigate its effects.
We use ∆( S ) to mean the set of distributions over S and focus on a Markov Decision Process (MDP) parameterized by (cid:104)S , A , T , r, T (cid:105) , where S is the state space, A is the action space,
T : S ×
A → ∆( S ) is the transition operator, r : S × A → [ − 1 , 1] is the reward function, and T is the horizon of the problem.
Let J ( π ) =
E τ ∼ π
[ (cid:80) Tt =1 r ( s t , a t )] , Π ⊆ {S → ∆( A ) } be the policy class we optimize over and d π be the visitation distribution of policy π .
In the presence of unobserved confounding, the trajectories generated by the expert can be captured by the structural causal model (SCM) in Fig.
3 Fig.
4 captures both the exogenous and endogenous noise settings.
In the exogenous noise setting, the confounder u t − 1 could be a persistent noise that affects pairs of actions, while in the endogenous noise setting, u t − 1 can be thought of as a response to a part of the state that the learner does not have access to (see Table 1 for several example)
In either setting, the confounding travels through the dynamics to inﬂuence the next state, leading to spurious correlations between the recorded states and actions.
We can also see this correlative effect by writing out the structural equations:
[Equation]
a t −a t ... ...
s t −s t u t −u t u t −
X
Y
Z π E π E T
Fig.
4 also tells us that Z = s t − 1 satisﬁes the three conditions to make it a valid instrument for countering the effects of U =
u t − 1 .
Intuitively, this is because the past state is independent of the current confounder, allowing it to function as an independent source of randomness.
One can imagine longer time-scale correlations induced between actions than just one step confounding – our approaches naturally extend to this setting by using a state further back in the past as the instrument.
Unlike standard imitation learning approaches like behavior cloning which attempt to recover E [ a | s ] , an approach based on IVR enables us to instead recover the interventional effect of the policy E [ π E ( s ) | s ] =
E [ a | do ( s )] .
Conceptually, E [ a | do ( s )] is asking what the expert would do on average if they were placed in state s , i.e. the kind of answer we would get from a queryable expert in DAgger (Ross et al., 2011) or DAeQuIL (Swamy et al., 2021).
However, as we are only interested in the result of queries on states from expert demonstrations, we are able to get a similar effect via IVR to an interactive approach without requiring access to a queryable expert.
We now build upon this intuition to derive two algorithms.
A LGORITHMS FOR C AUSAL I MITATION L EARNING
We now present two approaches for causal imitation learning that can be seen as applications of the generative modeling and game-theoretic approaches of Sec. 3.
At their core, both algorithms are attempting to minimize a PRMSE objective,
[Equation]
instead of the usual IL objective,
[Equation]
In both the exogenous and endogenous settings, minimizing equation 14 corresponds to recovering E [ a | do ( s )] .
What differs is the performance implications of doing so, as we now discuss further.
5. E XOGENOUS N OISE
Exogenous noise is present both in the demonstrations as well as at test time.
Our goal in this setting is to eliminate the effect of the confounder so at test time we do not needlessly reproduce its effects (e.g. the increased swerving in our quadcopter example).
Notice that under exogenous noise, minimizing (15) to 0 would not recover the expert’s policy while minimizing Equation (14) would.
First, let a distribution P ( U ) be c-Total Variation stable (Bassily et al., 2021) if:
[Equation]
This property is satisﬁed by a wide variety of distributions.
For example, for standard normal random variables, c = 1 / 2 .
Next, in our setting, the measure of ill-posedness (Dikkala et al., 2020; Chen &
Pouzo, 2012) is
[Equation]
We prove the following bound on policy performance in Appendix B:
Theorem 3.
Assume P ( U ) is c -TV Stable exog.
noise, π E is deterministic, and let κ (Π) be the measure of the ill-posedness of the problem.
Then, PRMSE ( π ) ≤ (cid:15) ⇒ J
( π E ) − J ( π ) ≤ cκ (Π) (cid:15)T 2 .
Intuitively, κ (Π) measures the strength of the strength of the instrument.
Consider the extreme case where s (cid:48)
=
s .
Then, κ (Π) = 1 .
As the past state becomes a weaker instrument, κ (Π) > 1 .
Thus, if the confounding affects multiple timesteps, we would expect κ (Π) to grow as one needs to reach further back in time to ﬁnd a valid instrument, leading to a looser performance bound.
We investigate the effect of the length of confounding on the ill-posedness of the problem empirically in Sec. 6. E NDOGENOUS N OISE
Endogenous noise is present only in the expert demonstrations and not applied during learner rollouts. Thus, in contrast to the exogenous setting, we do not need to eliminate the effect of the confounder to perform as well as the expert. Instead, we hope to effectively reduce our uncertainty over the confounder (e.g. the state of the trafﬁc light). We begin by deﬁning the following policies:
[Equation] where supp denotes support. We prove the following results under endog.
noise in Appendix A: Lemma 1. There exist MDPs for which π E , π BC , and π IV have different trajectory distributions. Lemma 2.
If reward r is a function of state and action only, J ( π E ) =
J ( π BC ) always, while there exist MDPs for which J ( π E ) >
J ( π IV ) . Lemma 3. If reward r ∈ {S × A × U → R } (i.e. the reward additionally depends on the con- founder), then there exist MDPs for which J ( π E ) >
J ( π ) [Equation]
One take-away from these lemmas is the fundamental difﬁculty of producing a value equivalent policy to the expert’s under endogenous noise, a result that was concurrently derived via a graphical condition by Kumor et al.
(Defn. 2.3). Consider, for example, an expert driver that stops at an intersection when a trafﬁc light is red.
If the learner does not see this light, there is no way for them to ensure they match the behavior of such an expert. However, in the special case where the reward function does not directly depend on the confounder (i.e. eliminating the natural reward function that penalizes the learner for not obeying the trafﬁc light), π BC is value equivalent to the expert, while, perhaps counter-intuitively, causally consistent π IV is not. The value difference is because π IV marginalizes out u t − 1 by sampling from P ( u t − 1 ) while π BC samples from P ( u t − 1 | s t ) (see proof of Lemma 1 in Appendix A). π BC is therefore better able to estimate the value of the confounder by utilizing the information in the current state, which is advantageous in the endogenous setting.
5. W ITH A S IMULATOR : D OUB IL
Algorithm 1 DoubIL Input: Dataset D E of expert trajectories, Policy class Π , Simulator (cid:98) T Output: Trained policy ππ 1 = arg min π ∈ Π E s,a ∼D E [ − log π ( a | s )] { Train preliminary policy via moment-matching. } D IV = { (
[Equation]
s, [Equation]
s
[Equation]
[Equation]
D E } { Pass π 1 ’s actions through simulator. } π
[Equation]
arg [Equation] ] { Train ﬁnal policy on new dataset and output. }
[Figure]
inputs to the second stage by resimulating state transitions.
Algorithm 1 can be seen as a variation of generative modeling approach of Sec. 3 and Hartford et al.
(2017) where one leverages knowledge of one factor of the P ( X | z ) distribution and just learns the other factor.
Via the Markov assumption, we can factorize P ( X | z ) =
P ( S (cid:48) | s )
= (cid:80) a ∈A P ( a | s ) T ( s, a ) .
Assuming access to a simulator (cid:98) T that closely approximates the true transition dynamics, we can instead focus on learning the P ( a | s ) component: the standard imitation learning task.
Notably, this ﬁrst-stage policy is biased as it includes the effect of the confounder: P ( a | s ) =
P ( U + π E ( s ) | s ) .
However, when we use it to simulate transitions, the next states that are produced no longer have the particular instantiation of the confounder present in the recorded dataset’s next actions.
Using a tilde to denote a fresh draw from a distribution, simulated states are drawn from
[Equation]
19) while the observed next actions are drawn from
[Equation]
Notice that there are no shared noise terms.
This allows us to apply standard imitation learning to this new dataset of ( (cid:101) s t , a t ) to learn a causally consistent policy.
The two applications of imitation learning lead us to term this algorithm DoubIL .
We can translate the guarantee of Theorem 1 to our factored context:
Lemma 4.
Assume we learn a π 1 ( s ) s.t.
[Equation]
Then, optimizing the second-stage MSE to (cid:15) corresponds to recovering a π 2 s.t.
[Equation]
We prove this lemma in Appendix A. Combining this lemma with Theorem 3 allows one to derive a performance bound of J ( π E ) − J ( π ) ≤ cκ (Π)( √ δ + √ (cid:15) )
T 2 under exogenous noise.
We note that one could simply learn the mapping P ( s (cid:48) | s ) but this can be far less sample efﬁcient than merely learning a policy when |A| ≤ |S| , as is often true in practice.
5. W ITHOUT STATE RE - SAMPLING : R ESIDU IL
Algorithm 2 ResiduIL
Input: Dataset D E of expert trajectories, Policy class Π , Discriminator class F , Learning rate η
Output: Trained policy π
Set π ∈ Π , f ∈ F , (cid:101) g π = 0 , (cid:101) g f = 0 while π not satisfactory do
[Equation]
{ Payoff of zero-sum game. }
[Equation]
{ Perform Optimistic Mirror Descent. }
π ← π − η
[Equation]
g
[Equation] f ← [Equation] (cid:101) g
[Equation] ← g
[Equation] , (cid:101) g f ← g f end while Algorithm 2 is the direct application of the game-theoretic approach of Sec.
and Dikkala et al.
(2020) to imitation learning.
We term it ResiduIL because the adversary attempts to predict the residual between the learner and the expert’s actions while the learner attempts to minimize this residual.
Notably, this algorithm can be run completely ofﬂine (i.e. without access to a simulator).
We use the Optimistic Mirror Descent approach of Syrgkanis et al.
(2015) to ﬁnd approximate Nash equilibria in our experiments.
Once again, we can extend our past results to the IL setting:
Lemma 5.
An (cid:15) -approximate equilibrium for the policy player corresponds to recovering a policy
π s.t PRMSE ( π ) ≤ √ (cid:15) .
This lemma dovetails with Theorem 3 to prove that J ( π E ) − J ( π ) ≤ cκ (Π) √ (cid:15)T 2 under exogenous noise (Appendix A).
E XPERIMENTS
We test DoubIL and ResiduIL on a slightly modiﬁed version of the OpenAI Gym (Brockman et al., 2016)
LunarLander-v2 environment against a behavioral cloning baseline.
We generate [Figure] demonstrations by simulating rollouts of an expert policy trained via PPO (Schulman et al., 2017), adding fresh Gaussian noise to the expert’s action as well as cached noise from the last timestep. The latter noise is the confounder.
See Appendix B for full parameters. We judge policy quality by computing the MSE between the output of a deconfounded expert query and a learner’s proposed actions on states from expert rollouts. We see that both of our methods are able to more closely match E [ a | do ( s )] than behavioral cloning, especially in the low-data regime (Fig. 6, left). We also measure the MSE on states from deconfounded expert rollouts – while there are no clear guarantees on this state distribution, we see that our methods generalize better than BC empirically (Fig. 6, middle).
One might wonder how, given a dataset of expert demonstrations, one detects whether there is unobserved confounding in the data.
We can answer this question by comparing the results of behavioral cloning and either of our above algorithms.
We prove the following in Appendix A:
Lemma 6.
Assume π BC ( s ) =
E [ a | s ] and π IV ( s ) =
E [ a | do ( s )] .
Then, E [ u | s ] = π BC ( s ) − π IV ( s ) .
The implication of this lemma is that comparing the outputs of IVR-based procedures to behavioral cloning can help us detect causal confounding – if they greatly differ with a sufﬁciently sized dataset, there is likely an unobserved confounding effect in our data.
Moreover, the states where they differ represent the parts of the state space where the inﬂuence of the confounder is highest.
Fig.
6 right is an empirical example of how the test of Lemma 6 can be used to identify areas of the state space where the effect of the confounder is especially strong (e.g. the center).
[Figure]
we vary the number of steps a confounder sticks around for.
For linear problems, we can bound κ (Π) (the measure of illposedness) via an eigenvalue ratio (Dikkala et al., 2020).
Extending our previous model to include the effect of the last H confounders ( a t [Equation] E
[Equation]
s t )
+ (cid:80) tj = t − H u j . )
, we arrive at the bound [Equation] [Equation] We compute this quantity empirically for a linear-quadratic problem with Gaussian confounding and plot results in Fig.
As expected, we see that increasing the length of confounding leads to weaker instruments as one has to use states further back in time.
Theorem 3 tells us that under exogenous noise, we should expect a weaker instrument to lead to a larger performance gap between the learner and expert.
See Appendix B for full experimental setup details.
C ONCLUSION
We present a model that captures confounding in imitation learning and derive two algorithms,
DoubIL and ResiduIL , that are able to utilize history as an instrument to mitigate the effects of unobserved confounders.
We prove performance bounds and validate their empirical efﬁcacy under exogenous noise.
We also discuss the challenges of learning with endogenous noise.
A P ROOFS A. P ROOF OF V ALIDITY OF I NSTRUMENT
Proof.
We check the instrument conditions in order:
Unconfounded Instrument : Z ⊥⊥ U :
The Z → X ← U , V → X ← U , and X → Y ← U triples are blocked by standard d-separation rules (Pearl et al., 2016).
All paths from Z to U must pass through one of these triples so Z ⊥⊥ U .
Exclusion :
Z ⊥⊥ Y | X, U : The Z → X → Y , X ← U → Y , and V → X → Y triples are blocked by standard d-separation rules.
All paths from Z to Y must pass through one of these triples so Z ⊥⊥ Y | X, U . 3.
Relevance : Z (cid:54)⊥⊥ X : There is a Z → X edge, which is assumed to be non-degenerate.
Thus, Z is a valid instrument for determining the causal relationship between X and Y .
A. P ROOF OF T HEOREM
Proof.
We simplify notation for clarity in our proof.
Consider two vectors of the same dimension, a and b .
Assume that (cid:80) Ni a 2 i ≤ (cid:15) and (cid:80)
Ni b 2 i ≤ δ .
This implies that (cid:107) a (cid:107) 2 ≤ √ (cid:15) and (cid:107) b (cid:107) 2 ≤ √ δ .
Then, by the triangle inequality, (cid:107) a − b (cid:107) 2 ≤ (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 ≤ √ (cid:15) + √ δ .
Setting a i = (cid:112)
P ( z )( E [ Y | z ] − E ˆ x ∼ g
( z ) [ (cid:98) h
(ˆ x )]) and b i = (cid:112) P ( z )( E ˆ x ∼ g
( z ) [ (cid:98) h
(ˆ x )] − E [ (cid:98) h
( x ) | z ]) proves that
[Equation]
⇒
[Equation]
A. P ROOF OF T HEOREM
Proof.
The population version of (11) is
[Equation]
[Equation]
An (cid:15) -approximate equilibrium is an ( (cid:98) h, (cid:98) f ) pair such that:
[Equation]
[Equation]
[Equation]
Taking the derivative w
[Equation]
r
[Equation]
t f ( z ) of the payoff and setting it equal to
[Equation]
we arrive at
[Equation]
P
[Equation] [Equation]
P
[Equation]
z [Equation] f ( z ) =
⇒ f
( z ) =
E
[ Y − (cid:98) h
( X ) | z ] .
Plugging this back into (49) gives us the inequality
[Equation]
Assuming we are in the realizable setting (e.g. h ( x )
=
E [ Y | do ( x )] ∈ H ), min h ∈H E [2( Y − h
( X ))
(cid:98) f
( Z ) − (cid:98) f 2 ( Z )] ≤ 0 .
Thus, we can write that:
[Equation] [Equation]
We note that Theorem 2 follows somewhat directly from the main theorems of Dikkala et al.
but that it was not stated in this precise form in their work.
A. P ROOF OF L EMMA
Proof.
We focus on the ﬁrst two timesteps of the problem.
By construction, p ( s 0 ) is the same for all three policies.
The following statements follow from standard conditional independence rules (Russell & Norvig, 2002):
[Equation]
The last equality follows from the fact s 0 has no parents.
Thus, when combined with the fact that the transition dynamics P ( s 1 | s 0 , a 0 ) are the same for all three policies, we arrive at the following equality:
[Equation]
The ﬁrst difference between the trajectory distributions starts at a 1 .
By deﬁnition, the expert chooses actions via
[Equation]
while the learners instead follow
[Equation]
The ﬁrst line follows from the fact u 1 ⊥⊥ u 0 , s 1 .
The last equality follows from Rule 2 of do-calculus (Pearl et al., 2016).
Thus, given the action distributions at the second step are not equal, so long as a 1 (cid:54)⊥⊥ u 0 and s 1 (cid:54)⊥⊥ u 0 , all three policies have different trajectory distributions.
A. P ROOF OF L EMMA
Proof.
We prove J ( π E ) =
J ( π BC ) via an application of the Performance Difference Lemma (PDL) (Kakade & Langford, 2002).
We note that this is equivalent to the graphical argument used by Kumor et al.
(Defn.
2.3 in their paper).
First, we note that p ( s 0 , a 0 ) is equal for both policies.
Assuming that p ( s t − 1 , a t − 1 ) is equal, p ( s t ) must be equal by the fact the transition dynamics are the same.
Then, via the fact that π BC matches the conditional distribution of a t | s t , p ( s t , a t ) must also be equal.
Thus, by induction, d π BC =
d π E .
This means that π BC is deﬁned properly everywhere within it’s own state visitation distribution.
We then apply the PDL as follows:
[Equation]
The third equality follows from the fact the learner’s actions are independent of the confounder, so under d π BC , P ( u | s ) =
P ( u ) .
From the proof of the previous lemma, p ( a 1 | s 1 ) (cid:54) = p ( a 1 | do ( s 1 )) in general, so π
IV and π E can have different state-action visitation distributions after the ﬁrst timestep.
Therefore, for a two-step problem where a 1 (cid:54)⊥⊥ u 0 and s 1 (cid:54)⊥⊥ u 0 ,
[Equation]
44) for reward function
[Equation]
A. P ROOF OF L EMMA
Proof.
Let u t be a Rademacher random variable (1 w.p. 12 and − 1 otherwise) ∀ t ∈
[ T ] .
Let a t | s t , u t , u t − 1 =
u t and there be a single ﬁxed state s that no action can leave.
Set r
( s, a t , u t )
{+= 
+}
[ a t = u t ] .
Then, J ( π E ) = T .
Notice that p (+1 | s ) =
p ( − 1 | s ) =
p (+1 | do ( s ))
= p ( − 1 | do ( s ))
= 0 .
Thus, J ( π E ) > J ( π IV ) =
J ( π BC ) =
T 2 , the expected number of correct answers of randomly guessing the outcome of a fair coin T times.
Furthermore, notice that T 2 is the best any π ∈ Π can do.
One can see this by considering the problem with T = 1 .
If there was a policy π with J ( π ) > 12 , one would be able to use such a policy to predict the outcome of a fair coin better than the Bayesoptimal classiﬁer for coin-betting ( p ( heads )
= p ( tails ) = 12 ), violating the fact the Bayes-optimal classiﬁer minimizes Bayes error.
A. P ROOF OF L EMMA
Proof.
Notice that
[Equation]
[Equation]
can be re-written as
[Equation]
Thus, the proof of Theorem 4 holds as written.
A. P ROOF OF L EMMA
An (cid:15) -approximate equilibrium for the policy player is a π such that
[Equation]
With a change of notation
[Equation]
we can re-write this as:
[Equation]
Thus [Equation] the proof of Theorem 2 holds as written.
A. P ROOF OF T HEOREM
Proof.
By deﬁnition,
[Equation]
[Equation] Recall that the measure of ill-posedness of the problem (Dikkala et al., 2020; Chen & Pouzo, 2012) can be deﬁned as
[Equation]
Directly,
[Equation]
We repeat the deﬁnition of total variation stability of a distribution P ( U ) :
[Equation]
We proceed by noting that TV-stability implies that ∀ s ∈ S ,
[Equation]
[Equation]
By Jensen’s inequality
[Equation]
Taking the square root of both sides
[Equation]
we arrive at
[Equation]
Lastly [Equation] we apply the Performance Difference Lemma of Kakade & Langford (2002) as follows:
[Equation]
We use the fact that the same u would be added to both the learner and the expert’s actions and that rewards are in the range [ − 1 , 1] in the third step.
A. P ROOF OF L EMMA
Proof.
[Equation]
[Equation]
B E XPERIMENT D ETAILS B. L UNAR L ANDER E XPERIMENTS
For ease of simulation, we remove the legs from the LunarLander vehicle (the joints connecting them to the main body have a state that is not recorded in the observed state), remove the dispersion noise, and generate trajectories with a ﬁxed ground layout.
For all learned functions, we use two-layer ReLu MLPs with 64 hidden units.
We use the Adam optimizer (Kingma & Ba, 2014) for behavioral cloning and DoubIL and use the optimistic variant for ResiduIL .
We apply a weight decay of 1e-3 to all.
We train all methods for 50k steps.
[Table]
For computational ease, we only learn the mean of P ( a | s ) for DoubIL and add fresh standard normal noise on-top of it to simulate drawing actions.
For more complex noise models, one would need to use a moment matching algorithm (Swamy et al., 2021) in the ﬁrst stage.
Importantly, DoubIL suffers from a “double-sample” issue (Baird, 1995) where multiple independent samples of g ( z ) are required to compute gradients of (cid:98) h .
To see this, note that the gradient with respect to h of (10) is
[Equation] [Table]
Notice that ˆ x appears under two separate expectations that are then multiplied together.
To get an unbiased estimate of this product, two samples of ˆ x are required, one for each expectation.
Therefore, it is most correct to use multiple samples from g ( z ) for each update.
Thus, for implementing the “double samples” for the gradient, we compute E
[ a (cid:48) − π ( s (cid:48) ) | s ] and E 2 [ a (cid:48) − π ( s (cid:48) ) | s ] using independent samples.
Then, we apply a stop-gradient operator to the former expectation before taking a product between the expectations and averaging over s : [Equation]
[Equation]
This loss function has the correct gradient as it uses independent samples for computing the two expectations.
[Table]
B. LQG E XPERIMENTS
We compute the optimal policy for the following canonical linear system via solving a Discrete-Time
Algebraic Ricatti Equation via the standard iterative method:
[Equation]
[Equation]
This is the dynamics of a “sliding brick on a frozen lake.”
We then simulate rollouts of 200 timesteps with u t being drawn i.i.d.
from the standard normal distribution.
We confound actions with the sum of confounders going H steps back:
[Equation]
We simulate 1000 such rollouts to compute (21) empirically.
We calculate E [ X | z ] =
E [ s t | s t − H ]
=
( A + BK ∗ )
H s t − H analytically instead of via samples due to the small value of the quantity in comparison to the variance of the noise.