Learning Deep Features in Instrumental Variable Regression
Anonymous authors
Paper under double-blind review
Abstract
Instrumental variable (IV) regression is a standard strategy for learning causal relationships between confounded treatment and outcome variables from observational data by utilizing an instrumental variable, which affects the outcome only through the treatment.
In classical IV regression, learning proceeds in two stages: stage 1 performs linear regression from the instrument to the treatment; and stage 2 performs linear regression from the treatment to the outcome, conditioned on the instrument.
We propose a novel method, deep feature instrumental variable regression (DFIV) , to address the case where relations between instruments, treatments, and outcomes may be nonlinear.
In this case, deep neural nets are trained to deﬁne informative nonlinear features on the instruments and treatments.
We propose an alternating training regime for these features to ensure good end-to-end performance when composing stages 1 and 2, thus obtaining highly ﬂexible feature maps in a computationally e ﬃ cient manner.
DFIV outperforms recent state-of-the-art methods on challenging IV benchmarks, including settings involving high dimensional image data.
DFIV also exhibits competitive performance in o ﬀ -policy policy evaluation for reinforcement learning, which can be understood as an IV regression task.
Introduction
The aim of supervised learning is to obtain a model based on samples observed from some data generating process, and to then make predictions about new samples generated from the same distribution.
If our goal is to predict the e ﬀ ect of our actions on the world, however, our aim becomes to assess the inﬂuence of interventions on this data generating process.
To answer such causal questions, a supervised learning approach is inappropriate, since our interventions, called treatments , may a ﬀ ect the underlying distribution of the variable of interest, which is called the outcome .
To answer these counterfactual questions, we need to learn how treatment variables causally a ﬀ ect the distribution process of outcomes, which is expressed in a structural function .
Learning a structural function from observational data (that is, data where we can observe, but not intervene) is known to be challenging if there exists an unmeasured confounder, which inﬂuences both treatment and outcome.
To illustrate: suppose we are interested in predicting sales of airplane tickets given price.
During the holiday season, we would observe the simultaneous increase in sales and prices.
This does not mean that raising the prices causes the sales to increase.
In this context, the time of the year is a confounder, since it a ﬀ ects both the sales and the prices, and we need to correct the bias caused by it.
One way of correcting such bias is via instrumental variable (IV) regression (Stock and
Trebbi, 2003).
Here, the structural function is learned using instrumental variables, which only a ﬀ ect the treatment directly but not the outcome.
In the sales prediction scenario, we can use supply cost shifters as the instrumental variable since they only a ﬀ ect the price (Wright, 1928; Blundell et al., 2012).
Instrumental variables can be found in many contexts, and IV regression is extensively used by economists and epidemiologists.
For example, IV regression is used for measuring the e ﬀ ect of a drug in the scenario of imperfect compliance (Angrist et al., 1996), or the inﬂuence of military service on lifetime earnings (Angrist, 1990).
In this work, we propose a novel IV regression method, which can discover non-linear causal relationships using deep neural networks.
Classically, IV regression is solved by the two-stage least squares (2SLS) algorithm; we learn a mapping from the instrument to the treatment in the ﬁrst stage and learn the structural function in the second stage as the mapping from the conditional expectation of the treatment given the instrument (obtained from stage 1) to the outcome.
Originally, 2SLS assumes linear relationships in both stages, but this has been recently extended to non-linear settings.
One approach has been to use non-linear feature maps.
Sieve IV (Newey and Powell, 2003;
Chen and Christensen, 2018) uses a ﬁnite number of basis functions explicitly speciﬁed.
Kernel IV (KIV) (Singh et al., 2019) and Dual IV regression (Muandet et al., 2019) extend sieve IV to allow for an inﬁnite number of basis functions using reproducing kernel Hibert spaces (RKHS).
Although these methods enjoy desirable theoretical properties, the ﬂexibility of the model is limited, since all existing work uses the prespeciﬁed features.
Another approach is to perform the stage 1 regression through conditional density estimation (Carrasco et al., 2007; Darolles et al., 2011; Hartford et al., 2017).
One advantage of this approach is that it allows for ﬂexible models, including deep neural nets, as proposed in the DeepIV algorithm of (Hartford et al., 2017).
It is known, however, that conditional density estimation is costly and often su ﬀ ers from high variance when the treatment is high-dimensional.
More recently, Bennett et al.
(2019) have proposed DeepGMM, a method inspired by the optimally weighted Generalized Method of Moments (GMM) (Hansen, 1982) to ﬁnd a structural function ensuring that the regression residual and the instrument are independent.
Although this approach can handle high-dimensional treatment variables and deep NNs as feature extractors, the learning procedure might not be as stable as 2SLS approach, since it involves solving a smooth zero-sum game as when training Generative Adversarial Networks (Wiatrak et al., 2019).
In this paper, we propose Deep Feature Instrumental Variable Regression (DFIV), which aims to combine the advantages of all previous approaches, while avoiding their limitations.
In DFIV, we use deep neural nets to adaptively learn feature maps in the 2SLS approach, which allows us to ﬁt highly nonlinear structural functions, as in DeepGMM and DeepIV.
Unlike DeepIV, DFIV does not rely on conditional density estimation.
Like sieve IV and
KIV, DFIV learns the conditional expectation of the feature maps in stage 1 and uses the predicted features in stage 2, but with the additional advantage of learned features.
We empirically show that DFIV performs better than other methods on several IV benchmarks, and apply DFIV successfully to o ﬀ -policy policy evaluation, which is a fundamental problem in Reinforcement Learning (RL).
The paper is structured as follows.
In Section 2, we formulate the IV regression problem and introduce two-stage least-squares regression.
In Section 3, we give a detailed description of our DFIV method.
We demonstrate the empirical performance of DFIV in Section 4, covering three settings: a classical demand prediction example from econometrics, a challenging IV setting where the treatment consists of high-dimensional image data, and the problem of o ﬀ -policy policy evaluation in reinforcement learning.
Preliminaries 2. Problem Setting of Instrumental Variable Regression
[Figure]
We begin with a description of the IV setting.
We observe a treatment X 2 X , where X ⇢ R d X , and an outcome Y
Y , where Y ⇢ R .
We also have an unobserved confounder that a ﬀ ects both X and Y .
This causal relationship can be represented with the following structural causal model:
[Equation]
where f struct is called the structural function, which we assume to be continuous, and " is an additive noise term.
This speciﬁc confounding assumption is necessary for the IV problem.
In Bareinboim and Pearl (2012), it is shown that we cannot learn f struct if we allow any type of confounders.
The challenge is that E [ " | X ] 6 = 0 , which reﬂects the existence of a confounder.
Hence, we cannot use ordinary supervised learning techniques since f struct ( x ) 6 = E [ Y | X = x ] .
Here, we assume there is no observable confounder but we may easily include this, as discussed in Appendix B.
To deal with the hidden confounder " , we assume to have access to an instrumental variable
Z 2 Z which satisﬁes the following assumption.
Assumption 1.
The conditional distribution P ( X | Z ) is not constant in Z and E [ " | Z ] = 0 .
Intuitively, Assumption 1 means that the instrument Z induces variation in the treatment
X
but is uncorrelated with the hidden confounder " .
Again, for simplicity, we assume
Z ⇢ R d Z .
The causal graph describing these relationships is shown in Figure 1 1 .
Note that the instrument Z cannot have an incoming edge from the latent confounder that is also a parent of the outcome.
Given Assumption 1, we can see that the function f struct satisﬁes the operator equation
E [ Y | Z ] =
E [ f struct ( X ) | Z ] by taking expectation conditional on Z of both sides of (1).
Newey and Powell (2003) provide the necessary and su ﬃ cient conditions, known as completeness assumption, to ensure identiﬁability of f struct ( X ) .
Solving this equation, however, is known to be ill-posed (Nashed and Wahba, 1974).
To address this, recent works (Carrasco et al., 2007; Darolles et al., 2011; Muandet et al., 2019; Singh et al., 2019) minimize the following regularized loss L to obtain the estimate ˆ f struct :
[Equation]
where F is an arbitrary space of continuous functions and ⌦ ( f ) is a regularizer on f .
2. Two Stage Least Squares Regression
A number of works (Newey and Powell, 2003; Singh et al., 2019) tackle the minimization problem (2) using two-stage least squares (2SLS) regression, in which the structural function is modeled as f struct ( x ) =
u > ( x ) , where u is a learnable weight vector and ( x ) is a vector of ﬁxed basis functions.
For example, linear 2SLS used the identity map ( x ) =
x , while sieve IV (Newey and Powell, 2003) uses Hermite polynomials.
In the 2SLS approach, an estimate ˆ u is obtained by solving two regression problems successively.
In stage 1, we estimate the conditional expectation E X | z
[ ( X )] as a function of z .
Then in stage 2, as E X | z [ f ( X )]
= u > E X | z
[ ( X )] , we minimize L with E X | z
[ f ( X )] being replaced by the estimate obtained in stage 1.
Speciﬁcally, we model the conditional expectation as E X | z
[ ( X )]
= V  ( z ) , where  ( z ) is another vector of basis functions and V is a matrix to be learned.
Again, there exist many choices for  ( z ) , which can be inﬁnite-dimensional, but we assume the dimensions of ( x ) and  ( z ) to be d 1 , d 2 < 1 respectively.
In stage 1, the matrix V is learned by minimizing the following loss,
[Equation]
where  1 > 0 is a regularization parameter.
This is a linear ridge regression problem with multiple targets, which can be solved analytically.
In stage 2, given ˆ V , we can obtain u by minimizing the loss
[Equation]
where  2 > 0 is another regularization parameter.
Stage 2 corresponds to a ridge linear regression from ˆ V  ( Z ) to Y , and also enjoys a closed-form solution.
Given the learned weights ˆ u , the estimated structural function is ˆ f struct ( x ) =
ˆ u > ( x ) .
DFIV Algorithm
In this section, we develop the DFIV algorithm.
Similarly to Singh et al.
(2019), we assume that we do not necessarily have access to observations from the joint distribution of ( X, Y, Z ) .
Instead, we are given m observations of ( X, Z ) for stage 1 and n observations of ( Y, Z ) for stage 2.
We denote the stage 1 observations as ( x i , z i ) and the stage 2 observations as (˜ y
i , ˜ z i ) .
If observations of ( X, Y, Z ) are given for both stages, we can evaluate the out-of-sample losses, and these losses can be used for hyper-parameter tuning of  1 ,  2 (Appendix A).
DFIV uses the following models
[Equation]
and
[Equation]
where u 2 R d 1 and V 2 R d 1 ⇥ d 2 are the parameters, and ✓ X ( x ) 2 R d 1 and  ✓ Z ( z ) 2 R d 2 are the neural nets parameterised by ✓ X and ✓ Z , respectively.
As in the original 2SLS algorithm, we learn E X | z
[ ✓ X ( X )] in stage 1 and f struct ( x ) in stage 2.
In addition to the weights u and V , however, we also learn the parameters of the feature maps, ✓ X and ✓ Z .
Hence, we need to alternate between stages 1 and 2, since the conditional expectation E X | z
[ ✓ X ( X )] changes during training.
Stage 1 Regression
The goal of stage 1 is to estimate the conditional expectation
E X | z
[ ✓ X ( X )] ' V ✓ Z ( z ) by learning the matrix V and parameter ✓ Z , with ✓ X = ˆ ✓ X given and ﬁxed.
Given the stage 1 data ( x i , z i ) , this can be done by minimizing the empirical estimate of L 1 ,
[Equation]
Note that the feature map ˆ ✓ X
( X ) is ﬁxed during stage 1, since this is the “target variable.”
If we ﬁx ✓ Z , the minimization problem (6) reduces to a linear ridge regression problem with multiple targets, whose solution as a function of ✓ X and ✓ Z is given analytically by
[Equation]
where  1 , 1 are feature matrices deﬁned as
[Equation]
and  1 =
[  ✓ Z ( z 1 ) , . . .
,  ✓ Z ( z m )] > 2 R m ⇥
d 2 .
We can then learn the parameters ✓ Z of the adaptive features ✓ Z by minimizing the loss L ( m )at V
=
ˆ V ( m ) (ˆ ✓ X , ✓ Z ) using gradient descent.
For simplicity, we introduce a small abuse of notation by denoting as ˆ ✓ Z the result of a user-chosen number of gradient descent steps on the loss (6) with ˆ V ( m ) (ˆ ✓ X , ✓ Z ) from (7), even though ˆ ✓ Z need not attain the minimum of the non-convex loss (6).
We then write ˆ
V ( m ) :
= ˆ V ( m ) (ˆ ✓ X , ˆ ✓ Z ) .
While this trick of using an analytical estimate of the linear output weights of a deep neural network might not lead to signiﬁcant gains in standard supervised learning, it turns out to be very important in the development of our 2SLS algorithm.
As shown in the following section, the analytical estimate ˆ V ( m ) ( ✓ X , ˆ ✓ Z ) (now considered as a function of ✓ X ) will be used to backpropagate to ✓ X in stage 2.
Stage 2 Regression In stage 2, we learn the structural function by computing the weight vector u and parameter ✓ X while ﬁxing ✓ Z = ˆ ✓ Z , and thus the corresponding feature map  ˆ ✓ Z ( z ) .
Given the data (˜ y i , ˜ z i ) , we can minimize the empirical version of L 2 , deﬁned as ˆ u ( n ) , ˆ ✓ X = arg min
[Equation]
Again, for a given ✓ X , we can solve the minimization problem (8) for u as a function of ˆ
V ( m ) :
= ˆ V ( m ) ( ✓ X , ˆ ✓ Z ) by a linear ridge regression ˆ
[Equation]
where  2 = [  ˆ ✓ Z
(˜ z 1 ) , . .
,  ˆ ✓ Z
(˜ z n )] > 2 R n ⇥ d 2 and y 2 =
[˜ y 1 , . . . , ˜ y n ] > 2 R n .
The loss L ( n )explicitly depends on the parameters ✓ X
and we can backpropagate it to ✓ X via ˆ V ( m ) ( ✓ X , ˆ ✓ Z ) , even though the samples of the treatment variable
X do not appear in stage 2 regression.
We again introduce a small abuse of notation for simplicity, and denote by ˆ ✓ X the estimate obtained after a few gradient steps on (8) with ˆ u ( n ) ( ✓ X , ˆ ✓ Z ) from (9), even though ˆ ✓ X need not minimize the non-convex loss (8).
We then have ˆ u ( n ) = ˆ u ( n ) (ˆ ✓ X , ˆ ✓ Z ) .
After updating ˆ ✓ X , we need to update ˆ ✓ Z accordingly.
We do not attempt to backpropagate through the estimate ˆ ✓ Z to do this, however, as this would be too computationally expensive; instead, we alternate stages 1 and 2.
We also considered updating ˆ ✓ X and ˆ ✓ Z jointly to optimize the loss L ( n ) 2 , but this fails, as discussed in Appendix E.
Computational Complexity
The computational complexity of the algorithm is
O ( md 1 d 2 + d 32 ) for stage 1, while stage 2 requires additional O ( nd 1 d 2 + d 31 ) computations.
This is small compared to KIV (Singh et al., 2019), which takes O ( m 3 ) and O ( n 3 ) , respectively.
We can further speed up the learning by using mini-batch training as shown in Algorithm 1.
Input: Stage 1 data ( x i , z i ) , Stage 2 data (˜ y i ˜
z i ) , Regularization parameters (  1 ,  2 ) .
Initial values ˆ ✓ X , ˆ ✓ Z .
Mini-batch size ( m b , n b ) .
Number of updates in each stage ( T 1 , T 2 ) .
Output: Estimated structural function ˆ f struct ( x )
Here, ˆ V ( m b ) and ˆ u ( n b ) are the functions given by (7) and (9) calculated using mini-batches of data.
Similarly, L ( m b )and L ( n b )are the stage 1 and 2 losses for the mini-batches.
We recommend setting the batch size large enough so that ˆ V ( m b ) , ˆ u ( n b ) do not diverge from ˆ V ( m ) , ˆ u ( n ) computed on the entire dataset.
Furthermore, we observe that setting T 1 > T 2 , i.e. updating ✓ Z more frequently than ✓ X , stabilizes the learning process.
Experiments
In this section, we report the empirical performance of the DFIV method.
The evaluation considers both low and high-dimensional treatment variables.
We used the demand design dataset of Hartford et al.
(2017) for benchmarking in the low and high-dimensional cases,
and we propose a new setting for the high-dimensional case based on the dSprites dataset (Matthey et al., 2017).
In the deep RL context, we also apply DFIV to perform o ﬀ -policy policy evaluation (OPE).
The network architecture and hyper-parameters are provided in
Appendix F.
The algorithms in the ﬁrst two experiments are implemented using PyTorch (Paszke et al., 2019) and the OPE experiments are implemented using TensorFlow (Abadi et al., 2015) and the Acme RL framework (Ho ﬀ man et al., 2020).
The code is included in the supplemental material.
4. Demand Design Experiments
The demand design dataset is a synthetic dataset introduced by Hartford et al.
(2017) that is now a standard benchmarking dataset for testing nonlinear IV methods.
In this dataset, we aim to predict the demands on airplane tickets Y given the price of the tickets P .
The dataset contains two observable confounders, which are the time of year T 2 [0 , 10] and customer groups S 2 { 1 , ..., 7 } that are categorized by the levels of price sensitivity.
Further, the noise in Y and P is correlated, which indicates the existence of an unobserved confounder.
The strength of the correlation is represented by ⇢ 2 [0 , 1] .
To correct the bias caused by this hidden confounder, the fuel price C is introduced as an instrumental variable.
Details of the data generation process can be found in Appendix D.1.
In DFIV notation, the treatment is X = P , the instrument is Z = C , and ( T, S ) are the observable confounders.
We compare the DFIV method to three leading modern competitors, namely KIV (Singh et al., 2019), DeepIV (Hartford et al., 2017), and DeepGMM (Bennett et al., 2019).
We used the DFIV method with observable confounders, as introduced in Appendix B. Note that
DeepGMM does not have an explicit mechanism for incorporating observable confounders.
The solution we use, proposed by Bennett et al.
(2019, p. 2), is to incorporate these observables in both instrument and treatment; hence we apply DeepGMM with treatment X
=
( P, T, S ) and instrumental variable Z
=
( C, T, S ) .
Although this approach is theoretically sound, this makes the problem unnecessary di ﬃ cult since it ignores the fact that we only need to consider the conditional expectation of P given Z .
We used a network with a similar number of parameters to DeepIV as the feature maps in
DFIV and models in DeepGMM.
We tuned the regularizers  1 ,  2 as discussed in Appendix A, with the data evenly split for stage 1 and stage 2.
We varied the correlation parameter ⇢ and dataset size, and ran 20 simulations for each setting.
Results are summarized in Figure 2.
We also evaluated the performance via the estimation of average treatment e ﬀ ect and conditional average treatment e ﬀ ect, which is presented in Appendix D.
Next, we consider a case, introduced by Hartford et al.
(2017), where the customer type
S 2 { 1 , . . .
, 7 } is replaced with an image of the corresponding handwritten digit from the
MNIST dataset (LeCun and Cortes, 2010).
This reﬂects the fact that we cannot know the exact customer type, and thus we need to estimate it from noisy high-dimensional data.
Note that although the confounder is high-dimensional, the treatment variable is still realvalued, i.e. the price P of the tickets.
Figure 3 presents the results for this high-dimensional confounding case.
Again, we train the networks with a similar number of learnable parameters to DeepIV in DFIV and DeepGMM, and hyper-parameters are set in the way discussed in
Appendix A. We ran 20 simulations with data size n + m = 5000 and report the mean and standard error.
Our ﬁrst observation from Figure 2 and 3 is that the level ⇢ of correlation has no signiﬁcant impact on the error under any of the IV methods, indicating that all approaches correctly account for the e ﬀ ect of the hidden confounder.
This is consistent with earlier results on this dataset using DeepIV and KIV (Hartford et al., 2017; Singh et al., 2019).
We note that DeepGMM does not perform well in this demand design problem.
This may be due to the current DeepGMM approach to handling observable confounders, which might not be optimal.
KIV performed reasonably well for small sample sizes and low-dimensional data, but it did less well in the high-dimensional MNIST case due to its less expressive features.
In high dimensions, DeepIV performed well, since the treatment variable is unidimensional.
However, DFIV performed consistently better than all other methods in both low and high dimensions, which suggests it can learn a ﬂexible structural function in a stable manner.
4. dSprites Experiments
To test the performance of DFIV methods for a high dimensional treatment variable, we utilized the dSprites dataset (Matthey et al., 2017).
This is an image dataset described by ﬁve latent parameters ( shape, scale, rotation, posX and posY ).
The images are 64 ⇥ 64 = 4096 -dimensional.
In this experiment, we ﬁxed the shape parameter to heart , i.e. we only used heartshaped images.
An example is shown in Figure 5.
From this dataset, we generated data for IV regression in which we use each ﬁgure as treatment variable X .
Hence, the treatment variable is 4096-dimensional in this experiment.
To make the task more challenging, we used posY as the hidden confounder, which is not revealed to the model.
We used the other three latent variables as the instrument variables Z .
The structural function f struct and outcome Y are deﬁned as
[Equation]
where each element of the matrix
A 2 R 10 ⇥ 4096 is generated from Unif(0 .
0 , 1 . 0) and ﬁxed throughout the experiment.
See Appendix D.3 for the detailed data generation process.
We tested the performance of DFIV with KIV and DeepGMM, where the hyper-parameters are determined as in the demand design problem.
The results are displayed in Figure 4.
DFIV consistently yields the best performance of all the methods.
DeepIV is not included in the ﬁgure because it fails to give meaningful predictions due to the di ﬃ culty of performing conditional density estimation for the high-dimensional treatment variable.
The performance of KIV su ﬀ ers since it lacks the feature richness to express a high-dimensional complex structural function.
Although DeepGMM performs comparatively to DFIV, we observe some instability during the training, see Appendix D.4.
4. Off-Policy Policy Evaluation Experiments
We apply our IV methods to the o ﬀ -policy policy evaluation (OPE) problem (Sutton and
Barto, 2018), which is one of the fundamental problems of deep RL.
In particular, it has been realized by Bradtke and Barto (1996) that 2SLS could be used to estimate a linearly parameterized value function, and we use this reasoning as the basis of our approach.
Let us consider the RL environment h S , A , P, R, ⇢ 0 ,  i , where S is the state space, A is the action space, P :
S ⇥ A ⇥ S !
[0 , 1] is the transition function, R : S ⇥
A ⇥ S ⇥
R !
R is the reward distribution, ⇢ 0 :
S !
[0 , 1] is the initial state distribution, and discount factor  2 (0 , 1] .
Let ⇡ be a policy, and we denote ⇡ ( a | s ) as the probability of selecting action a in stage s 2 S .
Given policy ⇡ , the Q -function is deﬁned as
[Equation]
with a t ⇠ ⇡ ( · | s t ) , s t +1 ⇠ P ( ·| s t , a t ) , r t ⇠ R ( ·| s t , a t , s t +1 ) .
The goal of OPE is to evaluate the expectation of the Q -function with respect to the initial state distribution for a given target policy ⇡ , E s ⇠ ⇢ 0 ,a | s ⇠ ⇡ [ Q ⇡ ( s, a )] , learned from a ﬁxed dataset of transitions ( s, a, r, s 0 ) , where s and a are sampled from some potentially unknown distribution µ and behavioral policy ⇡ b ( ·| s ) respectively.
Using the Bellman equation satisﬁed by Q ⇡ , we obtain a structural causal model of the form (1),
[Equation]
where X = ( s, a, s 0 , a 0 ) , Z = ( s, a ) , Y = r .
We have that E [ " ] = 0 , E [ " | X ] 6 = 0 , and
Assumption 1 is veriﬁed.
Minimizing the loss (2) for the structural causal model (10) corresponds to minimizing the following loss L OPE
[Equation]
and we can apply any IV regression method to achieve this.
In Appendix C, we show that minimizing L OPE corresponds to minimizing the mean squared Bellman error (MSBE) (Sutton and Barto, 2018, p. 268) and we detail the DFIV algorithm for OPE.
Note that
MBSE is also the loss minimized by the residual gradient (RG) method proposed in (Baird, 1995) to estimate Q -functions.
However, this method su ﬀ ers from the “double-sample” issue, i.e. it requires two independent samples of s 0 starting from the same ( s, a ) due to the inner conditional expectation (Baird, 1995), whereas IV regression methods do not su ﬀ er from this issue.
We evaluate DFIV on three BSuite (Osband et al., 2019) tasks: catch, mountain car, and cartpole.
See Section D.6.1 for a description of those tasks.
The original system dynamics are deterministic.
To create a stochastic environment, we randomly replace the agent action by a uniformly sampled action with probability p 2 [0 , 0 . 5] .
The noise level p controls the level of confounding e ﬀ ect.
The target policy is trained using DQN (Mnih et al., 2015), and we subsequently generate an o ﬄ ine dataset for OPE by executing the policy in the same environment with a random action probability of 0.2 (on top of the environment’s random action probability p ).
We compare DFIV with KIV, DeepIV, and DeepGMM; as well as Fitted Q Evaluation (FQE) (Le et al., 2019; Voloshin et al., 2019), a specialized approach designed for the OPE setting, which serves as our “gold standard” baseline (Paine et al., 2020) (see Section D.6.2 for details).
All methods use the same network for value estimation.
Figure 6 shows the absolute error of the estimated policy value by each method with a standard deviation from 5 runs.
In catch and mountain car, DFIV comes closest in performance to FQE, and even matches it for some noise settings, whereas DeepGMM is somewhat worse in catch, and signiﬁcantly worse in mountain car.
In the case of cartpole,
DeepGMM performs somewhat better than DFIV, although both are slightly worse than
FQE.
DeepIV and KIV both do poorly across all RL benchmarks.
Conclusion
We have proposed a novel method for instrumental variable regression, Deep Feature IV (DFIV), which performs two-stage least squares regression on ﬂexible and expressive features of the instrument and treatment.
As a contribution to the IV literature, we showed how to adaptively learn these feature maps with deep neural networks.
We also showed that the o ﬀ -policy policy evaluation (OPE) problem in deep RL can be interpreted as a nonlinear IV regression, and that DFIV performs competitively in this domain.
This work thus brings the research worlds of deep o ﬄ ine RL and causality from observational data closer.
In terms of future work, it would be interesting to adapt the ideas from (Angrist and
Krueger, 1995; Angrist et al., 1999; Hansen and Kozbur, 2014) to select the regularization hyperparameters of DFIV as well as investigate generalizations of DFIV beyond the additive model (1) as considered in (Carrasco et al., 2007, Section 5.4).
In RL, problems with additional confounders are common, see e.g. (Namkoong et al., 2020; Shang et al., 2019), and we believe that adapting DFIV to this setting will be of great value.