Off-Policy Evaluation for Action-Dependent Non-Stationary Environments
Abstract
Methods for sequential decision-making are often built upon a foundational assumption that the underlying decision process is stationary.
This limits the application of such methods because real-world problems are often subject to changes due to external factors ( passive non-stationarity), changes induced by interactions with the system itself ( active non-stationarity), or both ( hybrid non-stationarity).
In this work, we take the ﬁrst steps towards the fundamental challenge of on-policy and off-policy evaluation amidst structured changes due to active, passive, or hybrid non-stationarity.
Towards this goal, we make a higher-order stationarity assumption such that non-stationarity results in changes over time, but the way changes happen is ﬁxed.
We propose, OPEN, an algorithm that uses a double application of counterfactual reasoning and a novel importance-weighted instrument-variable regression to obtain both a lower bias and a lower variance estimate of the structure in the changes of a policy’s past performances.
Finally, we show promising results on how OPEN can be used to predict future performances for several domains inspired by real-world applications that exhibit non-stationarity.
Introduction
Methods for sequential decision making are often built upon a foundational assumption that the underlying decision process is stationary [Sutton and Barto, 2018].
While this assumption was a cornerstone when laying the theoretical foundations of the ﬁeld and while is often reasonable, it is seldom true in practice and can be unreasonable [Dulac-Arnold et al., 2019].
Instead, real-world problems are subject to non-stationarity that can be broadly classiﬁed as (a) Passive: where the changes to the system are induced only by external (exogenous) factors, (b) Active: where the changes result due to the agent’s past interactions with the system and (c) Hybrid: where both passive and active changes can occur together [Khetarpal et al., 2020].
There are many applications that are subject to active, passive, or hybrid non-stationarity, and where the stationarity assumption may be unreasonable.
Consider methods for automated healthcare where we would like to use the data collected over past decades to ﬁnd better treatment policies.
In such cases, not only might there have been passive changes due to healthcare infrastructure changing over time, but active changes might also occur because of public health continuously evolving based on the treatments made available in the past, thereby resulting in hybrid non-stationarity.
Similar to automated healthcare, other applications like online education, product recommendations, and in fact almost all human-computer interaction systems need to not only account for the continually drifting behavior of the user demographic, but also how the preferences of users may change due to interactions with the system [Theocharous et al., 2020].
Even social media platforms need to account for the partisan bias of their users that change due to both external political developments and increased self-validation resulting from previous posts/ads suggested by the recommender system itself
[Cinelli et al., 2021, Gillani et al., 2018].
Similarly, motors in a robot suffer wear and tear over time not only based on natural corrosion but also on how vigorous past actions were.
However, conventional off-policy evaluation methods [Precup, 2000, Jiang and Li, 2015, Xie et al., 2019] predominantly focus on the stationary setting.
These methods assume availability of either (a) resetting assumption to sample multiple sequences of interactions from a stationary environment with a ﬁxed starting state distribution (i.e., episodic setting), or (b) ergodicity assumption such that interactions can be sampled from a steady-state/stationary distribution (i.e., continuing setting).
For the problems of our interest, methods based on these assumptions may not be viable.
For e.g., in automated healthcare, we have a single long history for the evolution of public health, which is neither in a steady state distribution nor can we reset and go back in time to sample another history of interactions.
As discussed earlier, because of non-stationarity the transition dynamics and reward function in the future can be different from the ones in the past, and these changes might also be dependent on past interactions.
In such cases, how do we even address the fundamental challenge of off-policy evaluation , i.e., using data from past interactions to estimate the performance of a new policy in the future?
Unfortunately, if the underlying changes are arbitrary, even amidst only passive nonstationarity it may not be possible to provide non-vacuous predictions of a policy’s future performance [Chandak et al., 2020a].
Thankfully, for many real-world applications there might be (unknown) structure in the underlying changes.
In such cases, can the effect of the underlying changes on a policy’s performance be inferred, without requiring estimation of the underlying model/process?
Prior work has only shown that this is possible in the passive setting.
This raises the question that we aim to answer:
How can one provide a uniﬁed procedure for (off) policy evaluation amidst active, passive, or hybrid non-stationarity, when the underlying changes are structured?
Contributions: To the best of our knowledge, our work presents the ﬁrst steps towards addressing the fundamental challenge of off-policy evaluation amidst structured changes due to active or hybrid non-stationarity.
Towards this goal, we make a higher-order stationarity assumption, under which the non-stationarity can result in changes over time, but the way changes happen is ﬁxed.
Under this assumption, we propose a model-free method that can infer the effect of the underlying nonstationarity on the past performances and use that to predict the future performances for a given policy.
We call the proposed method OPEN: off-policy evaluation for non-stationary domains.
On domains inspired by real-world applications, we show that OPEN often provides signiﬁcantly better results not only in the presence of active and hybrid non-stationarity, but also for the passive setting where it even outperforms previous methods designed to handle only passive non-stationarity.
OPEN primarily relies upon two key insights: (a) For active/hybrid non-stationarity, as the underlying changes may dependend on past interactions, the structure in the changes observed when executing the data collection policy can be different than if one were to execute the evaluation policy.
To address this challenge, OPEN makes uses counterfactual reasoning twice and permits reduction of this off-policy evaluation problem to an auto-regression based forecasting problem.
(b) Despite reduction to a more familiar auto-regression problem, in this setting naive least-squares based estimates of parameters for auto-regression suffers from high variance and can even be asymptotically biased.
Finally, to address this challenge, OPEN uses a novel importance-weighted instrument-variable (auto-)regression technique to obtain asymptotically consistent and lower variance parameter estimates.
Related Work
Off-policy evaluation (OPE) is an important aspect of reinforcement learning [Precup, 2000, Thomas et al., 2015, Sutton and Barto, 2018] and various techniques have been developed to construct efﬁcient estimators for OPE [Jiang and Li, 2015, Thomas and Brunskill, 2016, Munos et al., 2016,
Harutyunyan et al., 2016, Espeholt et al., 2018, Xie et al., 2019].
However, these work focus on the stationary setting.
Similarly, there are various methods for tackling non-stationarity in the bandit setting [Moulines, 2008, Besbes et al., 2014, Seznec et al., 2018, Wang et al., 2019a].
In contrast, the proposed work focuses on methods for sequential decision making.
Literature on off-policy evaluation amidst non-stationarity for sequential decision making is sparse.
Perhaps the most closely related works are by Thomas et al.
[2017], Chandak et al.
[2020b], Xie et al. [2020a], Poiani et al.
[2021], Liotet et al.
While these methods present an important stepping stone, such methods are for passive non-stationarity and, as we discuss using the toy example in
Figure 1, may result in undesired outcomes if used as-is in real-world settings that are subject to active or hybrid non-stationarity.
Consider a robot that can perform a task each day either by ‘walking’ or ‘running’.
A reward of 8 is obtained upon completion using ‘walking’, but ‘running’ ﬁnishes the task quickly and results in a reward of 10 .
However, ‘running’ wears out the motors, thereby increasing the time to ﬁnish the task the next day and reduces the returns for both ‘walking’ and ‘running’ by a small factor, α ∈ (0 , 1) .
Here, methods for tackling passive non-stationarity will track the best policy under the assumption that the changes due to damages are because of external factors and would fail to attribute the cause of damage to the agent’s decisions.
Therefore, as on any given day ‘running’ will always be better, every day these methods will prefer ‘running’ over ‘walking’ and thus aggravate the damage.
Since the outcome on each day is dependent on decisions made during previous days this leads to active non-stationarity, where ‘walking’ is better in the long run.
Finding a better policy ﬁrst requires a method to evaluate a policy’s (future) performance, which is the focus of this work.
Notice that the above problem can also be viewed as a task with effectively a single lifelong episode.
However, as we discuss later in Section 4, approaches such as modeling the problem as a large stationary POMDP or as a continuing average-reward MDP with a single episode may not be viable.
Further, non-stationarity can also be observed in multi-agent systems and games due to different agents/players interacting with the system.
However, often the goal in these other areas is to search for (Nash) equilibria, which may not even exist under hybrid non-stationarity.
Non-stationarity may also result due to artifacts of the learning algorithm even when the problem is stationary.
While relevant, these other research areas are distinct from our setting of interest and we discuss them and others in more detail in Appendix B.
Non-Stationary Decision Processes
We build upon the formulation used by past work [Xie et al., 2020a, Chandak et al., 2020b] and consider that the agent interacts with a lifelong sequence of partially observable Markov decision processes (POMDPs), ( M i ) ∞ i =1 .
However, unlike prior problem formulations, we account for active and hybrid non-stationarity by considering POMDP M i +1 to be dependent on both on the POMDP
M i and the decisions made by the agent while interacting with M i .
We provide a control graph for this setup in Figure 2.
For simplicity of presentation, we will often ignore the dependency of M i + on M i − k for k > 0 , although our results can be extended for settings with k > 0 .
Notation:
Let M be a ﬁnite set of POMDPs.
Each POMDP M i ∈ M is a tuple ( O , S , A , Ω i , P i , R i , µ i ) , where O is the set of observations, S is the set of states, and A is the set of actions, which are the same for all the POMDPs in M .
For simplicity of notation, we assume
M , S , O , A are ﬁnite sets, although our results can be extended to settings where these sets are inﬁnite or continuous.
Let Ω i : S×O → [0 , 1] be the observation function, P i : S×A×S → [0 , 1] be the transition function, µ i : S → [0 , 1] be the starting state distribution, and R i : S ×
A → [ − R max , R max ] be the reward function with 0 ≤ R max < ∞ .
Let π : O × A → [0 , 1] be any policy and Π be the set of all policies.
Let H i :=
( O ti , A ti , R ti )
Tt =1 be a sequence of at most T interactions in M i , where O ti , A ti , R ti are the random variables corresponding to the observation, action, and reward at the step t .
Let G i :
=
(cid:80) Tt =1 R ti be an observed return and J i ( π ) :
= E π
[ G i | M i ] be the performance of π on M i .
Let H be the set of possible interaction sequences, and ﬁnally let T : M × H × M
→ [0 , 1] be the transition function that governs the non-stationarity in the POMDPs.
That is, T ( m, h, m (cid:48) )
=
Pr( M i +
= m (cid:48) | M i = m, H i = h ) .
[Figure]
Problem Statement: We look at the fundamental problem of evaluating the performance of a policy π in the presence of non-stationarity.
Let ( H i ) ni
=1 be the data collected in the past by interacting using policies ( β i ) ni
=1 .
Let D n be the dataset consisting of ( H i ) ni
=1 and the probabilities of the actions taken by ( β i ) ni
=1 .
Given D n , we aim to evaluate the expected future performance of π if it is deployed for the next L episodes (each a different POMDP), that is J ( π ) :
= E π (cid:104)(cid:80) n + L k = n +1 J k
( π ) (cid:12)(cid:12) (cid:12) ( H i ) ni
=1 (cid:105) .
We call it the on-policy setting if ∀ i, β i = π , and the off-policy setting otherwise.
Notice that even in the on-policy setting, naively aggregating observed performances from ( H i ) ni =1 may not be indicative of J ( π ) as M k for k > n may be different than M ∈ ( M i ) ni
=1 due to non-stationarity.
Understanding Structural Assumptions
A careful reader would have observed that instead of considering interactions with a sequence of
POMDPs ( M i ) ni
=1 that are each dependent on the past POMDPs and decisions, an equivalent setup might have been to consider a ‘chained’ sequence of interactions ( H 1 , H 2 , ..., H n ) as a single episode in a ‘mega’ POMDP comprised of all M ∈ M .
Consequently, J ( π ) would correspond to the expected future return given ( H i ) ni
=1 .
Tackling this single long sequence of interactions using the continuing/average-reward setting is not generally viable because methods for these settings rely on an ergodicity assumption (which implies that all states can always be revisited) that may not hold in the presence of non-stationarity.
For instance, in the earlier example of automated healthcare, it is not possible to revisit past years.
To address the above challenge, we propose introducing a different structural assumption.
Particularly, framing the problem as a sequence of POMDPs allows us to split the single sequence of interactions into multiple (dependent) fragments, with additional structure linking together the fragments.
Speciﬁcally, we make the following intuitive assumption.
Assumption 1.
∀ m ∈ M such that the performance J ( π ) associated with m is j ,
[Equation]
Assumption 1 characterizes the probability that π ’s performance will be j i +1 in the i + 1 th episode when the policy π (cid:48) is executed in the i th episode.
To understand Assumption 1 intuitively, consider a ‘meta-transition’ function that characterizes Pr( J i +1 ( π ) | J i ( π ) , π (cid:48) ) similar to how the standard transition function in an MDP characterizes Pr( S t +1 | S t , A t ) .
While the underlying changes actually happen via T , Assumption 1 imposes the following two conditions: (a) A higher-order stationarity condition on the meta-transitions under which non-stationarity can result in changes over time, but the way the changes happen is ﬁxed , and (b) Knowing the past performance(s) of a policy π provides sufﬁcient information for the meta-transition function to model how the performance will change upon executing any (possibly different) policy π (cid:48) .
For example, in the earlier toy robot domain, given the current performance there exists an (unknown) oracle that can predict the performance for the next day if the robot decides to ‘run’/‘walk’.
Assumption 1 is beneﬁcial as it implicitly captures the effect of both the underlying passive and active non-stationarity by modeling the conditional distribution of the performance J i +
( π ) given
J i ( π ) , when executing any (different) policy π (cid:48) .
At the same time, notice that it generalizes (a) the stationary setting, where ∀ π ∈ Π , ∀ i > 0 , J i +1 ( π ) =
J i ( π ) , and (b) only passive non-stationarity, which is a special case of (1) wherein π (cid:48) does not inﬂuence the outcome, i.e., ∀ π a , π b ∈ Π 2 , ∀ i > 0 , {+Pr( J i +
+}
( π ) =
J i +
| J i ( π ) =
j ; π a ) =
Pr( J i +1 ( π ) =
J i +
| J i ( π ) =
j ; π b ) .
Remark 1.
In some cases, it may be beneﬁcial to relax Assumption 1 such that instead of using
Pr( J i +1 ( π ) | J i ( π ); π (cid:48) ) in (1) , one considers Pr( J i +1 ( π )
| ( J i − k
( π )) pk =0 ; π (cid:48) ) .
This can be considered similar to the p-Markov MDP where the transitions are characterized using Pr( S t +1 | ( S t − i ) pi =0 , A t ) .
While we consider this general setting for our empirical results, for simplicity, to present the key ideas we will consider (1) .
We provide a detailed discussion on cases where we expect such an assumption to be (in)valid , and also other potential assumptions in Appendix C.
Model-Free Policy Evaluation
In this section we discuss how under Assumption 1 , we can perform model-free off-policy evaluation amidst passive, active, or hybrid non-stationarity.
The high level idea can be decomposed into the following: (a) Obtain estimates of ( J i ( π )) ni
=1 using ( H i ) ni
=1 (red arrows in Figure 3), and (b) Use the estimates of ( J i ( π ))
n i =1 to infer the effect of the underlying non-stationarity on the performance, and use that to predict ( J i ( π )) n + L i = n +1 (blue arrows in Figure 3).
( J i ( π ))
ni =1 could have been directly estimated if we had access to ( M i ) ni =1 .
However, how do we estimate ( J i ( π )) ni
=1 when we only have ( H i ) ni
=1 collected using interactions via possibly different data collecting policies ( β i ) ni
=1 ?
To estimate ( J i ( π )) ni
=1 , we use the collected data D n and aim to answer the following counterfactual question: what would the performance of π would have been, if π was used to interact with M i instead of β i ?
To answer this, we make the following standard support assumption [Thomas et al., 2015, Thomas and Brunskill, 2016, Xie et al., 2019] that says that any action that is likely under π is also sufﬁciently likely under the policy β i for all i .
Assumption 2.
∀ o ∈ O , ∀ a ∈ A , and ∀ i ≤ n , π ( o,a ) β i ( o,a ) is bounded above by a (unknown) constant c .
Under Assumption 2 , an unbiased estimate of J i ( π ) can be obtained using common off-policy evaluation methods like importance sampling (IS) or per-decision importance sampling (PDIS) [Precup, 2000], ∀ i, (cid:98) J i ( π ) :
= (cid:80) Tt =1 ρ ti R ti , where, ρ ti :=
(cid:81) tj =1 π ( O ji ,A ji )
β
i ( O ji ,A ji ) .
This (cid:98) J i
( π ) provides an estimate of J i ( π ) associated with each M i and policy π , as needed for the red arrows in Figure [-3.5.
-]3.
Double Counterfactual Reasoning Having obtained the estimates for ( J i ( π )) ni
=1 , we now aim to estimate how the performance of π changes due to the underlying non-stationarity.
Recall that under active or hybrid non-stationarity, changes in a policy’s performance due to the underlying non-stationarity is dependent on the past actions.
From Assumption 1 , let
[Equation]
denote how the performance of π changes between episodes, if π (cid:48) was executed.
Here J i +1 ( π ) is a random variable because of stochasticity in H i (i.e., how π (cid:48) interacts in M i ), as well as in the meta-transition from POMDP M i to M i +1 .
Similarly, let ∀
[Equation]
be some (unknown) function parameterized by θ π ∈ Θ , which denotes the expected performance of π in episode i + 1 , if in episode i , π ’s performance was J i ( π ) and π (cid:48) was executed.
Parameters θ π depend on π and thus f can model different types of changes to the performance of different policies.
Recall from Figure 3 (blue arrows), if we can estimate f ( · , π ; θ π ) to infer how J i ( π ) changes due to the underlying non-stationarity when interacting with π , then we can use it to predict ( J i ( π )) n + L i = n + when π is deployed in the future.
In the following, we will predominantly focus on estimating f ( · , π ; θ π ) using past data D n .
Therefore, for brevity we let f ( · ; θ π ) :
= f ( · , π ; θ π ) .
If pairs of ( J i ( π ) , J i +1 ( π )) are available when the transition between M i and M i +1 occurs due to execution of π , then one could auto-regress J i +
( π ) on J i ( π ) to estimate f ( · ; θ π ) and model the changes in the performance of π .
However, the sequence ( (cid:98) J i ( π ))
ni =1 obtained from counterfactual reasoning cannot be used as-is for auto-regression.
This is because the changes that occurred between
M i and M i +1 are associated with the execution of β i , not π .
For example, recall the toy robot example in Figure 1.
If data was collected by mostly ‘running’, then the performance of ‘walking’ would decay as well.
Directly auto-regressing on the past performances of ‘walking’ would result in how the performance of ‘walking’ would change when actually executing ‘running’ .
However, if we want to predict performances of ‘walking’ in the future, what we actually want to estimate is how the performance of ‘walking’ changes if ‘walking’ is actually performed .
To resolve the above issue, we ask another counter-factual question: What would the performance of π in M
i +1 have been had we executed π , instead of β i , in M i ?
In the following theorem we show how this question can be answered with a second application of the importance ratio ρ i :=
ρ Ti .
Theorem 1.
Under Assumptions 1 and 2 , ∀ m ∈ M such that the performance J ( π ) associated with m is j , E
[Equation]
See Appendix D.1 for the proof.
Intuitively, as β i and β i +1 were used to collect the data in i and i + 1 th episodes, respectively, Theorem 1 uses ρ i to ﬁrst correct for the mismatch between π and β i that inﬂuences how M i changes to M i +1 due to interactions H i .
Secondly, (cid:98) J i +1 corrects for the mismatch between π and β i +1 for the sequence of interactions
H i +1 in M
i +1 .
5.3 Importance-
Weighted IV-Regression
An important advantage of Theorem 1 is that given
J i ( π ) , ρ i (cid:98) J i +1 ( π ) provides an unbiased estimate of E π [ J i +1 ( π )
| J i ( π )] , even though π may not have been used for data collection.
This permits using Y i :
=
ρ i (cid:98) J i +1 ( π ) as a target for predicting the next performance given X i :=
J i ( π ) , i.e., to estimate f
( J i ( π ); θ π ) through regression on ( X i , Y i ) pairs.
However, notice that performing regression on the pairs ( X i =
J i ( π ) , Y i =
ρ (cid:98) J i +1 ( π ))
n − 1 i =1 may not be directly possible as we do not have J i ( π ) ; only unbiased estimates (cid:98) J i ( π ) of J i ( π ) .
This is problematic because in least-squares regression, while noisy estimates of the target variable Y i are ﬁne, noisy estimates of the input variable
X i may result in estimates of θ π that are not even asymptotically consistent even when the underlying f is a linear function of its inputs.
To see this clearly, consider the following naive estimator,
[Equation]
Because (cid:98) J i ( π ) is an unbiased estimate of J π , without loss of generality, let (cid:98)
J i ( π ) =
J i ( π ) + η i , where η i is mean zero noise.
Let N :=
[ η 1 , η 2 , ..., η n − 1 ] (cid:62) and J := [ J 1 ( π ) , J [Equation]
, ..., J n [Equation] 1 ( π )] [Equation]
θ naive can now be expressed as (see Appendix D.2),
[Equation]
Observe that N (cid:62) N in (2) relates to the variances of the mean zero noise variables η i .
The greater the variances, the more ˆ θ naive would be biased towards zero (if ∀ i, η i = 0 , then the true θ π is trivially recovered).
Intuitively, when the variance of η i is high, noise dominates and the structure in the data gets suppressed even in the large-sample regime.
Unfortunately, the importance sampling based estimator (cid:98) J i ( π ) in the sequential decision making setting is infamous for extremely high variance [Thomas et al., 2015].
Therefore, ˆ θ naive can be extremely biased and will not be able to capture the trend in how performances are changing, even in the limit of inﬁnite data and linear f .
The problem may be exacerbated when f is non-linear.
Reduction To mitigate the bias stemming from noise in input variables, we introduce a novel instrument variable (IV)
[Pearl et al., 2000] regression method for tackling non-stationarity.
Instrument variables Z represent some side-information and were originally used in the causal literature to mitigate any bias resulting due to spurious correlation, caused by unobserved confounders, between the input and the target variables.
For mitigating bias in our setting, IVs can intuitively be considered as some side-information to ‘denoise’ the input variable before performing regression.
For this IV-regression, an ideal IV is correlated with the input variables (e.g., (cid:98) J i ( π ) ) but uncorrelated with the noises in the input variable (e.g., η i ).
We propose leveraging statistics based on past performances as an IV for (cid:98)
J i ( π ) .
For instance, using
Z i :
= (cid:98) J i − 1 ( π ) as an IV for (cid:98)
J i ( π ) .
Notice that while correlation between J i − 1 ( π ) and J i ( π ) can directly imply correlation between (cid:98) J i − 1 ( π ) and (cid:98)
J i ( π ) , values of J i − 1 ( π ) and J i ( π ) are dependent on non-stationarity in the past.
Therefore, we make the following assumption, which may easily be satisﬁed when the consecutive performances do not change arbitrarily.
Assumption 3.
∀ i,
[Equation]
However, notice that the noise in (cid:98)
J i ( π ) can be dependent on (cid:98) J i − 1 ( π ) .
This is because non-stationarity can make H i − 1 and H i dependent, which are in turn used to estimate ( cid:98) J i − 1 ( π ) and (cid:98)
J i ( π ) , respectively.
Nevertheless, perhaps interestingly, we show that despite not being independent, (cid:98) J i − 1 ( π ) is uncorrelated with the noise in (cid:98) J i ( π ) .
Theorem 2.
Under Assumptions 1 and 2 , ∀ i,
[Equation]
See Appendix D.3 for the proof.
Finally, as IV regression requires learning an additional function g :=
R → R parameterized by ϕ ∈ Ω (intuitively, think of this as a denoising function), we let (cid:98) J i − 1 ( π ) be an IV for (cid:98)
J i ( π ) and propose the following IV-regression based estimator,
[Equation]
Theorem 3.
Under Assumptions 1 , 2 , and 3 , if f and g are linear functions of their inputs, then ˆ θ n is a strongly consistent estimator of θ π , i.e., ˆ θ n a.s. −→ θ π .
(See Appendix D.3 for the proof.)
Remark 2.
Other choices of instrument variables Z i (apart from Z i = (cid:98) J i − 1 ( π ) ) are also viable.
We discuss some alternate choices in
Appendix E.
These other IVs can be used in (3) and (4) by replacing (cid:98) J i − 1 ( π ) with the alternative Z i .
Remark 3.
As discussed earlier, it may be beneﬁcial to model J i +1 ( π ) using ( J k ( π ))
ik = i − p +1 with p > 1 .
The proposed estimator can be easily extended by making f dependent on multiple past terms ( X k )
ik = i − p
+1 , where ∀ k, X k :=
g (( (cid:98) J l ( π ))
k − 1 l = k − p ; ˆ φ ) .
We discuss this in more detail in Appendix E.
The proposed procedure is also related to methods that use lags of the time series as instrument variables [Bellemare et al., 2017, Wilkins, 2018, Wang and Bellemare, 2019].
Remark 4.
An advantage of the model-free setting is that we only need to consider changes in J ( π ) , which is a scalar statistic.
For scalar quantities, linear auto-regressive models have been known to be useful in modeling a wide variety of time-series trends.
Nonetheless, non-linear functions like
RNNs and LSTMs [Hochreiter and Schmidhuber, 1997] may also be leveraged using deep instrument variable methods [Hartford et al., 2017, Bennett et al., 2019, Liu et al., 2020, Xu et al., 2020].
As required for the blue arrows in Figure 3, f ( · ; ˆ θ n ) can now be used to estimate the expected value E π[-[ J i +
-]
[ J i +1 ( π ) | J i ( π )] under hybrid non-stationarity.
Therefore, using f ( · ; ˆ θ n ) we can now autoregressively forecast the future values of ( J i ( π )) n + L i = n +1 and obtain an estimate for J ( π ) .
A complete algorithm for the proposed procedure is provided in Appendix E.1. 5.3.2 Variance Reduction As discussed earlier, importance sampling results in noisy estimates of
J i ( π ) .
During regression, while high noise in the input variable leads to high bias, high noise in the target variables leads to high variance parameter estimates.
Unfortunately, (3) and (4) have target variables containing ρ i (and ρ i +1 ) which depend on the product of importance ratios and can thus result in extremely large values leading to higher variance parameter estimates.
The instrument variable technique helped in mitigating bias.
To mitigate variance, we draw inspiration from the reformulation of weighted-importance sampling presented for the stationary setting by
Mahmood et al.
[2014], and propose the following estimator,
[Equation]
where ¯ ρ i :=
[Equation]
[Equation]
where ρ † i :=
[Equation]
where G i is the return observed for M i .
Intuitively, instead of importance weighting the target , we importance weight the squared error, proportional to how likely that error would be if π was used to collect the data.
Since dividing by any constant does not affect ˜ ϕ n and ˜ θ n , the choice of ¯ ρ i and ρ † i ensures that both ¯ ρ i and ρ † i ∈ [0 , 1] , thereby mitigating variance but still providing consistency.
Theorem 4.
Under Assumptions 1 , 2 , and 3 , if f and g are linear functions of their inputs, then ˜ θ n is a strongly consistent estimator of θ π , i.e., ˜ θ n a.s. −→ θ π .
(See Appendix D.3 for the proof.)
Empirical Analysis
This section presents both qualitative and quantitative empirical evaluations using several environments inspired by real-world applications that exhibit non-stationarity.
In the following paragraphs, we ﬁrst brieﬂy discuss different algorithms being compared and answer three primary questions.
OPEN:
We call our proposed method OPEN: off-policy evaluation for non-stationary domains with structured passive, active, or hybrid changes.
It is based on our bias and variance reduced estimator developed in (5) and (6).
Appendix E.1 contains the complete algorithm.
Pro-WLS: For the baseline, we use Prognosticator with weighted least-squares (Pro-WLS) [Chandak et al., 2020b].
This method is designed to tackle only passive non-stationarity.
WIS:
A weighted importance sampling based estimator that ignores presence of non-stationarity completely [Precup, 2000].
Q1.
(Qualitative Results)
What is the impact of the two stages of the OPEN algorithm?
In Figure 4 we present a step by step breakdown of the intermediate stages of a single run of OPEN on the RoboToy domain from Figure 1.
It can be observed that OPEN is able to extract the effect of the underlying active non-stationarity on the performances and also detect that the evaluation policy π that ‘runs’ more often will cause an active harm, if deployed in the future.
Q2.
(Quantitative Results)
What is the effect of different types and rates of non-stationarity?
Besides the toy robot from Figure 1, we provide empirical results on three other domains inspired by real-world applications that exhibit non-stationarity.
Appendix E.3 contains details for each, including how the evaluation policy and the data collecting policy were designed for them.
Non-stationary Mountain Car:
In real-world mechanical systems, motors undergo wear and tear over time based on how vigorously they have been used in the past.
To simulate similar performance degradation, we adapt the classic (stationary) mountain car domain [Sutton and Barto, 2018].
We modify the domain such that after every episode the effective acceleration force is decayed proportional to the average velocity of the car in the current episode.
This results in active non-stationarity, where the change in the system is based on the actions taken by the agent in the past.
individual patients should account for the physiological and lifestyle changes of the patient over time.
To simulate such a scenario we use an open-source implementation [Xie, 2019] of the U.S. Food and Drug Administration (FDA) approved Type-1 Diabetes Mellitus simulator (T1DMS)
[Man et al., 2014] for the treatment of
Type-1 diabetes, where we induced non-stationarity by oscillating the body parameters (e.g., rate of glucose absorption, insulin sensitivity, etc.)
between two known conﬁgurations available in the simulator.
This induces passive non-stationarity, that is, changes are not dependent on past actions.
MEDEVAC:
This domain stands for medical evacuation using air ambulances.
This domain was developed by Robbins et al.
[2020] for optimally routing air ambulances to provide medical assistance in regions of conﬂict.
Based on real-data, this domain simulates the arrival of different events, from different zones, where each event can have different priority levels.
Serving higher priority events yields higher rewards.
A good controller decides whether to deploy, and which MEDEVAC to deploy, to serve any event (at the risk of not being able to serve a new high-priority event if all ambulances become occupied).
Here, the arrival rates of different events can change based on external incidents during conﬂict.
Similarly, the service completion rate can also change based on how frequently an ambulance is deployed in the past.
To simulate such non-stationarity, we oscillate the arrival rate of the incoming high-priority events, which induces passive non-stationarity.
Further, to induce wear and tear, we decay the service rate of an ambulance proportional to how frequently the ambulance was used in the past.
This induces active non-stationarity.
The presence of both active and passive changes makes this domain subject to hybrid non-stationarity.
Figure 5 presents the (absolute) bias and MSE incurred by different algorithms for predicting the future performance of the evaluation policy π .
As expected, the baseline method WIS that ignores the non-stationarity completely fails to capture the change in performances over time.
Therefore, while WIS works well for the stationary setting, as the rate of non-stationarity increases, the bias incurred by WIS grows.
In comparison, the baseline method Pro-WLS that can only account for passive non-stationarity captures the trend better than WIS, but still performs poorly in comparison to the proposed method OPEN that is explicitly designed to handle active/hybrid non-stationarity.
Perhaps interestingly, for the Diabetes domain which only has passive non-stationarity, we observe that OPEN performs better than Pro-WLS.
As we discuss later, this can be attributed to the sensitivity of Pro-WLS to its hyper-parameters.
While OPEN incorporated one variance reduction technique, it can be noticed when the rate of non-stationarity is high, variance can sometimes still be high thereby leading to higher MSE.
We discuss potential negative impacts of this in Appendix A. Incorporating (partial) knowledge of the underlying model and developing doubly-robust version of OPEN could potentially mitigate variance further.
We leave this extension for future work.
Q3.
(Ablations Results)
How robust are the methods to hyper-parameters?
Due to space constraints, we defer the empirical results and discussion for this to Appendix E.5.
Overall, we observe that the proposed method OPEN being an auto-regressive method can extrapolate/forecast better and is thus more robust to hyper-parameters (number of past terms to condition, as discussed in Remark 3 ) than Pro-WLS that uses Fourier bases for regression (where the hyperparameter is the order of Fourier basis) and is not as good for extrapolation.
Conclusion
We took the ﬁrst steps for addressing the fundamental question of off-policy evaluation under the presence of non-stationarity.
Towards this goal we discussed the need for structural assumptions and developed a model-free procedure OPEN and presented ways to mitigate its bias and variance.
Empirical results suggests that OPEN can now not only enable practitioners to predict future performances amidst non-stationarity but also identify policies that may be actively causing harm or damage.
In the future, OPEN can also be extended to enable control of non-stationary processes.
Acknowledgements
Research reported in this paper was sponsored in part by a gift from Adobe, NSF award #2018372.
This work was also funded in part by the U
S. Army Combat Capabilities Development Command (DEVCOM) Army Research Laboratory under Cooperative Agreement W911NF-17-2-0196 and Support Agreement No. USMA21050.
The views expressed in this paper are those of the authors and do not reﬂect the ofﬁcial policy or position of the United States Military Academy, the United States Army, the Department of Defense or the United States Government. The U S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Off-Policy Evaluation for Action-Dependent Non-Stationary Environments
(Appendix)
Contents
A FAQs: Frequently Asked Questions[-
-]
A.1 How does the stationarity condition for a time-series differ from that in RL?
A.2 Can the POMDP during each episode (Figure itself be {+non-stationary? A.
+}
What if it is known ahead of time that the non-stationarity is passive only
A.4 How should different non-stationarities be treated in the on-policy setting?
A.5 Can you tell us more about when would Assumption 1 be (in)valid? A
6 What are the limitations and potential negative impacts of the work
B Extended Related Work
B.1 Off-policy evaluation in stationary domains
B.2 Non-stationarity in stationary domains
B.3 Single Episode Continuing setting
B.4 Non-stationarity in MDPs/Bandits
B.5 Multi-agent Games
B.6 Dynamical Systems and Time-Series Analysis
C Discussion on the Structural Assumption[-
-]
C.1 Latent Variables
D Proofs for Theoretical Results
D.1 Double Counterfactual Reasoning D
2 Asymptotic bias of ˆ θ naive
D.3 Importance-Weighted IV-Regression
E Empirical Details
E.1 Algorithm
E.2 Implementation and Hyper-parameters
E.3 Environments
E.4 Additional Results
E 5 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A FAQs: Frequently Asked Questions
A.1 How does the stationarity condition for a time-series differ from that in RL?
Conventionally, stationarity is the time-series literature refers to the condition where the distribution (or few moments) of a ﬁnite sub-sequence of random-variables in a time-series remains the same as we shift it along the time index axis
[Cox and Miller, 2017].
In contrast, the stationarity condition in the RL setting implies that the environment is ﬁxed [Sutton and Barto, 2018].
This makes the performance J ( π ) of any policy π to be a constant value throughout.
In this work, we use ‘stationarity’ as used in the RL literature.
A.2 Can the POMDP during each episode (Figure 2) itself be non-stationary?
Any source of non-stationarity can be incorporated in the (unobserved) state to induce another stationary POMDP (from which we can obtain a single sequence of interaction).
The key step towards tractability is Assumption 1 that enforces additional structure on the performance of any policy across the sequence of (non-)stationary POMDPs.
A.
What if it is known ahead of time that the non-stationarity is passive only?
In such cases where the underlying changes are independent of the past actions, E β
[Equation]
[Equation]
π ) | J i ( π )] , for any policies β 1 and β 2 .
Therefore, there is no need for double-counterfactual reasoning to correct for the changes observed in the past.
Particularly, in
Theorem 1 the second use of importance sampling can be avoided as E
[Equation]
E
[Equation]
M i ( π ) (cid:105) under passive non-stationarity.
Rest procedure
OPEN can be modiﬁed accordingly.
A.4 How should different non-stationarities be treated in the on-policy setting?
Perhaps interestingly, OPEN makes no effective distinction between active and passive nonstationarity in the on-policy setting.
Notice that in the on-policy setting, importance ratios ρ =everywhere, therefore the use of double counterfactual reasoning has no impact.
Intuitively, in the on-policy setting, there is no need to dis-entagle the active and passive sources of non-stationarity, as the prediction needs to be made about the same policy that was used during data collection.
A
5 Can you tell us more about when would Assumption 1 be (in)valid?
Yes, we provide a detailed discussion on Assumption 1 in Appendix C.
A.6 What are the limitations and potential negative impacts of the work?
Our work presents the ﬁrst few steps towards off-policy evaluation in the presence of non-stationarity.
Towards this goal, we used Assumption 1 to enforce a higher-order stationarity condition.
We have provided extended discussion regarding the same in Appendix C and a practitioner should carefully analyze their problem setup to conclude if the assumption holds (at least approximately).
Further, often off-policy evaluation is used in safety-critical settings, where it is important to provide conﬁdence intervals [Thomas et al., 2015, 2019, Jiang and Huang, 2020].
Because of our use of instrument variables, our estimator may have high-variance.
This can be explained by observing the closed form equation in (22) obtained using the IV procedure.
Here, Z is the instrument variable and if it is weakly correlated with X (i.e,.
Z (cid:62) X has a small magnitude) then ( Z (cid:62) X ) − 1 can be large thereby increasing variance.
However, our proposed method OPEN only provides point-estimates and thus using it as-is in safety critical settings would be irresponsible.
If the application does exhibit non-stationarity, a practitioner may have to make a tough choice between prior methods that provide conﬁdence intervals under the stationarity assumption, or the proposed method that may be applicable to their non-stationary setting but does not provide any conﬁdence intervals.
B Extended Related Work
In this section we discuss several different research directions that are relevant to the topic of this paper.
We refer the readers to the work by Padakandla [2020], Khetarpal et al.
[2020] for a more exhaustive survey.
B.1 Off-policy evaluation in stationary domains
In the off-policy RL setup, there is a large body of literature that tackles the off-policy estimation problem.
One line of work leverages dynamic programming [Puterman, 1990, Sutton and Barto, 2018] to develop off-policy estimators [Boyan, 1999, Sutton et al., 2008, 2009, Mahmood et al., 2014, 2015].
Several recent approaches also build upon a dual perspective for dynamic programming [Puterman, 1990, Wang et al., 2007, Nachum and Dai, 2020] for performing off-policy evaluation [Liu et al., 2018, Xie et al., 2019, Jiang and Huang, 2020, Uehara et al., 2020, Dai et al., 2020, Feng et al., 2021].
These works require fully-observable states.
Other direction of work takes Monte-Carlo perspective to perform trajectory based importance sampling and are applicable to stationary setting with partial observability [Precup, 2000, Thomas et al., 2015, Jiang and Li, 2015, Thomas and
Brunskill, 2016].
The proposed work builds upon this direction.
Several works have also discussed various techniques for variance reduction [Jiang and Li, 2015,
Thomas and Brunskill, 2016, Munos et al., 2016, Harutyunyan et al., 2016, Liu et al., 2018, Espeholt et al., 2018, Nachum et al., 2019, Yang et al., 2020, Yuan et al., 2021].
However, these methods are restricted to stationary domains.
B.2 Non-stationarity in stationary domains
In the face of uncertainty, prior works often opt for exploratory or safe behavior by acting optimistically or pessimistically, respectively.
This is often achieved by using the collected data to dynamically modify the observed rewards for any state-action pair by either providing bonuses [Agarwal et al., 2020, Taiga et al., 2021] or penalties [Buckman et al., 2020, Cetin and Celiktutan, 2021].
One could view this as an instance of active non-stationarity.
Similarly, in temporal-difference (TD) methods the target for the value function keeps changing and such changes are also dependent on the data collected in the past [Sutton and Barto, 2018].
However, we note that such non-stationarities are only artifacts of the learning algorithm as the underlying domain remains stationary throughout.
In contrast, the focus of our work is on settings where the underlying domain is non-stationary.
B.3 Single Episode Continuing setting
As discussed in Section 4, non-stationarity can be alternatively modeled using a single long episode in a stationary POMDP.
From this point of view, one may wonder if the average-reward/continuing setting [Sutton and Barto, 2018] could be useful?
While there have been off-policy evaluation methods designed to tackle the continuing setting [Liu et al., 2018, Nachum et al., 2019, Yang et al., 2020], they require two important conditions that are no applicable for our setting: (a) They assume access to the true underlying state such that there is no partial-observability, and (b) They assume that the transition tuples are sampled from the stationary state-visitation distribution of a policy.
In the non-stationary setting that we consider, we may not have data from any stationary state visitation distribution, and we may not have access to the true underlying states either.
B.4 Non-stationarity in MDPs/Bandits
Several prior methods have considered tackling non-stationarity for reinforcement learning problems.
For instance, a Hidden-Mode MDP is a setting that assumes that the environment changes are conﬁned to a few hidden modes, where each mode represents a unique MDP.
This provides a tractable way to model a limited number of MDPs [Choi et al., 2000, Basso and Engel, 2009], or perform updates using mode-change detection [Da Silva et al., 2006, Padakandla et al., 2019, Alegre et al., 2021].
Similarly there are methods [Xie et al., 2020a] based on hidden-parameter MDPs [Doshi-Velez and
Konidaris, 2016] that consider a more general setup where the hidden variable can be continuous.
Alternatively, many methods [Thomas et al., 2017, Jagerman et al., 2019, Chandak et al., 2020b, Zhou et al., 2020, Poiani et al., 2021, Liotet et al., 2021] have considered time-dependent MDPs [Rachelson et al., 2009].
Aspects related to safety and conﬁdence intervals have also been explored [Ammar et al., 2015, Chandak et al., 2020a, 2021].
However, the focus of these methods are on settings with passive non-stationarity, where the past actions do not inﬂuence the underlying non-stationarity.
Our works extends this direction of research to provide off-policy evaluation amidst active and hybrid non-stationarity as well.
Non-stationary multi-armed bandits (NMAB) capture the setting where the horizon length is one, but the reward distribution changes over time [Moulines, 2008, Besbes et al., 2014, Russac et al., 2019,
Vernade et al., 2020].
Many variants of NMAB, like cascading non-stationary bandits [Wang et al., 2019a, Li and de Rijke, 2019] and rotting bandits [Levine et al., 2017, Seznec et al., 2018] have also been considered.
In contrast, this work focuses on methods that generalize to the sequential decision making setup where the horizon length can be more than 1.
B.5 Multi-agent Games
Non-stationarity also occurs in multiplayer games
[Singh et al., 2000, Bowling, 2005, Conitzer and
Sandholm, 2007] where the opponent can change their strategy as a response to the agent’s previous decisions.
These types of changes are related to active non-stationarity that we consider in this work.
In such games, opponent modeling has been shown to be useful and regret bounds for multi-player games [Zhang and Lesser, 2010, Mealing and Shapiro, 2013, Foster et al., 2016, Foerster et al., 2018].
Further, often these games still assume that the underlying system/environment (excluding other players) is stationary and focus on searching for (Nash) equilibria.
Similarly, non-stationarities are also induced in the multi-agent systems where an agent tries to inﬂuence other agents [Jaques et al., 2019, Wang et al., 2019b, Xie et al., 2020b, Wang et al., 2021].
However, under general non-stationarity, the underlying system may also change and thus there may not even exist any ﬁxed equilibria.
Perhaps a more relevant setting would be that of evolutionary/dynamics games, where the pay-off matrix and speciﬁcation of the game can change over time [Gemp and Mahadevan, 2017,
Hennes et al., 2019].
Such methods, however, do not leverage any underlying structure in how the game is changing nor do they account for settings where the changes might be a consequence of past interactions of the agent.
While relevant, these other research areas are distinct from our setting of interest.
B.6 Dynamical Systems and Time-Series Analysis
The proposed method for modeling the evolution of a policy’s performance over time using stochastic estimates of past performances may be reminiscent of state-space methods (e.g., Kalman ﬁltering) for dynamical systems [Hamilton, 1994].
However, in comparison to these methods, we do not need to model noise variables, which could have been challenging in our case as noise is heteroskedastic because of past (off-policy) performance estimates being computed using data from different behavior policies.
Further the form of OPEN estimator allows leveraging (accelerated) gradient descent based optimizers to obtain the solution instead of relying on computationally expensive closed-form solutions that are typically needed by state-space models.
Due to this, in practice our method can also be used with non-linear functions f (e.g., recurrent neural network based auto-regressive models).
Different applications of time-series analysis have also discussed the use of lags as instruments [Achen, 2000, Reed, 2015, Bellemare et al., 2017, Wilkins, 2018, Wang and Bellemare, 2019].
Our use case differs from these prior works in that we look at the full sequential decision making setup for reinforcement learning, and also consider a novel importance-weighted instrument-variable regression model.
C Discussion on the Structural Assumption
Assumption 1 states that ∀ m ∈ M such that the performance J ( π ) associated with m is j ,
[Equation]
As discussed earlier, consider a ‘meta-transition’ function that characterizes Pr( J i +1 ( π ) | J i ( π ) , π (cid:48) ) similar to how the standard transition function in an MDP characterizes Pr( S t +1 | S t , A t ) .
This assumption is imposing the following two conditions: (a) A higher-order stationarity condition on the meta-transitions under which non-stationarity can result in changes over time, but the way the changes happen is ﬁxed , and (b) Knowing the past performance(s) of a policy π provides sufﬁcient information for the meta-transition function to model how the performance will change upon executing any (possibly different) policy π (cid:48) .
We provide some examples in Figure 6 to demonstrate few settings to discuss the applicability of this assumption.
C.1 Latent Variables
Instead of enforcing structure on the performances, a possible alternative could have been to enforce structure on how the underlying latent variable (e g., friction of a motor, interests of a user) are changing over time.
While this might be more intuitive for some, just considering structure on this latent variable need not be sufﬁcient.
Dealing with latent/hidden variables can particularly challenging in the off-policy setting, as it may often not be possible (unless additional assumptions are enforced) to infer the latent variable using just the observations from past interactions, even in the stationary setting [Tennenholtz et al., 2020, Namkoong et al., 2020, Shi et al., 2021, Bennett et al., 2021].
Further, the end goal is to estimate the performance of a policy in the future.
Therefore, even if we could infer the possible latent variables for the future episodes, it would still require additional regularity conditions on the (unknown) function that maps from the latent variable to the performance associated with it for any given policy.
Without that it would not be possible to generalize what would the performance be for the inferred latent variables of the future.
And as we discuss in Figure 7, these two assumptions on (a) the structure of how the latent variable could change, and (b) the regularity condition on how the latent variable impacts the performance, can often be reduced to a single condition directly on the structure of how the performances are changing.
D Proofs for Theoretical Results D.1 Double Counterfactual Reasoning
Theorem 1.
Under Assumptions 1 and 2 , ∀ m ∈ M such that the performance J ( π ) associated with m is
[Equation]
[Equation]
Proof.
In the following, to make the dependence of trajectories explicit, we will additionally deﬁne ρ ( h ) and g ( h ) to be the importance ratios and the return associated with a trajectory h .
Using this notation, it can be observed that,
[Equation]
where (a) follows from the law of total probability, (b) follows from the chain rule of probability, (c) follows using conditional independence, where m i +1 is independent of π given h i and M i because of the meta-transition function T , and h
i +1 i independent of h
i and M i given m i +
and π , (d) follows from the use of importance sampling to switch the sampling distribution under Assumption 2 , and (e) follows from re-arrangement of terms.
Finally, ρ i and ρ i +1 are the random variables corresponding the importance ratios in episodes
i and i + 1 .
Random variable G i +1 corresponds to the return under β in episode i + 1 .
Now notice that
[Equation]
where ( f ) follows from Assumption 1 .
Finally, combining (7) and (8),
[Equation]
Similarly, under a more generalized Assumption 1 , where ∀ m ∈ M ,
[Equation]
then similar steps as earlier can be used to conclude that
[Equation]
Note that no additional importance correction is needed in (9) compared to (8).
The term ρ i only shows up to correct for the transition between M i and M i +1 due to the meta-transition function
T ( m, h, m (cid:48) )
=
Pr( M i +
= m (cid:48) | M i = m, H i = h ) .
This independence on the choice of p also holds if
T is non-Markovian in the previous M i values.
Although, additional importance correction would be required if T is dependent on multiple past H i terms.
D.2 Asymptotic bias of ˆ θ
naive
Recall that ˆ θ naive is given by,
[Equation]
Because (cid:98) J i ( π ) is an unbiased estimate of J π , let (cid:98) J i
( π ) =
J i ( π ) + η i , where η i is a mean zero noise.
Let N :=
[ η 1 , η 2 , ..., η n − 1 ] (cid:62) and J :=
[ J 1 ( π ) , J 2 ( π ) , ..., J n − 1 ( π )] (cid:62) .
When f is a linear function of its inputs, expected value
E π
[ J i +1 ( π ) |
J i ( π )]
= J i
θ
π .
Also, as ρ i
ˆ J i +
( π ) is an unbiased estimator for J i ( π ) θ π given J i ( π ) , let ρ i
ˆ J i +1 ( π ) =
J i ( π ) θ π + ζ i , where ζ i is mean zero noise.
Let N 2 :=
[ ζ 1 , ζ 2 , ..., ζ n − 1 ] (cid:62) then θ naive can be expressed as,
[Equation]
In the limit, using continuous mapping theorem when the inverse in (10) exists,
[Equation]
Observe that both N and N 2 are mean zero and uncorrelated with each other and also with J .
Therefore, the terms corresponding to J (cid:62) N , J (cid:62) N 2 , and N (cid:62) N 2 in (11) will be zero almost surely due to Rajchaman’s strong law of large numbers for uncorrelated random variables [Rajchman, 1932,
Chandra, 1991].
However, the term corresponding to N (cid:62) N will not be zero in the limit, and instead roughly result in (average of the) variances of η i .
Consequently, this results in,
[Equation]
Theorem 2.
Under Assumptions 1 and 2 , ∀ i,
[Equation]
Proof.
[Equation]
Focusing on term (II),
[Equation]
where (a) follows from the fact that under Assumption 2 , (cid:98) J i +1 ( π ) is an unbiased estimator for
J i +1 ( π )
[Thomas, 2015].
Focusing on term (I) and using the law of total expectation,
[Equation]
Expanding term (III) further using the law of total expectation,
[Equation]
where in (b) the outer expectation is over the next environment M i +1 given that the current performance estimate is (cid:98)
J i ( π ) and that β i was used for interaction in episode i .
The inner expectation is over (cid:98) J i +1 ( π ) and the trajectory used for estimating (cid:98) J i +1 ( π ) is collected using β in the environment
M i +1 .
Step (c) follows from the fact that conditioned on the environment M i +1 , interactions in M i + are independent of quantities observed in the episodes before i + 1 .
Finally, step (d) follows from observing that
[Equation]
where (e) follows from the fact that under Assumption 2 , (cid:98) J i +1 ( π ) is an unbiased estimator of the performance of π for the given environment M i +1 .
Therefore both (a) and (b) in (12) are zero, and we conclude the result.
Theorem 3.
Under Assumptions 1 , 2 , and 3 , if f and g are linear functions of their inputs, then ˆ θ n is a strongly consistent estimator of θ π , i.e., ˆ θ n a.s. −→ θ π .
(See Appendix D.3 for the proof.)
Proof.
For the linear setting, ˆ θ n can be expressed as,
[Equation] [Equation]
where
[Equation]
Before moving further, we introduce some additional notations.
Particularly, we will use matrix based notations such that it provides more insights into how the steps would work out for other choices of instrument variables as well.
[Equation]
where the diag corresponds to a diagonal matrix with off-diagonals set to zero.
In the following, we split the proof in two parts: (a) we will ﬁrst show that
[Equation]
and then (b) using this simpliﬁed form for ˆ θ n we will show that ˆ θ n a.s. −→ θ π .
Part (a) Solving (13) in matrix form,
[Equation]
Similarly, solving (14) in matrix form,
[Equation]
Now substituting the value of ¯X 2 in [Equation]
[Equation]
Using (15) to substitute the value of ˆ φ n in (17),
[Equation]
Using matrix operations to expand the transposes in (18),
[Equation]
[Equation]
Similarly, using matrix operations to expand inverses in (19) (colored underlines are used to match the terms before expansion in (19) and after expansion in (20)),
[Equation]
[Equation]
Notice that several terms in (20) cancel each other out, therefore,
[Equation]
As a side remark [Equation] we note that if we replace X 1 in the above steps with an appropriate instrument variable Z 1 , then similar steps will follow and will result in
[Equation]
Part (b) Now when f ( J i ( π ); θ π ) :=
E π [ J i +1 ( π ) |
J i ( π )] is a linear function,
[Equation]
where U i +1 is a bounded mean zero noise (which depends on the interaction H i by π ).
Using
Theorem 1 , let
[Equation]
and its unbiased estimate be
[Equation]
For the regression, since (cid:98) J i ( π ) is an unbiased estimate of the input J i
( π ) and (cid:98) Y
i +1 is an unbiased estimate of the target E π
[ J i +1 ( π ) |
J i ( π )] [Equation] these can be equivalently expressed as,
[Equation]
where V i ( H i ) is some bounded mean-zero noise (dependent on the unbiased estimate made using
H i ) and W i +
( H i , H i +1 ) is also a bounded mean-zero noise (dependent on the unbiased estimate made using H i and H i +1 ).
Before moving further [Equation] we deﬁne some additional notation,
[Equation]
] (cid:62)
[Equation]
Using (23) note that (cid:98) Y 3 =
Λ 2 X 3 , therefore (21) can be expressed as,
[Equation]
Unrolling value of (cid:98) Y 3 in (25) using relations from (23) and (24),
[Equation]
Expanding (26),
[Equation]
Evaluating the value of (27) in the limit,
[Equation]
It can be now seen from (28) that if in the limit the terms inside the paranthesis are zero, then we would obtain our desired result.
Focusing on the term (a) and using the continuous mapping theorem,
[Equation]
where Assumption 3 ensures that X 1 and X 2 are correlated and thus their dot product is not zero.
Notice that term (c) (29) can be expressed as 1 n (cid:80) n − 1 i =
X i − 1 V i .
Further, recall from Theorem 2 that
V i is a mean zero random variable uncorrelated with X i − 1 for all i .
Further, V i and X i − 1 are also bounded for all i as both rewards and importance ratios are bounded (Assumption 2 ), and T is ﬁnite.
Now, for α i :=
X i − 1 V
i observe that E [ α i ] =
E [ X i − 1 E
[ V i | X i − 1 ]]
=
E [ X i − 1 0] =
and thus α i is a bounded and mean zero random variable ∀ i .
Therefore, as ( c ) is an average of α variables, it follows from the Rajchaman’s strong law of large numbers for uncorrelated random variables [Rajchman, 1932, Chandra, 1991] that term under ( c ) is zero almost surely.
Thus,
[Equation]
Similarly, for term (b) in (28) observe that both U 3 and W 3 are zero mean random variables uncorrelated with X 1 .
Therefore, term (b) in (28) is also zero in the limit almost surely.
It can now be concluded from (28) that
[Equation]
Theorem 4.
Under Assumptions 1 , 2 , and 3 , if f and g are linear functions of their inputs, then ˜ θ n is a strongly consistent estimator of θ π , i.e., ˜ θ n a.s. −→ θ π .
(See Appendix D.3 for the proof.)
Proof.
For the linear setting, ˜ θ n can be expressed as,
[Equation]
where
[Equation]
Notice that as dividing the objective by a positive constant does not change the optima, we drop the denominator terms in
[Equation]
for the purpose of the analysis.
Before moving further [Equation] we introduce some additional notations besides the ones introduced in the proof of Theorem 3 ,
[Equation]
Solving (30) in matrix form,
[Equation]
Similarly, solving (31) in matrix form,
[Equation]
) where (a) follows from the fact that ρ i ρ i +
G i +
=
ρ i (cid:98) J i +1 ( π ) .
Now substituting the value of ¯X 2 in (32) similar to (17) and (18) in the proof of Theorem 3 ,
[Equation]
Similarly, using matrix operations to expand inverses in (33) (colored underlines are used to match the terms before expansion in (33) and after expansion in (34)) and multiplying and dividing by n ,
[Equation]
Now focusing on the term underlined in green, in the limit, lim n →∞
[Equation]
where in (a) we deﬁned random variable ρ i ρ i +1 as its expected value
E β i ,β i +
[ ρ i ρ i +1 ] plus a mean zero noise ε i .
Step (b) follows from ﬁrst observing that
ρ
i and ρ i +1 are uncorrelated.
Therefore E β i ,β i +
[ ρ i ρ i +1 ]
=
E β i
[ ρ i ]
E β i +
[ ρ i +1 ] = 1 as the expected value of importance ratios is 1 [Thomas, 2015].
Similarly, ε i is uncorrelated with (cid:98) J i − 1 ( π ) , i.e., the expected value E β i ,β i +1 (cid:104)
ε i | (cid:98) J i − 1 ( π ) (cid:105) =
E β i ,β
[Equation]
+
[ ε i ] = 0 for any given J i − 1 ( π ) .
(Intuitively, this step can be seen analogous to the derivation of PDIS, where the expected value of future IS ratios is always one, irrespective of the past events that it has been conditioned on).
Now notice that the random variable ζ i :=
ε i (cid:98) J i − 1 ( π ) (cid:98) J
i − 1 ( π ) (cid:62) is bounded and has mean zero for all i .
Therefore, while ζ i and ζ j may be dependent, they are uncorrelated for all i (cid:54) =
j .
Using strong law of large number for uncorrelated random variables [Rajchman, 1932, Chandra, 1991] the second term in (a) is zero almost surely.
Similarly, it can be observed that 1 n X (cid:62) 1 Λ 2 X 1 converges to 1 n X (cid:62) 1 X 1 .
Therefore using (35) in (34), and using the continuous mapping theorem,
[Equation]
Notice that Assumption 3 ensures that X 1 and X 2 are correlated and thus their dot product is not zero.
Further, several terms in (36) cancel each other out, therefore,
[Equation]
Now proof can be completed similarly to the part (b) of the proof of Theorem 3 .
E Empirical Details E.1 Algorithm
In Section 5 we established the key insight for how to forecast the next performance based on a single previous performance, when the true performance trend of a policy can be modeled auto-regressively using a single past term.
However, as noted in Remark 3 and Figure 7 using more terms can provide more ﬂexibility in the the type of trends that can be modeled.
Therefore, we leverage statistics based on multiple past terms to form the instrument variable Z i .
One immediate choice for Z i is (cid:98)
J i ( π ) .
However, we found that the high variance of IS estimate makes (cid:98) J i ( π ) a weak instrument variable [Pearl et al., 2000], that is not strongly correlated with J i {++
+} ( π ) .
Better choices of Z i may be the ones that are strongly correlated with J i +
( π ) but uncorrelated with the noise in the (cid:98) J
i +1 ( π ) estimate.
We found that an alternate choice of Z i composed of the unweighted return G i and a WIS estimate for J i ( π ) (where the normalization is done only using the importance ratios from episodes before i ) to be more useful.
Speciﬁcally, we let Z i :=
[ G i , (cid:101)
J i ( π )] , where
[Equation]
It can be observed similar to Theorem 2 that this Z i is uncorrelated with the noise in (cid:98) J
i +1 ( π ) as well.
Further, the weighted version (cid:101) J i ( π ) suffers less from variance and we found it to be more strongly correlated with J i +1 ( π ) .
Further, often the performance of the behavior policy is positively/negatively correlated with the performance of the evaluation policy and thus G i tends to be correlated with
J i +1 ( π ) as well.
One could also explore other potential IVs; we leave this for future work.
Now using past p values of Z i to form the complete instrument variable, where p is a hyper-parameter, we use the following importance weighted instrument-variable regression,
[Equation]
where,
[Equation]
Once ˜ θ n is obtained, we use it to auto-regressively forecast the future performances.
Particularly, we use ( ¯ J k ) n + L k = n +1 as the predicted performances for the next L episodes, where
[Equation]
While our theoretical results were established for the setting where there is only a single regressor ( p = 1 ), a more generalized theoretical result for p > 1 may be possible using the concepts of endogenous and exogenous regressors.
Particularly, let [ ..., X i , X i +1 , X i +2 ,
X i +3 , ... ] , be observations from an
{+AR (2) time-series sequence where X i +3 depends on X i +
+}
and X i +2 .
Here, using X i +1 as the only instrument variable for X i +2 is not possible as X i +3 is correlated with X
i +1 .
However, Z = X i or even Z
=
[ X i , X i +1 ] may form a valid instrument for X i +2 as neither the noise in X i +3 nor the noise in X i +2 is correlated with at least one component of Z , i.e., X i .
For precise instrument relevance conditions and additional discussion, we refer the reader to the works by Abbott [2007],
Cameron [2019], Parker [2020].
We leave this theoretical extension for the future work.
E.2 Implementation and Hyper-parameters
For the Pro-WLS baseline, we use the weighted least-squares procedure using the Fourier basis features [Chandak et al., 2020b].
The hyper-parameter for this baseline is the number of Fourier terms d that should be used to estimate the performance trend.
We found that setting d to be too high results in extremely high-variance and setting it to a lower value fails to capture the trend in performance.
Therefore, based on ablation studies in Figure 10 we set d = 5 for all the experiments.
For OPEN, the hyper-parameter corresponds to the number of terms to condition on during autoregression.
Based on ablation studies in Figure 10 we set p =
( 15% of the number of episodes in the data) for all the experiments.
For each environment, we collect data consisting of 2000 episodes of interaction using the behavior policy, and predict the expected future returns if executing the evaluation policy for the nextepisodes.
The behavior policy and the evaluation policy for each domain are described in Section E.3.
Since the future outcomes are stochastic, to evaluate the true expected future performance in (3), we create digital-clones of the environment after data has been collected using the behavior policy.
Using these clones, we compute the average of 30 possible futures when executing the evaluation policy.
This estimate of the expected future returns are then used as the ground truth for comparison with the predictions made by the algorithms.
For Figure 5, | bias | was computed using the absolute value of the difference between (a) the predicted future performance averaged across 30 trials and (b) the ground truth future performance.
That is, for an estimator ˆ J of J , the bias is |
J − E
[ ˆ J ] | .
Because of this, 30 trials only gives us a point estimate for bias.
(Notice that using the absolute value of the difference between (a) the predicted future performance for each trial and (b) the true future performance’, averaged across 30 trials, will provide an estimate of E [ | J − ˆ J | ] , which would not capture the bias but will be more like the variance (using
L1/absolute distance instead of L2)).
E.3 Environments
We provide empirical results on four non-stationary environments: a toy robot environment, nonstationary mountain car, diabetes treatment, and MEDEVAC domain for routing air ambulances.
Details for each of these environments are provided in this section.
For all of the above environments, we regulate the ‘speed’ of non-stationarity to characterize an algorithms’ ability to adapt.
Higher speed corresponds to a faster rate of non-stationarity; A speed of zero indicates that the environment is stationary.
RoboToy
This domain corresponds to the toy robot scenario depicted in Figure 1.
Here, a robot can accomplish a task using either by ‘running’ or ‘walking’.
Robot ﬁnishes a task faster when ‘running’ than ‘walking’ and thus the reward received at the end of ’running’ is higher.
However, ‘running’ causes more wear and tear on the robot, thereby degrading the performance of both ‘running’ or ‘walking’ in the future.
Since the past interactions inﬂuence the non-stationarity, this is an instance of active non-stationarity.
To perform more ablations on our algorithms, we also simulated a RoboToy-Passive domain, where there is no active non-stationarity as above.
Instead, the reward obtained at the end of executing the options ‘walking’ or ‘running’ ﬂuctuate across episodes.
Therefore, the changes to the underlying system are independent of the actions taken by the agent in the past.
For both the active and passive version of this domain, we collect data using a behavior policy that chooses ‘walking’ more frequently, and the evaluation policy is designed such that it chooses ‘running’ more frequently.
Non-stationary Mountain Car:
In real-world mechanical systems, motors undergo wear and tear over time based on how vigorously they have been used in the past.
To simulate similar performance degradation, we adapt the classic (stationary) mountain car domain [Moore, 1990].
We modify the domain such that at every episode the effective acceleration force is decayed proportional to the average velocity of the car in the previous episode.
This results in active non-stationarity as the change in the system is based on the actions taken by the agent in the past.
Similar to the works by [Thomas, 2015, Jiang and Li, 2015], we make use of macro-actions to repeat an action 10 times, which helps in reducing the effective horizon length of each episode.
The maximum number of step per episode using these macros is 30.
For our experiments, using an actor-critic algorithm
[Sutton and Barto, 2018]
we ﬁnd a near-optimal policy π on the stationary version of the mountain car domain, which we use as the evaluation policy.
Let π rand be a random policy with uniform distribution over the actions.
Then we deﬁne the behavior policy β ( o, a ) : = 0 .
5 π ( o, a ) + 0 .
5 π rand ( o, a ) for all states and actions.
Type-1 Diabetes Management:
Automated healthcare systems that aim to personalise for individual patients should account for the physiological changes of the patient over time.
To simulate such a scenario we use an open-source implementation [Xie, 2019] of the U.S. Food and Drug Administration (FDA) approved [-Type-
-]Type-1 Diabetes Mellitus simulator (T1DMS)
[Man et al., 2014] for the treatment of Type-1 diabetes, where we induced non-stationarity by oscillating the body parameters (e.g., rate of glucose absorption, insulin sensitivity, etc.) between two known conﬁgurations available in the simulator.
This induces passive non-stationarity, that is, changes are not dependent on past actions.
Each step of an episode corresponds to a minute ( 1440 timesteps–one for each minute in a day) in an in-silico patient’s body and state transitions are governed by a continuous time non-linear ordinary differential equation (ODE)
[Man et al., 2014].
This makes the problem particularly challenging as it is unclear how the performance trends of policies vary in this domain when the physiological parameters of the patient are changed.
Notice that as the parameters that are being oscillated are inputs to a non-linear ODE system, the exact trend of performance for any policy is unknown.
This more closely reﬂects a real-world setting where Assumption 1 might not hold, as every policy’s performance trend in real-world problems cannot be expected to follow any speciﬁc trend exactly –one can only hope to obtain a coarse approximation of the trend.
For our experiments, using an actor-critic algorithm
[Sutton and Barto, 2018]
we ﬁnd a near-optimal policy π on the stationary version of this domain, which we use as the evaluation policy.
The policy learns the CR and CF parameters of a basal-bolus controller.
Let π rand be a random policy with uniform distribution over actions.
Then we deﬁne the behavior policy
β
( o, a ) :
= 0 .
5 π ( o, a ) + 0 .
5 π rand ( o, a ) for all states and actions.
MEDEVAC:
This domain stands for med ical evac uation using air ambulances.
This domain was developed by Robbins et al.
[2020] for optimally routing air ambulances to provide medical assistance in regions of conﬂict.
This domain divides the region of conﬂict into 34 mutually exclusive zones, and has 4 air ambulances to serve all zones when an event occurs.
Based on real-data, this domain simulates the arrival of different events, from different zones, where each event can have 3 different priority levels.
Serving higher priority events yields higher rewards.
If an ambulance is assigned to an event, it will ﬁnish the assignment in a time dependent on the distance between the base of the ambulance and the zone of the corresponding event.
While engaged in an assignment, that ambulance is no longer available to serve other events.
A good controller decides whether to deploy, and which
MEDEVAC to deploy, to serve any event (at the risk of not being able to serve a new high-priority event if all ambulances become occupied).
The original implementation of the domain assumes that the arrival rates of the events and the time taken by an ambulance to complete an event follow a Poisson process with a ﬁxed rate.
However, in reality, the arrival rates of different events can change based on external incidents during conﬂict.
Similarly, the completion rate can also change based on how frequently an ambulance is deployed.
To simulate such non-stationarity, we oscillate the arrival rate of the incoming high-priority events, which induces passive non-stationarity.
Further, to induce wear and tear, we slowly decay the rate at which an ambulance can ﬁnish an assignment.
This decay is proportional to how frequently the ambulance was used in the past.
This induces active non-stationarity.
The presence of both active and passive changes makes this domain subject to hybrid non-stationarity.
Similar to other domains, we used an actor-critic algorithm
[Sutton and Barto, 2018]
we ﬁnd a near-optimal policy π on the stationary version of this domain, which we use as the evaluation policy.
Let π rand be a random policy with uniform distribution over the actions.
Then we deﬁne the behavior policy β ( o, a ) : = 0 .
5 π ( o, a ) + 0 .
5 π rand ( o, a ) for all states and actions.
E.4 Additional Results
While the primary focus of this chapter was to develop methods to handle active/hybrid nonstationarity, we observed that the proposed method OPEN also provides beneﬁts over the earlier algorithm Pro-WLS even when it is known that there is only passive non-stationarity in the environment.
E.4.1 Single Run
Similar to Figure 4, in Figure 8 we present a step by step breakdown of the intermediate stages of a single run of OPEN on the RoboToy-Passive domain.
Here the trend in how the performance of the evaluation policy was changing in the past remains the same in the future.
When only passive non-stationarity is present, the double counter-factual correction performed by OPEN is superﬂuous.
However, it can be observed that OPEN can still correctly identify the trend and provide useful predictions of π ’s future performance.
E.4.2 Summary Plots
In Figure 9 we provide bias and MSE analysis of different algorithms on the domains that exhibit passive non-stationarity.
Except for the stationary setting, where WIS has the best performance overall, we observe that for all other settings in the plot, OPEN performs better than both Pro-WLS and WIS consistently.
The blue curve corresponds to the performances J i ( π ) for the past episodes.
As there is no active non-stationarity, the choice of actions executed does not impact the underlying non-stationarity.
Therefore, J i ( π ) follows the same trend in future as it did in the past.
The blue and gray curves are unknown to the algorithm.
OPEN ﬁrst uses historical data to obtain counterfactual estimates of J i ( π ) for the past episodes.
One can see the high-variance in these estimates (notice the change in the y-scale) due to the use of importance sampling.
Before naively auto-regressing, OPEN ﬁrst aims to denoise the past performance estimates using the ﬁrst stage of instrument variable regression.
Since p = 300 , the ﬁrstterms were not denoised.
It can be observed that OPEN successfully denoises the importance sampling estimates.
Using the denoised estimates of past performances, with the second use of counterfactual reasoning, OPEN performs the second stage of regression to forecast the future performance when π will be deployed.
In the passive setting, use of double-counterfactual is superﬂuous but OPEN is still able to correctly predict the future performance.
One thing that particularly stands out in these plots is the poor performance of Pro-WLS, despite being designed for the passive setting.
We observed that because of the choice of parametric regression using the Fourier basis, Pro-WLS tends to suffer from high bias when the number of Fourier terms is not sufﬁcient to model the underlying trend.
Also, if the number of Fourier terms is increased naively, then they overﬁt the data and extrapolate poorly, thereby resulting in high-variance.
In contrast, our method is based on an auto-regressive based time-series forecast that is more robust to the model choice (we kept the number of lag terms for auto-regression as p = 300 for OPEN for all our experiments).
To obtain all the results for Figure 5 and Figure 9, in total 30 different seeds were used for each speed of each domain for each algorithm to get the standard error.
The authors had shared access to a computing cluster, consisting of 50 compute nodes with 28 cores each, which was used to run all the experiments.
E.5 Ablation Study
In this section we study the sensitivity to hyper-parameters for the proposed method OPEN and the baseline method Pro-WLS [Chandak et al., 2020b].
The hyper-parameter for OPEN corresponds to the number of past terms to condition on for auto-regression, as discussed in Remark 3 .
The hyper-parameter for Pro-WLS corresponds to the order of Fourier bases required for parametric regression.
In Figure 10 we present the results for how the performance of the methods vary for different choices of hyper-parameters.