Optimizing for the Future in Non-Stationary MDPs
Abstract
Most reinforcement learning methods are based upon the key assumption that the transition dynamics and reward functions are ﬁxed, that is, the underlying Markov decision process is stationary.
However, in many real-world applications, this assumption is violated and using existing algorithms may result in a performance lag.
To proactively search for a good future policy, we present a policy gradient algorithm that maximizes a forecast of future performance.
This forecast is obtained by ﬁtting a curve to the counter-factual estimates of policy performance over time, without explicitly modeling the underlying non-stationarity.
The resulting algorithm amounts to a non-uniform reweighting of past data, and we observe that minimizing performance over some of the data from past episodes can be beneﬁcial when searching for a policy that maximizes future performance.
We show that our algorithm, called Prognosticator, is more robust to non-stationarity than two online adaptation techniques, on three simulated problems motivated by real-world applications.
1. Introduction
Policy optimization algorithms in RL are promising for obtaining general purpose control algorithms. They are designed for Markov decision processes (MDPs), which model a large class of problems (Sutton & Barto, 2018). This generality can facilitate application to a variety of realworld problems.
However, most existing algorithms assume that the environment remains stationary over time.
This assumption is often violated in practical problems of interest.
For example, consider an assistive driving system.
Over time, tires suffer from wear and tear, leading to in- creased friction and thus, change in the system dynamics
Similarly, in almost all human-computer interaction applications, e.g., automated medical care, dialogue systems, and marketing, human behavior changes over time.
In such scenarios, if the automated system is not adapted to take such changes into account, or if it is adapted only after observing such changes, then the system might quickly become suboptimal, incurring severe loss (Moore et al., 2014).
This raises our main question: how do we build systems that proactively search for a policy that will be good for the future MDP?
In this paper we present a policy gradient based approach to search for a policy that maximizes the forecasted future performance when the environment is non-stationary.
To capture the impact of changes in the environment on a policy’s performance, ﬁrst, the performance of the policy during the past episodes is estimated using counter-factual reasoning.
Subsequently, a regression curve is ﬁt to these estimates to model the performance trend of the policy over time, thereby enabling the forecast of future performance.
By differentiating this performance forecast with respect to the parameters of the policy being evaluated, we obtain a gradient-based optimization procedure that proactively searches for a policy that will perform well in the future.
Recently, Al-Shedivat et al.
(2017) and Finn et al. (2019) also presented methods that search for initial parameters that are effective when the objective is changing over time. These approaches are complementary to our own, as they could be additionally applied to set the initial parameters of our algorithms. In our empirical study, we show how the adaptation procedure of their methods alone can result in a performance lag that is mitigated by our method, which explicitly captures the trend of the objective due to nonstationarity.
A detailed survey on other approaches can be found in the work by Padakandla (2020). the focus of this work. Here, ‘non-stationarity’ refers to the transition dynamics and reward function of an environment changing across episodes as described in Section 3. Advantages: The proposed method has the following advantages: (a) It does not require modeling the transition function, reward function, or how either changes in a nonstationary environment, and thus scales gracefully with respect to the number of states and actions in the environment.
(b) Irrespective of the complexity of the environment or the policy parameterization, it concisely models the effect of changes in the environment on a policy’s performance using a univariate time-series.
(c) It is data-efﬁcient in that it leverages all available data. (d) It mitigates performance lag by proactively optimizing performance for episodes in both the immediate and near future.
(e) It degenerates to an estimator of the ordinary policy gradient if the system is stationary, meaning that there is little reason not to use our approach if there is a possibility that the system might be non-stationary.
Limitations: The method that we propose is limited to settings where (a) non-stationarity is governed by an exogenous process (i.e., past actions do not impact the underlying non-stationarity), which has no auto-correlated noise, and (b) performance of every policy changes smoothly over time and has no abrupt breaks/jumps.
Further, we found that our method is sensitive to a hyper-parameter that trades off exploration and exploitation in the non-stationary setting. We conclude the paper with a discussion of these limitations.
2. Notation
An MDP M is a tuple ( S , A , P , R , γ, d 0 ) , where S is the set of possible states, A is the set of actions, P is the transition function, R is the reward function, γ is the discount factor, and d 0 is the start state distribution.
Let R ( s, a ) denote the expected reward of taking action a in state s .
For any given set X , we use ∆( X ) to denote the set of distributions over X .
A policy π : S → ∆( A ) is a distribution over the actions conditioned on the state.
When π is parameterized using θ ∈ R d , we denote it as π θ .
In a non-stationary setting, as the MDP changes over time, we use M k to denote the MDP during episode k .
In general, we will use sub-script k to denote the episode number and a super-script t to denote the time-step within an episode.
S tk , A tk , and
R tk are the random variables corresponding to the state, action, and reward at time step t , in episode k .
Let H k denote a trajectory in episode k : ( s 0 k , a 0 k , r 0 k , s 1 k , a 1 k , ..., s Tk ) , where T is the ﬁnite horizon.
The value function evaluated at state s , during episode k , under a policy π is v πk ( s ) =
E [ (cid:80) T − t j =0 γ j R t + j k | S tk = s, π ] , where condition- ing on π denotes that the trajectory in episode k was sampled using π .
The start state objective for a policy π , in episode k , is deﬁned as J k ( π ) :
= (cid:80) s d 0 ( s ) v πk ( s ) .
Let
J ∗ k = max π J k
( π ) be the performance of an optimal policy for M k .
Often we write θ in place of π θ when the dependence on θ is important.
3. Problem Statement
To model non-stationarity, we let an exogenous process change the MDP from M k to M k +1 , i.e., between episodes.
Let {M k } ∞ k =1 represent a sequence of MDPs, where each
MDP M k is denoted by the tuple ( S , A , P k , R k , γ, d 0 ) .
In many problems, like adapting to friction in robotics, human-machine interaction, etc., the transition dynamics and rewards function change, but every other aspect of the
MDP remains the same throughout.
Therefore, we assume that for any two MDPs, M k and M k +1 , the state set S , the action set A , the starting distribution d 0 , and the discount factor γ are the same.
If the exogenous process changing the MDPs is arbitrary and changes it in unreasonable ways, then there is little hope of ﬁnding a good policy for the future MDP as M k +1 can be wildly different from everything the agent has observed by interacting with the past MDPs, M 1 , ..., M k .
However, in many practical problems of interest, such changes are smooth and have an underlying (unknown) structure.
To make the problem tractable, we therefore assume that both the transition dynamics ( P 1 , P 2 , ... ) , and the reward functions ( R 1 , R 2 , ... ) vary smoothly over time in a way that ensures there are no abrupt jumps in the performance of any policy.
Problem Statement.
We seek to ﬁnd a sequence of policies that minimizes lifelong regret:
[Equation]
4. Related Work
The problem of non-stationarity has a long history and no effort is enough to thoroughly review it.
Here, we brieﬂy touch upon the most relevant work and defer a more detailed literature review to the appendix.
A more exhaustive survey can be found in the work by Padakandla (2020).
Perhaps the work most closely related to ours is that of AlShedivat et al.
They consider a setting where an agent is required to solve test tasks that have different transition dynamics than the training tasks.
Using meta-learning, they aim to use training tasks to ﬁnd an initialization vector for the policy parameters that can be quickly ﬁne-tuned when facing tasks in the test set.
In many real-world prob- lems, however, access to such independent training tasks may not be available a priori .
In this work, we are interested in the continually changing setting where there is no boundary between training and testing tasks.
As such, we show how their proposed online adaptation technique that ﬁne-tunes parameters, by discarding past data and only using samples observed online, can create performance lag and can therefore be data-inefﬁcient.
In settings where training and testing tasks do exist, our method can be leveraged to better adapt during test time, starting from any desired parameter vector.
Recent work by Finn et al.
(2019) aims at bridging both the continuously changing setting and the train-test setting for supervised-learning problems.
They propose continuously improving an underlying parameter initialization vector and running a Follow-The-Leader (FTL) algorithm (Shalev-Shwartz et al., 2012) every time new data is observed.
A naive adaption of this for RL would require access to all the underlying MDPs in the past for continuously updating the initialization vector, which would be impractical.
Doing this efﬁciently remains an open question and our method is complementary to choosing the initialization vector.
Additionally, FTL based adaptation always lags in tracking optimal performance as it uniformly maximizes performance over all the past samples that might not be directly related to the future.
Further, we show that by explicitly capturing the trend in the non-stationarity, we can mitigate this performance lag resulting from the use of an
FTL algorithm during the adaptation process.
The problem of adapting to non-stationarity is also related to continual learning (Ring, 1994), lifelong-learning (Thrun, 1998), and meta-learning (Schmidhuber, 1999).
Several meta-learning based approaches for ﬁne-tuning a (mixture of) trained model(s) using samples observed during a similar task at test time have been proposed (Nagabandi et al., 2018a;b).
Other works have also shown how models of the environment can be used for continual learning (Lu et al., 2019) or using it along with a model predictive control (Wagener et al., 2019).
Concurrent work by Xie et al.
(2020) also demonstrates how modeling the changes in a dynamic-parameter MDP can be useful.
We focus on the model-free paradigm and our approach is complementary to these model-based methods. More importantly, in many real-world applications, it can be infeasible to update the system frequently if it involves high computational or monetary expense.
In such a case, even optimizing for the immediate future might be greedy and sub-optimal.
The system should optimize for a longer term in the future, to compensate for the time until the next update is performed.
None of the prior approaches can efﬁciently tackle this problem.
5. Optimizing for the Future
The problem of minimizing lifelong regret is straightforward if the agent has access to sufﬁcient samples, in advance, from the future environment, M k +1 , that it is going to face (where k denotes the current episode number).
That is, if we could estimate the start-state objective, J k +1 ( π ) , for the future MDP M k +1 , then we could search for a policy π whose performance is close to J ∗ K +1 .
However, obtaining even a single sample from the future is impossible, let alone getting a sufﬁcient number of samples.
This necessitates rethinking the optimization paradigm for searching for a policy that performs well when faced with the future
unknown
MDP.
There are two immediate challenges here:
How can we estimate J k +1 ( π ) without any samples from M k +1 ?
How can gradients, ∂J
k +1 ( π ) /∂θ , of this future performance be estimated?
In this section we address both of these issues using the following idea.
When the transition dynamics ( P 1 , P 2 , ... ) , and the reward functions ( R 1 , R 2 , ... ) are changing smoothly the performances ( J 1 ( π ) , J 2 ( π ) , ... ) of any policy π will also vary smoothly over time.
The impact of smooth changes in the environment thus manifests as smooth changes in the performance of any policy, π .
In cases where there is an underlying, unknown, structure in the changes of the environment, one can now ask: if the performances
J 1: k ( π ) :
= ( J 1 ( π ) , ..., J k ( π )) of π over the course of past episodes were known, can we analyze the trend in its past performances to ﬁnd a policy that maximizes future performance J k +1 ( π ) ?
5.1. Forecasting Future Performance
In this section we address the ﬁrst challenge of estimating future performance J k +1 ( π ) and pose it as a time series forecasting problem.
Broadly, this requires two components: (a) A procedure to compute past performances, J 1: k ( π ) , of π .
(b) A procedure to create an estimate,
ˆ J k +1 ( π ) , of π ’s future performance,
J k +1 ( π ) , using these estimated values from component (a).
An illustration of this idea is provided in Figure 1.
a Component (a).
As we do not have access to the past MDPs for computing the true values of past performances, J 1: k ( π ) , we propose computing estimates, ˆ J 1: k ( π ) , of them from the observed data.
That is, in a non-stationary MDP, starting with the ﬁxed transition matrix P 1 and the reward function R 1 , we want to estimate the performance J i ( π ) of a given policy in episode i ≤ k .
Leveraging the fact that the changes to the underlying MDP are due to an exogenous processes, we can estimate J i ( π ) as,
[Equation]
where P i and R i are also random variables.
Next we describe how an estimate of J i ( π ) can be obtained from (1) using information only from the i th episode.
To get an unbiased estimate, ˆ J i ( π ) , of π ’s performance during episode i , consider the past trajectory H i of the i th episode that was observed when executing a policy β i .
By using counter-factual reasoning (Rosenbaum & Rubin, 1983) and leveraging the per-decision importance sampling (PDIS) estimator (Precup, 2000), an unbiased estimate of
J i ( π ) is thus given by:
[Equation]
It is worth noting that computing (2) does not require storing all the past policies
β i , one needs to only store the actions and the probabilities with which these actions were chosen.
Component (b).
To obtain the second component, which captures the structure in ˆ J 1: k ( π ) :
= ( ˆ J 1 ( π ) , ..., ˆ J k ( π )) and predicts future performances, we make use of a forecasting function Ψ that estimates future performance ˆ J k +
( π ) conditioned on the past performances:
[Equation]
While Ψ can be any forecasting function, we consider Ψ to be an ordinary least squares (OLS) regression model with parameters w ∈ R d × 1 , and the following input and output variables,
[Equation]
For any x ∈ X , let φ ( x ) ∈ R 1 × d denote a d -dimensional basis function for encoding the time index.
For example, an identity basis φ ( x ) :
=
{ x, 1 } , or a Fourier basis, where φ ( x ) := { sin(2 πnx | n ∈ N > 0 ) }∪{ cos(2 πnx )
| n ∈ N > 0 }∪{ 1 } .
Let Φ ∈ R k ×
d be the corresponding basis matrix.
The solution to above least squares problem is w =
(Φ (cid:62) Φ) − 1 Φ (cid:62) Y (Bishop, 2006) and the forecast of the future performance can be obtained using,
[Equation]
This procedure enjoys an important advantage – by just using a univariate time-series to estimate future performance, it bypasses the need for modeling the environment, which can be prohibitively hard or even impossible.
Further, note that Φ (cid:62) Φ ∈ R d × d , where d << k typically, and thus the cost of computing the matrix inverse is negligible.
These advantages allows this procedure to gracefully scale to more challenging problems, while being robust to the size, |S| , of the state set or the action set |A| .
5.2. Differentiating Forecasted Future Performance
In the previous section, we addressed the ﬁrst challenge and showed how to proactively estimate future performance, ˆ
J k +1 ( θ ) , of a policy π θ by explicitly modeling the trend in its past performances
ˆ J 1: k ( θ ) .
In this section, we address the second challenge to facilitate a complete optimization procedure.
A pictorial illustration of the idea is provided in
Figure 2.
Gradients for ˆ J k +1 ( θ ) with respect to θ can be obtained as follows,
[Equation]
The decomposition in (5) has an elegant intuitive interpretation.
The terms assigned to ( a ) in (5) correspond to how the future prediction would change as a function of past outcomes, and the terms in ( b ) indicate how the past outcomes would change due to changes in the parameters of the policy π θ .
In the next paragraphs, we discuss how to obtain the terms ( a ) and ( b ) .
To obtain term (a), note that in (4), ˆ J i ( θ ) corresponds to the i th element of Y , and so using (3) the gradients of the terms (a) in (5) are,
[Equation]
where [ Z ] i represents the i th element of a vector Z .
Therefore, (6) is the gradient of predicted future performance with respect to an estimated past performance .
The term ( b ) in (5) corresponds to the gradient of the PDIS estimate ˆ J i ( θ ) of the past performance with respect to policy parameters θ .
The following Property provides a form for (b) that makes its computation straightforward.
Property (PDIS gradient) . Let [Equation] =
[Equation] dθ [Equation]
t
[-(cid:81) lj =
 π θ ( A ji | S ji ) β i ( A ji | S ji ) , then-][Equation]
[Equation]
Proof.
See Appendix B.
5.3. Algorithm
We provide a sketch of our proposed Prognosticator procedure for optimizing the future performance of the policy in Algorithm 1.
To make the method more practical, we incorporated two additional modiﬁcations to reduce computational cost and variance.
1 Input Learning-rate η , time-duration δ , entropy-regularizer λ 2 Initialize Forecasting function Ψ , Buffer B 3 while True do # Record a new batch of trajectories using π θ for episode = 1 , 2 , ..., δ do [Equation]
[Equation] [Equation] [Equation] r 0: T ) }
B .
insert ( h )
# Update for future performance for i
= 1 , 2 , ... do # Evaluate past performances for k = 1 , 2 , ..., | B | do
[-= (cid:80) Tt =
-]
[Equation]
# Future forecast and its gradient
[Equation] [Equation]
First, it is often desirable to perform an update only after a certain episode interval δ to reduce computational cost.
This raises the question: if a newly found policy will be executed for the next δ episodes, should we choose this new policy to maximize performance on just the single next episode, or to maximize the average performance over the next δ episodes?
An advantage of our proposed method is that we can easily tune how far in the future we want to optimize for.
Thus, to minimize lifelong regret, we propose optimizing for the mean performance over the next δ episodes.
That is,
[Equation]
Second, notice that if the policy becomes too close to deterministic, there would be two undesired consequences.
(a The policy will not cause exploration, precluding the agent from observing any changes to the environment in states that it no longer revisits—changes that might make entering those states worthwhile to the agent.
(b In the future when estimating ˆ J k +1 ( θ ) using the past performance of θ , importance sampling will have high variance if the policy executed during episode k + 1 is close to deterministic.
To mitigate these issues, we add an entropy regularizer H during policy optimization.
More details are available in
Appendix D.
6. Understanding the Behavior of Prognosticator
Notice that as the scalar term (a) is multiplied by the PDIS gradient term (b) in (5), the gradient of future performance can be viewed as a weighted sum of off-policy policy gradients.
In Figure 3, we provide visualization of the weights
∂ ˆ J 100 ( θ ) /∂
ˆ J i ( θ ) for PDIS gradients of each episode i , when the performance for 100 th episode is forecasted using data from the past 99 episodes.
For the speciﬁc setting when Ψ is an OLS estimator, these weights are independent of
Y in (6) and their pattern remains constant for any given sequence of MDPs.
Importantly, note the occurrence of negative weights in
Figure 3 when the identity basis or Fourier basis is used, suggesting that the optimization procedure should move towards a policy that had lower performance in some of the past episodes.
While this negative weighting seems unusual at ﬁrst glance, it has an intriguing interpretation.
We discuss this in the following paragraphs.
To better understand these negative weights, consider a qualitative comparison when weights from different methods in Figure 3 are used along with the performance estimates of policies π 1 and π 2 in Figure 1.
Despite having lower estimates of return everywhere, π 2 ’s rising trend suggests that it might have higher performance in the future, that is, J k +1 ( π 2 ) > J k +
( π 1 ) .
Existing online learning methods like FTL, maximize performance on all the past data uniformly (green curve in Figure 3).
Similarly, the exponential weights (red curve in Figure 3) are representative of approaches that only optimize using data from recent episodes and discard previous data.
Either of these methods that use only non-negative weights can never capture the trend to forecast J k +1 ( π 2 ) >
J k +1 ( π 1 ) .
However, the weights obtained when using the identity basis would facilitate minimization of performances in the distant past and maximization of performance in the recent past.
Intuitively, this means that it moves towards a policy whose performance is on a linear rise, as it expects that policy to have better performance in the future.
While weights from the identity basis are useful for forecasting whether J k +1 ( π 2 )
> J k +1 ( π 1 ) , it cannot be expected that the trend will always be linear as in Figure 1.
To be more ﬂexible and allow for any smooth trend, we opt to use the Fourier basis in our experiments.
Observe the alternating sign of weights in Figure 3 when using the Fourier basis.
This suggests that the optimization procedure will take into account the sequential differences in performances over the past, thereby favoring the policy that has shown the most performance increments in the past.
This also avoids restricting the performance trend of a policy to be linear.
7. Mitigating Variance
While model-free algorithms for ﬁnding a good policy are scalable to large problems, they tend to suffer from highvariance (Greensmith et al., 2004).
In particular, the use of importance sampling estimators can increase the variance further (Guo et al., 2017).
In our setup, high variance in estimates of past performances ˆ J 1: k ( π ) of π can hinder capturing π ’s performance trend, thereby making the forecasts less reliable.
Notice that a major source of variance is the availability of only a single trajectory sample per MDP M i , for all
i ∈ N .
If this trajectory H i , generated using β
i is likely when using β i , but has near-zero probability when using π then the estimated ˆ J i ( π ) is also nearly zero.
While ˆ J i ( π ) is an unbiased estimate of J i ( π ) , information provided by this single H i is of little use to evaluate J i ( π ) .
Subsequently, discarding this from time-series analysis, rather than setting it to be 0 , can make the time series forecast more robust against outliers.
In comparison, if trajectory H i is unlikely when using β i but likely when using π , then not only is
H i very useful for estimating J i ( π ) but it also has a lower chance of occurring in the future, so this trajectory must be emphasized when making a forecast.
Such a process of (de-)emphasizing estimates of past returns using the collected data itself can introduce bias, but this bias might be beneﬁcial in this few-sample regime.
To capture this idea formally, we build upon the insights of
Hachiya et al.
(2012) and Mahmood et al.
(2014), who draw an equivalence between weighted least-squares (WLS) estimation and the weighted importance sampling (WIS) (Precup, 2000) estimator.
Particularly, let G
i
=
(cid:80) Tt =
γ t R ti be the discounted return of the i th trajectory observed from a stationary MDP, and ρ † i :=
ρ i (0 , T ) be the importance ratio of the entire trajectory.
The WIS estimator, ˆ J † ( π ) , of the performance of π in a stationary MDP can then be obtained as,
[Equation]
To mitigate variance in our setup, we propose extending
WIS.
In the non-stationary setting, to perform WIS while capturing the trend in performance over time, we use a modiﬁed forecasting function Ψ † , which is a weighted leastsquares regression model with a d − dimensional basis function φ , and parameters w † ∈ R d × 1 ,
[Equation]
Let Λ ∈ R k × k be a diagonal weight matrix such that Λ ii =
ρ † i , let Φ ∈ R k × d be the basis matrix, and let the following be input and output variables,
[Equation]
The solution to the weighted least squares problem in (7) is then given by w † = (Φ (cid:62) ΛΦ) − 1 Φ (cid:62) Λ Y and the forecast of the future performance can be obtained using,
[Equation]
ˆ J † k +
( π ) has several desired properties.
It incorporates a notion of how relevant each observed trajectory is towards forecasting, while also capturing the desired trend in performance.
The forecasts are less sensitive to the importance sampling variances and the entire process is still fully differentiable.
8. Strictly Generalizing the Stationary Setting
As the agent is unaware of how the environment is changing, a natural question to ask is what if the agent wrongly assumed a stationary environment was non-stationary?
What is the quality of of the agent’s performance forecasts?
What is the impact of the negative weights on past evaluations of a policy’s performance?
Here we answer these questions.
Before stating the formal results, we introduce some necessary notation and two assumptions.
Let J ( π ) be the performance of policy π for a stationary MDP.
Let ˆ J k + δ ( π ) and ˆ J † k + δ ( π ) be the non-stationary importance sampling (NIS) and non-stationary weighted importance sampling (NWIS) estimators of performance δ episodes in future.
Further, let the basis function φ used for encoding the time index in both Ψ and Ψ † be such that it satisﬁes the following conditions: (a) φ ( · ) always contains 1 to incorporate a bias/intercept coefﬁcient in least-squares regression (for example, φ ( · ) =
[ φ 1 ( · ) , ..., φ d − 1 ( · ) , 1] , where φ i ( · ) are arbitrary functions).
(b) Φ has full column rank such that (Φ (cid:62) Φ) − 1 exists.
Both these properties are trivially satisﬁed by most basis functions.
With this notation and these assumptions, we then have the following results indicating that NIS is unbiased and consistent like ordinary importance sampling and NWIS is biased and consistent like weighted importance sampling.
Theorem 1 (Unbiased NIS) .
For all δ ≥ 1 , ˆ J k + δ ( π ) is an unbiased estimator of J ( π ) , that is E [ ˆ J k
+ δ ( π )]
=
J ( π ) .
Theorem 2 (Biased NWIS) .
For all δ ≥ 1 , ˆ J † k + δ ( π ) may be a biased estimator of J ( π ) , that is, E
[ ˆ J † k
+ δ ( π )]
(cid:54) =
J ( π ) always.
Theorem 3 (Consistent NIS) .
For all δ ≥ 1 , ˆ J k + δ ( π ) is a consistent estimator of J ( π ) , that is as N → ∞ , ˆ J N + δ ( π ) a.s. −→ J ( π ) .
Theorem 4 (Consistent NWIS) .
For all δ ≥ 1 , ˆ J † k + δ ( π ) is a consistent estimator of J ( π ) , that is as N → ∞ ,
ˆ J † N + δ ( π ) a.s. −→ J
( π ) .
Proof. See Appendix A for all of these proofs.
Since NWIS is biased and consistent like the WIS estimator, we expect it to have similar variance reduction properties that can potentially make the optimization process more efﬁcient in a non-stationary MDP.
9. Empirical Analysis
This section presents empirical evaluations using several environments inspired by real-world applications that exhibit non-stationarity.
In the following paragraphs, we brieﬂy discuss each environment; a more detailed description is available in Appendix D.
This environment is based on an open-source implementation (Xie, 2019) of the FDA approved Type-1 Diabetes Mellitus simulator (T1DMS) (Man et al., 2014) for treatment of Type-1 Diabetes.
Each episode consists of a day in an in-silico patient’s life.
Consumption of a meal increases the blood-glucose level in the body and if the blood-glucose level becomes too high, then the patient suffers from hyperglycemia and if the level becomes too low, then the patient suffers from hypoglycemia.
The goal is to control the bloodglucose level of a patient by regulating the insulin dosage to minimize the risk associated with both hyper and hypoglycemia.
However, the insulin sensitivity of a patient’s internal body organs vary over time, inducing non-stationarity that should be accounted for.
In the T1DMS simulator, we induce this non-stationarity by oscillating the body parameters (e.g., insulin sensitivity, rate of glucose absorption, etc.)
between two known conﬁgurations available in the simulator.
Non-stationary Recommender System: In this environment a recommender engine interacts with a user whose interests in different items ﬂuctuate over time.
In particular, the rewards associated with each item vary in seasonal cycles.
The goal is to maximize revenue by recommending an item that the user is most interested in at any time.
Non-stationary Goal Reacher: This is a 2D environment with four (left, right, up, and down) actions and a continuous state set representing the Cartesian coordinates.
The goal is to make the agent reach a moving goal position.
For all of the above environments, we regulate the speed of non-stationarity to characterize an algorithms’ ability to adapt.
Higher speed corresponds to a greater amount of nonstationarity; A speed of zero indicates that the environment is stationary.
We consider the following algorithms for comparison:
Prognosticator: Two variants of our algorithm, Pro-OLS and Pro-WLS , which use OLS and WLS estimators for Ψ .
ONPG: Similar to the adaptation technique presented by AlShedivat et al.
(2017), this baseline performs purely online optimization by ﬁne-tuning the existing policy using only the trajectory being observed online.
FTRL-PG:
Similar to the adaptation technique presented by Finn et al.
(2019), this baseline performs Follow-the-
(regularized)-leader optimization by maximizing performance over both the current and all the past trajectories.
9.1. Results
In the non-stationary recommender system, as the exact value of J ∗ k is available from the simulator, we can compute the true value of regret.
However, for the non-stationary goal reacher and diabetes treatment environment, as J ∗ k is not known for any k , we use a surrogate measure for regret.
That is, let ˜ J ∗ k be the maximum return obtained in episode k by any algorithm, then we use ( (cid:80) Nk
=
( ˜ J ∗ k −
J k ( π ))) /
( (cid:80) Nk
=
˜ J ∗ k ) as the surrogate regret for a policy π .
In the non-stationary recommender system, all the methods perform nearly the same when the environment is stationary.
FTRL-PG has a slight edge over ONPG when the environment is stationary as all the past data is directly indicative of the future MDP.
It is interesting to note that while FTRL-PG works the best for the stationary setting in the recommender system and the goal reacher task, it is not the best in the diabetes treatment task as it can suffer from high variance.
We discuss the impact of variance in later paragraphs.
With the increase in the speed of non-stationarity, performance of both the baselines deteriorate quickly.
Of the two, ONPG is better able to mitigate performance lag as it discards all the past data.
In contrast, both the proposed methods, Pro-OLS and Pro-WLS, can leverage all the past data to better capture the impact of non-stationarity and thus are consistently more robust to the changes in the environment.
In the non-stationary goal reacher environment, a similar trend as above is observed.
While considering all the past data equally is useful for FTRL-PG in the stationary setting, it creates drastic performance lag as the speed of the non-stationarity increases.
Between Pro-OLS and Pro-WLS, in the stationary setting, once the agent nearly solves the task all subsequent trajectories come from nearly the same distribution and thus the variance resulting from importance sampling ratio is not severe.
In such a case, where the variance is low, Pro-WLS has less advantage over Pro-OLS and additionally suffers from being biased.
However, as the nonstationarity increases, the optimal policy keeps changing and there is a higher discrepancy between distributions of past and current trajectories.
This makes the lower variance property of Pro-WLS particularly useful.
Having the ability to better capture the underlying trend, both Pro-OLS and Pro-WLS consistently perform better than the baselines when there is non-stationarity.
The non-stationary diabetes treatment environment is particularly challenging as it has a continuous action set.
This makes importance sampling based estimators subject to much higher variance.
Consequently, Pro-OLS is not able to reliably capture the impact of non-stationarity and performs similar to ONPG.
In comparison, the combination of both high variance and performance lag makes FTRL-PG perform poorly across all the speeds.
The most advantageous algorithm in this environment is Pro-WLS.
As it is designed to better tackle variance stemming from importance sampling, Pro-WLS is able to efﬁciently use the past data to capture the underlying trend and performs well across all the speeds of non-stationarity.
10. Conclusion
We presented a policy gradient-based algorithm that combines counter-factual reasoning with curve-ﬁtting to proactively search for a good policy for future MDPs.
Irrespective of the environment being stationary or non-stationary, the proposed method can leverage all the past data, and in nonstationary settings it can pro-actively optimize for future performance as well.
Therefore, our method provides a single solution for mitigating performance lag and being data-efﬁcient.
While the proposed algorithm has several desired properties, many open questions remain.
In our experiments, we noticed that the proposed algorithm is particularly sensitive to the value of the entropy regularizer λ .
Keeping λ too high prevents the policy from adapting quickly.
Keeping λ too low lets the policy overﬁt to the forecast and become close to deterministic, thereby increasing the variance for subsequent importance sampling estimates of policy performance.
While we resorted to hyper-parameter search, leveraging methods that adapt λ automatically might be fruitful (Haarnoja et al., 2018).
Our framework highlights new research directions for studying bias-variance trade-offs in the non-stationary setting.
While tackling the problem from the point of view of a univariate time-series is advantageous as the model-bias of the environment can be reduced, this can result in higher variance in the forecasted performance.
Developing lower variance off-policy performance estimators is also an active research direction which directly complements our algorithm.
In particular, often a partial model of the environment is available and using it through doubly-robust estimators (Jiang & Li, 2015; Thomas & Brunskill, 2016) is an interesting future direction.
Further, there are other forecasting functions, like kernel regression, Gaussian Processes, ARIMA, etc., and some break-point detection algorithms that can potentially be used to incorporate more domain knowledge in the forecasting function Ψ , or make Ψ robust to jumps and auto-correlations in the time series. 11. Acknowledgement
Part of the work was done when the ﬁrst author was an intern at Adobe Research, San Jose.
The research was later supported by generous gifts from Adobe Research.
We are thankful to Ian Gemp, Scott M. Jordan, and Chris Nota for insightful discussions and for providing valuable feedback.
Research reported in this paper was also sponsored in part by the CCDC Army Research Laboratory under Cooperative Agreement W911NF-17-2-0196 (ARL IoBT CRA).
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the Army Research Laboratory or the U S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. Appendix A. Properties of NIS and NWIS Estimators
Here we provide proofs for the properties of NIS and NWIS estimators.
While NIS and NWIS are developed for nonstationary setting, these properties ensure that these estimators strictly generalize for the stationary setting as well.
That is, when used in stationary setting, NIS estimator is both unbiased and consistent like the PDIS estimator, and NWIS estimator is biased and consistent like the WIS estimator; when NIS and NWIS are used in non-stationary setting they can provide more accurate estimates of a future performance as they explicitly model the trend of policy’s performance over time.
Our proof technique draws inspiration from the results presented by Mahmood et al.
The primary difference is that their results are for estimators that are developed only for the stationary setting and can not be readily used for the non-stationary setting, unlike ours.
The key modiﬁcation that we make to leverage their proof technique is that instead of using the features of the state as the input and the observed return from that corresponding state as the output to the regression function, we use the features of the time index of an episode as the input and the observed return for that corresponding episode as the output.
In their setup, as states are stochastically drawn from a distribution their analysis is not directly applicable for our setting, where inputs (time indices) form a deterministic sequence.
For analysis of our estimators, we leverage techniques discussed by Greene (2003) for analyzing properties of the ordinary least squares estimator.
Before proceeding, we impose the following constraints on the set of policies, and the basis function φ used for encoding the time index in both Ψ and Ψ † .
(a) φ ( · ) always contains 1 to incorporate a bias coefﬁcient in least-squares regression (for example, φ ( · ) =
[ φ 1 ( · ) , ..., φ d − 1 ( · ) , 1] , where φ i ( · ) are arbitrary functions).
(b) There exists a ﬁnite constant C 1 , such that ∀ i, | φ i ( · ) | < C 1 .
(c) Φ has full column rank such that ( [Equation] (cid:62) [Equation]
1 exists. (d) We only consider set of policies Π that have non-zero probability of taking any action in any state.
That is ∃ C 2 > 0 , such that ∀ π ∈ Π , ∀ s ∈ S , ∀ a ∈ A , π ( a | s ) >
C 2 .
Satisfying condition (a) is straightforward as it is already done by all basis functions.
Intuitively, this constraint ensure that the regression problem is not ill-deﬁned and there exists a model in the model class that can capture a ﬁxed constant, which corresponds to absence of any trend.
This is useful for our purpose as in the stationary setting, there exists no trend in the expected performance between the episodes for any given policy.
The forecaster should be capable to infer this fact from the data and represent it.
That is, the optimal model parameter is
[Equation]
such that for any k ,
[Equation]
Conditions (b) and (c) are also readily satisﬁed by popular basis functions.
For example, features obtained using Fourier basis is bounded by [ − 1 , 1] , and features from polynomial/identity basis are also bounded when inputs are adequately normalized.
Further, when the basis function does not repeat any feature, and number of samples are more than number of features, condition (c) is satisﬁed.
This ensures the the least-squares problem is well-deﬁned and has a unique-solution.
Condition (d) ensures that the denominator in any importance ratio is always bounded below, such that the importance ratios are bounded above.
This condition implies that the importance sampling estimator for any evaluation policy has ﬁnite variance.
Use of entropy regularization with common policy parameterizations (softmax/Gaussian) can prevent violation of this condition. all [Table] can
[Table]
for that environment as well
[Table] A.1. Finite Sample Properties
In this subsection, ﬁnite sample properties of NIS and NWIS are presented.
Speciﬁcally, it is established that NIS is an unbiased estimator, whereas NWIS is a biased estimator of J ( π ) , the performance of a policy π in a stationary MDP.
Theorem 1 (Unbiased NIS) .
For all δ ≥ 1 , ˆ J k + δ ( π ) is an unbiased estimator of J ( π ) , that is E [ ˆ J k
+ δ ( π )]
=
J ( π ) .
Proof.
Recall from (4) that
[Equation]
Therefore, the expected value of ˆ J k + δ ( π ) is
[Equation]
As Y =
[ ˆ J 0 ( π ) , ..., ˆ J k ( π )] (cid:62) and the MDP is stationary, expected value of each element of Y is J ( π ) .
Let [ J ( π )] denote the vector of size similar to Y , where all elements are J ( π ) , then
[Equation]
Now using (8) in (9),
[Equation]
Proof.
(Alternate)
Here we present an alternate proof for Theorem 1 which does not require invoking w ∗ .
[Equation]
where (a) is the dot product written as summation, and (b) holds because the multiplicative constants are ﬁxed values, as given in (6).
Since the environment is stationary, ∀ i E
[ Y i ] =
J ( π ) , therefore,
[Equation]
In the following we focus on the terms inside the summation in (10).
Without loss of generality, assume that the for a given matrix of features Φ , the feature corresponding to value 1 is in the last column of Φ .
Let A :=
(Φ (cid:62) Φ) − 1 Φ (cid:62) ∈ R
d × k , and let B := Φ[1 , ..., k ; 1 , ..., k − 1] ∈ R k × ( d − 1) be the submatrix of Φ such that B has all features of Φ except the ones column, 1 ∈ R k × 1 .
Let I be the identity matrix ∈ R k × k , then it can seen that (Φ (cid:62) Φ) [Equation] (Φ
[Equation]
Φ) can be expressed as,
[Equation]
In (11), as the j th row in last column of I corresponds to the dot product of the j th row of A , A j , with 1 ,
[Equation]
[Equation]
Equation (12) ensures that the summation of all rows of A , except the last, sum to 0 , and the last one sums to 1 .
Now, let φ ( k + δ ) :
= [ φ 1 ( k + δ ) , φ 2 ( k + δ ) , ..., φ d − 1 ( k + δ ) , 1] ∈ R 1 × d .
Therefore,
[Equation]
[Equation]
Therefore, combining (13) with (10), E (cid:104)
ˆ J k
+ δ ( π ) (cid:105) =
J ( π ) .
Theorem 2 (Biased NWIS) .
For all δ ≥ 1 , ˆ J † k + δ ( π ) may be a biased estimator of J ( π ) , that is, E
[ ˆ J † k
+ δ ( π )]
(cid:54) =
J ( π ) always.
Proof.
We prove this result using a simple counter-example.
Consider the following basis function, φ ( · ) =
[1] , then,
[Equation]
which is the WIS estimator.
Therefore, as WIS is a biased estimator, NWIS is also a biased estimator of J ( π ) .
A.2. Large Sample Properties
In this subsection, large sample properties of NIS and NWIS are presented.
Speciﬁcally, it is established that both NIS and
NWIS are consistent estimators of, J ( π ) , the performance of a policy π in a stationary MDP.
Theorem 3 (Consistent NIS) .
For all δ ≥ 1 , ˆ J k + δ ( π ) is a consistent estimator of J ( π ) , that is as N → ∞ , ˆ J N + δ ( π ) a.s. −→
J ( π ) .
Proof.
Using (4),
[Equation]
As Y =
[ ˆ J 0 ( π ) , ...
[Equation]
ˆ J N ( π )] (cid:62) and the MDP is stationary, each element of Y is an unbiased estimate of J ( π ) .
In other words, ∀ i ∈ [0 , k ] , ˆ J i ( π ) =
J ( π ) + (cid:15) i , where (cid:15) i is a mean zero error.
Let (cid:15) ∈ R N × 1 be the vector containing all the error terms (cid:15) i .
Now using (8),
[Equation] [Equation]
Using Slutsky’s Theorem,
[Equation]
where Q = (cid:16) lim N →∞ 1 N Φ (cid:62) Φ (cid:17) , and it holds from Grenander’s conditions that Q − 1 exists.
Informally, Grenander’s conditions require that no feature degenerates to a sequence of zeros, no feature of a single observation dominates the sum of squares of its series, and the Φ (cid:62) Φ matrix always has full rank.
These conditions are easily satisﬁed for most popular basis functions used to create input features.
For formal deﬁnitions of these conditions, we refer the reader to Chpt.
5, Greene (2003).
In the following, we restrict our focus to the term inside the brackets in the second term of (15).
The mean of that term is,
[Equation]
Since the mean is 0 , variance of that term is given by,
[Equation]
As each policy has a non-zero probability of taking any action in any state, the variance of PDIS (or the standard IS) estimator is bounded and thus each element of (cid:15)(cid:15) (cid:62) is bounded.
Further, as φ i ( · ) is bounded, each element of Φ is also bounded.
Therefore, as
[Equation]
As mean is 0 and variance asymptotes to 0 , then as N → ∞ , 1 N Φ (cid:62) (cid:15) a.s. −→ 0 .
Combining this with (15),
[Equation]
Theorem 4 (Consistent NWIS) .
For all δ ≥ 1 , ˆ J † k + δ ( π ) is a consistent estimator of J ( π ) , that is as N → ∞ , ˆ J † N + δ ( π ) a.s. −→
J ( π ) .
Proof.
Recall from (7),
[Equation]
Consistency of ˆ J † N + δ ( π ) can be proved similarly to the proof of Theorem 3 .
Note that over here Y =
[ G 0 , ..., G k ] (cid:62) contains the returns for each episode, Λ Y denotes the unbiased estimates for J ( π ) .
Therefore, similar to (14),
[Equation]
Using Slutsky’s Theorem
[Equation]
Now restricting our focus to the terms in the ﬁrst bracket in (16),
[Equation]
where to obtain (a), let ˜ ρ k := ρ † k − E
[ ρ † k ] be a mean 0 random variable.
Then (1 /N )
(cid:80) ρ † k φ ( k ) (cid:62) φ ( k ) =
(1 /N )
( (cid:80) Nk
=1 ˜ ρ k φ ( k ) (cid:62) φ ( k ) + E [ ρ † k ] (cid:80) Nk
=1 φ ( k ) (cid:62) φ ( k )) .
Now using the strong law for weighted sums of independent random variables (Theorem 1.1, Cuzick (1995)) and under the condition that both ρ † and φ ( · ) are bounded, (1 /N )
(cid:80) Nk
=1 ˜ ρ k φ ( k ) (cid:62) φ ( k ) a.s. → 0 in limit.
Therefore, (1 /N )
(cid:80) ρ † k φ ( k ) (cid:62) φ ( k ) =
E [ ρ † k ]((1 /N )
(cid:80) Nk
=1 φ ( k ) (cid:62) φ ( k )) in limit as N → ∞ .
Consequently, (b) is obtained using the fact that the expected value of importance ratios is 1 (Lemma 3, (Thomas, 2015)).
Substituting (17) in (16),
[Equation]
[Equation]
where (18) follows from simpliﬁcation used for (15) in the proof of Theorem 3 .
B. Gradient of PDIS Estimator
Recall that NIS and NWIS estimators build upon PDIS and WIS estimators by using them along with OLS and WLS regression, respectively.
Consequently, gradients of NIS and NWIS estimators with respect to the policy parameters can be decomposed into terms which consists of gradients of PDIS and WIS estimators with respect to the policy parameters.
Here we provide complete derivations for obtaining a straightforward equation for computing the gradients of the PDIS and the WIS estimators with respect to the policy parameters.
These might also be of independent interest when dealing with off-policy policy optimization in purely stationary MDPs.
Property 1 (PDIS Gradient) .
Let ρ i (0 , l ) :
=
(cid:81)
l j =
π θ ( A j i |
S
j i ) β
i ( A ji | S ji ) , then
[Equation]
i ( θ
[Equation]
T (cid:88) t = ∂ log π θ
( A ti | S ti )
∂θ (cid:32) T (cid:88) l = t
ρ i (0 , l ) · γ l · R li (cid:33) .
[Table]
Proof.
Recall from (2),
[-( A li | S li ) (cid:33) ∂ log (cid:16)(cid:81) tl =
 π θ ( A li | S li ) (cid:17) ∂θ γ t R ti = T (cid:88) t = (cid:32) t (cid:89) l = π θ ( A li | S li ) β-][Equation]
Computing the gradient of ˆ J i ( θ ) ,
[Equation]
(19) where, in the last step, instead of the summation over partial derivatives of log π θ for each weight ρ ( · , · ) , we consider the alternate form where summation is over the importance weights ρ ( · , · ) for each partial derivative of log π θ .
To see this step clearly, let Ψ ti =
∂ log π θ
( A ti | S ti ) /∂θ , then Table 1 shows all the terms in (19).
The last step in simpliﬁcation corresponds to taking column-wise sum instead of row wise sum in Table 1.
C. Detailed Literature Review
The problem of non-stationarity has a long history.
In the operations research community, many dynamic sequential decisionmaking problems are modeled using inﬁnite horizon non-homogeneous MDPs (Hopp et al., 1987).
While estimating an optimal policy is infeasible under an inﬁnite horizon setting when the dynamics are changing and a stationary distribution cannot be reached, researchers have been interested in identifying sufﬁcient forecast horizons for performing near-optimal planning (Garcia & Smith, 2000; Cheevaprawatdomrong et al., 2007; Ghate & Smith, 2013) or robust policy iteration (Sinha & Ghate, 2016).
In contrast, non-stationary multi-armed bandits (NMAB) capture the setting where the horizon length is one, but the reward distribution changes over time (Moulines, 2008; Besbes et al., 2014).
Many variants of NMAB, like cascading non-stationary bandits (Wang et al., 2019b; Li & de Rijke, 2019) and rotten bandits (Levine et al., 2017; Seznec et al., 2018) have also been considered.
In optimistic online convex optimization, researchers have shown that better performance can be achieved by updating the parameters using predictions of the gradient of the future loss, based on past gradients (Rakhlin &
Sridharan, 2013; Yang & Mohri, 2016; Mohri & Yang, 2016; Wang et al., 2019a).
Non-stationarity also occurs in multi-player games, like rock-paper-scissors, where each episode is a single one-step interaction (Singh et al., 2000; Bowling, 2005; Conitzer & Sandholm, 2007).
Opponent modeling in games has been shown to be useful and regret bounds for multi-player games where players can be replaced with some probability p , i.e., the game changes slowly over time, has also been established (Zhang & Lesser, 2010; Mealing & Shapiro, 2013; Foster et al., 2016;
Foerster et al., 2018).
However, learning sequential strategies in a non-stationary setting is still an open research problem.
For episodic non-stationary MDPs, researchers have also looked at providing regret bounds for algorithms that exploit oracle access to the current reward and transition functions (Even-Dar et al., 2005; Yu & Mannor, 2009;
Abbasi et al., 2013;
Lecarpentier & Rachelson, 2019; Li et al., 2019).
Alleviating oracle access by performing a count-based estimate of the reward and transition function based on the recent history of interactions has also been proposed (Gajane et al., 2018; Cheung et al., 2019).
For tabular MDPs, past data from a non-stationary MDP can be used to construct a maximum-likelihood estimate model (Ornik & Topcu, 2019) or a full Bayesian model (Jong & Stone, 2005) of the transition dynamics.
Our focus is on the setting which is not restricted to tabular representations.
A Hidden-Mode MDP is an alternate setting that assumes that the environment changes are conﬁned to a small number of hidden modes, where each mode represents a unique MDP.
This provides a more tractable way to model a limited number of
MDPs (Choi et al., 2000; Basso & Engel, 2009; Mahmud & Ramamoorthy, 2013), or perform model-free updates using mode-change detection (Padakandla et al., 2019).
In this work, we are interested in the continuously changing setting, where the number of possible MDPs is unbounded.
Tracking has also been shown to play an important role in non-stationary domains.
Thomas et al.
(2017) and Jagerman et al.
(2019) have proposed policy evaluation techniques in a non-stationary setting by tracking a policy’s past performances.
However, they do not provide any procedure for searching for a good future policy.
To adapt quickly in non-stationary tasks, TIDBD (Kearney et al., 2018) and AdaGain (Jacobsen et al., 2019) perform TD-learning while also automatically (de-)emphasizing updates to (ir)relevant features by modulating the learning rate of the parameters associated with the respective features.
Similarly, Abdallah & Kaisers (2016) propose repeating a Q-value update inversely proportional to the probability with which an action was chosen to obtain a transition tuple.
In this work, we aim at going beyond tracking and pro-actively optimizing for the future.
D. Implementation details D.1. Environments
We provide empirical results on three non-stationary environments: diabetes treatment, recommender system, and a goal-reacher task.
Details for each of these environments are provided in this section.
Non-stationary Diabetes Treatment: This MDP models the problem of Type-1 diabetes management.
At a high-level, the body of a person with Type-1 diabetes does not produce enough insulin , a hormone that promotes absorption of glucose from the blood.
Consumption of a meal increases the blood-glucose level in the body and if the blood-glucose level becomes too high, then the patient can suffer from Hyperglycemia .
Insulin injections can be used externally to reduce the level but if the level becomes too low, then the patient suffers from Hypoglycemia .
While either of the extremes is undesirable,
Hypoglycemia is more dangerous and can triple the ﬁve-year mortality rate for a person with diabetes (Man et al., 2014).
Autonomous medical support system have been proposed to decide how much insulin should be injected to keep a person’s blood glucose levels near ideal levels (Bastani, 2014).
The parameters of such a medical support system are set by a doctor speciﬁcally for each patient.
However, due to non-stationarities induced over time as a consequence of changes in the body mass index, the insulin sensitivity of the pancreas, diet, etc., the parameters of the controller need to be re-adjusted regularly.
Currently, this requires re-visiting the doctor.
A viable reinforcement learning solution to this non-stationary problem could enable the automatic tuning of these parameters for patients without regular access to a physician.
To model this MDP we use an open-source implementation (Xie, 2019) of the U.S. Food and Drug Administration (FDA)
approved Type-1 Diabetes Mellitus simulator (T1DMS) (Man et al., 2014) for treatment of Type-1 diabetes, where we induce non-stationarity by oscillating the body parameters between two known conﬁgurations.
Each episode consists of a day ( 1440 timesteps, where each timestep corresponds to a minute) in an in-silico patient’s life and the transition dynamics of a patient’s body for each second is governed by a continuous time ordinary differential equation (ODE) (Man et al., 2014).
After each second the insulin controller is used to inject the desired amount of insulin for controlling the blood glucose.
For controlling the insulin injection, we use a parameterized policy based on the amount of insulin that a person with diabetes is instructed to inject prior to eating a meal (Bastani, 2014):
[Equation]
where ‘current blood glucose’ is the estimate of the person’s current blood glucose level, ‘target blood glucose’ is the desired blood glucose, ‘meal size’ is the estimate of the size of the meal the patient is about to eat, and CR and CF are two real-valued parameters, that must be tuned based on the body parameters to make the treatment effective.
Non-stationary Recommender System: During online recommendation of movies, tutorials, advertisements and other products, the system needs to interact and personalize for each user.
However, the interests of an user for different items among the products that can be recommended ﬂuctuate over time.
For example, interests during online shopping can vary based on seasonality or other unknown factors.
This environment models the desired recommender system setting where reward (interest of the user) associated with each item changes over time.
Figure 6 (left) plots how the reward associated with each item changes over time, for each of the considered ‘speeds’ of non-stationarity.
The goal for the reinforcement learning agent is to maximize revenue by recommending the item which the user is most interested at any time.
Non-stationary Goal Reacher: For an autonomous robot dealing with tasks in the open-world it is natural for the problem speciﬁcation to change over time.
An ideal system should quickly adapt to the changes and still complete the task.
To model the above setting, this environment considers a task of reaching a non-stationary goal position.
That is, the location of the goal position keeps slowly moving around with time.
The goal of the reinforcement learning agent is to control the four (left, right, up, and down) actions to move the agent towards the goal as quickly as possible given the real valued
Cartesian coordinates of the agent’s current location.
Maximum time given to the agent to reach the goal is 15 steps.
D.2. Hyper-parameters
For both the variants of the proposed Prognosticator algorithms, we use Fourier basis to encode the time index while performing (ordinary/weighted) least squares estimation.
Since Fourier basis requires inputs to be normalized with | x | ≤ 1 , we normalize each time index by dividing it by K + δ , where K is the current time and δ is the maximum time in future we need the forecast for.
Further, as we are regressing only on time (which are all positive values), is does not matter whether the function of policy performance over time is odd ( Ψ( x ) = − Ψ
( − x ) ) or not.
Therefore, we drop all the terms in the basis corresponding to sin( · ) which are useful for modeling odd functions.
This reduces the number of parameters to be estimated by half.
Finally, instead of letting n ∈ N , we restrict it to a ﬁnite set { 1 , ..., d − 1 } , where d is a ﬁxed constant that determines the size of feature vector for each input.
In all our experiments, d was a hyper-parameter chosen from { 3 , 5 , 7 } .
Other hyper-parameter ranges were common for all the algorithms.
The discounting factor γ was kept ﬁxed to 0 .
99 and learning rate η was chosen from range [5 e − 5 , 5 e − 2] .
Entropy regularizer λ was chosen from range [0 , 1 e − 2]
Batch size δ was chosen from the set { 1 , 3 , 5 } .
Inner optimization over past data for the proposed methods and FTRL-PG was run for { 10 , 20 , 30 } × δ iterations.
Inner optimization for ONPG corresponds to one iteration over all the trajectories collected in the current batch.
Past algorithms have shown that clipping the importance weights can improve stability of reinforcement learning algorithms (Schulman et al., 2017).
Similarly, we clip the maximum value of the importance ratio to a value chosen from { 5 , 10 , 15 } .
Additionally, value functions learned online is popularly used as a baseline/control-variate in policy gradient based algorithms.
Whether to use of such a value function or not was decided from { True, False } .
As the non-stationary diabetes treatment problem has a continuous action space, the policy was parameterized with a Gaussian distribution having a variance chosen from [0 . 5 , 2 . 5] .
For the non-stationary goal-reacher environment, the policy was parameterized using a two-layer neural network with number of hidden nodes chosen from { 16 , 32 , 64 } .
In total, 2000 settings for each algorithm, for each domain, were uniformly sampled (loguniformly for learning rates and λ ) from the mentioned hyper-parameter ranges/sets.
Results from the best performing setting is reported in all the plots.
Each hyper-parameter setting was ran using 10 seeds for the non-stationary diabetes treatment (as it was time intensive to run a continuous time ODE for each step) and 30 seeds for the other two environments to get the standard error of the mean performances.
The authors had shared access to a computing cluster, consisting of 50 compute nodes with 28 cores each, which was used to run all the experiments.
E. Detailed Empirical Results
Complexity analysis (space, time, and sample size) Space requirement for our algorithms and FTRL is linear in the number of episodes seen in the past, whereas it is constant for ONPG as it discards all past data.
Computational cost of our algorithm is also similar to FTRL as the only additional cost is that of differentiating through least-squares estimators which involves computing (Φ (cid:62) Φ) − 1 or (Φ (cid:62) ΛΦ) − 1 .
This additional overhead is negligible as these matrices are of size d × d , where d is the size of feature vector for time index and d << N , where N is the number of past episodes.
Figures 5, 6, and 7 present an empirical estimate of sample efﬁciency.
Performance over time: In Figure 5, summary statistics of the results were presented.
In this section we present all the results in details.
Figure 6 plots the performances of all the algorithms for individual episodes as the user interests changes over time in the recommender system environment.
In this environment as the true rewards for each of the item is directly available we provide a visual plot for it as well in Figure 6 (left).
Notice the shape of the performance curve for the proposed methods, which closely captures the trend of maximum reward attainable over time.
Figure 7 plots the performances of all the algorithms for non-stationary goal-reacher and diabetes treatment environments.
In these environments, the maximum achievable performance for each episode is not readily available.