D EEP Q-L EARNING WITH L OW S WITCHING C OST
Anonymous authors
Paper under double-blind review
A BSTRACT
We initiate the study on deep reinforcement learning problems that require low switching cost, i.e., a small number of policy switches during training.
Such a requirement is ubiquitous in many applications, such as medical domains, recommendation systems, education, robotics, dialogue agents, etc, where the deployed policy that actually interacts with the environment cannot change frequently.
Our paper investigates different policy switching criteria based on deep Q-networks and further proposes an adaptive approach based on the feature distance between the deployed Q-network and the underlying learning Q-network.
Through extensive experiments on a medical treatment environment and a collection of the
Atari games, we ﬁnd our feature-switching criterion substantially decreases the switching cost while maintains a similar sample efﬁciency to the case without the low-switching-cost constraint.
We also complement this empirical ﬁnding with a theoretical justiﬁcation from a representation learning perspective.
I NTRODUCTION
Reinforcement learning (RL) is often used for modeling real-world sequential decision-making problems such as medical domains, personalized recommendations, hardware placements, database optimization, etc.
For these applications, oftentimes it is desirable to restrict the agent from adjusting its policy frequently.
In medical domains, changing a policy requires a thorough approval process by experts.
For large-scale software and hardware systems, changing a policy requires to redeploy the whole environment.
Formally, we would like our RL algorithm admits a low switching cost .
In this setting, it is required that the deployed policy that interacts with the environment cannot change many times.
In some real-world RL applications such as robotics, education, and dialogue system, changing the deployed policy frequently may cause high costs and risks.
Gu et al.
(2017) trained robotic manipulation by decoupling the training and experience collecting threads; Mandel et al.
(2014) applied RL to educational games by taking a data-driven methodology for comparing and validating policies ofﬂine, and run the strongest policy online; Jaques et al.
(2019) developed an off-policy batch RL algorithms for dialog system, which can effectively learn in an ofﬂine fashion, without using different policies to interact with the environment.
All of these work avoid changing the deployed policy frequently as they try to train the policy ofﬂine effectively or validate the policy to determine whether to deploy it online.
For RL problem with a low switching cost constraint, the central question is how to design a criterion to decide when to change the deployed policy .
Ideally, we would like this criterion to have the following four properties:
Low switching cost: This is the purpose of this criterion.
An algorithm equipped with this policy switching criterion should have low switching cost.
High Reward: Since the deployed policy determines the collected samples and the agent uses fewer deployed policies, the collected data may not be informative enough to learn the optimal policy with high reward.
We need this criterion to deploy policies that can collect informative samples.
Sample Efﬁciency: Since the agent only uses a few deployed policies, there may be more redundant samples, which will not be collected if the agent switches the policy frequently.
We would like algorithms equipped with a criterion with similar sample efﬁciency as the case without the low switching cost constraint.
Generality: We would like this criterion to be effective not only on a speciﬁc task but also broadly effective on a wide range of tasks.
In this paper, we take a step toward this important problem.
We focus on designing a principled policy switching criterion for deep Q-networks (DQN) learning algorithms, which have been widely used in applications.
For example, Ahn & Park (2020) apply DQN to control balancing between different HVAC systems, Ao et al.
(2019) propose a thermal process control method based on DQN, and Chen et al.
(2018) try to apply it to online recommendation.
Notably these applications all require low switching cost.
Our paper conducts a systematic study on DQN with low switching cost.
Our contributions are summarized below.
Our Contributions
• We conduct the ﬁrst systematic empirical study on benchmark environments that require modern reinforcement learning algorithms.
We test two naive policy switching criteria: 1) switching the policy after a ﬁxed number of steps and 2) switching the policy after an increasing step with a ﬁxed rate.
We ﬁnd that neither criterion is a generic solution because sometimes they either cannot ﬁnd the best policy or signiﬁcantly decrease the sample efﬁciency.
• Inspired by representation learning theory, we propose a new feature-based switching criterion that uses the feature distance between the deployed Q-network and the underlying learning Q-network.
Through extensive experiments, we ﬁnd our proposed criterion is a generic solution – it substantially decreases the switching cost while maintains a similar performance to the case without the low-switching-cost constraint.
• Along the way, we also derive a deterministic Rainbow DQN (Hessel et al., 2018), which may be of independent interest.
Organization This paper is organized as follows.
In Section 2, we review related work.
In Section 3, we describe our problem setup and review necessary backgrounds.
In Section 4, we describe deterministic Rainbow DQN with the low switching cost constraint.
In Section 5, we introduce our feature-based policy switching criterion and its theoretical support.
In Section 6, we conduct experiments to evaluate different criteria.
We conclude in Section 7 and leave experiment details to appendix.
R ELATED W ORK
Low switching cost algorithms were ﬁrst studied in the bandit setting (Auer et al., 2002; CesaBianchi et al., 2013).
Existing work on RL with low switching cost is mostly theoretical.
To our knowledge, Bai et al.
(2019) is the ﬁrst work that studies this problem for the episodic ﬁnite-horizon tabular RL setting.
Bai et al.
(2019) gave a low-regret algorithm with an O (cid:0) H 3 SA log ( K ) (cid:1) local switching upper bound where S is the number of stats, A is the number of actions, H is the planning horizon and K is the number of episodes the agent plays.
The upper bound was improved in Zhang et al.
(2020b;a).
The only empirical study on RL with switching cost is Matsushima et al.
(2020), which proposed a concept of deployment efﬁciency and gave a model-based algorithm.
During the training process, the algorithm ﬁxes the number of deployments, trains a dynamics model ensemble, and updates the deployed policy alternately.
After each deployment, the deployed policy collects transitions in the real environment to enhance the models, and then the models optimize the policy by providing imagined trajectories.
In other words, they reduce the number of deployments by training on simulated environments.
Our goal is different: we design a criterion to decide when to change the deployed policy, and this criterion could be employed by model-free algorithms.
There is a line of work on ofﬂine RL (also called Batch RL) methods, where the policy does not interact with the environment directly and only learns from a ﬁxed dataset (Lange et al., 2012; Levine et al., 2020).
Some methods Interpolate ofﬂine and online methods, i.e., semi-batch RL algorithms (Singh et al., 1995; Lange et al., 2012), which update the policy many times on a large batch of transitions.
However, the switching cost is not their focus.
P RELIMINARIES 3. M ARKOV D ECISION P ROCESS
Throughout our paper, we consider the episodic Markov decision model ( S , A , H, P, r) .
In this model, S is the state space,
A is the action space,
H ∈ Z + is the planning horizon.
P is the transition operator where P ( x (cid:48)
| x, a ) denotes the transition probability of taking action a from state x to state x (cid:48) .
r : S ×A → R is the reward function.
A policy is a mapping from a state to an action, π : S → A .
In this paper, we focus on deterministic policies as required by motivating applications.
The dynamics of the episodic MDP can be view as the interaction of an agent with the environment periodically.
We let K be the total number of episodes the agent plays.
At the beginning of an episode k ∈ [ K ] , the agent chooses a policy π k .
The initial x k 1 ∈ S is sampled from a distribution, and the agent is then in step 1 .
At each step h ∈
[ H ] in this episode, based on the current state x kh ∈ S the agent chooses the action a kh = π k ( x kh ) .
The environment will give the reward for the step r ( x kh , a kh ) and move the agent to the next state x h +1 ∼ P ( ·| x kh , a kh ) .
The episode automatically ends when the agent reaches the step H + 1 .
Q -function is used to evaluate the long-term value for the action a and subsequent decisions.
The
Q -function of a policy π at time step h is deﬁned as follows:
[Equation]
The goal of this agent is to ﬁnd a policy π ∗ which maximizes the expected reward, π ∗ = arg max π E π (cid:104)(cid:80)
Hh =1 r h (cid:105) .
Ideally, we want to use a few episodes ( K ) as possible to learn π ∗ .
3. S WITCHING C OST
The concept of switching cost is used to quantify the adaptability of RL algorithms.
The switching cost is deﬁned as the number of policy changes of deployed policies in the running of the algorithm in K episodes, namely:
[Equation]
The goal of this paper is to equip algorithm with a criterion that learns π ∗ using a few episodes while at the same time has small N switch .
3. D EEP Q -L EARNING
If we can estimate Q -function for each state and action pair well, there is no difﬁculty in ﬁnding π ∗ .
For example, we can always select a ∗ =
arg max a Q ( x, a ) .
However, it is not easy to learn Q value estimates for each state and action pair, especially when the state or the action space is large.
In deep Q -learning (DQN), Mnih et al. (2015) combine deep networks and reinforcement learning successfully by using a deep neural network to approximate Q ( x, a ) .
Given the current state x h , the agent selects a action a h greedily based on the Q ( x h , a ) , then the state move to x h +1 and a reward r t +1 is obtained.
The transition ( x t , a t , r t +1 , x t +1 ) is saved to the replay memory buffer.
At each time, a batch of transitions is sampled from this buffer, and the parameters of neural networks are optimized by using stochastic gradient descent to minimize the loss
[Equation]
where γ h +1 is the chosen discount of time step h
+ 1 , θ is the parameters of the online network, and ¯ θ represents the parameters of the target network.
The gradient of the loss is back-propagated only to update θ , while ¯ θ is not optimized directly.
DQN has been successful as it led to superhuman performance on several Atari games.
Nevertheless, there are also several limitations of this algorithm and many extensions have been proposed.
Rainbow (Hessel et al., 2018) combines six of these extensions and obtains an excellent and stable performance on many Atari games.
In the
Appendix A, we review these tricks.
In this paper, we focus on Rainbow DQN with low switching cost.
3. C OUNT -B ASED E XPLORATION
In many real-world scenarios that require low switching cost, it is also required to use deterministic policies, especially in applications mentioned above.
Exploring strategies like (cid:15) -greedy and noisy net make the policy stochastic, which we cannot use here.
Count-based exploration algorithms are known to perform near-optimally when used in reinforcement learning for solving tabular MDPs.
Tang et al.
(2017) applied these algorithms into high-dimensional state spaces successfully.
They discretized the state space with a hash function φ : S → Z .
Then an exploration bonus r + ( x ) = β √ n ( φ ( x )) is added to the reward function and the agent is trained with the modiﬁed reward r + r + .
Note that with the count-based exploration, the policy is deterministic.
D ETERMINISTIC R AINBOW DQN WITH A P OLICY S WITCHING C RITERION
In this section, we ﬁrst introduce how to implement a deterministic Rainbow DQN with a policy switching criterion.
This implementation combines six DQN tricks and the policy is always deterministic.
4. D ETERMINISTIC R AINBOW DQN
We ﬁrst discuss how to make Rainbow DQN always output deterministic policies.
Recall that to explore more efﬁciently, Rainbow adopts the Noisy Net (Fortunato et al., 2018), which makes the policy stochastic.
To obtain a deterministic policy and keep exploring, we remove Noisy Net and employ the Count-Based exploration described in the previous section.
When the deployed policy interacts with the environment, it selects the action which maximizes the Q -function.
After obtaining the reward by taking this action, an exploration bonus r + = β √ n ( φ ( x )) is added into the reward.
4. DQN WITH A P OLICY S WITCHING C RITERION
Besides, Rainbow updates the policies which interacts with the environment directly.
These policies usually switch millions of times during the training process (since the parameters are updated millions of times).
In our implementation, we add an online policy in addition to the deployed policy that interacts with the environment.
The deployed policy collects the data for the experience replay buffer, while the online policy is frequently updated when training.
The deployed policy is replaced with the online policy when they meet the criterion we will discuss soon.
The algorithm of deterministic switching Rainbow is shown in Algorithm 1.
4. P OLICY S WITCHING C RITERIA
Here we ﬁrst introduce two straightforward policy switching criteria.
Fixed Interval Switching This is the simplest criterion, we switch the deployed policy with a ﬁxed interval.
Under the FIX n criterion, we switch the deployed policy whenever the online policy is updated n times where n is a per-speciﬁed number.
We will specify n in our experiments.
Adaptive Interval Switching This criterion aims to switch the deployed policy frequently at the ﬁrst and reduce the switching speed gradually.
Under the Adaptive n2m criterion, we increase the deployment interval from n to m .
The interval between the i -th deployment and the ( i + 1) -th deployment is min(( i + 1) × n, m ) .
We will specify n and m in our experiments.
Fixed interval switching FIX n input n (accumulated updates) output bool ( n (accumulated updates) ≥ n ) (cid:46)
Adaptive interval switching Adaptive n2m input n (accumulated updates) , n (deployment) output bool ( n (accumulated updates) ≥ min( n (deployment) + 1) × n, m ) (cid:46) Feature-based switching F a input f deployed , f online , D
Sample minibatch B of transitions ( x h ) from D with probability p h
Compute the the similarity sim ( x h ) =
[Equation]
( x h )
[Equation]
|
Compute the average similarity in sampled batch sim ( B ) =
(cid:80) x ∈ B sim ( x ) || B || output bool ( sim ( B ) ≤ a )
These two criteria and our proposed new criterion are summarized in Algorithm 2.
Unfortunately, as will be shown in our experiments, these two criteria do not perform well.
This requires us to design new principled criterion.
F EATURE - BASED S WITCHING C RITERION
In the section, we describe our new policy switching criterion based on feature learning.
We ﬁrst describe this criterion, and then we provide some theoretical justiﬁcation from a representation learning point of view.
Feature-based Switching Criterion We adopt the view the DQN learns to extract informative features of the states of environments.
Our proposed criterion tries to switch the deployed policy according to the extracted feature.
When deciding whether to switch the deployed policy or not, we ﬁrst sample a batch of states B from the experience replay buffer, and then extract the feature of all states by both the deployed deep Q-network and online deep Q -network.
For a state x , the extracted feature are denoted as f deployed ( x ) and f online ( x ) , respectively.
The similarity score between f deployed and f online on state x is deﬁned as
[Equation]
We then compute the averaged similarity score on the batch of states B
[Equation]
With a hyper-parameter a ∈ [0 , 1] , the feature-based policy switching criterion is to change the deployed policy whenever sim ( B ) ≤ a .
Theoretical Justiﬁcation Our criterion is inspired by representation learning.
To illustrate the idea, we consider the following setting.
Suppose we want to learn f ( · ) , a representation function that maps the input to a k -dimension vector.
We assume we have input-output pairs ( x, y ) with y = (cid:104) w, f ∗ ( x ) (cid:105) for some underlying representation function f ∗ ( · ) and a linear predictor w ∈ R k .
For ease of presentation, let us assume we know w , and our goal is to learn the underlying representation which together with w gives us 0 prediction error.
Suppose we have data sets D 1 and D 2 .
We use D 1 to train an estimator of f ∗ , denoted as f 1 , and
D 1 ∪ D 2 to train another estimator of f ∗ , denoted as f 1+2 .
The training method is empirical risk minimization, i.e.,
[Equation]
where F is some pre-speciﬁed representation function class.
The following theorem suggests if the similarity score between f 1 and f 1+2 is small, then f 1 is also far from the underlying representation f ∗ .
Theorem 1.
Suppose f 1 and f 1+2 are trained via aforementioned scheme.
There exist dataset D 1 ,
D 2 , function class F and w such that if the similarity score between f 1 and f 1+2 on D 1+2 is smaller than α , then the prediction error of f 1 on D 1+2 is 1 − α .
The proof is deferred to Appendix B where we give explicitly constructions.
Theorem 1 suggests that in certain scenarios, if the learned representation has not converged (the similarity score is small), then it cannot be the optimal representation which in turn will hurt the prediction accuracy.
Therefore, if we ﬁnd the similarity score is small, we should change the deployed policy.
E XPERIMENTS
In this section, we conduct experiments to evaluate different policy switching criteria on DQN.
We study several Atari game environments along and an environment for simulating sepsis treatment for
ICU patients.
We evaluate the efﬁciency among different switching criteria in these environments.
Implementation details and hyper-parameters are listed in the Appendix A.
6. E NVIRONMENTS
GYMIC GYMIC is an OpenAI gym environment for simulating sepsis treatment for ICU patients to an infection, where sepsis is caused by the body’s response to an infection and could be lifethreatening.
GYMIC built an environment to simulate the MIMIC sepsis cohort, where MIMIC is an open patient EHR dataset from ICU patients.
This environment generates a sparse reward, the reward is set to +15 if the patient recovered and -15 if the patient died.
This environment hasclinical features and a 5 × 5 action space.
For the GYMIC, we display the learning curve of 1. million steps of the environment, after which the reward converge.
We choose this environment because it is simulating a real-world problem that requires low switching cost.
For Atari games, all the experiments were training for the environment stepping 3.5 million times.
Atari 2600 Atari 2600 games are widely employed to evaluate the performance of DQN based agents.
We also evaluate the efﬁciency among different switching criteria on several games, such as
Pong, Road Runner, Beam Rider, etc.
6. R ESULTS AND D ISCUSSIONS
For all environments, we evaluate a feature-based criterion F 0 .
98 , three ﬁxed interval criteria covering a vast range FIX 10 2 , FIX 10 3 and FIX 10 4 , and an adaptive criterion increasing the deploying interval from 100 to 10,000.
Besides the switching criteria we discussed above, we use “None” to indicate an experiment without the low-switching-cost constraint where deployed policy kept in sync with online policy all the time, notice that this experiment is equivalent to FIX 1 .
The ﬁgure above is the learning cure of reward, while the ﬁgure below displays switching are
GYMIC As shown in ﬁgure 1, none of the switching criteria affects the performance, but they can reduce the number of policy switches drastically, which indicates that reducing the switching cost in such a medical environment could be feasible.
In particular, we ﬁnd FIX 10and F 0 .
98 are the two best criteria to reduce switching cost.
Finally, this criterion keeps a good performance with the minimal switching cost.
Atari 2600 GYMIC may be too simple, and we should compare the performances among different criteria in some more difﬁcult environments.
We evaluate the performance of different criteria when playing the Atari games, which are imagebased environments.
In particular, the state space is much more complex.
Figure 2 shows the results of 6 Atari games.
In each subgraph, the upper curves are about the rewards of steps, while the curves blow is about switching cost.
First we observe that overall trend, higher switching cost leads to better performance.
In general
Rainbow DQN with no switching cost constrain often gives the best performance.
Also, FIX 10enjoys better performance than FIX 10 3 and
FIX 10 4 .
In some games such as Qbert and
Riverraid, although FIX 10 4 and Adaptive lead to lower switching cost, they fail to learn how to play the games well.
Therefore, they are not desired generic policy switching criterion.
Secondly, we observe that changing the policy with an adaptive interval may be better than a ﬁxed interval.
Focusing on the criteria FIX 10 4 and Adaptive 10 2 to 10 4 , the adaptive criterion switches the online policy fast at ﬁrst and decrease its switching speed gradually, and in the end, the adaptive criterion would have the same speed as the ﬁxed one.
Therefore, there is no signiﬁcant difference between the total switching cost of these two criteria.
However, we could observe that the adaptive criterion’s performance is better than the ﬁxed one when playing Beam Rider, Pong, Qbert, and they obtain similar performances in the rest games.
Lastly, we ﬁnd our proposed feature-based criterion ( F 0 . 98 ) is the desired on that satisfy all four properties we discussed in Section 1.
It signiﬁcantly reduces the switching cost compared to “None”, and is smaller than FIX 10 2 .
While it incurs higher switching cost than FIX 10 3 , FIX 10 4 , and
Adaptive 10 2 to 10 4 , on all environments feature-based criterion consistently perform as well as “None” in the sense that 1) it ﬁnds the optimal policy eventually, 2) it has the same sample efﬁciency as “None”.
On the other hand, other criteria sometime have signiﬁcantly worse performance compared to “None”, so none of them is a generic solution.
C ONCLUSION
In this paper, we focus on the concept of switching cost and take a step toward designing a generic solution for reducing the switching cost while maintaining the performance.
Inspired by representation learning theory, we proposed a new feature-based policy switching criterion for deep Q-learning methods.
Through experiment on one medical simulation environment and six Atari games, we ﬁnd our proposed criterion signiﬁcantly reduces the switching cost and at the same time enjoys the same performance as the case where there is no switching cost constraint.
We believe our paper is just the ﬁrst step on this important problem.
One interesting question is how to design principled policy switching criteria for policy-based and model-based methods.
Another direction is to give provable guarantees for these policy switching criteria that work for methods dealing with large state space in contrast to existing analyses are all about tabular RL (Bai et al., 2019; Zhang et al., 2020b;a).
A D ETAILS OF EXPERIMENTS A. DETAILED ALGORITHM
For completeness, we introduce the extensions of DQN and display our detailed algorithm.
Double Q -learning Conventional Q-learning is affected by an overestimation bias, due to the maximization step in Equation 3, Double Q -learning (Hasselt et al., 2016) address this problem by decoupling.
They use the loss
[Equation]
This change reduce harmful overestimations that were present for DQN, which leads to a improvement.
Multi-step learning Q -learning accumulates a single reward, Sutton (2005) use a forward-view multi-step accumulated reward long ago, where a n-step accumulated reward is deﬁned as
[Equation]
and the ﬁnal loss is
[Equation]
Multi-step targets often lead to faster learning (Sutton & Barto, 2005)
Dueling networks Wang et al.
(2016) splits the DQN network into two streams called value stream
V and advantage stream A , these two strems share the convolutional encoder f , and action value
Q ( x, a ) could be computed as:
[Equation]
Prioritized replay DQN samples uniformly from the replay buffer, to sample more frequently from those transitions from which the policy can learn more and quickly, Schaul et al.
samples transitions with probability p h relative to the last encountered absolute TD error
[Equation]
where ω is a hyper-parameter, and new transitions always have the maximum priority when they enter the buffer to ensure a bias towards unseen transitions.
Distributional RL Instead of approximating the expected return as DQN, Bellemare et al.
(2017) proposed a method to approximate the distribution of returns on a a discrete support z , z is a vector with N atoms atoms and is deﬁned as
[Equation]
where V min and V max is the minimal and maximal value in this support.
The approximating distribution d h is deﬁned on this support and the probability p iθ ( x h , a h ) on each atom i as d h
= ( z, p θ ( x h , a h ))
.The
goal is to update the trainable parameters θ to match this distribution with the actual distribution of returns.
To learn the probability p iθ for each i with a variant of Bellman’s equation, they minimize the Kullbeck-Leibler divergence D KL (Φ z d (cid:48)
h || d h ) between the distribution d h and the target distribution d (cid:48)
h :=
( r h +1 + γ h +1 z, p ¯ θ ( x h +1 , arg max a (cid:48)
q ¯ θ ( x h +1 , a (cid:48) ))) , where Φ z is a L2-projection from the target distribution to the ﬁxed support z and q ¯ θ ( x h +1 , a ) =
z T p ¯ θ ( x h +1 , a )
Nosiy Net To address the limitations of exploring using (cid:15) -greedy policies, Fortunato et al.
(2018) propose a noisy linear layer
[Equation]
to replace the standard linear y
=
b + Wx , where (cid:15) b and (cid:15) w are random variables, (cid:12) denotes the element-wise product.
Rainbow combines all of these 6 extensions, To make the policy deterministic but still keep exploring during training, we remove the Noisy Net and adopt Count-Based exploration.
The detailed algorithm is as follows.
[Table]
( x, a ) 14: 15: (cid:46) Training process 16: set up environment 17: n (accumulated updates) = 0 , n (deployment) =18: for h = 1 to H max do 19: if Termination then 20: reset environment 21: end if 22: select a h = arg max a Q deployed ( x h , a ) 23:
Execute action a h in emulator and observe reward r h and state x h + 24: compute the hash codes through for x h , φ ( x h ) = sgn ( Ag ( x h )) 25: (cid:46)
A is a matrix with i.i.d.
entries drawn from a standard Gaussian distribution N (0 , 1) and g is a ﬂat function 26: update the hash table counts, n ( φ ( x h ))
=
n ( φ ( x h ))
+27: update the reward r h
= r h + β √ n ( φ ( x h )) 28: store ( x h , a h , r h , x h +1 ) in D 29: if h > H start then 30: sample minibatch of transitions ( x h (cid:48) , a h (cid:48) , w h (cid:48) , r h (cid:48) , r h (cid:48)
+1 ..., r h (cid:48)
+ n − 1 , x h (cid:48) + n ) from D with probability p h (cid:48) 31: (cid:46) w h (cid:48) is the importance-sampling weight for transitions at h (cid:48)
32: compute the multi-step reward r ( n ) h (cid:48)
= (cid:80) n − 1 k =
γ n r h (cid:48)
+ k 33: a ∗ = arg max a Q online
( x h (cid:48)
+ n , a ) , m i = 0 , i ∈ 1 , 2 , ..., N atoms 34: for j = 1 to N atoms do 35:
[Equation]
γ n z j ] .clip
( V min , V max ) 36: ∆ z =
( V max − V min ) /
( N atoms − 1) 37: b j =
( ˆ T z j − V min ) /
∆ z, l = (cid:98) b j (cid:99) , u = (cid:100)
b j (cid:101) 38:
[Equation]
39: end for 40:
[Equation]
41: update p h (cid:48) by || D KLh (cid:48)
|| ω 42: update parameters in P online by a gradient descent step on D KLh (cid:48)
× w h (cid:48)
43: n (accumulated updates) =
n (accumulated updates)
+44: if h % H target then 45: update P target by the parameters of P online 46: end if 47: if J ( f d , f o , D, n (accumulated updates) , n(deployment))
=
true then 48: update P deployed by the parameters of P online 49: n (accumulated updates) = 0 , n (deployment) =
n (deployment) +50: end if 51: end if 52: end for
A. H YPE - PARAMETERS
Table 1 lists the basic hyper-parameters of the algorithm, all of our experiments share these hyperparameters except the experiments on GYMIC adopt the H target as 1 K .
Most of these parameter are the same with raw Rainbow algorithm.
For the count base exploration, the bonus is β set to 0.01.
[Table]
prioritization we use the recommended proportional variant, with importance sampling from 0.4 to 1, the prioritization ω is set to 0.5.
In addition, we employ N atoms = 51 , V min = − 10 , V max =for distributional RL and n = 3 for multi-step returns.
Finally the count-base bonus is set to 0.
Table 2 lists the rest hyper-parameters for experiments on GYMIC.
Since there are 46 clinical features in this environment, we stack 4 consecutive states to compose a 184-dimensional vector as the input for the state encoder f online ( f deployed or f target ).
The state encoder is a 2-layer MLP with hidden size 128.
Table 3 shows the additional hyper-parameters for experiments on Atari games.
The observations are grey-scaled and resized to 84 × 84 tensor, and 4 consecutive frames are concatenated as a single state, each action selected by the agent is repeated for 4 times.
The state encoder is composed ofconvolutional layers with 32, 64 and 64 channels, which use 8x8, 4x4, 3x3 ﬁlters and strides of 4, 2, 1 respectively.
[Table] [Table]
A. R EWARD AND SWITCHING COST
In the end, we list the value of the switching cost and reward of different criteria when the environment take 1.5 million steps and 3 million steps in Table 4.
B P ROOF FOR S ECTION
Proof of Theorem 1.
We let w
=
) ∈
R k be a k -dimensional all one vector.
We let
F = { f : f ( x ) =
(2 σ ( (cid:104) v 1 , x (cid:105) )
− 1 , 2 σ ( (cid:104) v 2 , x (cid:105) ) − 1 , . .
, 2 σ ( (cid:104) v k , x (cid:105) ) − 1) }
⊂ { R k
→ R k } with σ ( · ) being the ReLU activation function 1 and v i ∈ { e i , − e i } where e i ∈
R k denotes the vector that only the i -th coordinate is 1 and others are 0 .
We assume k is an even number and αk is an integer for simplicity.
We let the underlying f ∗ be the vector correspond to ( e 1 , e 2 , . . .
, e k ) .
We let
D 1 = { ( e 1 , 1) , ( e 2 , 1) , . .
, ( e (1 − α ) k , 1) } and D 2 = { ( e (1 − α ) k
+1 , 1) , . . .
, ( e k , 1) } .
Because we use the ERM training scheme, it is clear that the training on D 1 ∪ D 2 will recover f ∗ , i.e., f 1+2 = f ∗ because if it is not f ∗ is better solution ( f ∗ has 0 error ) for the empirical risk.
Now if the similarity score between f 1 and f 1+2 is smaller than α , it means for f 1 , its corresponding { v (1 − α )
k +1 , . .
. , v k } are not correct.
In this case, f 1 ’s prediction error is at least 1 − α on D 1 ∪D 2 , because it will predict 0 on all inputs of D 2 .
1 , 500 , s t
e p s e nv i r on m e n t c
r
it e r
i a
N on e
F I X 
 F I X
F I X
A d a p ti v e
F 0 .
b a ttl e z on e
S w it
c
h
i ng c o s t . .
R e w a r d .±445 . .± . .± . .± . .± . .± .
G a p
N / A - .- .- .- .- .b
ea m r i
d
e r
S w it
c
h
i ng c o s t . .
R e w a r d .± 202 1 . .± . .± . .± . .± . .± .
G a p
N / A -.-.- .- .-.pong
S w it
c
h
i ng c o s t . .
[Table]
[Table]
R e w a r d −.± 8 .− 2 .± 9 .−.± 4 .−.±
.−
± 10 .
[Table] [Table]
G a p
N / A 10 .
[Table] [Table]
[Table] [Table]
qb e r t
S w it
c
h
i ng c o s t
[Table] [Table]
[Table] [Table]
[Table]
R e w a r d .±189 . .±
[Table]
±
[Table] [Table]
±
[Table] [Table]
±
[Table]
.±
[Table]
G a p
N / A -.
[Table] [Table]
[Table] [Table]
[Table] [Table]
r i v e rr a i d
S w it
c
h
i ng c o s t . .
[Table]
[Table] [Table]
[Table]
R e w a r d .±87 . .±
[Table]
.± 870 . .±
[Table] [Table]
±
[Table] [Table]
±
[Table]
G a p
N / A
[Table] [Table]
[Table] [Table]
[Table] [Table]
r o a d r unn e r
S w it
c
h
i ng c o s t . .
R e w a r d .±513 . .± . .± . .± . .± . .± .
G a p
N / A . .
.3 , 000 , s t
e
p s b a ttl e z on e
S w it
c
h
i ng c o s t . .
R e w a r d .±119 . .± . .± . .± . .± . .± .
G a p
N / A - .- .- .- .- .b
ea m r i
d
e r
S w it
c
h
i ng c o s t . .
R e w a r d .±530 . .± . .±
[Table]
[Table] [Table]
[Table]
G a p
N / A
[Table] [Table]
[Table] [Table]
[Table] [Table]
pong
S w it
c
h
i ng c o s t .
[Table] [Table]
R e w a r d 17 .± 6 .
[Table] [Table]
[Table] [Table]
G a p
N / A
[Table] [Table]
[Table]
qb e r t
S w it
c
h
i ng c o s t . .
[Table]
R e w a r d .±486 . .± . .
[Table]
[Table] [Table]
[Table] [Table]
[Table]
G a p
N / A -
[Table] [Table]
[Table] [Table]
r i v e rr a i d
S w it
c
h
i ng c o s t
[Table]
[Table]
R e w a r d .±08 . .±
351 . .±
939 . .±
821 . .±
938 . .±
G a p
N / A 31 .64 .- .- .-.r
o a d r unn e r
S w it
c
h
i ng c o s t . .
R e w a r d .±429 . .± . .± . .± . .± . .± .
G a p
N / A - .-.- . . .
T a b l e 4 :
W e li s t t h e v a l u e o f t
h e s w it
c h
i ng c o s t a nd r e w a r d o f d i ff e r e n t c
r
it e r i a w h e n t h e e nv
i r on m e n t t a k e s 1 .m illi on s t e p s a nd 3 m illi on s t e p s .
“ R e w a r d ” c o rr e s pond s t o t h e a b s o l u t e v a l u e o f t h e r e w a r d , a nd “ G a p ” d e no t e s t h e d i ff e r e n ce b e t w ee n t
h e r e w a r d und e r a s p ec i
ﬁ c c r
it
e
r
i on a nd “ N on e . ”
A nd “ S w it
c
h
i ng
C o s t ” c o rr e s pond s t o t
h e s w it
c h
i ng
c o s t und e r a c r it e r i o n a t t h
i s ti m
e s t e p .