B ATCH R EINFORCEMENT L EARNING T HROUGH C ONTINUATION M ETHOD
Anonymous authors
Paper under double-blind review
A BSTRACT
Many real-world applications of reinforcement learning (RL) require the agent to learn from a ﬁxed set of trajectories, without collecting new interactions.
Policy optimization under this setting is extremely challenging as: 1) the geometry of the objective function is hard to optimize efﬁciently; 2) the shift of data distributions causes high noise in the value estimation.
In this work, we propose a simple yet effective policy iteration approach to batch RL using global optimization techniques known as continuation.
By constraining the difference between the learned policy and the behavior policy that generates the ﬁxed trajectories, and continuously relaxing the constraint, our method 1) helps the agent escape local optima; 2) reduces the error in policy evaluation in the optimization procedure.
We present results on a variety of control tasks, game environments and a recommendation task to empirically demonstrate the efﬁcacy of our proposed method.
I NTRODUCTION
While RL is fundamentally an online learning paradigm, many practical applications of RL algorithms, e.g., recommender systems [5, 7] or autonomous driving [36], fall under the batch RL setup.
Under this setting, the agent is asked to learn its policy from a ﬁxed set of interactions collected by a different (and possibly unknown) policy commonly referred to as the behavior policy, without the ﬂexibility to gather new interactions.
Realizing the interactive nature of online RL has been hindering its wider adoptions, researchers strive to bring these techniques ofﬂine [24, 11, 20, 23, 31, 12, 21, 2, 32, 8].
We focus on policy optimization under batch RL setup.
As pointed out in [3, 26], even with access to the exact gradient, the loss surface of the objective function maximizing the expected return is difﬁcult to optimize, leading to slow convergence.
Chen et al.
[8] show that the objective function of expected return exhibits sub-optimal plateaus and exponentially many local optima in the worst case.
Batch setup makes the learning even harder as it adds large variance to the gradient estimate, especially when the learned policy differs from the behavior policy used to generate the ﬁxed trajectories.
Recent works propose to constrain the size of the policy update [27, 28] or the distance between the learned policy and the behavior policy [14, 21].
The strength of that constraint is a critical hyperparameter that can be hard to tune [28], as a loose constraint does not alleviate the distribution shift while a strict one results in conservative updates.
Here we propose to address the challenges using continuation methods [35, 6, 17].
Continuation methods attempt to solve the global optimization problem by progressively solving a sequence of new objectives that can be optimized more efﬁciently and then trace back the solutions to the original one.
We change the objective function of policy optimization by including an additional term penalizing the KL divergence between the parameterized policy π θ and the behavior policy.
We then gradually decrease the weight of that penalty, eventually converging to optimizing the expected return.
With this additional constraint, we beneﬁt from more accurate policy evaluation in the early stage of training as the target policy is constrained to be close to the behavior policy.
As training continues, we relax the constraint and allow for more aggressive improvement over the behavior policy as long as the policy evaluation is still stable and relatively reliable, i.e. with a small enough variance.
By doing so, the proposed method exhaustively exploits the information in the collected trajectories while avoiding the overestimation of state-action pairs that lack support.
The contributions of this paper are as follows: (1) We propose a soft policy iteration approach to batch RL through the continuation method.
(2) We theoretically verify that in the tabular setting with exact gradients, maximizing KL regularized expected return leads to faster convergence than optimizing the expected return alone.
Also, our method converges to the globally optimal policy if there are sufﬁcient data samples for accurate value estimation.
(3) We demonstrate the effectiveness of our method in reducing errors in value estimation using visualization; (4) We empirically verify the advantages of our method over existing batch RL methods on various complex tasks.
R ELATED W ORK
Batch Reinforcement Learning.
Off-policy reinforcement learning has been extensively studied [11, 20, 30, 23, 31], with many works [12, 21, 2] focusing on variants of Q-learning.
Fujimoto et al.
[12], Kumar et al.
[21] investigated the extrapolation error in batch RL resulting from the mismatch of state-action visitation distribution between the ﬁxed dataset and the current policy, and proposed to address it by constraining the action distribution of the current policy from deviating much from the training dataset distribution.
Recent works [29, 33] studied policy iteration under batch RL.
The Q function is estimated in the policy evaluation step without special treatment while the policy updates are regularized to remain close to the prior policy with a ﬁxed constraint.
To further reduce uncertainty in Q learning, an ensemble of Q networks [21, 29] and distributional Q-function [2, 33] are introduced for the value estimation.
[34, 18] use the KL divergence between the the target policy and the behavior policy as a regularization term in the policy update and/or value estimation.
The constraint is controlled by a ﬁxed weight of the KL regularization or a ﬁxed threshold for the
KL divergence.
While all of these works apply a ﬁxed constraint determined by a sensitive hyperparameter to control the distance between the behavior/prior policy and the target policy, we focus on gradually relaxed constraints.
Constrained Policy Updates.
Several works [27, 1, 15] studied constrained policy updates in online settings.
Kakade & Langford [19] show that large policy updates can be destructive, and propose a conservative policy iteration algorithm to ﬁnd an approximately optimal policy.
Schulman et al.
[27] constrain the KL divergence between the old policy and new policy to guarantee policy improvement in each update.
Grau-Moya et al.
[15] force the policy to stay close to a learned prior distribution over actions, deriving a mutual-information regularization between state and action.
Cheng et al.
[9] propose to regularize in the function space.
Again these methods focused on a ﬁxed constraint while we are interested in continuing relaxing the constraint to maximize the expected return eventually.
Also none of these methods have been extensively tested for batch RL with ﬁxed training data.
Continuation Method.
Continuation method [35] is a global optimization technique.
The main idea is to transform a nonlinear and highly non-convex objective function to a series of smoother and easier to optimize objective functions.
The optimization procedure is successively applied to the new functions that are progressively more complex and closer to the original non-context problem, to trace their solutions back to the original objective function.
Chapelle et al.
[6] use the continuation method to optimize the objective function of semi-supervised SVMs and reach lower test error compared with algorithms directly minimizing the original objective.
Hale et al.
[17] apply the continuation method to l1-regularized problems and demonstrate better performance for compressed sensing problems.
Inspired by prior works, we employ the continuation method to transform the objective of batch RL problems by adding regularization.
We gradually decrease the regularization weight to trace the solution back to the original problem.
M ETHOD
In classical RL, an agent interacts with the environment while updating its policy.
At each step t , the agent observes a state s t ∈ S , selects an action a t ∈ A according to its policy to receive a reward r t = r
( s t , a t ) : S ×
A → R and transitions to the next state s t +1 ∼ P ( ·| s t , a t ) .
The state value of a policy π is V π ( s ) =
E s
= s,a t ∼ π ( ·| s t ) ,s t +1 ∼P ( ·| s t ,a t ) [ (cid:80) ∞ t =
γ t r ( s t , a t )] .
γ ∈ [0 , 1] is the discounting factor.
At each step, the agent update its policy so that the expected return V π ( ρ ) =
E s ∼ ρ
[ V π ( s )] (where ρ is the initial state distribution) is maximized when acting accordingly.
In batch RL, the agent is not allowed to interact with the environment during learning.
Instead it has access to a ﬁxed set of trajectories sampled from the environment according to some behavior policy 1 .
A trajectory { ( s 0 , a 0 , r 0 ) , ( s 1 , a 1 , r 1 ) , · · · , ( s T , a T , r T ) } is generated by sampling s 0 from the initial state distribution ρ and the action a t ∼ β ( ·| s t ) at the state s t and s t +1 ∼ P ( ·| s t , a t ) for each step t ∈ [0 , 1 , · · · , T ] .
The length T can vary between trajectories.
We then convert the trajectories to a dataset D
= { ( s i , a i , r i , s (cid:48) i ) }
Ni =1 , where s (cid:48)
i is the next state after s i in a trajectory.
The goal is to learn a parameterized policy π θ with the provided dataset to maximize the expected return V π ( ρ ) .
In Sec. 3.1, we will ﬁrst introduce a new objective function ˜ V π,τ ( ρ ) , i.e. the expected return of policy π with KL regularization weight τ .
With exact gradients, ˜ V π,τ ( ρ ) can be optimized more efﬁciently than the original objective V π ( ρ ) .
With the continuation method, solving a sequence of optimization problems for ˜ V π,τ ( ρ ) with decaying value of τ converges toward optimizing V π ( ρ ) and makes the optimization easier.
In Sec. 3.2, we derive soft policy iteration with KL regularization to optimize ˜ V π,τ ( ρ ) , without the assumption of exact gradients.
Finally, in Sec. 3.3, we propose a practical batch RL algorithm with value estimation for target policy based on this theory.
3. O PTIMIZING E XPECTED R ETURN WITH KL REGULARIZATION
In batch RL, the distribution of the trajectories generated by the behavior policy can be very different from that of the learned policy.
We thus restrict the learned policy to stay close to the behavior policy via the regularization of KL divergence.
Deﬁne the soft state value of policy π as
[Equation]
where the temperature parameter τ controls the deviation from β , the new objective function becomes ˜ V π,τ ( ρ ) =
E s ∼ ρ
[ ˜ V π,τ ( s )] .
This KL regularized objective differs from the original objective
V π ( ρ ) , which however can be recovered as τ → 0 .
As pointed out in [3], even with exact gradients, the objective function V π ( ρ ) is still difﬁcult to optimize due to its highly non-smooth landscape.
Mei et al.
[26] further prove that, in a tabular setting with softmax parameterized policy and exact gradients, the vanilla policy gradient method (i.e. directly maximizing V π ( ρ ) with gradient descent) converges to the global optimal policy at a convergence rate O (1 /t ) , while the entropy-regularized policy gradient enjoys a signiﬁcantly faster linear convergence rate O ( e − t ) .
Motivated by this line of work, we investigate the convergence of optimizing ˜ V π,τ ( ρ ) with the exact gradient descent and compare it with the vanilla policy gradient method.
We study the smoothness and Łojasiewicz inequality for the function ˜
V π,τ ( ρ ) to prove the convergence rate, similar to [26].
The detailed proofs of the following theorems are in the appendix.
Theorem 1.
In the tabular setting with softmax parameterized policy π θ , maximizing ˜ V π,τ ( ρ ) using policy gradient with the learning rate η =
[Equation]
) , for all t > 1 , we have
[Equation]
where π ∗ τ is the optimal policy maximizing
˜ V π,τ ( ρ ) , M is the bound of the absolute value of r ( s, a )+ τ log β
( a | s ) , A is the size of action space, S is the size of state space, C τ ∝
[Equation]
and C is a constant independent with t and τ .
Theorem 1 states that KL regularized expected return can be optimized with a convergence rate
O ( e − t ) rather than the O (1 /t ) , the convergence rate of vanilla policy gradient for expected return alone.
The faster convergence inspires us to optimize ˜ V π,τ ( ρ ) to reach policy π ∗ τ , then use π ∗ τ as initialization, gradually decrease the temperature τ towards 0, and eventually move from π ∗ τ to π ∗ = arg max π V π ( ρ ) .
With a reasonable value of τ , we enjoy a linear convergence rate toward π ∗ τ from the randomly initialized policy π θ .
As τ decreases, π ∗ τ gets closer to π ∗ .
The ﬁnal optimization of V π θ ( ρ ) from π ∗ τ can be much faster than from a randomly initialized π θ .
[Equation]
[Figure]
We construct a toy example to illustrate this motivation.
In the grid world (Fig. 1a), the start state, annotated with ‘S’, is in the center and the terminal states are marked in yellow.
There are only two states with positive rewards (0.9 and 1).
There are four actions { up, down, left, right } .
A badly initialized policy π θ 0 is shown as arrows in Fig. 1a).
The initialization results in a poor policy, having high tendency to go right toward a terminal state with zero reward.
The vanilla policy gradient method (i.e. maximizing V π ( ρ ) with true gradient) starting from this initial point takes more than 7000 iterations to escape a sub-optimal solution (Fig. 1b).
In contrast, we escape the sub-optimal solution much faster when applying the continuation method to update the policy with the gradients of ˜ V π,τ ( ρ ) , where the behavior policy β ( ·| s ) =
[ u 1 , u 2 , u 3 , u 4 ] with u i , i = 1 , 2 , 3 , 4 randomly sampled from U [0 , 1] and normalized for each state s .
In Fig.
1b, as we decrease τ , the value of learned policy π θ i for each iteration i quickly converges to the optimal value.
In other words, optimizing a sequence of objective functions ˜ V π,τ ( ρ ) can reach the optimal solution for V π ( ρ ) signiﬁcantly faster.
1: Initialize: actor network π θ , ensemble critic network { Q φ (1) , Q φ (2) , · · · , Q φ ( K ) } , behavior policy network β ψ , penalty coefﬁcient τ , decay rate λ , number of iterations
I for each τ 2: Input: training dataset D
= { ( s i , a i , r i , s (cid:48) i ) } Ni = 3: for update j = 0 , 1 , · · · do 4:
Sample batch of data { ( s i , a i , r i ) }
Bi
=1 from D 5: # Learn the behavior policy with behavior cloning objective 6:
Update ψ to maximize 1 B (cid:80) Bi
=1 log β ψ ( a i | s i ) 7: # Train the critic network 8:
Update φ ( k ) to minimize the temporal difference 1 B (cid:80) Bi
=1 (cid:0) r i + γV ( s (cid:48) i ) − Q φ ( k ) ( s i , a i ) (cid:1)9: where V ( s ) =
1 K (cid:80)
Kk =1 E a ∼ π θ ( ·| s )
( Q φ ( k ) ( s, a )) − τKL ( π θ ( ·| s ) | β ψ ( ·| s )) 10: # Train the actor network 11:
Update θ to maximize 1 B (cid:80) Bi
=1 (cid:104) 1 K (cid:80)
Kk =1 E a ∼ π θ ( ·| s i ) ( Q φ ( k ) ( s i , a )) − τKL ( π θ ( ·| s i ) | β ψ ( ·| s i ))
(cid:105) 12: # Decay the weight of KL regularization τ for every I updates 13: if j mod I = 0 then 14: τ ← τ ∗ λ 15: end if 16: end for
3. S OFT P OLICY I TERATION WITH KL REGULARIZATION
As explained in the previous section, we focus on the new objective function ˜ V π,τ ( ρ ) , which can be optimized more efﬁciently, and use continuation method to relax toward optimizing V π ( ρ ) .
Batch
RL adds the complexity of estimating the gradient of ˜ V π,τ ( ρ ) with respect to π from a ﬁxed set of trajectories.
We propose to adapt soft actor-critic[16], a general algorithm to learn optimal maximum entropy policies in batch RL for our use case.
We change the entropy regularization to KL regularization and derive the soft policy iteration to learn KL regularized optimal policy.
For a policy π and temperature τ , the soft state value is deﬁned in Eq.
1 and soft Q function is deﬁned as:
[Equation]
In the step of soft policy evaluation, we aim to compute the value of policy π according to the minimum KL divergence objective ˜ V π,τ ( ρ ) =
E s ∼ ρ
[ ˜ V π,τ ( s )] .
According to Lemma 1 in Appendix, the soft Q value can be computed by repeatedly applying the soft bellman backup operator.
[Equation]
) , where V
[Equation]
In the step of policy improvement, we maximize the expected return based on Q-value evaluation with the KL divergence regularization.
The following policy update can be guaranteed to result in an improved policy in terms of its soft value (Lemma 2 in Appendix).
[Equation]
where
[Equation]
The soft policy iteration algorithm alternates between the soft policy evaluation and soft policy improvement, and it will provably converge to the optimal policy maximizing the objective ˜ V π,τ ( ρ ) .
Theorem 2.
Repeated application of soft policy evaluation and soft policy improvement converges to a policy π ∗ τ such that ˜ Q π ∗ τ ,τ ( s, a ) ≥ ˜ Q π,τ ( s, a ) for any π ∈ Π and ( s, a ) ∈ S × A .
The soft policy iteration ﬁnds a policy π ∗ τ with optimal soft Q value for each state-action pair and hence gets the optimal value of ˜ V π,τ ( ρ ) .
Here we propose to use the soft policy iteration to solve objectives ˜
V π,τ ( ρ ) with decreasing value of τ and move back to the objective V π ( ρ ) as τ = 0 .
The method is guaranteed to asymptotically converge to the optimal policy π ∗ for the objective V π ( ρ ) .
Theorem 3.
Let π ∗ τ ( a | s ) be the optimal policy from soft policy iteration with ﬁxed temperature τ .
We have π ∗ τ ( a | s ) ∝ exp (cid:16) ˜ Q π ∗ τ,τ ( s,a ) τ (cid:17) β ( a | s ) .
As τ → 0 , π ∗ τ ( a | s ) will take the optimal action a ∗ with optimal Q value for state s .
[Figure]
3. E RROR IN V ALUE E STIMATE
In the previous section, we show that the soft policy iteration with the continuation method provably converges to the global optimal policy maximizing expected return.
However, in batch RL with a ﬁxed dataset and limited samples, we cannot perform the soft policy iteration with KL regularization in its exact form.
Speciﬁcally, in the policy evaluation step, when the learned policy π deviates for the behavior policy β , and chooses the state-action pair ( s, a ) rarely visited by β , the estimation of target r ( s, a ) + γ E s (cid:48) ∼P ( ·| s,a ) ( V ( s (cid:48) )) can be very noisy.
The error in the value estimate Q ( s, a ) will be further propagated to other state-action pairs through the bellman update.
Finally, inaccurate value estimation will cause errors in the policy improvement step, resulting in a worse policy.
On the other hand, if we constrain the learned policy π to be very close to the behavior policy β , we can expect the policy evaluation to be reliable and safely update the learned policy.
The tight constraint however prevents π to be much better than β due to the conservative update.
On the grid world, we study this problem of value estimation with different values of τ .
Figure 2 visualizes the propagation of Q value estimation errors and the learned policies.
We assume a mediocre behavior policy tending to move left and down.
For the rarely visited states in the upper right part of the grid, there are errors in the value estimation of ˜ Q π,τ ( s, a ) , i.e. | Q ( s, a ) − ˜ Q π,τ ( s, a ) | >where Q ( s, a ) is the Q value we learn during training and ˜ Q π,τ ( s, a ) is the ground truth soft Q value.
Because the bad initial policy (Fig. 1a) tends to move towards the right part, without a strong
KL regularization, the policy evaluation can be problematic due to the errors of value estimation in the right part of the grid world.
In Fig.
2, the ﬁrst row shows that errors even propagate to the frequently visited states by the behavior policy.
On the other hand, when we set a large value of τ = 1 (second row), the error | Q ( s, a ) − ˜ Q π,τ ( s, a ) | is smaller.
Yet the performance of the learned policy is not much better than the behavior policy.
Our continuation method gradually moves the policy update between these two spectra.
The value estimation beneﬁts from the gradually relaxed
KL regularization and the errors remain small.
The last column of Fig.
2 visualizes the learned policy in these methods.
With constant τ = 0 .
001 , the wrong value estimates in some states mislead the agent.
It fails to visit any terminal state and gets stuck at the state in dark orange.
With constant τ = 1 , the tight constraint of KL divergence makes the learned policy close to the behavior policy, mostly visiting the left bottom part of the environment.
With continuation method, the agent learns to always take the optimal path moving left directly and obtains the highest expected return.
More details of this example are provided in the appendix.
In the toy example, gradually relaxing KL regularization towards zero alleviates the propagation of errors in the soft Q estimate and helps the agent converge to the optimal policy.
In more complicated domains, we ﬁnd that as τ decays close to 0, the policy evalua- tion is still erroneous.
To mitigate this issue, we introduce an ensemble of critic networks { Q φ (1) , Q φ (2) , · · · , Q φ ( K ) } to approximate the soft Q value, and monitor the variance of value estimation in different critic networks to measure the uncertainty.
Given a batch of data samples { s i } Bi =1 ⊂ D , var ( Q π ) = 1 B (cid:80) Bi
=1 E a ∼ π ( ·| s i ) var ( Q φ (1) ( s i , a ) , Q φ (2) ( s i , a ) , · · · , Q φ ( k ) ( s i , a )) indicates whether the current policy π tends to take actions with highly noisy value estimation.
Our method is summarized in Algorithm 1.
Instead of running the soft policy evaluation and policy improvement until convergence, we alternate between optimizing the critic network and actor network with stochastic gradient descent.
We set τ to large value initially and let the KL divergence term dominate the objective, thus performing behavior cloning.
We record a moving average of the
Q value estimation variance var ( Q π,τ 0 ) over 1000 updates at the end of the phase.
After that, we decay the temperature gradually with = 0 .
9 every I steps.
When the moving average of the Q value estimation variance var ( Q π,τ ) is large compared with the initial value var ( Q π,τ 0 ) (i.e. at the end of behavior cloning), we no longer trust the value estimate under the current temperature τ and take the policy checkpointed before the temperature decays to this τ as our solution.
E XPERIMENTS 4. M UJOCO
We evaluate our method with several baselines on continuous control tasks.
We train a Proximal
Policy Optimization agent [28] with entropy regularization for 1000 million steps in the environments.
We parameterize the policy using Gaussian policies where the mean is a linear function of the agent’s state θ T s and variance is an identity matrix to keep the policy simple as introduced in [3].
To generate training datasets D with varying quality, we construct the behavior policy by mixing the well-trained policy N ( θ Topt s, 0 .
5 I ) , i.e. checkpoint with the highest score during training, and a poor policy N ( θ T 0 s, 0 .
5 I ) , i.e. checkpoint at the beginning of the training, with the weight α .
Then the behavior policy β ( ·| s ) is N (((1 − α )
θ opt + αθ 0 )
T s, 0 .
5 I ) .
We generate trajectories and store a total of one million data samples from the mixed behavior for different values of the coefﬁcient α .
The architecture of the target policy is the same as the behavior policy.
We consider four baseline approaches: BCQ [14], BEAR [21], ABM+SVG [29], CRR [33], CQL [22], BRAC [34].
For a fair comparison, the architectures of the critic network and the policy network are the same in the baselines and our method, except BCQ which has no policy network.
To evaluate and compare the methods, we run the learned policy in the environments for 100 episodes and report the average episode reward in Fig 3.
As for the continuation method, we report the score of policy from checkpointed last with reasonable value estimation variance, as explained in Section 3.3.
For the baselines, we report the score of the ﬁnal policy when we terminate the training at 1.5M updates.
[Table]
Tab. 1 shows that our method outperforms all the baselines on 5 settings.
On the dataset with relatively reasonable quality (i.e. α = 0 . 2 , 0 . 4 , 0 . 6 ), ours performs comparable or better than the baselines.
With α = 0 . 2 , i.e., close to optimal behavior policy, all the methods perform similarly and one can achieve a good return by simply cloning the behavior policy.
With α = 0 . 8 , i.e., low-quality behavior policy, there are few good trajectories in the dataset for any methods to learn.
The advantage of our method is most obvious when α = 0 .
6 (Fig. 3), as the dataset contains trajectories of both
[Table]
high and low cumulative rewards.
Our method can learn from the relatively large number of good trajectories and at the same time deviate from the behavior policy to avoid those bad trajectories and achieve higher rewards.
In Fig.
3, ‘Constant’ is the method of optimizing KL regularized expected reward with constant value of τ .
We search several values of τ and report the best result.
We can see that gradually relaxing constraint performs better than the ﬁxed constraint.
In Fig. 3(left), as τ decays to close to 0, the learned policy can degrade due to errors in the Q estimation, the stopping condition explained in Section 3.3 is however able to identify a good policy before the degenerating point.
More experimental details are in the Appendix.
4. A TARI
We further study our method on several Atari games from the Arcade Learning Environment (ALE) [4].
The rich observation space requires more complicated policies and makes policy optimization even more challenging.
We focus on eight games and generate the datasets as discussed in Fujimoto et al.
We use a mediocre DQN agent, trained online for 10 million timesteps ( million frames).
The performance of the DQN agent is shown as ‘Online DQN’ in Fig.
We add exploratory noise on the DQN agent (at 10 million timesteps) to gather a new set of 10 million transitions, similar to [13].
The line ”Behavior” in Fig.
4 shows the average of trajectory reward in the dataset D .
The dataset D is used to train each off-policy agent.
We compare with BCQ[13] and
REM[2] because they are recently proposed ofﬂine RL algorithms and work well on Atari domain.
For evaluation, we run 10 episodes on the Atari games with the learned policies and record the average episode reward (Fig. 4).
Tab.
2 summarizes the performance of BCQ, REM and CQL after 6M updates.
For our method, we report the score before the variance of Q estimate becomes too high.
Our approach achieves higher scores than the baselines on 7 out of 8 games, and perform comparably on the other one.
Agarwal et al.
[2] reports that REM performs well in the dataset consisting of the entire replay experiences collected in the online training of the DQN agent for 50M timesteps (200M frames).
We hypothesize that learning on the entire replay experience makes the setup easier as the training dataset contains more exploratory and higher quality trajectories.
With the dataset of much smaller size and worse quality, REM performs poorly in this single behavioral policy setting.
We use the same architecture of the critic network for both our method and BCQ with ensemble of 4 Q networks.
As mentioned in [13], BCQ only matches the performance of the online DQN on most games.
In contrast, ours is able to outperforms online DQN signiﬁcantly on several games.
4. R ECOMMENDER
We also showcase our proposed method for building a softmax recommender agent.
We use a publicly available dataset MovieLens-1M, a popular benchmark for recommender system.
There are 1 million ratings of 3,900 movies (with the title and genre features) from 6,040 users (with demographic features).
The problem of recommending movies for each user can be converted to a
[Figure]
[Table]
We report the mean standard deviation of the over contextual bandit problem, where we aim to learn a target policy π θ ( a | s ) selecting the proper action (movie) a for each state (user) s to get a high reward (rating) r in a single step.
The ratings are converted to binary reward using a cutoff of 4.
To evaluate whether a learned target policy works well, ideally we should run the learned policy in real recommendation environments.
However, such environments for online test are rarely publicly available.
Thus, we use the online simulation method.
We train a simulator to predict the immediate binary feedback from user and movie features, and the well-trained simulator can serve as a proxy of the real online environments, because it outputs the feedback for any user-movie pair.
Similar to [25], we train the simulator with all records of logged feedback in MovieLens-1M dataset.
The behavior policy is trained with partial data in
MovieLens-1M. We then construct the bandit datasets D
= { s i , a i , r i , β ( a i | s i ) } Ni =1 of different size and quality, by using different behavior policies β to select movies a i for users s i and getting the binary feedback r i from the well-trained simulator.
We train ofﬂine RL agents on the generated dataset D and use the simulator to evaluate the learned policies on a held-out test set of users.
We compare our method with two baselines, as they are commonly used in current industrial recommender systems [10, 7].
(1) Cross-Entropy: a supervised learning method for the softmax recommender where the learning objective is the cross-entropy loss J CE ( θ ) =
− 1 N (cid:80)
Ni =1 r
i log π θ ( a i | s i ) (2) IPS: the off-policy policy gradient method introduced in [7] with the learning objective
J IPS ( θ ) =
[Equation]
(cid:80)
[Equation]
=
[Equation] [Equation]
θ ( a i | s i ))
β ( s i ,a i ) r
i log π θ ( a i | s i ) where sg indicates a stop-gradient operation.
J IPS ( θ ) produces the same gradient as that of the function 1 N (cid:80)
Ni =1 r i π θ ( a i | s i )
β ( a i | s i ) , the expected return with importance sampling.
(3) Ours: in the bandit setting, we simply perform IPS with gradually decaying KL regularization since estimating the soft Q from bellman update is not needed.
Tab.clearly demonstrates the advantage of our proposed method over the baselines.
IPS can be viewed as vanilla policy gradient with importance sampling to correct the distribution shift.
Our method clearly outperforms it across datasets collected using different behavior policies.
C ONCLUSION
We propose a simple yet effective approach, soft policy iteration algorithm through continuation method to alleviate the two challenges in policy optimization under batch reinforcement learning: (1) highly non-smooth objective function which is difﬁcult to optimize (2) high variance in value estimates.
We provide theoretical ground and visualization tools to help understand this technique.
We demonstrate its efﬁcacy on multiple complex tasks.
A PPENDIX : A C ONVERGENCE FOR S OFT P OLICY I TERATION
Lemma 1.
Consider the soft Bellman backup operator T π,τ Q ( s, a )
=
r ( s, a ) + γ E s (cid:48) ∼P ( ·| s,a ) ( V ( s (cid:48) )) where V ( s ) =
E a ∼ π ( ·| s ) (cid:104) Q ( s, a ) − τ log π ( a | s ) β ( a | s ) (cid:105) , then the sequence Q ,
T π,τ Q , ( T π,τ ) 2 Q , · · · , ( T π,τ ) k Q will converge to the soft value of π as k → ∞ .
Proof.
We prove the soft Bellman backup operator is a contraction.
If we apply T π to two different value functions Q and Q (cid:48) , the max norm distance (cid:107) Q − Q (cid:48)
(cid:107) =
max s,a | Q ( s, a ) − Q
[Equation]
( s, a ) | shrinks.
[Equation]
Therefore
[Equation]
the sequence
[Equation]
only has one ﬁxed point.
Consider the soft Q value of policy π ,
[Equation]
we have T π,τ ˜ Q π,τ
[Equation]
s t
[Equation]
t
[Equation]
=
˜ Q π
[Equation]
Thus, ( T π,τ ) k Q will converge to the ﬁxed point ˜ Q π,τ .
Lemma 2.
let π old ∈ Π and π new ( ·| s )
= arg
[Equation]
˜
[Equation]
old
[Equation]
a
[Equation]
Then ˜ Q π new ,τ ( s, a ) ≥ ˜ Q π old ,τ ( s, a ) for all ( s, a ) .
Proof.
Due to the deﬁnition of π new , for any state s t , we have E
[Equation] [Equation]
Theorem 2.
Repeated application of soft policy evaluation and soft policy improvement converges to a policy π ∗ τ such that ˜ Q π ∗ τ ,τ ( s, a ) ≥ ˜ Q π,τ ( s, a ) for any π ∈ Π and ( s, a ) ∈ S × A .
Proof.
Let π i be the policy at iteration i.
The sequence Q π i is monotonically increasing.
Since Q π is bounded above (because both reward and KL divergence are bounded)
[Equation]
the sequence converges to some π ∗ .
We will still need to show that π ∗ is indeed optimal.
At convergence
[Equation]
it must be case that:
[Equation]
So for any π ∈ Π
[Equation]
we have
[Equation]
We denote this optimal policy π ∗ as π ∗ τ in the following.
Consider the optimization problem with hard constraint:
[Equation]
s
[Equation]
|
[Equation]
| (cid:88)
i
[Equation] [Equation]
[Equation]
| A | (cid:88)
i
[Equation]
π ( a i | s ) log π ( a i
[Equation]
s )
[Equation]
( a i | s ) s.t.
| A | (cid:88) i =
π ( a i | s ) =
Due to KKT condition, let ˜ Q π ∗ τ ,τ ( s, a )
− τ log π ( a i | s ) β
( a i | s ) − τ − λ = 0 .
We have π ∗ τ ( a | s ) =
[Equation]
Theorem 3.
Let π ∗ τ ( a | s ) be the optimal policy from soft policy iteration with ﬁxed temperature τ .
We have π ∗ τ ( a | s ) ∝ exp (cid:16) ˜ Q π ∗ τ,τ ( s,a ) τ (cid:17) β ( a | s ) .
As τ → 0 , π ∗ τ ( a | s ) will take the optimal action a ∗ with optimal Q value for state s .
Assume there exists a set of optimal actions X ( s ) for state s .
For each action in X ( s ) , its soft Q value is the optimal for state s .
To simplify the notation, in the following proof, we use Q ∗ τ to replace ˜
Q π ∗ τ ,τ .
Assume a j ∈ X ( s ) , then π ∗ τ ( a j | s ) =
[Equation]
For the second term in the denominator, we show it converges to 0 as τ → 0 .
[Equation] [Equation]
β (
[Equation]
s )
As τ → 0 , exp (cid:16) Q ∗ τ ( s,a sub ) − Q ∗ τ ( s,a opt ) τ (cid:17) → 0 and (cid:80) a i (cid:54)∈ X ( s ) β ( a i | s ) β ( a j | s ) is a constant.
Thus, as τ → 0 , π ∗ τ ( a j | s ) → 1 (cid:80) ai ∈ X ( s ) β ( ai | s ) β ( aj |
[Equation]
= β ( a j | s ) (cid:80) ai ∈ X
( s ) β ( a i | s ) if a j ∈ X ( s ) .
Assume a j (cid:54)∈ X ( s ) , then π ∗ τ ( a j | s ) = 1 (cid:80) ai
∈ X ( s )
[Equation]
(cid:16)
[Equation]
Obviously, exp
[Equation]
(cid:17)
[Equation]
j | s ) → + ∞ as τ → and exp
[Equation]
(cid:17)
[Equation]
> 0 .
Thus, as τ → 0 , π ∗ τ ( a j | s ) → 0 if a j (cid:54)∈ X ( s ) .
B C ONVERGENCE R ATE OF KL REGULARIZED P OLICY G RADIENT
Recall our deﬁnitions:
[Equation]
Mei et al.
[26] proved that policy gradient with a softmax parameterization and true gradient optimizes V π ( ρ ) at a O (1 /t ) convergence rate, while the entropy regularized objective V π ( ρ ) +
τ H ( ρ, π ) with H ( ρ, π ) =
E s 0 ∼ ρ,a t ∼ π ( ·| s t ) ,s t +1 ∼P ( ·| s t
[Equation]
a t ) [
[Equation]
∞
[Equation]
=
[Equation]
t log π ( a t | s t ))] enjoys a signiﬁcantly faster linear convergence rate O ( e − t ) .
With a behavior policy β and the ”temperature” τ > 0 that determines the strength of the regularization, we deﬁne the value of the policy π with the KL divergence regularization as
[Equation]
(cid:19)(cid:35) (8)
Similarly we prove policy gradient optimizes ˜ V π,τ ( ρ ) at a O ( e − t ) convergence rate.
Here we explain the sketch of the proof.
˜ V π,τ ( ρ ) can be re-written as ˆ V π,τ ( ρ ) + τ H ( ρ, π ) where ˆ V π,
[Equation]
ˆ V π,τ ( ρ ) in eq.(8) is the same as V π ( ρ ) in eq.(7) if we replace r ( s t , a t ) with r ( s t , a t )+ τ log β ( a t
| s t ) .
Mei et al.
[26] assume that r ( s t , a t ) ∈ [0 , 1] .
Here we assume r ( s t , a t )+ τ log β ( a t
| s t ) ∈ [ − M, M ] with a constant M , effectively requiring the behavior policy to span the entire state-action space.
We can then adapt the proof in [26] for the new objective ˜ V π,τ ( ρ ) .
We conclude that in t iterations, the learned policy π θ t is approaching the optimal policy π ∗ τ for the new objective ˜ V π,τ ( ρ ) , satisfying ˜ V
π ∗ τ ,τ ( ρ ) − ˜ V π θt ,τ ( ρ ) ≤
[Equation]
, where C
[Equation]
=
[Equation]
(cid:107) d π ∗ τµ µ (cid:107) − 1 ∞ and µ is the initial state distribution used in the policy optimization algorithm.
To simplify the notation, in the following proofs, we use ˜ V π ( ρ ) =
˜ V π,τ ( ρ ) and ˆ V π
( ρ )
=
ˆ V π,τ ( ρ ) .
Assumption 1 (Bounded Reward and Behavior Probability) .
r ( s, a ) + τ log β ( a | s ) ∈ [ − M, M ] , ∀ ( s, a ) where M is a positive constant.
Deﬁnition 1 (Smoothness) .
A function f : Θ → R is λ
-smooth if for all θ, θ
(cid:48) ∈ Θ , (cid:12)
[Equation]
Lemma 3 (Monotone Increasing) .
Assume a function f : Θ → R is λ -smooth, and it is updated with θ t +
=
θ t + η ∂f ( θ ) ∂θ and η = 2 λ , then the function f ( θ ) is monotone increasing.
[Equation]
Deﬁnition 2.
Given a vector θ ∈ R [ K ] and the probability distribution π θ = softmax ( θ ) =
expθ (cid:80)
k expθ ( k ) , then H ( π θ ) is the Jacobian of the
[Table]
[Equation]
[Table]
R K ×
[Table]
Lemma 4 (Smoothness) .
ˆ V π θ ( ρ ) is 8 M (1 − γ ) 3 -smooth.
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
i
[Table]
S
[Table]
is the i,j-th element of
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Deﬁne P
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
) For any vector x ∈ R S , the l ∞ norm is
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
a
[Equation]
[Table]
[Equation]
Consider the KL regularized value function of π θ α .
[Equation]
where e s is an indicator vector for the starting state s , M ( α )
=
( Id − P ( α )) − 1 =
(cid:80) ∞ t =0 γ t P ( α ) t , and r θ α ( s )
= (cid:80) a π θ α ( a | s )( r ( s, a )
+ τ log β ( a | s )) .
Taking derivative with respect to α :
[Equation]
Taking second derivative with respect to α :
[Equation]
[Table]
[Equation] [Equation]
is the
[Equation]
-th element of the second derivative )
[Equation]
i
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
For the third term of Eq.
[Equation]
For the last term of Eq.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Denote θ ξ = θ + ξ ( θ (cid:48) − θ ) , where ξ ∈ [0 , 1] , according to Taylor’s theorem,
[Equation]
Lemma 5 (Smoothness) .
H ( ρ, π θ ) is 4+8log A (1 − γ ) 3 -smooth, where A = |A| is the total number of actions.
Lemma 6 (Soft Policy Gradient) .
It holds that
[Equation] [Equation]
where ˜
A π θ ( s, a ) is the ’soft’ advantage function deﬁned as:
˜ A
[Equation]
s, a
[Equation]
=
[Equation]
Q
[Equation]
a
[Equation]
−
[Equation]
− ˜
[Equation]
a
[Equation]
Lemma 7.
d
[Equation]
s
[Equation]
Lemma 8 (Non-uniform Lojasiewicz) .
Suppose initial state distribution in the policy gradient algorithm µ ( s ) > 0 for all state s ∈ S
[Equation]
∂θ
[Equation]
d
[Equation]
Lemma 9.
Using the algorithm with soft policy gradient, we have inf t ≥
[Equation]
min s,a π θ t ( a | s ) >
The augmented value function ˜
V
π θ ( ρ ) is monotonically increasing following the gradient update with proper η due to smoothness.
˜ V π θ ( ρ ) is upper bounded as:
[Equation]
˜ V π θ ( ρ ) is lower bounded as:
[Equation]
Because entropy function is positive )
[Equation]
˜ Q
π θ ( s, a ) is lower bounded as:
[Equation]
According to monotone convergence theorem, ˜ V π θ ( ρ ) converges to a ﬁnite value, π θ t ( a | s ) → π θ ∞ ( a | s ) .
For any state s ∈ S , deﬁne the following sets: A 0 ( s ) =
{ a : π θ ∞ ( a | s ) = 0 } ,
A + ( s ) =
{ a : π θ ∞ ( a | s ) > 0 } .
We prove that A 0 ( s ) = ∅ by contradiction.
Suppose that ∃ s ∈ S ,such that A 0 ( s ) is non-empty.
For any a 0 ∈ A 0 ( s ) , we have π θ t ( a 0 | s ) →
π θ ∞ ( a 0 | s ) = 0 , which implies − log π
θ t ( a 0 | s ) → ∞ .
There exists t 0 > 0 , such that ∀ t ≥ t 0 , − log π θ t ( a 0 | s ) ≥ 2 M + τ log A τ (1 − γ ) .
According to Lemma 6, ∀ t ≥ t 0 :
[Equation]
This means θ t ( s, a 0 ) is always increasing ∀ t ≥ t 0 , which implies that θ ∞ ( s, a 0 ) is lower bounds by a constant c , and thus exp { θ ∞ ( s, a 0 ) } ≥ e c > 0 .
According to π θ ∞ ( a 0 | s ) =
[Equation]
} = 0 , we have (cid:80) a exp { θ ∞ ( s, a ) } = ∞ .
On the other hand, for any a + ∈ A + ( s ) , according to π θ ∞ ( a + | s ) =
exp { θ ∞ ( s,a + ) } (cid:80) a exp { θ ∞ ( s,a ) } > 0 , we have exp { θ ∞ ( s, a + ) } = ∞ , ∀ a + ∈ A + ( s ) , which implies (cid:80)
[Equation]
θ ∞
[Equation]
s, a +
[Equation]
= ∞ .
Note that ∀ t ,the summation of logit incremental over all actions is zero.
[Equation]
∀ t ≥ t 0 , (cid:80) a 0 ∈A 0 ( s ) ∂
˜ V πθt ( µ ) ∂θ t ( s,a 0 ) ≥ 0 , then (cid:80) a + ∈A + ( s ) ∂
˜ V πθt ( µ ) ∂θ t ( s,a + ) ≤ 0 .
So (cid:80) a + ∈A + ( s ) θ ∞ ( s, a + ) will always decrease for all large enough t > t 0 .
This is contradiction with (cid:80) a + ∈A + ( s ) θ ∞ ( s, a + ) = ∞ .
To this point, we have shown A 0 ( s ) = ∅ for any state s .
At the convergence point π θ ∞ ( ·| s ) , the gradient is zero.
[Equation]
H ( π θ ∞ ( ·| s ))
(cid:104)
˜ Q π θ ∞ ( s, · ) − τ log π θ ∞ ( ·| s ) (cid:105) = 0 .
Because of Lemma 7, d π θ ∞ µ ( s ) > 0 , for all state s .
Therefore H ( π θ ∞ ( ·| s ))
(cid:104)
˜ Q π θ ∞ ( s, · ) − τ log π θ ∞ ( ·| s ) (cid:105) = 0 .
H ( π θ ∞ ( ·| s )) has eigenvaluewith multiplicity 1, and the corresponding eigenvector is c · 1 for some constant c ∈ R .
Therefore,
˜ Q π θ ∞ ( s, · ) − τ log π θ ∞ ( ·| s )
=
c · 1 , which is equivalent to π θ ∞ ( ·| s ) = softmax ( ˜ Q π θ ∞ ( s, · ) /τ ) .
Because τ ∈ Ω(1) > 0 , − M 1 − γ ≤ ˜ Q π θ ∞ ( s, a ) ≤ M + τ log A 1 − γ , we have π θ ∞ ( a | s ) ∈ Ω(1) .
Since π θ t ( a | s ) → π θ ∞ ( a | s ) > 0 , there exists t 0 > 0 such that ∀ t ≥ t 0 , 0 .
9 π θ ∞ ( a | s ) ≤ π θ t ( a | s ) ≤ 1 .
1 π θ ∞ ( a | s ) , which means inf t ≥ t 0 min s,a π θ t ( a | s ) ∈ Ω(1) .
[Equation]
Theorem 4.
Suppose µ ( s ) > 0 for all state s .
Using entropy regularized softmax policy gradient algorithm with η = (1 − γ ) 3 (8 M + τ (4+8log A )) and π θ 1 ( a | s ) ∈ Ω(1) , ∀ ( s, a ) ,
[Equation]
for all t > 0 , where C τ , Ω(1) > 0 are independent with t .
According to the soft sub-optimality lemma,
[Equation]
Because of the property of smoothness, ˜ V π θ ( µ ) =
ˆ V π θ ( µ ) +
[Equation]
H ( µ, π θ ) is λ -smooth with λ = 8 M + τ (4+8log A ) (1 − γ ).
Denote ˜ δ t =
˜ V π ∗ τ ( µ ) − ˜ V π θt ( µ ) , ˜ δ t − 1 − ˜ δ t =
˜ V π θt ( µ ) − ˜ V π θt +1 ( µ ) ≤ − (cid:42) ∂
˜ V π θt ( µ ) ∂θ t , θ t +1 − θ t (cid:43) + 4 M + τ (2 + 4 log A ) (1 − γ )(cid:107) θ t +1 − θ t (cid:107) 2= (cid:18) − η + 4 M + τ (2 + 4 log A ) (1 − γ )· η (cid:19) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∂
˜ V π θt ( µ ) ∂θ t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  = − (1 − γ )
M + τ (8 + 16 log A ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∂
˜ V π θt ( µ ) ∂θ t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  ( Because η = (1 − γ )8 M + τ (4 + 8 log A )) ≤ − (1 − γ )
M + τ (8 + 16 log A ) · 2 τ
S · min s µ ( s ) · (cid:20) min s,a π θ t ( a | s ) (cid:21)· (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) d π ∗ τ µ d π θt µ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
−∞ · (cid:104) ˜
V π ∗ τ ( µ ) − ˜ V π θt ( µ ) (cid:105) ≤ − (1 − γ )(8 M/τ + 4 + 8 log A ) · S · min s µ ( s ) (cid:20) min s,a π θ t ( a | s ) (cid:21)· (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
d π ∗ τ µ (1 − γ ) µ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
d
[Equation]
[Table] [Table]
[Table] [Table]
[Equation]
(cid:21)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
µ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
−∞ ( t − 1)  
M +
[Equation]
log A 1 − γ (37)
Thus
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
µ (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
∞ ( t − 1)  
M
[Equation]
A
[Equation]
γ
[Equation]
38) where C τ =
[Equation]
[Table]
[Figure]
C E XPERIMENTAL D ETAILS C.
G RID W ORLD
We design the grid world of size 9 × 9 .
The action space includes actions: up, down, left, right.
The reward is 0 at most states, while there are positive rewards 0.9 and 1 at two terminal states.
In each episode, the agent starts from the state in the center annotated with ’S’, walks around the environment, and terminates the episode only when the agent visits any of the four terminal states (yellow squares at the edge of the grid work).
As explained in Section 3.3, we study the performance of soft policy iteration algorithm with different value of τ .
The behavior policy is mediocre and tends to move left and down with higher probability at each state.
We collect 10000 transitions in the domain according to the behavior policy .
We visualize the behavior policy in Fig.
In each state, the length of the arrow is proportional to the probability of taking the actions in four directions.
We also visualize the visitation count of each state in the collected dataset.
The darker color means more visitation.
We can see the behavior policy mostly move around the left bottom corner.
We consider optimizing ˜
V
π,τ ( ρ ) with different value of τ .
It could be constant value in { 1 , 0 . 1 , 0 . 01 , 0 . 001 , 0 } .
For the continuation method, we initially set the value of τ as 1, and decay it with τ ← 0 .
1 τ at every 100 iteration.
We search the hyper-parameter of learning rate in { 5 , 1 , 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 , 0 . 001 } .
The learning curves in Fig.
5 verify that the continuation method has better performance than the baselines with a ﬁxed constraint.
Visualization of the learned policies for different methods is in Fig.
With a constant τ = 1 , the learned policy performs similarly to the behavior policy.
With a constant τ = 0 .
001 , the learned policy crashes because the value estimate is noisy.
With decaying value of τ , the error in Q value estimate is reduced and the relaxing constraint allows the agent to deviate much from the behavior policy.
C. M UJOCO
As mentioned in the main paper, the architecture of the critic network and the policy network is kept the same in the baselines and our method.
The critic network consists of 4 independent Q–networks, which share the same feedforward network architecture.
The state and action are concatenated and then passed through 2 hidden layers of size 400 and 300.
The policy network is a simple linear network, with the input and output corresponding to the state and the action respectively.
ReLU is used as the activation function in all hidden layers.
We use 2 separate Adam optimizers for policy and critic network learning and the learning rates are set to 0 .
For all the baselines, we do a grid search of their algorithm–speciﬁc hyper–parameters for each game and each dataset.
This parameter is ﬁxed in 5 independent runs.
C.2. C ONTINUOUS BCQ
For continuous BCQ, we mostly follow their ofﬁcial implementation 2 and hyper–parameter settings.
BCQ uses a perturbation model to adjust the action output with an added residual, which is in the range of [ − Φ , Φ ].
We do a grid search of the max perturbation Φ over { 0 . 1 , 0 . 2 , 0 . 3 } .
C.2. BEAR
For BEAR, we also follow the ofﬁcial implementation 3 .
In BEAR, a threshold (cid:15) is used to constrain the MMD distance between the unknown behaviour policy β and the target policy π .
It is set to be 0.05 in the original experiments.
We instead do a grid search of (cid:15) over { 0 . 05 , 0 . 1 , 0 . 2 } .
BEAR also uses a VAE network to generate actor with distribution similar as the unknown behavior policy.
We keep the same architecture as their implementation for VAE and use an Adam optimizer with learning rate 0 .
C.2. ABM+SVG
For ABM, we use the batch size of 100, target network update period 1.
The KL divergence between the learned policy and the prior policy is constrained by (cid:15) in ABM, which is set to 0.2 for SVG in the original work.
We do a grid search of (cid:15) over { 0 . 05 , 0 . 1 , 0 . 2 } .
For the additional prior policy network, we also use an Adam optimizer with learning rate 0 . 001 .
C.2. CRR
In CRR training, we use the batch size of 128, target network update period 1.
We use 4 samples to compute the advantage as the original paper does.
For fair comparison, we keep the critic architecture the same as others, instead of the distributional one.
We focus on the ’mean exp’ variant of CRR, because it performs well in CRR paper, while another variant ’max binary’ is roughly equivalent to
ABM+SVG, which we have run as a separate baseline.
To decide on the appropriate temperature in the scalar function f , we swept β over { 0 . 01 , 0 . 1 , 1 , 10 } , where β is ﬁxed as 1 in the original paper.
C.2. CQL
For CQL, we mostly follow the ofﬁcial implementation 4 but make the network architecture same as the other methods.
In CQL, α is the coefﬁcient to control the conservative, lower bound Q–function and there are two variants, CQL ( ρ ) and CQL ( H ) .
We search the hyper-parameter for both variants on our datasets.
For CQL ( ρ ) , we search the Lagrange threshold τ over { 0 . 1 , 0 .
For
CQL ( H ) , we search the ﬁxed tradeoff factor α over { 0 . 1 , 1 , 10 , 100 , 1000 , 10000 } .
Besides, since our buffer size is 1 million, a half of the buffer size in the original D4RL experiments, we also reduce the batch size from 256 to 100 to keep it consistent with the other methods.
The remaining hyper parameters are kept the same as the default value in CQL.
C.2. C ONSTANT
To compare with the cases that there’s no decay of the KL weight in our method, we set KL weight to be 10, 1, 0.1 separately and ﬁx the decay to be 1.
We run this method on the dataset with α = 0 .on
all three games.
C.2. BRAC
For BRAC, we adapt the code of the ofﬁcial implementation 5 to use the same network architecture and datasets as the other methods.
BRAC has different design choices (value penalty and/or policy regularization).
We use both value penalty in the Q function objective and policy update with KL regularization, which is reported to have better performance than policy regularization only.
We use the KL divergence in the primal form to keep the setting the same as our method.
We search the hyper-parameter for both ﬁxed and adaptive version to set the regularization coefﬁcient α .
For the ﬁxed version, we search α over the range { 0 . 1 , 1 , 10 } . For the adaptive version, we search the threshold (cid:15) , which is the same as the threshold used in BEAR, over the range { 0 . 05 , 0 . 5 , 2 } .
We ﬁnd the ﬁxed version has a better performance, which is consistent with the conclusion from BRAC paper.
Besides, we also change the learning rate of the policy network to 0 .
001 , which is the same as ours.
Other hyper-parameters are kept the same as the default value.
The ﬁxed version of BRAC is equivalent to our ”Constant” baseline, though there are many differences in implementation details.
For example, following the ofﬁcial implementation of BRAC, there is a regularization for the weight of network parameters in BRAC baseline but ”Constant KL” baseline does not have this term.
BRAC has two training phases, the training of the behavior policy for 30K updates, and the training of BRAC with the ﬁxed behavior policy trained before.
However, ”Constant” trains the behavior policy simultaneously throughout the whole procedure.
In the computation of KL divergence, BRAC samples 4 actions for one state to approximate the KL divergence but ”Constant” uses the closed–form solution to calculate KL divergence between two normal distributions.
C.2. O URS
Initially in the ﬁrst J iterations we learn behavior policy β ψ from the data and train the target policy π θ only minimizing KL divergence between π θ and β ψ .
To ﬁnd the well-trained policy with good performance before the value estimate becomes quite noisy, we record the variance in Q estimate var ( Q π θi ) for each update i .
In each run, we calculate the average of the variance in 1000 iterations at the end of behavior cloning, i.e. x =
[Equation]
1000 (cid:80)
[Equation] i = [Equation] − 1000 var π θi ( Q ) .
This is a reference point of the Q variance for this run.
As training continues, at the iteration j , we monitor the average of Q variance in the most recent 1000 iterations y =
[Equation]
(cid:80)
[Equation]
=
[Equation]
var ( Q π θi ) . If the average of Q variance is larger than 1.5 times the reference point, i.e. yx > 1 . 5 , the current value of τ may be not reliable and we report the score of policy checkpointed with the previous value of τ .
Until the end of training, if we always have yx ≤ 1 . 5 , then we ﬁnd the policy iteration is stable and we report the ﬁnal performance of the trained agent.
For our continuation method, accross the different datasets, we set most hyper-parameters the same.
For the ﬁrst J = 500 K updates, we train the policy network with only KL divergence loss to conduct behavior cloning, which is equivalent to the case that τ → ∞ .
We set the decay rate λ = 0 .
9 and decay the weight KL term with τ ← λ ∗ τ for every I = 10000 updates.
We conduct hyper-parameter search over the initial value of τ in the set { 10000 , 1000 , 100 } .
C. A TARI
To generate the dataset D for the Atari games, we train the DQN agents using the standard online procedure to 10 million timesteps on the environment with sticky action.
To generate the trajectory, with probability of 0.8, we uses (cid:15) -greedy with noise (cid:15)
= 0 .
2 for the whole episode to take actions.
With probability of 0.2, we use the noise (cid:15) = 0 .
001 for the whole trajectory.
As explained in [13], we can ensure the dataset includes trajectories reaching the good performance of the DQN agent as well as trajectories with exploratory behavior.
C.3. DISCRETE BCQ
We refer to the ofﬁcial implementation of discrete BCQ[13], training the the behavioral policies, generating datasets and training the BCQ model.
The 84 × 84 × 3 RGB image is fed into a convolutional neural network.
The input image ﬁrst goes through an 8 × 8 convolution with 32 ﬁlters and stride 4, then a 4 × 4 convolution with 64 ﬁlters and 2 stride, followed by a 3 × 3 convolution with 64 ﬁlters and 1 stride, with ReLU activations.
There are 2 heads, each being a fully connected layer, for Q–network output and generative model output after the convolutional neural network.
Both fully connected layers have hidden size 512 and use the ReLU activation.
The convolutional neural network is shared between the Q–network and the generative model.
A ﬁnal softmax layer is used after the output of the generative model to recover the probability for each action.
We search of BCQ threshold over { 0 . 1 , 0 . 3 , 0 . 5 } to ﬁnd the best parameter for each game.
We use the same hyper–parameter as the original implementation for training and only change the evaluation (cid:15) in the testing time to keep consistent with other baselines and our method.
C.3. REM
For REM training, we completely follow the implementation from the ofﬁcial codebase 6 released by [2].
The architecture is exactly the same as [2] while the dataset is from a single behavior policy.
shared with discrete BCQ and our method.
We use the multi-head architecture with 200 heads and do not change any hyper parameter.
C.3. CQL
The ofﬁcial implementation of CQL 7 is based on REM, so we use the same way to adapt the code and run experiments on our dataset.
We follow CQL experiments on Atari, using CQL ( H ) with a ﬁxed tradeoff factor α .
We search α over { 0 . 5 , 1 , 4 } for the best hyper-parameter for each game.
Other hyper-parameters are kept the same as the default setting in the ofﬁcial code.
C.3. O URS
In our method, we also use an ensemble critic network as we do in Mujoco.
The architecture of each Q-network in the critic network is the same as the Q-network architecture used by discrete
BCQ, with a 3–layer convolution and a fully connect layer.
The policy network also uses the same architecture.
For the ﬁrst J = 500 K updates, we train the policy network with only KL divergence loss to conduct behavior cloning, which is equivalent to the case that τ → ∞ .
We set initial weight of the KL term as 1 , the decay rate λ = 0 .
9 and decay the weight KL term with τ ← λ ∗ τ for every I =updates.
We pick the learned policy with a reasonable value of τ using the same way as in Mujoco experiment.
C. R ECOMMENDER
Here we explain more details about the experiments for a softmax recommender.
In the MovieLen10M, the rating is of 5-score.
Similarly to [25], we view the scores 4 or 5 as positive feedback from users but the ratings less than 4 as negative feedback.
In the bandit setting, the expected return of the policy π can be expressed as V π ( ρ ) =
E s ∼ ρ,a ∼ β ( ·| s ) ( π ( a | s ) β ( a | s ) r ( s, a )) .
So we do not use the critic network to estimate the soft Q value.
We convert the objective ˜ V π,τ ( ρ ) to E s ∼ ρ,a ∼ β ( ·| s ) ( π ( a | s ) β ( a | s ) r ( s, a )) −
[Equation]
E s ∼ ρ KL ( π ( ·| s ) | β
·| s )) in the bandit setting.
Here the initial state distribution should be sampling the users randomly from the dataset, and we set the objective function as J Ours (
[Equation]
) =
[Equation]
N
[Equation]
The architecture of the simulator, the behavior policies and target policy are the same as introduced in [25].
The simulator is trained with all 1 million records of (user, movie, feedback) in the
MovieLen-1M dataset.
We train the behavior policy with 500, 5000, 10000 samples respectively to get the behavior policy with different performance.
Then 6040 users in Movie-Len dataset, the movie can be selected from the behavior policy, and the simulator outperforms the feedback.
If the behavior policy recommends 5 movies out of the 3900 choices for each user, we have a dataset D with around 30,000 samples.
If the behavior policy recommend 10 movies for each user, we have a dataset D with around 60,000 samples
With the generated datasets of different sizes and different qualities, we train the target policy π θ using three methods: ’Cross-Entropy’, ’IPS’ and ’Ours’.
The objective functions are optimized for 400 epochs, with Adagrad optimizer and the learning rate 0.05 and the batch size 256.
For our continuation method, the value of τ is initialized as 1 and decay with the rate λ
= 0 .
9 in every epoch.
The users in MovieLens-1M dataset are split into validation set (2000 users) and test set (4040 users).
In each epoch, we measure the precision at 10 for the learned policy on the validation set.
We take the policy achieving best performance on the validataion set during training, and test it for the held out users to report the performances in Table 3.
D V ARIANCE OF E NSEMBLE C RITIC N ETWORKS
We found the error in Q estimation is highly correlated with the variance of ensemble critic network.
If the learned policy π tends to select the state-action pairs with high variance in ensemble networks, the error in the Q estimation on these state-action pairs will propagate to the other state-action pairs and mislead the policy update.
So we measure E a ∼ π (cid:2) var ( Q φ (1) ( s, a ) , Q φ (2) ( s, a ) , · · · , Q φ ( K ) ( s, a )) (cid:3) as the criteria to determine proper value of τ .
When this statistics is quite large, the policy tends to select actions with highly erroneous value estimation and we no longer trust the policy with smaller value of τ .
D. T RAIN THE C RITIC N ETWORKS WITH THE S AME D ATA
When we train the critic networks with the same data, the discrepancy in value estimation comes from the initialization of different critic networks.
For state-action pairs frequently visited by the behavior policy, these data are frequently sampled to update the Q network.
Therefore, the error in Q estimation and variance in Q ensemble on these state-action pairs diminishes as training goes on.
But for state-action pairs less frequently visited by the behavior data, they are rarely used to update the Q network.
The variance in Q ensemble are larger compared to the other state-action pairs.
In Figure 6, for each state-action pair in the grid world environment, we visualize the error | 1 K (cid:80)
Kk =1 Q Φ ( k ) ( s, a ) − ˜ Q π,τ ( s, a ) | and the variance var ( Q φ (1) ( s, a ) , Q φ (2) ( s, a ) , · · · , Q φ ( K ) ( s, a )) in the grid world for each state-action pair.
It shows that the error becomes highly correlated with the variance as training goes on.
Obviously, the frequently visited state-action pairs in the dataset has lower error and variance during training.
[Figure]
D. T RAIN THE C RITIC N ETWORKS WITH THE D IFFERENT D ATA
We have an alternative way to train the critic network with delete-d jackknife resampling.
It is a more standard approach to estimate the variance through data perturbation.
To train each of 4 critic networks, we leave out 14 data samples in the dataset.
We assume ˆ Q ( s, a ) is an estimator of ˜ Q π,τ ( s, a ) .
We consider the expected squared estimation error.
E
[Equation]
The ﬁrst term is the bias of the estimator and the second term is the variance of the estimator.
In our case, we use the average of the ensemble Q networks ˆ Q ( s, a ) = 1 K (cid:80)
Kk =1 Q φ ( k ) ( s, a ) to estimate the soft Q value.
With jackknife re-sampling, the variance of the estimator ˆ Q ( s, a ) = 1 K (cid:80)
Kk =1 Q φ ( k ) ( s, a ) can be approximated by var ( Q φ (1) ( s, a ) , Q φ (2) ( s, a ) , · · · , Q φ ( K ) ( s, a )) = 1 K ( K − 1) (cid:80)
Kk =1 (cid:104) Q φ ( k ) ( s, a ) − 1 K (cid:80) Kj =1 Q φ ( j ) ( s, a ) (cid:105) 2 .
Thus, the squared estimation error is closely related with the variance in the ensemble Q φ (1) ( s, a ) , Q φ (2) ( s, a ) , · · · , Q φ ( K ) ( s, a ) .
With this method, the empirical results on Mujoco and Atari are similar to the scores we achieved when training critic networks with the same data (Table 4& 5).
Therefore, we validate the use of variance of the ensemble network to detect the Q estimation error and then stop annealing τ on
Mujoco and Atari as well.
Hopper
HalfCheetah
Walker
[Table]
Same
[Table]
Jackknife
[Table]
E S OFT P OLICY I TERATION VIA L AGRANGIAN D UALITY
The discounted state-action visitations d π of π can be deﬁned as:
[Equation]
The visitation satisﬁes the single-step transpose Bellman recurrence:
[Equation]
where P π ∗ is the transpose policy transition operator:
[Equation]
Note that if we deﬁne the policy transition operator
[Equation]
We have (cid:80) s,a y ( s, a ) P π ∗ x ( s, a ) =
(cid:80) s,a x ( s, a ) P π y ( s, a ) .
The objective function ˜ V π,τ ( ρ ) can be expressed as ˜ V π,τ ( ρ ) =
(cid:80) s,a d π ( s, a ) r ( s, a ) − τ (cid:80) s,a d π ( s, a ) log π ( a | s ) β ( a | s ) .
Consider the optimization problem:
[Equation]
Application of Lagrange duality to the above problem yields max π min Q L ( Q, π ) where
[Equation]
and Q ( s, a ) are dual variables
[Equation]
We can rewrite L ( Q, π ) :
[Equation]
,a
Q ( s, a ) ρ ( s ) π ( a | s ) − τ (cid:88) s,a ρ ( s ) π ( a | s ) log π ( a | s ) β ( a | s ) + (cid:88) s,a d π ( s, a ) r ( s, a ) + γ (cid:88) s,a ( Q ( s, a ) − τ log π ( a | s ) β ( a | s ))
P π ∗ d
π ( s, a ) − (cid:88) s,a d π ( s, a ) Q ( s, a )
= (cid:88) s,
a
Q ( s, a ) ρ ( s ) π ( a | s ) − τ (cid:88) s,a ρ ( s ) π ( a | s ) log π ( a | s ) β ( a | s ) + (cid:88) s,a d π ( s, a ) r ( s, a ) + γ (cid:88) s,a d π ( s, a ) P π ( Q ( s, a ) − τ log π ( a | s ) β ( a | s ) ) − (cid:88) s,a d π ( s, a ) Q ( s, a ) =
E s ∼ ρ (cid:2)
E a ∼ π ( ·| s ) Q ( s, a ) − KL ( π ( ·| s ) (cid:107) β ( ·| s )) (cid:3) + (cid:88) s,a d π ( s, a ) (cid:20) r
( s, a ) + γ P π
( Q ( s, a ) − τ log π ( a | s ) β ( a | s ))
− Q ( s, a ) (cid:21)
In our soft policy evaluation step, we repeatedly apply the soft Bellman operator.
At the convergence, the Q value satisﬁes T π,τ Q ( s, a ) =
Q ( s, a ) , i.e. r ( s, a )+ γ P π ( Q ( s, a ) − τ log π ( a | s ) β ( a | s ) )
− Q ( s, a ) = 0 .
Thus, the second part of L ( Q, π ) is 0.
In our soft policy improvement step, for each state s , we optimize π ( ·| s ) to maximize E a ∼ π ( ·| s ) Q ( s, a ) − KL ( π ( ·| s ) (cid:107) β ( ·| s )) .
Thus, the ﬁrst part of L ( Q, π ) is maximized.
Therefore, the dual objective function bears some resemblance to the our soft policy iteration.
F P ERFORMANCE ON D ATASETS OF D IFFERENT Q UALITY
We study the performance of our method on datasets of different quality.
As shown in Figure 7, our method is able to learn a policy superior to the behavior policy and the gap is obvious especially when α ∈ [0 . 4 , 0 . 6] .
In other words, when the behavior policy is mediocre, our method has a good chance to learn from the good trajectories in the dataset and achieve a better score than the average score of the dataset.
We further investigate the distribution of the trajectory rewards in the datasets.
In Figure 7, we observe that our method does not perform so well in comparison with the behavior policy on the datasets (Hopper, α = 0 . 2 ), (Hopper, α = 0 . 8 ), (Walker, α = 0 . 2 ), (Walker, α = 0 . 8 ).
In Figure 8, we ﬁnd the problem of these four dataset is that there is only a small fraction of trajectories with cumulative rewards better than the average trajectory reward plus the standard deviation in the dataset.
We study the percentage of trajectories with reward better than one standard deviation of the mean (i.e. %1 σ trajectory).
In Figure 9, we see the percentage of 1 σ trajectory highly correlates with the improvement, where the improvement is the average trajectory reward of our learned policy minus the average trajectory reward of the dataset.
If there is a reasonable proportion (i.e. > 10% ) of %1 σ trajectory in the dataset, our method can learn from these good trajectories and perform signiﬁcantly better than the average trajectory reward in the dataset.