Original: K EEP C ALM AND C ARRY O FFLINE : P OLICY R EFINEMENT IN O FFLINE R EINFORCEMENT L EARNING Anonymous authors Paper under double-blind review
Revised: Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning

Original: A BSTRACT
Revised: Abstract

Original: The ability to discover optimal behaviour from fixed data sets has the potential to transfer the successes of reinforcement learning (RL) to domains where data collection is acutely problematic.
Revised: The ability to discover optimal behaviour from fixed data sets has the potential to transfer the successes of reinforcement learning (RL) to domains where data collection is acutely problematic.

Original: In this offline setting a key challenge is overcoming overestimation bias for actions not present in data which, without the ability to correct for via interaction with the environment, can propagate and compound during training,thus leading to highly sub-optimal policies.
Revised: In this offline setting, a key challenge is overcoming overestimation bias for actions not present in data which, without the ability to correct for via interaction with the environment, can propagate and compound during training, leading to highly sub-optimal policies.

Original: One simple method to reduce this bias is to introduce a policy constraint via behavioural cloning (BC), which encourages agents to pick actions closer to the source data.
Revised: One simple method to reduce this bias is to introduce a policy constraint via behavioural cloning (BC), which encourages agents to pick actions closer to the source data.

Original: By finding the right balance between RL and BC such approaches have been shown to be surprisingly effective while requiring minimal changes to the underlying algorithms they are based on.
Revised: By finding the right balance between RL and BC such approaches have been shown to be surprisingly effective while requiring minimal changes to the underlying algorithms they are based on.

Original: To date, this balance has been held constant but in this work we explore the idea of tipping this balance towards RL following initial training.
Revised: To date this balance has been held constant, but in this work we explore the idea of tipping this balance towards RL following initial training.

Original: Using TD3BC we demonstrate that by continuing to train a policy offline while reducing the influence of the BC component we can produce refined policies that outperform the original baseline, as well as match or exceed the performance of more complex alternative approaches.
Revised: Using TD3-BC, we demonstrate that by continuing to train a policy offline while reducing the influence of the BC component we can produce refined policies that outperform the original baseline, as well as match or exceed the performance of more complex alternatives.

Original: Furthermore, we show these refined policies can be fine-tuned online while largely mitigating severe performance drops.
Revised: Furthermore, we demonstrate such an approach can be used for stable online fine-tuning, allowing policies to be safely improved during deployment.

Original:  I NTRODUCTION
Revised:  Introduction

Original: In offline reinforcement learning (offline-RL) (Lange et al., 2012) an agent seeks to discover an optimal policy purely from a fixed data set.
Revised: In offline reinforcement learning (offline-RL) [15] an agent seeks to discover an optimal policy purely from a fixed data set.

Original: This is a challenging task as, unlike traditional reinforcement learning, an agent can no longer interact with the environment to assess the quality of exploratory actions and thus cannot correct for errors in value estimates.
Revised: This is a challenging task as, unlike traditional reinforcement learning, an agent can no longer interact with the environment to assess the quality of exploratory actions and thus cannot correct for errors in value estimates.

Original: These errors can propagate and compound during training leading to highly sub-optimal policies or, in the worst case, a complete failure to learn (Levine et al., 2020).
Revised: These errors can propagate and compound during training leading to highly sub-optimal policies or, in the worst case, a complete failure to learn [16].

Original: A common approach to overcoming this problem is to perform some kind of regularisation during training, encouraging updates during policy evaluation and/or policy improvement to stay close to underlying data.
Revised: A common approach to overcoming this problem is to perform some kind of regularisation during training, encouraging updates during policy evaluation and/or policy improvement to stay close to underlying data.

Original: One way to achieve this is to modify existing off-policy algorithms to directly incorporate behavioural cloning (BC), a technique that has been shown to be computationally efficient, easy to implement and able to produce policies that compete with more complex alternatives (Fujimoto & Gu, 2021).
Revised: One way to achieve this is to modify existing off-policy algorithms to directly incorporate behavioural cloning (BC), a technique that has been shown to be computationally efficient, easy to implement and able to produce policies that compete with more complex alternatives [6].

Original: What’s more, such approaches require minimal changes to the underlying algorithms they are based on, making performance gains easier to attribute and hyperparameter tuning less burdensome, both desirable properties where online interaction is limited.
Revised: Furthermore, such approaches require minimal changes to the underlying algorithms they are based on, making performance gains easier to attribute and hyperparameter tuning less burdensome, both desirable properties when online interaction is limited.

Original: Finding the optimal level of behavioural cloning is key; too little runs the risk of error propagation but too much limits the agent’s ability to learn policies beyond those which generated the data.
Revised: Finding the optimal level of behavioural cloning is key: too little runs the risk of error propagation, but too much limits the agent’s ability to learn policies beyond those which generated the data.

Original: To date, such approaches have maintained a constant level throughout training, but in this work we explore the idea of reducing the influence of behavioural cloning following an initial period of learning.
Revised: To date, such approaches have maintained a constant level throughout training, but in this work we explore the idea of reducing the influence of behavioural cloning following an initial period of learning.

Original: Using TD3-BC as the foundation we show that by continuing to train offline in this way we are able to produce refined policies that improve over the baseline, particularly in cases wherethe data is sub-optimal.
Revised: Using TD3-BC as the foundation, we show that by continuing to train offline in this way we are able to produce refined policies that improve over the baseline, particularly in cases where data is sub-optimal.

Original: In addition, we show we are able to transition to online fine-tuning while maintaining a similar level of performance without having to make any additional implementation adjustments.
Revised: Deployment of policies following offline training opens up the possibility of further improvement via online fine-tuning. Such a transition should ideally result in negligible performance drops, but the sudden change in setting often means policies get worse before they get better [24]. We investigate whether our offline approach can also be extended to this setting and find we are able to fine-tune both baseline and refined policies while largely mitigating these stability issues.

Original: Despite the simplicity of our of approach we are able to match or exceed the performance of stateof-the-art alternatives.
Revised: For both offline learning and online fine-tuning our approach maintains the simplicity of TD3-BC, yet we are able to match or exceed the performance of state-of-the-art alternatives.

Original:  B ACKGROUND
Revised:  Background

Original: We follow standard practice and define a Markov decision process (MDP) with state space S , action space A , transition dynamics T ( s ′ | s, a ) , reward function R ( s, a ) and discount factor γ ∈ [0 , 1] (Sutton & Barto, 2018).
Revised: We follow previous work and define a Markov decision process (MDP) with state space S , action space A , transition dynamics T ( s ′ | s, a ) , reward function R ( s, a ) and discount factor γ ∈ [0 , 1] [20].

Original: An agent interacts with this MDP by following a policy π ( a | s ) , which can be deterministic or stochastic.
Revised: An agent interacts with this MDP by following a policy π ( a | s ) , which can be deterministic or stochastic.

Original: The goal of reinforcement learning is to discover an optimal policy π ∗ ( a | s ) that maximises the expected discounted sum of rewards E π ( (cid:80) ∞ t =

Revised: The goal of reinforcement learning is to discover an optimal policy π ∗ ( a | s ) that maximises the expected discounted sum of rewards E π ( (cid:80) ∞ t =

Original: γ t r ( s t , a t )) .
Revised: γ t r ( s t , a t )) .

Original: In Actor-Critic methods this is achieved by alternating between policy evaluation and policy improvement using Q-functions Q π ( s, a ) , which estimate the value of taking action a in state s following policy π thereafter.
Revised: In Actor-Critic methods this is achieved by alternating between policy evaluation and policy improvement using Q-functions Q π ( s, a ) , which estimate the value of taking action a in state s following policy π thereafter.

Original: Policy evaluation consists of updating Q-values based on the Bellman expectation equation Q π ( s, a ) =
Revised: Policy evaluation consists of updating Q-values based on the Bellman expectation equation Q π ( s, a ) =

Original: r ( s, a )+ γ E s ′ ∼ T,a ′ ∼ π ( Q π ( s ′ , a ′ )) with improvement coming from updating policy parameters to maximise Q π ( s, a ) .
Revised: r ( s, a ) + γ E s ′ ∼ T,a ′ ∼ π ( Q π ( s ′ , a ′ )) with improvement coming from updating policy parameters to maximise Q π ( s, a ) .

Original: In offline reinforcement learning the agent no longer has access to the environment and instead must learn solely from pre-existing interactions D
Revised: In offline reinforcement learning the agent no longer has access to the environment and instead must learn solely from pre-existing interactions D

Original: = ( s i , a i , r i , s ′ i ) .
Revised: = ( s i , a i , r i , s ′ i ) .

Original: In this setting, agents struggle to learn effective policies due to overestimation bias for state-action pairs absent in D , which compounds and propagates during training leading to highly sub-optimal policies or a complete collapse of the learning process (Levine et al., 2020).
Revised: In this setting, agents struggle to learn effective policies due to overestimation bias for state-action pairs absent in D , which compounds and propagates during training leading to highly sub-optimal policies or a complete collapse of the learning process [16].

Original: TD3-BC (Fujimoto & Gu, 2021) is an adaptation of TD3 (Fujimoto et al., 2018) to the offline setting, adding a behavioural cloning term to policy updates to encourage agents to stay close to actions in D .
Revised: TD3-BC [6] is an adaptation of TD3 [7] to the offline setting, adding a behavioural cloning term to policy updates to encourage agents to stay close to actions in D .

Original: The updated policy objective is:
Revised: The updated policy objective is:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] The hyperparamter α controls the balance between reinforcement learning and behavioural cloning.
Revised: The hyperparamter α controls the balance between reinforcement learning and behavioural cloning.

Original: In order to keep this balance in check, Q-values (which scale with rewards and hence can vary across tasks) are normalised by dividing by the mean of the absolute values.
Revised: In order to keep this balance in check, Q-values (which scale with rewards and hence can vary across tasks) are normalised by dividing by the mean of the absolute values.

Original: States are also normalised to approximate an N (0 , 1) distribution:
Revised: States are also normalised to have mean 0 and standard deviation 1 :

Original: [Equation]
Revised: [Equation]

Original: where ϵ = 10 − 3 is a small normalisation constant.
Revised: where ϵ = 10 − 3 is a small normalisation constant.

Original: The only remaining task it to find a value of α that both prevents the propagation of overestimation bias and allows the agent to learn a policy that improves on the data.
Revised: The only remaining task it to find a value of α that both prevents the propagation of overestimation bias and allows the agent to learn a policy that improves on the data.

Original:  R ELATED WORK
Revised:  Related work

Original: While TD3-BC uses behavioural cloning explicitly in policy updates, other approaches such as BCQ (Fujimoto et al., 2019), PLAS (Wu et al., 2019), BRAC (Wu et al., 2019) and BEAR (Kumar et al., 2019) utilise it more implicitly, training a ConditionalVariational AutoEncoder (CVAE)
Revised: While TD3-BC uses behavioural cloning explicitly in policy updates, other approaches such as BCQ [8], PLAS [22], BRAC [22] and BEAR [13] utilise it more implicitly, training a Conditional

Original: (Sohn et al., 2015) to mimic the data distribution and either optimise CVAE generated actions (BCQ, PLAS) or minimise divergence metrics between the CVAE and the policy (BRAC, BEAR).
Revised: Variational AutoEncoder (CVAE) [19] to mimic the data distribution and either optimise CVAE generated actions (BCQ, PLAS) or minimise divergence metrics between the CVAE and the policy (BRAC, BEAR).

Original: Such methods do successfully combat overestimation bias but tend to perform worse than simpler methods such as TD3-BC, particularly for multimodal distributions which are hard to model accurately.
Revised: Such methods do successfully combat overestimation bias, but tend to perform worse than simpler methods such as TD3-BC, particularly for multimodal distributions which are hard to model accurately.

Original: Other examples of using behavioural cloning less directly include Implicit Q-learning (IQL) (Kostrikov et al., 2021b) which combines expectile regression and advantage weighted behavioural cloning to train agents without having to evaluate actions outside the dataset, and Fisher-BRC (Kostrikov et al., 2021a) which uses a cloned behaviour policy for critic regularisation via the Fisher divergence metric.
Revised: Other examples of using behavioural cloning less directly include Implicit Q-learning (IQL) [12] which combines expectile regression and advantage weighted behavioural cloning to train agents without having to evaluate actions outside the dataset, and Fisher-BRC [11] which uses a cloned behaviour policy for critic regularisation via the Fisher divergence metric.

Original: Both methods perform comparably to TD3-BC but also suffer to the same degree when learning from sub-optimal data.
Revised: Both methods perform comparably to TD3-BC but also suffer to the same degree when learning from sub-optimal data.

Original: Conservative Q-learning (CQL) (Kumar et al., 2020) focuses specifically on critic regularisation by augmenting the Bellman expectation equation by “pushing down” on Q ( s, a ) for actions outside the dataset and “pushing up” on Q ( s, a ) for actions in the dataset.
Revised: Conservative Q-learning (CQL) [14] focuses specifically on critic regularisation by augmenting the Bellman expectation equation by “pushing down” on Q ( s, a ) for actions outside the dataset and “pushing up” on Q ( s, a ) for actions in the dataset.

Original: CQL performs well across a wide range of tasks but is computationally expensive and often relies on implementation tweaks to achieve such levels of performance (Fujimoto & Gu, 2021).
Revised: CQL performs well across a wide range of tasks but is computationally expensive and often relies on implementation tweaks to achieve such levels of performance [6].

Revised: Online fine-tuning following offline learning has largely focused on final performance, but approaches such as REDQ+AdaptiveBC [24] also recognise the importance of stability.

Revised: By adapting the BC component based on online returns, REDQ+AdaptiveBC is able to improve policies online in a reasonably stable manner, although requires the use of ensembles and domain knowledge to do so.

Original: Finally, we note that behavioural cloning has also been used in the online setting with a focus on data efficiency, such as combining with DDPG (Nair et al., 2018) or SAC (Nair et al., 2020).
Revised: Finally, we note that behavioural cloning has also been used in the online setting with a focus on data efficiency, such as combining with DDPG [18] or SAC [17].

Original:  P OLICY REFINEMENT
Revised:  TD3-BC improvements 4.1 Offline policy refinement

Original: The relative simplicity and competitive performance of TD3-BC makes it an attractive proposition for offline-RL.
Revised: The relative simplicity and competitive performance of TD3-BC makes it an attractive proposition for offline-RL.

Original: There are however occasions where it falls short, particularly when the underlying data is sub-optimal.
Revised: There are however occasions where it falls short, particularly when the underlying data is sub-optimal.

Original: The BC component which is so crucial to preventing overestimation bias is the same component which limits the agent’s potential.
Revised: The BC component which is so crucial for preventing overestimation bias is the same component which limits the agent’s potential.

Original: But is there a way to overcome this  without increasing  complexity?
Revised: Is there a way to overcome this limitation without increasing the algorithm’s complexity?

Original: We note that at the start of training too low a value of α fails to prevent overestimation bias.
Revised: We note that, at the start of training, too low a value of α fails to prevent overestimation bias.

Original: When α is sufficiently high this overestimation bias is mitigated leading to a more accurate critic, albeitmore limited to the state-action region present in D .
Revised: When α is sufficiently high, this overestimation bias is mitigated leading to a more accurate critic, albeit limited to the state-action region contained in D .

Original: However, in learning to produce reasonable estimates in this region it may then be possible for the critic to improve its estimates outside this region, allowing the value of α to be reduced to a level which at the start was too small.
Revised: However, in learning to produce reasonable estimates in this region, it may then be possible for the critic to improve its estimates outside this range, allowing the value of α to be reduced to a level which at the start was too small.

Original: In other words, the critic learns from actions close to the ground truth before generalising to actions further away: learning to walk before learning to run.
Revised: In other words, the critic learns from actions close to the ground truth before generalising to actions further away: learning to walk before learning to run.

Original: With this in mind we modify the training procedure of TD3-BC to include an additional period of policy refinement, reducing the value of α and carrying-on offline.
Revised: With this in mind we modify the training procedure of TD3-BC to include an additional period of policy refinement, where we continue to train offline with a reduced value of α .

Original: To prevent overestimation bias reappearing we focus solely on the policy, performing no further updates to the critic.
Revised: To prevent overestimation bias reappearing we focus solely on the policy, performing no further updates to the critic.

Original: We illustrate this procedure in Figure 1 and provide full details in Appendix A. is trained on actions close to the data to avoid overestimation bias. Next, the agent is trained on actions further from the data to refine the policy prior to deployment. Following this procedure allows better generalisation to unseen stateaction pairs that an agent may poorly estimate without prior training. Green area represents action space permitted by the value of α .
Revised: We illustrate this procedure in Figure 1 and provide full details in Section 4.3.

Original: While there are many options available for reducing α , for the purpose of this work we simply scale by a constant λ ≥ 1 , giving rise to a new BC coefficient α ′ such that:
Revised: While there are many options available for reducing α , for the purpose of this work we simply scale by a constant λ ≥ 1 , giving rise to a new BC coefficient α ′ such that:

Original: [Equation]
Revised: [Equation]

Original: E XPERIMENTS
Revised: 4.2 Online fine-tuning

Revised: Once deployed, an agent has the opportunity for continued improvement via online fine-tuning. While one option is to simply remove the BC constraint imposed during offline learning, this often leads to an initial period of policy degradation owing to the sudden shift from the offline to online setting.

Original: We evaluate our method using MuJoCo tasks from the D4RL benchmark (Todorov et al., 2012), (Fu et al., 2020) and compare to a number of state-of-the-art approaches mentioned in Section 3. This benchmark contains a variety of tasks aimed at testing an agent’s ability to learn from sub-optimal or narrow data distributions as well as expert demonstrations. Each experiment is repeated across five random seeds with reported results the mean normalised score ± one standard deviation for 
Revised: In many situation this would be deemed unacceptable, hence approaches that promote stability as well as performance are desired. In light of this, we devise an alternative approach for transitioning to the online setting.

Original: 50 evaluations (10 from each seed). Scores of 0 and 100 represent equivalent random and expert policies, respectively. Further details, including network architecture and hyperparameters, can be found in Appendix B.
Revised: We initialise a new replay buffer and train both critic and policy on additional interactions collected from the environment.

Original: Offline policy refinement: Following the procedure in (Fujimoto & Gu, 2021) we train an agent for 1M gradient steps with  α = 0 . 4 to produce a baseline policy. We then perform policy refinement for a further 250k steps, setting λ = 5 for all tasks except hopper-medium-expert and halfcheetahexpert/medium-expert where λ = We also conduct a number of ablation studies to highlight the importance of the design decisions outlined in Section 4.
Revised: The influence of the BC term is reduced based on exponential decay with decay rate κ α :

Original: These results are presented in Appendix C.

Original: Online fine-tuning: We assess each agent’s ability to transition from offline to online learning.

Original: We initialise a new replay buffer and populate with 5k ( s, a, r, s ′ )

Original: samples collected using the offline trained policy. We then train both critic and policy for an additional 245k environment interactions (250k in total), decaying α exponentially from α start = 0 . 4 to α end = 0 . While we acknowledge other TD3-BC based approaches such as REDQ+AdaptiveBC (Zhao et al., 2021), we find our simple procedure achieves a similar level of performance and stability without the need for domain knowledge or ensemble methods.
Revised: [Equation]

Original: Results are summarised in Table 1. We denote the baseline policy as TD3-BC and refined policy as TD3-BC-PR. Corresponding online fine-tuned policies are suffixed -FT.
Revised: start

Original: Offline results for BC, CQL and IQL are taken from (Kostrikov et al., 2021b) with additional comparisons provided in Appendix

Original: D.

Original: [Table]
Revised: [Equation]

Original: For see the produces the or performs poorly TD3-BC is now able to compete with or surpass other leading methods across all tasks, despite being the simplest to implement. In terms of online learning we see from Table 1 that both baseline and refined policies benefit from fine-tuning. In Figure 2 we plot the corresponding learning curves and (un-normalised) scores across the 5 random seeds, where evaluation takes place every 5k environment interactions.
Revised: Here, α start and α end are the initial and final values of α , respectively, and N is the number of decay steps.

Original: The solid line represents the mean, shaded area ± one standard deviation and dashed line performance prior to fine tuning. We see in general that learning is relatively stable with very few catastrophic performance drops, comparable to results presented in (Zhao et al., 2021) despite being much more straightforward. The only exception to this is hopper-medium-expert which takes nearly the entire training period to stabilise.  D ISCUSSION , LIMITATIONS AND FUTURE WORK
Revised: Full details are again provided in Appendix 4.3.

Original: In our experiments we have shown that simple approaches to offline learning and online fine-tuning can produce policies that match or exceed more complex alternatives. The policy improvement procedure set out in Section 4 allows TD3-BC to overcome some of its shortcoming relating to suboptimal data, while still adhering to  the core principles of minimalism. Resulting policies can, for the most part, be fine-tuned online with little or no detrimental impact on performance. There are however several limitations to our work.
Revised: We adopt this approach over REDQ+AdaptiveBC to avoid the need for domain knowledge and ensemble methods, staying as close as possible to the original TD3-BC algorithm.

Original: First, our policy refinement process is sensitive to the value of λ , particularly for the medium-expert data sets (see Appendix E).

Original: While we note much published research in offline-RL reports results for optimised hyperparameters across tasks, such a procedure is unrealistic for real-world applications where online fine-tuning is limited or prohibited. Second, our fine-tuning procedure isn’t completely immune to performance drops, which may be considered unacceptable for high-risk environments. Finally, we have only applied our method to dense-reward environments, but equally important are those where rewards are sparse. In future work we look to address these shortcomings, devising methods to tune and/or set the value of λ offline and provide more stability during online fine-tuning, as well as evaluate both procedures in sprase-reward/goal-conditioned settings.
Revised: 4.3 Algorithms

Original: We also hope to provide a theoretical analysis to complement our empirical results.

Original: In addition, we plan to incorporate policy refinement into online reinforcement learning as a means to improve data efficiency.  C ONCLUSION In this work we have highlighted the potential of simple approaches for offline reinforcement learning and subsequent online fine-tuning. We have shown how deficiencies with TD3-BC can be overcome using a policy refinementprocedure andhow policies can be fine-tuned online in a relatively stable manner.
Revised: For completeness, in Algorithms 1 and 2 we outline our policy refinement and online fine-tuning procedures.

Original: We have discussed limitations of our work and directions for future research. A CKNOWLEDGEMENTS We would like to acknowledge Weights & Biases (Biewald, 2020) as the online platform we used for experiment tracking and visualizations to develop insights for this paper A P OLICY REFINEMENT AND ONLINE - TUNING PROCEDURES
Revised: Corresponding parameter updates and notation are as follows.

Original: Let θ i and θ
Revised: Let θ i and θ

Original: i represent  parameters for i Q-networks and target Q-networks, respectively. Let ϕ and ϕ ′ represent  parameters for a policy network and target policy network, respectively.
Revised: i represent the parameters of the i th Q-network and target Q-network, respectively, and ϕ and ϕ ′ represent the parameters for a policy network and target policy network, respectively.

Original: LetB be a sample of transitions from a replay buffer, α a BC regulariser and τ a target network update rate
Revised: Let α represent the BC coefficient, τ the target network update rate, ϵ policy noise variance and B a sample of transitions from dataset D .

Original: Define Q-networks update via gradient descent:
Revised: Each Q-network update is performed through gradient descent using:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where a ′ = clip ( π ϕ ′ ( s ′ )
Revised: where a ′ = clip ( π ϕ ′ ( s ′ )

Original: + noise, -0.5, 0.5) and noiseis sampled from an N (0 , ϵ ) distribution.
Revised: + noise, -0.5, 0.5) with noise sampled from an N (0 , ϵ ) distribution.

Original: Define policy network update via gradient ascent:
Revised: The policy network update is performed through gradient ascent using:

Original: [Equation]
Revised: [Equation]

Original: Define target network updates via polyak averaging:
Revised: Finally, target networks are updated using Polyak averaging:

Original: [Equation]
Revised: [Equation]

Original: With these updates defined we outline TD3-BC, policy improvement and online fine-tuning in Algorithm 1: Algorithm /or
Revised: Algorithm 1 TD3-BC with policy refinement

Original: Require: Dataset D , discount factor γ , target update rate τ , policy noise  ϵ and BC regulariser α
Revised: Require: Dataset D , discount factor γ , target update rate τ , policy noise variance ϵ and BC coefficient α

Original: Initialise critic parameters θ i , policy parameters ϕ and corresponding target parameters θ ′ i , ϕ ′ .
Revised: Initialise critic parameters θ i , policy parameters ϕ and corresponding target parameters θ ′ i , ϕ ′ .

Original: TD3-BC for j = 0 to J do
Revised: TD3-BC for j = 0 to J do

Original: Sample minibatch of transitions ( s, a, r, s ′ ) from D
Revised: Sample minibatch of transitions ( s, a, r, s ′ ) from D

Original: Update critic parameters θ i using equation (2)
Revised: Update critic parameters θ i using equation (3)

Original: Update policy parameters ϕ
Revised: Update policy parameters ϕ

Original: using equation (3)
Revised: using equation (4)

Original: Update target network parameters θ ′ i , ϕ ′ using equations (4) end for
Revised: Update target network parameters θ ′ i , ϕ ′ using equations (5) end for

Original: Policy refinement
Revised: Policy refinement

Original: Set scaling factor λ and replace α with α ′ as per equationfor k = 0 to K do
Revised: Set scaling factor λ and replace α with α ′ as per equationfor k = 0 to K do

Original: Sample minibatch of transitions ( s, a, r, s ′ ) from D
Revised: Sample minibatch of transitions ( s, a, r, s ′ ) from D

Original: Update policy parameters ϕ
Revised: Update policy parameters ϕ

Original: using equation (3)
Revised: using equation (4)

Original: Update target network parameters ϕ
Revised: Update target network parameters ϕ

Original: ′ using equations (4) end for
Revised: ′ using equations (5) end for

Original: Online fine-tuning
Revised: Require: Discount factor γ , target update rate τ , exploration noise σ , decay parameters α start , α end , N and BC coefficient α Initialise pre-trained critic parameters θ i , policy parameters ϕ and corresponding target parameters θ ′ i , ϕ ′ .

Original: Initialise environment and replay buffer R .
Revised: Initialise environment and replay buffer R .

Original: Set exploration noise σ and decay parameters α start , α end

Original: Set decay rate κ α = exp (cid:2) 1 N log (cid:0) α end α start(cid:1)(cid:3) for m = 0 to M do
Revised: Set decay rate κ α as per equation Set α = α start for m = 0 to M do

Original: Act in environment with exploration, a ∼ π ϕ ( s ) + N (0 , σ )
Revised: Act in environment with exploration, a ∼ π ϕ ( s ) + N (0 , σ )

Original: Store resulting transitions ( s, a, r, s ′ ) in R end for for n = 0 to N do
Revised: Store resulting transition ( s, a, r, s ′ ) in R end for for n = 0 to N do

Original: Act in environment with exploration, a ∼ π ϕ ( s ) + N (0 , σ )
Revised: Act in environment with exploration, a ∼ π ϕ ( s ) + N (0 , σ )

Revised: Store resulting transition ( s, a, r, s ′ ) in R

Original: Sample minibatch of transitions ( s, a, r, s ′ ) from R
Revised: Sample minibatch of transitions ( s, a, r, s ′ ) from R

Original: Update critic parameters θ i using equation (2)
Revised: Update critic parameters θ i using equation (3)

Original: Update policy parameters ϕ
Revised: Update policy parameters ϕ

Original: using equation (3)
Revised: using equation (4)

Original: Update target network parameters θ ′ i , ϕ ′ using equations (4)
Revised: Update target network parameters θ ′ i , ϕ ′ using equations (5)

Original: Update decay rate α ← κ α α end for
Revised: Update decay rate α ← κ α α end for

Original:  B E XPERIMENTAL DETAILS TD3-BC hyperparameters and network architecture
Revised:  Experiments

Revised: We evaluate our method using MuJoCo tasks from the open-source D4RL 1 benchmark [21] [5] and compare to a number of state-of-the-art approaches mentioned in Section 3. This benchmark contains a variety of tasks aimed at testing an agent’s ability to learn from sub-optimal or narrow data distributions as well as expert demonstrations. For the tasks considered in this paper the data sets comprise the following

Revised: • Expert 1M interactions collected from an agent trained to expert level using SAC • Medium 1M interactions collected from an agent trained to 1/3 expert level using SAC • Medium-Expert The combined interactions from expert and medium (2M total) • Medium-Replay

Revised: The replay buffer from training the medium-level agent (variable size)

Revised: Each experiment is repeated across five random seeds and we report the mean normalised score ± one standard deviation for 50 evaluations ( 10 from each seed).

Revised: Scores of 0 and 100 represent equivalent random and expert policies, respectively.

Revised: Further details, including network architecture and hyperparameters, can be found in Appendix A.1.

Revised: Offline policy refinement: Following the procedure in [6], we train an agent for 1M gradient steps with α = 0 .

Revised: 4 to produce a baseline policy.

Revised: We then perform policy refinement for a further 250k steps, setting λ = 5 for all tasks except hopper-medium-expert and halfcheetah-expert/medium-expert where λ = 1 .

Revised: We also conduct a number of ablation studies to highlight the importance of the design decisions outlined in Section 4.1.

Revised: These results are presented in Appendix A.2.

Revised: Online fine-tuning: We assess each agent’s ability to transition from offline to online learning, focusing on stability as well as performance.

Revised: We initialise a new replay buffer and populate with 5k ( s, a, r, s ′ )

Revised: samples collected using the offline trained policy.

Revised: We then train both critic and policy for an additional N =245k environment interactions (250k in total), decaying α exponentially from α start = 0 .

Revised: 4 to α end = 0 .

Revised: Results are summarised in Table 1.

Revised: We denote the baseline policy as TD3-BC and refined policy as

Revised: TD3-BC-PR.

Revised: Corresponding online fine-tuned policies are suffixed -FT.

Revised: Offline results for BC, CQL and IQL are taken from [12] with additional comparisons provided in Appendix A.3.

Revised: [Table]

Revised: For offline learning, we observe that policy refinement produces policies that outperform the original baseline.

Revised: This is particularly the case when data is sub-optimal, i.e. medium/medium-replay or where

Revised: TD3-BC performs poorly, i.e. hopper-medium/medium-replay.

Revised: For data sets where TD3-BC already achieves expert or near-expert performance, refined policies match or exceed this.

Revised: TD3-BC is now able to compete with or surpass other leading methods across all tasks while remaining the simplest to implement.

Revised: For online learning, we see from Table 1 that both baseline and refined policies benefit from finetuning in terms of performance.

Revised: To assess stability, we plot the corresponding learning curves in

Revised: Figure 2 using un-normalised scores as the performance metric, with evaluation taking place every 5k iterations.

Revised: The solid line represents the mean, shaded area ± one standard deviation and dashed line performance prior to fine tuning.

Revised: Learning is relatively stable with very few severe performance drops and comparable to results presented in [24].

Revised: The only exception to this is hopper-medium-expert, which takes nearly the entire training period to stabilise.

Revised: Note this procedure works equally well for baseline and refined policies, showing there is no detrimental impact of offline policy refinement prior to online fine-tuning.

Revised:  Discussion and conclusion

Revised: In this paper we have presented approaches for improving the performance of TD3-BC during offline learning and online fine-tuning.

Revised: We have shown empirically how deficiencies with TD3-BC can be overcome using a policy refinement procedure and how such an approach can also be used for stable online fine-tuning.

Revised: In both cases our results match or exceed the performance of state-of-the-art alternatives while being much simpler.

Revised: There are however several limitations to our work.

Revised: First, our policy refinement process is sensitive to the value of λ , particularly for the medium-expert data sets (see Appendix A.4).

Revised: While much published research in offline-RL report results for optimised hyperparameters across tasks, such a procedure is unrealistic for real-world application where online fine-tuning is limited or prohibited.

Revised: Second, our fine-tuning procedure is not completely immune to performance drops, which may be considered unacceptable for high-risk environments.

Revised: Finally, we have only applied our method to dense-reward environments, but equally important are settings where rewards are sparse.

Revised: In future work we look to address these shortcomings as well as conduct a theoretical analysis to complement our empirical results.

Revised: In addition, we plan to incorporate our approach into online reinforcement learning as a means of improving data efficiency.

Revised: By showing it is possible to train performant policies through simple modifications to TD3-BC we hope to highlight the potential of minimalist approaches and encourage researchers to explore the possibilities within this framework.

Revised: Acknowledgements

Revised: AB acknowledges support from a University of Warwick and University Hospitals Birmingham NHS

Revised: Foundation Trust collaborative doctoral training scholarship.

Revised: GM acknowledges support from a

Revised: UKRI AI Turing Acceleration Fellowship (EPSRC EP/V024868/1).

Revised: We also acknowledge Weights &

Revised: Biases [2] as the online platform used for experiment tracking and visualizations to develop insights for this work.

Original: In the original TD3-BC (Fujimoto & Gu, 2021) the hyperparamater α is attached to Q-values but since our focus in the BC component we attach to the MSE.
Revised: In the original TD3-BC [6] the hyperparamater α is attached to Q-values but since our focus in the BC component we attach to the MSE.

Original: This is an equivalent set-up if α = 0 .
Revised: This is an equivalent set-up if α = 0 .

Original: 4 , the inverse of the original value α = 2 .
Revised: 4 , the inverse of the original value α = 2 .

Original: All other details are the same, as per Table
Revised: All other details are the same, as per Table

Original: Policy refinement hyperparameter
Revised: [Table]

Original: Table 3 reiterates the value of the scaling parameter λ used in our experiments.
Revised: Table 3 reiterates the value of the scaling parameter λ used in our experiments.

Revised: Our procedure for tuning λ emulates that of the TD3-BC paper.

Revised: We used hopper-medium and hopper-expert as test cases and tuned over range λ

Revised: = (1 , 2 , 3 , 4 , 5) .

Revised: Based on the resulting policy performances, we chose λ = 5 and applied this to the remaining data sets.

Revised: Performance was improved over the baseline for all data sets apart from hopper-medium-expert, halfcheetah-medium-expert and halfcheetah-expert where performance was significantly worse.

Original:  Offline policy evaluation procedure
Revised: For these three, we hypothesised the critic was unable to generalised to actions beyond the region it was trained on and hence reduced the value of λ as per the above range. We found that for λ > 1 policies were still inferior to their baselines and hence λ =was chosen. Offline policy evaluation procedure

Original: Policy evaluation takes place online in the simulated MuJoCo environment.
Revised: Policy evaluation takes place online in the simulated MuJoCo environment.

Original: The reported results are mean normalised scores ± one standard deviation across evaluations and seeds, where scores of 0 and 100 represent random and expert policies, respectively.
Revised: The reported results are mean normalised scores ± one standard deviation across evaluations and seeds, where scores ofand 100 represent random and expert policies, respectively.

Original: Further details can be found in the D4RL paper (Fu et al., 2020).
Revised: Further details can be found in the D4RL paper [5].

Original: Online policy evaluation procedure
Revised: Online policy evaluation procedure

Original: Policy evaluation takes place online in the simulated MuJoCo environment.
Revised: Policy evaluation takes place online in the simulated MuJoCo environment.

Original: The reported results are mean un-normalised scores ± one standard deviation across seeds
Revised: The reported results are mean un-normalised scores ± one standard deviation across seeds

Original: [Table]
Revised: [Table]

Original: [Table]

Original: Hardware All experiments were conducted on a single machine with the following hardware specification: Intel Core i9 9900K CPU, 64GB RAM, 1TB SSD, 2x NVIDIA GeForce RTX 2080Ti 11GB TURBO GPU.
Revised: Hardware All experiments were conducted on a single machine with the following hardware specification: Intel Core i9 9900K CPU, 64GB RAM, 1TB SSD, 2x NVIDIA GeForce RTX 2080Ti 11GB TURBO GPU.

Original: C A BLATIONS
Revised: A.2 Ablations

Original: In Section 4 we note two key designs decisions as part of policy refinement.
Revised: In Section 4.1 we note two key designs decisions as part of policy refinement.

Original: The first is reducing the value of α after initial training, allowing us to choose values that at the start would be too low.
Revised: The first is reducing the value of α after initial training, allowing us to choose values that at the start would be too low.

Original: The second is only training the policy while keep the critic fixed in efforts to prevent overestimation bias reappearing.
Revised: The second is only training the policy while keep the critic fixed in efforts to prevent overestimation bias reappearing.

Original: To check the relative importance of each of these we carry out the following ablations:
Revised: To check the relative importance of each of these we carry out the following ablations:

Original: • Ablation 1: Run baseline TD3-BC with the value of α ′ from Section• Ablation 2: During policy refinement train both the policy and critic.
Revised: • Ablation 1: Run baseline TD3-BC with the value of α ′ from Section• Ablation 2: During policy refinement train both the policy and critic.

Original: Note, to reach an equivalent 250k policy updates the total additional gradient steps is 500k as the critic-actor update ratio is 2:1.
Revised: Note, to reach an equivalent 250k policy updates the total additional gradient steps is 500k as the critic-actor update ratio is 2:1.

Original: For completeness we also run the baseline TD3-BC with α = 0 .
Revised: For completeness we also run the baseline TD3-BC with α = 0 .

Original: 4 for 1.5M gradient steps (1.5M critic, 750k policy) to show additional training alone isn’t the cause of policy improvements.
Revised: 4 for 1.5M gradient steps (1.5M critic, 750k policy) to show additional training alone isn’t the cause of policy improvements.

Original: We denote this Ablation 3.
Revised: We denote this Ablation 3.

Original: Results are summarised in Table 4.
Revised: Results are summarised in Table 4.

Original: Ablation 1 results show that while setting α = α ′ works for a few data sets, for most this leads to a collapse in learning and highly sub-optimal policies.
Revised: Ablation 1 results show that while setting α = α ′ works for a few data sets, for most this leads to a collapse in learning and highly sub-optimal policies.

Original: Ablation 2 results show that training the policy and critic leads to collapse for certain data sets, and even when it doesn’t the results aren’t as strong as training the policy alone.
Revised: Ablationresults show that training the policy and critic leads to collapse for certain data sets, and even when it doesn’t the results aren’t as strong as training the policy alone.

Original: Ablation 3 results show additional training alone can’t match the performance of policy refinement when data sets are sub-optimal (medium/medium-replay).[Table]
Revised: Ablation 3 results show additional training alone can’t match the performance of policy refinement when data sets are sub-optimal (medium/medium-replay).

Original: While some data sets are agnostic to these choices, in general they are required to attain the perfor

Original: D A DDITIONAL COMPARISONS FOR OFFLINE -RL
Revised: A.3 Additional comparisons for offline-RL

Original: In Table 5 we provide additional comparisons for a range of offline approaches, including some model-based methods not covered in Section 3.
Revised: In Table 5 we provide additional comparisons for a range of offline approaches, including some model-based methods not covered in Section 3.

Original: Specifically we compare to:
Revised: Specifically we compare to:

Original: • Advantage weighted actor-critic (AWAC)
Revised: • Advantage weighted actor-critic (AWAC)

Original: (Nair et al., 2020) • One-step Exp.
Revised: [17] • One-step Exp.

Original: Weight (OnestepRL) (Brandfonbrener et al., 2021) • Decision Transformer (DT) (Chen et al., 2021) • Diffuser (Janner et al., 2022) • CQL + Model Based Policy Optimization (COMBO) 
Revised: Weight (OnestepRL) [3] • Decision Transformer (DT) [4] • Diffuser [9] • CQL + Model Based Policy Optimization (COMBO) [23] • Model-based Offline Reinforcement Learning (MOReL) [10] • Model-based Offline Planning (MBOP) [1]

Original: (Yu et al., 2021)
Revised: [Table]

Original: • Model-based Offline Reinforcement Learning (MOReL) (Kidambi et al., 2020) • Model-based Offline Planning (MBOP) (Argenson & Dulac-Arnold, 2020)
Revised: of design decisions in policy refinement. While some data sets are agnostic to these choices, in general they are required to attain the performance levels reported in the paper. The results marked with a single asterisk are the same as baseline TD3-BC and the results marked with a double asterisk are the same as Ablation 3 (as λ = 1 for these data sets).

Original: We note some comparisons aren’t directly equivalent due to differences in data sets, using -v0 instead of -v2.
Revised: We note some comparisons aren’t directly equivalent due to differences in data sets, using -v0 instead of -v2.

Original: Since we use the latest -v2 we mark -v0 approaches with an asterisk.
Revised: Since we use the latest -v2 we mark -v0 approaches with an asterisk.

Original: AWAC, OnestepRL and DT are taken from (Kostrikov et al., 2021b) with Diffuser, COMBO, MOReL and MBOP taken from their respective papers.
Revised: AWAC, OnestepRL and DT are taken from [12] with Diffuser, COMBO, MOReL and MBOP taken from their respective papers.

Original: As can be seen, TD3-BC-PR remains competitive despite its minimalist approach.
Revised: As can be seen, TD3-BC-PR remains competitive while adhering to the minimalist approach.

Original: [Table]
Revised: [Table]

Original: E M EDIUM -E XPERT PERFORMANCE FOR λ =
Revised: A.4 Medium-Expert performance for λ =

Revised: As mentioned in Section 6 our policy refinement procedure is sensitive to the quality of the data set.

Revised: This is particularly the case for hopper/halfcheetah-medium-expert where, as can be seen in Table 6, using λ = 5 leads to inferior performance than the TD3-BC baseline:

Original: [Table]
Revised: [Table]