Original: F INE -T UNING O FFLINE P OLICIES WITH O PTIMISTIC A CTION S ELECTION
Revised: F INE -T UNING O FFLINE P OLICIES WITH O PTIMISTIC A CTION S ELECTION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Offline reinforcement learning algorithms can train performant policies for hard tasks using previously-collected datasets.
Revised: Offline reinforcement learning algorithms can train performant policies for hard tasks using previously-collected datasets.

Original: However, the quality of the offline dataset often limits the levels of performance possible.
Revised: However, the quality of the offline dataset often limits the levels of performance possible.

Original: We consider the problem of improving offline policies through online fine-tuning.
Revised: We consider the problem of improving offline policies through online fine-tuning.

Original: Offline RL requires a pessimistic training objective to mitigate distributional shift between the trained policy and the offline behavior policy, which will make the trained policy averse to picking novel actions.
Revised: Offline RL requires a pessimistic training objective to mitigate distributional shift between the trained policy and the offline behavior policy, which will make the trained policy averse to picking novel actions.

Original: In contrast, online RL requires exploration, or optimism.
Revised: In contrast, online RL requires exploration, or optimism.

Original: Thus, fine-tuning online policies with the offline training objective is not ideal.
Revised: Thus, fine-tuning online policies with the offline training objective is not ideal.

Original: Additionally, loosening the fine-tuning objective to allow for more exploration can potentially destroy the behaviors learned in the offline phase because of the sudden and significant change in the optimization objective.
Revised: Additionally, loosening the fine-tuning objective to allow for more exploration can potentially destroy the behaviors learned in the offline phase because of the sudden and significant change in the optimization objective.

Original: To mitigate this challenge, we propose a method to facilitate exploration during online fine-tuning that maintains the same training objective throughout both offline and online phases, while encouraging exploration.
Revised: To mitigate this challenge, we propose a method to facilitate exploration during online fine-tuning that maintains the same training objective throughout both offline and online phases, while encouraging exploration.

Original: We accomplish this by changing the action-selection method to be more optimistic with respect to the Q-function.
Revised: We accomplish this by changing the action-selection method to be more optimistic with respect to the Q-function.

Original: By choosing to take actions in the environment with higher expected Q-values, our method is able to explore and improve behaviors more efficiently, obtaining 56% more returns on average than the alternative approaches on several locomotion, navigation, and manipulation tasks.
Revised: By choosing to take actions in the environment with higher expected Q-values, our method is able to explore and improve behaviors more efficiently, obtaining 56% more returns on average than the alternative approaches on several locomotion, navigation, and manipulation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Offline reinforcement learning (RL) algorithms show strong performance on different tasks even when the available data contains sub-optimal behaviors such as random exploration or another RL agent’s behavior.
Revised: Offline reinforcement learning (RL) algorithms show strong performance on different tasks even when the available data contains sub-optimal behaviors such as random exploration or another RL agent’s behavior.

Original: However, the quality of the offline data can limit the performance of an offline
Revised: However, the quality of the offline data can limit the performance of an offline

Original: RL agent, and in these cases, further online fine-tuning can help improve the policy using additional environment interactions.
Revised: RL agent, and in these cases, further online fine-tuning can help improve the policy using additional environment interactions.

Original: A key challenge to online fine-tuning is that the training mechanisms designed for the offline setting are not effective at improving the policy online; offline learning requires a conservative objective to mitigate distributional shift between the learned policy and the behavior policy.
Revised: A key challenge to online fine-tuning is that the training mechanisms designed for the offline setting are not effective at improving the policy online; offline learning requires a conservative objective to mitigate distributional shift between the learned policy and the behavior policy.

Original: However, during the online fine-tuning phase, the conservative objective causes the agent to keep doing the same behaviors, preventing it from exploring as needed to further improve the policy.
Revised: However, during the online fine-tuning phase, the conservative objective causes the agent to keep doing the same behaviors, preventing it from exploring as needed to further improve the policy.

Original: In this paper, we aim to mitigate these challenges and develop a more effective method for offline-to-online RL.
Revised: In this paper, we aim to mitigate these challenges and develop a more effective method for offline-to-online RL.

Original: This challenge of misaligned priorities between the offline and online training phases could in principle be tackled by switching training objectives when the online phase starts.
Revised: This challenge of misaligned priorities between the offline and online training phases could in principle be tackled by switching training objectives when the online phase starts.

Original: For example, a policy trained with an offline RL algorithm such as CQL (Kumar et al., 2020) could be fine-tuned using an off-policy online algorithm designed for efficient exploration, such as SAC (Haarnoja et al., 2018).
Revised: For example, a policy trained with an offline RL algorithm such as CQL (Kumar et al., 2020) could be fine-tuned using an off-policy online algorithm designed for efficient exploration, such as SAC (Haarnoja et al., 2018).

Original: However, significantly changing the training objective between the offline and online phases can cause instability in the performance, potentially resulting in the destruction of behaviors learned during the offline phase (Nair et al., 2020).
Revised: However, significantly changing the training objective between the offline and online phases can cause instability in the performance, potentially resulting in the destruction of behaviors learned during the offline phase (Nair et al., 2020).

Original: It can also exacerbate distributional shift between the offline data and the learned policy (Lee et al., 2022) since the more exploratory training objective will make the policy deviate from the offline behavior policy even more, resulting in unsatisfactory fine-tuning performance.
Revised: It can also exacerbate distributional shift between the offline data and the learned policy (Lee et al., 2022) since the more exploratory training objective will make the policy deviate from the offline behavior policy even more, resulting in unsatisfactory fine-tuning performance.

Original: The problem of online fine-tuning from offline training presents an impasse: on the one hand, online fine-tuning requires efficient exploration, which is not feasible using the conservative offline training objective; on the other hand, changing the objective might destroy the behaviors learned by the offline training.
Revised: The problem of online fine-tuning from offline training presents an impasse: on the one hand, online fine-tuning requires efficient exploration, which is not feasible using the conservative offline training objective; on the other hand, changing the objective might destroy the behaviors learned by the offline training.

Original: A key insight of our method is that we collect optimistic data without changing the training objective.
Revised: A key insight of our method is that we collect optimistic data without changing the training objective.

Original: To collect such exploratory data, we aim to use the knowledge embedded in the
Revised: To collect such exploratory data, we aim to use the knowledge embedded in the

Original: Q-function to direct exploration, i.e., selecting actions that are estimated to be better than the ones given by the policy.
Revised: Q-function to direct exploration, i.e., selecting actions that are estimated to be better than the ones given by the policy.

Original: Concretely, during online fine-tuning, we can execute actions that have high estimated Q-values, thus providing exploratory data that can be used to improve the policy, while using the same offline objective with this new data.
Revised: Concretely, during online fine-tuning, we can execute actions that have high estimated Q-values, thus providing exploratory data that can be used to improve the policy, while using the same offline objective with this new data.

Original: The key contribution of this paper is an exploration technique that allows for stable and efficient offline-to-online fine-tuning.
Revised: The key contribution of this paper is an exploration technique that allows for stable and efficient offline-to-online fine-tuning.

Original: Our approach is simple, as it only modifies the action-selection process, and can in principle be implemented on top of any offline learning algorithm that doesn’t explicitly penalize the Q-values of out-of-distribution state-action pairs.
Revised: Our approach is simple, as it only modifies the action-selection process, and can in principle be implemented on top of any offline learning algorithm that doesn’t explicitly penalize the Q-values of out-of-distribution state-action pairs.

Original: We show results built on top of LAPO (Chen et al., 2022) and IQL (Kostrikov et al., 2021) for several locomotion, navigation, and manipulation tasks, and find that by using this action-selection mechanism agents get an average of 56% more returns than the next best prior method.
Revised: We show results built on top of LAPO (Chen et al., 2022) and IQL (Kostrikov et al., 2021) for several locomotion, navigation, and manipulation tasks, and find that by using this action-selection mechanism agents get an average of 56% more returns than the next best prior method.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Exploration mechanisms: One simple technique for exploration is to apply noise or randomness to the agent’s behavior (Mnih et al., 2015; Lillicrap et al., 2015; Schulman et al., 2015).
Revised: Exploration mechanisms: One simple technique for exploration is to apply noise or randomness to the agent’s behavior (Mnih et al., 2015; Lillicrap et al., 2015; Schulman et al., 2015).

Original: Other approaches explore in a more targeted way through novelty seeking (Houthooft et al., 2016;
Revised: Other approaches explore in a more targeted way through novelty seeking (Houthooft et al., 2016;

Original: Bellemare et al., 2016; Pathak et al., 2017; Ostrovski et al., 2017; Fu et al., 2017a; Burda et al., 2018b;a) and entropy regularization (Pong et al., 2019;
Revised: Bellemare et al., 2016; Pathak et al., 2017; Ostrovski et al., 2017; Fu et al., 2017a; Burda et al., 2018b;a) and entropy regularization (Pong et al., 2019;

Original: Eysenbach et al., 2018; Florensa et al., 2017; Gregor et al., 2016).
Revised: Eysenbach et al., 2018; Florensa et al., 2017; Gregor et al., 2016).

Original: These prior exploration methods are focused on online RL and are ill-suited for offline-to-online fine-tuning on their own, as exploration can negatively impact offline training.
Revised: These prior exploration methods are focused on online RL and are ill-suited for offline-to-online fine-tuning on their own, as exploration can negatively impact offline training.

Original: Our exploration mechanism is based on modifying the action-selection mechanism to choose actions more optimistically using a Q-function pre-trained with offline RL.
Revised: Our exploration mechanism is based on modifying the action-selection mechanism to choose actions more optimistically using a Q-function pre-trained with offline RL.

Original: Offline to online fine-tuning: The problem of leveraging offline data during online RL has been widely studied especially when the offline data corresponds to demonstrations.
Revised: Offline to online fine-tuning: The problem of leveraging offline data during online RL has been widely studied especially when the offline data corresponds to demonstrations.

Original: Prior work has proposed a variety of approaches for this setting, ranging from online inverse RL and imitation learning (Lu et al., 2022; Ziebart et al., 2008;
Revised: Prior work has proposed a variety of approaches for this setting, ranging from online inverse RL and imitation learning (Lu et al., 2022; Ziebart et al., 2008;

Original: Finn et al., 2016; Fu et al., 2017b;
Revised: Finn et al., 2016; Fu et al., 2017b;

Original: Ho & Ermon, 2016; Kostrikov et al., 2018)
Revised: Ho & Ermon, 2016; Kostrikov et al., 2018)

Original: to using the demonstrations to initialize the policy or replay buffer (Peters & Schaal, 2008; Vecerik et al., 2017; Hester et al., 2018;
Revised: to using the demonstrations to initialize the policy or replay buffer (Peters & Schaal, 2008; Vecerik et al., 2017; Hester et al., 2018;

Original: Rajeswaran et al., 2018; Zhu et al., 2018a; Gupta et al., 2019; Zhu et al., 2018b; Kober & Peters, 2009).
Revised: Rajeswaran et al., 2018; Zhu et al., 2018a; Gupta et al., 2019; Zhu et al., 2018b; Kober & Peters, 2009).

Original: We consider a more general case where the offline data may contain low or mixed quality data.
Revised: We consider a more general case where the offline data may contain low or mixed quality data.

Original: The simplest approach for the general case of offline to online RL is to fine-tune online using the same offline objective (Kumar et al., 2020;
Revised: The simplest approach for the general case of offline to online RL is to fine-tune online using the same offline objective (Kumar et al., 2020;

Original: Kostrikov et al., 2021; Nair et al., 2020;
Revised: Kostrikov et al., 2021; Nair et al., 2020;

Original: Lyu et al., 2022); however, this approach may be too conservative during the online phase, resulting in too little exploration and plateauing performance.
Revised: Lyu et al., 2022); however, this approach may be too conservative during the online phase, resulting in too little exploration and plateauing performance.

Original: Instead of maintaining the same offline training objective during the online phase, Wu et al.
Revised: Instead of maintaining the same offline training objective during the online phase, Wu et al.

Original: (2022) propose to gradually make the training objective less conservative.
Revised: (2022) propose to gradually make the training objective less conservative.

Original: As we empirically show, this approach does not significantly improve exploration for the considered baselines.
Revised: As we empirically show, this approach does not significantly improve exploration for the considered baselines.

Original: Lee et al.
Revised: Lee et al.

Original: (2022) instead propose to target the over-conservatism of using an offline training objective during the online phase with a prioritized replay buffer that prefers more on-policy samples.
Revised: (2022) instead propose to target the over-conservatism of using an offline training objective during the online phase with a prioritized replay buffer that prefers more on-policy samples.

Original: This approach could in principle be combined with our method for enhanced exploration, but we leave this direction for future work.
Revised: This approach could in principle be combined with our method for enhanced exploration, but we leave this direction for future work.

Original: Unlike these past works, we develop a simple and sample-efficient fine-tuning approach that effectively trade-offs conservatism in the offline phase and optimism in the online phase without sacrificing stability.
Revised: Unlike these past works, we develop a simple and sample-efficient fine-tuning approach that effectively trade-offs conservatism in the offline phase and optimism in the online phase without sacrificing stability.

Original: Offline meta reinforcement learning: Offline meta RL is related to online fine-tuning, since it deals with efficient online adaptation using an offline dataset.
Revised: Offline meta reinforcement learning: Offline meta RL is related to online fine-tuning, since it deals with efficient online adaptation using an offline dataset.

Original: However, the meta-RL domain assumes access to a multi-task offline dataset, and trains an agent to quickly adapt to new tasks (Mitchell et al., 2021; Dorfman et al., 2021; Pong et al., 2022).
Revised: However, the meta-RL domain assumes access to a multi-task offline dataset, and trains an agent to quickly adapt to new tasks (Mitchell et al., 2021; Dorfman et al., 2021; Pong et al., 2022).

Original: As opposed to offline meta RL, our work (and offline-to-online fine-tuning in general) does not assume access to multi-task data, and deals with adaptation and improvement in the context of the same offline task.
Revised: As opposed to offline meta RL, our work (and offline-to-online fine-tuning in general) does not assume access to multi-task data, and deals with adaptation and improvement in the context of the same offline task.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We address policy learning in a Markov decision process (MDP), defined by the tuple ( S, A, p, r ) , where S and A denote the state and action spaces.
Revised: We address policy learning in a Markov decision process (MDP), defined by the tuple ( S, A, p, r ) , where S and A denote the state and action spaces.

Original: The unknown state transition probability distribution p : S × S ×
Revised: The unknown state transition probability distribution p : S × S ×

Original: A → [0 , ∞ ) represents the probability density of the next state s ′ ∈ S given the current state s ∈ S and action a ∈ A .
Revised: A → [0 , ∞ ) represents the probability density of the next state s ′ ∈ S given the current state s ∈ S and action a ∈ A .

Original: The reward function assigns a sparse reward to every state-action transitions r : S ×
Revised: The reward function assigns a sparse reward to every state-action transitions r : S ×

Original: A → R .
Revised: A → R .

Original: 3. O FFLINE R EINFORCEMENT L EARNING
Revised: 3. O FFLINE R EINFORCEMENT L EARNING

Original: In the offline RL setting, an agent only has access to a dataset of experiences D = { ( s, a, r, s ′ )
Revised: In the offline RL setting, an agent only has access to a dataset of experiences D = { ( s, a, r, s ′ )

Original: i } that are collected prior to the RL training using an unknown behavior policy π β .
Revised: i } that are collected prior to the RL training using an unknown behavior policy π β .

Original: Off-policy RL approaches can leverage the offline dataset to train a critic network (Q-function)
Revised: Off-policy RL approaches can leverage the offline dataset to train a critic network (Q-function)

Original: Q πθ ( s, a ) with parameters θ , which estimates the long-term discounted reward attained by executing action a at state s and following the policy π thereafter.
Revised: Q πθ ( s, a ) with parameters θ , which estimates the long-term discounted reward attained by executing action a at state s and following the policy π thereafter.

Original: The critic network can then be used to update the policy in an actor-critic training scheme.
Revised: The critic network can then be used to update the policy in an actor-critic training scheme.

Original: The critic network can be trained using the following temporal difference learning objective:
Revised: The critic network can be trained using the following temporal difference learning objective:

Original: [Equation]
Revised: [Equation]

Original: where ˆ θ denotes the delayed parameters of the Q-function.
Revised: where ˆ θ denotes the delayed parameters of the Q-function.

Original: Notice that a ′ is a potentially out-ofdistribution action, so Q θ ( s ′ , a ′ ) might give an incorrect value.
Revised: Notice that a ′ is a potentially out-ofdistribution action, so Q θ ( s ′ , a ′ ) might give an incorrect value.

Original: The max operator evaluated over many potentially-incorrect values is then likely to update the Q-function with incorrect information, resulting in sub-optimal policies.
Revised: The max operator evaluated over many potentially-incorrect values is then likely to update the Q-function with incorrect information, resulting in sub-optimal policies.

Original: To alleviate this problem, prior offline RL methods propose two general solutions: (1) assigning low Q-values to the out-of-distribution actions by training a Qfunction with a pessimistic objective (Kumar et al., 2020; Lyu et al., 2022), and (2) training the
Revised: To alleviate this problem, prior offline RL methods propose two general solutions: (1) assigning low Q-values to the out-of-distribution actions by training a Qfunction with a pessimistic objective (Kumar et al., 2020; Lyu et al., 2022), and (2) training the

Original: Q-function using equation 1 while constraining the policy to only produce actions contained in the offline dataset D (Kostrikov et al., 2021; Chen et al., 2022).
Revised: Q-function using equation 1 while constraining the policy to only produce actions contained in the offline dataset D (Kostrikov et al., 2021; Chen et al., 2022).

Original: The action-selection mechanism we present in this work relies on sampling the Q-function for slightly out-of-distribution state-action pairs and obtaining non-pessimistic value estimates.
Revised: The action-selection mechanism we present in this work relies on sampling the Q-function for slightly out-of-distribution state-action pairs and obtaining non-pessimistic value estimates.

Original: Our method is therefore only applicable to the second class of offline RL algorithms that constrain the actor, and not the critic.
Revised: Our method is therefore only applicable to the second class of offline RL algorithms that constrain the actor, and not the critic.

Original: 3. O FFLINE - TO -O NLINE F INE -T UNING
Revised: 3. O FFLINE - TO -O NLINE F INE -T UNING

Original: As shown in Figure 1, having trained a policy and a Q-function with offline data, the goal of online fine-tuning is to further improve the behavior of the learned policy by allowing the agent to interact
Revised: As shown in Figure 1, having trained a policy and a Q-function with offline data, the goal of online fine-tuning is to further improve the behavior of the learned policy by allowing the agent to interact

Original: [Figure]
Revised: [Figure]

Original: with the environment.
Revised: with the environment.

Original: In every episode of the online fine-tuning phase, new samples are collected by unrolling the latest policy in the environment and are added to the replay buffer.
Revised: In every episode of the online fine-tuning phase, new samples are collected by unrolling the latest policy in the environment and are added to the replay buffer.

Original: Then, the policy and the Q-function are updated by the actor-critic learning algorithm which is employed for the online fine-tuning phase.
Revised: Then, the policy and the Q-function are updated by the actor-critic learning algorithm which is employed for the online fine-tuning phase.

Original: Offline-to-online fine-tuning presents two closely related challenges: (1) conservative offline learning objectives limit agents’ exploration power, which is vital for efficient online learning, and (2) changing training objectives when switching from the offline phase to the online phase can cause a significant drop in performance, due to distributional shift and the sudden change in objective.
Revised: Offline-to-online fine-tuning presents two closely related challenges: (1) conservative offline learning objectives limit agents’ exploration power, which is vital for efficient online learning, and (2) changing training objectives when switching from the offline phase to the online phase can cause a significant drop in performance, due to distributional shift and the sudden change in objective.

Original: A suitable method for online fine-tuning should handle the transition from offline to online while improving the policy performance by carefully exploring the environment.
Revised: A suitable method for online fine-tuning should handle the transition from offline to online while improving the policy performance by carefully exploring the environment.

Original:  O BJECTIVE SWITCHING BETWEEN OFFLINE AND ONLINE PHASES
Revised:  O BJECTIVE SWITCHING BETWEEN OFFLINE AND ONLINE PHASES

Original: The pessimistic objectives in offline RL algorithms prevents proper exploration in online settings and limits the agent from efficiently improving its performance.
Revised: The pessimistic objectives in offline RL algorithms prevents proper exploration in online settings and limits the agent from efficiently improving its performance.

Original: As shown in Figure 2(b), online training with CQL does not notably improve the learning performance even though the agent is no longer bounded by the offline setting.
Revised: As shown in Figure 2(b), online training with CQL does not notably improve the learning performance even though the agent is no longer bounded by the offline setting.

Original: This limited performance in the online setting can be explained by the restricted exploration capability of the agent.
Revised: This limited performance in the online setting can be explained by the restricted exploration capability of the agent.

Original: CQL avoids querying out-of-distribution actions, hence it stabilizes the offline training by explicitly assigning low Q-values to actions outside the support of the offline dataset.
Revised: CQL avoids querying out-of-distribution actions, hence it stabilizes the offline training by explicitly assigning low Q-values to actions outside the support of the offline dataset.

Original: As a side-effect, CQL would have difficulties in efficiently exploring the environment, and therefore it demonstrates a poor performance at improving the policy in online settings.
Revised: As a side-effect, CQL would have difficulties in efficiently exploring the environment, and therefore it demonstrates a poor performance at improving the policy in online settings.

Original: As discussed earlier, a naive solution to this problem would be to pre-train a constrained policy with offline RL, and then modify the objective at the fine-tuning phase by lifting the constraints or making them less conservative.
Revised: As discussed earlier, a naive solution to this problem would be to pre-train a constrained policy with offline RL, and then modify the objective at the fine-tuning phase by lifting the constraints or making them less conservative.

Original: However, our empirical studies show that this approach do not perform well in practice.
Revised: However, our empirical studies show that this approach do not perform well in practice.

Original: Figure 2(b) shows the results of using the soft actor-critic (SAC) algorithm for online fine-tuning of a policy trained with CQL.
Revised: Figure 2(b) shows the results of using the soft actor-critic (SAC) algorithm for online fine-tuning of a policy trained with CQL.

Original: SAC leverages a maximum entropy objective to encourage exploration which is helpful to improve the performance at the online fine-tuning phase.
Revised: SAC leverages a maximum entropy objective to encourage exploration which is helpful to improve the performance at the online fine-tuning phase.

Original: However, as shown by the figure, fine-tuning the CQL policy with SAC leads to an immediate drop in performance, which takes about 15,000 online environment steps to recover.
Revised: However, as shown by the figure, fine-tuning the CQL policy with SAC leads to an immediate drop in performance, which takes about 15,000 online environment steps to recover.

Original: We hypothesize that this drop in performance stems from the sudden change in the Q-function objective (from CQL to
Revised: We hypothesize that this drop in performance stems from the sudden change in the Q-function objective (from CQL to

Original: SAC).
Revised: SAC).

Original: As shown in figure 2(c), the critic loss spikes once the objectives changes.
Revised: As shown in figure 2(c), the critic loss spikes once the objectives changes.

Original: Additionally, as shown in figure 2(a), updating the objective results in 40 x over-estimation of the Q-values for at least several thousands of steps after switching the objective.
Revised: Additionally, as shown in figure 2(a), updating the objective results in 40 x over-estimation of the Q-values for at least several thousands of steps after switching the objective.

Original: This renders the pre-trained Q-function suboptimal and in need of more updates with online data.
Revised: This renders the pre-trained Q-function suboptimal and in need of more updates with online data.

Original: Updating the policy using this sub-optimal
Revised: Updating the policy using this sub-optimal

Original: Q-function can destroy the behaviors that were learned during the offline training phase.
Revised: Q-function can destroy the behaviors that were learned during the offline training phase.

Original: We may encounter a similar problem even for a minor modification of the objective.
Revised: We may encounter a similar problem even for a minor modification of the objective.

Original: Figureillustrates the online fine-tuning performance for CQL, AWAC, and IQL on the HalfCheetah environment with medium offline dataset for three settings: (1) no change in the objective, (2) changing the hyper-parameters at the beginning of the online fine-tuning phase to make the objective less
Revised: Figureillustrates the online fine-tuning performance for CQL, AWAC, and IQL on the HalfCheetah environment with medium offline dataset for three settings: (1) no change in the objective, (2) changing the hyper-parameters at the beginning of the online fine-tuning phase to make the objective less

Original: [Figure]
Revised: [Figure]

Original: conservative, and (3) linearly updating the values of the hyper-parameters over time so as to make the change as smooth as possible.
Revised: conservative, and (3) linearly updating the values of the hyper-parameters over time so as to make the change as smooth as possible.

Original: The hyper-parameters are the Lagrange threshold for CQL, the
Revised: The hyper-parameters are the Lagrange threshold for CQL, the

Original: Lagrange multiplier for AWAC, and the expectile value for IQL.
Revised: Lagrange multiplier for AWAC, and the expectile value for IQL.

Original: As shown by the figure, abruptly changing the CQL objective to be less conservative negatively affect the learning performance while smoothly updating the objective yields a similar level of performance as training with a fixed objective.
Revised: As shown by the figure, abruptly changing the CQL objective to be less conservative negatively affect the learning performance while smoothly updating the objective yields a similar level of performance as training with a fixed objective.

Original: IQL shows slightly better performance when the objective is updated abruptly, and AWAC training does not seem to be affected by the change in the hyper-parameter.
Revised: IQL shows slightly better performance when the objective is updated abruptly, and AWAC training does not seem to be affected by the change in the hyper-parameter.

Original: To conclude, our empirical results show that making the objective less conservative during the online phase does not significantly improve the online fine-tuning performance.
Revised: To conclude, our empirical results show that making the objective less conservative during the online phase does not significantly improve the online fine-tuning performance.

Original:  O PTIMISTIC A CTION S ELECTION M ECHANISM
Revised:  O PTIMISTIC A CTION S ELECTION M ECHANISM

Original: To explore efficiently without changing the offline training objective, we take advantage of the knowledge that the Q-function might contain about behaviors proximal to the ones the current policy produces.
Revised: To explore efficiently without changing the offline training objective, we take advantage of the knowledge that the Q-function might contain about behaviors proximal to the ones the current policy produces.

Original: Provided that the Q-function has been trained so as to not explicitly penalize outof-distribution actions (as, for instance, CQL would), we can expect the Q-function to sometimes assign reasonable values to slightly out-of-distribution state-action pairs.
Revised: Provided that the Q-function has been trained so as to not explicitly penalize outof-distribution actions (as, for instance, CQL would), we can expect the Q-function to sometimes assign reasonable values to slightly out-of-distribution state-action pairs.

Original: In this section we describe the offline to online optimistic fine-tuning (O3F) algorithm, an action-selection process that samples multiple action-candidates, and queries the Q-function to decide which action to explore.
Revised: In this section we describe the offline to online optimistic fine-tuning (O3F) algorithm, an action-selection process that samples multiple action-candidates, and queries the Q-function to decide which action to explore.

Original: 5. O PTIMISM VIA A CTION R ANKING
Revised: 5. O PTIMISM VIA A CTION R ANKING

Original: We leverage the knowledge embedded in the Q-function to choose actions that are better than the ones given by the policy.
Revised: We leverage the knowledge embedded in the Q-function to choose actions that are better than the ones given by the policy.

Original: However, only slightly out-of-distribution samples from the Q-function can be trusted since value estimates for state-action pairs not supported by the training data can have arbitrarily-incorrect values.
Revised: However, only slightly out-of-distribution samples from the Q-function can be trusted since value estimates for state-action pairs not supported by the training data can have arbitrarily-incorrect values.

Original: This motivates the design of our action-selection mechanism, as it only queries the Q-function for actions close to the ones given by the pre-trained policy.
Revised: This motivates the design of our action-selection mechanism, as it only queries the Q-function for actions close to the ones given by the pre-trained policy.

Original: As demonstrated by Algorithm 1, O3F requires an offline dataset, a policy and an ensemble of critic networks pretrained with the offline data, and the offline update rule for actor-critic training that is used for offline pre-training.
Revised: As demonstrated by Algorithm 1, O3F requires an offline dataset, a policy and an ensemble of critic networks pretrained with the offline data, and the offline update rule for actor-critic training that is used for offline pre-training.

Original: We initialize a replay buffer with the offline dataset.
Revised: We initialize a replay buffer with the offline dataset.

Original: In every step of online fine-tuning,
Revised: In every step of online fine-tuning,

Original: O3F randomly samples N candidate actions from a normal distribution, c i ∼ N ( a, σ 2 ) , where the mean a is the action given by the policy, a ∼ π ϕ ( s ) , and σ is a small constant value.
Revised: O3F randomly samples N candidate actions from a normal distribution, c i ∼ N ( a, σ 2 ) , where the mean a is the action given by the policy, a ∼ π ϕ ( s ) , and σ is a small constant value.

Original: The candidate actions are then ranked according to some scoring function f
Revised: The candidate actions are then ranked according to some scoring function f

Original: ( s, a ) → R , and the candidate with the highest score gets executed in the environment.
Revised: ( s, a ) → R , and the candidate with the highest score gets executed in the environment.

Original: The new online experience is then incorporated into the replay buffer.
Revised: The new online experience is then incorporated into the replay buffer.

Original: After every step, O3F updates the actor and the critic using a batch of data from the replay buffer and the provided offline update rule.
Revised: After every step, O3F updates the actor and the critic using a batch of data from the replay buffer and the provided offline update rule.

Original: 5. U NCERTAINTY O PTIMISM S CORING F UNCTION
Revised: 5. U NCERTAINTY O PTIMISM S CORING F UNCTION

Original: Require: Offline algorithm Update ( ϕ ( t ) , { θ i } ( t ) , D )
Revised: Require: Offline algorithm Update ( ϕ ( t ) , { θ i } ( t ) , D )

Original: Require:
Revised: Require:

Original: Offline policy π ϕ , Critic ensemble { Q θ i ( s, a ) }
Revised: Offline policy π ϕ , Critic ensemble { Q θ i ( s, a ) }

Original: Require:
Revised: Require:

Original: Offline dataset D 1: for step t in 1 , . . .
Revised: Offline dataset D 1: for step t in 1 , . . .

Original: , T do 2: a ∼ π ϕ ( a | s t ) 3:
Revised: , T do 2: a ∼ π ϕ ( a | s t ) 3:

Original: Sample n action-candidates c i ∼ N ( a, σ 2 ) 4:
Revised: Sample n action-candidates c i ∼ N ( a, σ 2 ) 4:

Original: Get best candidate ranking by expected Q-value.
Revised: Get best candidate ranking by expected Q-value.

Original: a t = arg max i 1 m (cid:80) mj
Revised: a [Equation] max [Equation]

Original: =1 (cid:2) Q θ j
Revised: (cid:2) Q [Equation]

Original: ( s t , c i ) (cid:3) 5: s t +1 ∼ p ( s t
Revised: ( s t , c i ) (cid:3) 5: s t +1 ∼ p ( s t

Original: +1 | s t , a t ) 6:
Revised: +1 | s t , a t ) 6:

Original: D ← D ∪ { ( s t , a t , r ( s t , a t ) , s t +1 ) } 7: ϕ, { θ i } ← Update ( ϕ ( t ) , { θ i } ( t ) , D ) 8: end for
Revised: D ← D ∪ { ( s t , a t , r ( s t , a t ) , s t +1 ) } 7: ϕ, { θ i } ← Update ( ϕ ( t ) , { θ i } ( t ) , D ) 8: end for

Original: We consider two design choices for the scoring function f
Revised: We consider two design choices for the scoring function f

Original: ( s, a ) : (1) the estimated Q-value, i.e., f ( s, a )
Revised: ( s, a ) : (1) the estimated Q-value, i.e., f ( s, a )

Original: =m (cid:80) mj
Revised: =m (cid:80) mj

Original: =1 (cid:2) Q θ j
Revised: =1 (cid:2) Q θ j

Original: ( s, a ) (cid:3) , and (2) an upper confidence bound of the estimated Qvalues which we refer to as the uncertainty optimism scoring function.
Revised: ( s, a ) (cid:3) , and (2) an upper confidence bound of the estimated Qvalues which we refer to as the uncertainty optimism scoring function.

Original: In both cases, O3F only queries actions with trusted values, i.e., actions close to the policy output.
Revised: In both cases, O3F only queries actions with trusted values, i.e., actions close to the policy output.

Original: This is particularly important for the uncertainty optimism scoring function, as the uncertainty of the value estimates farther away from the policy output can be arbitrarily high, which results in favoring out-of-distribution actions.
Revised: This is particularly important for the uncertainty optimism scoring function, as the uncertainty of the value estimates farther away from the policy output can be arbitrarily high, which results in favoring out-of-distribution actions.

Original: We model the uncertainty of the valueestimates by using an ensemble of independently-trained Q-functions.
Revised: We model the uncertainty of the valueestimates by using an ensemble of independently-trained Q-functions.

Original: Let { Q θ i ( s, a ) → R } be the ensemble of Q-functions with parameters { θ i } .
Revised: Let { Q θ i ( s, a ) → R } be the ensemble of Q-functions with parameters { θ i } .

Original: We define Q mean ( s, a )
Revised: We define Q mean ( s, a )

Original: = 1 m (cid:80) mi =1 (cid:2) Q
Revised: = 1 m (cid:80) mi =1 (cid:2) Q

Original: θ i ( s, a ) (cid:3) , and Q std ( s, a ) =
Revised: θ i ( s, a ) (cid:3) , and Q std ( s, a ) =

Original: (cid:113) 1 m (cid:80) mi =1 | Q
Revised: (cid:113) 1 m (cid:80) mi =1 | Q

Original: θ i ( s, a ) − Q mean ( s, a ) | 2 .
Revised: θ i ( s, a ) − Q mean ( s, a ) | 2 .

Original: The proposed scoring function is then the upper confidence bound given by (Chen et al., 2017; Ghadirzadeh et al., 2016):
Revised: The proposed scoring function is then the upper confidence bound given by (Chen et al., 2017; Ghadirzadeh et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: where ω is a hyper-parameter balancing the optimistic level of the scoring function.
Revised: where ω is a hyper-parameter balancing the optimistic level of the scoring function.

Original: For two action candidates with similar estimated values, the upper confidence bound selects the action with higher prospects to encourage exploration.
Revised: For two action candidates with similar estimated values, the upper confidence bound selects the action with higher prospects to encourage exploration.

Original: 5. I MPLEMENTATION DETAILS FOR O3F
Revised: 5. I MPLEMENTATION DETAILS FOR O3F

Original: We evaluate our method O3F applied on top of two offline RL algorithms, LAPO (Chen et al., 2022) and
Revised: We evaluate our method O3F applied on top of two offline RL algorithms, LAPO (Chen et al., 2022) and

Original: IQL (Kostrikov et al., 2021).
Revised: IQL (Kostrikov et al., 2021).

Original: For both algorithms, the offline training phases are carried out in a similar fashion to the original works.
Revised: For both algorithms, the offline training phases are carried out in a similar fashion to the original works.

Original: For LAPO, we find that training the critic in a manner similar to
Revised: For LAPO, we find that training the critic in a manner similar to

Original: Chen et al.
Revised: Chen et al.

Original: (2021) improves sample-efficiency.
Revised: (2021) improves sample-efficiency.

Original: In particular, we train an ensemble of 5 Q-functions, and use a random subset of the Q-functions to generate the training target.
Revised: In particular, we train an ensemble of 5 Q-functions, and use a random subset of the Q-functions to generate the training target.

Original: This modification lets us update the critic 20 times per environment step during the online fine-tuning phase.
Revised: This modification lets us update the critic 20 times per environment step during the online fine-tuning phase.

Original: This style of critic-training made IQL unstable, and thus we do not include it in the final method for IQL.
Revised: This style of critic-training made IQL unstable, and thus we do not include it in the final method for IQL.

Original: Instead, IQL’s critic is trained the same way as the original paper, with the distinction that we train 5 Q-functions instead of 2 to be consistent with the LAPO-based implementation.
Revised: Instead, IQL’s critic is trained the same way as the original paper, with the distinction that we train 5 Q-functions instead of 2 to be consistent with the LAPO-based implementation.

Original: Our method introduces 3 hyper-parameters: N , the number of action-candidates to generate; σ , the standard deviation of the normal distribution; and ω , the optimism weight for scoring function.
Revised: Our method introduces 3 hyper-parameters: N , the number of action-candidates to generate; σ , the standard deviation of the normal distribution; and ω , the optimism weight for scoring function.

Original: We found that N = 100 works well on every considered task.
Revised: We found that N = 100 works well on every considered task.

Original: On locomotion tasks, we found σ = 0 .to yield the best performance, and every other task uses σ = 0 .
Revised: On locomotion tasks, we found σ = 0 .to yield the best performance, and every other task uses σ = 0 .

Original: We hypothesize that because of the dense reward function of the locomotion tasks, the Q-function can potentially be estimated more globally which enables O3F to evaluate a wider range of actions, and not only the ones located close to the policy output.
Revised: We hypothesize that because of the dense reward function of the locomotion tasks, the Q-function can potentially be estimated more globally which enables O3F to evaluate a wider range of actions, and not only the ones located close to the policy output.

Original: On section 6.3 we compare the performances of ω = 1 and ω = 0 , and find that the uncertainty optimism scoring function improves performance for the halfcheetah-medium-v task, decreases performance on kitchen-mixed-v0, and maintains performance on antmaze-largediverse-v1.
Revised: On section 6.3 we compare the performances of ω = 1 and ω = 0 , and find that the uncertainty optimism scoring function improves performance for the halfcheetah-medium-v task, decreases performance on kitchen-mixed-v0, and maintains performance on antmaze-largediverse-v1.

Original: To avoid unnecessary complexity, the rest of the experiments use ω = 0 .
Revised: To avoid unnecessary complexity, the rest of the experiments use ω = 0 .

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: The goal of our experiments is to, first and foremost, evaluate how O3F compares to prior methods.
Revised: The goal of our experiments is to, first and foremost, evaluate how O3F compares to prior methods.

Original: Moreover, we aim to study whether O3F is compatible with multiple base offline RL methods and to ablate the role of different design choices within O3F.
Revised: Moreover, we aim to study whether O3F is compatible with multiple base offline RL methods and to ablate the role of different design choices within O3F.

Original: [Figure]
Revised: [Figure]

Original: 6. E XPERIMENTAL S ET -U P
Revised: 6. E XPERIMENTAL S ET -U P

Original: Tasks.
Revised: Tasks.

Original: We evaluate O3F on a variety of locomotion, manipulation
Revised: We evaluate O3F on a variety of locomotion, manipulation

Original: [Figure]
Revised: [Figure]

Original: and navigation tasks from the
Revised: and navigation tasks from the

Original: D4RL benchmark Fu et al.
Revised: D4RL benchmark Fu et al.

Original: We consider twelve total problems: six in simple environments and six in more complex environments.
Revised: We consider twelve total problems: six in simple environments and six in more complex environments.

Original: For the simple tasks, we use the D4RL Gym-MuJoCo locomotion tasks consisting of the HalfCheetah, Hopper
Revised: For the simple tasks, we use the D4RL Gym-MuJoCo locomotion tasks consisting of the HalfCheetah, Hopper

Original: [Figure]
Revised: [Figure]

Original: and Walker2D domains.
Revised: and Walker2D domains.

Original: Since we want to evaluate the ability to improve with online experience, we focus on the random and medium offline datasets, which have the lowest offline performance (as opposed to, e.g., using the expert dataset, which leaves little room for further improvement).
Revised: Since we want to evaluate the ability to improve with online experience, we focus on the random and medium offline datasets, which have the lowest offline performance (as opposed to, e.g., using the expert dataset, which leaves little room for further improvement).

Original: The random dataset consists of experiences collected by a random policy, while the medium dataset uses experiences collected by a mediocre policy trained with early stopping.
Revised: The random dataset consists of experiences collected by a random policy, while the medium dataset uses experiences collected by a mediocre policy trained with early stopping.

Original: Since the locomotion tasks are relatively simple and straightforward, we also evaluate on the ant-maze and Franka kitchen environments, which are considerably more difficult.
Revised: Since the locomotion tasks are relatively simple and straightforward, we also evaluate on the ant-maze and Franka kitchen environments, which are considerably more difficult.

Original: The ant-maze domain requires both learning to walk, as well as reaching the goal in a maze.
Revised: The ant-maze domain requires both learning to walk, as well as reaching the goal in a maze.

Original: We consider the three diverse ant-maze datasets, which involve mazes of increasing difficulty.
Revised: We consider the three diverse ant-maze datasets, which involve mazes of increasing difficulty.

Original: The Franka kitchen domain involves controlling a simulated Franka robot in a kitchen environment solving a sequence of tasks.
Revised: The Franka kitchen domain involves controlling a simulated Franka robot in a kitchen environment solving a sequence of tasks.

Original: We consider all three Franka kitchen datasets in the D4RL benchmark: “complete”, “mixed”, and “partial”.
Revised: We consider all three Franka kitchen datasets in the D4RL benchmark: “complete”, “mixed”, and “partial”.

Original: Comparisons.
Revised: Comparisons.

Original: We evaluate the performance of O3F implemented on top of both LAPO (Chen et al., 2022) and IQL (Kostrikov et al., 2021) in comparison to the prior work on offline-to-online fine-tuning.
Revised: We evaluate the performance of O3F implemented on top of both LAPO (Chen et al., 2022) and IQL (Kostrikov et al., 2021) in comparison to the prior work on offline-to-online fine-tuning.

Original: In particular, we consider three strong prior methods as baselines: (1) CQL (Kumar et al., 2020), (2)
Revised: In particular, we consider three strong prior methods as baselines: (1) CQL (Kumar et al., 2020), (2)

Original: IQL
Revised: IQL

Original: (Kostrikov et al., 2021) and AWAC (Nair et al., 2020).
Revised: (Kostrikov et al., 2021) and AWAC (Nair et al., 2020).

Original: CQL and IQL are used in offline-to-online fine-tuning settings by adding new experiences to the replay buffer and training with the same offline procedure provided all offline and online data.
Revised: CQL and IQL are used in offline-to-online fine-tuning settings by adding new experiences to the replay buffer and training with the same offline procedure provided all offline and online data.

Original: AWAC (Nair et al., 2020) is specifically designed for the online fine-tuning setting.
Revised: AWAC (Nair et al., 2020) is specifically designed for the online fine-tuning setting.

Original: For CQL and AWAC, we use d3rlpy (Takuma Seno, 2021), an offline RL library that provides implementations of multiple offline RL algorithms, and allows for online fine-tuning.
Revised: For CQL and AWAC, we use d3rlpy (Takuma Seno, 2021), an offline RL library that provides implementations of multiple offline RL algorithms, and allows for online fine-tuning.

Original: For IQL, we use the
Revised: For IQL, we use the

Original: [Figure]
Revised: [Figure]

Original: original paper’s implementation.
Revised: original paper’s implementation.

Original: For all three prior methods, we use the original hyper-parameters, including the policy and the critic function network architecture.
Revised: For all three prior methods, we use the original hyper-parameters, including the policy and the critic function network architecture.

Original: Every experiment starts with training a model only using the offline dataset, like the original work.
Revised: Every experiment starts with training a model only using the offline dataset, like the original work.

Original: Then, we initialize a replay buffer with the offline dataset, and proceed to alternate between taking a step in the environment and adding the experience to the replay buffer, and training the agent with the original offline procedure.
Revised: Then, we initialize a replay buffer with the offline dataset, and proceed to alternate between taking a step in the environment and adding the experience to the replay buffer, and training the agent with the original offline procedure.

Original: 6. M AIN R ESULTS
Revised: 6. M AIN R ESULTS

Original: Figure 4 shows the comparison of our method (O3F on top of LAPO) to the three prior methods.
Revised: Figure 4 shows the comparison of our method (O3F on top of LAPO) to the three prior methods.

Original: Comparisons of O3F with LAPO and IQL are made in section 6.3.
Revised: Comparisons of O3F with LAPO and IQL are made in section 6.3.

Original: O3F performs comparably to or better than all of the prior methods in all the tasks except for hopper-random-v2, in which
Revised: O3F performs comparably to or better than all of the prior methods in all the tasks except for hopper-random-v2, in which

Original: AWAC obtains the best performance.
Revised: AWAC obtains the best performance.

Original: O3F generally does not see a drop in performance after the start of the online fine-tuning phase, which shows that acting optimistically after the offline training phase can improve performance without sacrificing stability.
Revised: O3F generally does not see a drop in performance after the start of the online fine-tuning phase, which shows that acting optimistically after the offline training phase can improve performance without sacrificing stability.

Original: We find that CQL and AWAC are unable to solve the Franka kitchen and AntMaze tasks, with the exception of AWAC on the simplest
Revised: We find that CQL and AWAC are unable to solve the Franka kitchen and AntMaze tasks, with the exception of AWAC on the simplest

Original: AntMaze setting (umaze), while O3F on top of LAPO reliably outperforms IQL on these tasks.
Revised: AntMaze setting (umaze), while O3F on top of LAPO reliably outperforms IQL on these tasks.

Original: For the simpler locomotion tasks, O3F outperforms all the other methods for halfcheetah-random-v2, walker2d-random-v2, and halfcheetah-medium-v2, performs comparatively to the best prior method on walker2d-medium-v2 and hopper-medium-v2, and underperforms AWAC for hopper-random-v2.
Revised: For the simpler locomotion tasks, O3F outperforms all the other methods for halfcheetah-random-v2, walker2d-random-v2, and halfcheetah-medium-v2, performs comparatively to the best prior method on walker2d-medium-v2 and hopper-medium-v2, and underperforms AWAC for hopper-random-v2.

Original: On average, O3F performs 56% better than the next best method (IQL).
Revised: On average, O3F performs 56% better than the next best method (IQL).

Original: 6. F URTHER C OMPARISONS AND A BLATIONS
Revised: 6. F URTHER C OMPARISONS AND A BLATIONS

Original: Action candidate ranking : We compare O3F with the simple baseline of adding random noise to the actions.
Revised: Action candidate ranking : We compare O3F with the simple baseline of adding random noise to the actions.

Original: O3F on top of both LAPO and IQL implements the action-selection mechanism described in Section 5.1 and the implementation details in Section 5.3.
Revised: O3F on top of both LAPO and IQL implements the action-selection mechanism described in Section 5.1 and the implementation details in Section 5.3.

Original: In contrast, we compare to the original approaches of LAPO and IQL with noise added to the actions during online data collection, i.e. with noise ϵ ∼ N (0 , σ 2 ) (where σ is the same as in O3F).
Revised: In contrast, we compare to the original approaches of LAPO and IQL with noise added to the actions during online data collection, i.e. with noise ϵ ∼ N (0 , σ 2 ) (where σ is the same as in O3F).

Original: Results for this experiment can be seen in Figure 5.
Revised: Results for this experiment can be seen in Figure 5.

Original: First, we find that O3F is compatible with both LAPO and IQL, matching or outperforming the performance of the base algorithm.
Revised: First, we find that O3F is compatible with both LAPO and IQL, matching or outperforming the performance of the base algorithm.

Original: For LAPO, our approach improves performance on halfcheetah-medium-v2 by 58%, on antmaze-large-diverse-v1 by 28%, and maintains the performance for kitchen-mixed-v0, where performance was already close to saturation (we find that
Revised: For LAPO, our approach improves performance on halfcheetah-medium-v2 by 58%, on antmaze-large-diverse-v1 by 28%, and maintains the performance for kitchen-mixed-v0, where performance was already close to saturation (we find that

Original: LAPO does not surpass 0.75 returns on kitchen-mixed-v0 even with further fine-tuning.)
Revised: LAPO does not surpass 0.75 returns on kitchen-mixed-v0 even with further fine-tuning.)

Original: For IQL,
Revised: For IQL,

Original: O3F improves performance by 14% on halfcheetah-medium-v2, and by 39% on kitchen-mixed-v0, while maintaining performance on antmaze-large-diverse-v1.
Revised: O3F improves performance by 14% on halfcheetah-medium-v2, and by 39% on kitchen-mixed-v0, while maintaining performance on antmaze-large-diverse-v1.

Original: Uncertainty optimism scoring function : We compare two designs for the scoring function in our method: (1) scoring action candidates by estimating their value using the Q-function ensemble (this corresponds to f ϕ ( s, a ) from Eq.
Revised: Uncertainty optimism scoring function : We compare two designs for the scoring function in our method: (1) scoring action candidates by estimating their value using the Q-function ensemble (this corresponds to f ϕ ( s, a ) from Eq.

Original: 2 when ω
Revised: 2 when ω

Original: = 0 ); and (2) the uncertainty optimism scoring function introduced in section 5.2 ( f ϕ ( s, a ) when ω
Revised: = 0 ); and (2) the uncertainty optimism scoring function introduced in section 5.2 ( f ϕ ( s, a ) when ω

Original: = 1 ).
Revised: = 1 ).

Original: We show results for 3 environments (halfcheetahmedium-v2, antmaze-large-diverse-v1, and kitchen-mixed-v0) on Figure 6.
Revised: We show results for 3 environments (halfcheetahmedium-v2, antmaze-large-diverse-v1, and kitchen-mixed-v0) on Figure 6.

Original: The uncertainty opti-
Revised: The uncertainty opti-

Original: [Figure]
Revised: [Figure]

Original: ω =  doesn’t take uncertainty into account.
Revised: ω = 0 doesn’t take uncertainty into account.

Original:  find that the uncertainty optimism scoring function performs best in the Franka kitchen domain, but is less advantageous in the other two domains.
Revised: We find that the uncertainty optimism scoring function performs best in the Franka kitchen domain, but is less advantageous in the other two domains.

Original: mism scoring function outperformed the baseline scoring function on the kitchen environment, but it underperformed in the halfcheetah-medium-v2 environment.
Revised: mism scoring function outperformed the baseline scoring function on the kitchen environment, but it underperformed in the halfcheetah-medium-v2 environment.

Original: On the antmaze-large-diverse-v there weren’t significant differences in performance between the two scoring functions.
Revised: On the antmaze-large-diverse-v there weren’t significant differences in performance between the two scoring functions.

Original: To avoid unnecessary complexity, we focus our experiments on the baseline scoring function ( f ϕ ( s, a ) where ω = 0 ).
Revised: To avoid unnecessary complexity, we focus our experiments on the baseline scoring function ( f ϕ ( s, a ) where ω = 0 ).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a simple yet efficient offline-to-online fine-tuning strategy to further improve a pretrained offline policy.
Revised: We presented a simple yet efficient offline-to-online fine-tuning strategy to further improve a pretrained offline policy.

Original: The goal of online fine-tuning is to improve the agent’s performance by properly exploring the environment without forgetting the behaviors already learned by the offline training.
Revised: The goal of online fine-tuning is to improve the agent’s performance by properly exploring the environment without forgetting the behaviors already learned by the offline training.

Original: However, most offline RL algorithms are pessimistic in that they prohibit the agent from exploring state-action pairs not supported by the offline data.
Revised: However, most offline RL algorithms are pessimistic in that they prohibit the agent from exploring state-action pairs not supported by the offline data.

Original: Therefore, it is challenging to fine-tune the learned policy to acquire new behaviors.
Revised: Therefore, it is challenging to fine-tune the learned policy to acquire new behaviors.

Original: We empirically showed that fine-tuning with the same objective does not result in a satisfactory fine-tuning performance, and making the objective less conservative negatively affects the behaviors already learned by the offline training.
Revised: We empirically showed that fine-tuning with the same objective does not result in a satisfactory fine-tuning performance, and making the objective less conservative negatively affects the behaviors already learned by the offline training.

Original: We presented O3F, an offline to online optimistic fine-tuning algorithm which facilitates exploration without changing the objective.
Revised: We presented O3F, an offline to online optimistic fine-tuning algorithm which facilitates exploration without changing the objective.

Original: O3F improves exploration by using an action-selection method which favors optimistic actions with respect to a Q-function pre-trained with offline data.
Revised: O3F improves exploration by using an action-selection method which favors optimistic actions with respect to a Q-function pre-trained with offline data.

Original: We empirically showed that fine-tuning with O3F can on average achieve 56% more returns compared to IQL, AWAC and CQL baselines on several locomotion, navigation and manipulation tasks.
Revised: We empirically showed that fine-tuning with O3F can on average achieve 56% more returns compared to IQL, AWAC and CQL baselines on several locomotion, navigation and manipulation tasks.

Original: Limitations: There are two important limitations for the proposed algorithm: (1) it is only applicable to offline RL methods that do not explicitly penalize the Q-values of out-of-distribution state-action pairs, and (2) it can only refine the behaviors already learned in the offline phase, without the capability of discovering new behaviors.
Revised: Limitations: There are two important limitations for the proposed algorithm: (1) it is only applicable to offline RL methods that do not explicitly penalize the Q-values of out-of-distribution state-action pairs, and (2) it can only refine the behaviors already learned in the offline phase, without the capability of discovering new behaviors.

Original: We argue that the second limitation is generally applicable to all fine-tuning methods that are built on top of pessimistic offline RL algorithms.
Revised: We argue that the second limitation is generally applicable to all fine-tuning methods that are built on top of pessimistic offline RL algorithms.

Original: We will study this limitation in more details as a part of our future work.
Revised: We will study this limitation in more details as a part of our future work