Paper: Bayesian Offline-to-Online Reinforcement Learning : A Realist Approach (ICLR-2024)

Reviewer comment (Experiments):
Experiments were solely performed on the less challenging locomotion tasks. Obtaining results from the more demanding antmaze tasks would provide stronger evidence.

Author response:
**A for W1:**
The results of Antmaze tasks are shown in Table 2 in the original paper. As suggested, we conducted more experiments on the Antmaze tasks.
The following table shows that BOORL outperforms various baselines on these challenging tasks.


| | BOORL | ODT | Cql-QL | PEX | 
| :---: | :---: | :---: | :---: | :---: |
| antmaze-umaze-v2 | 80.0$\rightarrow$**100.0** | 55.0$\rightarrow$85.0 | 80.0$\rightarrow$**100.0** | 80.0$\rightarrow$95.0 |
| antmaze-umaze-diverse-v2 | 75.0$\rightarrow$**95.0** | 50.0$\rightarrow$80.0 | 75.0$\rightarrow$**95.0** | 75.0$\rightarrow$80.0 |
| antmaze-medium-play-v2 | 50.0$\rightarrow$**100.0** | 0.0$\rightarrow$0.0 |  60.0$\rightarrow$90.0 | 70.0$\rightarrow$85.0 |
| antmaze-medium-diverse-v2 | 60.0$\rightarrow$**85.0** | 0.0$\rightarrow$0.0 | 70.0$\rightarrow$**85.0** | 60.0$\rightarrow$**85.0** |
| antmaze-large-play-v2 | 60.0$\rightarrow$**75.0** | 0.0$\rightarrow$0.0 | 25.0$\rightarrow$55.0 | 50.0$\rightarrow$55.0|
| antmaze-large-diverse-v2 | 50.0$\rightarrow$**80.0** | 0.0$\rightarrow$0.0 | 35.0$\rightarrow$70.0 | 45.0$\rightarrow$55.0|

Table 1. Comparison results between BOORL and baselines in Antmaze tasks with the normalized score metric, averaged over five random seeds with standard deviation.

Reviewer comment (Reproducibility):
The performance of PEX significantly deviates from the original results and my personal experience, indicating a potential bug in the code or suboptimal parameter tuning.

Author response:
**A for W2:**
In the offline-to-online setting, we aim to have a swift and robust improvement for the offline policy during the online phase.
For this reason, we present the performance of the online fine-tuning policy of 200k steps, while the original PEX paper presents results of 1M steps.
We provide the comparative results of 1M steps in Appendix G. As shown in the result, PEX's performance aligns with the results of the original paper, and BOORL significantly outperforms PEX.

Reviewer comment (Presentation):
The legend in Figure 2 can be put to the top of two images to avoid overlapping with the curves.

Author response:
**A for W3:**
Thanks for your suggestions; we have updated the figures in the revised version.

Reviewer comment (Experiments):
There are some missing SOTA baselines for offline-to-online fine-tuning in the experiments: Reincarnating RL [1] and InAC [2]. Moreover, the current results of PEX seems to be problematic.

Author response:
**A for Q1:**
As suggested, we compare BOORL with the SOTA baselines, Reincarnating RL and InAC.
The experimental results in the following table show that BOORL outperforms these two baselines. Reincarnating RL proposes a decayed behavior cloning constraint (or cross-entropy loss in discrete action space), but the choice of the scheduling of the decay process can greatly affect the performance. InAC provides strong offline performance, but there is no explicit mechanism for accelerating online fine-tuning.


| | BOORL | Reincarnating RL | InAC |
| :---: | :---: | :---: | :---: |
| hopper-random-v2 | 8.8$\rightarrow$**75.7** | 8.1$\rightarrow$8.6 | 32.0$\rightarrow$35.1 |
| hopper-medium-v2 | 61.9$\rightarrow$**109.8** | 60.7$\rightarrow$93.0 | 60.3$\rightarrow$80.9 |
| hopper-medium-replay-v2 | 75.7$\rightarrow$**111.1** | 73.1$\rightarrow$73.3 | 92.1$\rightarrow$97.8 |
| hopper-medium-expert-v2 | 89.0$\rightarrow$103.4 | 87.9$\rightarrow$**111.0** | 93.8$\rightarrow$93.3 |
| hopper-expert-v2 | 111.5$\rightarrow$**109.2** | 101.5$\rightarrow$106.1 | 103.4$\rightarrow$**108.7** |
| walker2d-random-v2 | 4.8$\rightarrow$**93.6** | 4.6$\rightarrow$6.9 | 2.0$\rightarrow$8.5 |
| walker2d-medium-v2 | 83.6$\rightarrow$**107.7** | 81.2$\rightarrow$95.0 | 82.7$\rightarrow$89.7 |
| walker2d-medium-replay-v2 | 69.1$\rightarrow$**114.4** | 67.8$\rightarrow$92.0 | 69.8$\rightarrow$95.5 |
| walker2d-medium-expert-v2 | 110.8$\rightarrow$**116.2** | 108.9$\rightarrow$111.1 | 109.0$\rightarrow$112.2 |
| walker2d-expert-v2 | 110.0$\rightarrow$**118.6** | 109.7$\rightarrow$110.8 | 110.6$\rightarrow$110.9 |
| halfcheetah-random-v2 | 10.7$\rightarrow$**97.7** | 8.9$\rightarrow$43.6 | 9.5$\rightarrow$42.1 |
| halfcheetah-medium-v2 | 47.9$\rightarrow$**98.7** | 46.0$\rightarrow$65.5 | 48.3$\rightarrow$63.8 |
| halfcheetah-medium-replay-v2 | 44.5$\rightarrow$**91.5** | 45.2$\rightarrow$60.2 | 44.3$\rightarrow$55.6 |
| halfcheetah-medium-expert-v2 | 77.7$\rightarrow$**97.9** | 78.4$\rightarrow$92.6 | 83.5$\rightarrow$92.9 |
| halfcheetah-expert-v2 | 97.5$\rightarrow$**98.4** | 96.7$\rightarrow$**97.8** | 93.6$\rightarrow$92.4 |
| $\delta_{\rm sum}(0.2\rm M)$ | **540.4** | 189.7 | 144.5 |

Table 1. Comparison results between BOORL and additional baselines with the normalized score metric, averaged over five random seeds with standard deviation.

Reviewer comment (Presentation):
In Figure 5, the three curves usually have the same starting points except for "hopper-medium-replay-v2", "walker-medium-replay-v2", and "halfcheetah-medium-expert-v2". Why does BOORL have a different value at step 0 in these tasks?

Author response:
**A for Q2:**
The main reason is that we adopt an ensemble method.
During the offline phase, we train $N$ policies and Q-value networks, and at each step, we select a policy randomly to generate an action. Each policy and Q-value network is trained on different samples due to the use of masks (i.e., bootstrapping). Therefore, our method has a different starting point for some tasks compared to standard offline methods. The performance of our method reduces to the same performance as TD3+BC at step 0 if we set $N= 1$.

Reviewer comment (Theory):
From my understanding, the proposition 3.3 is trying to argue that using UCB in online stage will cause the performance drop? I am not sure that one-step suboptimality corresponds exactly to the "performance drop".

Author response:
Proposition 3.3 is used to show that there is no per-episode performance guarantee for UCB (but a cumulative regret guarantee) due to optimistic exploration, and the performance for one episode can be arbitrarily worse when the online phase starts. On the contrary, offline learned policy has an $O(1/\sqrt{N})$ per-episode performance guarantee. Therefore, Proposition 3.3 explains why changing the offline algorithm to an online one can lead to a performance drop at the early stage.

Reviewer comment (Novelty):
At the ending remark of the theory section, the paper mentions that [Song et al., 2022] benefits only when offline data has sufficient coverage, but to my best knowledge it seems that [Song et al., 2022] indeed requires and benefit from sufficient coverage from offline data. So to my understanding the current paper is achieving a best-of-both-world (kind of, not exactly) results of [Xie et al, 2021] and [Song et al., 2022].

Author response:
We thank the Reviewer for pointing this out. More precisely, Xie et al., 2021 analyze the benefit of online exploration when offline data only has partial coverage, while Song et al., 2022 propose Hy-Q, which has an oracle property ($O(\sqrt{T})$ regret compared with **any** policy $\pi_e$) and is computationally efficient by incorporating offline data. Unlike the previous results, our result shows that using offline data with full coverage can improve over the $O(\sqrt{T})$ online regret bound by adopting a Bayesian method. We made this point more clear in the revised version.

Reviewer comment (Theory):
Some indexing on $h$ seems to be off in eq 3. Also in eq 4, are the linear features not $h$-dependent?

Author response:
We thank the Reviewer for pointing it out, and we have corrected it in the revised version. A $h$-independent feature is usually assumed as in previous linear MDP literature [3,4]. We can concatenate $h$-dependent features for all $H$ steps to get an $h$-independent one so that $h$-dependent features are a special case of our definition.

Reviewer comment (Novelty):
In Section 4's discussion of the replay buffer, the authors employ a symmetric sampling design, a technique previously proposed and validated in multiple prior works, e.g., [BSKL23] and [Ross et al., 2012]. It is essential to ensure that these references are comprehensively cited to acknowledge the contributions they provide and give readers a better context.

Author response:
Thanks for your comment. We have corrected it in the modified version.

Reviewer comment (Writing):
It looks like the information ratio $\Gamma_t$ lacks of a formal definition in the paper. If so, please include this in the revision.

Author response:
We thank the Reviewer for pointing this out. The information ratio $\Gamma_t$ is the ratio between the suboptimality and the information gain [6]. We have added a concrete definition for $\Gamma_t$ in the Definition 2.2 in the revised version.

Reviewer comment (Experiments):
Figure 2 / Appendix E: the experimental setting is slightly unclear. For UCB and LCB, what is the algorithm applied in the offline phase? Wouldn't a fair comparison for TS be using LCB at the offline phase and UCB at the online phase? The authors should also explicitly state the bandit setting (e.g., the distribution arm probabilities) for people to replicate the experiments.

Author response:
We would like to clarify that there is no explicit "offline" stage for offline-to-online RL, theoretically. The choice of algorithms in the offline phase does not change the overall regret curve since it only affects the performance at step 0. For UCB/LCB algorithms, we treat all the offline data as past online experiences to calculate the statistics. For example, when we have $M$ offline data points and $N$ online data points for some arm $a$, then we use these $N+M$ data points to estimate $\mu(a)$ and $\sigma(a)$ and choose the best arm to act according to the UCB/LCB principle. We let all algorithms use both offline and online data to ensure a fair comparison.

In the bandit setting, we let each arm $a_n$ have a probability $\theta_n$ to yield a reward of 1 and a probability $1-\theta_n$ to yield a reward of 0. The parameters $\theta_n$ are i.i.d. drawn from a Beta distribution for all arms $a_n$. We use 10 arms and 1000 offline data points, and the online phase lasts for 100000 steps. 
We have included the implementation for the bandit settings in the supplementary material to ensure better replication.

Reviewer comment (Theory):
A naive extension would be to use pessimistic TS (cf., [A23]) in the offline phase + optimistic TS (cf., [HZHS23]) in the online phase. Would the same analysis framework apply?

Author response:
As discussed in W3, the choice of algorithms in the offline stage theoretically does not change the overall performance. While optimistic TS provides a nice frequentist guarantee compared to vanilla TS, it suffers from the same issue as UCB, which can lead to a significant performance drop at an early stage due to optimistic exploration.

Reviewer comment (Theory):
The key part of the algorithm is not clear. Specifically, the mask m samples from a distribution M. What is the format of M and how does it initialize and update? what is the difference between the usage of m compared to the original ensemble method? In the online part, it seems that the distribution M can be seen as the priori according definition of Bayesian. However, the authors choose to use uniform distribution to choose policy, which is the same as the original ensemble method. Could the authors explain it?

Author response:
We process the mask distribution consistent with [1] to simulate the bootstrapping process. 
Specifically, $ M$ is a Bernoulli distribution with a parameter $ p$, and we sample mask $ m_1, m_2,..., m_N$ i.i.d. from Bernoulli ($ p$). Therefore, each mask generates a bootstrapped dataset (i.e., iid sampling with replacement) from the original one. We draw new masks for each new online data point when it is added to the replay buffer, but the masks are not updated after that.
Each policy trained with one bootstrapped dataset can be seen as a sample from the posterior [1]. Therefore
 uniformly choosing a policy in the sampled set of $N$ policies is an approximation of sampling from the posterior from a Bayesian point of view.




The original ensemble method uses the same dataset and different initialization of neural networks to train ensembled policies. While this method can reduce variance, it does not properly reflect epistemic uncertainty [3]. The usage of the mask is derived from the bootstrapping method where each policy uses its own dataset, i.i.d. sampled with replacement from the original dataset. This helps diversify the policies and can reflect the posterior properly.