Paper: Improving Offline RL by Blending Heuristics (ICLR-2024)

Reviewer comment (Theory):
$\tilde{r}$ dependence on the behavior policy may cause problems when data is collected using multiple behavior policies. Consider a dataset with data from a mixture of policies (say the medium replay case in D4RL), and for simplicity suppose we assume a constant lambda, say 0.5. Now, since the reward $\tilde{r}$ is conditioned on the behavior policy, doesn’t the effective MDP (reconstructed using the relabeled dataset) become non stationary? Is the performance not affected much due to the deterministic nature of the environments chosen for evaluation?

Author response:
> r dependence on the behavior policy may cause problems when data is collected using multiple behavior policies.

In the theoretical analysis, we made a simplifying assumption that the heuristic $h$ is a function of the state in Theorem 1. In this case, HUBL can be viewed as solving a stationary reshaped MDP. In Theorem 2, we further consider a special case that  $h(s)$ is the value function of a policy e.g. the behavior policy $\mu$ when proving an exact finite sample upper bound. This extra assumption that the heuristic is a policy’s value function is made as a sufficient condition to establish the heuristic is improvable (see [Definition 4.1, Cheng et al., 2021]), which roughly means that the heuristic value is attainable and as a result can guide the learning (i.e. $\tilde{V}^*(s) \geq h(s)$ [Proposition 4.4, Cheng et al., 2021]). This sufficient condition is used to show the second term in the regret  $h(s’) - \tilde{V}^\pi(s’)$ can be non-positive: 

$\textnormal{Regret}(\pi,  h, \lambda):= 
         \tilde{V}^{\pi^*}(d_0) - \tilde{V}^{{\pi}}(d_0) + \frac{\gamma}{1-\gamma}E_{(s,a,s')\sim d^\pi}[\lambda(s') ( h(s') - \tilde{V}^{{\pi}}(s')) | s, s'\in\Omega ]$

On the other hand, we totally agree that in practical scenarios with mixed behavior policies, the estimated heuristics in Section 4.2 may lead to a non-stationary MDP. As discussed above, the current analysis would work through if this heuristic (in expectation) turns out to be improvable. We think analyzing the exact condition for this to happen in a scenario of mixed policies is interesting, but it’s beyond the scope of the current work. When the resulting heuristic is not improvable, it eventually will lead to an extra error term in Section 4.2. As a result, the performance of HUBL is affected by how well the heuristic function is calculated, which is affected by many factors like the randomness of the environment as you pointed out, the amount of data, the coverage of the data, and so on. Specifically for the randomness in environments, we provided experiments of HUBL with stochastic environments in Section D.3, where HUBL still provides performance improvement.

Reviewer comment (Experiments):
Choosing $\lambda$ may not be straightforward. The choice of $\lambda$ should be such that it is high for trajectories with high returns, and low otherwise. But this is hard to determine if the dataset consists of trajectories that perform equally well. This can be seen in results shown in figure 2, as significant improvement is seen on datasets with a mixture of behavior policies because there is a way to determine the appropriate choice of $\lambda$ based on relative performance. Now, suppose you have a dataset with low rewards (the random variant in D4RL), then the ranking method will still assign high ranks to most trajectories, and this might result in poor performance as the Monte Carlo estimates using the behavior policy might cause instability.

Author response:
> Choosing lambda may not be straightforward 

When the trajectories perform equally well, HUBL with Rank labeling behaves exactly as you expected: the trajectories will have similar lambda. However, this does not mean that HUBL will necessarily hurt the performance, as there is $\alpha <1$ discounting the amount of heuristics blended into bootstrapping. Empirically, we applied HUBL with Rank labeling on hopper-random-v2, walker2d-random-v2 and halfcheetah-random-v2. The results are provided below. We can see that HUBL is able to improve the performance. However, in this setting, the performance improvement of HUBL is limited. We believe the reason is as you suggested: the $\lambda$ function could not find well-performing trajectories to blend with bootstrapping. A better design for $\lambda$ is required for such scenarios, which is one of our future directions. We will include such discussions in the final version   of the submission. 
|                       | Relative Performance Improvement of HUBL | Base  |
| --------------------- | ---------------------------------------- | ----- |
| hopper-random-v2      | 4%                                       | 15.18 |
| walker2d-random-v2    | 7%                                       | 3.94  |
| halfcheetah-random-v2 | 0%                                       | 26.1  |

Reviewer comment (Theory):
Why is the training of a value function needed during step 2?

Author response:
> Why is the training of a value function needed in step 2.

We are not quite sure which step you are referring to as we do not train a value function in Step 2 on page 4. We train a value function in Step 1: Computing heuristic $h_t$ on page 4 (footnote 3). That is because sometimes a trajectory ends not due to reaching a terminal stage but due to a timeout. When this happens, the simple Monte-Carlo sum of the rewards underestimates the value of this trajectory. Therefore, we need to estimate a value function to add at the end of such trajectories.

Reviewer comment (Writing):
For someone not already familiar with offline RL, it can be challenging to follow the comprehensive theoretical analysis developed in this paper, especially in the appendices.

Author response:
Thank you for pointing out this presentation issue. We have uploaded an updated draft by adding more intuitions and explanations to the proof in Appendix A and B. Such modifications are highlighted in blue.

Reviewer comment (Experiments):
I believe the manuscript needs a brief discussion on how the proposed model would be applied to problems with sparse rewards. For instance, in environments like Antmaze, has it been tested?

Author response:
Following your suggestion, we conducted additional experiments on applying HUBL with TD3+BC and Rank labeling to antmaze-medium-diverse-v0,  antmaze-medium-play-v0, antmaze-umaze-diverse-v0, and antmaze-umaze-v0. The results are provided below. HUBL is able to provide performance improvement to TD3+BC in the considered environments. 
|                           | Relative Performance Improvement of HUBL | Base |
| ------------------------- | ---------------------------------------- | ---- |
| antmaze-medium-diverse-v0 | 67%                                      | 30.0 |
| antmaze-medium-play-v0    | 17%                                      | 60.0 |
| antmaze-umaze-diverse-v0  | 61%                                      | 60.0 |
| antmaze-umaze-v0          | 43%                                      | 70.0 |

Theoretically, we did not pose any assumptions on whether the reward is sparse or not. Potentially, HUBL can be more helpful in sparse-reward environments if given a proper heuristic.  The reason is that even if the reward is sparse, the heuristic can be dense and guides decision-making. We defer a thorough analysis of HUBL in sparse-reward environments to future work. We will include such discussions in the final version of the submission.

Reviewer comment (Presentation):
The presentation of the experimental results in the graphs lacks clarity. The absence of a horizontal baseline at 0 makes it unclear whether there's an improvement or decline. A horizontal baseline at 0 should be added, and different colors could be considered to depict increases and decreases.

Author response:
We have uploaded an updated draft, where the plots and tables are modified as you suggested. 
For figures, a horizontal line at 0 is added to each plot, and bars are colored in red and blue, with blue for positive performance improvement and red for negative improvement.

Reviewer comment (Presentation):
The experimental tables in the appendix have a similar problem. The best performances should be bolded for easier readability.

Author response:
We have uploaded an updated draft, where the plots and tables are modified as you suggested. 
For tables, the best performing configuration in each row is highlighted in bold.

Reviewer comment (Experiments):
As the authors discussed in the limitations section, offline datasets based on disconnected transition tuples are challenging to utilize with the HUBL trick unless heuristic values are computed during the construction of datasets.

Author response:
Although having trajectory data is a prerequisite of the current design of HURL, most real world decision-making data are actually collected as trajectories. Collecting independent transitions is rarely feasible. Thus, while we impose an assumption that might seem strong in the mathematical sense, it is actually weak in practice.

Reviewer comment (Presentation):
Please improve the presentation of figures and tables in the paper as mentioned above.

Author response:
We have uploaded an updated draft, where the plots and tables are modified as you suggested. 
For figures, a horizontal line at 0 is added to each plot, and bars are colored in red and blue, with blue for positive performance improvement and red for negative improvement.
For tables, the best performing configuration in each row is highlighted in bold.

Reviewer comment (Theory):
On tasks where baseline offline RL algorithms already perform well, HUBL might decrease the performance. Is there a way to ensure that its enhancements are consistently non-negative?

Author response:
Indeed, HUBL may occasionally hurt performance empirically on some datasets – this can happen in situations where offline RL algorithms already attain near-optimal results. Such performance decrease is more due to *finite sample error*, as the room for improvement on these datasets is small – for any offline learning technique. Nonetheless, in most medium-expert datasets where the base RL method already performs well, HUBL still provides positive performance improvement. Overall,  the main motivation behind HUBL is to consistently improve the performance of the base algorithm whenever the baseline algorithm performs suboptimally on its own (compared with other offline RL algorithms), and HUBL succeeds at this, sometimes with a relative improvement up to 50%. 

It is an interesting direction to design a method that can guarantee non-negative *expected* improvement. The tricky bit is that different offline RL methods have different characteristics like regret, approximation error, optimization error and so on. Therefore, it is hard to design one $\lambda$ that is rigorously guaranteed to improve expected performance for various offline RL methods without overtunning $\lambda$ for each problem.

Reviewer comment (Evaluation):
On D4RL, HUBL shows significant improvements over baseline algorithms for the hopper task several times. I wonder whether there exist any shared or general characteristics in situations where HUBL offers substantial advantages. Could this be discussed, or did I perhaps overlook any mention of this aspect?

Author response:
The empirical performance improvement of HUBL is most significant when the base offline RL method shows inconsistent performance and performs suboptimally compared to other offline RL algorithms. This can be seen from the results in Appendix D.2, where we provide the performance of the base RL algorithms in the experiments.