Paper: Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment (ICLR-2024)

Reviewer comment (Experiments):
The approach introduces an additional hyperparameter $\beta$ that must be tuned. I am also unaware of how sensitive the approach is to this hyperparameter (whether most selected values will work well and beat baselines or whether only a small handful are appropriate).

Author response:
> How sensitive is the approach to the $\beta$ hyperparameter (whether most selected values will work well and beat baselines or whether only a small handful are appropriate)?

We ran ROAM with 4 different values (0.01, 0.1, 0.5, 0.9) of $\beta$ in each simulated suite and show the performance below. For both evaluations, 3 out of 4 of these values (all except 0.01) outperform all the other baselines. 

|      | Dynamic Friction | Dynamic Stiffness |
|------|------------------|-------------------|
| beta | Avg # Steps      | Avg # Steps       |
|------|------------------|-------------------|
| 0.01  | 7610 +- 854      | 2698 +- 844       |
| 0.1    | 2082 +- 382      | 1331 +- 263       |
| 0.5  | 772 +- 179       | 628 +- 19         |
| 0.9  | 1466 +- 534      | 735 +- 54         |

Reviewer comment (Theory):
The approach somewhat changes the definition of the Bellman operator such that it also contains a notion of the policies propensity to have encountered a given state instead of being based solely on the expected cumulative reward. Moreover, some values of $\beta$ appear to have potentially odd behaviors. For example, selecting $\beta=1$ appears to make the Bellman operator no longer depend on the reward signal?

Author response:
> Some values of $\beta$ appear to have potentially odd behaviors. For example, selecting $\beta=1$ appears to make the Bellman operator no longer depend on the reward signal. Can the authors clarify this?

Thanks for bringing this up, and we apologize for any confusion. In practice, it makes sense to use ROAM only with values of $\beta < 1$, and we only use such $0 < \beta < 1$ in our experiments. 

Thank you for your reply and we apologize for the confusion. In our initial code implementation, we did not include the (1-beta) term in front of the first term of the loss and simply scaled the second term with a range of beta values (which has the same effect). We ended up reporting those values in the first version. We reran ROAM with the above listed beta values, where we include the (1-beta) on the first term, which gives the same performance as the previous implementation. We ask that you may take a look at the updated pdf, where everything is now consistent. Please let us know if we can clarify this further. We thank you for your detailed attention, which has significantly improved the clarity of our paper.

Reviewer comment (Experiments):
There is perhaps some issues of fairness compared to RMA and HLC baselines. The authors use a state-of-the-art RLPD actor-citric method for their base learning approach. If RMA and HLC baseline methods also use actor-critic agents, were they also updated to use RLPD? If not, I would perhaps be concerned that the performance benefits reported may be in part due to RLPD instead of the author’s proposed method.

Author response:
> There is perhaps some issues of fairness compared to RMA and HLC baselines. The authors use a state-of-the-art RLPD actor-citric method for their base learning approach. If RMA and HLC baseline methods also use actor-critic agents, were they also updated to use RLPD? If not, I would perhaps be concerned that the performance benefits reported may be in part due to RLPD instead of the author’s proposed method.

Thanks for this question, and we apologize for any confusion. We built all methods, including ROAM, RMA, and HLC on top of the same state-of-the art implementation of SAC as the base learning approach. For all comparisons, we additionally use a high UTD ratio, dropout, and layernorm, following DroQ (Hiraoka et al. 2022), and for any method that does fine-tuning, we use 50/50 sampling following RLPD (Ball et al. 2022). We have revised the paper to clarify this.