Paper: SEER: Towards Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation (ICLR-2024)

Reviewer comment (Novelty):
The proposed algorithm seems to only modify the existing framework of maximum entropy RL a little bit and not very novel.

Author response:
**A1**: Thanks for your question. The general framework of PbRL consists of the reward learning and policy learning. SEER employs SAC as its backbone algorithm in policy learning, similar to baselines like PEBBLE [1], SURF [2], and MRN [3]. Main Contirbutions lie in, SEER leverages the non-parametric model to sample informative trajectories and regularize neural Q function for feedback-efficient learning.

Reviewer comment (Theory):
The paper only shows that empirical Q function is a lower bound of the ground-truth optimal Q function. I think the distribution-constrained loss is more reasonable if the authors can further show the empirical Q function is close to the ground-truth optimal Q function.

Author response:
**A2**: Thank you for your question. In response to your concern about the empirical Q function's closeness to the ground-truth optimal Q function, we have conducted additional experiments. The results are as follows:
  
Table 1: MSE of $\hat{Q}$ and $Q_\theta$ across ten runs  
| Task/Q | $\hat{Q}$ | $Q_\theta$|  
| -------- | -------- | -------- |  
| Push-5x5-1   | 0.05±0.07  | 0.06±0.10 |  
| Block-shaped | 0.07±0.03  | 0.08 ±0.03|  

This result represents the performance of the empirical Q and the neural Q  across different tasks. As indicated by the result, the empirical Q closely approximates the ground-truth optimal Q value.

Reviewer comment (Theory):
Why do the authors use the KL divergence between $\hat{\pi}(s)$ and $\pi_{ soft(\theta)}(s)$ as the regularization term? A more intuitive choice would be the direct distance between $Q_{\theta}$ and $\hat{Q}$.

Author response:
**A3**: Thanks for your thoughtful question. The decision to use KL divergence is rooted in our objective to align the learned policy as closely as possible with the optimal policy derived from the Q-function. This choice is inspired by the SAC, where the KL divergence is used to minimize the distance between the current policy and the exponentiated softmax of the Q-function, as detailed in [4] Equation 4.

Reviewer comment (Experiments):
The main weakness with the paper is in the experimental evaluation. All of the other baselines test on the same broad suite of simulated robotics tasks. Since this paper follows from those papers and directly compares against them, those environments should definitely be included in the evaluation.

Author response:
**A1**: Thank you for your valuable feedback. SEER primarily utilizes a discrete graph for conservative estimations, which inherently aligns more with discrete action domains and therefore is not very suitable for continuous action domains. However, we highlight that SEER demonstrates the benefit of regularization based on conservative estimation for feedback-efficient learning. This finding opens up a wide range of research possibilities for future studies. In terms of addressing continuous action scenarios, we are considering the adoption of alternative approaches, such as offline RL methods [1], to achieve conservative estimation without explicitly constructing a graph.

Reviewer comment (Writing):
Another weakness is with the presentation. There are a lot of grammar mistakes throughout, for example the first sentence in abstract should be “One of the challenges… “ also in abstract “optimizing policies”, also citation (III and Sadigh) should be Hejna III and Sadigh.

Author response:
**A2**: Thank you for your valuable feedback regarding the presentation aspects of our paper. We will review the entire manuscript to correct these issues. Regarding the citations, we will ensure that all references, such as 'Hejna III and Sadigh', are correctly formatted and cited according to the required style guidelines.

Reviewer comment (Theory):
I comprehend the author's assertion regarding the conservatism of the empirical Q-function. However, I am seeking clarification regarding the formal properties of the resulting policy, denoted as SAC $\pi_{\phi}$. Are we anticipating it to exhibit conservatism or optimism? Additionally, the author contends that it "aligns with human preference." Could this alignment be elucidated in a more rigorous manner?

Author response:
**A1**: Thanks for your question. In our approach, the primary objective is to obtain an accurate Q function to enhance policy learning in PbRL. To this end, we employ an empirical Q-value, which embodies a conservative estimation approach, to constrain the neural Q function. This strategy is designed to accelerate and refine the policy learning process. Additionally, the term 'aligns with human preference' in this context implies that a more precise Q-value estimation correlates with greater alignment to human preferences within the PbRL framework.

Reviewer comment (Theory):
The author state that "theoretically, we demonstrate that the empirical Q-function is a lower-bound of ...." in the Abstract. However, it is challenging to discern the precise details from Theorem 3.1. in a main text. Several elements remain undefined, such as the exact meanings of $Q_t$ and $\hat Q_t$ in the main text, as well as the underlying assumptions (e.g., do we need assumptions for rewards to say $\hat Q_t$ converges to $Q^{\star}$?).

Author response:
Regarding Theorem 3.1, we acknowledge that it assumes environments are tabular. In light of this, we will revise the theorem section to more clearly state that our analysis is conducted within the context of a finite state-action space. Meanwhile, this revision will include providing the Bellman optimality equation for $Q_t$ to ensure it is clearly defined and understood.

Reviewer comment (Theory):
The proposed framework appears to be tailored for tabular settings, primarily due to its reliance on an empirical Q-function. How does the author intend to extend this approach to accommodate continuous settings, particularly in terms of both algorithmic and theoretical considerations?

Author response:
**A3**: Thank you for your insightful question. To adapt our framework to continuous settings, there are several strategies. Firstly, it can consider the incorporation of discretization techniques to transform continuous action spaces into discrete equivalents. This method is common in similar contexts; for instance, RT-2 [1] successfully applies it for discretizing actions in robotic arm applications. Secondly, it can also introduce an action translator to generate continuous actions, akin to the VMG model [2], which uses a graph-structured world model and convert generated actions via an action translator.

Reviewer comment (Experiments):
In a related context, update (3) seems somewhat naive in addressing the data coverage concern. I agree it might be beneficial to differentiate between actions that have not been visited, actions that have been visited. But how can we distinguish actions that fall in between – perhaps those visited frequently versus those visited infrequently?

Author response:
**A4**: Thank you for your valuable feedback regarding update rule (3) in our paper. In our method, we use $N(s,a,s')$ to represent the number of visits to different actions. The empirical dynamics $\widehat{p}(s'|s,a) = N(s, a, s')/\sum_{s'} N(s, a, s')$ already encapsulate a distinction in visit frequencies. For instance, actions that have been visited frequently will have a higher value of $N(s, a, s')$, reflecting in a higher empirical probability, whereas infrequently visited actions will have lower values. This differentiation is crucial for ensuring that our update mechanism does not overlook less explored actions.