Paper: Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The dependence on a hand-picked segment length K. While having some parts of the algorithm heuristically tuned is generally not a problem, I find this parameter to possibly be challenging to pick (due to its dependence on the task) and time-varying. My intuition is that the problem does not appear in the selected experiments due to their simplicity. I guess it will be much more challenging as soon as the time derivative of the reward drastically changes over time. A cue in this direction is the failure case on the table tennis task, which shows these characteristics (when the ball is close to the end-effector, the robot needs very high-resolution updates, while a longer horizon is sufficient in the waiting times). This problem is not unusual in most robot tasks: for example, in visual navigation or locomotion, the interdependence between states is rarely constant (e.g., when a sudden obstacle appears, one needs to react fast). I think the paper would benefit from studying this aspect in more detail. Could there be a connection between the derivative of the value function and the segment length K? Can it be learned with the task?

Author response:
- The segment length, or the number of segments per trajectory, does indeed affect performance. However, we observed that within an optimal range, such as 10-25 segments per trajectory, this parameter is not overly sensitive compared to other hyperparameters. Thus, we use K=25 in all of our experiments. A detailed study on the influence of the number of segments is provided in the updated supplementary material, Section R.5, which was previously reported in Section C.7.
- The approach we use in TCE for learning value functions aligns with that of standard RL methods, such as PPO. In our model, the segment length K influences only the policy update and does not affect the learning of the value function. This method of leveraging value functions is a key factor in why TCE achieves faster learning speeds compared to BBRL[4] in most tasks we reported. However, it also leads to a similar performance decline in the table tennis task, mirroring the challenges faced by other step-based RL methods that rely on value function learning.

Reviewer comment (Evaluation):
Some of the results in the experimental setup are difficult to interpret. In Fig. 6b-c, all methods appear to perform similarly in the dense and sparse setting, which I find surprising. Even more so, almost all approaches (including gSDE, which achieves the best performance) perform better with sparse rewards than dense ones. At least for step-wise methods, dense should do better. Why does this happen?

Author response:
- Dense and sparse rewards settings    
    We attribute the observed phenomenon to two main factors. Firstly, as previously discussed in [4], sparse reward settings often provide a clearer task description, aligning the reward structure more directly with the success evaluation criteria. Secondly, for some of the step-based algorithms, we employed a discount factor of 1.0, as opposed to the usual 0.99. This adjustment mitigates the task's sparsity, potentially enhancing the performance of these methods. However, since the task is constrained by a limited number of time steps, the episode return remains within a finite range. Despite these considerations, some step-based algorithms still exhibit poorer performance compared to their counterparts in dense reward settings.

Reviewer comment (Experiments):
The results should be reported for a longer number of interactions. In Fig. 5, 6, and 7, the methods do not appear to be fully converged. Showing results up to convergence (as in Fig. 5a) would give a better intuition about the approach's strength in comparison to the baselines.

Author response:
- Longer training iterations
    
    Previsouly, we found a few baelines collapsed after a certain training iterations. And for off-policy methods, due to their update-to-data (UTD) ratio of 1, as suggested in references [6, 7], their training often take 1-2 days in the reported tasks. However, the main challenge is not the collection of samples, but the network updates and the overhead involved in data exchange between the policy and the environments. Considering that off-policy methods are designed for enhanced sample efficiency, their failure to learn effectively from a data volume comparable to that utilized by on-policy methods highlights their inferior performance in the tasks evaluated in the paper.

Reviewer comment (Experiments):
The baselines are not defined (e.g., what paper is BBRL?).

Author response:
BBRL stands for Deep Black Box Reinforcement Learning, as described in [4], an episodic RL method that learns trajectories directly in the parameter space. We will update our paper to offer a proper introduction of it.

Reviewer comment (Theory):
The paper lacks a detailed derivation of the objective in the proposed multi-step setup.

Author response:
- Our learning objective as presented in Equation (8) can be viewed as an approximation of the episodic RL’s learning objective in Equation (1), rather than being derived from other Policy Gradient Method. This approximation is achieved by replacing the trajectory likelihood and advantage with their respective segment-wise versions.

Reviewer comment (Experiments):
There is no discussion and possibly no experiments on the value of K, nor a more in-depth analysis to justify the experimental setup.

Author response:
- The segment length, or the number of segments per trajectory, does indeed affect performance. However, we observed that within an optimal range, such as 10-25 segments per trajectory, this parameter is not overly sensitive compared to other hyperparameters. Thus, we use K=25 in all of our experiments. A detailed study on the influence of the number of segments is provided in the updated supplementary material, Section R.5, which was previously reported in Section C.7.
- The approach we use in TCE for learning value functions aligns with that of standard RL methods, such as PPO. In our model, the segment length K influences only the policy update and does not affect the learning of the value function. This method of leveraging value functions is a key factor in why TCE achieves faster learning speeds compared to BBRL[4] in most tasks we reported. However, it also leads to a similar performance decline in the table tennis task, mirroring the challenges faced by other step-based RL methods that rely on value function learning.

Reviewer comment (Novelty):
The paper would benefit from a more organized exposition of its contributions. It appears that applying policy gradient to segments is the primary novel component. And this work integrates techniques from Li et al. (2023) and Otto et al. (2021). A clearer differentiation from these prior works would enhance the clarity of the paper.

Author response:
While building upon the foundations laid by previous approaches like ProDMP and TRPL, we believe that TCE stands out as a significant algorithmic advancement in RL. As a novel framework, TCE distinguishes itself by integrating step-based information into the policy updates of episodic RL. This integration not only boosts update efficiency but also maintains trajectory smoothness and supports correlated action exploration, both temporally and across degrees of freedom (DoF). TCE is the first work to successfully merge the benefits of both RL methodologies, marking a novel advancement in the field.

Reviewer comment (Experiments):
I feel that the current experimental setup does not sufficiently justify certain design choices. Consider incorporating ablation studies to address the following:  
a. The necessity of the differentiable policy projection, as compared to alternatives like those used in PPO.  
b. The benefits of employing full covariance over diagonal variance in the trajectory representation, especially in light of the increased computational cost.

Author response:
We include additional study in the updated supplementary material. Section R.1 highlights the advantages of using full covariance in enhancing trajectory smoothness. Section R.2 examines the capacity of various methods to model movement correlation, with TCE and BBRL Cov demonstrating the greatest flexibility in trajectory sampling. However, it's crucial to note that the ability to model movement correlation is not the only factor contributing to successful performance. The efficiency of policy updates plays a significant role as well, which is a key reason why TCE outperforms BBRL in the majority of tasks we have reported. Similarly, PINK's performance limitations can be attributed to constraints in learning its Q-function, a challenge it shares with SAC. These factors may account for their relatively poorer performance in some tasks.

Reviewer comment (Presentation):
Formatting: Certain pages in the main pdf are images, making it impossible to do a text search in the pdf and hindering readability.

Author response:
We apologize for the inconvenience, this problem is fixed in the updated version of the paper.

Reviewer comment (Theory):
Is there a feasible approach to directly represent each segment, rather than representing the entire trajectory?

Author response:
Some studies, such as [5], have employed movement primitives to represent individual segments. However, this approach has been shown to underperform compared to methods that model the entire trajectory, like BBRL, as previously discussed in [4].

Reviewer comment (Presentation):
I suggest the authors to spend more time re-organizing this paper. The overall description is very messy at present. There is no good connection between the various parts, making it difficult to follow the storyline.

Author response:
We are grateful for the time and effort you have dedicated to reviewing our submission. In response to your valuable feedback, we are currently re-structuring the storyline of our paper to enhance its readability. An updated version will be provided shortly. In the meantime, we have addressed the other concerns and questions you raised to the best of our knowledge, as detailed below.

### **[Update Log, Nov. 18.2023]**

Address the major concerns of all reviewers.

Upload additional experiment and ablation study to Supplementary Material.

### **[Update Log, Nov. 22, 2023]**
Restructure the paper's storyline, and address the remaining concerns from the reviewers

### **Writing:**

- We agree with you that the storyline and the connection to other works need to be improved. The revised version will be updated shortly.

Reviewer comment (Writing):
The writing of this paper should be improved. The current expression is too redundant, especially in the LIKELIHOOD COMPUTATION OF A SAMPLED TRAJECTORY SEGMENTS section. The author should reasonably remove some content to refine it.

Author response:
The main updates in the manuscript that related to your feedback:

- Revise the abstract and restructure the introduction section to explicitly define the concepts of Step and Episodic RL.
- Clarify the term "Open the Black Box" in the title, as well as the research gap that we want to solve.
- Emphasize the main contribution of the paper, and discuss how our framework differs from the existing ones.
- Split the previous Section 2 "Background and Related Works" into 
  - a) "Preliminaries" as Section 2, focusing on the methods foundational to our framework.
  - b) "Related Works" as a new Section 4, discussing literature approaches that address similiar research gaps.
- Link Episodic RL to the MDP in Section preliminaries.
- Review and refine the claims at the end of Section 2.1, ensuring they are supported by relevant references.
- For each preliminary method discussed in Section 2, clearly state the specific problem it targets and its relevance to Episodic RL.
- Provide detailed explanations of the mathematical variables used in our framework, including their dimensions.
- Remove redundant content in Section 3 and streamline its structure for better clarity and conciseness.
- Include a discussion on the works closely related to ours, with a special focus on the development of Episodic RL
- Discuss BBRL (Deep Black Box Reinforcement Learning) in detial as it is highly relevant to our work.

Reviewer comment (Evaluation):
The simulation experiment results do not verify the effectiveness of the method. Compared with strong step-based methods, the proposed method does not demonstrate the obvious final performance improvement. Although the variance of the proposed method is the lowest in the CONTACT-RICH MANIPULATION section, the gSDE approach achieves better performance in 2 of 3 tasks. Moreover, the BBRL Cov. algorithm achieves a 20% higher success rate with faster convergence in the HITTING TASK.

Author response:
- While we acknowledge that certain baselines, such as gSDE in the box-pushing sparse task and BBRL-cov in table tennis, outperform TCE in specific scenarios, we believe that TCE consistently delivers comparable results across most tasks. Notably, in the MetaWorld tasks, TCE not only achieved the highest asymptotic performance but also maintained the most consistent performance profile. This consistency is characterized by successfully solving a majority of tasks and maintaining a high success rate in those it can solve. Although gSDE shows strong results in box-pushing tasks, its overall performance in the MetaWorld's 50 tasks is moderate, and it fails entirely in the table tennis task. In stark contrast, TCE still manages to achieve a commendable success rate of 65% in the latter.

Reviewer comment (Experiments):
The authors are suggested to experiment with some real scenes. The simulation environments now set by the authors can not provide a big enough challenge to verify the robustness of the algorithm.

Author response:
- Real Scenes
    
    We agree that comparing these methods in real-world environments would be very useful. However, as mentioned in our paper, many RL baselines have a problem with smooth movements in their trajectories, which could harm real hardware. Following reviewer dLeV's suggestion, we added a new section on trajectory smoothness in the updated supplementary material (Section R.1). This section shows that most step-based RL methods have very low smoothness, even with well-trained policies. On the other hand, using movement primitives has been successful in real tasks, like imitation learning [1] and Sim2Real transfer in RL [2]. Yet, for real-world applications, we believe it essential to first test and confirm the effectiveness of the policy in a simulation environment before transitioning it to real-world scenarios.
    

**References**

[1] Gomez-Gonzalez, Sebastian, et al. "Adaptation and robust learning of probabilistic movement primitives." IEEE Transactions on Robotics 36.2 (2020): 366-379.

[2] Klink, Pascal, Hany Abdulsamad, Boris Belousov, and Jan Peters. "Self-paced contextual reinforcement learning." In Conference on Robot Learning, pp. 513-529. PMLR, 2020.

Reviewer comment (Novelty):
The novelty of the algorithm is incremental. It is a direct application of Li et al. 2023 to the RL regime, by (1) changing the action space to the parameter space of trajectories, and (2) enlarging the timestep to the segmentation scale.

Author response:
### **Novelty & Opening the Black Box**

Despite the leverage of previous approaches, we believe TCE has made a distinct algorithmic advancement in the field of RL. As a novel framework, TCE uniquely incorporates step-based information into the policy updates of episodic RL, simultaneously preserving trajectory smoothness and facilitating correlated action exploration both temporally and across DoF. Thus, by effectively integrating the strengths of the two RL methodologies, TCE successfully 'opens the black box’.

Reviewer comment (Theory):
Since this is still an RL approach, define the Markov Decision Process first. What is the action space? What is the state space? Now you can explain how your approach differs from traditional RL more clearly, by providing us the details in this different MDP. A tricky question here is that since MDP is Markovian, why do you need the temporal correlation?

Author response:
### **Writing**:

- Figure 1 is an illustration, not an experiment
- We will add a MDP description in the updated paper. However, our learning framework maintains the Markovian property, as discussed in the Chapter 3.1 of [9]. The Markovian property specifically concerns the state transition probability, $p(s_{t+1}, r_{t} |~s_{t}, a_{t})$, indicating that the next state and reward depend solely on the current state and action. It does not impose any assumptions on the policy or the decision process.
- d_mean and d_cov are general notations for discrepancy measures, we specifically use KL-divergence in this work as it has been shown to be effective for policy learning within the parameter space of Motion Primitives [5].

Reviewer comment (Presentation):
As a reader, I feel many unimportant texts are taking too many places, which makes it not easy to understand the paper. For example, Sections 3.1, 3.2, and 3.4 don't need so many places to mention, since they are merely (a) complete-trajectory sampling, (b) GAE, and (c) TRPO-like. Instead, providing the pseudo-code for the whole learning process would be more valuable as a complement to Figure 3. Currently, there are too many technical things and curves in Figure 3, which makes it hard to understand the learning procedure from this single figure.

Author response:
Dear reviewer dLeV,

We are glad to hear that our responses and updates to the paper have addressed your major concerns. Your feedback has also provided us with valuable insights on how to evaluate the effectiveness of our approach from various aspects. We truly appreciate your suggestions.
Regarding the pseudo-code, we actually added an algorithm box in the supplementary material, **Section R.8**. You can click this link to get the latest version of it:

https://openreview.net/attachment?id=mnipav175N&name=supplementary_material

Regarding the four MetaWorld tasks, we are equally curious about the challenges they present. We observed that the BBRL method, which closely related to our work, encounters similar difficulties in these tasks. Interestingly, there are discussions in the Metaworld GitHub repository indicating potential issues with the reward configurations for button press family. Notably, two of our unsuccessful tasks are part of this category, suggesting that these potential bugs might be a reason for the failures we've encountered.

- Discussions to these potential bugs in Metaworld

  https://github.com/Farama-Foundation/Metaworld/issues/389

  https://github.com/Farama-Foundation/Metaworld/issues/433


Thank you and best regards,

TCE Authors

Reviewer comment (Evaluation):
The result is not sufficient to show the significance of the proposed method.

Author response:
### **Performance against Baselines**
While we recognize that the performance gains in certain environments may be marginal, it is crucial to highlight that our algorithm, TCE, consistently demonstrates superior or at least comparable results in the majority of tasks when benchmarked against robust baselines. A notable example is the table tennis task, where gSDE failed entirely, yet our algorithm achieved a 60% success rate. We acknowledge TCE's underperformance in 5 out of the 50 Metaworld tasks. However, considering the wide variety of task types in Metaworld, this is not unusual. Similar to other baseline methods, which also exhibit variable performance across different tasks, TCE has not been specifically optimized for any individual task. Despite this, it has shown exceptional overall performance across the comprehensive Metaworld benchmark. This not only underscores TCE's robustness but also its adaptability, making it a highly effective method in a broad spectrum of scenarios.

Reviewer comment (Evaluation):
After checking Appendix C.2, I found that the baselines perform generally well when TCE works well. However, there exist some situations where TCE significantly fails and the other baselines still work well, and these situations are not rare, including LeverPull, StickPull, Hammer, ButtonPressTopdown, and ButtonPressTopdownWall.

Author response:
Dear reviewer dLeV,

We are glad to hear that our responses and updates to the paper have addressed your major concerns. Your feedback has also provided us with valuable insights on how to evaluate the effectiveness of our approach from various aspects. We truly appreciate your suggestions.
Regarding the pseudo-code, we actually added an algorithm box in the supplementary material, **Section R.8**. You can click this link to get the latest version of it:

https://openreview.net/attachment?id=mnipav175N&name=supplementary_material

Regarding the four MetaWorld tasks, we are equally curious about the challenges they present. We observed that the BBRL method, which closely related to our work, encounters similar difficulties in these tasks. Interestingly, there are discussions in the Metaworld GitHub repository indicating potential issues with the reward configurations for button press family. Notably, two of our unsuccessful tasks are part of this category, suggesting that these potential bugs might be a reason for the failures we've encountered.

- Discussions to these potential bugs in Metaworld

  https://github.com/Farama-Foundation/Metaworld/issues/389

  https://github.com/Farama-Foundation/Metaworld/issues/433


Thank you and best regards,

TCE Authors

Reviewer comment (Evaluation):
The curves for SAC and PINK are terminated after 10M and 27M interactions respectively, whereas the other methods terminate at 50M interactions, which is an unfair comparison. It is hard to show the efficiency of this algorithm unless these baselines are finished. The reason is: in many cases, these baselines would raise their success rates nonlinearly during the training process, and we cannot conclude by only inspecting their intermediate performances.

Author response:
### **Training horizon of off-policy methods**
The original SAC employs an update-to-data (UTD) ratio of 1, as suggested in references [10], a configuration known for balancing training stability with sample efficiency. However, this ratio does not leverage the advantages of parallel processing across multiple environments. The main challenge is not the collection of samples, but the network updates and the overhead involved in data exchange between the policy and the environments. This issue often restricts training to a relatively lower number of samples, typically ≤ 10M steps, as described in [10]. Considering that off-policy methods are designed for enhanced sample efficiency, their failure to learn effectively from a data volume comparable to that utilized by on-policy methods highlights their inferior performance in the tasks evaluated in the paper.

Reviewer comment (Experiments):
Several key ablation studies are missing to test the soundness of each proposed component. See the questions.

Author response:
### **Ablation Studies:**

- We include a smoothness evaluation in the updated supplementary material, Section R.1. The evaluation is conducted using three common smoothness metrics in robotics and motion planning. The comparison shows TCE and the BBRL Cov have achieved the highest smoothness among all the approaches.
- In Section R.2 of the updated supplementary material, we present a qualitative evaluation of the action correlation for all methods. This evaluation highlights the varying capacities of these methods in modeling movement correlations. Both TCE and BBRL Cov successfully model correlations across both time and inter-DoF. In contrast, the original BBRL and PINK are limited to modeling correlations within individual DoF. gSDE demonstrates the capability to model temporal correlations, but only over a few consecutive time steps. Notably, PPO, TRPL, and SAC do not engage in action correlation modeling.
    
    However, it is important to emphasize that the capability to model movement correlation is not the sole determinant of successful performance. The BBRL, for instance, is hindered by an inefficient policy update strategy. Similarly, PINK's performance limitations can be attributed to constraints in learning its Q-function, a challenge it shares with SAC. These factors may account for their relatively poorer performance in some tasks.
    
- We agree that including tests for PPO and SAC with Movement Primitives (MP) action space is an important ablation study. Previous work [5] has already shown the ineffectiveness of combining PPO with Movement Primitives (MP).
    
    In the updated supplementary material, Section R.3, we include an additional ablation study where we combined SAC with MP parameters as the action space. The results from this study suggest that this combination is not effective. We believe this ineffectiveness arises from the increased dimensionality of the action space, which significantly complicates the learning process for the Q-function.
    
- Using p(w) instead of p(y) for likelihood is exactly what BBRL [5] did. We already included that as a baseline, but we agree that more discussion of the method should be included in the experiment section to clarify the differences, and we are currently working on that.

Reviewer comment (Writing):
What does black box in the title stand for? Can you help me accurately define it? Does it mean the dynamics is black-box, or the policy approximator (neural network) is black-box? Though this term is repeated several times in the paper, there is no accurate definition in the paper. Furthermore, since the title says 'open the black box', is your algorithm a white-box algorithm?

Author response:
### **Episodic RL & Black-Box Optimization**

Episodic Reinforcement Learning (RL), as discussed in references [1,2,3,4], is a distinct branch in the evolution of RL that emphasizes maximizing the return of an entire episode, rather than focusing on the internal dynamics of the environment. This approach shifts the search for solutions from the per-step action space to the trajectory parameter space, thereby promoting broader exploration. In this context, entire trajectories are often treated as single data points, and models are typically trained using black-box optimization methods [5,6,7,8]

The term 'black box' in our title reflects the essence of these methods, rooted in black-box optimization principles, which tend to overlook the detailed step-based information acquired during interactions with the environment. While these methods excel in expansive exploration and trajectory smoothness, they generally require a larger number of samples for effective policy training. In contrast, step-based RL methods have shown substantial improvements in learning efficiency by leveraging this step-based information.

Reviewer comment (Experiments):
Is Figure 1 a real experiment, or just an illustrative explanation?

Author response:
### **Writing**:

- Figure 1 is an illustration, not an experiment
- We will add a MDP description in the updated paper. However, our learning framework maintains the Markovian property, as discussed in the Chapter 3.1 of [9]. The Markovian property specifically concerns the state transition probability, $p(s_{t+1}, r_{t} |~s_{t}, a_{t})$, indicating that the next state and reward depend solely on the current state and action. It does not impose any assumptions on the policy or the decision process.
- d_mean and d_cov are general notations for discrepancy measures, we specifically use KL-divergence in this work as it has been shown to be effective for policy learning within the parameter space of Motion Primitives [5].

Reviewer comment (Experiments):
Notably, the off-policy methods SAC and PINK were trained with less samples than used for on-policy methods due to their limitations in parallel environment utilization. What does the limitation refer to here? Why cannot SAC and PINK be deployed to parallel training? (Also, 'fewer' samples, not 'less' samples).

Author response:
### **Training horizon of off-policy methods**
The original SAC employs an update-to-data (UTD) ratio of 1, as suggested in references [10], a configuration known for balancing training stability with sample efficiency. However, this ratio does not leverage the advantages of parallel processing across multiple environments. The main challenge is not the collection of samples, but the network updates and the overhead involved in data exchange between the policy and the environments. This issue often restricts training to a relatively lower number of samples, typically ≤ 10M steps, as described in [10]. Considering that off-policy methods are designed for enhanced sample efficiency, their failure to learn effectively from a data volume comparable to that utilized by on-policy methods highlights their inferior performance in the tasks evaluated in the paper.

Reviewer comment (Evaluation):
Smoothness comparison. Define a metric for smoothness, and evaluate how all the final trajectories for all the methods perform under this metric. Since you claim that TCE helps the smoothness, then only showing the success rate is not sufficient.

Author response:
### **Ablation Studies:**

- We include a smoothness evaluation in the updated supplementary material, Section R.1. The evaluation is conducted using three common smoothness metrics in robotics and motion planning. The comparison shows TCE and the BBRL Cov have achieved the highest smoothness among all the approaches.
- In Section R.2 of the updated supplementary material, we present a qualitative evaluation of the action correlation for all methods. This evaluation highlights the varying capacities of these methods in modeling movement correlations. Both TCE and BBRL Cov successfully model correlations across both time and inter-DoF. In contrast, the original BBRL and PINK are limited to modeling correlations within individual DoF. gSDE demonstrates the capability to model temporal correlations, but only over a few consecutive time steps. Notably, PPO, TRPL, and SAC do not engage in action correlation modeling.
    
    However, it is important to emphasize that the capability to model movement correlation is not the sole determinant of successful performance. The BBRL, for instance, is hindered by an inefficient policy update strategy. Similarly, PINK's performance limitations can be attributed to constraints in learning its Q-function, a challenge it shares with SAC. These factors may account for their relatively poorer performance in some tasks.
    
- We agree that including tests for PPO and SAC with Movement Primitives (MP) action space is an important ablation study. Previous work [5] has already shown the ineffectiveness of combining PPO with Movement Primitives (MP).
    
    In the updated supplementary material, Section R.3, we include an additional ablation study where we combined SAC with MP parameters as the action space. The results from this study suggest that this combination is not effective. We believe this ineffectiveness arises from the increased dimensionality of the action space, which significantly complicates the learning process for the Q-function.
    
- Using p(w) instead of p(y) for likelihood is exactly what BBRL [5] did. We already included that as a baseline, but we agree that more discussion of the method should be included in the experiment section to clarify the differences, and we are currently working on that.

Reviewer comment (Experiments):
The significance of the exact probability calculation. If you abandon this component and just use the normal \(p(w)\) instead of \(p(y)\), how would the performance change empirically?

Author response:
### **Ablation Studies:**

- We include a smoothness evaluation in the updated supplementary material, Section R.1. The evaluation is conducted using three common smoothness metrics in robotics and motion planning. The comparison shows TCE and the BBRL Cov have achieved the highest smoothness among all the approaches.
- In Section R.2 of the updated supplementary material, we present a qualitative evaluation of the action correlation for all methods. This evaluation highlights the varying capacities of these methods in modeling movement correlations. Both TCE and BBRL Cov successfully model correlations across both time and inter-DoF. In contrast, the original BBRL and PINK are limited to modeling correlations within individual DoF. gSDE demonstrates the capability to model temporal correlations, but only over a few consecutive time steps. Notably, PPO, TRPL, and SAC do not engage in action correlation modeling.
    
    However, it is important to emphasize that the capability to model movement correlation is not the sole determinant of successful performance. The BBRL, for instance, is hindered by an inefficient policy update strategy. Similarly, PINK's performance limitations can be attributed to constraints in learning its Q-function, a challenge it shares with SAC. These factors may account for their relatively poorer performance in some tasks.
    
- We agree that including tests for PPO and SAC with Movement Primitives (MP) action space is an important ablation study. Previous work [5] has already shown the ineffectiveness of combining PPO with Movement Primitives (MP).
    
    In the updated supplementary material, Section R.3, we include an additional ablation study where we combined SAC with MP parameters as the action space. The results from this study suggest that this combination is not effective. We believe this ineffectiveness arises from the increased dimensionality of the action space, which significantly complicates the learning process for the Q-function.
    
- Using p(w) instead of p(y) for likelihood is exactly what BBRL [5] did. We already included that as a baseline, but we agree that more discussion of the method should be included in the experiment section to clarify the differences, and we are currently working on that.

Reviewer comment (Evaluation):
One thing I am wondering is how this temporal correlation helps the sampling complexity. Can you provide the final Sigma for each DoF? Ideally, this matrix should be far from the diagonal matrix, otherwise, we can just use the step-wise algorithm.

Author response:
### **Ablation Studies:**

- We include a smoothness evaluation in the updated supplementary material, Section R.1. The evaluation is conducted using three common smoothness metrics in robotics and motion planning. The comparison shows TCE and the BBRL Cov have achieved the highest smoothness among all the approaches.
- In Section R.2 of the updated supplementary material, we present a qualitative evaluation of the action correlation for all methods. This evaluation highlights the varying capacities of these methods in modeling movement correlations. Both TCE and BBRL Cov successfully model correlations across both time and inter-DoF. In contrast, the original BBRL and PINK are limited to modeling correlations within individual DoF. gSDE demonstrates the capability to model temporal correlations, but only over a few consecutive time steps. Notably, PPO, TRPL, and SAC do not engage in action correlation modeling.
    
    However, it is important to emphasize that the capability to model movement correlation is not the sole determinant of successful performance. The BBRL, for instance, is hindered by an inefficient policy update strategy. Similarly, PINK's performance limitations can be attributed to constraints in learning its Q-function, a challenge it shares with SAC. These factors may account for their relatively poorer performance in some tasks.
    
- We agree that including tests for PPO and SAC with Movement Primitives (MP) action space is an important ablation study. Previous work [5] has already shown the ineffectiveness of combining PPO with Movement Primitives (MP).
    
    In the updated supplementary material, Section R.3, we include an additional ablation study where we combined SAC with MP parameters as the action space. The results from this study suggest that this combination is not effective. We believe this ineffectiveness arises from the increased dimensionality of the action space, which significantly complicates the learning process for the Q-function.
    
- Using p(w) instead of p(y) for likelihood is exactly what BBRL [5] did. We already included that as a baseline, but we agree that more discussion of the method should be included in the experiment section to clarify the differences, and we are currently working on that.