Paper: Decoupled Actor-Critic (ICLR-2024)

Reviewer comment (Experiments):
The paper does not compare against the ensemble-based methods [e.g., REDQ], which I believe is relevant and necessary.

Author response:
We kindly ask the Reviewer to investigate Figures 1, 9 & 10 of the original manuscripts to find the evaluation of DAC against exactly REDQ. As follows, REDQ achieves significantly worse results than other considered algorithms in the high replay benchmark. This result is consistent with [1], where SR-SAC (high replay SAC with resets) massively outperforms REDQ across various replay regimes. For completeness, we included both REDQ and SR-SAC in the evaluation presented in Figure 1, but dropped it in most of the latter evaluations because of substantial performance differences between REDQ and other algorithms.

Reviewer comment (Experiments):
On the general applicability of the idea, will this idea work with different value-to-policy generation rules?

Author response:
Our approach is trivially applicable to other value-based continuous actions algorithms like DDPG, TD3 or Dreamer. It would require some accommodations to be applied to discrete actions algorithms like Rainbow or BBF. We are happy to expand on the answer if the Reviewer shares which particular algorithm the Reviewer is interested in.

Reviewer comment (Presentation):
There is no text pointing to Figure 1 and Figure 2. Also, many of the abbreviations are used since the beginning of the paper but are introduced at very late stages. The overall presentation of the paper can be improved.

Author response:
We thank the Reviewer for this suggestion! We added text in the main body that describes why Figures 1 & 2 are important for our narrative. We make some additional changes to make sure that abbreviations are explained earlier in the text.

Reviewer comment (Evaluation):
The authors do not disclose any pitfalls of the DAC algorithm. Is it consistently better than SAC/TD3/other baselines? This is a huge claim, and would definitely be a huge strength of the work if it is true.

Author response:
We ask the Reviewer to note that our original manuscript contains a “Limitations” section, where we describe pitfalls of our proposed DAC algorithm. Those limitations include:

1. Complexities of the tandem setting - “DAC divergence minimization presents unique optimization challenges. Unlike typical uses of KL divergence, where the target distribution remains fixed (eg. Variational Autoencoders (VAE)), DAC deals with a constantly evolving policy that is continually improving. Consequently, the optimistic actor needs to keep up with the conservative actor’s changes. As depicted in Figure 4, DAC heavily relies on maintaining a low divergence between the actors. While DAC adjustment mechanisms proved effective in the tested environments, there is no guarantee that they will suffice in more complex ones.”
2. Optimistic actor architecture - “The second drawback of DAC lies in its inherent use of two actor networks, which results in slightly increased memory and computational demands compared to the standard Soft Actor-Critic (SAC) approach. In practice, the wall-clock time of DAC is around 10% greater than that of SAC and is indistinguishable from the overhead induced by OAC, which requires additional backpropagation through the critic ensemble. Moreover, since DAC initializes both actors with identical parameters, they must share the same network architecture. However, as indicated by Figure 4, simply copying parameters between them offers only minimal performance enhancement. In light of this, we believe that the necessity for identical architectures can be mitigated by employing techniques like delayed policy updates or by learning rate scheduling.” 

As DAC builds on SAC, we focus on limitations stemming from components specific to DAC. We add a line in the limitations section mentioning that DAC inherits limitations of SAC.

Reviewer comment (Theory):
It is known in the literature that off-policy learning can suffer from the [tandem problem]. How do you solve such difficulty when updating the conservative actor using the data generated by the optimistic actor?

Author response:
We kindly ask the Reviewer to note that Section 3.2 of our original manuscript describes the KL penalty term that ensures that the two policies do not diverge substantially. In this section we write: “KL divergence, the second objective term, regularizes the optimistic policy. (...) Using KL stabilizes the off-policy learning by ensuring that the sampled trajectories are probable under the conservative actor policy.” Furthermore, in Section A2 we write: “the optimistic policy objective features a KL divergence penalty. As shown in ablation studies in Figure 4, such a penalty is of crucial importance for DAC performance. The penalty is applied only to the optimistic actor and serves multiple purposes: (...)”. To this end, the KL penalty term is introduced specifically to restrict the negative effects of the tandem setting. Furthermore, we employ hard parameter copying as a final fail-safe mechanism against divergence of two actors. As such, DAC goes a long way to restrict the negative effects of the tandem setting on learning.

Reviewer comment (Reproducibility):
What is the backbone of DAC? How do the authors explicitly model the variance and mean of the policy?

Author response:
DAC is just SAC with an additional “optimistic” actor network (which is updated according to a rule described in Section 3.2 & 3.3). In Section 3 we write: “DAC introduces two distinct actor networks: an optimistic one and a conservative one. (...) While policies can be represented by various parameterized distributions, we implement both actors as simple diagonal normal distributions, transformed by the hyperbolic tangent activation.” Additionally, Section A.2 goes in detail about why we used soft learning paired with hyperbolic tangent transformed policies.

Reviewer comment (Presentation):
It is unclear how does Figure 2b and 2c shows the critic disagreement for different actors?

Author response:
Figure 2b & 2c show conservative and optimistic policies and related sampled state-action samples (denoted by different dot colors - red for optimistic and black for conservative). As follows, the policies are not massively different which is due to a small KL divergence between the two policies which is enforced by DAC. Despite similarity of the policies, the optimistic policy allocates some of the allowed KL budget to achieve bigger variance than the baseline conservative policy (indicated by the smaller likelihoods of the optimistic policy); secondly, the optimistic policy leads the agent to the corners of the state-action space which as shown in Figure 2a yields most critic disagreement. If the Reviewer wishes, we are happy to create a new graph where the optimistic policy is less penalized for the divergence and should thus lead to more contrasting policies.

Reviewer comment (Experiments):
Dreamer results in Figure 11 are looking good, while harder tasks such as Quadruped Run and Humanoid Walk are missing – Dreamer-v2 is able to solve these tasks well even for visual-control, why not compare on tasks like Quadruped/Humanoid/etc. (or even extend to visual control)?

Author response:
> Dreamer results in Figure 11 are looking good, while harder tasks such as Quadruped Run and Humanoid Walk are missing – Dreamer-v2 is able to solve these tasks well even for visual-control, why not compare on tasks like Quadruped/Humanoid/etc.?

We used the results provided by the authors of DreamerV3 in their official repository. As such, we restricted evaluation to the 18 DMC tasks considered in DreamerV3 prio evaluation. DreamerV3 has pretty substantial compute requirements as compared to model-free algorithms. To that end, we are unsure if the potential small changes to the evaluation (there are already 18 tasks in the benchmark) mandate the cost of running Dreamer on additional environments.

> (or even extend to visual control)?

We agree that prio-based control is not where Dreamer shines. However, we also think that using DAC in visual tasks would require non-trivial architectural choices. Such non-trivial choices stem from the intersection of using two actor networks, the common practice in visual control to use a shared encoder for actor-critic structure, and the recently researched impact of gradient weighting between actor-critic in visual tasks [1]. As such, we would have to ablate on various design choices regarding architecture and gradient passing in our extended actor-critic context. As such, we think that extending DAC to visual control validates a separate research endeavor. We kindly ask the Reviewer to note that it is a common practice to evaluate a continuous-control RL algorithm on a prio-based benchmark alone [2, 3, 4].

Reviewer comment (Experiments):
More complex Control Suite tasks (Figures 12 & 13) like Humanoid Walk/Run should be run for longer as they have not converged, yet, while recent papers have also evaluated on the Dog domain.

Author response:
> More complex Control Suite tasks (Figures 12 & 13) like Humanoid Walk/Run should be run for longer as they have not converged yet, while recent papers have also evaluated on the Dog domain

DMC 500k/1mln has become a popular evaluation protocol used in recent works [1,5,6]. We think that running many tasks of varying difficulty for a consistent amount of timesteps makes sense from the aggregate evaluation perspective. However, we also agree that harder, unsolved tasks are more interesting than well researched propositions like cheetah. To this end, we added a new experiment: dog {walk, run, trot} and humanoid {walk, run} on 3mln environment steps - following the Reviewer’s expectations. We describe the experiment in our meta-response. There, we compare SR-SAC, DAC and a recent baseline pointed by the Reviewer (TD-MPC). Please find a summary table below:

|    task   | SR-SAC |  DAC |   DAC+  | TD-MPC | DAC+ | TD-MPC |
|:---------:|:------:|:----:|:-------:|:------:|:----:|:------:|
|    mean   |   214  |  439 | **607** |   546  |  **756** |   725  |
| env steps |  1mln  | 1mln |   1mln  |  1mln  | 3mln |  3mln  |

In particular, SR-SAC does not seem to work on dog domain. This result is consistent with the evaluation performed in the TD-MPC manuscript.

Reviewer comment (Experiments):
SAC is in general a good baseline, however, it would be nice to also compare performance to a more “DMC-native” baseline such as D4PG or (D)MPO to provide another reference point.

Author response:
> SAC is in general a good baseline, however, it would be nice to also compare performance to a more “DMC-native” baseline such as D4PG or (D)MPO to provide another reference point

We are happy to share that we added D4PG and MPO to the Dreamer evaluation. The results are consistent with the Dreamer manuscript (ie. DAC > DreamerV3 > D4PG > MPO)

Reviewer comment (Writing):
There are quite a few missing articles / words + typos that should be fixed.

Author response:
> There are quite a few missing articles / words + typos that should be fixed

We made a sweep through the manuscript, we will make sure that the camera-ready version is fixed!

Reviewer comment (Evaluation):
Fish and Swimmer are very stochastic tasks due to random goal placement, but the results in Figure 12 & 13 still look extremely variable with confidence intervals barely visible. Could you double check how the evaluations are computed? Do all runs use the same evaluation seed (e.g. same eval goal across all seeds)?

Author response:
> Fish and Swimmer are very stochastic tasks due to random goal placement, but the results in Figure 12 & 13 still look extremely variable with confidence intervals barely visible. Could you double check how the evaluations are computed? Do all runs use the same evaluation seed (e.g. same eval goal across all seeds)?

We built our implementations on the codebase associated with [6] and did not change the environment wrappers in any way. We think that the variability of our results is consistent with [6], considering the differences in the evaluation protocol (10 vs. 5 seeds; 3 std vs. 95% CI)

Reviewer comment (Experiments):
It might be worth briefly discussing the general impact of replay ratios across algorithm implementation, as the “low replay” regime with 3 gradient steps per 1 environment steps seems to be significantly higher than e.g., Acme’s MPO default of 1 gradient steps per 8 environment steps (or 1/1 for Acme’s SAC). The impact of replay ratios can be wild.

Author response:
> It might be worth briefly discussing the general impact of replay ratios across algorithm implementation, as the “low replay” regime with 3 gradient steps per 1 environment steps seems to be significantly higher than e.g., Acme’s MPO default of 1 gradient steps per 8 environment steps (or 1/1 for Acme’s SAC). The impact of replay ratios can be wild.

We are happy to add an appendix section discussing the impact of replay ratio on RL algorithms. Indeed, we believe many approaches could be revisited, given that (replay ratio + resets) is applicable to almost any off-policy algorithm. Unfortunately, some robot hardware might impose inherent restrictions to the replay ratio. As such, we believe researching new algorithms in both high and low replay settings is valuable. Furthermore, we would like to note that the baselines ran within our experiments (ie. Figure 6) use consistent replay ratio and reset schedules.

Reviewer comment (Presentation):
The caption of Table 2 mentions 10 HARD DMC tasks, while Pendulum Swingup and Cartpole Swingup Sparse would not be considered hard (arguably, Cheetah, Quadruped, Fish aren’t hard either)?

Author response:
> The caption of Table 2 mentions 10 HARD DMC tasks, while Pendulum Swingup and Cartpole Swingup Sparse would not be considered hard (arguably, Cheetah, Quadruped, Fish aren’t hard either)?

We thank the Reviewer for pointing this mistake out. Indeed, the model-free evaluation considers tasks from easy to hard levels of difficulty. We change the Table 2 caption in the manuscript.

Reviewer comment (Experiments):
Have you also tried DroQ as an even more recent addition / alternative to REDQ?

Author response:
> Have you also tried DroQ as an even more recent addition / alternative to REDQ?

Unfortunately we haven’t tried DroQ. We used the JaxRL library which features only an implementation of REDQ.

Reviewer comment (Experiments):
Have you tried an ablation study on the number of critics? What patterns would you expect?

Author response:
> Have you tried an ablation study on the number of critics? What patterns would you expect?

Increasing the critic ensemble size in DAC is an interesting research theme. Approaches like REDQ take a subset of the ensemble for every TD calculation. We think that such design choice would have to be ablated in the context of DAC. To cut to the chase, we believe that ensemble-based DAC is an interesting direction due to the synergies specific to DAC which are absent in standard SAC (ie. learning based on the ensemble statistics). Since enlarging the critic ensemble would both imply substantially higher compute requirements and distort the evaluation against baselines we decided to leave it for future work.

Reviewer comment (Theory):
The proposed algorithm is mostly designed by heuristics and the implementation details are not theoretically justified. Although theoretical support is not considered necessary for a good paper, empirical evidence is expected to verify the critical claims and algorithmic designs of this paper.

Author response:
We would like to note that DAC follows fundamental concepts in RL theory, such as the actor-critic framework (stemming from the Policy Gradient Theorem), optimism in the face of uncertainty and off-policy learning. Its design incorporates widely researched principles of policy and value iteration in the context of maximum entropy formulation, which all are foundational for the field of RL. Even the automatic optimization of optimism and KL regularization can easily be linked to dual optimization [3], used in similar contexts in many other RL algorithms [4, 5]. However, we realize that our manuscript focuses on evaluating implementation of theoretically founded principles in practice, and as such could be considered an empirical work. 

>  I expect empirical evidence to verify the critical claims/algorithmic designs of this paper 

Figures 4 & 5 extensively evaluate the performance impact of many design decisions (10 DAC simplifications & 14 hyperparameter configurations). We note that 7/10 DAC simplifications and 14/14 hyperparameter configurations perform better than baseline SAC.

Reviewer comment (Theory):
Since the proposed algorithm is essentially a synthesis of different intuitive designs, a discussion of where the algorithm would converge to should be provided. The paper currently introduces different updating rules without presenting a clear objective function (possibly with constraints) for Algorithm 1, which would allow readers to easily see what it is optimizing. The authors are asked to write this down in the rebuttal.

Author response:
Furthermore, since the proposed algorithm is basically a synthesis of different intuitive designs, a discussion of where the algorithm would converge to should be provided.

In Section 3.2 we write: “This mechanism allows for separate entropy for TD learning and exploration while retaining standard convergence guarantees of AC algorithms. In fact, (...) it follows that in the limit both actors recover a policy that differs only by the level of entropy”. Furthermore, since SAC is an off-policy algorithm and DAC differs only by samples that land in the experience buffer, we believe it trivially follows that DAC retains convergence guarantees of SAC. 

> Currently the paper is written in a way that different updating rules are introduced; I expect to see a clear objective function (maybe with constraints) of Algorithm 1, so readers can easily see what it is optimizing, can the authors write it down in the rebuttal?

The current version of the paper features a pseudo-code with links to every equation used in the calculation. Furthermore, Sections 3 & A2 detail what and why DAC is optimizing. If the Reviewer points us to what particularly the Reviewer finds confusing, we are happy to adjust the text.

Reviewer comment (Theory):
When simultaneously maximizing both lower and upper bound of the Q values, there is a concern that it might squash all action values higher and still result in overestimation.

Author response:
We think that there is a misunderstanding. Overestimation is a property of the critic and is believed to stem from TD learning [2,5,6]. DAC features no “simultaneous maximization of lower and upper bound of the Q-values” and the critic used in DAC learns using a traditional clipped-double Q-learning [4,6]. The lower and upper bounds policies are optimized by conservative and optimistic actors respectively (not ‘simultaneously’), and the optimistic actor does not interact with TD learning in DAC design. As such, there is no reason for overestimation in DAC beyond overestimation that occurs in SAC. This is underlined in Section 1 of our manuscript: “DAC employs two actors, each independently optimized using gradient backpropagation with different objectives. The optimistic actor is trained to maximize an optimistic Q-value upper-bound while adjusting optimism levels automatically. This actor is responsible for exploration (sampling transitions added to the experience buffer). In contrast, the conservative actor is trained using standard lower-bound soft policy learning and is used for sampling temporal-difference (TD) targets and evaluation.”

Reviewer comment (Experiments):
The added optimism in the proposed method is essentially an exploration strategy, so some baselines that aim at exploration should also be compared.

Author response:
> The added optimism is essentially an exploration strategy, some baselines aiming at exploration should be also compared. 

We evaluate against OAC and TOP (and SR versions thereof) which are baselines aiming at exploration via optimism. Additionally, we evaluate 10 simplifications of DAC which also feature optimism-driven exploration. Does the Reviewer have any specific optimism-based model-free continuous control baselines in mind?

Reviewer comment (Experiments):
An ablation study should be provided to justify the following design choices: the effect of ensemble (and whether those baselines also use ensembles), and the necessity of optimizing optimism and KL weight (could some intuitive choices be used instead of learning them).

Author response:
> Ablation study should be provided to justify the following design choices: the effect of ensemble (and do those baselines use ensemble too?)

Using a critic ensemble has been standard in modern RL since TD3/SAC (which introduced clipped double Q-learning based on an ensemble of 2 critics) [1,2,5]. In Section 4 we write: “all algorithms except RedQ use the same network architectures and a standard ensemble of two critics”. As such, we believe that evaluating the effect of ensemble is outside of scope of this paper. If the Reviewer is curious about the effect of even bigger ensemble in continuous-control RL we kindly point to the following works [7,8]

> the necessity of optimizing optimism and KL weight (can you use some intuitive choices instead of learning them)?

Figure 4 of the original manuscript presents evaluation of various DAC design decisions, including not optimizing optimism and KL weight, optimizing only one of them and optimizing both. Section 3 discusses why it is hard to find “some intuitive choices instead of learning them” and Figure 3 shows an experiment backing that claim.

Reviewer comment (Reproducibility):
Clarification is requested on how the order of the updating rules 10-15 is decided and how their learning rates are chosen.

Author response:
> Any comments how do you decide the order of the updating rules 10-15? 

We added our additional modules (optimistic actor, optimism and KL) at the end of the update loop (after standard SAC updates). The optimistic actor update is dependent on optimism and KL. As such, we decided to update it ahead of the dual optimization modules. The order of optimism and KL is irrelevant as these modules are independent.

> And how their learning rates are chosen? 

For the actor-critic modules, we use the same learning rates as the baselines proposed in earlier works. That includes the optimistic actor. For the optimism and KL modules, we took the same learning rate but divided it by 10 arbitrarily. We did not investigate the effects of varying learning rates.

Reviewer comment (Experiments):
The algorithm introduces many more hyperparameters compared to commonly seen SAC or TD3 due to the added components in the losses. The comparison to other baselines is not considered fair unless evidence is provided that similar efforts have been made to thoroughly sweep the baseline’s hyperparameters.

Author response:
We would like to respond to the above assertion in three points:

1. SAC/TD3 are arguably the most popular RL algorithms at the time of writing this response. Both were evaluated numerous times on DMC locomotion, which is one of the most popular RL continuous-control benchmarks. As such, we believe that the hyperparameter settings used for SAC/TD3 were arguably more “sweeped” than those of DAC. The hyperparameter choices used for our implementations of SAC and SR-SAC are used in the current DMC locomotion SOTA (SR-SAC) [9].

2. We kindly ask the Reviewer to note that reusing hyperparameters proposed in the earlier works is a common practice, especially in the case of such well-known benchmarks as DMC/gym [1,2,4,5,6,7,8,9,10,11].

3. Finally, we would like to point towards experiments presented in Figure 5 of our original manuscript. There, we evaluate 14 hyperparameter configurations of DAC (our entire sweep), as well as the reference SAC performance. According to the results, all 14/14 DAC configurations significantly outperform SAC. 

Considering the above arguments, we kindly ask the Reviewer to reconsider his “unfair comparison” assertion.

Reviewer comment (Theory):
Regarding Eq. 7, clarification is needed on how the gradient with respect to optimistic actor parameters is calculated. It appears that the gradient should also be propagated through/to the first two Q functions, and these Qs are interdependent with the actor.

Author response:
> Eq 7, how do you calculate the gradient w.r.t optimistic actor parameters, it appears the gradient should be also propagated through/to the first two Q functions and these Qs are interdependent with actor. 

The Eq 7 is implemented the same way a standard backprob through critic is implemented in DDPG, D4PG, SAC, TD3, OAC, TOP etc. It is not required to update the critics parameters as a result of this update, we can just use the gradient calculation to propagate it further to the actor. If the Reviewer wishes, we can further expand on how it is usually implemented.

Reviewer comment (Novelty):
The paper’s motivation is to avoid overestimation while maintaining good exploration. It seems intuitive to combine some exploration method with methods that mitigate overestimation, and designing such a baseline should not be difficult, as mitigating overestimation typically requires multiple critics and uncertainty estimates could be derived from the ensemble for exploration purposes. The authors should clarify what the proposed algorithm’s advantage is.

Author response:
> The motivation of the paper is to avoid overestimation while keep good exploration. Isn’t it quite intuitive to combine some exploration method with methods that mitigate overestimation? it shouldn’t be difficult to design such a baseline as mitigating overestimation typically require multiple critics and uncertainty estimate could be derived from the ensemble for exploration purpose. What is the proposed algorithm’s advantage? 

We evaluate against OAC, TOP, SAC, REDQ and TD3 which all “combine some exploration method with methods that mitigate overestimation” and all use multiple critics to derive an uncertainty estimate “for exploration purpose”. Furthermore, in Figure 4 we consider 10 DAC simplifications, which all as well fit the Reviewer’s description. We discuss the advantage of our approach in Section 3. 

> it shouldn’t be difficult to design such a baseline

We design and evaluate a variety of baselines common to DAC. The results are presented in Figures 4 & 6. We are happy to add more baseline designs to our evaluations, if the Reviewer shares more details about the Reviewer’s envisioned baseline.

Reviewer comment (Experiments):
The experimental design of studying replay ratio appears to be disconnected from the primary motivation of the paper and should be placed in the appendix.

Author response:
> The experimental design of studying replay ratio appears to disconnect with the primary motivation of this paper; this should be put in the appendix.

We argument why we study two replay ratio regimes in Section 1 & 4. In our view it is an important aspect in the light of modern RL studies, which was repeatedly shown to play a substantial role in the performance of RL algorithms. Also, as Reviewer XQ6P wanted us to expand the discussion of replay ratios, we would like to keep the analysis in the main body of the paper.