Paper: Rethinking Actor-Critic: Successive Actors for Critic Maximization (ICLR-2024)

Reviewer comment (Novelty):
The key parts of the successive actor-critic modules adopt both ‘deep-set’ and ‘FiLM’ methods, but lack a description of potential advantages and an explanation of alternative methods.

Author response:
In Appendix C.3 and C.4, we added the descriptions of Deepset along with other design choices of list-summarizers as well as Feature-wise Linear Modulation (FiLM). In short, Deepset effectively aggregates the information in the list input while FiLM helps to condition on preceding list-input. Those modules contribute to better training the list of actor-critics in SAVO.
- We also added ablations in Appendix.D.2 & D.3 to compare (i) FiLM v/s no-FiLM and (ii) the choice of DeepSet v/s LSTM v/s Transformers to aggregate the information from previous selected actions. The results validated our choice of DeepSet as it is more scalable to an increasing number of actor-critic pairs. Also, we acknowledge that this design choice depends on the environment and the use-case as some tasks in continuous domains show that LSTM / Transformers outperform Deepset.

Reviewer comment (Presentation):
Legend should be added to FIG. 7, although it is related to FIG. 3.

Author response:
We have fixed all the suggested typos, missing legends, and source code naming in our revision. Thank you for your detailed suggestions.

Reviewer comment (Reproducibility):
Although there is not enough time to go through the source code carefully, it is recommended that the method abbreviation be consistent in the code(called FLAIR) and the paper(called SAVO).

Author response:
We have fixed all the suggested typos, missing legends, and source code naming in our revision. Thank you for your detailed suggestions.

Reviewer comment (Theory):
While the idea of using a series of successive actor-critic pairs to gradually refine/restrict the action space is interesting, this also means that the action selection decision from the policy network may be highly sensitive to the minor nuances of the learned critic. This often introduces bias to the learning process, resulting in degraded learning stability and restricted exploration. The downside of using multiple successive actor-critic pairs should be extensively examined in this paper, with a solid theoretical foundation explaining why successively restricting the action space based on the trained critic will not affect the learning stability. It is also important to explain why this approach helps to improve the effectiveness of exploration, rather than restricting exploration, as claimed by the authors.

Author response:
`Action selection decision`
This is true for all actor-critic methods, especially those learned with deep learning. However, the key goal of any RL agent is to take actions that maximize its expected return in the environment, which is why actor-critic algorithms learn actors that can find actions to maximize the Q-values. This does not create any *bias* in learning of the actors, because their true objective is to maximize the critic. While the Q-function itself can be biased during deep RL training, it is still the best estimate of the objective for the actor. Our method does not degrade the primary critic’s function which is already based on stabilizing tricks adopted from TD3, like twin critics, delayed updates, and soft-action evaluation. Thus, stability of training of the primary critic is not negatively affected at all by our method. And the actor learned by our method is guaranteed to be better than a single actor, because of the max operator in Line 13 of Algorithm 1, which ensures the action is always at least as good as the one suggested by a single actor.
Please also refer to combined response for more details.

`Exploration`
- Exploration for each successive actor $\pi_i$ still follows the standard OU Noise based exploration from DDPG, TD3. The successive actors provide an action that has a higher Q-value than a single actor would have provided, but the exploration term is still added over the actions while acting in the environment. We have modified the Algorithm 1 to clarify this important detail.
- Furthermore, the sample efficiency improvements across 14 tasks (Fig.4 using RLiable as AC suggested) empirically justifies that our method never hurts exploration in the environment.

Reviewer comment (Novelty):
Since the newly proposed algorithm uses an ensemble of actor-critic pairs, it is related to ensemble actor-critic algorithms. In section 2, it seems necessary for the authors to review existing ensemble actor-critic methods and clearly highlight the key novelty of the new algorithm compared to existing ensemble algorithms. Furthermore, the experimental study should include more state-of-the-art ensemble baselines to clearly show the advantages of the new algorithm over existing ensemble algorithms.

Author response:
### New Ensemble Baseline
Novelty over ensemble methods: Ours is not a typical ensemble method, where usually the constituents of the ensemble are symmetric. In fact, ours is a sequential ensemble, where successive actor-critic pairs optimize different optimization problems, building up on the past actor-critic pairs.
Our added visual result in `Fig.3` shows how this successive optimization eases the Q-function optimization landscape, which indeed helps the successive actors to find more globally optimal actions than before, with respect to the current Q-function

Reviewer comment (Writing):
Some parts of the new algorithm design lack technical clarity. In particular, it is not clear how deep-set is used to produce Z as a concatenation of previously selected actions and state. It is also unclear how FiLM is used to enable a policy to choose its actions conditional on Z. The motivations and rationales for using deep-set and FiLM should be clearly explained and strongly justified.

Author response:
### Technical explanation and analysis of FiLM, DeepSet
- In Appendix C.3 and C.4, we added the descriptions of Deepset along with other design choices of list-summarizers as well as Feature-wise Linear Modulation (FiLM). In short, Deepset effectively aggregates the information in the list input while FiLM helps to condition on preceding list-input. Those modules contribute to better training the list of actor-critics in SAVO.
- We also added ablations in Appendix.D.2 & D.3 to compare (i) FiLM v/s no-FiLM and (ii) the choice of DeepSet v/s LSTM v/s Transformers to aggregate the information from previous selected actions. The results validated our choice of DeepSet as it is more scalable to an increasing number of actor-critic pairs. Also, we acknowledge that this design choice depends on the environment and the use-case as some tasks in continuous domains show that LSTM / Transformers outperform Deepset.

Reviewer comment (Experiments):
Although the authors stated that their new ensemble technique can be applied to many different actor-critic algorithms, they focus primarily on applying the technique to TD3. To demonstrate the wide applicability of the new technique, the authors should study its possible application to other algorithms, such as SAC.

Author response:
`Baseline Selection of TD3`
We choose TD3 because of its unified applicability to both action-representation-based discrete action spaces and continuous action space. Particularly, the Wolpertinger algorithm [Dulac et al 2015] is based on the DDPG algorithm and is the most commonly used base algorithm for large discrete action spaces. While we agree with the reviewer’s suggestion that combining SAVO with SAC would be good to show wider applicability of our method, we think this extension might offer more flexibility in the design choices. Thus, for this project, we decided to stick to the simple architecture of TD3 as described above.

Reviewer comment (Theory):
Some mathematical formulas in the paper are not easy to understand. For example, it is unclear how to find the optimal action a' based on the primary critic, as part of the final training objective, which is further conditional on $\Pi$. The formula for the policy gradient in the final training objective also misses some brackets such as }. It remains questionable why the policy gradient formula is valid—i.e., what kind of gradient is being calculated and why the gradient allows the trained policy network to maximize the expected return. A more detailed and thorough theoretical analysis is necessary to justify the validity and effectiveness of the proposed training objective.

Author response:
`How to find the optimal action a' based on the primary critic`
We improved the clarity of writing in the approach section and added the following qualitative result to support our claim;
``[New Figure.3]`` The optimization landscape of Q-value concretely represents the landscape where the N-D action space represents the independent variable and the 1-D Q-value represents the dependent variable. For visualization, we map the action space to 2-D when N > 1. Fig.3 demonstrates that the Q-function becomes more easy to globally optmize with every iteration of our successive actor-critics. This is because the locally optimal peaks below past actions’ Q-values are pruned away from the optimization landscape. As we can observe:
The number of local optima are reduced in $Q_0 \to Q_1 \to Q_2$, as most action values are shifted up in the Q-value space. This enables successive actors $\pi_0 \to \pi_1 \to \pi_2$ to focus on more optimal regions of the true Q-value space.
The actions $a_1$ and $a_2$ found by $\pi_1$ and $\pi_2$ are indeed more optimal than $\pi_0$ whose actions were only locally optimal peaks, but globally suboptimal.

Reviewer comment (Theory):
In Line 13 of Algorithm 1, the method is based on the estimated Q-function. Due to estimation noise/error, it is not guaranteed that the new action selection method is definitely better than using a single actor. The intuitive discussion (including illustrative figures) about the stability issue or the direct advantage of identifying the best possible action is not sufficiently convincing. A more thorough theoretical analysis is needed to justify these major claims.

Author response:
`Action selection decision`
This is true for all actor-critic methods, especially those learned with deep learning. However, the key goal of any RL agent is to take actions that maximize its expected return in the environment, which is why actor-critic algorithms learn actors that can find actions to maximize the Q-values. This does not create any *bias* in learning of the actors, because their true objective is to maximize the critic. While the Q-function itself can be biased during deep RL training, it is still the best estimate of the objective for the actor. Our method does not degrade the primary critic’s function which is already based on stabilizing tricks adopted from TD3, like twin critics, delayed updates, and soft-action evaluation. Thus, stability of training of the primary critic is not negatively affected at all by our method. And the actor learned by our method is guaranteed to be better than a single actor, because of the max operator in Line 13 of Algorithm 1, which ensures the action is always at least as good as the one suggested by a single actor.
Please also refer to combined response for more details.

Reviewer comment (Theory):
It is unclear how the policy gradient should be calculated for the new form of actor-critics. Specifically, does Q_i in the new form satisfy any Bellman equation and why? Why can the effective state space be expanded with actions chosen by other actors? What is the new definition of policy gradient on the basis of the expanded state space, especially when the action component of the state space is affected by the learning process of other actors (and thus the expanded state space may no longer satisfy the Markov property)? More detailed mathematical derivation could help clarify these concerns.

Author response:
### Validity of policy gradient objective
Each successive actor-critic pair solves its own deterministic policy gradient objective, where only the effective state space is modified. Conventional actor-critics take the form $\pi(s, a)$ and $Q(s, a)$. Whereas, each of our successive actor-critics take the form $\pi_i(\{s, a_{0:i-1}\}, a)$ and $Q_i(\{s, a_{0:i-1}\}, a, a)$. Thus, only the effective state space of the MDP is modified, and we can directly apply any actor-critic algorithm to train these actor-critics (we use TD3).

Reviewer comment (Theory):
There is an open question as to why the new policy gradient formula is valid—specifically, what kind of gradient is being calculated and why the gradient allows the trained policy network to maximize the expected return.

Author response:
### Validity of policy gradient objective
Each successive actor-critic pair solves its own deterministic policy gradient objective, where only the effective state space is modified. Conventional actor-critics take the form $\pi(s, a)$ and $Q(s, a)$. Whereas, each of our successive actor-critics take the form $\pi_i(\{s, a_{0:i-1}\}, a)$ and $Q_i(\{s, a_{0:i-1}\}, a, a)$. Thus, only the effective state space of the MDP is modified, and we can directly apply any actor-critic algorithm to train these actor-critics (we use TD3).

Reviewer comment (Writing):
The clarity of certain statements remains in question, as referred to in the previous section on the clarity of expressions.

Author response:
`Language clarifications`
As per the reviewer’s suggestion, we have improved the language-ambiguous stataments:
"navigate the action-value landscape more proficiently" → “globally optimize the Q-value landscape”
An actor is trained with gradient ascent over the Q-value landscape and can get stuck in local optima, which we resolve.
"distribute the optimization load over to the critic" → “utilize the critic to evaluate multiple actions”.
"This can make the learning procedure even slower than the original inefficiencies of the actor" → “A larger action space in the joint-actor-critic model present another optimization challenge, making this solution infeasible”.

Reviewer comment (Experiments):
The experimental results are not sufficiently reliable. The baselines are outdated. There is no mention of hyperparameter sensitivity or setting experiments.

Author response:
## Added more experimental benchmarks  
> “The experimental results are not sufficiently reliable.” "The experiment in the Appendix only has the Easy environments, what about the other Hard environments?”  

`[Figure 2]` We added more continuous control environments in the paper, and report results across 11 environments.  
`[Figure 4(c)]` We also report the results using the RLiable library (also suggested by AC) to demonstrate that our method outperforms the baselines and ablations reliably across the environments.  

## Clarifications  
> “There is no mention of hyperparameter sensitivity or setting experiments.”  
- Appendix E.2 and E.3 already provides a discussion of hyperparameter sensitivity and setting details across all baselines.  
- To reiterate, the sensitive hyperparameters were found to be learning rates and network sizes of actors and critics, and were searched for each baseline.  
- `[Figure 4 (c)]` A key hyperparameter of our approach is $K$, the number of actor-critic pairs. Figure.4 shows the improvement in performance as $K$ increases in continuous control tasks. However, we chose $K$=3 across environments as that is enough to show significant gain with our method.  
- `[Appendix D2, D3]` We also add experiments to show different design choices are viable: FiLM v/s No-FiLM for conditioning, and DeepSet v/s LSTM v/s Transformer for past action summarization.

Reviewer comment (Novelty):
In Relate Work, the introduction of prior work is outdated. Please supplement it with the latest relevant work.

Author response:
## Supplemented discussion in Related Work  
`[Section 2]` Added discussion on evolutionary methods and ensemble methods, and also restructured the related work section significantly.

Reviewer comment (Experiments):
The work presented in this paper seems to fall under the domain of ensemble methods. It may be necessary to supplement it with relevant work and introduce ensemble-based value optimization algorithms as additional baselines.

Author response:
## Supplemented discussion in Related Work  
`[Section 2]` Added discussion on evolutionary methods and ensemble methods, and also restructured the related work section significantly.

Reviewer comment (Writing):
In Algorithm 1, the "state s" in lines 8 and 10 need clarification.

Author response:
## Clarifications  
> “In Algorithm 1, the "state s" in lines 8 and 10 need clarification.”  
- We correct the typo in lines 8-10, as $s$ → $s_t$. $s_t$ is the state observed in Line 5.

Reviewer comment (Theory):
Does the proposed method in this paper suffer from the problem of action values overestimation?

Author response:
## Clarifications  
> "Does the proposed method in this paper suffer from the problem of action values overestimation?”  
- Since our algorithm is based on TD3, the problem of overestimation is reduced by the presence of twin critics and delayed updates. We clarify this in Algorithm 1 to alleviate the concern of overstimation. To further justify, in the Q-learning update of primary critic in Line 19 of Algorithm 1, our method only modifies the search of the Q-value maximizing action. This is a key requirement of the DPG algorithm and the training objective of DPG, DDPG, TD3, etc. Works like DDPG and TD3 alleviate the issues of overestimation of Q-functions, and our implementation being based on TD3, also benefits from those solutions. All the baselines and ablations also equivalently benefit from this, and do not suffer from overestimation any more than TD3.

Reviewer comment (Experiments):
Why only select 3 seeds in some experiments, such as Figure 4?

Author response:
## Clarifications  
> “Why only select 3 seeds in some experiments, such as Figure 4?”  
- All analysis results have been updated with 5 seeds. No trend is affected.

Reviewer comment (Experiments):
Is the modification of the experimental setup fair? Other baselines may not have been designed specifically to address the problem presented in this paper.

Author response:
## Clarifications  
> “Is the modification of the experimental setup fair? Other baselines may not have been designed specifically to address the problem presented in this paper.  
- To exemplify the problem of challenging Q-function landscapes, we curate hard benchmarks of continuous control tasks, but we also reported results on the original easy benchmarks with relatively simple optimization landscapes, in Figure 2. SAVO outperforms baselines in both the easy and hard versions of the tasks, but more significantly in the hard versions. This shows that our algorithm’s contribution is more robust to challenging optimization landscapes, that would be more common as RL scales up to harder problems. This is already evident in our results in the discrete action space environments, where the optimization landscape is already challenging because of the presence of only few valid points in the action representation space, that correspond to actual discrete actions.  

- The baselines from various prior works (Figure 2) are indeed incapable of addressing the problems of Q-function optimization presented in this paper. That is the key claim that we are making. All the implementations of the baselines are fair and based on the same code skeleton (code is attached in supplementary), and we clearly demonstrate that it is because of the better optimization of the Q-function by SAVO’s successive actors than baseline actors like single actors, Sampling-augmented actors, CEM Actors, and Ensemble of Actors.

Reviewer comment (Evaluation):
Due to the utilization of ensemble methods, I am concerned about the computational efficiency of the algorithm. Please supplement the experiments or provide an analysis.

Author response:
## Computational Efficiency  
“I am concerned about the computational efficiency of the algorithm. Please supplement the experiments or provide an analysis.”  
`[Figure 4 (d)]` Thank you for this suggestion. We have added a complexity v/s performance analysis that shows the computational requirement increase is negligible. Please refer to the combined response.

Reviewer comment (Experiments):
The actual benefit of the proposed algorithm is unclear — it is not clear whether the improvement comes from finding the maximum action, from ensemble, or from exploration.

Author response:
### Ensemble actors  
[`Figure 2`, `Section 2.3`] As per your suggestion, we implement an ensemble-actors approach where we train multiple actors with the same critic, but explore using the critic maximizing action found by the actors. This approach helps disentangle SAVO’s ability to prune the optimization landscape from the improved exploration due to the presence of multiple actors. Please refer to combined response for further details, and Section 2.3 for a discussion and references of past work.

Reviewer comment (Novelty):
Highly related works are missing.

Author response:
We appreciate the reviewer’s suggestions and pointing out the relevant work and baselines from prior work. We have added a discussion and comparison to all the suggested baselines:

Reviewer comment (Theory):
The connection between finding the maximum action and improved sample efficiency is not supported; please justify this connection and address whether it would introduce overestimation that could hurt learning.

Author response:
> “The connection between finding the maximum action and improved sample efficiency is not supported”  
- We bring attention to Figure 4(a) and Figure(b) where we already show that SAVO results in an improved Q-value of its actions, which translates into a sample efficiency improvement when evaluating SAVO with all its actors v/s only a single actor.  

> “Would it introduce overestimation?”  
- No. SAVO only improves the actor’s capability to find the Q-value maximizing action, which is the key goal of the base algorithm of DPG / DDPG / TD3.  
- There is still exploration on top of the action found by SAVO, so the exploration is not hindered.

Reviewer comment (Experiments):
An ensemble-based exploration method should be added for comparison, as ensemble methods are known to enhance sample efficiency. Adding such a comparison would also help verify whether the main benefit comes from finding the maximum action or from exploration; if it is the latter, then the pitch of the paper should be modified and another set of baselines aiming at better exploration should be compared.

Author response:
### Ensemble actors  
[`Figure 2`, `Section 2.3`] As per your suggestion, we implement an ensemble-actors approach where we train multiple actors with the same critic, but explore using the critic maximizing action found by the actors. This approach helps disentangle SAVO’s ability to prune the optimization landscape from the improved exploration due to the presence of multiple actors. Please refer to combined response for further details, and Section 2.3 for a discussion and references of past work.  

### Evolutionary Actors  
[`Figure 2`, `Section 2.4`] As per your suggestion, we add:  
- **CEM-actor baseline**: Algorithms like QT-Opt, CGP, CEM-RL, and GRAC employ CEM as the actor in online RL training. We implement QT-Opt style CEM, where the critic is still trained with our TD3-augmented tricks. These algorithms involving CEM require infeasible amounts of repetitive evaluations of the Q-function and do not scale well to high-dimensional action spaces (Yan et al., 2019). Note that CGP, CEM-RL and GRAC can be considered as suboptimal versions of a good CEM actor in the QT-Opt, because they already use CEM as a guide to train their actors. Thus, we only implement and compare against the QT-Opt style CEM baseline.  
- **Greedy-actor** critic baseline: emulates CEM by sampling from a high-entropy action proposal policy, evaluating these actions with the Q-function, and training the actor to greedily follow the best actions. However, since this approach also depends on gradient ascent and samples actions around the mean actions of a single-actor, thus limiting its ability to find the globally optimal action.  
- **Greedy-TD3**: We implement another version of Greedy-actor-critic in TD3 style training, where the critic now benefits from TD3 update tricks, and the exploration is performed with OUNoise added to the mean action of the stochastic greedy policy.

Reviewer comment (Experiments):
In the algorithm, it seems that at each time step, both policy and critic parameters are updated K times; do the authors do the same for the baselines?

Author response:
> “It seems at each time step, the algorithm update both policy and critic parameters K times?”  
- No. We clarify that, as shown in Lines 16 and 17 of Algorithm 1, each individual $Q_i$ and $\pi_i$ is only updated once every iteration. $K$ denotes the number of actor-critic pairs, as written in the first line of Algorithm 1. So, the for loop in line 15 is to denote all the actor-critic pairs. To further clarify, all the baselines and ablations are based on the same algorithm design and implemented in the same code file, for fairness.

Reviewer comment (Experiments):
Baselines suggested by the missing related works should be added.

Author response:
We appreciate the reviewer’s suggestions and pointing out the relevant work and baselines from prior work. We have added a discussion and comparison to all the suggested baselines:

Reviewer comment (Novelty):
Several highly relevant references are missing, including:  
1. A model reference adaptive search method for global optimization by Jiaqiao et al.  
2. Q-learning for continuous actions with cross-entropy guided policies by Riley et al.  
3. Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvement by Samuel et al.  
4. CEM-RL: Combining evolutionary and gradient-based methods for policy search by Alois et al.  
5. Wire fitting algorithm by Baird et al. (likely titled RL with high-dimensional continuous actions).

Author response:
We appreciate the reviewer’s suggestions and pointing out the relevant work and baselines from prior work. We have added a discussion and comparison to all the suggested baselines:

Reviewer comment (Novelty):
Among the missing works, references 2, 3, and 4 are highly relevant and should be compared. The authors should explain the differences between their work and these existing methods and comment on the significance of such differences, as this is considered one of the critical weaknesses of the work.

Author response:
### Evolutionary Actors  
[`Figure 2`, `Section 2.4`] As per your suggestion, we add:  
- **CEM-actor baseline**: Algorithms like QT-Opt, CGP, CEM-RL, and GRAC employ CEM as the actor in online RL training. We implement QT-Opt style CEM, where the critic is still trained with our TD3-augmented tricks. These algorithms involving CEM require infeasible amounts of repetitive evaluations of the Q-function and do not scale well to high-dimensional action spaces (Yan et al., 2019). Note that CGP, CEM-RL and GRAC can be considered as suboptimal versions of a good CEM actor in the QT-Opt, because they already use CEM as a guide to train their actors. Thus, we only implement and compare against the QT-Opt style CEM baseline.  
- **Greedy-actor** critic baseline: emulates CEM by sampling from a high-entropy action proposal policy, evaluating these actions with the Q-function, and training the actor to greedily follow the best actions. However, since this approach also depends on gradient ascent and samples actions around the mean actions of a single-actor, thus limiting its ability to find the globally optimal action.  
- **Greedy-TD3**: We implement another version of Greedy-actor-critic in TD3 style training, where the critic now benefits from TD3 update tricks, and the exploration is performed with OUNoise added to the mean action of the stochastic greedy policy.