Paper:
CEM-RL: COMBINING EVOLUTIONARY AND GRADIENT-BASED METHODS FOR POLICY SEARCH
Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search. The former is widely applicable and rather stable, but suffers from low sample efficiency. By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstable and highly sensitive to hyper-parameter setting. So far, these families of methods have mostly been compared as competing tools. However, an emerging approach consists in combining them so as to get the best of both worlds. Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL algorithm. In this paper, we propose a different combination scheme using the simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy gradient (TD3), another off-policy deep RL algorithm which improves over DDPG. We evaluate the resulting method, CEM-RL, on a set of benchmarks classically used in deep RL. We show that CEM-RL benefits from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efficiency.

Review:
Gradient-free evolutionary search methods for Reinforcement Learning are typically very stable, but scale poorly with the number of parameters when optimizing highly-parametrized policies (e.g. neural networks). Meanwhile, gradient-based deep RL methods, such as DDPG are often sample efficient, particularly in the off-policy setting when, unlike evolutionary search methods, they can continue to use previous experience to estimate values. However, these approaches can also be unstable.
This work combines the well-known CEM search with TD3 (an improved variant of DDPG). The key idea of of this work is in each generation of CEM, 1/2 the individuals are improved using TD3 (i.e. the RL gradient). This method is made more practical by using a replay buffer so experience from previous generations is used for the TD3 updates and importance sampling is used to improve the efficiency of CEM.
This work shows, on some simple control tasks, that this method appears to result in much stronger performance compared with CEM, and small improvements over TD3 alone. It also typically out-performs ERL.
Intuitively, it seems like it may be possible to construct counter-examples where the gradient updates will prevent convergence. Issues of convergence seem like they deserve some discussion here and potentially could be examined empirically (is CEM-TD3 converging in the swimmer?).
The justification that the method of Khadka & Tumer (2018) cannot be extended to use CEM, since the RL policies do not comply with the covariance matrix is unclear to me. Algorithm 1, step 20, the covariance matrix is updated after the RL step so regardless of how the RL policies are generated, the search distribution on the next distribution includes them. In both this work, and Khadka & Tumer, the RL updates lead to policies that differ from the search distribution (indeed that is the point), and there is no guarantee in this work that the TD3 updates result in policies close to the starting point. It sees like the more important distinction is that, in this approach, the information flows both from ES to RL and vice-versa, rather than just from RL to ES.
One view of this method would be that it is an ensemble method for learning the policy [e.g. similar to Osband et al., 2016 for DQN]. This could be discussed and a relevant control would be to keep a population (ensemble) of policies, but only update using RL while sharing experience across all actors. This would isolate the ensemble effect from the evolutionary search.
Minor issues:
- The ReLU non-linearity in DDPG and TD3 prior work is replaced with tanh. This change is noted, but it would be useful to make a least a brief (i.e. one sentence) comment on the motivation for this change.
- The paper is over the hard page limit for ICLR so needs to be edit to reduce the length.
Osband I, Blundell C, Pritzel A, Van Roy B. Deep exploration via bootstrapped DQN. InAdvances in neural information processing systems 2016 (pp. 4026-4034).