  We introduce ES-ENAS, a simple neural architecture search (NAS) algorithm for
the purpose of reinforcement learning (RL) policy design, by combining
Evolutionary Strategies (ES) and Efficient NAS (ENAS) in a highly scalable and
intuitive way. Our main insight is noticing that ES is already a distributed
blackbox algorithm, and thus we may simply insert a model controller from ENAS
into the central aggregator in ES and obtain weight sharing properties for
free. By doing so, we bridge the gap from NAS research in supervised learning
settings to the reinforcement learning scenario through this relatively simple
marriage between two different lines of research, and are one of the first to
apply controller-based NAS techniques to RL. We demonstrate the utility of our
method by training combinatorial neural network architectures for RL problems
in continuous control, via edge pruning and weight sharing. We also incorporate
a wide variety of popular techniques from modern NAS literature, including
multiobjective optimization and varying controller methods, to showcase their
promise in the RL field and discuss possible extensions. We achieve >90%
network compression for multiple tasks, which may be special interest in mobile
robotics with limited storage and computational resources.
