Paper:
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS (Vinyals et al., 2015;Pham et al., 2018;Zoph & Le, 2017) and ES (Salimans et al., 2017) in a highly scalable and intuitive way. By defining the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efficient learned edge-partitionings. For several RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as 17 weight parameters, providing > 90% compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices (Choromanski et al., 2018), while still maintaining good reward. We believe that our work is one of the first attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics (Gage, 2002) with limited storage and computational resources.

Review:
Summary: This paper focuses on neural architecture search for constructing compact RL policies. It combines ideas from the popular ENAS and ES methods for optimisation. Recent work defined a family of compact policies by imposing a fixed Toeplitz structure. This paper introduces the so-called “chromatic network” architecture, which partitions weights of the RL network into tied sub-groups. The partitioning is searched with ENAS and shared weights are updated via ES. Experiments on continuous control benchmarks show that good performance can be obtained using a very small number of parameters. Favourable reward-compression outcomes can be achieved compared to some baseline alternatives.
To my understanding the main contributions are (1) the Chromatic parameter sharing scheme and (2) the combined ENAS+ES learning procedure.  
Pros:
+ Chromatic networks provide a neat idea for managing compact networks via parameter sharing. (Although not dramatically novel given cited work by Salimans, Gaier, etc)
+ Searching both the partitioning + weight values with ENAS and ES provides a nice way to do learning & compact architecture search simultaneously.
+ Results generally show favourable compression/reward performance vs Mask, Toeplitz  and Circulant baselines. 
Cons & Questions: 
0. Motivation. The paper claims that RL can be high-dimensional with millions of parameters, and so there is a need to apply NAS to RL. But experiments are conducted in low dimensional environments with only 100s of parameters. The empirical validation doesn’t match the motivating scenario. We would want to see results on successful compression of larger vision-based networks to be fully persuaded. Without this it undermines the significance of the paper. The paper makes a claim about embedded devices, but this is unconvincing, as for these proprioceptive control tasks, the uncompressed networks are already small enough to run on most embedded devices. 
 1. ENAS vs Chromatic. Both use RL controllers. The main difference to ENAS seems to be the Chromatic sharing scheme, and use of ES rather than Backprop to update the weights. Some points of motivation/justification are not very clear after reading: (1) Why vanilla ENAS can’t be used for RL? (with the modification of replacing standard backdrop with any standard continuous control RL algorithm for weight updates). It is sort of claimed that ENAS can’t be used, it’s not obvious to me that this is true. (2) How is the  Chromatic strategy specific for RL? In principle it should also be usable in SL. Assessing its performance in SL would be more broadly relevant, interesting and significant since there is more prior compression work in SL.
2. The paper is a bit vague about how number of partitions is set. Is it a user-specifiable parameter, if so how do we set it? It’s suggested that partition number is included in the reward function, but then it’s even more unclear how to calibrate this to hit a particular performance or size target (or optimize performance for fixed size, size for fixed performance constraint) that a user may require in practice.
3. The real (wall-clock) running time for different NAS methods should be reported in the results. 
4. Baselines. The presented results are a reasonable start, but they are all very much variations within the same family. One would like to know how the current method compares to: (i) Direct application of ENAS (See also Q1), (ii) Training the full network and applying standard NN pruning techniques (such as magnitude based), (iii) Baseline of training a comparably small sized network directly. (ivv) Importantly, since the size of networks here are very small, classic neuro-evolution algorithms from evolutionary robotics such as NEAT (Stanley & Miikkulainen) seem to provide a reasonable alternative. It’s hard to know if to be impressed with the current result or not without seeing the results for a decent NEAT-like competitor. (See also Q0).
Other:
A. A presentation of the algorithm with pseudocode would help the reader follow an overview of the algorithm. 
B. Is the RL network/method used for the tasks the same as the network used in paper “Structured Evolution with Compact Architectures for Scalable Policy Optimization”? If so what is the source of the discrepancy between Tab 2 here and Tab 1 in that paper? 
C. The title doesn’t make the content obvious to a naive prospective reader. At least something like “RL with Chromatic Networks for Compact Architecture Search” would be more informative.
D. Eq (1) has a vspace error.