Differentiable Architecture Search for Reinforcement Learning
Abstract In this paper, we investigate the fundamental question: To what extent are gradient-based neural architecture search (NAS) techniques applicable to RL?
Using the original DARTS as a convenient baseline, we discover that the discrete architectures found can achieve up to 250% performance compared to manual architecture designs on both discrete and continuous action space environments across off-policy and on-policy RL algorithms, at only 3x more computation time.
Furthermore, through numerous ablation studies, we systematically verify that not only does DARTS correctly upweight operations during its supernet phrase, but also gradually improves resulting discrete cells up to 30x more efficiently than random search, suggesting DARTS is surprisingly an effective tool for improving architectures in RL.
1 Introduction and Motivation
[Figure]
Over the past few years, Differentiable Architecture Search (DARTS) (Liu et al., 2019) has dramatically risen in popularity as a method for neural architecture search (NAS), with multiple modifications and improvements (Chu et al., 2020c; Li et al., 2021;
Chu et al., 2020a; Liang et al., 2019; Wang et al., 2021a; Chu et al., 2020b;
Hundt et al., 2019; Wang et al., 2021b; Chen and Hsieh, 2020; Zela et al., 2020) constantly proposed at a rapid speed.
From a broader perspective, one may wonder how far we may push the limits of differentiable search, as differentiability is the cornerstone of all deep learning research.
One such very large application space is in reinforcement learning (RL), with DARTS being a natural fit due to its simplicitly and modularity in integrating with large distributed RL training pipelines.
Shockingly however, among the vast amounts of literature on DARTS, there are virtually no works addressing its viability in RL.
This can be attributed to the fact that RL fundamentally does not follow the same optimization paradigm as supervised learning (SL).
The end goal of RL is not to simply minimize the loss or accuracy over a fixed dataset, but rather to improve a policy’s reward over an entire environment whose training data is generated by the policy itself.
This scenario raises the possibility of a negative feedback loop in which a poorly trained policy may achieve trivial reward even if it successfully optimizes its loss to zero, and thus the loss is not an indicator of a policy’s true performance.
As DARTS is purely reliant on optimizing the architecture topology by using only the loss as the search signal, therein lies the simple question: Would DARTS even work in RL?[-
-]
© 2022 the authors, released under CC BY 4.
It is highly valuable to answer this question, as recently, works in larger-scale RL suggest that policy architecture designs greatly affect various metrics such as generalization, transferrability, and efficiency.
One surprising phenomenon found on Procgen (Cobbe et al., 2020), a procedural generation benchmark for RL, was that "IMPALA-CNN", a residual convolutional architecture from (Espeholt et al., 2018), could substantially outperform "NatureCNN", the standard 3-layer architecture used for Atari (Mnih et al., 2013), in both generalization and sample complexity under limited and infinite data regimes respectively (Cobbe et al., 2019).
Furthermore, in robotics subfields such as grasping (Kalashnikov et al., 2018; Rao et al., 2020), cameras collect very detailed real-world images (ex: 472 x 472, 4x larger than ImageNet (Russakovsky et al., 2015)) for observations which require deep image encoders consisting of more than 15 convolutions, raising concerns on efficiency and speed in policy training and inference.
As such RL policy networks gradually become larger and more sophisticated, so does the need for understanding and automating such designs.
In this paper, we show that DARTS can in fact find better architectures efficiently in RL, one of the first instances in which the loss does not directly affect the final objective.
This work may be of interest to both the gradient-based NAS community as means to expand to the RL domain, and the AutoRL community as a practical toolset and method for co-training policies and neural architectures for better performing agents.
Our contributions are:
• We conceptually identify the key differences between SL and RL in terms of their usage of the loss function, which raise important hypothetical questions and issues about whether DARTS is applicable to RL.
In particular, these deal with the quality of the training signal to the architecture variables during supernet training, and downstream effects on discrete cells during evaluation.
• Empirically, we find that DARTS is in fact compatible with several on-policy and off-policy algorithms including PPO (Schulman et al., 2017), Rainbow-DQN (Hessel et al., 2018), and SAC (Haarnoja et al., 2018).
The discrete architectures found can reach up to 250% performance compared to manual architecture designs on discrete action (e
g. Procgen) and continuous control (e.g. DM-Control) environments, at only 3x more computation time.
• Through comprehensive ablation studies, we show the supernet successfully trains, and reasonably upweights optimal operations.
We further verify both qualitatively and quantitatively that discretized cells gradually evolve to better architectures.
However, we also demonstrate how this can fail, especially if the corresponding supernet fails to train, with further extensive ablations in the Appendix.
Related Works.
Recently, there have been a flurry of works modifying many components in the
RL pipeline, both manually and automatically, as part of the broader Automated Reinforcement
Learning (AutoRL) (Parker-Holder et al., 2022) field. Specifically for architecture components, manual modifications include (Raileanu and Fergus, 2021; Nafi et al., 2021; Tang and Ha, 2021) which have shown great success in improving metrics such as sample complexity and generalization, especially on the Procgen benchmark. However, very few works have considered the possibility of actually automating the search for new architectures, i.e. "NAS for RL", specifically for large-scale modern convolutional networks. Most previous NAS for RL works only involve small policies trained via blackbox/evolutionary optimization methods, which include (Song et al., 2021; Gaier and Ha, 2019; Stanley and Miikkulainen, 2002; Stanley et al., 2009), utilizing CPU workers for forward pass evaluations rather than exact gradient computation on GPUs.
Such methods are usually unable to train policies involving more than 10K+ parameters due to the sample complexity of zeroth order methods in high dimensional parameter space (Agarwal et al., 2010).
The only previous known application of gradient-based routing is (Akinola et al., 2021), which searches for the optimal way of combining observation and action tensors together in off-policy QT-OPT (Kalashnikov et al., 2018), but does Discretization: Discretize Evaluation not search for image encoders nor uses the supernet for inference, as it trains using off-policy robotic data collected independently.
This leaves the applicability of DARTS to inference-dependent RL as an open question addressed in our work.
2 Problem Overview and Method
DARTS Preliminaries.
Since we only use the original DARTS (Liu et al., 2019) to reduce confounding factors, we thus give a very brief overview of DARTS to save space.
More comprehensive details can be found in Appendix G.5 and the original paper.
DARTS optimizes substructures called cells, where each cell contains 𝐼 intermediate nodes organized in a directed acyclic graph, where each node 𝑥 ( 𝑖 ) , represents a feature map, and each edge ( 𝑖, 𝑗 ) consists of an operation (op) 𝑜 ( 𝑖,𝑗 ) , with later nodes 𝑥 ( 𝑗 ) merged (e.g. summation) from some previous 𝑜 ( 𝑖,𝑗 )
( 𝑥 ( 𝑖 ) ) .
A DARTS supernet is constructed by continuously relaxing selection of ops in O , via softmax weighting, i.e. 𝑜 ( 𝑖,𝑗 ) ( 𝑥 ( 𝑖 ) ) =
(cid:205) 𝑜 ∈ O 𝑝 ( 𝑖,𝑗 )
𝑜 · 𝑜 ( 𝑥 ( 𝑖 ) ) , where 𝑝 ( 𝑖,𝑗 ) 𝑜 = exp ( 𝛼 ( 𝑖,𝑗 ) 𝑜 ) (cid:205) 𝑜
(cid:48)∈ O exp ( 𝛼 ( 𝑖,𝑗 ) 𝑜 (cid:48) ) . The cell’s output is by default the result of a Conv1x1 op on the depthwise concatenation of all intermediate node features, although this may be changed (e.g. by simply outputting the last intermediate node’s features). We denote the collection of all architecture variables 𝑎 ( 𝑖,𝑗 ) 𝑜 as 𝛼 . Denote the total set of possible operations in our searchable network as O =
O 𝑏𝑎𝑠𝑒 ∪{ Zero, Skip } which must contain Zero and Skip Connection ops, while O 𝑏𝑎𝑠𝑒 is user-defined.
Denote 𝛼 to be the pre-softmax trainable architecture variables in the supernet.
A predefined loss function L ( 𝜃 ) over neural network weights 𝜃 will thus be redefined as L ( 𝜃, 𝛼 ) when under DARTS’s search mode , where the original model 𝑓 𝜃 will be replaced with a dense supernet 𝑓 𝜃,𝛼 .
During evaluation time, a trained 𝛼 ∗ is then discretized into a sparser final cell 𝛿 ( 𝛼 ∗ ) by representing each edge ( 𝑖, 𝑗 ) with the highest softmax weighted op, i.e. arg max 𝑜 ∈ O ,𝑜 ≠ 𝑧𝑒𝑟𝑜 𝑝 ( 𝑖,𝑗 ) 𝑜 , and then retaining only the top 𝐾 incoming edges for each intermediate node. We thus retrain over the new loss L 𝛿 ( 𝛼 ∗ ) ( 𝜙 ) , now dependent on only fresh sparse weights 𝜙 to obtain the final reported metric.
RL Preliminaries.
For RL notation, given an MDP M , denote 𝑠 𝑡 , 𝑎 𝑡 ,𝑟 𝑡 as state, action, reward respectively at time 𝑡 .
𝜋 is the policy and D is the replay buffer containing collected trajectories
𝜏 = ( 𝑠 0 , 𝑎 0 ,𝑟 0 ,𝑠 1 , . . . ) .
The goal is to maximize 𝐽 ( 𝜋 ) =
E 𝜏 ∼ 𝜋 (cid:2)(cid:205) 𝑡 ≥ 0 𝑟 𝑡 (cid:3) , the expected cumulative reward using policy 𝜋 .
In most RL algorithms, there is the notion of a neural network torso or encoder 𝑓 𝜃 mapping the state 𝑠 to a final feature vector when forming 𝜋 .
In the DARTS case, we use a supernet encoder 𝑓 𝜃,𝛼 leading to a supernet policy denoted as 𝜋 𝜃,𝛼 and also a corresponding discretized-cell policy 𝜋 𝜙,𝛿 ( 𝛼 ∗ ) for evaluation.
2.1 Methodology
SL features the notion of training and validation sets, with corresponding losses L 𝑆𝐿𝑡𝑟𝑎𝑖𝑛 , L 𝑆𝐿𝑣𝑎𝑙 , where the learning procedure consists of a bilevel optimization problem and the goal is to find
[Equation]
) where 𝜃 ∗ = arg min 𝜃 L 𝑆𝐿𝑡𝑟𝑎𝑖𝑛 ( 𝜃, 𝛼 ) .
In this paper, we do not need to use the original bilevel optimization framework, as we are optimizing sample complexity and raw training performance, which are standard metrics in RL.
Furthermore, bilevel optimization is notoriously difficult and unstable, sometimes requiring special techniques (Liu et al., 2018;
Dong and Yang, 2019; Hundt et al., 2019; Li et al., 2020; Chu et al., 2020b,a,c; Liang et al., 2019; Wang et al., 2021b) specific to SL optimization, which can confound the results and message of our paper.
We thus joint optimize both 𝜃 and 𝛼 , and the full procedure is concisely summarized in Algorithm 1 as "RL-DARTS".
However, this is easier said than done, as we explain the core issues of applying DARTS to RL below.
SL vs RL DARTS.
Fundamentally, SL relies on a fixed dataset D 𝑆𝐿 = {( 𝑥 𝑖 ,𝑦 𝑖 ) | 𝑖 ≥ 1 } , in which the loss is defined as L 𝑆𝐿 ( 𝜃 ) =
E ( 𝑥,𝑦 )∼ D [ ℓ ( 𝑓 𝜃 ( 𝑥 ) ,𝑦 )] where ℓ (·) is defined as mean squared error, cross-entropy loss, or negative log-likelihood depending on application.
These losses are strongly correlated or even equivalent to the final objective (e.g. accuracy or density estimation), and this reason can be considered a significantly contributing factor to the success of DARTS in SL.
Unfortunately in RL, there are no such guarantees that minimizing the loss L 𝑅𝐿 (·) necessarily improves the true objective 𝐽 (·) , for two primary reasons:
The RL agent’s dataset (a.k.a.
replay buffer)
D 𝑅𝐿 =
{ 𝜏 𝑖 | 𝑖 ≥ 1 } is significantly non-stationary and self-dependent, as it constantly changes based on the current performance of data collection actors, which themselves are functions of 𝜃 .
Thus, a negative feedback loop may arise, where 𝜃 produces an actor which collects poor training data, leading to convergence towards an even poorer 𝜃 (cid:48) .
While the loss L 𝑅𝐿 (·) converges to 0 over low quality data, the reward 𝐽 (·) still does not increase.
This issue is commonplace in RL, such as in any environments which require exploration.
Hypothetically in the DARTS case, 𝛼 can potentially produce the same negative feedback loop by converging to subpar operations and impairing supernet training, also leading to a poor discrete cell 𝛿 ( 𝛼 ) .
The losses are considerably more complex and utilize multiple auxiliary losses which are used to assist training but never used during evaluation.
For PPO, the loss is defined as L 𝑅𝐿𝑃𝑃𝑂 ( 𝜃 ) =
E 𝜏 ∼ D 𝑅𝐿
[ 𝐿 𝐶𝐿𝐼𝑃 ( 𝜃 ) − 𝐿 𝑉𝐹 ( 𝜃 ) + H ( 𝜃 )] .
However, the value function (corresponding to 𝐿 𝑉𝐹 ) nor the policy entropy (corresponding to H ) are never used to evaluate final reward.
This is similarly the case for off-policy algorithms such as SAC which strongly emphasizes maximizing the entropy H and Rainbow-DQN which also uses value functions to assist training.
The question is thus raised in the DARTS case as to whether such auxiliary losses are actually useful signals, or inhibit proper training of 𝛼 , whose main goal is only maximize 𝐽 (·) using discrete cell 𝛿 ( 𝛼 ) .
The core theme of our experiments will be understanding whether these are obstacles to
DARTS’s application to RL.
3 Experiments
Experiment Setup: To verify that every component of RL-DARTS works as intended, we seek to answer all questions below, by first presenting end-to-end results, and then further key ablation studies:
End-to-End Performance:
Overall, how do the final discrete cells perform at evaluation, and what gains can we obtain from architecture search?
Furthermore, how does RL-DARTS compare against random search?
Supernet Training: During supernet training, how does the 𝛼 change?
Does 𝛼 converge towards a sparse solution and select good operations over supernet training?
Discrete Cells: Even if the supernet trains, do the corresponding discrete cells also improve in evaluation performance throughout 𝛼 ’s training?
What kinds of failure modes occur?
Following common NAS practices (Zoph et al., {+
 ;+} Pham et al., 2018; Zoph and Le, 2016), we construct our supernet (with 𝐼 intermediate cells) by stacking both normal ( 𝑁 times) and reduction cells ( 𝑅 times) together into blocks , which are themselves also stacked together 𝐷 times (see Figure 2).
Reduction cells apply a stride of 2 on the input.
Each block possesses its own convolutional channel depth, used throughout all cells in the block.
During search, we train a smaller supernet (i.e. depth 16) to reduce computation time, but evaluate final discretized cells on larger models (i.e. depth 64), with 𝐷 = 3 layers for cheap large-scale runs and 𝐷 = 5 layers for fine-grained A/B testing. [Figure]
For the operation search space, we consider the following base ops O 𝑏𝑎𝑠𝑒 and corresponding algorithms:
• Classic on PPO: O 𝑏𝑎𝑠𝑒,𝑁 =
{Conv3x3+ReLU, Conv5x5+ReLU, Dilated3x3+ReLU, Dilated5x5+ReLU} for normal ops and O 𝑏𝑎𝑠𝑒,𝑅 = {Conv3x3, MaxPool3x3, AveragePool3x3} for reduction ops, which is standard in supervised learning (Liu et al., 2019; Zoph et al., 2018; Pham et al., 2018).
• Micro on Rainbow and SAC: We also propose O 𝑏𝑎𝑠𝑒,𝑁
= {Conv3x3, ReLU, Tanh}, a more fine-grained and novel search space which has not been used previously in SL.
The inclusion of Tanh is motivated by its use previously for continuous control architectures (Salimans et al., ; Song et al., 2020).
For benchmarks, we primarily use Procgen (Cobbe et al., 2019, 2020) for discrete action spaces, with PPO (Schulman et al., 2017) and Rainbow DQN (Hessel et al., 2018) as training algorithms, with Procgen’s difficulty set to "easy" similar to other works (Raileanu et al., 2020; Raileanu and Fergus, 2021; Parker-Holder et al., 2021a).
Procgen comprehensively evaluates all aspects of RL-DARTS, as it possesses a diverse selection of 16 games, each with infinite levels to simulate large data regimes where episodes may drastically change, relevant for generalization.
It further uses the IMPALA-CNN architecture (Espeholt et al., 2018) as a strong hand-designed baseline, and can be seen as a specific instance of the stacked cell design in Figure 2, where its "Reduction Cell" consists of a Conv3x3 and MaxPool3x3 (Stride 2) with 𝑅 = 1 and its "Normal Cell" consists of a residual layer with Conv3x3’s and ReLU’s, with 𝑁
= 2.
For fair comparisons to IMPALA-CNN, we use ( 𝑁, 𝑅, 𝐼 ) =
( 1 , 1 , 4 ) on the classic search space, while ( 𝑁, 𝑅, 𝐼 ) =
( 2 , 0 , 4 ) on the Micro search space, where reduction ops default to IMPALA-CNN’s in order to avoid hidden confounding effects when visualizing Micro cells.
In addition, we also assess DARTS’s viability in continuous control which is common in robotics tasks.
We use the common DM-Control benchmark (Tassa et al., 2018) along with the popular SAC algorithm.
We use 𝑁 = 3 , 𝐼 = 4 , 𝐾 = 1 with "Micro" search space to remain fair to the 4-layer convolutional encoder baseline observing images of size 64 × 64 (full details in Appendix G).
Unless specified, we by default use consistent hyperparameters (found in Appendix G) for all comparisons found inside a figure, although learning rate and minibatch size may be altered when training models of different sizes due to GPU memory limits.
Thus, even though RL is commonly sensitive to hyperparameters (Zhang et al., 2021), we surprisingly find that once a pre-existing
RL baseline has already been setup, incorporating DARTS requires no extra cost in tuning, as evidence of its ease-of-use.
3.1 End-to-End Results on Multi-task, Discrete and Continuous Control Tasks
Multi-game Search.
We first begin with the most surprising and largest end-to-end result in terms of scale of data and compute shown in Figure 3:
By training a supernet across infinite levels across all 16 Procgen games to find a single transferrable cell, we are able to achieve up to 250 % evaluation performance compared to the IMPALA-CNN baseline over select environments.
This is achieved using Rainbow with our proposed "Micro" search space, where a learner performs gradient updates over actor replay data (with normalized rewards) from all games.
We further display the discovered discrete cell and the supernet joint-training procedure in
Figure 4.
Interestingly, the discrete cell uses nonlinearities over all intermediate connections, with convolutions only used via the merge operation for the output.
Single-game Search.
We further compare DARTS’s end-to-end performance against random search, but more appropriately applied to single game scenarios.
On the PPO side, we use the "Classic" search space (total size 4 × 10 11 , see Appendix H.1).
For a fair comparison, we ensure total wall-clock time (with same hardware) stays equal, as common in (Liu et al., 2019).
Since in Appendix A, Table 4, a PPO supernet takes 2.5x longer to reach 25M steps, this is rounded to a random search budget of 3 cells to be trained with depths 16 × 3 for 25M steps.
The best of the 3 cells is used for full evaluation.
In Table 1, on average, random search underperforms significantly.
[Table]
In Figure 5 we further find that RL-DARTS is capable of finding game-specific architectures from scratch which outperform IMPALA-CNN on select environments such as Plunder and Heist, while maintaining competitive performance on others.
100 Random Cell Comparison.
To further present a more comprehensive comparison against random search, and understand how strong IMPALA-CNN is as a baseline, we compare against 100 unique random cells. To manage the computational load, we performed the study over two selected environments, as shown in Figure 6 and find that all of the random cells underperform against both IMPALA-CNN and DARTS . This demonstrates that DARTS possesses a strong search capability, achieving 100 / 3 ≈ 30 𝑥 efficiency over random search and can discover architectures that match in complexity and performance of highly-tuned, expert designed architectures such as IMPALA-CNN.
DM-Control with Soft Actor-Critic.
DARTS also consistently finds better and stable architectures over multiple lightweight environments involving continuous control (see Figure 7) trained up to 1M steps.
Even though the 4-layer encoder network used (details in Appendix G) is significantly smaller than IMPALA-CNN, we find that there is still room for architectural improvement.
3.2 Role of Supernet Training
In Figure we verify that training the supernet end-to-end works effectively, even with minimal hyperparameter tuning .
Furthermore, the training only requires at 3x more compute time, with extensive efficiency metrics calculated in Appendix A.
However, as mentioned in Subsec.
2.1, it is unclear whether the right signals are provided to operation routing variables 𝛼 via the RL training loss, and whether 𝛼 produces the correct behavior, which we investigate.
Appendix Conveniently in Figure we find that 𝛼 strongly downweights all base ops (in particular, 5x5 ops) except for the standard Conv3x3+ReLU.
This provides an opportunity to understand whether 𝛼 downweights suboptimal ops throughout training.
We confirm this result in Tableby evaluating standard IMPALA-CNN cells using either purely 3x3 or 5x5 convolutions for the whole network and demonstrating that the 3x3 setting outperforms the 5x5 setting (especially in limited data, e.g. 200-level training/test regime), suggesting the signaling ability of 𝛼 on op choice.
[Table]
steps) on Dodgeball.
Learning curves can be found in Appendix D.1, Figure 15.
Answering the converse question is just as important: Can any supernet train?
The supernet possesses an incredibly dense set of weights, and thus one might wonder whether trainability occurs with any search space or settings.
We answer in the negative, where a poorly designed supernet can fail.
To show this clearly, we remove all ReLU nonlinearities from the "Classic" search space ops used for PPO, as well as simply freeze 𝛼 to be uniform for Rainbow, and find both cases produce poor training in Table 3.
Thus, the supernet in RL provides important search signals in terms of reward and α during training, especially on the quality of a search space. [Table] 25M
[Table]
3.3 Discrete Cell Improvement
We further must analyze whether discrete cells also improve, as they are used for final evaluation and deployment.
Strong supernet performance (via continuous relaxation) does not necessarily imply strong evaluation cell performance (via discretization), due to the existence of integrality gaps (Wang et al., 2021b,a).
Nevertheless, we demonstrate that even using the default 𝛿 discretization from (Liu et al., 2019) leads to both quantitative and qualitative improvements.
Quantitative improvement.
As 𝛼 changes during supernet training, so do the outputs of the discretization procedure on 𝛼 .
We collect all distinct discrete cells { 𝛿 ( 𝛼 1 ) , 𝛿 ( 𝛼 2 ) , . . .
} into a sequence, and evaluate each cell’s performance max 𝜙 𝐽 ( 𝜋 𝜙,𝛿 ( 𝑎 𝑖 ) ) ∀ 𝑖 via training from scratch for 25M steps (Figure 9).
The performance generally improves over time, indicating that supernet optimization selects better cells.
However, we find that such behavior can be environment-dependent, as some environments possess less monotonic evaluation curves (see Fig.
18 in Appendix D).
Qualitative improvement.
We visualize discrete cells 𝛿 ( 𝛼 𝑠𝑡𝑎𝑟𝑡 ) , 𝛿 ( 𝛼 𝑒𝑛𝑑 ) from the start and end of supernet training with Rainbow on the Starpilot environment in Fig.
The earlier cell consists of only linear operators is clearly a poor design in comparison to the later cell.
We find similar cell evolution results for PPO in Figures 16, 17 in Appendix D.2, displaying more sophisticated yet still interpretable changes in cell topology.
We further answer one of the most common questions in DARTS research: What is the direct relationship between a supernet and its discrete cell?
In Figure 11, we provide two supernet runs along with their corresponding discrete cells, side by side in order to answer this question.
While "Supernet 1" is a standard successful training run, "Supernet 2" is a failed run which can commonly occur due to the inherent sensitivity and variance in RL training.
As it turns out, this directly leads to differences between their corresponding discrete cells both quantitatively and qualitatively as well, in which "Discretized Cell 1"’s design appears to make sense and train properly, while "Discretized Cell 2" is clearly a suboptimal design, and fails to train at all.
Thus, a major reason for why a discretized cell may underperform is if its corresponding supernet fails to learn .
[Figure] We investigate further in Appendix D.3, Figure 19, and find that supernet and discrete cell rewards are indeed correlated, after adjusting for environment-dependent factors, suggesting that search quality can be improved via both better supernet training (Raileanu et al., 2020; Zhang et al., 2021), as well as better discretization procedures (Liang et al., 2019; Wang et al., 2021b).
4 Conclusions, Limitations, and Broader Impact Statement
Conclusion.
Even though RL uses complex loss functions defined over nonstationary data, we empirically showed that nonetheless, DARTS is capable of improving policy architectures in a minimally invasive and efficient way (only 3x more compute time) across several different algorithms (PPO, Rainbow, SAC) and environments (Procgen, DM-Control).
Our paper is the first to have comprehensively provided evidence for the applicability of softmax/gradient-based architecture search outside of standard classification and SL.
Limitations.
To avoid confounding factors and for simplicity, our paper uses the default DARTS method.
However, we outline multiple possible improvements in Appendix F, as RL is a completely new frontier for which to understand softmax routing and continuous relaxation techniques.
Future Work and Broader Impact.
In this paper, we have provided concrete evidence that architecture search can be conducted practically and efficiently in RL, via DARTS.
We believe that this could start important initiatives into finding better and more efficient (Cai et al., 2019)
architectures for large-scale robotics (James et al., 2019; Akinola et al., 2021), transferable architectures in offline RL (Levine et al., 2020), as well as RNNs for memory (Pritzel et al., 2017; Fortunato et al., 2019; Kapturowski et al., 2019) and adaptation (Duan et al., 2016; Wang et al., 2017).
Other NAS methods’ applicability in RL may also be investigated, especially ones which utilize blackbox optimization controllers, such as multi-trial evolution (Real et al., 2019) or ENAS (Pham et al., 2018).
Furthermore, we believe our work’s potential negative impacts are generally equivalent to ones found in general NAS methods, such as sacrificing model interpretability in order to achieve higher objectives.
For the field of RL specifically, this may warrant more attention in AI safety when used for real world robotic pipelines. Furthermore, as with any NAS research, the initial phase of discovery and experimentation may contribute to carbon emissions due to the computational costs of extensive tuning. However, this is usually a means to an end, such as an efficient search algorithm, which this paper proposes with no extra hardware costs. 5 Reproducibility Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We have comprehensively demonstrated the behavior of all components in DARTS when applied inside an RL pipeline.
(b) Did you describe the limitations of your work?
[Yes] Yes, see Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Yes, see Section 4.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] Yes, see Section 4.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A]
No theoretical results provided.
(b) Did you include complete proofs of all theoretical results?
[N/A] No proofs provided.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results, including all requirements (e.g., requirements.txt with explicit version), an instructive README with installation, and execution commands (either in the supplemental material or as a url)?
[Yes] We have included the core code for creating a DARTS supernet and discrete cell, as well as training code for Procgen on both PPO and Rainbow, and addition added the modified files from an opensource variant of SAC on DM-Control.
We have also added the README on guidance for code organization and running experiments, as well as a requirements.txt .
The code can be found at https://github.com/google/brain_autorl/tree/main/rl_darts .
(b) Did you include the raw results of running the given instructions on the given code and data?
[Yes] We have provided the data related to the core results of this paper.
(c) Did you include scripts and commands that can be used to generate the figures and tables in your paper based on the raw results of the code, data, and instructions given?
[Yes] We have provided relevant figure-plotting utilities in the code submission.
(d) Did you ensure sufficient code quality such that your code can be safely executed and the code is properly documented?
[Yes] The code was formatted by a strict automatic Python lint-checker, as well as reviewed by other individuals.
The code also provides tests for modules, which can be used to understand the intended use.
(e) Did you specify all the training details (e.g., data splits, pre-processing, search spaces, fixed hyperparameter settings, and how they were chosen)?
[Yes] We discuss the explicit hyperparameters in Appendix G, as well as search space details in Section 3.
(f) Did you ensure that you compared different methods (including your own) exactly on the same benchmarks, including the same datasets, search space, code for training and hyperparameters for that code?
[Yes] We ensured the same evaluation protocol via making sure all depths remain the same (16 for supernets, 64 for discrete cells, and 𝐷 = 3 layers for cheaper large-scale runs or 𝐷 =for fine-grained A/B testing).
We also used the exact same hyperparameters for all discrete/baseline runs, while only needing to change minibatch size + learning rate in accomodate networks with larger GPU memory sizes.
(g) Did you run ablation studies to assess the impact of different components of your approach?
[Yes] For ablation studies, we provided a large set of studies as seen throughout Subsections 3.2 and 3.3 as well as in the Appendix.
(h)
Did you use the same evaluation protocol for the methods being compared?
[Yes] In terms of comparing NAS methods, we indeed controlled for confounding factors by using the same hardware,
as shown in Table 4 in Appendix A. We further performed side-by-side comparisons for RL training curves using environment steps, which is standard.
(i) Did you compare performance over time?
[Yes] Table 4 in Appendix A contains wall-clock time comparisons.
In terms of performance comparisons over time, we showed the performance of RL-DARTS’s discrete cells over the search procedure / supernet training in Figure 9.
(j) Did you perform multiple runs of your experiments and report random seeds?
[Yes] For seeded runs, as standard in RL, we performed 3-seeded runs for all experiments.
(k) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] For seeded runs, as standard in RL, we performed 3-seeded runs for all experiments and reported error bars.
(l) Did you use tabular or surrogate benchmarks for in-depth evaluations?
[Yes] We extensively covered the comparisons against random search in Subsection 3.1, over all 16 games as well as a large scale 100 random cell comparison in Figure 6.
(m) Did you include the total amount of compute and the type of resources used (e.g., type of gpus, internal cluster, or cloud provider)?
[Yes] See Table 4 in Appendix A for GPU and compute time used for all evaluations.
We applied this to all 16 games on Procgen.
(n) Did you report how you tuned hyperparameters, and what time and resources this required (if they were not automatically tuned by your AutoML method, e.g. in a nas approach; and also hyperparameters of your own method)?
[Yes] This is written in detail in Appendix G as well as Section 3. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We have cited Procgen and DM-Control.
(b) Did you mention the license of the assets?
[N/A] Both are publicly licensed and freely available.
(c) Did you include any new assets either in the supplemental material or as a url?
[Yes] We included the relevant publicly available code for environments and algorithms in Section G. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] Benchmarks are publicly available, consent not needed.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] No personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] No research was conducted on human subjects or crowdsourcing.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (irb) approvals, if applicable?
[N/A]
No research was conducted on human subjects or crowdsourcing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
No research was conducted on human subjects or crowdsourcing.