Original: RL-DARTS: D IFFERENTIABLE A RCHITECTURE S EARCH FOR R EINFORCEMENT L EARNING
Revised: RL-DARTS: D IFFERENTIABLE A RCHITECTURE S EARCH FOR R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, Differentiable Architecture Search (DARTS) has become one of the most popular Neural Architecture Search (NAS) methods successfully applied in supervised learning (SL).
Revised: Recently, Differentiable Architecture Search (DARTS) has become one of the most popular Neural Architecture Search (NAS) methods successfully applied in supervised learning (SL).

Original: However, its applications in other domains, in particular for reinforcement learning (RL), has seldom been studied.
Revised: However, its applications in other domains, in particular for reinforcement learning (RL), has seldom been studied.

Original: This is due in part to
Revised: This is due in part to

Original: RL possessing a signiﬁcantly different optimization paradigm than SL, especially with regards to the notion of replay data, which is continually generated via inference in RL.
Revised: RL possessing a signiﬁcantly different optimization paradigm than SL, especially with regards to the notion of replay data, which is continually generated via inference in RL.

Original: In this paper, we introduce RL-DARTS, one of the ﬁrst applications of end-to-end DARTS in RL to search for convolutional cells, applied to the challenging, inﬁnitely procedurally generated Procgen benchmark.
Revised: In this paper, we introduce RL-DARTS, one of the ﬁrst applications of end-to-end DARTS in RL to search for convolutional cells, applied to the challenging, inﬁnitely procedurally generated Procgen benchmark.

Original: We demonstrate that the beneﬁts of DARTS become ampliﬁed when applied to RL, namely search efﬁciency in terms of time and compute, as well as simplicity in integration with complex preexisting RL code via simply replacing the image encoder with a
Revised: We demonstrate that the beneﬁts of DARTS become ampliﬁed when applied to RL, namely search efﬁciency in terms of time and compute, as well as simplicity in integration with complex preexisting RL code via simply replacing the image encoder with a

Original: DARTS supernet, compatible with both off-policy and on-policy RL algorithms.
Revised: DARTS supernet, compatible with both off-policy and on-policy RL algorithms.

Original: At the same time however, we provide one of the ﬁrst extensive studies of DARTS outside of the standard ﬁxed dataset setting in SL via RL-DARTS.
Revised: At the same time however, we provide one of the ﬁrst extensive studies of DARTS outside of the standard ﬁxed dataset setting in SL via RL-DARTS.

Original: We show that throughout training, the supernet gradually learns better cells, leading to alternative architectures which can be highly competitive against manually designed policies, but also verify previous design choices for RL policies.
Revised: We show that throughout training, the supernet gradually learns better cells, leading to alternative architectures which can be highly competitive against manually designed policies, but also verify previous design choices for RL policies.

Original:  I NTRODUCTION AND M OTIVATION
Revised:  I NTRODUCTION AND M OTIVATION

Original: Over the last decade, recent advancements in deep reinforcement learning have heavily focused on algorithmic improvements (Hessel et al., 2018; Wang et al., 2016), data augmentation (Raileanu et al., 2020; Kostrikov et al., 2020), infrastructure upgrades (Espeholt et al., 2018; Horgan et al., 2018), and even hyperparameter tuning (Zhang et al., 2021; Faust et al., 2019).
Revised: Over the last decade, recent advancements in deep reinforcement learning have heavily focused on algorithmic improvements (Hessel et al., 2018; Wang et al., 2016), data augmentation (Raileanu et al., 2020; Kostrikov et al., 2020), infrastructure upgrades (Espeholt et al., 2018; Horgan et al., 2018), and even hyperparameter tuning (Zhang et al., 2021; Faust et al., 2019).

Original: The automation of such RL methods have given rise to the ﬁeld of Automated Reinforcement Learning (AutoRL).
Revised: The automation of such RL methods have given rise to the ﬁeld of Automated Reinforcement Learning (AutoRL).

Original: Surprisingly however, one relatively underdeveloped and unexplored area is automating large-scale policy architecture design.
Revised: Surprisingly however, one relatively underdeveloped and unexplored area is automating large-scale policy architecture design.

Original: Recently, works in larger-scale RL suggest that designing a policy’s architecture can be just as important as the algorithm or quality of data, if not more, for various metrics such as generalization, transferrability, and efﬁciency.
Revised: Recently, works in larger-scale RL suggest that designing a policy’s architecture can be just as important as the algorithm or quality of data, if not more, for various metrics such as generalization, transferrability, and efﬁciency.

Original: One surprising phenomenon found on Procgen (Cobbe et al., 2020), a procedural generation benchmark for RL, was that “IMPALA-CNN”, a residual convolutional architecture from (Espeholt et al., 2018), could substantially outperform “NatureCNN”, the standard 3-layer convolutional architecture used for Atari (Mnih et al., 2013), in both generalization and sample complexity under limited and inﬁnite data regimes respectively (Cobbe et al., 2019).
Revised: One surprising phenomenon found on Procgen (Cobbe et al., 2020), a procedural generation benchmark for RL, was that “IMPALA-CNN”, a residual convolutional architecture from (Espeholt et al., 2018), could substantially outperform “NatureCNN”, the standard 3-layer convolutional architecture used for Atari (Mnih et al., 2013), in both generalization and sample complexity under limited and inﬁnite data regimes respectively (Cobbe et al., 2019).

Original: Furthermore, in robotics subﬁelds such as grasping (Kalashnikov et al., 2018;
Revised: Furthermore, in robotics subﬁelds such as grasping (Kalashnikov et al., 2018;

Original: Rao et al., 2020), cameras collect very detailed real-world images (ex: 472 x 472, 4x larger than ImageNet (Russakovsky et al., 2015)) for observations which require deep image encoder architectures, consisting of more than 15 convolutions, raising concerns on efﬁciency and speed in policy training and inference.
Revised: Rao et al., 2020), cameras collect very detailed real-world images (ex: 472 x 472, 4x larger than ImageNet (Russakovsky et al., 2015)) for observations which require deep image encoder architectures, consisting of more than 15 convolutions, raising concerns on efﬁciency and speed in policy training and inference.

Original: As such RL policy networks gradually become larger and more sophisticated, so does the need for understanding and automating such designs.
Revised: As such RL policy networks gradually become larger and more sophisticated, so does the need for understanding and automating such designs.

Original: Many efﬁcient Neural Architecture Search (NAS) methods utilize the one-shot setup, where only one set of shared weight parameters is trained through an ensemble of models, otherwise known as a supernet.
Revised: Many efﬁcient Neural Architecture Search (NAS) methods utilize the one-shot setup, where only one set of shared weight parameters is trained through an ensemble of models, otherwise known as a supernet.

Original: Differentiable Architecture Search (Liu et al., 2019), trains the supernet by inserting architecture selection variables inside the network in a continuous relaxation approach, in order to be differentiable with respect to a loss.
Revised: Differentiable Architecture Search (Liu et al., 2019), trains the supernet by inserting architecture selection variables inside the network in a continuous relaxation approach, in order to be differentiable with respect to a loss.

Original: These architecture search parameters are then used to select a ﬁnal discretized perception module to be retrained from scratch in an evaluation phase.
Revised: These architecture search parameters are then used to select a ﬁnal discretized perception module to be retrained from scratch in an evaluation phase.

Original: The differentiable aspect of DARTS is highly attractive to RL, which possess two complications which SL does not:
Revised: The differentiable aspect of DARTS is highly attractive to RL, which possess two complications which SL does not:

Original: RL is an entire system .
Revised: RL is an entire system .

Original: In most RL algorithms involving auto-differentiation, there are multiple different phases: a collection/exploration phase which collects environment data via forward passes/inference of a policy, in addition to a training phase which updates parameters for potentially multiple network components (e.g. both the policy and value function in policy gradient methods).
Revised: In most RL algorithms involving auto-differentiation, there are multiple different phases: a collection/exploration phase which collects environment data via forward passes/inference of a policy, in addition to a training phase which updates parameters for potentially multiple network components (e.g. both the policy and value function in policy gradient methods).

Original: There can also even be a separate evaluation phase (disjoint from evaluation in NAS) different from the collection phase (e
Revised: There can also even be a separate evaluation phase (disjoint from evaluation in NAS) different from the collection phase (e

Original: g. using argmax greedy evaluation instead of the stochastic actor, for both Q-learning and policy gradient methods).
Revised: g. using argmax greedy evaluation instead of the stochastic actor, for both Q-learning and policy gradient methods).

Original: Such phases are also commonly distributed over different machines, adding an extra layer of complexity via inter-machine communication.
Revised: Such phases are also commonly distributed over different machines, adding an extra layer of complexity via inter-machine communication.

Original: RL training curves possess considerably more variance and noise than SL curves, with a much larger room for error due to sensitivity to hyperparameters (Andrychowicz et al., 2020), which originally led to the criterion of reporting mean and standard deviation for 3 seeded runs (Henderson et al., 2018).
Revised: RL training curves possess considerably more variance and noise than SL curves, with a much larger room for error due to sensitivity to hyperparameters (Andrychowicz et al., 2020), which originally led to the criterion of reporting mean and standard deviation for 3 seeded runs (Henderson et al., 2018).

Original: In fact, this noise can be nearly unbounded , as in many cases, an unlucky and catastrophic seeded training run can simply lead to a trivial reward even with an optimal architecture.
Revised: In fact, this noise can be nearly unbounded , as in many cases, an unlucky and catastrophic seeded training run can simply lead to a trivial reward even with an optimal architecture.

Original: Such noise may potentially be problematic for fast convergence for blackbox/evolutionary optimization methods.
Revised: Such noise may potentially be problematic for fast convergence for blackbox/evolutionary optimization methods.

Original: Furthermore, the RL loss contains important information not conveyed by just pure reward (e
Revised: Furthermore, the RL loss contains important information not conveyed by just pure reward (e

Original: g. value function error and entropy penalty, as well as potentially exploration/novelty bonuses), whereas information may be lost when simply summarizing all progress to a single scalar point of data.
Revised: g. value function error and entropy penalty, as well as potentially exploration/novelty bonuses), whereas information may be lost when simply summarizing all progress to a single scalar point of data.

Original: DARTS solves (1) as it integrates naturally within the RL system with minimal modiﬁcations to the code and pipeline, by simply replacing the image encoder in standard RL algorithms.
Revised: DARTS solves (1) as it integrates naturally within the RL system with minimal modiﬁcations to the code and pipeline, by simply replacing the image encoder in standard RL algorithms.

Original: For (2), architecture search parameters in the DARTS supernet are joint trained alongside weight parameters, and can thus also use information from multiple auxiliary losses, which can be much richer signals than potentially sparse rewards.
Revised: For (2), architecture search parameters in the DARTS supernet are joint trained alongside weight parameters, and can thus also use information from multiple auxiliary losses, which can be much richer signals than potentially sparse rewards.

Original: We represent this approach in Figure 1.
Revised: We represent this approach in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: components of a standard training pipeline, which may potentially
Revised: components of a standard training pipeline, which may potentially

Original: However, since there has been no previous proof of DARTS’s feasibility in standard RL, one may hypothesize a few subtle concerns.
Revised: However, since there has been no previous proof of DARTS’s feasibility in standard RL, one may hypothesize a few subtle concerns.

Original: While
Revised: While

Original: DARTS in SL is simpler and can focus more on generalization, a core major challenge in RLDARTS is the feasability of training the supernet itself due to a non-stationary data distribution.
Revised: DARTS in SL is simpler and can focus more on generalization, a core major challenge in RLDARTS is the feasability of training the supernet itself due to a non-stationary data distribution.

Original: This is especially because using a differentiable loss (ex: Bellman error) as the search objective is a double-edged sword in RL, as loss is deﬁned with respect to the replay buffer, whose data is collected by the policy itself via inference on the architecture.
Revised: This is especially because using a differentiable loss (ex: Bellman error) as the search objective is a double-edged sword in RL, as loss is deﬁned with respect to the replay buffer, whose data is collected by the policy itself via inference on the architecture.

Original: In SL, the data distribution is independent of the architecture, and the loss (e.g. cross-entropy or negative log-likelihood) is strongly correlated or even equivalent to the objective (e.g. accuracy or density estimation (Parmar et al., 2018)).
Revised: In SL, the data distribution is independent of the architecture, and the loss (e.g. cross-entropy or negative log-likelihood) is strongly correlated or even equivalent to the objective (e.g. accuracy or density estimation (Parmar et al., 2018)).

Original: However in RL, a bad architecture could lead to a poor initial distribution of data, and thus a local optimum where the loss is zero while the policy is still poor.
Revised: However in RL, a bad architecture could lead to a poor initial distribution of data, and thus a local optimum where the loss is zero while the policy is still poor.

Original: This is inherently due to the fact that the data in the replay buffer may not sufﬁciently cover the state and action space of the entire environment, and is common in cases such as sparse reward problems.
Revised: This is inherently due to the fact that the data in the replay buffer may not sufﬁciently cover the state and action space of the entire environment, and is common in cases such as sparse reward problems.

Original: This issue occurs even if the replay buffer data is ﬁxed, which is the basis of ofﬂine RL (Levine et al., 2020).
Revised: This issue occurs even if the replay buffer data is ﬁxed, which is the basis of ofﬂine RL (Levine et al., 2020).

Original: The other question occurs during the discretization phase of DARTS, as any continuous relaxation method must deal with integrality gaps when rounding back to a discrete space, with this gap dependent on the optimization landscape.
Revised: The other question occurs during the discretization phase of DARTS, as any continuous relaxation method must deal with integrality gaps when rounding back to a discrete space, with this gap dependent on the optimization landscape.

Original: At the core of multiple discretization procedures (Liu et al., 2019; Chu et al., 2020c; Hundt et al., 2019) lies the idea of op selection based on edge weights.
Revised: At the core of multiple discretization procedures (Liu et al., 2019; Chu et al., 2020c; Hundt et al., 2019) lies the idea of op selection based on edge weights.

Original: However, RL’s optimization landscape is signiﬁcantly different (Ilyas et al., 2020; Ahmed et al., 2019) from SL, and thus even if the supernet trains in RL, it is unclear if e.g. DARTS’s default discretization procedure leads to a better discrete cell, which is the original goal of DARTS and NAS in general.
Revised: However, RL’s optimization landscape is signiﬁcantly different (Ilyas et al., 2020; Ahmed et al., 2019) from SL, and thus even if the supernet trains in RL, it is unclear if e.g. DARTS’s default discretization procedure leads to a better discrete cell, which is the original goal of DARTS and NAS in general.

Original: In this paper, we carefully analyze the extent at which DARTS can optimize perception modules for RL environments.
Revised: In this paper, we carefully analyze the extent at which DARTS can optimize perception modules for RL environments.

Original: While there is a rich series of active study on the theoretical and empirical behavior of DARTS optimization in SL (Wang et al., 2021b; Chen & Hsieh, 2020; Li et al., 2021), our work is one of the ﬁrst to study its optimization in RL, a completely different paradigm.
Revised: While there is a rich series of active study on the theoretical and empirical behavior of DARTS optimization in SL (Wang et al., 2021b; Chen & Hsieh, 2020; Li et al., 2021), our work is one of the ﬁrst to study its optimization in RL, a completely different paradigm.

Original: Through extensive experiments, we ultimately validate that even vanilla DARTS is capable of architecture search in RL, laying a core foundation for future work in NAS for RL.
Revised: Through extensive experiments, we ultimately validate that even vanilla DARTS is capable of architecture search in RL, laying a core foundation for future work in NAS for RL.

Original: In summary, our contributions are:
Revised: In summary, our contributions are:

Original: • We demonstrate DARTS can be integrated with existing RL algorithms in a minimally invasive and easy-to-use manner, agnostic to the speciﬁc pipeline.
Revised: • We demonstrate DARTS can be integrated with existing RL algorithms in a minimally invasive and easy-to-use manner, agnostic to the speciﬁc pipeline.

Original: Remarkably, on the challenging Procgen benchmark featuring inﬁnite data, the DARTS supernet is able to train from scratch and efﬁciently produce similar training curves as baselines using both on-policy (PPO) and off-policy (Rainbow) methods, both of which require inference on the network.
Revised: Remarkably, on the challenging Procgen benchmark featuring inﬁnite data, the DARTS supernet is able to train from scratch and efﬁciently produce similar training curves as baselines using both on-policy (PPO) and off-policy (Rainbow) methods, both of which require inference on the network.

Original: • Due to the common issue of integrality gaps between supernet cells and discrete cells, we verify both qualitatively via visualizations and quantitatively via rewards, that discretized cells start with suboptimal architectures and gradually evolve to better architectures.
Revised: • Due to the common issue of integrality gaps between supernet cells and discrete cells, we verify both qualitatively via visualizations and quantitatively via rewards, that discretized cells start with suboptimal architectures and gradually evolve to better architectures.

Original: However, we also demonstrate how this can fail, especially if the corresponding supernet fails to train, with further extensive ablations in the Appendix.
Revised: However, we also demonstrate how this can fail, especially if the corresponding supernet fails to train, with further extensive ablations in the Appendix.

Original: • By evaluating the ﬁnal discrete cells at the end of supernet training, we discover that many environments beneﬁt from learned, custom-tailored architectures, suggesting that humandesigned architectures can be suboptimal.
Revised: • By evaluating the ﬁnal discrete cells at the end of supernet training, we discover that many environments beneﬁt from learned, custom-tailored architectures, suggesting that humandesigned architectures can be suboptimal.

Original: Furthermore, we ﬁnd that joint-training a supernet on multiple environments can potentially produce a single transferrable cell.
Revised: Furthermore, we ﬁnd that joint-training a supernet on multiple environments can potentially produce a single transferrable cell.

Original: This opens up questions for future study, as large-scale RL can potentially beneﬁt from more applications of NAS techniques.
Revised: This opens up questions for future study, as large-scale RL can potentially beneﬁt from more applications of NAS techniques.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Most previous AutoRL methods can be thought of in terms of blackbox and evolutionary methods.
Revised: Most previous AutoRL methods can be thought of in terms of blackbox and evolutionary methods.

Original: For NAS on RL, these include (Song et al., 2021;
Revised: For NAS on RL, these include (Song et al., 2021;

Original: Gaier & Ha, 2019; Stanley & Miikkulainen, 2002;
Revised: Gaier & Ha, 2019; Stanley & Miikkulainen, 2002;

Original: Stanley et al., 2009) which utilize CPU workers for forward pass evaluations rather than exact gradient computation on GPUs.
Revised: Stanley et al., 2009) which utilize CPU workers for forward pass evaluations rather than exact gradient computation on GPUs.

Original: Such methods are usually unable to train policies involving more than 10K+ parameters due to the sample complexity of zeroth order methods in high dimensional parameter space (Agarwal et al., 2010).
Revised: Such methods are usually unable to train policies involving more than 10K+ parameters due to the sample complexity of zeroth order methods in high dimensional parameter space (Agarwal et al., 2010).

Original: However, such methods have proved useful on smaller search spaces for RL, such as hyperparameter optimization (Zhang et al., 2021; Franke et al., 2020; Parker-Holder et al., 2020; Jaderberg et al., 2017) and algorithm search/“learning to learn” methods (Faust et al., 2019; Co-Reyes et al., 2021).
Revised: However, such methods have proved useful on smaller search spaces for RL, such as hyperparameter optimization (Zhang et al., 2021; Franke et al., 2020; Parker-Holder et al., 2020; Jaderberg et al., 2017) and algorithm search/“learning to learn” methods (Faust et al., 2019; Co-Reyes et al., 2021).

Original: In terms of differentiable search in RL, methods for tuning hyperparameters and environments include (Zahavy et al., 2020; Xu et al., 2018;
Revised: In terms of differentiable search in RL, methods for tuning hyperparameters and environments include (Zahavy et al., 2020; Xu et al., 2018;

Original: Hu et al., 2020).
Revised: Hu et al., 2020).

Original: The only previous known application of
Revised: The only previous known application of

Original: DARTS is (Akinola et al., 2021), which searches for the optimal way of combining the observation and action tensors together in off-policy QT-OPT (Kalashnikov et al., 2018).
Revised: DARTS is (Akinola et al., 2021), which searches for the optimal way of combining the observation and action tensors together in off-policy QT-OPT (Kalashnikov et al., 2018).

Original: However, this approach does not modify the image encoder nor uses the supernet for inference, as off-policy robotic data collected independently from the actual policy is used for training, which leaves the applicability of
Revised: However, this approach does not modify the image encoder nor uses the supernet for inference, as off-policy robotic data collected independently from the actual policy is used for training, which leaves the applicability of

Original: DARTS to inference-dependent RL as an open question addressed in our work.
Revised: DARTS to inference-dependent RL as an open question addressed in our work.

Original:  M ETHODOLOGY 3. V ANILLA DARTS AND NAS P RELIMINARIES
Revised:  M ETHODOLOGY 3. V ANILLA DARTS AND NAS P RELIMINARIES

Original: We provide a brief review of DARTS which can be pictorially summarized in Figure 2, although more details can be found in the original DARTS paper (Liu et al., 2019).
Revised: We provide a brief review of DARTS which can be pictorially summarized in Figure 2, although more details can be found in the original DARTS paper (Liu et al., 2019).

Original: DARTS optimizes substructures called cells, where each cell contains I intermediate nodes organized in a directed acyclic graph, where each node x ( i ) , represents a feature map, and each edge ( i, j ) consists of an operation (op) o
Revised: DARTS optimizes substructures called cells, where each cell contains I intermediate nodes organized in a directed acyclic graph, where each node x ( i ) , represents a feature map, and each edge ( i, j ) consists of an operation (op) o

Original: ( i,j ) , with later nodes x ( j ) merged (e.g. summation) from some previous o
Revised: ( i,j ) , with later nodes x ( j ) merged (e.g. summation) from some previous o

Original: ( i,j ) ( x ( i ) ) .
Revised: ( i,j ) ( x ( i ) ) .

Original: A DARTS supernet is constructed by continuously relaxing selection of ops in O , via softmax weighting, i.e. o ( i,j ) ( x ( i ) )
Revised: A DARTS supernet is constructed by continuously relaxing selection of ops in O , via softmax weighting, i.e. o ( i,j ) ( x ( i ) )

Original: = (cid:80) o ∈O p
Revised: = (cid:80) o ∈O p

Original: ( i,j ) o
Revised: ( i,j ) o

Original: · o ( x ( i ) ) , where p ( i,j ) o = exp( α ( i,j ) o ) (cid:80) o
Revised: · o ( x ( i ) ) , where p ( i,j ) o = exp( α ( i,j ) o ) (cid:80) o

Original: (cid:48)∈O exp( α ( i,j ) o (cid:48) ) .
Revised: (cid:48)∈O exp( α ( i,j ) o (cid:48) ) .

Original: The cell’s output is by default the result of a Conv1x1 op on the depthwise concatenation of all intermediate node features, although this may be changed (e.g. simply output last intermediate node’s features).
Revised: The cell’s output is by default the result of a Conv1x1 op on the depthwise concatenation of all intermediate node features, although this may be changed (e.g. simply output last intermediate node’s features).

Original: We denote the collection of all architecture variables a ( i,j ) o as α .
Revised: We denote the collection of all architecture variables a ( i,j ) o as α .

Original: Since the total op space O must contain the Zero and Skip
Revised: Since the total op space O must contain the Zero and Skip

Original: Connection ops, we mainly refer to the main op space as O base where O = O base ∪ { Zero, Skip } .
Revised: Connection ops, we mainly refer to the main op space as O base where O = O base ∪ { Zero, Skip } .

Original: In the standard optimization framework for SL where there is the notion of a validation set, the learning procedure consists of a bilevel optimization problem:
Revised: In the standard optimization framework for SL where there is the notion of a validation set, the learning procedure consists of a bilevel optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where L ( θ, α ) is a loss function deﬁned with respect to the supernet.
Revised: where L ( θ, α ) is a loss function deﬁned with respect to the supernet.

Original: α ∗ is then discretized into a ﬁnal cell, by representing each edge ( i, j ) with the highest softmax weighted op arg max [Equation]
Revised: α ∗ is then discretized into a ﬁnal cell, by representing each edge ( i, j ) with the highest softmax weighted op arg max o ∈O ,o (cid:54) =

Original: = [Equation] and then retaining only the top  2 incoming edges for each intermediate node.
Revised: zero p ( i,j ) o and then retaining only the top K = 2 incoming edges for each intermediate node.

Original: [Figure]
Revised: [Figure]

Original: Following common NAS practices (Zoph et al., 2018; Pham et al., 2018; Zoph & Le, 2016), we construct our supernet by stacking both normal ( N times) and reduction cells ( R times) together into blocks , which are themselves also stacked together D times (see Figure 2).
Revised: Following common NAS practices (Zoph et al., 2018; Pham et al., 2018; Zoph & Le, 2016), we construct our supernet by stacking both normal ( N times) and reduction cells ( R times) together into blocks , which are themselves also stacked together D times (see Figure 2).

Original: Normal cells preserve the input shape on output, while reduction cells apply a stride of 2 to reduce height and width immediately on the input.
Revised: Normal cells preserve the input shape on output, while reduction cells apply a stride of 2 to reduce height and width immediately on the input.

Original: Each block can have its own associated convolutional channel depth which is used throughout all cells in the block.
Revised: Each block can have its own associated convolutional channel depth which is used throughout all cells in the block.

Original: Another common NAS practice we adopt is to train a smaller supernet (i.e. depth 16) to reduce computation time, but evaluate ﬁnal discretized cells on larger models (i.e. depth 64), with D = 3 layers for cheap large-scale runs and D
Revised: Another common NAS practice we adopt is to train a smaller supernet (i.e. depth 16) to reduce computation time, but evaluate ﬁnal discretized cells on larger models (i.e. depth 64), with D = 3 layers for cheap large-scale runs and D

Original: = 5 layers for ﬁne-grained A/B testing.
Revised: = 5 layers for ﬁne-grained A/B testing.

Original: 3. RL-DARTS
Revised: 3. RL-DARTS

Original: For notation, we use standard conventions in the
Revised: For notation, we use standard conventions in the

Original: RL literature: for a given MDP M , we denote s t , a t , r t as the state, action, reward respectively at time t .
Revised: RL literature: for a given MDP M , we denote s t , a t , r t as the state, action, reward respectively at time t .

Original: π is the policy and D is the replay buffer which contains collected trajectories τ =
Revised: π is the policy and D is the replay buffer which contains collected trajectories τ =

Original: ( s 0 , a 0 , r 0 , s 1 , . . . ) .
Revised: ( s 0 , a 0 , r 0 , s 1 , . . . ) .

Original: The goal of the RL algorithm is to maximize J ( π ) =
Revised: The goal of the RL algorithm is to maximize J ( π ) =

Original: E τ ∼ π (cid:104)(cid:80) t ≥ 0 r t (cid:105) , the expected cumulative reward when using policy π .
Revised: E τ ∼ π (cid:104)(cid:80) t ≥ 0 r t (cid:105) , the expected cumulative reward when using policy π .

Original: In most RL algorithms, there is the notion of a neural network torso, or encoder which maps a state s (or sequence τ ) to a feature vector, which may be projected to correct action dimensions, behind the scenes in forming π .
Revised: In most RL algorithms, there is the notion of a neural network torso, or encoder which maps a state s (or sequence τ ) to a feature vector, which may be projected to correct action dimensions, behind the scenes in forming π .

Original: In the DARTS case, we use a supernet for this encoder instead, which leads to a policy denoted as π θ,α .Supernet training max ,α J
Revised: In the DARTS case, we use a supernet for this encoder instead, which leads to a policy denoted as π θ,α .

Original: π θ,α ) .

Original: Discretization

Original: Discretize α

Original: to construct evaluation policy π φ,δ

Original: α ∗ ) .

Original: Evaluation: Report max φ J ( π φ,δ ( α ∗ ) ) .

Original: The original bilevel optimization framework (Eq.
Revised: The original bilevel optimization framework (Eq.

Original: 1) is notoriously difﬁcult and unstable, sometimes requiring special techniques (Liu et al., 2018;
Revised: 1) is notoriously difﬁcult and unstable, sometimes requiring special techniques (Liu et al., 2018;

Original: Dong & Yang, 2019; Hundt et al., 2019;
Revised: Dong & Yang, 2019; Hundt et al., 2019;

Original: Li et al., 2020; Chu et al., 2020b;a;c; Liang et al., 2019; Wang et al., 2021b) speciﬁc to SL optimization.
Revised: Li et al., 2020; Chu et al., 2020b;a;c; Liang et al., 2019; Wang et al., 2021b) speciﬁc to SL optimization.

Original: To avoid such confounding factors, especially when DARTS has not yet been fully validated in RL, we may simply optimize sample complexity and raw training performance which are common metrics in RL.
Revised: To avoid such confounding factors, especially when DARTS has not yet been fully validated in RL, we may simply optimize sample complexity and raw training performance which are common metrics in RL.

Original: We thus joint optimize both θ and α :
Revised: We thus joint optimize both θ and α :

Original: [Equation]
Revised: [Equation]

Original: which leads to an optimal α ∗ .
Revised: which leads to an optimal α ∗ .

Original: With slight abuse of notation for convenience, denote π φ,δ ( α ∗ ) as the policy using the discretized cell δ ( α ∗ ) and L δ ( α ∗ ) ( φ ) to be the corresponding loss function, now dependent on only the new sparse weights φ .
Revised: With slight abuse of notation for convenience, denote π φ,δ ( α ∗ ) as the policy using the discretized cell δ ( α ∗ ) and L δ ( α ∗ ) ( φ ) to be the corresponding loss function, now dependent on only the new sparse weights φ .

Original: Note that under standard NAS procedure, we must then evaluate the discretized cell via training from scratch again to obtain the ﬁnal reward:
Revised: Note that under standard NAS procedure, we must then evaluate the discretized cell via training from scratch again to obtain the ﬁnal reward:

Original: [Equation]
Revised: [Equation]

Original: This can be concisely summarized in Algorithm 1, although we remind the reader that L ( · ) is deﬁned with respect to replay buffer D and is not exactly equivalent to optimizing J ( · ) .
Revised: This can be concisely summarized in Algorithm 1, although we remind the reader that L ( · ) is deﬁned with respect to replay buffer D and is not exactly equivalent to optimizing J ( · ) .

Original: Unless speciﬁed, we by default use consistent hyperparameters (found in Appendix F) for all comparisons found inside a ﬁgure, although learning rate and minibatch size may be altered when training models of different sizes due to GPU memory limits.
Revised: Unless speciﬁed, we by default use consistent hyperparameters (found in Appendix F) for all comparisons found inside a ﬁgure, although learning rate and minibatch size may be altered when training models of different sizes due to GPU memory limits.

Original: Thus, even though RL is commonly sensitive to hyperparameters (Zhang et al., 2021), we ﬁnd that once a pre-existing RL baseline has already been setup, incorporating
Revised: Thus, even though RL is commonly sensitive to hyperparameters (Zhang et al., 2021), we ﬁnd that once a pre-existing RL baseline has already been setup, incorporating

Original: DARTS requires no extra cost in tuning, as evidence of its ease-of-use.
Revised: DARTS requires no extra cost in tuning, as evidence of its ease-of-use.

Original:  E XPERIMENTS 4. E XPERIMENT S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENT S ETUP

Original: We perform key experiments to understand the behavior of RL-DARTS, and answer the following main questions in our analysis:
Revised: We perform key experiments to understand the behavior of RL-DARTS, and answer the following main questions in our analysis:

Original: Does the supernet train end-to-end at all or will it fail to even initially collect good online data?
Revised: Does the supernet train end-to-end at all or will it fail to even initially collect good online data?

Original: Does α converge towards a sparse solution over time, and what is the computational cost?
Revised: Does α converge towards a sparse solution over time, and what is the computational cost?

Original: Even if the supernet trains, do the corresponding discrete cells also improve in evaluation performance throughout α ’s training, despite common integrality gap issues?
Revised: Even if the supernet trains, do the corresponding discrete cells also improve in evaluation performance throughout α ’s training, despite common integrality gap issues?

Original: What kinds of failure modes occur?
Revised: What kinds of failure modes occur?

Original: How do the ﬁnal discrete cells perform at evaluation, and how does RL-DARTS compare against random search?
Revised: How do the ﬁnal discrete cells perform at evaluation, and how does RL-DARTS compare against random search?

Original: Can a large-scale joint optimization over multiple diverse environments lead to a single transferrable cell?
Revised: Can a large-scale joint optimization over multiple diverse environments lead to a single transferrable cell?

Original: In order to study the behavior of RL-DARTS, we use the Procgen benchmark (Cobbe et al., 2019; 2020), which readily provides pre-existing baselines for both PPO (Schulman et al., 2017) and
Revised: In order to study the behavior of RL-DARTS, we use the Procgen benchmark (Cobbe et al., 2019; 2020), which readily provides pre-existing baselines for both PPO (Schulman et al., 2017) and

Original: Rainbow DQN (Hessel et al., 2018) algorithms, over a diverse selection of 16 games, each with inﬁnite levels to benchmark RL-DARTS in large data regimes where episodes may drastically change, relevant for generalization.
Revised: Rainbow DQN (Hessel et al., 2018) algorithms, over a diverse selection of 16 games, each with inﬁnite levels to benchmark RL-DARTS in large data regimes where episodes may drastically change, relevant for generalization.

Original: We use the standard “easy” difﬁculty commonly benchmarked by a variety of other works (Raileanu et al., 2020; Raileanu & Fergus, 2021; Parker-Holder et al., 2021b).
Revised: We use the standard “easy” difﬁculty commonly benchmarked by a variety of other works (Raileanu et al., 2020; Raileanu & Fergus, 2021; Parker-Holder et al., 2021b).

Original: We also use the default IMPALA-CNN architecture (Espeholt et al., 2018) as a strong hand-designed baseline when comparing both supernet and discrete cell performances.
Revised: We also use the default IMPALA-CNN architecture (Espeholt et al., 2018) as a strong hand-designed baseline when comparing both supernet and discrete cell performances.

Original: IMPALA-CNN can be seen as speciﬁc instance of the stacked cell design in Subsection 3.1 and Figure 2, where its “Reduction
Revised: IMPALA-CNN can be seen as speciﬁc instance of the stacked cell design in Subsection 3.1 and Figure 2, where its “Reduction

Original: Cell” consists of a Conv3x3 and MaxPool3x3 (Stride 2) with R = 1 and its “Normal Cell” consists of a residual layer with Conv3x3’s and ReLU’s, with N = 2 .
Revised: Cell” consists of a Conv3x3 and MaxPool3x3 (Stride 2) with R = 1 and its “Normal Cell” consists of a residual layer with Conv3x3’s and ReLU’s, with N = 2 .

Original: More details can be found in the original paper (Espeholt et al., 2018).
Revised: More details can be found in the original paper (Espeholt et al., 2018).

Original: As for RL-DARTS’s search space, we consider the following base ops O base , corresponding algorithms, and ( N, R, I ) :
Revised: As for RL-DARTS’s search space, we consider the following base ops O base , corresponding algorithms, and ( N, R, I ) :

Original: • Classic + PPO: O base,N = { Conv3x3+ReLU, Conv5x5+ReLU, Dilated3x3+ReLU, Dilated5x5+ReLU } for normal ops and O base,R = { Conv3x3, MaxPool3x3, AveragePool3x3 } for reduction ops, which is standard in supervised learning (Liu et al., 2019; Zoph et al., 2018; Pham et al., 2018).
Revised: • Classic + PPO: O base,N = { Conv3x3+ReLU, Conv5x5+ReLU, Dilated3x3+ReLU, Dilated5x5+ReLU } for normal ops and O base,R = { Conv3x3, MaxPool3x3, AveragePool3x3 } for reduction ops, which is standard in supervised learning (Liu et al., 2019; Zoph et al., 2018; Pham et al., 2018).

Original: We use ( N, R, I ) =
Revised: We use ( N, R, I ) =

Original: (1 , 1 , 4) to be comparable to the baseline.
Revised: (1 , 1 , 4) to be comparable to the baseline.

Original: • Micro + Rainbow: O base,N = { Conv3x3, ReLU, Tanh } , a more ﬁne-grained and novel search space which has not been used previously in SL.
Revised: • Micro + Rainbow: O base,N = { Conv3x3, ReLU, Tanh } , a more ﬁne-grained and novel search space which has not been used previously in SL.

Original: The inclusion of Tanh is motivated by its use previously for continuous control architectures (Salimans et al., 2017; Song et al., 2020).
Revised: The inclusion of Tanh is motivated by its use previously for continuous control architectures (Salimans et al., 2017; Song et al., 2020).

Original: In order to avoid hidden confounding effects from the reduction cell when displaying the normal cell in ﬁgures, we use ( N, R, I ) =
Revised: In order to avoid hidden confounding effects from the reduction cell when displaying the normal cell in ﬁgures, we use ( N, R, I ) =

Original: (2 , 0 , 4) , where reduction ops default to IMPALA-CNN’s.
Revised: (2 , 0 , 4) , where reduction ops default to IMPALA-CNN’s.

Original: 4. T RAINING THE S UPERNET
Revised: 4. T RAINING THE S UPERNET

Original: In SL, it is expected to see the supernet’s asymptotic validation accuracy eventually reach a sufﬁcient threshold, comparable to (although slightly worse than) a discrete cell’s validation accuracy (Liu et al., 2019; Li et al., 2020).
Revised: In SL, it is expected to see the supernet’s asymptotic validation accuracy eventually reach a sufﬁcient threshold, comparable to (although slightly worse than) a discrete cell’s validation accuracy (Liu et al., 2019; Li et al., 2020).

Original: The question is, can an RL supernet achieve the same result for training reward?
Revised: The question is, can an RL supernet achieve the same result for training reward?

Original: As mentioned earlier in the introduction, this is unclear as a negative feedback loop can occur where initially, a suboptimal supernet collects poor data for training, which leads to an even worse supernet, as α may potentially upweight suboptimal ops.
Revised: As mentioned earlier in the introduction, this is unclear as a negative feedback loop can occur where initially, a suboptimal supernet collects poor data for training, which leads to an even worse supernet, as α may potentially upweight suboptimal ops.

Original: However, we ﬁnd that training the supernet end-to-end works effectively, even with minimal hyperparameter tuning for both PPO (Figure 3) and Rainbow (Figure 4), although PPO may take longer (50M steps) to reach comparable scores to IMPALA-CNN while Rainbow only needs 25M steps.
Revised: However, we ﬁnd that training the supernet end-to-end works effectively, even with minimal hyperparameter tuning for both PPO (Figure 3) and Rainbow (Figure 4), although PPO may take longer (50M steps) to reach comparable scores to IMPALA-CNN while Rainbow only needs 25M steps.

Original: However, this is a non-issue during discretization, as the α has already converged to a sparse solution (via softmax weight magnitudes) by 25M steps on PPO.
Revised: However, this is a non-issue during discretization, as the α has already converged to a sparse solution (via softmax weight magnitudes) by 25M steps on PPO.

Original: [Table]
Revised: [Table]

Original: Train)
Revised: Train)

Original: [Table]
Revised: [Table]

Original: Appendix
Revised: Appendix

Original: Interestingly, on Dodgeball with PPO, we ﬁnd that the process strongly downweights all base ops except for the standard Conv3x3+ReLU, with 5x5 ops possessing the lowest softmax weights.
Revised: Interestingly, on Dodgeball with PPO, we ﬁnd that the process strongly downweights all base ops except for the standard Conv3x3+ReLU, with 5x5 ops possessing the lowest softmax weights.

Original: Conveniently, this provides an opportunity to understand whether α downweights suboptimal ops throughout training.
Revised: Conveniently, this provides an opportunity to understand whether α downweights suboptimal ops throughout training.

Original: We conﬁrm this result in Table 1 by performing evaluations over standard IMPALA-CNN cells using either purely 3x3 or 5x5 convolutions for the whole network, and demonstrating that the 3x3 setting outperforms the 5x5 setting (especially in limited data, e.g.
Revised: We conﬁrm this result in Table 1 by performing evaluations over standard IMPALA-CNN cells using either purely 3x3 or 5x5 convolutions for the whole network, and demonstrating that the 3x3 setting outperforms the 5x5 setting (especially in limited data, e.g.

Original: 200-level training/test regime), suggesting the signaling ability of α on op choice.
Revised: 200-level training/test regime), suggesting the signaling ability of α on op choice.

Original: Although the
Revised: Although the

Original: Rainbow (25M Steps)
Revised: Rainbow (25M Steps)

Original: PPO (50M Steps)
Revised: PPO (50M Steps)

Original: [Table]
Revised: [Table]

Original: Trainable
Revised: Trainable

Original: [Table]
Revised: [Table]

Original: Uniform
Revised: Uniform

Original: [Table]
Revised: [Table]

Original: With ReLU
Revised: With ReLU

Original: No ReLU
Revised: No ReLU

Original: [Table]
Revised: [Table]

Original: training abilities of supernets in RL have been established, an important contrasting question now surfaces: Can any supernet train?
Revised: training abilities of supernets in RL have been established, an important contrasting question now surfaces: Can any supernet train?

Original: The supernet possesses an incredibly dense set of weights, and thus one might wonder whether trainability occurs with any search space or settings.
Revised: The supernet possesses an incredibly dense set of weights, and thus one might wonder whether trainability occurs with any search space or settings.

Original: We answer in the negative, where a poorly designed supernet can fail.
Revised: We answer in the negative, where a poorly designed supernet can fail.

Original: To show this clearly, we remove all
Revised: To show this clearly, we remove all

Original: ReLU nonlinearities from the “Classic” search space used for PPO (although ReLUs still exist in ﬁnal MLP projections), as well as simply freeze α to be uniform for Rainbow, and ﬁnd both cases produce poor training in Table 2.
Revised: ReLU nonlinearities from the “Classic” search space used for PPO (although ReLUs still exist in ﬁnal MLP projections), as well as simply freeze α to be uniform for Rainbow, and ﬁnd both cases produce poor training in Table 2.

Original: Thus, the supernet in RL provides important search signals in terms of reward and α during training, especially on the quality of a search space.
Revised: Thus, the supernet in RL provides important search signals in terms of reward and α during training, especially on the quality of a search space.

Original: We further provide computational efﬁciency metrics in Table 3, where we ﬁnd that the practical wall-clock time required for training the supernet (i.e. the search cost) is very comparable with
Revised: We further provide computational efﬁciency metrics in Table 3, where we ﬁnd that the practical wall-clock time required for training the supernet (i.e. the search cost) is very comparable with

Original: DARTS in SL (Liu et al., 2019), requiring only a few GPU days.
Revised: DARTS in SL (Liu et al., 2019), requiring only a few GPU days.

Original: We do note that unlike SL where the vast majority of the cost is due to the network, RL time cost is partially based on non-network factors such as environment simulation, and thus wall-clock times may change depending on speciﬁc implementation.
Revised: We do note that unlike SL where the vast majority of the cost is due to the network, RL time cost is partially based on non-network factors such as environment simulation, and thus wall-clock times may change depending on speciﬁc implementation.

Original: [Table]
Revised: [Table]

Original: 4. D ISCRETE C ELL I MPROVEMENT
Revised: 4. D ISCRETE C ELL I MPROVEMENT

Original: Due to the existence of integrality gaps in SL between a continuously relaxed supernet and its discrete cell (Wang et al., 2021b;a), this also raises the same question in RL.
Revised: Due to the existence of integrality gaps in SL between a continuously relaxed supernet and its discrete cell (Wang et al., 2021b;a), this also raises the same question in RL.

Original: We demonstrate that even using the default δ discretization from (Liu et al., 2019) leads to both qualitative and quantitative improvements, although we leave potential improvements to future works.
Revised: We demonstrate that even using the default δ discretization from (Liu et al., 2019) leads to both qualitative and quantitative improvements, although we leave potential improvements to future works.

Original: In terms of qualitative improvement , we display discretized cells when training a supernet with
Revised: In terms of qualitative improvement , we display discretized cells when training a supernet with

Original: Rainbow on the Starpilot environment in Figure 5.
Revised: Rainbow on the Starpilot environment in Figure 5.

Original: As we can see, the earlier cell consists of only linear operators is clearly a poor design in comparison to the later cell.
Revised: As we can see, the earlier cell consists of only linear operators is clearly a poor design in comparison to the later cell.

Original: We provide more extensive cell evolution ablations for PPO in Figures 14, 15 in Appendix C.2, displaying more sophisticated yet still interpretable changes in cell topology.
Revised: We provide more extensive cell evolution ablations for PPO in Figures 14, 15 in Appendix C.2, displaying more sophisticated yet still interpretable changes in cell topology.

Original: For quantitative improvement , as α changes during supernet training, so do the outputs of the discretization procedure on α .
Revised: For quantitative improvement , as α changes during supernet training, so do the outputs of the discretization procedure on α .

Original: We collect all distinct discrete cells { δ ( α 1 ) , δ ( α 2 ) , . . .
Revised: We collect all distinct discrete cells { δ ( α 1 ) , δ ( α 2 ) , . . .

Original: } into a sequence, and evaluate each cell’s performance max φ J ( π φ,δ ( a i ) )
Revised: } into a sequence, and evaluate each cell’s performance max φ J ( π φ,δ ( a i ) )

Original: ∀ i via training from scratch for 25M steps, displayed in Figure 6.
Revised: ∀ i via training from scratch for 25M steps, displayed in Figure 6.

Original: As we can see, the performance generally improves over time, indicating that supernet optimization selects better cells.
Revised: As we can see, the performance generally improves over time, indicating that supernet optimization selects better cells.

Original: However, we ﬁnd that such behavior can be environment-dependent, as some environments possess less monotonic evaluation curves (see Figure 16 in Appendix
Revised: However, we ﬁnd that such behavior can be environment-dependent, as some environments possess less monotonic evaluation curves (see Figure 16 in Appendix

Original: C).
Revised: C).

Original: One reason why a discretized cell may underperform is if its corresponding supernet fails to learn .
Revised: One reason why a discretized cell may underperform is if its corresponding supernet fails to learn .

Original: One simple reason is due to the inherent sensitivity and variance in RL training.
Revised: One simple reason is due to the inherent sensitivity and variance in RL training.

Original: For example, in
Revised: For example, in

Original: Figure 7, we see that when using Rainbow, bad training setups (ex: setting D too high) can create an unlearnable “Supernet 2”, subsequently leading to an unlearnable corresponding discrete cell, while the inverse occurs with “Supernet 1” and its own discrete cell.
Revised: Figure 7, we see that when using Rainbow, bad training setups (ex: setting D too high) can create an unlearnable “Supernet 2”, subsequently leading to an unlearnable corresponding discrete cell, while the inverse occurs with “Supernet 1” and its own discrete cell.

Original: The discrete cells’ rewards can also be qualitatively explained (ex: see Figure 7’s caption).
Revised: The discrete cells’ rewards can also be qualitatively explained (ex: see Figure 7’s caption).

Original: We perform further studies in Appendix
Revised: We perform further studies in Appendix

Original: C.3, Figure 17, showing that supernet and discrete cell rewards are indeed correlated, but also have environment-dependent integrality gaps, suggesting that search quality can be improved via both better supernet training (Raileanu et al., 2020; Zhang et al., 2021), as well as better discretization procedures (Liang et al., 2019; Wang et al., 2021b).
Revised: C.3, Figure 17, showing that supernet and discrete cell rewards are indeed correlated, but also have environment-dependent integrality gaps, suggesting that search quality can be improved via both better supernet training (Raileanu et al., 2020; Zhang et al., 2021), as well as better discretization procedures (Liang et al., 2019; Wang et al., 2021b).

Original: [Figure]
Revised: [Figure]

Original: 4. S INGLE G AME E VALUATIONS
Revised: 4. S INGLE G AME E VALUATIONS

Original: Since the results above establish RL-DARTS’s search capability, we now focus on comparisons between discrete architectures and baselines.
Revised: Since the results above establish RL-DARTS’s search capability, we now focus on comparisons between discrete architectures and baselines.

Original: We ﬁrst present results in the standard scenario, where
Revised: We ﬁrst present results in the standard scenario, where

Original: RL-DARTS is applied to each game independently, discretized at 25M steps.
Revised: RL-DARTS is applied to each game independently, discretized at 25M steps.

Original: [Table]
Revised: [Table]

Original: We use random search as an important baseline to gauge the search space’s difﬁculty (ex: “Classic” total size is 4 × 10 11 , see Appendix G.1).
Revised: We use random search as an important baseline to gauge the search space’s difﬁculty (ex: “Classic” total size is 4 × 10 11 , see Appendix G.1).

Original: For fair comparison, we ensure total wall-clock time (with same hardware) stays equal, as common in (Liu et al., 2019).
Revised: For fair comparison, we ensure total wall-clock time (with same hardware) stays equal, as common in (Liu et al., 2019).

Original: Since in Table 3, a PPO supernet takes 2.5x longer to reach 25M steps, this is rounded to a random search budget of 3 cells to be trained with depths 16 × 3 for 25M steps.
Revised: Since in Table 3, a PPO supernet takes 2.5x longer to reach 25M steps, this is rounded to a random search budget of 3 cells to be trained with depths 16 × 3 for 25M steps.

Original: The best of the 3 cells is used for full evaluation.
Revised: The best of the 3 cells is used for full evaluation.

Original: In Table 4, on average, random search underperforms signiﬁcantly, which also shows IMPALA-CNN is a strong hand-designed baseline.
Revised: In Table 4, on average, random search underperforms signiﬁcantly, which also shows IMPALA-CNN is a strong hand-designed baseline.

Original: However, in Figure 8, RL-DARTS is capable of ﬁnding architectures from scratch which outperform
Revised: However, in Figure 8, RL-DARTS is capable of ﬁnding architectures from scratch which outperform

Original: IMPALA-CNN on few environments such as Plunder and Heist, while maintaining competitive performance on others.
Revised: IMPALA-CNN on few environments such as Plunder and Heist, while maintaining competitive performance on others.

Original: On the Rainbow side, in Figure 9, we found that in certain cases, even 100 unique random cells underperform against
Revised: On the Rainbow side, in Figure 9, we found that in certain cases, even 100 unique random cells underperform against

Original: RL-DARTS!
Revised: RL-DARTS!

Original: 4. T RANSFER E VALUATION
Revised: 4. T RANSFER E VALUATION

Original: Instead of performing the RL-DARTS by joint training a supernet across inﬁnite levels within only one game , one may ask if it is possible to joint train a supernet across all games over all inﬁnite levels to ﬁnd a single transferrable cell , which may be of interest to practitioners seeking to ﬁnd a common architecture general to a broad class of RL problems.
Revised: Instead of performing the RL-DARTS by joint training a supernet across inﬁnite levels within only one game , one may ask if it is possible to joint train a supernet across all games over all inﬁnite levels to ﬁnd a single transferrable cell , which may be of interest to practitioners seeking to ﬁnd a common architecture general to a broad class of RL problems.

Original: We conﬁrm this approach using Rainbow, where a learner performs gradient updates over actor replay data (with normalized rewards) from all 16 games.
Revised: We conﬁrm this approach using Rainbow, where a learner performs gradient updates over actor replay data (with normalized rewards) from all 16 games.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our research has demonstrated that in a minimally invasive way, it is entirely possible to apply DARTS to RL, a completely different optimization paradigm than SL, which DARTS was originally proposed for.
Revised: Our research has demonstrated that in a minimally invasive way, it is entirely possible to apply DARTS to RL, a completely different optimization paradigm than SL, which DARTS was originally proposed for.

Original: We believe this work can open new doors to both NAS and RL communities alike.
Revised: We believe this work can open new doors to both NAS and RL communities alike.

Original: Especially for theorists and analysts of DARTS, RL is a completely new frontier for which to understand softmax routing and continuous relaxation techniques.
Revised: Especially for theorists and analysts of DARTS, RL is a completely new frontier for which to understand softmax routing and continuous relaxation techniques.

Original: Previous DARTS techniques (Wang et al., 2021b;
Revised: Previous DARTS techniques (Wang et al., 2021b;

Original: Chu et al., 2020a; Liang et al., 2019) may be tested in the RL setting, and new RL-speciﬁc ways of discretization and training may also be proposed, increasing RL-DARTS’s utility even more.
Revised: Chu et al., 2020a; Liang et al., 2019) may be tested in the RL setting, and new RL-speciﬁc ways of discretization and training may also be proposed, increasing RL-DARTS’s utility even more.

Original: For
Revised: For

Original: RL practitioners, RL-DARTS could be important to ﬁnding better and more efﬁcient (Cai et al., 2019)
Revised: RL practitioners, RL-DARTS could be important to ﬁnding better and more efﬁcient (Cai et al., 2019)

Original: architectures for large-scale robotics (James et al., 2019; Akinola et al., 2021), transferable architectures in ofﬂine RL (Levine et al., 2020), as well as RNNs for memory (Pritzel et al., 2017;
Revised: architectures for large-scale robotics (James et al., 2019; Akinola et al., 2021), transferable architectures in ofﬂine RL (Levine et al., 2020), as well as RNNs for memory (Pritzel et al., 2017;

Original: Fortunato et al., 2019; Kapturowski et al., 2019) and adaptation (Duan et al., 2016; Wang et al., 2017).
Revised: Fortunato et al., 2019; Kapturowski et al., 2019) and adaptation (Duan et al., 2016; Wang et al., 2017).

Original:  S TATEMENTS
Revised:  S TATEMENTS

Original: Ethics Statement: The only relevant potential concerns in our work occur normally with general NAS methods, which can sacriﬁce model interpretability in order to achieve higher objectives.
Revised: Ethics Statement: The only relevant potential concerns in our work occur normally with general NAS methods, which can sacriﬁce model interpretability in order to achieve higher objectives.

Original: For the ﬁeld of RL speciﬁcally, this may warrant more attention in AI safety when used for real world robotic pipelines.
Revised: For the ﬁeld of RL speciﬁcally, this may warrant more attention in AI safety when used for real world robotic pipelines.

Original: Furthermore, as with any NAS research, the initial phase of discovery and experimentation may contribute to carbon emissions due to the computational costs of extensive tuning.
Revised: Furthermore, as with any NAS research, the initial phase of discovery and experimentation may contribute to carbon emissions due to the computational costs of extensive tuning.

Original: However, this is usually a means to an end, such as an efﬁcient search algorithm, which this paper proposes with no extra hardware costs.
Revised: However, this is usually a means to an end, such as an efﬁcient search algorithm, which this paper proposes with no extra hardware costs.

Original: Reproducibility Statement: We base our statement on the “NAS Best Practices Checklist” (Lindauer &
Revised: Reproducibility Statement: We base our statement on the “NAS Best Practices Checklist” (Lindauer &

Original: Hutter, 2019), recently released to improve reproducibility in neural architecture search.
Revised: Hutter, 2019), recently released to improve reproducibility in neural architecture search.

Original: In terms of raw reproducibility, we discuss the explicit hyperparameters in Appendix F, as well as search space details in Subsection 4.1.
Revised: In terms of raw reproducibility, we discuss the explicit hyperparameters in Appendix F, as well as search space details in Subsection 4.1.

Original: The original DARTS implementation is readily available across Github (e.g. https://github.com/quark0/darts ) as it is a highly popular algorithm.
Revised: The original DARTS implementation is readily available across Github (e.g. https://github.com/quark0/darts ) as it is a highly popular algorithm.

Original: Similarly, Procgen is publicly available ( https://github.com/openai/procgen ) along with its training code and details (Cobbe et al., 2020).
Revised: Similarly, Procgen is publicly available ( https://github.com/openai/procgen ) along with its training code and details (Cobbe et al., 2020).

Original: For our case, we used TF-Agents for PPO, available at https://www.tensorflow.
Revised: For our case, we used TF-Agents for PPO, available at https://www.tensorflow.

Original: org/agents and Acme for Rainbow, available at https://github.com/deepmind/acme .
Revised: org/agents and Acme for Rainbow, available at https://github.com/deepmind/acme .

Original: Since our algorithm is relatively simple to assemble given preexisting pipelines, we believe that our approach should require minimal effort to reproduce.
Revised: Since our algorithm is relatively simple to assemble given preexisting pipelines, we believe that our approach should require minimal effort to reproduce.

Original: In terms of comparing NAS methods, we indeed controlled for confounding factors by using the same hardware, as shown in Table 3.
Revised: In terms of comparing NAS methods, we indeed controlled for confounding factors by using the same hardware, as shown in Table 3.

Original: For ablation studies, we provided a large set of studies as seen throughout Subsections 4.
Revised: For ablation studies, we provided a large set of studies as seen throughout Subsections 4.

Original: and 4.3 as well as in the Appendix.
Revised: and 4.3 as well as in the Appendix.

Original: We also ensured the same evaluation protocol via making sure all depths remain the same ( 16 for supernets, 64 for discrete cells, and D = 3 layers for cheaper large-scale runs or D =for ﬁne-grained A/B testing).
Revised: We also ensured the same evaluation protocol via making sure all depths remain the same ( 16 for supernets, 64 for discrete cells, and D = 3 layers for cheaper large-scale runs or D =for ﬁne-grained A/B testing).

Original: In terms of performance comparisons over time, we showed the performance of RL-DARTS’s discrete cells over the search procedure / supernet training in Figure 6.
Revised: In terms of performance comparisons over time, we showed the performance of RL-DARTS’s discrete cells over the search procedure / supernet training in Figure 6.

Original: We extensively covered the comparisons against random search in Subsection 4.4, over all 16 games as well as a large scale 100 random cell comparison in Figure 9.
Revised: We extensively covered the comparisons against random search in Subsection 4.4, over all 16 games as well as a large scale 100 random cell comparison in Figure 9.

Original: For seeded runs, as standard in RL, we performed 3-seeded runs for all experiments.
Revised: For seeded runs, as standard in RL, we performed 3-seeded runs for all experiments.

Original: In terms of hyperparameter tuning, we discuss this in Appendix F and also in Subsection 3.2.
Revised: In terms of hyperparameter tuning, we discuss this in Appendix F and also in Subsection 3.2.

Original: The time reported for all components of the NAS method can be found in Table and we explicitly reported all of the details of our experiments throughout the paper.
Revised: The time reported for all components of the NAS method can be found in Table and we explicitly reported all of the details of our experiments throughout the paper.

Original: A PPENDIX A R EBUTTAL A. T RAINING P ROCEDURE (E XTENDED )
Revised: A PPENDIX A R EBUTTAL A. T RAINING P ROCEDURE (E XTENDED )

Original: Below are PPO (Schulman et al., 2017) and Rainbow-DQN (Hessel et al., 2018) RL-DARTS variants.
Revised: Below are PPO (Schulman et al., 2017) and Rainbow-DQN (Hessel et al., 2018) RL-DARTS variants.

Original: Exact loss deﬁnitions and data collection procedures can be found in their respective papers.
Revised: Exact loss deﬁnitions and data collection procedures can be found in their respective papers.

Original: Algorithm 2:
Revised: Algorithm 2:

Original: Supernet
Revised: Supernet

Original: training:
Revised: training:

Original: Setup supernet encoder f θ e ,α with weights θ e .
Revised: Setup supernet encoder f θ e ,α with weights θ e .

Original: Initialize policy and value head projection weights W π ∈ R d, |A| , W v ∈ R d, 1 .
Revised: Initialize policy and value head projection weights W π ∈ R d, |A| , W v ∈ R d, 1 .

Original: Collect all trainable weights θ = { θ e , W π , W v } .
Revised: Collect all trainable weights θ = { θ e , W π , W v } .

Original: Setup policy π θ,α ( s ) ∼ softmax ( W π · f θ e ,α ( s )) .
Revised: Setup policy π θ,α ( s ) ∼ softmax ( W π · f θ e ,α ( s )) .

Original: Setup value function V θ,α ( s ) =
Revised: Setup value function V θ,α ( s ) =

Original: W v · f θ e ,α ( s ) .
Revised: W v · f θ e ,α ( s ) .

Original: Deﬁne standard PPO loss L ( θ, α ) using π θ,α and V θ,α .
Revised: Deﬁne standard PPO loss L ( θ, α ) using π θ,α and V θ,α .

Original: Perform PPO training via collecting data from π θ,α and SGD with ∇ θ,α L ( θ, α ) .
Revised: Perform PPO training via collecting data from π θ,α and SGD with ∇ θ,α L ( θ, α ) .

Original: Collect α ∗ from previous training procedure.
Revised: Collect α ∗ from previous training procedure.

Original: Discretization:
Revised: Discretization:

Original: Let δ ( α ∗ ) be the discrete cell constructed via Algorithm 4.
Revised: Let δ ( α ∗ ) be the discrete cell constructed via Algorithm 4.

Original: Evaluation:
Revised: Evaluation:

Original: Setup discretized cell encoder f φ e ,δ ( α ∗ ) .
Revised: Setup discretized cell encoder f φ e ,δ ( α ∗ ) .

Original: Initialize policy and value head projection weights W (cid:48)
Revised: Initialize policy and value head projection weights W (cid:48)

Original: π ∈ R d, |A| , W (cid:48) v ∈ R d, 1 .
Revised: π ∈ R d, |A| , W (cid:48) v ∈ R d, 1 .

Original: Collect all trainable weights φ = { φ e , W (cid:48) π , W (cid:48) v } .
Revised: Collect all trainable weights φ = { φ e , W (cid:48) π , W (cid:48) v } .

Original: Setup policy π φ,δ ( α ∗ ) ( s ) ∼ softmax ( W (cid:48) π · f φ e ,δ ( α ∗ ) ( s )) .
Revised: Setup policy π φ,δ ( α ∗ ) ( s ) ∼ softmax ( W (cid:48) π · f φ e ,δ ( α ∗ ) ( s )) .

Original: Setup value function V φ,δ ( α ∗ ) ( s ) =
Revised: Setup value function V φ,δ ( α ∗ ) ( s ) =

Original: W (cid:48) v · f φ e ,δ ( α ∗ ) ( s ) .
Revised: W (cid:48) v · f φ e ,δ ( α ∗ ) ( s ) .

Original: Deﬁne standard PPO loss L δ ( α ∗ ) ( φ ) using π φ,δ ( α ∗ ) and V φ,δ ( α ∗ ) .
Revised: Deﬁne standard PPO loss L δ ( α ∗ ) ( φ ) using π φ,δ ( α ∗ ) and V φ,δ ( α ∗ ) .

Original: Perform PPO training via collecting data from π φ,δ ( α ∗ ) and SGD with ∇ φ L δ ( α ∗ ) ( φ ) .
Revised: Perform PPO training via collecting data from π φ,δ ( α ∗ ) and SGD with ∇ φ L δ ( α ∗ ) ( φ ) .

Original: Report ﬁnal policy reward.
Revised: Report ﬁnal policy reward.

Original: Algorithm
Revised: Algorithm

Original: Supernet training:
Revised: Supernet training:

Original: Setup supernet encoder f θ e ,α with weights θ e .
Revised: Setup supernet encoder f θ e ,α with weights θ e .

Original: Initialize dueling network projections W v ∈ R d, 1 , W a ∈ R d, |A| .
Revised: Initialize dueling network projections W v ∈ R d, 1 , W a ∈ R d, |A| .

Original: Collect all trainable weights θ = { θ e , W v , W a } .
Revised: Collect all trainable weights θ = { θ e , W v , W a } .

Original: Setup value network V θ,α ( s )
Revised: Setup value network V θ,α ( s )

Original: =
Revised: =

Original: W v · f θ e ,α ( s ) .
Revised: W v · f θ e ,α ( s ) .

Original: Setup advantage network A θ,α ( s, a ) =
Revised: Setup advantage network A θ,α ( s, a ) =

Original: W a · f θ e ,α ( s ) .
Revised: W a · f θ e ,α ( s ) .

Original: Setup Q-network Q θ,α ( s, a ) =
Revised: Setup Q-network Q θ,α ( s, a ) =

Original: V θ,α ( s ) +
Revised: V θ,α ( s ) +

Original: A θ,α ( s, a ) − 1 |A| (cid:80) a (cid:48)
Revised: A θ,α ( s, a ) − 1 |A| (cid:80) a (cid:48)

Original: ∈A
Revised: ∈A

Original: A θ,α ( s, a (cid:48) ) .
Revised: A θ,α ( s, a (cid:48) ) .

Original: Deﬁne standard Rainbow loss L ( θ, α ) using Q θ,α .
Revised: Deﬁne standard Rainbow loss L ( θ, α ) using Q θ,α .

Original: Perform Rainbow training via collecting data from Q θ,α and SGD with ∇ θ,α L ( θ, α ) .
Revised: Perform Rainbow training via collecting data from Q θ,α and SGD with ∇ θ,α L ( θ, α ) .

Original: Collect α ∗ from previous training procedure.
Revised: Collect α ∗ from previous training procedure.

Original: DiscretizationLet δ ( α ∗ ) be the discrete cell constructed via Algorithm 4.
Revised: DiscretizationLet δ ( α ∗ ) be the discrete cell constructed via Algorithm 4.

Original: EvaluationSetupdiscretized cell encoder f φ e ,δ ( α ∗ ) with weights φ e .
Revised: EvaluationSetupdiscretized cell encoder f φ e ,δ ( α ∗ ) with weights φ e .

Original: Initialize dueling network projections W (cid:48) v ∈ R d, 1 , W (cid:48) a ∈ R d, |A|
Revised: Initialize dueling network projections W (cid:48) v ∈ R d, 1 , W (cid:48) a ∈ R d, |A|

Original: Collect all trainable weights φ = { φ e , W (cid:48) v , W (cid:48) a }
Revised: Collect all trainable weights φ = { φ e , W (cid:48) v , W (cid:48) a }

Original: Setup value network V φ,δ ( α ∗ ) ( s ) =
Revised: Setup value network V φ,δ ( α ∗ ) ( s ) =

Original: W (cid:48) v · f φ e ,δ ( α ∗ ) ( s )
Revised: W (cid:48) v · f φ e ,δ ( α ∗ ) ( s )

Original: Setup advantage network A φ,δ ( α ∗ ) ( s, a ) = W (cid:48)
Revised: Setup advantage network A φ,δ ( α ∗ ) ( s, a ) = W (cid:48)

Original: a · f φ e ,δ ( α ∗ ) ( s ) Setup Q-network
Revised: a · f φ e ,δ ( α ∗ ) ( s ) Setup Q-network

Original: Q φ,δ ( α ∗ ) ( s, a ) =
Revised: Q φ,δ ( α ∗ ) ( s, a ) =

Original: V φ,δ ( α ∗ ) (
Revised: V φ,δ ( α ∗ ) (

Original: [Equation]
Revised: [Equation]

Original: Deﬁne standard Rainbow loss L δ ( α ∗ ) ( φ ) using Q φ,δ ( α ∗ ) .
Revised: Deﬁne standard Rainbow loss L δ ( α ∗ ) ( φ ) using Q φ,δ ( α ∗ ) .

Original: Perform Rainbow training via collecting data from Q φ,δ ( α ∗ ) and SGD with ∇ φ L δ ( α ∗ ) ( φ ) .
Revised: Perform Rainbow training via collecting data from Q φ,δ ( α ∗ ) and SGD with ∇ φ L δ ( α ∗ ) ( φ ) .

Original: Report ﬁnal policy reward.
Revised: Report ﬁnal policy reward.

Original: Argmax:
Revised: Argmax:

Original: For ( i, j ) across all edges:
Revised: For ( i, j ) across all edges:

Original: Deﬁneedge strength w i,j = max o ∈O ,o (cid:54) =
Revised: Deﬁne

Original: zero p ( i,j ) o .
Revised: = = ,j ) o .

Original: Deﬁneedge op o ( i,j ) = arg max o ∈O ,o (cid:54) =
Revised: Deﬁne

Original: zero p ( i,j ) o .
Revised: = o o = ,j ) o .

Original: Prune:
Revised: Prune:

Original: For node j in all intermediate nodes:
Revised: For node j in all intermediate nodes:

Original: Sort input edge weights w i 1 ,j ≥ w i 2 ,j ≥ . . .

Original: ≥ w i k ,j . Retain only top 2 edges (

Original: , j i , j ) corresponding ops o i 1 ,j ) , o i 2 ,j in ﬁnal cell. Note that both RL-DARTS procedures can also be summarized in terms of raw code as simple one-line edits to the image encoder used (compressing the rest of the regular RL training pipeline code): def train(feature_encoder): """InitialRLalgorithm setup""" ... extra_variables = Wrap(feature_encoder) all_trainable_variables
Revised: Note that both RL-DARTS procedures can also be summarized in terms of raw code as simple one-line edits to the image encoder used (compressing the rest of the regular RL training pipeline code): def train(feature_encoder): """Initial RL algorithm setup""" ... extra_variables = Wrap(feature_encoder) all_trainable_variables

Original: = [feature_encoder.trainable_variables(), extra_variables] """Rest of RL algorithm setup""" ...
Revised: = [feature_encoder.trainable_variables(), extra_variables] """Rest of RL algorithm setup""" ...

Original: apply_gradients(loss, all_trainable_variables)
Revised: apply_gradients(loss, all_trainable_variables)

Original: Thus, the 3-step RL-DARTS procedure from Section 3.2 can be seen as:
Revised: Thus, the 3-step RL-DARTS procedure from Section 3.2 can be seen as:

Original: DARTSSuperNet = MakeSuperNet(ops, num_nodes) # Setup train(DARTSSuperNet)
Revised: DARTSSuperNet = MakeSuperNet(ops, num_nodes) # Setup train(DARTSSuperNet)

Original: # Supernet training
Revised: # Supernet training

Original: DiscretizedNet = DARTSSuperNet.discretize() # Discretization train (DiscretizedNet) #
Revised: DiscretizedNet = DARTSSuperNet.discretize() # Discretization train(DiscretizedNet)

Original: Evaluation A. P OSSIBLE I MPROVEMENTS TO RL-DARTS
Revised: Evaluation A. P OSSIBLE I MPROVEMENTS TO RL-DARTS

Original: These include:
Revised: These include:

Original: Discretization Changes: One may consider discretization based on the total reward J ( π θ,α ) , which may provide a better signal for the correct discrete architecture.
Revised: Discretization Changes: One may consider discretization based on the total reward J ( π θ,α ) , which may provide a better signal for the correct discrete architecture.

Original: This is due to the fact that the relative strengths of operation weights from α may not correspond to the best choices during discretization.
Revised: This is due to the fact that the relative strengths of operation weights from α may not correspond to the best choices during discretization.

Original: Wang et al., 2021b) considers iteratively pruning edges from the supernet based on maximizing validation accuracy changes.
Revised: (Wang et al., 2021b) considers iteratively pruning edges from the supernet based on maximizing validation accuracy changes.

Original: For RL, this would imply a variant of discretization dependent on multiple calculations of J ( π θ ∗ ,δ 1 ( α ∗ ) )
Revised: For RL, this would imply a variant of discretization dependent on multiple calculations of J ( π θ ∗ ,δ 1 ( α ∗ ) )

Original: − J ( π θ ∗ ,δ 2 ( α ∗ ) ) where θ ∗ consists the weights obtained during supernet training, as well as ﬁne-tuning
Revised: − J ( π θ ∗ ,δ 2 ( α ∗ ) ) where θ ∗ consists the weights obtained during supernet training, as well as ﬁne-tuning

Original: J ( π θ ∗ ,δ 1 ( α ∗ ) ) at every pruning step.
Revised: J ( π θ ∗ ,δ 1 ( α ∗ ) ) at every pruning step.

Original: These changes, in addition to the inherently noisy evaluations of J ( · ) , greatly increase the complexity of the discretization procedure, but are worth exploring in future work.
Revised: These changes, in addition to the inherently noisy evaluations of J ( · ) , greatly increase the complexity of the discretization procedure, but are worth exploring in future work.

Original: Changing the Loss / Regularization:
Revised: Changing the Loss / Regularization:

Original: Throughout this paper, we have found that vanilla
Revised: Throughout this paper, we have found that vanilla

Original: DARTS is able to train by simply optimizing α with respect to the loss, even though in principle, the loss is not strongly correlated to the actual reward in RL.
Revised: DARTS is able to train by simply optimizing α with respect to the loss, even though in principle, the loss is not strongly correlated to the actual reward in RL.

Original: Thus, it is curious to understand whether loss-based metrics or modiﬁcations may help improve RL-DARTS.
Revised: Thus, it is curious to understand whether loss-based metrics or modiﬁcations may help improve RL-DARTS.

Original: One such modiﬁcation is based on the observation that certain RL losses may not be required for training α .
Revised: One such modiﬁcation is based on the observation that certain RL losses may not be required for training α .

Original: In PPO, the entropy loss of π θ may not be necessary or useful for improving the search quality of α , and thus it may be better to perform a two step update by providing a different loss for α .
Revised: In PPO, the entropy loss of π θ may not be necessary or useful for improving the search quality of α , and thus it may be better to perform a two step update by providing a different loss for α .

Original: One may also consider searching for two separate encoders via two supernets, since both PPO and Rainbow feature separate networks, e.g. the policy π θ 1 ,αand value function V θ 2 ,α 2 for PPO and advantage function A θ 1 ,α 1 and value function V θ 2 ,α 2 for Rainbow.
Revised: One may also consider searching for two separate encoders via two supernets, since both PPO and Rainbow feature separate networks, e.g. the policy π θ 1 ,αand value function V θ 2 ,α 2 for PPO and advantage function A θ 1 ,α 1 and value function V θ 2 ,α 2 for Rainbow.

Original: Signaling Metrics and Early Stopping: Observing metrics throughout training allows for early stopping, which can reduce search cost and provide better discrete cells.
Revised: Signaling Metrics and Early Stopping: Observing metrics throughout training allows for early stopping, which can reduce search cost and provide better discrete cells.

Original: This includes metrics such as the strength of certain operation weights (Liang et al., 2019) as well as the
Revised: This includes metrics such as the strength of certain operation weights (Liang et al., 2019) as well as the

Original: Hessian with respect to α throughout training, i.e. ∇ 2 α L ( θ, α ) as found in (Zela et al., 2020).
Revised: Hessian with respect to α throughout training, i.e. ∇ 2 α L ( θ, α ) as found in (Zela et al., 2020).

Original: Furthermore, inspired by performance prediction methods (Mellor et al., 2020; Luo et al., 2018), one may analyze metrics such as the Jacobian Covariance, via the score deﬁned to be the − (cid:80)
Revised: Furthermore, inspired by performance prediction methods (Mellor et al., 2020; Luo et al., 2018), one may analyze metrics such as the Jacobian Covariance, via the score deﬁned to be the − (cid:80)

Original: Bi
Revised: Bi

Original: =1 (cid:2) log( σ i + ε )
Revised: =1 (cid:2) log( σ i + ε )

Original: + ( σ i + ε ) − 1 (cid:3) where ε = 10 − 5 is a stability constant and σ 1 ≤ . . .
Revised: + ( σ i + ε ) − 1 (cid:3) where ε = 10 − 5 is a stability constant and σ 1 ≤ . . .

Original: ≤ σ B are the eigenvalues of the correlation matrix corresponding to the Jacobian
Revised: ≤ σ B are the eigenvalues of the correlation matrix corresponding to the Jacobian

Original: J = (cid:104) ∂f∂s 1 , . .
Revised: J = (cid:104) ∂f∂s 1 , . .

Original: , ∂f∂s B (cid:105) T with B input images { s 1 , . . .
Revised: , ∂f∂s B (cid:105) T with B input images { s 1 , . . .

Original: , s B } .
Revised: , s B } .

Original: This metric/predictor has been found to be a strong signal for accuracy in SL NAS among many previous predictors (White et al., 2021).
Revised: This metric/predictor has been found to be a strong signal for accuracy in SL NAS among many previous predictors (White et al., 2021).

Original: However, for the RL case, just like the loss, these metrics must be deﬁned with respect to the current replay buffer D , and thus raises the question of what type of data is to be used for calculating these metrics.
Revised: However, for the RL case, just like the loss, the mentioned metrics must be deﬁned with respect to the current replay buffer D , and thus raises the question of what type of data is to be used for calculating these metrics.

Original: When using a reasonable variant where the data is collected from a pretrained policy, we ﬁnd that methods such as Jacobian Covariance do not provide meaningful feedback as shown in Figure R1.
Revised: When using a reasonable variant where the data is collected from a pretrained policy, we ﬁnd that methods such as Jacobian Covariance do not provide meaningful feedback as shown in Figure R1.

Original: Supernet Training:
Revised: Supernet Training:

Original: As seen from the results in Figure 7 (main body) and Appendix C.3, there is a correlation between supernet and discrete cell performances.
Revised: As seen from the results in Figure 7 (main body) and Appendix C.3, there is a correlation between supernet and discrete cell performances.

Original: However, this is affected by the environment used as well as integrality gaps between the continuous relaxation and discrete counterparts, and thus further exploration is needed before concluding that improving the supernet training leads to better discrete cell performances.
Revised: However, this is affected by the environment used as well as integrality gaps between the continuous relaxation and discrete counterparts, and thus further exploration is needed before concluding that improving the supernet training leads to better discrete cell performances.

Original: In any case, reasonable methods of improving the supernet can involve DARTS-agnostic modiﬁcations to the RL pipeline, including data augmentation (Kostrikov et al., 2020; Raileanu et al., 2020) as well as online hyperparameter tuning (Parker-Holder et al., 2020; 2021a).
Revised: In any case, reasonable methods of improving the supernet can involve DARTS-agnostic modiﬁcations to the RL pipeline, including data augmentation (Kostrikov et al., 2020; Raileanu et al., 2020) as well as online hyperparameter tuning (Parker-Holder et al., 2020; 2021a).

Original: Simple hyperparameter tuning (e.g. on the softmax temperature for calculating p ( i,j ) o ’s) also can be effective.
Revised: Simple hyperparameter tuning (e g. on the softmax temperature for calculating p ( i,j ) o ’s) also can be effective.

Revised: A. DM-C ONTROL WITH SAC (E XTRA B ENCHMARK ) R ESULTS

Revised: We also include results via using a Soft Actor Critic (SAC) (Haarnoja et al., 2018) variant of RL-DARTS on DM-Control (Tassa et al., 2018) with image-based observations resized to 64 × 64 with a frame-stacking of 3.

Revised: The code can be found in https://github.com/ google-research/pisac , where we disabled the predictive information loss to use only regular SAC.

Original:  B W HAT A FFECTS S UPERNET T RAINING ?
Revised: The baseline architecture is a 4-layer convolutional architecture found in https:// github.com/google-research/pisac/blob/master/pisac/encoders.py . Both our DARTS supernet and discrete cells use N = 3 , I = 4 , K = 1 using the “Micro” search space, with convolutional depths of 32 to remain fair to the baseline. Our results in Figure R2 thus also provide evidence of RL-DARTS’s ﬂexibility with yet a 3rd algorithm (SAC), as well as on continuous control, where discrete cells outperform the baseline. B W HAT A FFECTS S UPERNET T RAINING ?

Original: Given the positive training results we demonstrated in the main body of the paper, one may wonder, can any supernet, no matter how poorly designed or setup, still train well in the RL setting?
Revised: Given the positive training results we demonstrated in the main body of the paper, one may wonder, can any supernet, no matter how poorly designed or setup, still train well in the RL setting?

Original: If so, this would imply that the search method would not be producing meaningful, but instead, random signals.
Revised: If so, this would imply that the search method would not be producing meaningful, but instead, random signals.

Original: We refute this hypothesis by performing ablations over our supernet training in order to have a better understanding of what components affect its performance.
Revised: We refute this hypothesis by performing ablations over our supernet training in order to have a better understanding of what components affect its performance.

Original: We ultimately show that the search space and architecture variables play a very signiﬁcant role in its optimization, thus validating our method.
Revised: We ultimately show that the search space and architecture variables play a very signiﬁcant role in its optimization, thus validating our method.

Original: B. R OLE OF S EARCH S PACE
Revised: B. R OLE OF S EARCH S PACE

Original: We remove the ReLU nonlinearities from the ”Classic” search space, so that O base =
Revised: We remove the ReLU nonlinearities from the ”Classic” search space, so that O base =

Original: { Conv3x3,
Revised: { Conv3x3,

Original: Conv5x5, Dilated3x3, Dilated5x5 } and thus the DARTS cell consists of only linear operations.
Revised: Conv5x5, Dilated3x3, Dilated5x5 } and thus the DARTS cell consists of only linear operations.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 11, this leads to a dramatic decrease in supernet performance, providing evidence that the search space matters greatly.
Revised: 11, this leads to a dramatic decrease in supernet performance, providing evidence that the search space matters greatly.

Original: [Figure]
Revised: [Figure]

Revised: . U

Original: [Figure]
Revised: [Figure]

Original: We further demonstrate the importance of the architecture variables α on training.
Revised: We further demonstrate the importance of the architecture variables α on training.

Original: We see that in Fig.
Revised: We see that in Fig.

Original: 12, freezing α to be uniform throughout training makes the Rainbow agent unable to train at all.
Revised: 12, freezing α to be uniform throughout training makes the Rainbow agent unable to train at all.

Original: This suggests that it is crucial for α to properly route useful operations throughout the network
Revised: This suggests that it is crucial for α to properly route useful operations throughout the network

Original: C W HAT A FFECTS D ISCRETE C ELL P ERFORMANCE ? C. S OFTMAX W EIGHTS VS D ISCRETIZATION
Revised: C W HAT A FFECTS D ISCRETE C ELL P ERFORMANCE ? C. S OFTMAX W EIGHTS VS D ISCRETIZATION

Original: As seen from Figure 3 in the main body of the paper, the DARTS supernet strongly downweights
Revised: As seen from Figure 3 in the main body of the paper, the DARTS supernet strongly downweights

Original: Conv5x5+ReLU operations when using the ”Classic” search space with PPO.
Revised: Conv5x5+ReLU operations when using the ”Classic” search space with PPO.

Original: In order to verify the predictive power of the softmax weights, as a proxy, we thus also performed evaluations when using purely 3x3 or 5x5 convolutions on a large IMPALA-CNN with 64 × 5 depths.
Revised: In order to verify the predictive power of the softmax weights, as a proxy, we thus also performed evaluations when using purely 3x3 or 5x5 convolutions on a large IMPALA-CNN with 64 × 5 depths.

Original: We see that
Revised: We see that

Original: the
Revised: the

Original: Conv3x3 setting indeed outperforms Conv5x5, corroborating the results in which during training, α strongly upweights the Conv3x3+ReLU operation and downweights Conv5x5+ReLU.
Revised: Conv3x3 setting indeed outperforms Conv5x5, corroborating the results in which during training, α strongly upweights the Conv3x3+ReLU operation and downweights Conv5x5+ReLU.

Original: C. D ISCRETE C ELL E VOLUTIONS
Revised: C. D ISCRETE C ELL E VOLUTIONS

Original: Along with Figure 5 in the main body of the paper, we also compare extra examples of discretizations before and after supernet training, to display reasonable behaviors induced by the trajectory of α .
Revised: Along with Figure 5 in the main body of the paper, we also compare extra examples of discretizations before and after supernet training, to display reasonable behaviors induced by the trajectory of α .

Original: In Figure 14, we use PPO with the “Classic” search space, but instead use ( N, R, I ) =
Revised: In Figure 14, we use PPO with the “Classic” search space, but instead use ( N, R, I ) =

Original: (1 , 0 , 6) along with outputting the last node (instead of concatenation with a Conv1x1 for the output) to allow a larger normal cell search space and graph topology.
Revised: (1 , 0 , 6) along with outputting the last node (instead of concatenation with a Conv1x1 for the output) to allow a larger normal cell search space and graph topology.

Original: In Figure 15, the discretized cell initially uses a large number of skip connections as well as dead-end nodes.
Revised: In Figure 15, the discretized cell initially uses a large number of skip connections as well as dead-end nodes.

Original: However, at convergence, it eventually utilizes all nodes to compute the ﬁnal output.
Revised: However, at convergence, it eventually utilizes all nodes to compute the ﬁnal output.

Original: Curiously, we ﬁnd that the skip connection between the input and output appears commonly throughout many searches.
Revised: Curiously, we ﬁnd that the skip connection between the input and output appears commonly throughout many searches.

Original: For the Rainbow setting, in Figure 6 in the main body of the paper, we saw that when the search process is successful, the supernet’s training trajectory induces discretized cells which improve evaluation performance as well.
Revised: For the Rainbow setting, in Figure 6 in the main body of the paper, we saw that when the search process is successful, the supernet’s training trajectory induces discretized cells which improve evaluation performance as well.

Original: The cells discovered later generally perform better than cells discovered earlier in the supernet training process.
Revised: The cells discovered later generally perform better than cells discovered earlier in the supernet training process.

Original: In Figure 16, we show more examples of such evaluation curves.
Revised: In Figure 16, we show more examples of such evaluation curves.

Original: C. C ORRELATION B ETWEEN S UPERNET AND D ISCRETIZED C ELLS
Revised: C. C ORRELATION B ETWEEN S UPERNET AND D ISCRETIZED C ELLS

Original: Given that the discretized cell only explicitly depends on architecture variables α and not necessarily model weights θ , one may wonder: Is there a relationship between the rewards of the supernet and of its corresponding discretized cell?
Revised: Given that the discretized cell only explicitly depends on architecture variables α and not necessarily model weights θ , one may wonder: Is there a relationship between the rewards of the supernet and of its corresponding discretized cell?

Original: For instance, the degenerate/underperformance setting mentioned in Section 4.4 and Appendix C can be thought of as an extreme scenario.
Revised: For instance, the degenerate/underperformance setting mentioned in Section 4.4 and Appendix C can be thought of as an extreme scenario.

Original: At the same time, there could be an integrality gap, where there the discretization process δ ( α ) produces cells which give different rewards than the supernet.
Revised: At the same time, there could be an integrality gap, where there the discretization process δ ( α ) produces cells which give different rewards than the supernet.

Original: In order to make such an analysis comparing rewards, we ﬁrst must prevent confounding factors arising from Rainbow’s natural performance on an environment regardless of architecture.
Revised: In order to make such an analysis comparing rewards, we ﬁrst must prevent confounding factors arising from Rainbow’s natural performance on an environment regardless of architecture.

Original: We thus ﬁrst divide the supernet and discrete cell scores by the score obtained by the IMPALA-CNN baseline, where the baseline and discrete cells all used depths of 64 × 3 .
Revised: We thus ﬁrst divide the supernet and discrete cell scores by the score obtained by the IMPALA-CNN baseline, where the baseline and discrete cells all used depths of 64 × 3 .

Original: In Figure 17, when using Rainbow and observing across environments, we ﬁnd both high correlation and also integrality gaps: for some environments such as Ninja and Coinrun, there is a signiﬁcant correlation between supernet and discrete cell rewards, while for other environments such as Dodgeball, there is a signiﬁcant gap.
Revised: In Figure 17, when using Rainbow and observing across environments, we ﬁnd both high correlation and also integrality gaps: for some environments such as Ninja and Coinrun, there is a signiﬁcant correlation between supernet and discrete cell rewards, while for other environments such as Dodgeball, there is a signiﬁcant gap.

Original: This suggests that search quality can be improved via both better supernet training such as using hyperparameter tuning or data augmentation (Raileanu et al., 2020), as well as better discretization procedures such as early stopping and stronger pruning (Chu et al., 2020a; Liang et al., 2019; Wang et al., 2021b).
Revised: This suggests that search quality can be improved via both better supernet training such as using hyperparameter tuning or data augmentation (Raileanu et al., 2020), as well as better discretization procedures such as early stopping and stronger pruning (Chu et al., 2020a; Liang et al., 2019; Wang et al., 2021b).

Original: D N UMERICAL S CORES
Revised: D N UMERICAL S CORES

Original: In Tables 8 and 5b
Revised: In Tables 8 and 5b

Original: [Table]
Revised: [Table]

Original: we display the average normalized reward after 25M steps, as standard in Procgen (Cobbe et al., 2020), for a subset of environments in which RL-DARTS performs competitively.
Revised: we display the average normalized reward after 25M steps, as standard in Procgen (Cobbe et al., 2020), for a subset of environments in which RL-DARTS performs competitively.

Original: The normalized reward for each environment is computed as R norm =
Revised: The normalized reward for each environment is computed as R norm =

Original: ( R − R min ) /
Revised: ( R − R min ) /

Original: ( R max − R min ) where R max and R min are calculated using a combination of theoretical maximums and PPO-trained agents, and can be found in (Cobbe et al., 2020).
Revised: ( R max − R min ) where R max and R min are calculated using a combination of theoretical maximums and PPO-trained agents, and can be found in (Cobbe et al., 2020).

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: E E XTENDED L ARGE S CALE S TUDY ON R ANDOM C ELLS
Revised: E E XTENDED L ARGE S CALE S TUDY ON R ANDOM C ELLS

Original: For some environments such as Dodgeball and Maze, the gap between DARTS and random search can be dramatic.
Revised: For some environments such as Dodgeball and Maze, the gap between DARTS and random search can be dramatic.

Original: For each game, we trained 100 unique random cells using Rainbow + the “Micro” search space with depths 64 × 3 and found that all of the random cells underperformed against
Revised: For each game, we trained 100 unique random cells using Rainbow + the “Micro” search space with depths 64 × 3 and found that all of the random cells underperformed against

Original: DARTS in Figure 18.
Revised: DARTS in Figure 18.

Original: F H YPERPARAMETERS
Revised: F H YPERPARAMETERS

Original: In our code, we entirely use Tensorﬂow 2 for auto-differentation, as well as the April 2020 version of
Revised: In our code, we entirely use Tensorﬂow 2 for auto-differentation, as well as the April 2020 version of

Original: Procgen.
Revised: Procgen.

Original: For compute, we either used P100 or V100 GPUs based on convenience and availability.
Revised: For compute, we either used P100 or V100 GPUs based on convenience and availability.

Original: Below are the hyperparameter settings for speciﬁc methods.
Revised: Below are the hyperparameter settings for speciﬁc methods.

Original: For all training curves, we use the common standard for reporting in RL (i.e. plotting mean and standard deviation across 3 seeds)
Revised: For all training curves, we use the common standard for reporting in RL (i.e. plotting mean and standard deviation across 3 seeds)

Original: . F. DARTS
Revised: . F. DARTS

Original: Initially, we sweeped the softmax temperature in order to ﬁnd a stable default value that could be used for all environments.
Revised: Initially, we sweeped the softmax temperature in order to ﬁnd a stable default value that could be used for all environments.

Original: For PPO, the sweep was across the set { 5 . 0 , 10 . 0 , 15 . 0 } .
Revised: For PPO, the sweep was across the set { 5 . 0 , 10 . 0 , 15 . 0 } .

Original: For Rainbow, the sweep was across { 10 . 0 , 20 . 0 , 50 . 0 } .
Revised: For Rainbow, the sweep was across { 10 . 0 , 20 . 0 , 50 . 0 } .

Original: For tabular reported scores in Figures 8 and 5b, we used a consistent softmax temperature of 5.0 for
Revised: For tabular reported scores in Figures 8 and 5b, we used a consistent softmax temperature of 5.0 for

Original: PPO, and 10.0 for Rainbow.
Revised: PPO, and 10.0 for Rainbow.

Original: F. R AINBOW -DQN
Revised: F. R AINBOW -DQN

Original: We use Acme (Hoffman et al., 2020) for our code infrastructure.
Revised: We use Acme (Hoffman et al., 2020) for our code infrastructure.

Original: We use a learning rate 5 × 10 − 5 , batch size 256, n-step size 7, discount factor 0.99.
Revised: We use a learning rate 5 × 10 − 5 , batch size 256, n-step size 7, discount factor 0.99.

Original: For the priority replay buffer (Schaul et al., 2016), we use priority exponent 0.8, importance sampling exponent 0.2, replay buffer capacity 500K. For particular environments (Bigﬁsh, Bossﬁght, Chaser, Dodgeball, Miner, Plunder, Starpilot), we use n-step size 2 and replay buffer capacity 10K.
Revised: For the priority replay buffer (Schaul et al., 2016), we use priority exponent 0.8, importance sampling exponent 0.2, replay buffer capacity 500K. For particular environments (Bigﬁsh, Bossﬁght, Chaser, Dodgeball, Miner, Plunder, Starpilot), we use n-step size 2 and replay buffer capacity 10K.

Original: For C51 (Bellemare et al., 2017), we use 51 atoms, with v min = 0 , v max = 1 . 0 .
Revised: For C51 (Bellemare et al., 2017), we use 51 atoms, with v min = 0 , v max = 1 . 0 .

Original: As a preprocessing step, we normalize the environment rewards by dividing the raw rewards by the max possible rewards reported in (Cobbe et al., 2020).
Revised: As a preprocessing step, we normalize the environment rewards by dividing the raw rewards by the max possible rewards reported in (Cobbe et al., 2020).

Original: F. PPO
Revised: F. PPO

Original: We use TF-Agents (Guadarrama et al., 2018) for our code infrastructure, along with equivalent PPO hyperparameters found from (Cobbe et al., 2020).
Revised: We use TF-Agents (Guadarrama et al., 2018) for our code infrastructure, along with equivalent PPO hyperparameters found from (Cobbe et al., 2020).

Original: Due to necessary changes in minibatch size when applying DARTS modules or networks with higher GPU memory usage, we thus swept learning rate across { 1 × 10 − 4 , 2 .
Revised: Due to necessary changes in minibatch size when applying DARTS modules or networks with higher GPU memory usage, we thus swept learning rate across { 1 × 10 − 4 , 2 .

Original: 5 × 10 − 4 , 5 × 10 − 4 } and number of epochs across { 1 , 2 , 3 } .
Revised: 5 × 10 − 4 , 5 × 10 − 4 } and number of epochs across { 1 , 2 , 3 } .

Original: For all models, we use a maximum power of 2 minibatch size before encountering GPU out-ofmemory issues on a standard 16 GB GPU.
Revised: For all models, we use a maximum power of 2 minibatch size before encountering GPU out-ofmemory issues on a standard 16 GB GPU.

Original: Thus, for a 16 × 3 = [16 , 16 , 16] DARTS supernet, we set the minibatch size to be 256 , which is also used for evaluation with a 64 × 3 = [64 , 64 , 64] discretized
Revised: Thus, for a 16 × 3 = [16 , 16 , 16] DARTS supernet, we set the minibatch size to be 256 , which is also used for evaluation with a 64 × 3 = [64 , 64 , 64] discretized

Original: CNN.
Revised: CNN.

Original: Our hyperparameter gridsearch for the evaluation led to an optimal setting of learning rate = 1 × 10 − 4 and number of epochs = 1 .
Revised: Our hyperparameter gridsearch for the evaluation led to an optimal setting of learning rate = 1 × 10 − 4 and number of epochs = 1 .

Original: G M ISCELLANEOUS G. S EARCH S PACE S IZE
Revised: G M ISCELLANEOUS G. S EARCH S PACE S IZE

Original: Let O nz = |O| − 1 , the number of non-zero ops in O .
Revised: Let O nz = |O| − 1 , the number of non-zero ops in O .

Original: For a cell (normal or reduction), the ﬁrst intermediate node can only be connected to the input via a single op, and thus the choice is only
Revised: For a cell (normal or reduction), the ﬁrst intermediate node can only be connected to the input via a single op, and thus the choice is only

Original: O nz .
Revised: O nz .

Original: However, later intermediate nodes use  2 inputs which leads to a choice size of O 2 nz × (cid:0) i 2 (cid:1) where i is the index of the intermediate node.
Revised: However, later intermediate nodes use K = 2 inputs which leads to a choice size of O Knz × (cid:0) iK (cid:1) where i is the index of the intermediate node.

Original: Thus the total number of possible discrete cells is
Revised: Thus the total number of possible discrete cells is

Original: O nz · (cid:81) Ii = [Equation] (cid:0) O [Equation]
Revised: O nz · (cid:81) Ii =2 (cid:0) O Knz × (cid:0) iK (cid:1)(cid:1) .

Original: nz × (cid:0)

Original: i [Equation] (cid:1)(cid:1) . For the “Classic” search space, there are both normal and reduction cells to be optimized, with number of non-zero normal ops O nz,N = 5 and number of non-zero reduction ops
Revised: For the “Classic” search space, there are both normal and reduction cells to be optimized, with number of non-zero normal ops O nz,N = 5 and number of non-zero reduction ops

Original: O nz,R = 4 , with I = 4  for both.
Revised: O nz,R = 4 , with I = 4 , K = 2 for both.

Original: This leads to a total conﬁguration size of (cid:104)
Revised: This leads to a total conﬁguration size of (cid:104)

Original: O nz,N · (cid:81) Ii =2 (cid:0)
Revised: O nz,N · (cid:81) Ii =2 (cid:0)

Original: O 2 nz × (cid:0) i 2 (cid:1)(cid:1)(cid:105) ×
Revised: O 2 nz × (cid:0) i 2 (cid:1)(cid:1)(cid:105) ×

Revised: (cid:104)

Original: [Equation]
Revised: O nz,R · (cid:81) Ii =2 (cid:0) O 2 nz,R × (cid:0) i 2 (cid:1)(cid:1)(cid:105) ≈ 4 × 10 11 .