Original: R EINFORCEMENT L EARNING WITH C HROMATIC N ETWORKS
Revised: R EINFORCEMENT L EARNING WITH C HROMATIC N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS (Vinyals et al., 2015;
Revised: We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS (Vinyals et al., 2015;

Original: Pham et al., 2018; Zoph & Le, 2017) and ES (Salimans et al., 2017) in a highly scalable and intuitive way.
Revised: Pham et al., 2018; Zoph & Le, 2017) and ES (Salimans et al., 2017) in a highly scalable and intuitive way.

Original: By deﬁning the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efﬁcient learned edge-partitionings.
Revised: By deﬁning the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efﬁcient learned edge-partitionings.

Original: For several
Revised: For several

Original: RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as 17 weight parameters, providing > 90% compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices (Choromanski et al., 2018), while still maintaining good reward.
Revised: RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as 17 weight parameters, providing > 90% compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices (Choromanski et al., 2018), while still maintaining good reward.

Original: We believe that our work is one of the ﬁrst attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics (Gage, 2002) with limited storage and computational resources.
Revised: We believe that our work is one of the ﬁrst attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics (Gage, 2002) with limited storage and computational resources.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Consider a ﬁxed Markov Decision Process (MDP) M and an agent aiming to maximize its total expected/discounted reward obtained in the environment E governed by M .
Revised: Consider a ﬁxed Markov Decision Process (MDP) M and an agent aiming to maximize its total expected/discounted reward obtained in the environment E governed by M .

Original: An agent is looking for a sequence of actions a 0 , ..., a T − 1 leading to a series of steps maximizing this reward.
Revised: An agent is looking for a sequence of actions a 0 , ..., a T − 1 leading to a series of steps maximizing this reward.

Original: One of the approaches is to construct a policy π θ : S → A , parameterized by vector θ , which is a mapping from states to actions.
Revised: One of the approaches is to construct a policy π θ : S → A , parameterized by vector θ , which is a mapping from states to actions.

Original: Policy π θ determines actions chosen in states visited by an agent.
Revised: Policy π θ determines actions chosen in states visited by an agent.

Original: Such a reinforcement learning (RL) policy is usually encoded as a neural network, in which scenario parameters θ correspond to weights and biases of a neural network.
Revised: Such a reinforcement learning (RL) policy is usually encoded as a neural network, in which scenario parameters θ correspond to weights and biases of a neural network.

Original: Reinforcement learning policies π θ often consist of thousands or millions of parameters (e.g. when they involve vision as part of the state vector) and therefore training them becomes a challenging high-dimensional optimization problem.
Revised: Reinforcement learning policies π θ often consist of thousands or millions of parameters (e.g. when they involve vision as part of the state vector) and therefore training them becomes a challenging high-dimensional optimization problem.

Original: Deploying such high-dimensional policies on hardware raises additional concerns in resource constrained settings (e.g. limited storage), emerging in particular in mobile robotics (Gage, 2002).
Revised: Deploying such high-dimensional policies on hardware raises additional concerns in resource constrained settings (e.g. limited storage), emerging in particular in mobile robotics (Gage, 2002).

Original: The main question we tackle in this paper is the following:
Revised: The main question we tackle in this paper is the following:

Original: Are high dimensional architectures necessary for encoding efﬁcient policies and if not, how compact can they be in in practice?
Revised: Are high dimensional architectures necessary for encoding efﬁcient policies and if not, how compact can they be in in practice?

Original: We show that ﬁnding such compact representations is a nontrivial optimization problem despite recent observations that some hardcoded structured families (Choromanski et al., 2018) provide certain levels of compactiﬁcation and good accuracy at the same time.
Revised: We show that ﬁnding such compact representations is a nontrivial optimization problem despite recent observations that some hardcoded structured families (Choromanski et al., 2018) provide certain levels of compactiﬁcation and good accuracy at the same time.

Original: We model the problem of ﬁnding compact presentations by using a joint objective between the combinatorial nature of the network’s parameter sharing proﬁle and the reward maximization of
Revised: We model the problem of ﬁnding compact presentations by using a joint objective between the combinatorial nature of the network’s parameter sharing proﬁle and the reward maximization of

Original: RL optimization.
Revised: RL optimization.

Original: We leverage recent advances in the ENAS (Efﬁcient Neural Architecture Search) literature and theory of pointer networks (Vinyals et al., 2015; Pham et al., 2018; Zoph & Le, 2017) to optimize over the combinatorial component of this objective and state of the art evolution strategies (ES) methods (Choromanski et al., 2018; Salimans et al., 2017; Mania et al., 2018a) to optimize over the RL objective.
Revised: We leverage recent advances in the ENAS (Efﬁcient Neural Architecture Search) literature and theory of pointer networks (Vinyals et al., 2015; Pham et al., 2018; Zoph & Le, 2017) to optimize over the combinatorial component of this objective and state of the art evolution strategies (ES) methods (Choromanski et al., 2018; Salimans et al., 2017; Mania et al., 2018a) to optimize over the RL objective.

Original: We propose to deﬁne the combinatorial search space to be the the set of different edge-partitioning (colorings) into same-weight classes and construct policies with learned weight-sharing mechanisms.
Revised: We propose to deﬁne the combinatorial search space to be the the set of different edge-partitioning (colorings) into same-weight classes and construct policies with learned weight-sharing mechanisms.

Original: We call networks encoding our policies: chromatic networks .
Revised: We call networks encoding our policies: chromatic networks .

Original: We are inspired by two recent papers:
Revised: We are inspired by two recent papers:

Original: (Choromanski et al., 2018) and (Gaier & Ha, 2019).
Revised: (Choromanski et al., 2018) and (Gaier & Ha, 2019).

Original: In the former one, policies based on Toeplitz matrices were shown to match their unstructured counterparts accuracy-wise, while leading to the substantial reduction of the number of parameters from
Revised: In the former one, policies based on Toeplitz matrices were shown to match their unstructured counterparts accuracy-wise, while leading to the substantial reduction of the number of parameters from

Original: T t = w t 0 w t 1 w t 2 w t 3 w t 4 w tw t 6 w t 0 w t 1 w t 2 w t 3 w tw t 7 w t 6 w t 0 w t 1 w t 2 w tw t 8 w t 7 w t 6 w t 0 w t 1 w t    π t = w t 0 w t 1 w t 2 w t 3 w t 4 w t 5 w t 6 w t 7 w t(cid:18)
Revised: T t = w t 0 w t 1 w t 2 w t 3 w t 4 w tw t 6 w t 0 w t 1 w t 2 w t 3 w tw t 7 w t 6 w t 0 w t 1 w t 2 w tw t 8 w t 7 w t 6 w t 0 w t 1 w t    π t = w t 0 w t 1 w t 2 w t 3 w t 4 w t 5 w t 6 w t 7 w t(cid:18)

Original: (cid:19) thousands (Salimans et al., 2017) to hundreds (Choromanski et al., 2018).
Revised: (cid:19) thousands (Salimans et al., 2017) to hundreds (Choromanski et al., 2018).

Original: Instead of quadratic (in sizes of hidden layers), those policies use only linear number of parameters.
Revised: Instead of quadratic (in sizes of hidden layers), those policies use only linear number of parameters.

Original: The Toeplitz structure can be thought of as a parameter sharing mechanism, where edge weights along each diagonal of the matrix are the same (see: Fig. 1).
Revised: The Toeplitz structure can be thought of as a parameter sharing mechanism, where edge weights along each diagonal of the matrix are the same (see: Fig. 1).

Original: However, this is a rigid pattern that is not learned.
Revised: However, this is a rigid pattern that is not learned.

Original: We show in this paper that weight sharing patterns can be effectively learned, which further reduces the number of distinct parameters.
Revised: We show in this paper that weight sharing patterns can be effectively learned, which further reduces the number of distinct parameters.

Original: For instance, using architectures of the same sizes as those in (Choromanski et al., 2018), we can train effective policies for OpenAI Gym tasks with as few as 17 distinct weights.
Revised: For instance, using architectures of the same sizes as those in (Choromanski et al., 2018), we can train effective policies for OpenAI Gym tasks with as few as 17 distinct weights.

Original: The latter paper proposes an extremal approach, where weights are chosen randomly instead of being learned, but the topologies of connections are trained and thus are ultimately strongly biased towards RL tasks under consideration.
Revised: The latter paper proposes an extremal approach, where weights are chosen randomly instead of being learned, but the topologies of connections are trained and thus are ultimately strongly biased towards RL tasks under consideration.

Original: It was shown in (Gaier & Ha, 2019) that such weight agnostic neural networks (WANNs) can encode effective policies for several nontrivial RL problems.
Revised: It was shown in (Gaier & Ha, 2019) that such weight agnostic neural networks (WANNs) can encode effective policies for several nontrivial RL problems.

Original: WANNs replace conceptually simple feedforward networks with general graph topologies using NEAT algorithm (Stanley & Miikkulainen, 2002) providing topological operators to build the network.
Revised: WANNs replace conceptually simple feedforward networks with general graph topologies using NEAT algorithm (Stanley & Miikkulainen, 2002) providing topological operators to build the network.

Original: Our approach is a middle ground, where the topology is still a feedforward neural network, but the weights are partitioned into groups that are being learned in a combinatorial fashion using reinforcement learning.
Revised: Our approach is a middle ground, where the topology is still a feedforward neural network, but the weights are partitioned into groups that are being learned in a combinatorial fashion using reinforcement learning.

Original: While (Chen et al., 2015) shares weights randomly via hashing, we learn a good partitioning mechanisms for weight sharing.
Revised: While (Chen et al., 2015) shares weights randomly via hashing, we learn a good partitioning mechanisms for weight sharing.

Original: Our key observation is that ENAS and ES can naturally be combined in a highly scalable but conceptually simple way .
Revised: Our key observation is that ENAS and ES can naturally be combined in a highly scalable but conceptually simple way .

Original: To give context, vanilla NAS (Zoph & Le, 2017) for classical supervised learning setting (SL) requires a large population of 450 GPU-workers (child models) all training one-by-one, which results in many GPU-hours of training.
Revised: To give context, vanilla NAS (Zoph & Le, 2017) for classical supervised learning setting (SL) requires a large population of 450 GPU-workers (child models) all training one-by-one, which results in many GPU-hours of training.

Original: ENAS (Pham et al., 2018) uses weight sharing across multiple workers to reduce the time, although it can reduce computational resources at the cost of the variance of the controller’s gradient.
Revised: ENAS (Pham et al., 2018) uses weight sharing across multiple workers to reduce the time, although it can reduce computational resources at the cost of the variance of the controller’s gradient.

Original: Our method solves both issues (fast training time and low controller gradient variance) by leveraging a large population of much-cheaper CPU workers (300) increasing the effective batch-size of the controller, while also training the workers simultaneously via ES.
Revised: Our method solves both issues (fast training time and low controller gradient variance) by leveraging a large population of much-cheaper CPU workers (300) increasing the effective batch-size of the controller, while also training the workers simultaneously via ES.

Original: This setup is not possible in SL, as single CPUs cannot train large image-based classiﬁers in practice.
Revised: This setup is not possible in SL, as single CPUs cannot train large image-based classiﬁers in practice.

Original: Furthermore, this magnitude of scaling by numerous workers can be difﬁcult with policy gradient or Q-learning methods as they can be limited by GPU overhead due to exact-gradient computation.
Revised: Furthermore, this magnitude of scaling by numerous workers can be difﬁcult with policy gradient or Q-learning methods as they can be limited by GPU overhead due to exact-gradient computation.

Original: We believe that our work is one of the ﬁrst attempts to propose a ﬂexible, rigorous approach to training compact neural network architectures for RL problems.
Revised: We believe that our work is one of the ﬁrst attempts to propose a ﬂexible, rigorous approach to training compact neural network architectures for RL problems.

Original: Those may be of particular importance in mobile robotics (Gage, 2002) where computational and storage resources are very limited.
Revised: Those may be of particular importance in mobile robotics (Gage, 2002) where computational and storage resources are very limited.

Original: We also believe that this paper opens several new research directions regarding structured policies for robotics.
Revised: We also believe that this paper opens several new research directions regarding structured policies for robotics.

Original: 1. B ACKGROUND AND R ELATED W ORK
Revised: 1. B ACKGROUND AND R ELATED W ORK

Original: Network Architecture Search: The subject of this paper can be put in the larger context of Neural
Revised: Network Architecture Search: The subject of this paper can be put in the larger context of Neural

Original: Architecture Search (NAS) algorithms which recently became a proliﬁc area of research with already voluminous literature (see: (Elsken et al., 2019) for an excellent survey).
Revised: Architecture Search (NAS) algorithms which recently became a proliﬁc area of research with already voluminous literature (see: (Elsken et al., 2019) for an excellent survey).

Original: Interest in NAS algorithms started to grow rapidly when it was shown that they can design state-of-the-art architectures for image recognition and language modeling (Zoph & Le, 2017).
Revised: Interest in NAS algorithms started to grow rapidly when it was shown that they can design state-of-the-art architectures for image recognition and language modeling (Zoph & Le, 2017).

Original: More recently it was shown that NAS network generators can be improved to sample more complicated connectivity patterns based on random graph theory models such as Erdos-Renyi, Barabasi-Albert or Watts-Strogatz to outperform human-designed networks, e.g. ResNet and ShufﬂeNet on image recognition tasks (Xie et al., 2019).
Revised: More recently it was shown that NAS network generators can be improved to sample more complicated connectivity patterns based on random graph theory models such as Erdos-Renyi, Barabasi-Albert or Watts-Strogatz to outperform human-designed networks, e.g. ResNet and ShufﬂeNet on image recognition tasks (Xie et al., 2019).

Original: However to the best of our knowledge, applying NAS to construct compact RL policy architectures has not been explored before.
Revised: However to the best of our knowledge, applying NAS to construct compact RL policy architectures has not been explored before.

Original: On a broader scope, there has been little work in applying NAS to general RL policies, partially because unlike SL, RL policies are quite small and thus do not require a search space involving hyperparameter primitives such as number of hidden layers, convolution sizes, etc.
Revised: On a broader scope, there has been little work in applying NAS to general RL policies, partially because unlike SL, RL policies are quite small and thus do not require a search space involving hyperparameter primitives such as number of hidden layers, convolution sizes, etc.

Original: However, we show that the search space for RL policies can still be quite combinatorial, by examining partitionings of weights in our work.
Revised: However, we show that the search space for RL policies can still be quite combinatorial, by examining partitionings of weights in our work.

Original: Parameterizing Compact Architectures: Before NAS can be applied, a particular parameterization of a compact architecture deﬁning combinatorial search space needs to be chosen.
Revised: Parameterizing Compact Architectures: Before NAS can be applied, a particular parameterization of a compact architecture deﬁning combinatorial search space needs to be chosen.

Original: That leads to the vast literature on compact encodings of NN architectures.
Revised: That leads to the vast literature on compact encodings of NN architectures.

Original: Some of the most popular and efﬁcient techniques regard network sparsiﬁcation.
Revised: Some of the most popular and efﬁcient techniques regard network sparsiﬁcation.

Original: The sparsity can be often achieved by pruning already trained networks.
Revised: The sparsity can be often achieved by pruning already trained networks.

Original: These methods have been around since the 1980s, dating back to Rumelhart (Rumelhart, 1987; Chauvin, 1989; Mozer & Smolensky, 1989), followed shortly by Optimal Brain
Revised: These methods have been around since the 1980s, dating back to Rumelhart (Rumelhart, 1987; Chauvin, 1989; Mozer & Smolensky, 1989), followed shortly by Optimal Brain

Original: Damage (Cun et al., 1990), which used second order gradient information to remove connections.
Revised: Damage (Cun et al., 1990), which used second order gradient information to remove connections.

Original: Since then, a variety of schemes have been proposed, with regularization (Louizos et al., 2018) and magnitude-based weight pruning methods (Han et al., 2015; See et al., 2016;
Revised: Since then, a variety of schemes have been proposed, with regularization (Louizos et al., 2018) and magnitude-based weight pruning methods (Han et al., 2015; See et al., 2016;

Original: Narang et al., 2017) increasingly popular.
Revised: Narang et al., 2017) increasingly popular.

Original: The impact of dropout (Srivastava et al., 2014) has added an additional perspective, with new works focusing on attempts to learn sparse networks (Gomez et al., 2019).
Revised: The impact of dropout (Srivastava et al., 2014) has added an additional perspective, with new works focusing on attempts to learn sparse networks (Gomez et al., 2019).

Original: Another recent work introduced the Lottery Ticket Hypothesis (Frankle & Carbin, 2019), which captivated the community by showing that there exist equivalently sparse networks which can be trained from scratch to achieve competitive results.
Revised: Another recent work introduced the Lottery Ticket Hypothesis (Frankle & Carbin, 2019), which captivated the community by showing that there exist equivalently sparse networks which can be trained from scratch to achieve competitive results.

Original: Interestingly, these works consistently report similar levels of compression, often managing to match the performance of original networks with up to 90% fewer parameters.
Revised: Interestingly, these works consistently report similar levels of compression, often managing to match the performance of original networks with up to 90% fewer parameters.

Original: Those methods are however designed for constructing classiﬁcation networks rather than those encoding RL policies.
Revised: Those methods are however designed for constructing classiﬁcation networks rather than those encoding RL policies.

Original: Quantization: Weight sharing mechanism can be viewed from the quantization point of view, where pre-trained weights are quantized, thus effectively partitioned.
Revised: Quantization: Weight sharing mechanism can be viewed from the quantization point of view, where pre-trained weights are quantized, thus effectively partitioned.

Original: Examples include (Han et al., 2016), who achieve 49 x compression for networks applied for vision using both pruning and weight sharing (by quantization) followed by Huffman coding.
Revised: Examples include (Han et al., 2016), who achieve 49 x compression for networks applied for vision using both pruning and weight sharing (by quantization) followed by Huffman coding.

Original: However such partitions are not learned which is a main topic of this paper.
Revised: However such partitions are not learned which is a main topic of this paper.

Original: Even more importantly, in RL applications, where policies are often very sensitive to parameters’ weights (Iscen et al., 2018), centroid-based quantization is too crude to preserve accuracy.
Revised: Even more importantly, in RL applications, where policies are often very sensitive to parameters’ weights (Iscen et al., 2018), centroid-based quantization is too crude to preserve accuracy.

Original: Compact RL Policies In recent times there has been increased interest in simplifying RL policies.
Revised: Compact RL Policies In recent times there has been increased interest in simplifying RL policies.

Original: In particular, (Mania et al., 2018b) demonstrated that linear policies are often sufﬁcient for the benchmark MuJoCo locomotion tasks, while (Cuccu et al., 2019) found smaller policies could work for vision-based tasks by separating feature extraction and control.
Revised: In particular, (Mania et al., 2018b) demonstrated that linear policies are often sufﬁcient for the benchmark MuJoCo locomotion tasks, while (Cuccu et al., 2019) found smaller policies could work for vision-based tasks by separating feature extraction and control.

Original: Finally, recent work found that small, sparse sub-networks can perform better than larger over-parameterized ones (Frankle &
Revised: Finally, recent work found that small, sparse sub-networks can perform better than larger over-parameterized ones (Frankle &

Original: Carbin, 2019), inspiring applications in RL (Yu et al., 2019).
Revised: Carbin, 2019), inspiring applications in RL (Yu et al., 2019).

Original: Our main contributions are:
Revised: Our main contributions are:

Original: We propose a highly scalable algorithm by combining ENAS and ES for learning compact representations that learns effective policies with over 92% reduction of the number of neural network parameters (Section 3).
Revised: We propose a highly scalable algorithm by combining ENAS and ES for learning compact representations that learns effective policies with over 92% reduction of the number of neural network parameters (Section 3).

Original: To demonstrate the impact (and limitations) of pruning neural networks for RL, we adapt recent algorithms training both masks deﬁning combinatorial structure as well as weights of a deep neural network concurrently (see: (Lenc et al., 2019)).
Revised: To demonstrate the impact (and limitations) of pruning neural networks for RL, we adapt recent algorithms training both masks deﬁning combinatorial structure as well as weights of a deep neural network concurrently (see: (Lenc et al., 2019)).

Original: Those achieve state-ofthe-art results on various supervised feedforward and recurrent models.
Revised: Those achieve state-ofthe-art results on various supervised feedforward and recurrent models.

Original: We conﬁrm these ﬁndings in the RL setting by showing that good rewards can be obtained up to a high level of pruning.
Revised: We conﬁrm these ﬁndings in the RL setting by showing that good rewards can be obtained up to a high level of pruning.

Original: However, at the 80 - 90% level
Revised: However, at the 80 - 90% level

Original: we see a signiﬁcant decrease in performance which does not occur for the proposed by us chromatic networks (Section 4.1, Section 4.2).
Revised: we see a signiﬁcant decrease in performance which does not occur for the proposed by us chromatic networks (Section 4.1, Section 4.2).

Original: We demonstrate that ﬁnding efﬁcient weight-partitioning mechanisms is a challenging problem and NAS helps to construct distributions producing good partitionings for more difﬁcult RL environments (Section 4.3).
Revised: We demonstrate that ﬁnding efﬁcient weight-partitioning mechanisms is a challenging problem and NAS helps to construct distributions producing good partitionings for more difﬁcult RL environments (Section 4.3).

Original:  T HE A RCHITECTURE OF N EURAL A RCHITECTURE S EARCH
Revised:  T HE A RCHITECTURE OF N EURAL A RCHITECTURE S EARCH

Original: The foundation of our algorithm for learning structured compact policies is the class of ENAS methods (Pham et al., 2018).
Revised: The foundation of our algorithm for learning structured compact policies is the class of ENAS methods (Pham et al., 2018).

Original: To present our algorithm, we thus need to ﬁrst describe this class.
Revised: To present our algorithm, we thus need to ﬁrst describe this class.

Original: ENAS algorithms are designed to construct neural network architectures thus they aim to solve combinatorial-ﬂavored optimization problems with exponential-size domains.
Revised: ENAS algorithms are designed to construct neural network architectures thus they aim to solve combinatorial-ﬂavored optimization problems with exponential-size domains.

Original: The problems are cast as MDPs, where a controller encoded by the LSTM-based policy π cont ( θ ) , typically parameterized by few hundred hidden units, is trained to propose good-quality architectures, or to be more precise: good-quality distributions D ( θ ) over architectures.
Revised: The problems are cast as MDPs, where a controller encoded by the LSTM-based policy π cont ( θ ) , typically parameterized by few hundred hidden units, is trained to propose good-quality architectures, or to be more precise: good-quality distributions D ( θ ) over architectures.

Original: In standard applications the score of the particular distribution D ( θ ) is quantiﬁed by the average performance obtained by trained models leveraging architectures A ∼ D ( θ ) on the ﬁxed-size validation set.
Revised: In standard applications the score of the particular distribution D ( θ ) is quantiﬁed by the average performance obtained by trained models leveraging architectures A ∼ D ( θ ) on the ﬁxed-size validation set.

Original: That score determines the reward the controller obtains by proposing D ( θ ) .
Revised: That score determines the reward the controller obtains by proposing D ( θ ) .

Original: LSTM-based controller constructs architectures using softmax classiﬁers via autoregressive strategy, where controller’s decision in step t is given to it as an input embedding at time t + 1 .
Revised: LSTM-based controller constructs architectures using softmax classiﬁers via autoregressive strategy, where controller’s decision in step t is given to it as an input embedding at time t + 1 .

Original: The initial embedding that the controller starts with is an empty one.
Revised: The initial embedding that the controller starts with is an empty one.

Original: Weight Sharing Mechanism: ENAS introduces a powerful idea of a weight-sharing mechanism.
Revised: Weight Sharing Mechanism: ENAS introduces a powerful idea of a weight-sharing mechanism.

Original: The core concept is that different architectures can be embedded into combinatorial space, where they correspond to different subgraphs of the given acyclic directed base graph G (DAG).
Revised: The core concept is that different architectures can be embedded into combinatorial space, where they correspond to different subgraphs of the given acyclic directed base graph G (DAG).

Original: Weights of the edges of G represent the shared-pool W shared from which different architectures will inherit differently by activating weights of the corresponding induced directed subgraph.
Revised: Weights of the edges of G represent the shared-pool W shared from which different architectures will inherit differently by activating weights of the corresponding induced directed subgraph.

Original: At ﬁrst glance such a mechanism might be conceptually problematic since, a weight of the particular edge e belonging to different architectures
Revised: At ﬁrst glance such a mechanism might be conceptually problematic since, a weight of the particular edge e belonging to different architectures

Original: A
Revised: A

Original: i 1 , ...
Revised: i 1 , ...

Original: A i k (see: next paragraph for details regarding weight training) will be updated based on evaluations of all of them and
Revised: A i k (see: next paragraph for details regarding weight training) will be updated based on evaluations of all of them and

Original: different A
Revised: different A

Original: i s can utilize e in different ways.
Revised: i s can utilize e in different ways.

Original: However it turns out that this is desirable in practice.
Revised: However it turns out that this is desirable in practice.

Original: As authors of (Pham et al., 2018) explain, the approach is motivated by recent work on transfer and multitask learning that provides theoretical grounds for transferring weights across models.
Revised: As authors of (Pham et al., 2018) explain, the approach is motivated by recent work on transfer and multitask learning that provides theoretical grounds for transferring weights across models.

Original: Another way to justify the mechanism is to observe that ENAS tries in fact to optimize a distribution over architectures rather than a particular architecture and the corresponding shared-pool of weights W shared should be thought of as corresponding to that distribution rather than its particular realizations.
Revised: Another way to justify the mechanism is to observe that ENAS tries in fact to optimize a distribution over architectures rather than a particular architecture and the corresponding shared-pool of weights W shared should be thought of as corresponding to that distribution rather than its particular realizations.

Original: Therefore the weights of that pool should be updated based on signals from all different realizations.
Revised: Therefore the weights of that pool should be updated based on signals from all different realizations.

Original: Alternating Optimization: For a ﬁxed parameterization θ deﬁning policy π ( θ ) of the controller (and thus also proposed distribution over architectures D ( θ ) ), the algorithm optimizes the weights of the models using M architectures: A 1 , ..., A M sampled from D ( θ ) , where the sampling is conducted by the controller’s policy π cont ( θ ) .
Revised: Alternating Optimization: For a ﬁxed parameterization θ deﬁning policy π ( θ ) of the controller (and thus also proposed distribution over architectures D ( θ ) ), the algorithm optimizes the weights of the models using M architectures: A 1 , ..., A M sampled from D ( θ ) , where the sampling is conducted by the controller’s policy π cont ( θ ) .

Original: Models corresponding to A 1 , ..., A M are called child models .
Revised: Models corresponding to A 1 , ..., A M are called child models .

Original: At iteration k of the weight optimization process, a worker assigned to the architecture A i computes the gradient of the loss function L A i , B k corresponding to the particular batch B k of the training data with respect to the weights of the inherited edges from the base DAG.
Revised: At iteration k of the weight optimization process, a worker assigned to the architecture A i computes the gradient of the loss function L A i , B k corresponding to the particular batch B k of the training data with respect to the weights of the inherited edges from the base DAG.

Original: Since the loss function
Revised: Since the loss function

Original: L B k on W shared for a ﬁxed distribution D ( θ ) and batch B k at iteration k is deﬁned as an expected loss
Revised: L B k on W shared for a ﬁxed distribution D ( θ ) and batch B k at iteration k is deﬁned as an expected loss

Original: E A∼ π ( θ )
Revised: E A∼ π ( θ )

Original: [ L A , B k ( W A shared )] , where W A shared stands for the projection of the set of shared-pool weights W shared into edges deﬁned by the induced subgraph determined by A , its gradient with respect to W shared can be estimated via Monte Carlo procedure as:
Revised: [ L A , B k ( W A shared )] , where W A shared stands for the projection of the set of shared-pool weights W shared into edges deﬁned by the induced subgraph determined by A , its gradient with respect to W shared can be estimated via Monte Carlo procedure as:

Original: [Equation]
Revised: [Equation]

Original: After weight optimization is completed, ENAS updates the parameters θ of the controller responsible for shaping the distribution used to sample architectures A .
Revised: After weight optimization is completed, ENAS updates the parameters θ of the controller responsible for shaping the distribution used to sample architectures A .

Original: As mentioned before, this is done using reinforcement learning approach, where parameters θ are optimized to maximize the expected reward E A∼ π ( θ )
Revised: As mentioned before, this is done using reinforcement learning approach, where parameters θ are optimized to maximize the expected reward E A∼ π ( θ )

Original: [ R W A shared ( A )] , where this time set W shared is frozen and
Revised: [ R W A shared ( A )] , where this time set W shared is frozen and

Original: R W A shared ( A ) is given as the accuracy obtained by the model using architecture A and weights from W A shared on the validation set.
Revised: R W A shared ( A ) is given as the accuracy obtained by the model using architecture A and weights from W A shared on the validation set.

Original: Parameters θ are updated with the use of the REINFORCE algorithm (Williams, 1992).
Revised: Parameters θ are updated with the use of the REINFORCE algorithm (Williams, 1992).

Original: Controller’s Architecture: The controller LSTM-based architecture uses pointer networks (Vinyals et al., 2015) that utilize LSTM-encoder/decoder approach, where the decoder can look back and forth over input.
Revised: Controller’s Architecture: The controller LSTM-based architecture uses pointer networks (Vinyals et al., 2015) that utilize LSTM-encoder/decoder approach, where the decoder can look back and forth over input.

Original: Thus these models are examples of architectures enriched with attention.
Revised: Thus these models are examples of architectures enriched with attention.

Original:  T OWARDS C HROMATIC N ETWORKS - ENAS FOR G RAPH P ARTITIONINGS
Revised:  T OWARDS C HROMATIC N ETWORKS - ENAS FOR G RAPH P ARTITIONINGS

Original: Preliminaries: Chromatic networks are feedforward NN architectures, where weights are shared across multiple edges and sharing mechanism is learned via a modiﬁed ENAS algorithm that we present below.
Revised: Preliminaries: Chromatic networks are feedforward NN architectures, where weights are shared across multiple edges and sharing mechanism is learned via a modiﬁed ENAS algorithm that we present below.

Original: Edges sharing a particular weight form the so-called chromatic class .
Revised: Edges sharing a particular weight form the so-called chromatic class .

Original: An example of the learned partitioning is presented on Fig.
Revised: An example of the learned partitioning is presented on Fig.

Original: Learned weight-sharing mechanisms are more complicated than hardcoded ones from Fig.
Revised: Learned weight-sharing mechanisms are more complicated than hardcoded ones from Fig.

Original: 1, but lead to smaller-size partitionings.
Revised: 1, but lead to smaller-size partitionings.

Original: For instance,
Revised: For instance,

Original: Toeplitz sharing mechanism for architecture from Subﬁgure (b) of Fig.
Revised: Toeplitz sharing mechanism for architecture from Subﬁgure (b) of Fig.

Original: 2 requires 103 weightparameters, while ours: only 17 .
Revised: 2 requires 103 weightparameters, while ours: only 17 .

Original: Controller Architecture & Training:
Revised: Controller Architecture & Training:

Original: We use a standard ENAS reinforcement learning controller similar to (Pham et al., 2018), applying pointer networks.
Revised: We use a standard ENAS reinforcement learning controller similar to (Pham et al., 2018), applying pointer networks.

Original: Our search space is the set of all possible mappings Φ : E → { 0 , 1 , ..., M − 1 } , where E stands for the set of all edges of the graph encoding an architecture and M is theupper bound on the number of partitions.
Revised: Our search space is the set of all possible mappings Φ : E → { 0 , 1 , ..., M − 1 } , where E stands for the set of all edges of the graph encoding an architecture and M is the number of partitions, which the user sets.

Original: Thus as opposed to standard ENAS approach, where the search space consists of different subgraphs, we instead deal with different colorings/partitionings of edges of a given base graph.
Revised: Thus as opposed to standard ENAS approach, where the search space consists of different subgraphs, we instead deal with different colorings/partitionings of edges of a given base graph.

Original: The controller learns distributions
Revised: The controller learns distributions

Original: D ( θ ) over these partitionings.
Revised: D ( θ ) over these partitionings.

Original: The shared pool of weights W shared is a latent vector in R M , where different entries correspond to weight values for different partitions enumerated from 0 to M − 1 .
Revised: The shared pool of weights W shared is a latent vector in R M , where different entries correspond to weight values for different partitions enumerated from 0 to M − 1 .

Original: As for standard ENAS, the controller π ( θ ) consists of a encoder RNN and decoder RNN, in which the encoder RNN is looped over the embedded input data, while the decoder is looped to repeatedly output smaller primitive components of the ﬁnal output.
Revised: As for standard ENAS, the controller π ( θ ) consists of a encoder RNN and decoder RNN, in which the encoder RNN is looped over the embedded input data, while the decoder is looped to repeatedly output smaller primitive components of the ﬁnal output.

Original: In our setting, we do not possess natural numerical data corresponding to an embedding of the inputs which are partition numbers and edges.
Revised: In our setting, we do not possess natural numerical data corresponding to an embedding of the inputs which are partition numbers and edges.

Original: Therefore we also train an embedding (as part of controller’s parameter vector θ ) of both using tables: V edge : e → R d and V partition : { 0 , 1 , ..., M − 1 } → R d . 
Revised: Therefore we also train an embedding (as part of controller’s parameter vector θ ) of both using tables: V edge : e → R d and V partition : { 0 , 1 , ..., M − 1 } → R d . [Equation]

Original: [Figure]
Revised: [Figure]

Original: hidden layer encoded by two matrices.

Original: State and action dimensionalities are: s = 17 and a =respectively and hidden layer for the architecture from (b) is of size 41 .

Original: Thus the size of the matrices are: 17 × 6 for the linear policy from (a) and: 17 × 41 , 41 × 6 for the nonlinear one from (b).

Original: We denote by P a partitioning of edges and deﬁne the reward obtained by a controller for a ﬁxed distribution D ( θ ) produced by its policy π ( θ ) as follows:
Revised: We denote by P a partitioning of edges and deﬁne the reward obtained by a controller for a ﬁxed distribution D ( θ ) produced by its policy π ( θ ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where R max W shared ( P ) stands for the maximal reward obtained during weight-optimization phase of the policy with partitioning P and with initial vector of distinct weights W shared .
Revised: where R max W shared ( P ) stands for the maximal reward obtained during weight-optimization phase of the policy with partitioning P and with initial vector of distinct weights W shared .

Original: As for the ENAS setup, in practice this reward is estimated by averaging over M workers evaluating independently different realizations P of D ( θ ) .
Revised: As for the ENAS setup, in practice this reward is estimated by averaging over M workers evaluating independently different realizations P of D ( θ ) .

Original: The updates of θ are conducted with the use of REINFORCE.
Revised: The updates of θ are conducted with the use of REINFORCE.

Original: 3. W EIGHT U PDATES VIA ES
Revised: 3. W EIGHT U PDATES VIA ES

Original: As opposed to standard ENAS, where weights for a ﬁxed distribution D ( θ ) generating architectures were trained by backpropagation, we propose to apply recently introduced ES blackbox optimization techniques for RL (Choromanski et al., 2018).
Revised: As opposed to standard ENAS, where weights for a ﬁxed distribution D ( θ ) generating architectures were trained by backpropagation, we propose to apply recently introduced ES blackbox optimization techniques for RL (Choromanski et al., 2018).

Original: For a ﬁxed partitioning P , we deﬁne the loss
Revised: For a ﬁxed partitioning P , we deﬁne the loss

Original: L P ( W shared ) with respect to weights as the negated reward obtained by an agent applying partitioning P and with vector of distinct weights W shared .
Revised: L P ( W shared ) with respect to weights as the negated reward obtained by an agent applying partitioning P and with vector of distinct weights W shared .

Original: That expression as a function of W shared is not necessarily differentiable, since it involves calls to the simulator.
Revised: That expression as a function of W shared is not necessarily differentiable, since it involves calls to the simulator.

Original: This also implies that explicit backpropagation is not possible.
Revised: This also implies that explicit backpropagation is not possible.

Original: We propose to instead estimate the gradient of its Gaussian smoothing L σ P ( W shared ) deﬁned as: L σ P ( W shared ) =
Revised: We propose to instead estimate the gradient of its Gaussian smoothing L σP ( W shared ) deﬁned as: L σP ( W shared ) =

Original: E g ∈N (0 , I M )
Revised: E g ∈N (0 , I M )

Original: [ L P ( W shared + σ g )] for a ﬁxed smoothing parameter σ > .
Revised: [ L P ( W shared + σ g )] for a ﬁxed smoothing parameter σ > .

Original: We approximate its gradient given by: ∇ W shared L σP ( W shared ) =
Revised: We approximate its gradient given by: ∇ W shared L σP ( W shared ) =

Original: σ E g ∈N (0 , I M )
Revised: σ E g ∈N (0 , I M )

Original: [ L P ( W shared + σ g ) g ] with the following forward ﬁnite difference unbiased estimator introduced in (Choromanski et al., 2018):
Revised: [ L P ( W shared + σ g ) g ] with the following forward ﬁnite difference unbiased estimator introduced in (Choromanski et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: where g 1 , ..., g t are sampled independently at random from N (0 , I M ) and the pivot point is deﬁned as an average loss for a given set of weights W shared over partitionings sampled from π ( θ ) .
Revised: where g 1 , ..., g t are sampled independently at random from N (0 , I M ) and the pivot point is deﬁned as an average loss for a given set of weights W shared over partitionings sampled from π ( θ ) .

Original: We note that a subtle key difference here from vanilla ES is by using the pivot point E
Revised: We note that a subtle key difference here from vanilla ES is by using the pivot point E

Original: P ∼ π ( θ ) [ L P ( W shared )] as the expectation over a distribution of partitions P ∼ π ( θ ) rather than a static single-query objective L ( W ) used for e.g. a basic Mujoco task.
Revised: P ∼ π ( θ ) [ L P ( W shared )] as the expectation over a distribution of partitions P ∼ π ( θ ) rather than a static single-query objective L ( W ) used for e.g. a basic Mujoco task.

Original: From a broader perspective, this comes from the fact that ES can optimize any objective of the form E P ∼P
Revised: From a broader perspective, this comes from the fact that ES can optimize any objective of the form E P ∼P

Original: [ f ( W, P )] where P is any (possibly discrete) object with distribution P and W is a continuous weight vector, by assigning both a perturbed weight W + σ g and a sampled P to a CPU worker for function evaluation on f ( W + σ g , P ) .
Revised: [ f ( W, P )] where P is any (possibly discrete) object with distribution P and W is a continuous weight vector, by assigning both a perturbed weight W + σ g and a sampled P to a CPU worker for function evaluation on f ( W + σ g , P ) .

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: The experimental section is organized as follows:
Revised: The experimental section is organized as follows:

Original: • In Subsection 4.1 we show the limitations of the sparse network approach for compactifying RL policies on the example of state-of-the-art class of algorithms from (Lenc et al., 2019) that aim to simultaneously train weights and connections of neural network architectures.
Revised: • In Subsection 4.1 we show the limitations of the sparse network approach for compactifying RL policies on the example of state-of-the-art class of algorithms from (Lenc et al., 2019) that aim to simultaneously train weights and connections of neural network architectures.

Original: This approach is conceptually the most similar to ours.
Revised: This approach is conceptually the most similar to ours.

Original: • In Subsection 4.2 we present exhaustive results on training our chromatic networks with ENAS on OpenAI Gym and quadruped locomotion tasks.
Revised: • In Subsection 4.2 we present exhaustive results on training our chromatic networks with ENAS on OpenAI Gym and quadruped locomotion tasks.

Original: We compare sizes and rewards obtained by our policies with those using masking procedure from (Lenc et al., 2019), applying low displacement rank matrices for compactiﬁcation as well as unstructured baselines.
Revised: We compare sizes and rewards obtained by our policies with those using masking procedure from (Lenc et al., 2019), applying low displacement rank matrices for compactiﬁcation as well as unstructured baselines.

Original: • In Subsection 4.3 we analyze in detail the impact of ENAS steps responsible for learning partitions, in particular compare it with the performance of random partitionings.
Revised: • In Subsection 4.3 we analyze in detail the impact of ENAS steps responsible for learning partitions, in particular compare it with the performance of random partitionings.

Original: We provide more experimental results in the Appendix.
Revised: We provide more experimental results in the Appendix.

Original: 4. L EARNED S PARSE N ETWORKS VIA S IMULTANEOUS W EIGHT -T OPOLOGY T RAINING
Revised: 4. L EARNED S PARSE N ETWORKS VIA S IMULTANEOUS W EIGHT -T OPOLOGY T RAINING

Original: The mask m , drawn from a multinomial distribution, is trained in (Lenc et al., 2019) using ES and element-wise multiplied by the weights before a forward pass.
Revised: The mask m , drawn from a multinomial distribution, is trained in (Lenc et al., 2019) using ES and element-wise multiplied by the weights before a forward pass.

Original: In (Lenc et al., 2019), the sparsity of the mask is ﬁxed, however, to show the effect of pruning, we instead initialize the sparsity at 50% and increasingly reward smaller networks (measured by the size of the mask | m | ) during optimization.
Revised: In (Lenc et al., 2019), the sparsity of the mask is ﬁxed, however, to show the effect of pruning, we instead initialize the sparsity at 50% and increasingly reward smaller networks (measured by the size of the mask | m | ) during optimization.

Original: Using this approach on several Open AI Gym tasks, we demonstrate that masking mechanism is capable of producing compact effective policies up to a high level of pruning.
Revised: Using this approach on several Open AI Gym tasks, we demonstrate that masking mechanism is capable of producing compact effective policies up to a high level of pruning.

Original: At the same time, we show signiﬁcant decrease of performance at the 80 - 90% compression level, quantifying accurately its limits for RL tasks (see: Fig. 3).
Revised: At the same time, we show signiﬁcant decrease of performance at the 80 - 90% compression level, quantifying accurately its limits for RL tasks (see: Fig. 3).

Original: [Figure]
Revised: [Figure]

Original: 4. R ESULTS ON C HROMATIC N ETWORKS
Revised: 4. R ESULTS ON C HROMATIC N ETWORKS

Original: We perform experiments on the following OpenAI Gym tasks: Swimmer , Reacher , Hopper ,
Revised: We perform experiments on the following OpenAI Gym tasks: Swimmer , Reacher , Hopper ,

Original: HalfCheetah , Walker2d , Pusher , Striker , Thrower and Ant as well as quadruped locomotion task of forward walking from (Iscen et al., 2018).
Revised: HalfCheetah , Walker2d , Pusher , Striker , Thrower and Ant as well as quadruped locomotion task of forward walking from (Iscen et al., 2018).

Original: The performance of our algorithm constructing chromatic networks is summarized in Table 1.
Revised: The performance of our algorithm constructing chromatic networks is summarized in Table 1.

Original: We tested three classes of feedforward architectures: linear from (Mania et al., 2018a), and nonlinear with one or two hidden layers and tanh nonlinearities.
Revised: We tested three classes of feedforward architectures: linear from (Mania et al., 2018a), and nonlinear with one or two hidden layers and tanh nonlinearities.

Original: We see a general trend that increasing hidden layers while keeping number of partitions ﬁxed, improves performance as well as increasing the number of partitions while keeping the architecture ﬁxed.
Revised: We see a general trend that increasing hidden layers while keeping number of partitions ﬁxed, improves performance as well as increasing the number of partitions while keeping the architecture ﬁxed.

Original: The relation between the expressiveness of a linear, high-partition policy vs a hidden-layer, low-partition policy is however not well understood.
Revised: The relation between the expressiveness of a linear, high-partition policy vs a hidden-layer, low-partition policy is however not well understood.

Original: As shown in Table 1, this depends on the environment’s complexity as well.
Revised: As shown in Table 1, this depends on the environment’s complexity as well.

Original: For HalfCheetah, the linear 50-partition policy performs better than a hidden layer 17-partition policy, while this is reversed for for the Minitaur.
Revised: For HalfCheetah, the linear 50-partition policy performs better than a hidden layer 17-partition policy, while this is reversed for for the Minitaur.

Original: In Table 2 we directly compare chromatic networks with a masking approach as discussed in Section 4.1, as well as other structured policies (Toeplitz from (Choromanski et al., 2018) and circulant) and the unstructured baseline.
Revised: In Table 2 we directly compare chromatic networks with a masking approach as discussed in Section 4.1, as well as other structured policies (Toeplitz from (Choromanski et al., 2018) and circulant) and the unstructured baseline.

Original: In all cases we use the same hyper-parameters, and train until convergence for ﬁve random seeds.
Revised: In all cases we use the same hyper-parameters, and train until convergence for ﬁve random seeds.

Original: For masking, we report the best achieved reward with > 90% of the network pruned, making the ﬁnal policy comparable in size to the chromatic network.
Revised: For masking, we report the best achieved reward with > 90% of the network pruned, making the ﬁnal policy comparable in size to the chromatic network.

Original: For each class of policies, we compare the number of weight parameters used (“# of weight-params” ﬁeld), since the compactiﬁcation mechanism does not operate on bias vectors.
Revised: For each class of policies, we compare the number of weight parameters used (“# of weight-params” ﬁeld), since the compactiﬁcation mechanism does not operate on bias vectors.

Original: We also record compression in respect to unstructured networks in terms of the total number of parameters (“# compression” ﬁeld).
Revised: We also record compression in respect to unstructured networks in terms of the total number of parameters (“# compression” ﬁeld).

Original: This number determines the reduction of sampling complexity with respect to unstructured networks (which is a bottleneck of ES training), since the number of RL blackbox function F queries needed to train/up-train the policy is proportional to the total number of weights and biases of the corresponding network [Table]
Revised: This number determines the reduction of sampling complexity with respect to unstructured networks (which is a bottleneck of ES training), since the number of RL blackbox function F queries needed to train/up-train the policy is proportional to the total number of weights and biases of the corresponding network.

Original: Finally, for a working policy we report total number of bits required to encode it assuming that real values are stored in the ﬂoat format.
Revised: Finally, for a working policy we report total number of bits required to encode it assuming that real values are stored in the ﬂoat format.

Original: Note that for chromatic and masking networks this includes bits required to encode a dictionary representing the partitioning.
Revised: Note that for chromatic and masking networks this includes bits required to encode a dictionary representing the partitioning.

Original: Further details are given in 
Revised: Further details are given in the

Revised: Appendix (Section E).

Revised: Top two performing networks for each environment are in bold.

Revised: Chromatic networks are the only to provide big compression and quality at the same time across all tasks.

Original: Inference Time: Similarly to Toeplitz, chromatic networks also provide computational gains.
Revised: Inference Time: Similarly to Toeplitz, chromatic networks also provide computational gains.

Original: Using improved version of the mailman-algorithm (Liberty & Zucker, 2009), matrix-vector multipli- [Table] cation part of the inference can be run on the chromatic network using constant number of distinct weights and deployed on real hardware in time O
Revised: Using improved version of the mailman-algorithm (Liberty & Zucker, 2009), matrix-vector multiplication part of the inference can be run on the chromatic network using constant number of distinct weights and deployed on real hardware in time O

Original: ( mn log(max( m,n )) ) , where ( m, n ) is the shape of the matrix [Table]
Revised: ( mn log(max( m,n )) ) , where ( m, n ) is the shape of the matrix.

Original: 4. F URTHER ANALYSIS : RANDOM PARTITIONINGS VERSUS ENAS
Revised: 4. F URTHER ANALYSIS : RANDOM PARTITIONINGS VERSUS ENAS

Original: A natural question to ask is whether ENAS machinery is required or maybe random partitioning is good enough to produce efﬁcient policies.
Revised: A natural question to ask is whether ENAS machinery is required or maybe random partitioning is good enough to produce efﬁcient policies.

Original: To answer it, we trained joint weights for ﬁxed population of random partitionings without NAS, as well as with random NAS controller. 
Revised: To answer it, we trained joint weights for ﬁxed population of random partitionings without NAS, as well as with random NAS controller. [Table] [Table]

Original: Our experiments show that these approaches fail by producing suboptimal policies for harder tasks (see: Fig. 4).
Revised: Our experiments show that these approaches fail by producing suboptimal policies for harder tasks (see: Fig. 4).

Original: Note also that if a partitioning distribution is ﬁxed throughout the entire optimization, training policies for such tasks and restarting to ﬁx another partitioning or distribution can cause substantial waste of computational resources, especially for the environments requiring long training time.
Revised: Note also that if a partitioning distribution is ﬁxed throughout the entire optimization, training policies for such tasks and restarting to ﬁx another partitioning or distribution can cause substantial waste of computational resources, especially for the environments requiring long training time.

Original: However we also notice an intriguing fact that random partitionings still lead to nontrivial rewards.
Revised: However we also notice an intriguing fact that random partitionings still lead to nontrivial rewards.

Original: We ﬁnd it similar to the conclusions in NAS for supervised learning (Pham et al., 2018) - while training a random child model sampled from a reasonable search space may produce ≥ 80 % accuracy, the most gains from a controller will ultimately be at the tail end; i.e. at the 95% accuracies.
Revised: We ﬁnd it similar to the conclusions in NAS for supervised learning (Pham et al., 2018) - while training a random child model sampled from a reasonable search space may produce ≥ 80 % accuracy, the most gains from a controller will ultimately be at the tail end; i.e. at the 95% accuracies.

Original: [Figure]
Revised: [Figure]

Original: We observe that by training with ENAS, the entire optimization beneﬁts in two ways by: (1) selecting partitionings leading to good rewards, (2) resampling good partitionings based on the controller replay buffer, and breaking through local minima inherently caused by the weight-sharing mechanism maximizing average reward.
Revised: We observe that by training with ENAS, the entire optimization beneﬁts in two ways by: (1) selecting partitionings leading to good rewards, (2) resampling good partitionings based on the controller replay buffer, and breaking through local minima inherently caused by the weight-sharing mechanism maximizing average reward.

Original: We see these beneﬁts precisely when a new ENAS iteration abruptly increases the reward by a large amount, which we present on Fig.
Revised: We see these beneﬁts precisely when a new ENAS iteration abruptly increases the reward by a large amount, which we present on Fig.

Original:  C ONCLUSION & F UTURE W ORK
Revised:  C ONCLUSION & F UTURE W ORK

Original: We presented a principled and ﬂexible algorithm for learning structured neural network architectures for RL policies and encoded by compact sets of parameters.
Revised: We presented a principled and ﬂexible algorithm for learning structured neural network architectures for RL policies and encoded by compact sets of parameters.

Original: Our architectures, called chromatic networks, rely on partitionings of a small sets of weights learned via ENAS methods.
Revised: Our architectures, called chromatic networks, rely on partitionings of a small sets of weights learned via ENAS methods.

Original: Furthermore, we have also provided a scalable way of performing NAS techniques with RL policies which is not limited to weight-sharing, but can potentially also be used to construct several other combinatorial structures in a ﬂexible fashion, such as node deletions and edge removals.
Revised: Furthermore, we have also provided a scalable way of performing NAS techniques with RL policies which is not limited to weight-sharing, but can potentially also be used to construct several other combinatorial structures in a ﬂexible fashion, such as node deletions and edge removals.

Original: We showed that chromatic networks provide more aggressive compression than their state-of-the-art counterparts while preserving efﬁciency of the learned policies.
Revised: We showed that chromatic networks provide more aggressive compression than their state-of-the-art counterparts while preserving efﬁciency of the learned policies.

Original: We believe that our work opens new research directions, especially from using other combinatorial objects.
Revised: We believe that our work opens new research directions, especially from using other combinatorial objects.

Original: Detailed analysis of obtained partitionings (see: Appendix C) also shows that learned structured matrices are very different from previously used state-of-the-art (in particular they are characterized by high displacement rank), yet it is not known what their properties are.
Revised: Detailed analysis of obtained partitionings (see: Appendix C) also shows that learned structured matrices are very different from previously used state-of-the-art (in particular they are characterized by high displacement rank), yet it is not known what their properties are.

Original: It would be also important to understand how transferable those learned partitionings are across different RL tasks (see: Appendix D).
Revised: It would be also important to understand how transferable those learned partitionings are across different RL tasks (see: Appendix D).

Original: M AXIMUM
Revised: M AXIMUM

Original: [Figure]
Revised: [Figure]

Original: EWARD
Revised: EWARD

Original: [Figure]
Revised: [Figure]

Original: URVES
Revised: URVES

Original: In order to view the maximum rewards achieved during the training process, for each worker at every NAS iteration, we record the maximum reward within the interval [ NAS iteration · T, ( NAS iteration +1 ) · T ) , where T stadns for the current number of conducted timestpes.
Revised: In order to view the maximum rewards achieved during the training process, for each worker at every NAS iteration, we record the maximum reward within the interval [ NAS iteration · T, ( NAS iteration +1 ) · T ) , where T stadns for the current number of conducted timestpes.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: B E XACT S ETUP AND H YPERPARAMETERS B. C ONTROLLER S ETUP
Revised: B E XACT S ETUP AND H YPERPARAMETERS B. C ONTROLLER S ETUP

Original: We set LSTM hidden layer size to be 64, with 1 hidden layer.
Revised: We set LSTM hidden layer size to be 64, with 1 hidden layer.

Original: The learning rate was 0 .
Revised: The learning rate was 0 .

Original: 001 , and the entropy penalty strength was 0 .
Revised: 001 , and the entropy penalty strength was 0 .

Original: We used a moving average weight of 0 . 99 for the critic, and used a temperature of 1 . 0 for softmax, with the training algorithm as REINFORCE.
Revised: We used a moving average weight of 0 . 99 for the critic, and used a temperature of 1 . 0 for softmax, with the training algorithm as REINFORCE.

Original: B. P OLICY
Revised: B. P OLICY

Original: We use tanh non-linearities.
Revised: We use tanh non-linearities.

Original: For all the environments, we used reward normalization, and state normalization from (Mania et al., 2018a) except for Swimmer.
Revised: For all the environments, we used reward normalization, and state normalization from (Mania et al., 2018a) except for Swimmer.

Original: We further used action normalization for the Minitaur tasks.
Revised: We further used action normalization for the Minitaur tasks.

Original: B. ES-A LGORITHM
Revised: B. ES-A LGORITHM

Original: For ES-optimization, we used algorithm from (Choromanski et al., 2018), where we applied
Revised: For ES-optimization, we used algorithm from (Choromanski et al., 2018), where we applied

Original: Monte Carlo estimators of the Gaussian smoothings to conduct gradient steps using forward ﬁnitedifference expressions.
Revised: Monte Carlo estimators of the Gaussian smoothings to conduct gradient steps using forward ﬁnitedifference expressions.

Original: Smoothing parameter σ and learning rate η were: σ = 0 .
Revised: Smoothing parameter σ and learning rate η were: σ = 0 .

Original: 1 , η = 0 .
Revised: 1 , η = 0 .

Original: C P ARTITIONINGS M ETRICS
Revised: C P ARTITIONINGS M ETRICS

Original: We analyzed obtained partitionings to understand whether they admit simpler representations and how they relate to the partitionings corresponding to Toeplitz-like matrices that are used on a regular basis for the compactiﬁcation of RL policies.
Revised: We analyzed obtained partitionings to understand whether they admit simpler representations and how they relate to the partitionings corresponding to Toeplitz-like matrices that are used on a regular basis for the compactiﬁcation of RL policies.

Original: C. S INGLE P ARTITIONING M ETRICS
Revised: C. S INGLE P ARTITIONING M ETRICS

Original: Partitionings’ Entropies: We analyze the distribution of color assignments for network edges for a partitioning, by interpreting the number of edges assigned for each color as a count, and therefore a probability after normalizing by the total number of edges.
Revised: Partitionings’ Entropies: We analyze the distribution of color assignments for network edges for a partitioning, by interpreting the number of edges assigned for each color as a count, and therefore a probability after normalizing by the total number of edges.

Original: We computed entropies of the corresponding probabilistic distributions encoding frequencies of particular colors.
Revised: We computed entropies of the corresponding probabilistic distributions encoding frequencies of particular colors.

Original: We noticed that entropies are large, in particular the representations will not substantially beneﬁt from further compaciﬁcation using Huffman coding (see: Fig. 12).
Revised: We noticed that entropies are large, in particular the representations will not substantially beneﬁt from further compaciﬁcation using Huffman coding (see: Fig. 12).

Original: [Figure]
Revised: [Figure]

Original: Displacement Rank: Recall that the displacement rank (Sindhwani et al., 2015; Demmel & Koev) of a matrix R with respect to two matrices: F, A is deﬁned as the rank of the resulting matrix ∇ F,A ( R ) =
Revised: Displacement Rank: Recall that the displacement rank (Sindhwani et al., 2015; Demmel & Koev) of a matrix R with respect to two matrices: F, A is deﬁned as the rank of the resulting matrix ∇ F,A ( R ) =

Original: FR − RA .
Revised: FR − RA .

Original: Toeplitz-type matrices are deﬁned as those that have displacement rankwith respect to speciﬁc band matrices (Sindhwani et al., 2015; Demmel & Koev).
Revised: Toeplitz-type matrices are deﬁned as those that have displacement rankwith respect to speciﬁc band matrices (Sindhwani et al., 2015; Demmel & Koev).

Original: We further analyze the displacement ranks of the weight matrices for our chromatic networks, and ﬁnd that they are full displacement rank using band matrices ( F, A ) for both the Toeplitz- and the Toeplitz-Hankel-type matrices (see: Fig. 13).
Revised: We further analyze the displacement ranks of the weight matrices for our chromatic networks, and ﬁnd that they are full displacement rank using band matrices ( F, A ) for both the Toeplitz- and the Toeplitz-Hankel-type matrices (see: Fig. 13).

Original: [Figure]
Revised: [Figure]

Original: C. P AIRWISE P ARTITIONINGS M ETRICS
Revised: C. P AIRWISE P ARTITIONINGS M ETRICS

Original: We examine the partitionings produced by the controller throughout the optimization by deﬁning different metrics in the space of the partitionings and analyzing convergence of the sequences of produced partitionings in these matrics.
Revised: We examine the partitionings produced by the controller throughout the optimization by deﬁning different metrics in the space of the partitionings and analyzing convergence of the sequences of produced partitionings in these matrics.

Original: We view partitionings as clusterings in the space of all edges of the network.
Revised: We view partitionings as clusterings in the space of all edges of the network.

Original: Thus we can use standard cluster similarity metrics such as RandIndex (Rand, 1971) and Variation of Information (Meil˘a, 2003).
Revised: Thus we can use standard cluster similarity metrics such as RandIndex (Rand, 1971) and Variation of Information (Meil˘a, 2003).

Original: Distance metric counts the number of edges that reside in different clusters (indexed with the indices of the vector of distinct weights) in two compared partitionings/clusterings.
Revised: Distance metric counts the number of edges that reside in different clusters (indexed with the indices of the vector of distinct weights) in two compared partitionings/clusterings.

Original: We do not observe any convergence in the analyzed metrics (see: Fig.14).
Revised: We do not observe any convergence in the analyzed metrics (see: Fig.14).

Original: This suggests that the space of the partitionings found in training is more complex.
Revised: This suggests that the space of the partitionings found in training is more complex.

Original: We leave its analysis for future work.
Revised: We leave its analysis for future work.

Original: [Figure]
Revised: [Figure]

Original: D T RANSFERABILITY
Revised: D T RANSFERABILITY

Original: We tested transferability of partitionings across different RL tasks by using the top-5 partitionings (based on maximal reward) from HalfCheetah (one-hidden-layer network of size h = 41 ), and using them to train distinct weights (for the inherited partitionings) using vanilla-ES for Walker2d (both environments have state and action vectors of the same sizes).
Revised: We tested transferability of partitionings across different RL tasks by using the top-5 partitionings (based on maximal reward) from HalfCheetah (one-hidden-layer network of size h = 41 ), and using them to train distinct weights (for the inherited partitionings) using vanilla-ES for Walker2d (both environments have state and action vectors of the same sizes).

Original: This is compared with the results when random partitionings were applied.
Revised: This is compared with the results when random partitionings were applied.

Original: Results are presented on Fig.15.
Revised: Results are presented on Fig.15.

Original: We notice that transfered partitionings do not underperform.
Revised: We notice that transfered partitionings do not underperform.

Original: We leave understanding the scale in which these learned partitionings can be transfered across tasks to future work.
Revised: We leave understanding the scale in which these learned partitionings can be transfered across tasks to future work.

Original: [Figure]
Revised: [Figure]

Original: E E XPERIMENT S ETUP FOR B ASELINES IN T ABLE
Revised: E E XPERIMENT S ETUP FOR B ASELINES IN T ABLE

Original: We compare the Chromatic network with other established frameworks for structrued neural network architectures.
Revised: We compare the Chromatic network with other established frameworks for structrued neural network architectures.

Original: In particular, we consider Unstructured, Toeplitz, Circulant and a masking mechanism (Choromanski et al., 2018;
Revised: In particular, we consider Unstructured, Toeplitz, Circulant and a masking mechanism (Choromanski et al., 2018;

Original: Lenc et al., 2019).
Revised: Lenc et al., 2019).

Original: We introduce their details below.
Revised: We introduce their details below.

Original: Notice that all baseline networks share the same general architecture: 1 -hidden layer with h =units and tanh non-linear activation.
Revised: Notice that all baseline networks share the same general architecture: 1 -hidden layer with h =units and tanh non-linear activation.

Original: To be concrete, we only have two weight matrices W 1 ∈ R
Revised: To be concrete, we only have two weight matrices W 1 ∈ R

Original: |S|× h , W 2 ∈ R h ×|A| and two bias vectors b 1 ∈ R h , b 2 ∈ R |A| , where |S| , |A| are dimensions of state/action spaces.
Revised: |S|× h , W 2 ∈ R h ×|A| and two bias vectors b 1 ∈ R h , b 2 ∈ R |A| , where |S| , |A| are dimensions of state/action spaces.

Original: These networks differ in how they parameterize the weight matrices.
Revised: These networks differ in how they parameterize the weight matrices.

Original: Unstructured.
Revised: Unstructured.

Original: A fully-connected layer with unstructured weight matrix W ∈ R a × b has a total of ab independent parameters.
Revised: A fully-connected layer with unstructured weight matrix W ∈ R a × b has a total of ab independent parameters.

Original: Toeplitz.
Revised: Toeplitz.

Original: A toeplitz weight matrix W ∈ R a × b has a total of a + b − 1 independent parameters.
Revised: A toeplitz weight matrix W ∈ R a × b has a total of a + b − 1 independent parameters.

Original: This architecture has been shown to be effective in generating good performance on benchmark tasks yet compressing parameters (Choromanski et al., 2018).
Revised: This architecture has been shown to be effective in generating good performance on benchmark tasks yet compressing parameters (Choromanski et al., 2018).

Original: Circulant.
Revised: Circulant.

Original: A circulant weight matrix W ∈ R a × b is deﬁned for square matrices a = b .
Revised: A circulant weight matrix W ∈ R a × b is deﬁned for square matrices a = b .

Original: We generalize this deﬁnition by considering a square matrix of size n × n where n = max { a, b } and then do a proper truncation.
Revised: We generalize this deﬁnition by considering a square matrix of size n × n where n = max { a, b } and then do a proper truncation.

Original: This produces n independent parameters.
Revised: This produces n independent parameters.

Original: Masking.
Revised: Masking.

Original: One additional technique for reducing the number of independent parameters in a weight matrix is to mask out redundant parameters (Lenc et al., 2019).
Revised: One additional technique for reducing the number of independent parameters in a weight matrix is to mask out redundant parameters (Lenc et al., 2019).

Original: This slightly differs from the other aforementioned architectures since these other architectures allow for parameter sharing while the masking mechanism carries out pruning.
Revised: This slightly differs from the other aforementioned architectures since these other architectures allow for parameter sharing while the masking mechanism carries out pruning.

Original: To be concrete, we consider a fully-connected matrix
Revised: To be concrete, we consider a fully-connected matrix

Original: W ∈ R a × b with ab independent parameters.
Revised: W ∈ R a × b with ab independent parameters.

Original: We also setup another mask weight S ∈ R a × b .
Revised: We also setup another mask weight S ∈ R a × b .

Original: Then the mask is generated via
Revised: Then the mask is generated via

Original: [Equation]
Revised: [Equation]

Original: where softmax is applied elementwise and α is a constant.
Revised: where softmax is applied elementwise and α is a constant.

Original: We set α = 0 .
Revised: We set α = 0 .

Original: so that the softmax is effectively a thresolding function wich outputs near binary masks.
Revised: so that the softmax is effectively a thresolding function wich outputs near binary masks.

Original: We then treat the entire concatenated parameter θ =
Revised: We then treat the entire concatenated parameter θ =

Original: [ W, S ] as trainable parameters and optimize both using ES methods.
Revised: [ W, S ] as trainable parameters and optimize both using ES methods.

Original: At convergence, the effective number of parameter is ab · η where η is the proportion of M components that are non-zero.
Revised: At convergence, the effective number of parameter is ab · η where η is the proportion of M components that are non-zero.

Original: During optimization, we implement a simple heuristics that encourage sparse network: while maximize the true environment return R = (cid:80) t r t , we also maximize the number of mask entries that are zero 1 − η .
Revised: During optimization, we implement a simple heuristics that encourage sparse network: while maximize the true environment return R = (cid:80) t r t , we also maximize the number of mask entries that are zero 1 − η .

Original: The ultimate ES objective is: R (cid:48)
Revised: The ultimate ES objective is: R (cid:48)

Original: =
Revised: =

Original: β · R + (1 − β ) · (1 − η ) , where β ∈ [0 , 1] is a combination coefﬁcient which we anneal as training progresses.
Revised: β · R + (1 − β ) · (1 − η ) , where β ∈ [0 , 1] is a combination coefﬁcient which we anneal as training progresses.

Original: We also properly normalize R and (1 − η ) before the linear combination to ensure that the procedure is not sensitive to reward scaling.
Revised: We also properly normalize R and (1 − η ) before the linear combination to ensure that the procedure is not sensitive to reward scaling.