Original: RL X 2: T RAINING A S PARSE D EEP R EINFORCEMENT L EARNING M ODEL FROM S CRATCH
Revised: RL X 2: T RAINING A S PARSE D EEP R EINFORCEMENT L EARNING M ODEL FROM S CRATCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training deep reinforcement learning (DRL) models usually require high computation costs.
Revised: Training deep reinforcement learning (DRL) models usually require high computation costs.

Original: Therefore, compressing DRL models possesses immense potential for training acceleration and model deployment.
Revised: Therefore, compressing DRL models possesses immense potential for training acceleration and model deployment.

Original: However, existing methods that generate small models mainly adopt the knowledge distillation-based approach by iteratively training a dense network.
Revised: However, existing methods that generate small models mainly adopt the knowledge distillation-based approach by iteratively training a dense network.

Original: As a result, the training process still demands massive computing resources.
Revised: As a result, the training process still demands massive computing resources.

Original: Indeed, sparse training from scratch in DRL has not been well explored and is particularly challenging due to non-stationarity in bootstrap training.
Revised: Indeed, sparse training from scratch in DRL has not been well explored and is particularly challenging due to non-stationarity in bootstrap training.

Original: In this work, we propose a novel sparse DRL training framework, “the R igged R einforcement L earning L ottery” (RLx2), which builds upon gradient-based topology evolution and is capable of training a sparse DRL model based entirely on a sparse network.
Revised: In this work, we propose a novel sparse DRL training framework, “the R igged R einforcement L earning L ottery” (RLx2), which builds upon gradient-based topology evolution and is capable of training a sparse DRL model based entirely on a sparse network.

Original: Specifically, RLx2 introduces a novel multistep TD target mechanism with a dynamic-capacity replay buffer to achieve robust value learning and efficient topology exploration in sparse models.
Revised: Specifically, RLx2 introduces a novel delayed multi-step TD target mechanism with a dynamic-capacity replay buffer to achieve robust value learning and efficient topology exploration in sparse models.

Original: It also reaches state-of-the-art sparse training performance in several tasks, showing 7 . 5 × - 20 × model compression with less than 3% performance degradation and up to 20 × and 50 × FLOPs reduction for training and inference, respectively.
Revised: It also reaches state-of-the-art sparse training performance in several tasks, showing 7 . 5 × - 20 × model compression with less than 3% performance degradation and up to 20 × and 50 × FLOPs reduction for training and inference, respectively.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning (DRL) has found successful applications in many important areas, e.g., games (Silver et al., 2017), robotics(Gu et al., 2017) and nuclear fusion (Degrave et al., 2022).
Revised: Deep reinforcement learning (DRL) has found successful applications in many important areas, e.g., games (Silver et al., 2017), robotics(Gu et al., 2017) and nuclear fusion (Degrave et al., 2022).

Original: However, training a DRL model demands heavy computational resources.
Revised: However, training a DRL model demands heavy computational resources.

Original: For instance, AlphaGo-Zero for Go games (Silver et al., 2017), which defeats all Go-AIs and human experts, requires more than 40 days of training time on four tensor processing units (TPUs).
Revised: For instance, AlphaGo-Zero for Go games (Silver et al., 2017), which defeats all Go-AIs and human experts, requires more than 40 days of training time on four tensor processing units (TPUs).

Original: The heavy resource requirement results in expensive consumption and hinders the application of DRL on resource-limited devices.
Revised: The heavy resource requirement results in expensive consumption and hinders the application of DRL on resource-limited devices.

Original: Sparse networks, initially proposed in deep supervised learning, have demonstrated great potential for model compression and training acceleration of deep reinforcement learning.
Revised: Sparse networks, initially proposed in deep supervised learning, have demonstrated great potential for model compression and training acceleration of deep reinforcement learning.

Original: Specifically, in deep supervised learning, the state-of-the-art sparse training frameworks, e.g., SET (Mocanu et al., 2018) and
Revised: Specifically, in deep supervised learning, the state-of-the-art sparse training frameworks, e.g., SET (Mocanu et al., 2018) and

Original: RigL
Revised: RigL

Original: (Evci et al., 2020), can train a 90% -sparse network (i.e., the resulting network size is 10% of the original network) from scratch without performance degradation.
Revised: (Evci et al., 2020), can train a 90% -sparse network (i.e., the resulting network size is 10% of the original network) from scratch without performance degradation.

Original: On the DRL side, existing works including Rusu et al.
Revised: On the DRL side, existing works including Rusu et al.

Original: ; Schmitt et al.
Revised: ; Schmitt et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2019) succeeded in generating ultimately sparse DRL networks.
Revised: (2019) succeeded in generating ultimately sparse DRL networks.

Original: Yet, their approaches still require iteratively training dense networks, e.g., pre-trained dense teachers may be needed.
Revised: Yet, their approaches still require iteratively training dense networks, e.g., pre-trained dense teachers may be needed.

Original: As a result, the training cost for DRL remains prohibitively high, and existing methods cannot be directly implemented on resource-limited devices, leading to low flexibility in adapting the compressed DRL models to new environments, i.e., on-device models have to be retrained at large servers and re-deployed.
Revised: As a result, the training cost for DRL remains prohibitively high, and existing methods cannot be directly implemented on resource-limited devices, leading to low flexibility in adapting the compressed DRL models to new environments, i.e., on-device models have to be retrained at large servers and re-deployed.

Original: Training a sparse DRL model from scratch, if done perfectly, has the potential to significantly reduce computation expenditure and enable efficient deployment on resource-limited devices, and achieves excellent flexibility in model adaptation.
Revised: Training a sparse DRL model from scratch, if done perfectly, has the potential to significantly reduce computation expenditure and enable efficient deployment on resource-limited devices, and achieves excellent flexibility in model adaptation.

Original: However, training an ultra sparse network (e.g., 90% sparsity) from scratch in DRL is challenging due to the non-stationarity in bootstrap training.
Revised: However, training an ultra sparse network (e.g., 90% sparsity) from scratch in DRL is challenging due to the non-stationarity in bootstrap training.

Original: Specifically, in DRL, the learning target is not fixed but evolves in a bootstrap way (Tesauro et al., 1995), and the distribution of the training data can also be non-stationary (Desai et al., 2019).
Revised: Specifically, in DRL, the learning target is not fixed but evolves in a bootstrap way (Tesauro et al., 1995), and the distribution of the training data can also be non-stationary (Desai et al., 2019).

Original: Moreover, using a sparse network structure means searching in a smaller hypothesis (topology) space, which further reduces the learning target’s confidence.
Revised: Moreover, using a sparse network structure means searching in a smaller hypothesis (topology) space, which further reduces the learning target’s confidence.

Original: As a result, improper sparsification can cause irreversible damage to the learning path (Igl et al., 2021), resulting in poor performance.
Revised: As a result, improper sparsification can cause irreversible damage to the learning path (Igl et al., 2021), resulting in poor performance.

Original: Indeed, recent works (Sokar et al., 2021; Graesser et al., 2022) show that a direct adoption of a dynamic sparse training (DST) framework in DRL still fails to achieve good compression of the model for different environments uniformly.
Revised: Indeed, recent works (Sokar et al., 2021; Graesser et al., 2022) show that a direct adoption of a dynamic sparse training (DST) framework in DRL still fails to achieve good compression of the model for different environments uniformly.

Original: Therefore, the following interesting question remains open:
Revised: Therefore, the following interesting question remains open:

Original: Can an efficient DRL agent be trained from scratch with an ultra-sparse network throughout?
Revised: Can an efficient DRL agent be trained from scratch with an ultra-sparse network throughout?

Original: In this paper, we give an affirmative answer to the problem and propose a novel sparse training framework, “the R igged R einforcement L earning L ottery” (RLx2), for off-policy RL, which is the first algorithm to achieve sparse training throughout using sparsity of more than 90% with only minimal performance loss.
Revised: In this paper, we give an affirmative answer to the problem and propose a novel sparse training framework, “the R igged R einforcement L earning L ottery” (RLx2), for off-policy RL, which is the first algorithm to achieve sparse training throughout using sparsity of more than 90% with only minimal performance loss.

Original: RLx2 is inspired by the gradient-based topology evolution criteria in
Revised: RLx2 is inspired by the gradient-based topology evolution criteria in

Original: RigL (Evci et al., 2020) for supervised learning.
Revised: RigL (Evci et al., 2020) for supervised learning.

Original: However, a direct application of RigL does not achieve high sparsity, because sparse DRL models suffer from unreliable value estimation due to limited hypothesis space, which further disturbs topology evolution.
Revised: However, a direct application of RigL does not achieve high sparsity, because sparse DRL models suffer from unreliable value estimation due to limited hypothesis space, which further disturbs topology evolution.

Original: Thus, RLx2 is equipped with a  multi-step Temporal Difference (TD) target mechanism and a novel dynamic-capacity replay buffer to achieve robust value learning and efficient topology exploration.
Revised: Thus, RLx2 is equipped with a delayed multi-step Temporal Difference (TD) target mechanism and a novel dynamic-capacity replay buffer to achieve robust value learning and efficient topology exploration.

Original: These two new components address the value estimation problem under sparse topology, and together with RigL, achieve superior sparse-training performance.
Revised: These two new components address the value estimation problem under sparse topology, and together with RigL, achieve superior sparse-training performance.

Original: The main contributions of the paper are summarized as follows.
Revised: The main contributions of the paper are summarized as follows.

Original: • We investigate the fundamental obstacles in training a sparse DRL agent from scratch, and discover two key factors for achieving good performance under sparse networks, namely robust value estimation and efficient topology exploration.
Revised: • We investigate the fundamental obstacles in training a sparse DRL agent from scratch, and discover two key factors for achieving good performance under sparse networks, namely robust value estimation and efficient topology exploration.

Original: • Motivated by our findings, we propose RLx2, the first framework that enables DRL training based entirely on sparse networks.
Revised: • Motivated by our findings, we propose RLx2, the first framework that enables DRL training based entirely on sparse networks.

Original: RLx2 possesses two key functions, i
Revised: RLx2 possesses two key functions, i

Original: e., a gradient-based search scheme for efficient topology exploration, and a  multi-step TD target mechanism with a dynamic-capacity replay buffer for robust value learning.
Revised: e., a gradientbased search scheme for efficient topology exploration, and a delayed multi-step TD target mechanism with a dynamic-capacity replay buffer for robust value learning.

Original: • Through extensive experiments, we demonstrate the state-of-the-art sparse training performance of RLx2 with two popular DRL algorithms, TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018), on several MuJoCo
Revised: • Through extensive experiments, we demonstrate the state-of-the-art sparse training performance of RLx2 with two popular DRL algorithms, TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018), on several MuJoCo

Original: (Todorov et al., 2012) continuous control tasks.
Revised: (Todorov et al., 2012) continuous control tasks.

Original: Our results show up to 20 × model compression.
Revised: Our results show up to 20 × model compression.

Original: RLx2 also achieves 20 × acceleration in training and 50 × in inference in terms of floating-point operations (FLOPs).
Revised: RLx2 also achieves 20 × acceleration in training and 50 × in inference in terms of floating-point operations (FLOPs).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: We discuss the related works on training sparse models in deep supervised learning and reinforcement learning below.
Revised: We discuss the related works on training sparse models in deep supervised learning and reinforcement learning below.

Original: We also provide a comprehensive performance comparison in Table 1.
Revised: We also provide a comprehensive performance comparison in Table 1.

Original: Sparse Models in Deep Supervised Learning Han et al. (2015; 2016); Srinivas et al.
Revised: Sparse Models in Deep Supervised Learning Han et al. (2015; 2016); Srinivas et al.

Original: (2017); Zhu & Gupta (2018) focus on finding a sparse network by pruning pre-trained dense networks.
Revised: (2017); Zhu & Gupta (2018) focus on finding a sparse network by pruning pre-trained dense networks.

Original: Iterative
Revised: Iterative

Original: Magnitude Pruning (IMP) in Han et al. (2016) achieves a sparsity of more than 90% .
Revised: Magnitude Pruning (IMP) in Han et al. (2016) achieves a sparsity of more than 90% .

Original: Techniques including neuron characteristic (Hu et al., 2016), dynamic network surgery (Guo et al., 2016), derivatives (Dong et al., 2017; Molchanov et al., 2019b), regularization (Louizos et al., 2018; Tartaglione et al., 2018), and dropout (Molchanov et al., 2017) have also been applied in network pruning.
Revised: Techniques including neuron characteristic (Hu et al., 2016), dynamic network surgery (Guo et al., 2016), derivatives (Dong et al., 2017; Molchanov et al., 2019b), regularization (Louizos et al., 2018; Tartaglione et al., 2018), and dropout (Molchanov et al., 2017) have also been applied in network pruning.

Original: Another line of work focuses on the Lottery Ticket Hypothesis (LTH), first proposed in Frankle &
Revised: Another line of work focuses on the Lottery Ticket Hypothesis (LTH), first proposed in Frankle &

Original: Carbin (2019), which shows that training from a sparse network from scratch is possible if one finds a sparse “winning ticket” initialization in deep supervised learning.
Revised: Carbin (2019), which shows that training from a sparse network from scratch is possible if one finds a sparse “winning ticket” initialization in deep supervised learning.

Original: The LTH is also validated in other deep learning models (Chen et al., 2020; Brix et al., 2020; Chen et al., 2021).
Revised: The LTH is also validated in other deep learning models (Chen et al., 2020; Brix et al., 2020; Chen et al., 2021).

Original: Many works (Bellec et al., 2017; Mocanu et al., 2018; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019; Evci et al., 2020) also try to train a sparse neural network from scratch without having to pre-trained dense models.
Revised: Many works (Bellec et al., 2017; Mocanu et al., 2018; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019; Evci et al., 2020) also try to train a sparse neural network from scratch without having to pre-trained dense models.

Original: These works adjust structures of sparse networks during training, including Deep Rewiring (DeepR) (Bellec et al., 2017),
Revised: These works adjust structures of sparse networks during training, including Deep Rewiring (DeepR) (Bellec et al., 2017),

Original: Sparse Evolutionary Training (SET) (Mocanu et al., 2018), Dynamic Sparse Reparameterization (DSR) (Mostafa & Wang, 2019), Sparse
Revised: Sparse Evolutionary Training (SET) (Mocanu et al., 2018), Dynamic Sparse Reparameterization (DSR) (Mostafa & Wang, 2019), Sparse

Original: Networks from Scratch (SNFS) (Dettmers & Zettlemoyer, 2019), and Rigged Lottery (RigL)
Revised: Networks from Scratch (SNFS) (Dettmers & Zettlemoyer, 2019), and Rigged Lottery (RigL)

Original: (Evci et al., 2020).
Revised: (Evci et al., 2020).

Original: Works Single-Shot Network Pruning (SNIP)
Revised: Works Single-Shot Network Pruning (SNIP)

Original: (Lee et al., 2019) and Gradient Signal
Revised: (Lee et al., 2019) and Gradient Signal

Original: Preservation (GraSP) (Wang et al., 2020) focus on finding static sparse networks before training. 
Revised: Preservation (GraSP) (Wang et al., 2020) focus on finding static sparse networks before training. [Table]

Original: [Table]
Revised: [Table]

Original: Sparse Models in DRL Evci et al.
Revised: Sparse Models in DRL Evci et al.

Original: Sokar et al.
Revised: Sokar et al.

Original: (2021) show that finding a sparse model in
Revised: (2021) show that finding a sparse model in

Original: DRL is difficult due to training instability.
Revised: DRL is difficult due to training instability.

Original: Existing works (Rusu et al., [Table]
Revised: Existing works (Rusu et al., 2016;

Original: Schmitt et al.,
Revised: Schmitt et al.,

Original: [Table]
Revised: [Table]

Original: Zhang et al., 2019) leverage knowledge distillation with static data to avoid unstable training and obtain small dense agents.
Revised: Zhang et al., 2019) leverage knowledge distillation with static data to avoid unstable training and obtain small dense agents.

Original: Policy Pruning and Shrinking (PoPs) in Livne & Cohen (2020) obtains a sparse DRL agent with iterative policy pruning (similar to IMP).
Revised: Policy Pruning and Shrinking (PoPs) in Livne & Cohen (2020) obtains a sparse DRL agent with iterative policy pruning (similar to IMP).

Original: LTH in DRL is firstly investigated in Yu et al.
Revised: LTH in DRL is firstly investigated in Yu et al.

Original: (2020), and then Vischer et al.
Revised: (2020), and then Vischer et al.

Original: (2022) shows that a sparse winning ticket can also be found by behavior cloning (BC).
Revised: (2022) shows that a sparse winning ticket can also be found by behavior cloning (BC).

Original: Another line of works (Lee et al., 2021; Sokar et al., 2021; Arnob et al., 2021) attempts to train a sparse neural network from scratch without pre-training a dense teacher.
Revised: Another line of works (Lee et al., 2021; Sokar et al., 2021; Arnob et al., 2021) attempts to train a sparse neural network from scratch without pre-training a dense teacher.

Original: Group Sparse Training (GST) in Lee et al.
Revised: Group Sparse Training (GST) in Lee et al.

Original: (2021) utilizes block-circuit compression and pruning.
Revised: (2021) utilizes block-circuit compression and pruning.

Original: Sokar et al.
Revised: Sokar et al.

Original: (2021) proposes using SET in topology evolution in DRL and achieves 50% sparsity.
Revised: (2021) proposes using SET in topology evolution in DRL and achieves 50% sparsity.

Original: Arnob et al.
Revised: Arnob et al.

Original: (2021) proposes single-shot pruning (SSP) for offline RL.
Revised: (2021) proposes single-shot pruning (SSP) for offline RL.

Original: Graesser et al.
Revised: Graesser et al.

Original: (2022) finds that pruning often obtains the best results and plain dynamic sparse training methods, including SET and RigL, improves over static sparse training significantly.
Revised: (2022) finds that pruning often obtains the best results and plain dynamic sparse training methods, including SET and RigL, improves over static sparse training significantly.

Original: However, existing works either demands massive computing resources, e.g. pruning-based methods (Rusu et al., 2016; Schmitt et al., 2018;
Revised: However, existing works either demands massive computing resources, e.g. pruning-based methods (Rusu et al., 2016; Schmitt et al., 2018;

Original: Zhang et al., 2019; Livne & Cohen, 2020), or fail in ultra sparse models, e.g. DST-based methods (Sokar et al., 2021; Graesser et al., 2022).
Revised: Zhang et al., 2019; Livne & Cohen, 2020), or fail in ultra sparse models, e.g. DST-based methods (Sokar et al., 2021; Graesser et al., 2022).

Original: In this paper, we further improve the performance of DST by introducing a  multi-step TD target mechanism with a dynamic-capacity replay buffer, which effectively addresses the unreliability of fixed-topology models during sparse training.
Revised: In this paper, we further improve the performance of DST by introducing a delayed multi-step TD target mechanism with a dynamic-capacity replay buffer, which effectively addresses the unreliability of fixed-topology models during sparse training.

Original:  D EEP R EINFORCEMENT L EARNING P RELIMINARIES
Revised:  D EEP R EINFORCEMENT L EARNING P RELIMINARIES

Original: In reinforcement learning, an agent interacts with an unknown environment to learn an optimal policy.
Revised: In reinforcement learning, an agent interacts with an unknown environment to learn an optimal policy.

Original: The learning process is formulated as a Markov decision process (MDP) M = ⟨S , A , r, P , γ ⟩ , where S is the state space, A is the action space, r is the reward function, P denotes the transition matrix, and γ stands for the discount factor.
Revised: The learning process is formulated as a Markov decision process (MDP) M = ⟨S , A , r, P , γ ⟩ , where S is the state space, A is the action space, r is the reward function, P denotes the transition matrix, and γ stands for the discount factor.

Original: Specifically, at time slot t , given the current state s t ∈ S , the agent selects an action a t ∈
Revised: Specifically, at time slot t , given the current state s t ∈ S , the agent selects an action a t ∈

Original: A by policy π : S → A , which then incurs a reward r t ( s, a ) .
Revised: A by policy π : S → A , which then incurs a reward r t ( s, a ) .

Original: Denote the Q function associated with the policy π for state-action pair ( s, a ) as
Revised: Denote the Q function associated with the policy π for state-action pair ( s, a ) as

Original: [Equation]
Revised: [Equation]

Original:  In actor-critic methods (Silver et al., 2014), the policy π ( s ; ϕ ) is parameterized by a policy (actor) network with weight parameter ϕ , and the Q function Q ( s, a ; θ ) is parameterized by a value (critic) network with parameter θ .
Revised: i [Equation] [Equation] In actor-critic methods (Silver et al., 2014), the policy π ( s ; ϕ ) is parameterized by a policy (actor) network with weight parameter ϕ , and the Q function Q ( s, a ; θ ) is parameterized by a value (critic) network with parameter θ .

Original: The goal of the agent is to find an optimal policy π ∗ ( s ; ϕ ∗ ) which maximizes long-term cumulative reward, i.e., J ∗ = max ϕ E π ( ϕ ) [ (cid:80) Ti =0 γ{+T i =
+}
Revised: The goal of the agent is to find an optimal policy π ∗ ( s ; ϕ ∗ ) which maximizes long-term cumulative reward, i.e., J ∗ = max ϕ E π ( ϕ ) [ (cid:80) {+T i =
+}

Original:  i − t r
Revised: γ i − t r

Original: ( s i , a i ) | s 0 , a 0 ] .
Revised: ( s i , a i ) | s 0 , a 0 ] .

Original: There are various DRL methods for learning an efficient policy.
Revised: There are various DRL methods for learning an efficient policy.

Original: In this paper, we focus on off-policy
Revised: In this paper, we focus on off-policy

Original: TD learning methods, including a broad range of state-of-the-art algorithms, e.g., TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018).
Revised: TD learning methods, including a broad range of state-of-the-art algorithms, e.g., TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018).

Original: Specifically, the critic network is updated by gradient descent to fit the TD targets T 1 ( s, a ) generated by a target network Q ( s, a ; θ ′ ) , i.e., 
Revised: Specifically, the critic network is updated by gradient descent to fit the TD targets T 1 ( s, a ) generated by a target network Q ( s, a ; θ ′ ) , i.e., [Equation]

Original: [Equation]
Revised: [Equation]

Original: for each state-action pair ( s, a ) , where the action a = π ( s ; ϕ ) .
Revised: for each state-action pair ( s, a ) , where the action a = π ( s ; ϕ ) .

Original: The loss function of the value network is defined as the expected squared loss between the current value network and TD targets:
Revised: The loss function of the value network is defined as the expected squared loss between the current value network and TD targets:

Original: [Equation]
Revised: [Equation]

Original: The policy π ( s ; ϕ ) is updated by the deterministic policy gradient algorithm in Silver et al.
Revised: The policy π ( s ; ϕ ) is updated by the deterministic policy gradient algorithm in Silver et al.

Original: [Equation]
Revised: [Equation]

Original:  RL X 2: R IGGING THE L OTTERY IN DRL
Revised:  RL X 2: R IGGING THE L OTTERY IN DRL

Original: In this section, we present the RLx2 algorithm, which is capable of training a sparse DRL model from scratch.
Revised: In this section, we present the RLx2 algorithm, which is capable of training a sparse DRL model from scratch.

Original: An overview of the RLx2 framework on an actor-critic architecture is shown in Figure 1.
Revised: An overview of the RLx2 framework on an actor-critic architecture is shown in Figure 1.

Original: To motivate the design of RLx2, we present a comparison of four sparse DRL training methods using TD3 with different topology update schemes on InvertedPendulum-v2, a simple control task from MuJoCo, in Figure 2.
Revised: To motivate the design of RLx2, we present a comparison of four sparse DRL training methods using TD3 with different topology update schemes on InvertedPendulum-v2, a simple control task from MuJoCo, in Figure 2.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: From the results, we make the following important observations.
Revised: From the results, we make the following important observations.

Original: (i) Topology evolution is crucial .
Revised: (i) Topology evolution is crucial .

Original: It can be seen that a random static sparse network (SS) leads to much worse performance than RigL.
Revised: It can be seen that a random static sparse network (SS) leads to much worse performance than RigL.

Original: (ii)
Revised: (ii)

Original: Robust value estimation is essential
Revised: Robust value estimation is essential

Original: This 
Revised: This is

Original: is validated
Revised: validated

Original: by the comparison between
Revised: by the comparison between

Original: RigL
Revised: RigL

Original: and
Revised: and

Original: RigL+ Q ∗ , both using the same topology adjustment scheme but with different Q-values
Revised: RigL+ Q ∗ , both using the same topology adjustment scheme but with different Q-values

Original: Motivated by the above findings, RLx2 utilizes gradient-based topology adjustment, i.e., RigL (for topology evolution), and introduces a  multi-step TD target mechanism with a dynamic-capacity replay buffer (for robust value estimation).
Revised: Motivated by the above findings, RLx2 utilizes gradient-based topology adjustment, i.e., RigL (for topology evolution), and introduces a delayed multi-step TD target mechanism with a dynamiccapacity replay buffer (for robust value estimation).

Original: Below, we explain the key components of RLx2 in detail, to illustrate why RLx2 is capable of achieving robust value learning and efficient topology exploration simultaneously.
Revised: Below, we explain the key components of RLx in detail, to illustrate why RLx2 is capable of achieving robust value learning and efficient topology exploration simultaneously.

Original: 4. G RADIENT - BASED T OPOLOGY E VOLUTION
Revised: 4. G RADIENT - BASED T OPOLOGY E VOLUTION

Original: The topology evolution in RLx2 is conducted by adopting the RigL method (Evci et al., 2020).
Revised: The topology evolution in RLx2 is conducted by adopting the RigL method (Evci et al., 2020).

Original: Specifically, we compute the gradient values of the loss function with respect to link weights.
Revised: Specifically, we compute the gradient values of the loss function with respect to link weights.

Original: Then, we dynamically grow connections (connecting neurons) with large gradients and delete existing links with the smallest  weights.
Revised: Then, we dynamically grow connections (connecting neurons) with large gradients and delete existing links with the smallest absolute value of the weights.

Original: In this way, we obtain a sparse mask that evolves by self-adjustment.
Revised: In this way, we obtain a sparse mask that evolves by self-adjustment.

Original: The pseudo-code of our scheme is given in Algorithm 1, where ⊙ is an element-wise multiplication operator and M ϕ is a binary mask to represent the sparse architecture of the network ϕ .
Revised: The pseudo-code of our scheme is given in Algorithm 1, where ⊙ is an element-wise multiplication operator and M ϕ is a binary mask to represent the sparse architecture of the network ϕ .

Original: The update fraction anneals during the training process according to ζ t = ζ 0 2 (1 + cos( πtT end )) , where ζ 0 is the initial update fraction and T end is the total number of iterations.
Revised: The update fraction anneals during the training process according to ζ t = ζ 0 2 (1 + cos( πtT end )) , where ζ 0 is the initial update fraction and T end is the total number of iterations.

Original: Note that this topology adjustment happens very infrequently, i.e., every 10000 step, such that consumption of this step is negligible (detailed analysis in Appendix C.3).
Revised: Note that this topology adjustment happens very infrequently, i.e., every 10000 step, such that consumption of this step is negligible (detailed analysis in Appendix C.3).

Original: Also, finding top- k links with maximum gradients in Line 10 can be efficiently implemented (detailed in Appendix A.1).
Revised: Also, finding top- k links with maximum gradients in Line 10 can be efficiently implemented (detailed in Appendix A.1).

Original: Thus, the topology evolution can be implemented efficiently on resource-limited devices.
Revised: Thus, the topology evolution can be implemented efficiently on resource-limited devices.

Original: Algorithm et al., 2020)
Revised: Algorithm et al., 2020)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 4. R OBUST V ALUE L EARNING
Revised: 4. R OBUST V ALUE L EARNING

Original: [Figure]
Revised: [Figure]

Original: As discussed above, value function learning is crucial in sparse training.
Revised: As discussed above, value function learning is crucial in sparse training.

Original: Specifically, we find that under sparse models, robust value learning not only serves to guarantee the efficiency of bootstrap training as in dense DRL training, but also guides the gradientbased topology exploration of the sparse network during training.
Revised: Specifically, we find that under sparse models, robust value learning not only serves to guarantee the efficiency of bootstrap training as in dense models, but also guides the gradient-based topology exploration of the sparse network during training.

Original: Figure 3 compares the performance of the masks (sparse networks) obtained under RigL and RLx2 (which is RigL + r obust value l earning) on Ant-v3.
Revised: Figure 3 compares the performance of the masks (i.e., sparse network topology) obtained by RigL and RLx2 (i.e., RigL + r obust value l earning) on Ant-v3.

Original: Here we adopt the methods in (Frankle & Carbin, 2019) for evaluating the obtained sparse model: 1) first initialize a random sparse topology; 2) keep adjusting the topology during the training and obtain the final mask; 3) train a sparse agent with the obtained mask (the mask is fixed throughout this training phase, only the weights are restored to their initial values as in the first step at the beginning).
Revised: Here we use a method similar to (Frankle & Carbin, 2019) for evaluating the obtained sparse mask: 2 1) first initialize a random sparse topology; 2) keep adjusting the topology during the training and obtain the final mask; 3) train a sparse agent with the obtained mask (the mask is fixed throughout this training phase, only the weights are restored to their initial values as in the first step at the beginning).

Original: It can be clearly observed that the mask by RLx2 significantly outperforms that by solely using RigL (Please see Appendix C.4 for details and experiments in other environments, where similar results are observed).
Revised: It can be clearly observed that the mask by RLx2 significantly outperforms that by solely using RigL (Please see Appendix C.4 for details and experiments in other environments, where similar results are observed).

Original: To achieve robust value estimation and properly guide the topology search, RLx2 utilizes two major components: i)  multi-step TD targets to bootstrap value estimation; ii) anovel dynamic-capacity replay buffer to eliminate the potential data inconsistency due to policy change during training.
Revised: To achieve robust value estimation and properly guide the topology search, RLx2 utilizes two novel components: i) delayed multi-step TD targets to bootstrap value estimation; ii) a dynamic-capacity replay buffer to eliminate the potential data inconsistency due to policy change during training.

Original: 4.2. M ULTI - STEP TD T ARGET
Revised: 4.2. M ULTI - STEP TD T ARGET

Original: In TD learning, a TD target is generated, and the value network will be iteratively updated by minimizing a squared loss induced by the TD target.
Revised: In TD learning, a TD target is generated, and the value network will be iteratively updated by minimizing a squared loss induced by the TD target.

Original: Single-step methods generate the TD target by combining one-step reward and discounted target network output, i.e., T 1 = r t + γQ ( s t +1 , π ( s t +1 ); θ ) .
Revised: Single-step methods generate the TD target by combining one-step reward and discounted target network output, i.e., T 1 = r t + γQ ( s t +1 , π ( s t +1 ); θ ) .

Original: However, a sparse network parameter (cid:98) θ = θ ⊙ M θ , obtained from its dense counterpart θ , will inevitably reside in a smaller hypothesis space due to using fewer parameters.
Revised: However, a sparse network parameter (cid:98) θ = θ ⊙ M θ , obtained from its dense counterpart θ , will inevitably reside in a smaller hypothesis space due to using fewer parameters.

Original: This means that the output of the sparse value network (cid:98) θ can be unreliable and may lead to inaccurate value estimation.
Revised: This means that the output of the sparse value network (cid:98) θ can be unreliable and may lead to inaccurate value estimation.

Original: Denote the fitting error of the value network as ϵ ( s, a ) =
Revised: Denote the fitting error of the value network as ϵ ( s, a ) =

Original: Q ( s, a ; θ ) − Q π ( s, a ) .
Revised: Q ( s, a ; θ ) − Q π ( s, a ) .

Original: One sees that this error may be larger under a sparse model compared to that under a dense network.
Revised: One sees that this error may be larger under a sparse model compared to that under a dense network.

Original: To overcome this issue, we adopt a multi-step target, i.e., T n
Revised: To overcome this issue, we adopt a multi-step target, i.e., T n

Original: [Equation]
Revised: [Equation]

Original: γ n Q ( s t + n , π ( s t + n ); θ ) , where the target combines an N -step sample and the output of the sparse value network after N -step, both appropriately discounted.
Revised: γ n Q ( s t + n , π ( s t + n ); θ ) , where the target combines an N -step sample and the output of the sparse value network after N -step, both appropriately discounted.

Original: By doing so, we reduce the expected error between the TD target and the true target.
Revised: By doing so, we reduce the expected error between the TD target and the true target.

Original: Specifically, Eq.(4) shows the expected TD error between multi-step TD target T n and the true Q-value Q π associated with the target policy π , conditioned on transitions from behavior policy b (see detailed derivation in Appendix A.2).
Revised: Specifically, Eq.(4) shows the expected TD error between multi-step TD target T n and the true Q-value Q π associated with the target policy π , conditioned on transitions from behavior policy b (see detailed derivation in Appendix A.2).

Original: [Equation]
Revised: [Equation]

Original: The multi-step target has been studied in existing works (Bertsekas & Ioffe, 1996; Precup, 2000;
Revised: The multi-step target has been studied in existing works (Bertsekas & Ioffe, 1996; Precup, 2000;

Original: Munos et al., 2016) for improving TD learning.
Revised: Munos et al., 2016) for improving TD learning.

Original: In our case, we also find that introducing a multi- step target reduces the network fitting error by a multiplicative factor γ n , as shown in Eq.
Revised: In our case, we also find that introducing a multistep target reduces the network fitting error by a multiplicative factor γ n , as shown in Eq.

Original: On the other hand, it has been observed, e.g., in Fedus et al.
Revised: On the other hand, it has been observed, e.g., in Fedus et al.

Original: (2020), that an immediate adoption of multi-step TD targets may cause a larger policy inconsistency error (the first term in Eq.
Revised: (2020), that an immediate adoption of multi-step TD targets may cause a larger policy inconsistency error (the first term in Eq.

Original: Thus, we adopt a hybrid scheme to suppress policy inconsistency and further improve value learning.
Revised: Thus, we adopt a delayed scheme to suppress policy inconsistency and further improve value learning.

Original: Specifically, at the early stage of training, we use one-step TD targets to better handle the quickly changing policy during this period, where a multi-step target may not be meaningful.
Revised: Specifically, at the early stage of training, we use one-step TD targets to better handle the quickly changing policy during this period, where a multi-step target may not be meaningful.

Original: Then, after several training epochs, when the policy change becomes less abrupt, We permanently switch to multi-step TD targets, to exploit its better approximation of the value function.
Revised: Then, after several training epochs, when the policy change becomes less abrupt, We permanently switch to multi-step TD targets, to exploit its better approximation of the value function.

Original: 4.2. D YNAMIC - CAPACITY B UFFER
Revised: 4.2. D YNAMIC - CAPACITY B UFFER

Original: The second component of RLx2 for robust value learning is a novel dynamic buffer scheme for controlling data inconsistency.
Revised: The second component of RLx2 for robust value learning is a novel dynamic buffer scheme for controlling data inconsistency.

Original: Off-policy algorithms use a replay buffer to store collected data and train networks with sampled batches from the buffer.
Revised: Off-policy algorithms use a replay buffer to store collected data and train networks with sampled batches from the buffer.

Original: Their performances generally improve when larger replay capacities are used (Fedus et al., 2020).
Revised: Their performances generally improve when larger replay capacities are used (Fedus et al., 2020).

Original: However, off-policy algorithms with unlimited-size replay buffers can suffer from policy inconsistency due to the following two aspects. 
Revised: However, off-policy algorithms with unlimited-size replay buffers can suffer from policy inconsistency due to the following two aspects. (i Inconsistent multi- step targets:

Original: [Figure]
Revised: In

Original:  with multi-step TD targets, the value function is updated to minimize the squared loss in Eq.
Revised: off -policy algorithms with multi-step TD targets, the value function is updated to minimize the squared loss in Eq.

Original: (3) on transitions sampled from the replay buffer, i.e., the reward sequence r t , r t +1 , · · · , r t +n collected during training.
Revised: (3) on transitions sampled from the replay buffer, i.e., the reward sequence r t , r t + 1 , · · · , r t +

Revised: n collected during training.

Original: However
Revised: However, the

Original: , the fact that the policy can evolve during training means that the data in the replay buffer, used for Monte-Carlo approximation of the current policy π , may be collected under a different behavior policy b (Hernandez-Garcia & Sutton, 2019; Fedus et al., 2020).
Revised: fact that the policy can evolve during training means that the data in the replay buffer, used for MonteCarlo approximation of the current policy π , may be collected under a different behavior policy b (Hernandez-Garcia & Sutton, 2019; Fedus et al., 2020).

Original: As a result, it may lead to a large policy inconsistency error in Eq.
Revised: As a result, it may lead to a large policy inconsistency error in Eq.

Original: (4), causing inaccurate estimation.
Revised: (4), causing inaccurate estimation.

Original:  (ii) Mismatched training data: In practice, the agent minimizes the value loss (cid:98) L ( θ ) with respect to the sampled value in mini-batch B t , given by
Revised: [Figure] d i s t a n c [Figure] (ii) Mismatched training data: In practice, the agent minimizes the value loss (cid:98) L ( θ ) with respect to the sampled value in mini-batch B t , given by

Original: [Equation]
Revised: [Equation]

Original: Compared to Eq.
Revised: Compared to Eq.

Original: (3), the difference between the distribution of transitions in the mini-batch B t and the true transition distribution induced by the current policy also leads to a mismatch in the training objective (Fujimoto et al., 2019).
Revised: (3), the difference between the distribution of transitions in the mini-batch B t and the true transition distribution induced by the current policy also leads to a mismatch in the training objective (Fujimoto et al., 2019).

Original: Indeed, our analysis in Appendix A.4 shows that training performance is closely connected to policy consistency
Revised: Indeed, our analysis in Appendix A.4 shows that training performance is closely connected to policy consistency.

Original: Motivated by our analysis, we introduce a dynamically-sized buffer to reduce the policy gap based on the policy distance of the collected data.
Revised: Motivated by our analysis, we introduce a dynamically-sized buffer to reduce the policy gap based on the policy distance of the collected data.

Original: The formal scheme is given in Algorithm 3.
Revised: The formal scheme is given in Algorithm 3.

Original: Specifically, we introduce the following policy distance measure to evaluate the inconsistency of data in the buffer, i.e.,
Revised: We introduce the following policy distance measure to evaluate the inconsistency of data in the buffer, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where B denotes the current replay buffer, OldK ( B ) denotes the oldest K transitions in B , and π ( · ; ϕ ) is the current policy.
Revised: where B denotes the current replay buffer, OldK ( B ) denotes the oldest K transitions in B , and π ( · ; ϕ ) is the current policy.

Original: Here K is a hyperparameter Upon any update to ϕ and appending new transitions to B , we calculate thelatest D ( B , ϕ ) value.
Revised: Here K is a hyperparameter. We calculate the D ( B , ϕ ) value every ∆ b steps.

Original: If D ( B , ϕ ) gets above a certain prespecified threshold, we start to pop items from B in a First-In-First-Out (FIFO) order until this distance measure D becomes below the threshold.
Revised: If D ( B , ϕ ) gets above a certain pre-specified threshold D 0 , we start to pop items from B in a First-In-First-Out (FIFO) order until this distance measure D becomes below the threshold.

Original: A visualization of the number of stored samples(dynamic capacity) and the proposed policy distance metric during training is shown in Figure 4.
Revised: A visualization of the number of stored samples and the proposed policy distance metric during training is shown in Figure 4.

Original: We see that the policy distance oscillates in the early stage as the policy evolves, but it is tightly controlled and does not violate the threshold condition to effectively address the off-policyness issue.
Revised: We see that the policy distance oscillates in the early stage as the policy evolves, but it is tightly controlled and does not violate the threshold condition to effectively address the off-policyness issue.

Original: As the policy converges, the policy distance tends to decrease and converge (We also show in Appendix C.2 that the performanceof RLx2 is insensitive to the policy threshold).
Revised: As the policy converges, the policy distance tends to decrease and converge (Appendix C.8 also shows that the performance is insensitive to the policy threshold D 0 ).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section

Original: [Table] we investigate the performance improvement of RLx2 in Section 5. [Table] and the importance of each component in RLx2 in Section 5.2.
Revised: In this section, we investigate the performance improvement of RLx2 in Section 5.1, and the importance of each component in RLx2 in Section 5.2.

Original: In particular, we pay extra attention to the role topology evolution plays in sparse training in Section 5.3.
Revised: In particular, we pay extra attention to the role topology evolution plays in sparse training in Section 5.3.

Original: Our experiments are conducted in four popular MuJoCo environments: HalfCheetah-v3 (Hal.), Hopper-v3 (Hop.), Walker2d-v3 (Wal.) [Table]
Revised: Our experiments are conducted in four popular MuJoCo environments: HalfCheetah-v3 (Hal.), Hopper-v3 (Hop.), Walker2d-v3 (Wal.), and

Original: and Ant-v3 (Ant.),  for RLx2 with two off-policy algorithms, TD3 and SAC.
Revised: Ant-v3 (Ant.), 3 for RLx2 with two off-policy algorithms, TD3 and SAC.

Original: Instantiations of RLx2 on
Revised: Instantiations of RLx2 on

Original: TD3 and SAC are provided in Appendix B. Each result is averaged over eight random seeds.
Revised: TD3 and SAC are provided in Appendix B. Each result is averaged over eight random seeds.

Original: 5. C OMPARATIVE E VALUATION
Revised: C OMPARATIVE E VALUATION

Original: [Table]

Original: Table 2 summarizes the comparison results.
Revised: Table 2 summarizes the comparison results.

Original: In our experiments, we compare RLx2 with the following baselines: (i) Tiny , which uses tiny dense networks with the same number of parameters as the sparse model in training.
Revised: In our experiments, we compare RLx2 with the following baselines: (i) Tiny , which uses tiny dense networks with the same number of parameters as the sparse model in training.

Original: (ii) SS : using static sparse networks with random initialization.
Revised: (ii) SS : using static sparse networks with random initialization.

Original: (iii)
Revised: (iii)

Original: SET (Bellec et al., 2017), which uses dynamic sparse training by dropping connections according to the magnitude and growing connections randomly.
Revised: SET (Mocanu et al., 2018), which uses dynamic sparse training by dropping connections according to the magnitude and growing connections randomly.

Original: Please notice that the previous work (Sokar et al., 2021) also adopts the SET algorithm for topology evolution in reinforcement learning.
Revised: Please notice that the previous work (Sokar et al., 2021) also adopts the SET algorithm for topology evolution in reinforcement learning.

Original: Our implementations reach better performance due to different hyperparameters.
Revised: Our implementations reach better performance due to different hyperparameters.

Original: (iv) RigL
Revised: (iv) RigL

Original: (Evci et al., 2020), which uses dynamic sparse training by dropping and growing connections with magnitude and gradient criteria, respectively, the same as RLx2’s topology evolution procedure. 
Revised: (Evci et al., 2020), which uses dynamic sparse training by dropping and growing connections with magnitude and gradient criteria, respectively, the same as RLx2’s topology evolution procedure. [Table] [Table]

Original: In our experiments, we allow the actor and critic networks to take different sparsities.
Revised: In our experiments, we allow the actor and critic networks to take different sparsities.

Original: We define an ultimate compression ratio, i.e., the largest sparsity level under which the performance degradation under RLx2 is within ± %3 of that under the original dense models.
Revised: We define an ultimate compression ratio, i.e., the largest sparsity level under which the performance degradation under RLx2 is within ± %3 of that under the original dense models.

Original: This can also be understood as the minimum size of the sparse model with the full performance of the original dense model.
Revised: This can also be understood as the minimum size of the sparse model with the full performance of the original dense model.

Original: We present performance comparison results in Table 2 based on the ultimate compression ratio.
Revised: We present performance comparison results in Table 2 based on the ultimate compression ratio.

Original: The performance of each algorithm is evaluated with the average reward per episode over the last 30 policy evaluations of the training ( policy evaluation is conducted every 5000 steps).
Revised: The performance of each algorithm is evaluated with the average reward per episode over the last 30 policy evaluations of the training ( policy evaluation is conducted every 5000 steps).

Original: Hyperparameters are fixed in all four environments for TD3 and SAC, respectively, which are presented in Appendix C.2.
Revised: Hyperparameters are fixed in all four environments for TD3 and SAC, respectively, which are presented in Appendix C.2.

Original: Performance Table 2 shows RLx2 performs best among all baselines in all four environments by a large margin (except for a close performance with RigL with SAC in Hopper).
Revised: Performance Table 2 shows RLx2 performs best among all baselines in all four environments by a large margin (except for a close performance with RigL with SAC in Hopper).

Original: In addition, tiny dense (Tiny) and random static sparse networks (SS) performance are worst on average.
Revised: In addition, tiny dense (Tiny) and random static sparse networks (SS) performance are worst on average.

Original: SET and
Revised: SET and

Original: RigL are better yet fail to maintain the performance in Walker2d-v3 and Ant-v3, which means robust value learning is necessary under sparse training.
Revised: RigL are better yet fail to maintain the performance in Walker2d-v3 and Ant-v3, which means robust value learning is necessary under sparse training.

Original: To further validate the performance of RLx2, we compare the performance of different methods under different sparsity levels in Hopper-v3 and Antv3 in Figure 5, showing RLx2 has a significant performance gain over other baselines.
Revised: To further validate the performance of RLx2, we compare the performance of different methods under different sparsity levels in Hopper-v3 and Antv3 in Figure 5, showing RLx2 has a significant performance gain over other baselines.

Original: Model Compression RLx2 achieves superior compression ratios (the reciprocal of the total size), with minor performance degradation (less than 3% ).
Revised: Model Compression RLx2 achieves superior compression ratios (the reciprocal of the total size), with minor performance degradation (less than 3% ).

Original: Specifically, RLx2 with TD3 achieves 7 .
Revised: Specifically, RLx2 with TD3 achieves 7 .

Original: 5 × - 25 × model compression, with the best compression ratios of 25 × on Hopper-v3.
Revised: 5 × - 25 × model compression, with the best compression ratios of 25 × on Hopper-v3.

Original: The actor can be compressed for each environment by more than 96% , and the critic is compressed by 85% - 95% .
Revised: The actor can be compressed for each environment by more than 96% , and the critic is compressed by 85% - 95% .

Original: The results for SAC are similar.
Revised: The results for SAC are similar.

Original: RLx2 with SAC achieves a 5 × - 20 × model compression.
Revised: RLx2 with SAC achieves a 5 × - 20 × model compression.

Original: [Figure]
Revised: [Figure]

Original: Acceleration in FLOPs Different from knowledge-distillation/BC based methods, e.g., Livne &
Revised: Acceleration in FLOPs Different from knowledge-distillation/BC based methods, e.g., Livne &

Original: Cohen (2020); Vischer et al. (2022); Lee et al. (2021), RLx2 uses a sparse network throughout training.
Revised: Cohen (2020); Vischer et al. (2022); Lee et al. (2021), RLx2 uses a sparse network throughout training.

Original: Thus, it has an additional advantage of immensely accelerating training and saving computation, i.e., 12 × training acceleration and 20 × inference acceleration for RLx2-TD3, and 7 × training acceleration and 12 × inference acceleration for RLx2-SAC.
Revised: Thus, it has an additional advantage of immensely accelerating training and saving computation, i.e., 12 × training acceleration and 20 × inference acceleration for RLx2-TD3, and 7 × training acceleration and 12 × inference acceleration for RLx2-SAC.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: We conduct a comprehensive ablation study on the three critical components of RLx2 on TD3, i.e., topology evolution, multi-step TD target, and dynamic-capacity buffer, to examine the effect of each component in RLx2 and their robustness in hyperparameters.
Revised: We conduct a comprehensive ablation study on the three critical components of RLx2 on TD3, i.e., topology evolution, multi-step TD target, and dynamic-capacity buffer, to examine the effect of each component in RLx2 and their robustness in hyperparameters.

Revised: In addition, we provide the sensitivity analysis for algorithm hyper-parameters, e.g. initial mask update fraction, mask update interval, buffer adjustment interval, and buffer policy distance threshold, in Appendix C.8.

Original: Topology evolution RLx2 drops and grows connections with magnitude and gradient criteria, respectively, which has been adopted in RigL
Revised: Topology evolution RLx2 drops and grows connections with magnitude and gradient criteria, respectively, which has been adopted in RigL

Original: (Evci et al., 2020) for deep supervised learning.
Revised: (Evci et al., 2020) for deep supervised learning.

Original: To validate the necessity of our topology evolution criteria, we compare RLx2 with three baselines, which replace the topology evolution scheme in RLx2 with Tiny, SS and SET, while keeping other components in RLx2 unchanged, i.e., they are also equipped with multi-step targets and dynamiccapacity buffer.
Revised: To validate the necessity of our topology evolution criteria, we compare RLx2 with three baselines, which replace the topology evolution scheme in RLx2 with Tiny, SS and SET, while keeping other components in RLx2 unchanged.

Original: The left part of Table 3 shows that RigL as a topology adjustment scheme (the resulting scheme is RLx2 when using RigL) performs best among the four baselines.
Revised: 4 Thy evolution e left partpolog of Table 3 shows that RigL as a topology adjustment scheme (the resulting scheme is RLx2 when using RigL) performs best among the four baselines.

Original: We also observe that Tiny performs worst, which is consistent with the conclusion in existing works (Zhu & Gupta, 2018) that a sparse network may contain a smaller hypothesis space and leads to performance loss, which necessitates a topology evolution scheme
Revised: We also observe that Tiny performs worst, which is consistent with the conclusion in existing works (Zhu & Gupta, 2018) that a sparse network may contain a smaller hypothesis space and leads to performance loss, which necessitates a topology evolution scheme

Original: [Table]
Revised: [Table]

Original: Env. Topoloy Evolution

Original: [Table] 
Revised: [Table] [Figure]

Revised: A v e r a g e r e t u r

Original: [Figure]
Revised: [Figure]

Original: Multi-step TD targets We also compare different step lengths in multi-step TD targets for RLx2 in the right part of Table 3.
Revised: Multi-step TD targets We also compare different step lengths in multi-step TD targets for RLx2 in the right part of Table 3.

Original: We find that multi-step TD targets with a step length of 3 obtain the maximum performance.
Revised: We find that multi-step TD targets with a step length of 3 obtain the maximum performance.

Original: In particular, multi-step TD targets improve the performance dramatically in Hopper-v3 and Walker2d-v3, while the improvement in HalfCheetach-v3 and Ant-v3 is minor.
Revised: In particular, multi-step TD targets improve the performance dramatically in Hopper-v3 and Walker2d-v3, while the improvement in HalfCheetach-v3 and Ant-v3 is minor.

Original: Dynamic-capacity Buffer We compare differentreplay buffer sizing schemes, including our dynamic scheme and using different fixed buffer capacities or an unlimited buffer.
Revised: Dynamic-capacity Buffer We compare different buffer sizing schemes, including our dynamic scheme, different fixed-capacity buffers, and an unlimited buffer.

Original: Figure 6 shows that our dynamic-capacity buffer performs best among all settings of the buffer.
Revised: Figure 6 shows that our dynamiccapacity buffer performs best among all settings of the buffer.

Original: Smaller buffer capacity benefits the performance in the early stage but may reduce the final performance.
Revised: Smaller buffer capacity benefits the performance in the early stage but may reduce the final performance.

Original: This is because using a smaller buffer results in higher sample efficiency in the early stage of training but fails in reaching high performance in the long term, whereas a large or even unlimited one may perform poorly in all stages.
Revised: This is because using a smaller buffer results in higher sample efficiency in the early stage of training but fails in reaching high performance in the long term, whereas a large or even unlimited one may perform poorly in all stages.

Original: 5. W HY E VOLVE T OPOLOGY IN DRL?
Revised: 5. W HY E VOLVE T OPOLOGY IN DRL?

Original: Compared to dense networks, sparse networks have smaller hypothesis spaces.
Revised: Compared to dense networks, sparse networks have smaller hypothesis spaces.

Original: Even under the same sparsity, different sparse architectures correspond to different hypothesis spaces.
Revised: Even under the same sparsity, different sparse architectures correspond to different hypothesis spaces.

Original: As Frankle & Carbin (2019) has shown, some sparse architecture (e.g., the “winning ticket”) performs better than a random one.
Revised: As Frankle & Carbin (2019) has shown, some sparse architecture (e.g., the “winning ticket”) performs better than a random one.

Original: To emphasize the necessity of topology evolution in sparse training, we compare different sparse network architectures in Figure 7, including the random ticket (topology sampled at random and fixed throughout training), the winning ticket (topology from an RLx2 run and fixed throughout training), and a dynamic ticket (i.e., training using RLx2) under both reinforcement learning (RL) and behavior cloning (BC). 
Revised: To emphasize the necessity of topology evolution in sparse training, we compare different sparse network architectures in Figure 7, including the random ticket (topology sampled at random and fixed throughout training), the winning ticket (topology from an RLx2 run and fixed throughout training), and a dynamic ticket (i.e., training using RLx2) under both reinforcement learning (RL) and behavior cloning (BC). 5 From Figure 7(a), we see that RLx2 achieves the best

Original: [Figure]
Revised: [Figure]

Original: From Figure 7(a), we see that RLx2 achieves the best performance, which is comparable with that under the original dense model.
Revised: performance, which is comparable with that under the original dense model.

Original: Due to the potential data inconsistency problem in value learning and the smaller hypothesis search space under sparse networks, training with a single fixed topology does not fully reap the benefit of high sparsity and can cause significantly degraded performance.
Revised: Due to the potential data inconsistency problem in value learning and the smaller hypothesis search space under sparse networks, training with a single fixed topology does not fully reap the benefit of high sparsity and can cause significantly degraded performance.

Original: That is why the winning ticket and random ticket both lead to significant performance loss compared to RLx2.
Revised: That is why the winning ticket and random ticket both lead to significant performance loss compared to RLx2.

Original: On the other hand, Figure 7(b) shows that in BC tasks, the winning ticket and RLx perform almost the same as the dense model, while the random ticket performs worst.
Revised: On the other hand, Figure 7(b) shows that in BC tasks, the winning ticket and RLx2 perform almost the same as the dense model, while the random ticket performs worst.

Original: This indicates that an appropriate fixed topology can indeed be sufficient to reach satisfactory performance in BC, which is intuitive since BC adopts a supervised learning approach and eliminates non-stationarity due to bootstrapping training.
Revised: This indicates that an appropriate fixed topology can indeed be sufficient to reach satisfactory performance in BC, which is intuitive since BC adopts a supervised learning approach and eliminates non-stationarity due to bootstrapping training.

Original: In conclusion, we find that a fixed winning ticket can perform as well as a dynamic topology that evolves during the training in behavior cloning, while RLx2 outperforms the winning ticket in RL training.
Revised: In conclusion, we find that a fixed winning ticket can perform as well as a dynamic topology that evolves during the training in behavior cloning, while RLx2 outperforms the winning ticket in RL training.

Original: This observation indicates that topology evolution not only helps find the winning ticket in sparse DRL training but is alsoa necessary component of training a sparse DRL agent due to the extra non-stationary in bootstrapping training, compared to deep supervised learning.
Revised: This observation indicates that topology evolution not only helps find the winning ticket in sparse DRL training but is also necessary for training a sparse DRL agent due to the extra non-stationary in bootstrapping training, compared to deep supervised learning.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes a sparse training framework, RLx2, for off-policy reinforcement learning (RL).
Revised: This paper proposes a sparse training framework, RLx2, for off-policy reinforcement learning (RL).

Original: RLx2 utilizes gradient-based evolution to enable efficient topology exploration and establishes robust value learning using a  multi-step TD target mechanism with a dynamic-capacity replay buffer.
Revised: RLx2 utilizes gradient-based evolution to enable efficient topology exploration and establishes robust value learning using a delayed multi-step TD target mechanism with a dynamic-capacity replay buffer.

Original: RLx2 enables training an efficient DRL agent with minimal performance loss using an ultra-sparse network throughout training and removes the need for pre-training dense networks.
Revised: RLx2 enables training an efficient DRL agent with minimal performance loss using an ultrasparse network throughout training and removes the need for pre-training dense networks.

Original: Our extensive experiments on RLx2 with TD3 and SAC demonstrate state-of-the-art sparse training performance, showing a 7 . 5 × - 20 × model compression with less than 3% performance degradation and up to 20 × and 50 × FLOPs reduction in training and inference, respectively.
Revised: Our extensive experiments on RLx2 with TD3 and SAC demonstrate state-of-the-art sparse training performance, showing a 7 . 5 × - 20 × model compression with less than 3% performance degradation and up to 20 × and 50 × FLOPs reduction in training and inference, respectively.

Revised: It will be interesting future work to extend the RLx2 framework in more complex RL scenarios requiring more intense demand for computing resources, e.g., real-world problems instead of standard MuJoCo environments or multi-agent settings.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Experiment details (including an efficient implementation for RLx2, implementation details of the dynamic buffer, hyperparameters, and network architectures) are included in Appendix C for reproduction.
Revised: Experiment details (including an efficient implementation for RLx2, implementation details of the dynamic buffer, hyperparameters, and network architectures) are included in Appendix C for reproduction.

Original: The proof for our analysis of the dynamic buffer can be found in Appendix A.4.
Revised: The proof for our analysis of the dynamic buffer can be found in Appendix A.4.

Original: The code will be open-sourced upon publication of the paper
Revised: The code will be open-sourced upon publication of the paper

Original: Supplementary Materials AAdditional
Revised: Supplementary Materials A

Original:  Details for Section
Revised: Additional Details for Section

Original: A.1 Efficient Implementation for Algorithm 1 . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1 Efficient Implementation for Algorithm 1 . . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Derivation of Eq.
Revised: A.2 Derivation of Eq.

Original: (4) in Section 4.2.1 . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: (4) in Section 4.2.1 . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.3 Algorithm of Dynamic-capacity Buffer in Section 4.2.2 . . . . . . . . . . . . . . .
Revised: A.3 Algorithm of Dynamic-capacity Buffer in Section 4.2.2 . . . . . . . . . . . . . . .

Original: A.4 Detailed analysis of dynamic buffer . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.4 Detailed analysis of dynamic buffer . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B Details of RLx2 with TD3 and SAC
Revised: B Details of RLx2 with TD3 and SAC

Original: C Experimental Details
Revised: C Experimental Details

Original: C.1 Hardware Setup . .
Revised: C.1 Hardware Setup . .

Original: C.2 Hyperparameter Settings for Reproduction . . . . . . . . . . . . . . . . . . . . . .
Revised: C.2 Hyperparameter Settings for Reproduction . . . . . . . . . . . . . . . . . . . . . .

Original: C.3 Calculation of Model Size and FLOPs . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.3 Calculation of Model Size and FLOPs . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.4 Comparisons between Different Masks in Other Environments . . . . . . . . . . .
Revised: C.4 Comparisons between Different Masks in Other Environments . . . . . . . . . . .

Original: C.5 Training Curves of Comparative Evaluation in Section 5.1 . . . . . . . . .
Revised: C.5 Training Curves of Comparative Evaluation in Section 5.1 . . . . . . . . .

Revised: C.6 Standard Deviations of Results in Table 2 . . . . . . . . . . . . . . . . . . . . . .

Original: C.6 Supplementary Results for Ablation Study . . . . . . . . . . . . . . . . . . . . . .
Revised: C.7 Supplementary Results for Ablation Study . . . . . . . . . . . . . . . . . . . . . .

Original: C.7 Supplementary Results for Investigation of the Lottery Ticket Hypothesis . . . . . C.8 Standard Deviations of Results in Table 2 . . . . . . . . . . . . . . . . . . . . . . 
Revised: C.8 Sensitivity Analysis for Hyperparameters . . . . . . . . . . . . . . . . . . . . . . C.9 Additional Results in Humanoid-v3 . . . . . . . . . . . . . . . . . . . . . . . . . C.10 Supplementary Results

Original: C.9 Visualization of Sparse Models. . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.11 Visualization of Sparse Models

Original: A A DDITIONAL D ETAILS FOR S ECTION
Revised: A A DDITIONAL D ETAILS FOR S ECTION

Original: This section provides additional details for Section 4, including how to efficiently implement Algorithm 1 with limited resource, the derivation of Eq.
Revised: This section provides additional details for Section 4, including how to efficiently implement Algorithm 1 with limited resource, the derivation of Eq.

Original: (4) in Section 4.2.1, and the full algorithm of dynamic-capacity buffer in Section 4.2.2.
Revised: (4) in Section 4.2.1, and the full algorithm of dynamic-capacity buffer in Section 4.2.2.

Original: A. E FFICIENT I MPLEMENTATION FOR A LGORITHM
Revised: A. E FFICIENT I MPLEMENTATION FOR A LGORITHM

Original: For simplicity, in this section, we omit all indices of l in the symbols that appeared in Algorithm 1.
Revised: For simplicity, in this section, we omit all indices of l in the symbols that appeared in Algorithm 1.

Original:  Suppose layer l takes an n ( in ) -dimensional
Revised: Parameter Storing Suppose layer l takes an n ( in ) -dimensional

Original: vector x as the input, and outputs an n ( out ) -feature
Revised: vector x as the input, and outputs an n ( out ) -feature

Original: vector y via a linear transformation.
Revised: vector y via a linear transformation.

Original: Then the layer’s number of parameters is N = n ( in ) × n ( out ) .
Revised: Then the layer’s number of parameters is N = n ( in ) × n ( out ) .

Original: A naive implementation will be to store both θ ∈ R n ( out ) × n ( in ) and M θ ∈ { 0 , 1 } n ( out ) × n ( in ) in two dense n ( out ) × n ( in ) matrices in the memory.
Revised: A naive implementation will be to store both θ ∈ R n ( out ) × n ( in ) and M θ ∈ { 0 , 1 } n ( out ) × n ( in ) in two dense n ( out ) × n ( in ) matrices in the memory.

Original: In forward and backward propagations, one simply performs the dense-matrix-multiply-vector operation on θ and x .
Revised: In forward and backward propagations, one simply performs the dense-matrix-multiply-vector operation on θ and x .

Original: However, this implementation cannot enjoy any speed-up even when we are using a sparsity ratio s close to 1 , so the network is highly sparse.
Revised: However, this implementation cannot enjoy any speed-up even when we are using a sparsity ratio s close to 1 , so the network is highly sparse.

Original: Also, the actual memory occupied by the model is always proportional to N and irrelevant to s .Input output gradient matrix x ≥
Revised: Also, the actual memory occupied by the model is always proportional to N and irrelevant to s .

Original: x σ

Original: ≥

Original: x elements are 3 -tuples comparing elements according to their first components A better way is to store θ in a more compact manner, where only the non-zero indices (i.e., positions of the ones in M θ ) and their values.
Revised: However, a better way is to store θ in a more compact manner, where only the non-zero indices (i.e., positions of the ones in M θ ) and their values.

Original: As a result, the weights of the layer now occupies Θ((1 − s ) N ) memory, and the matrix-multiply-vector operation also just costs O ((1 − s ) N + n ( in ) + n ( out ) ) .
Revised: As a result, the weights of the layer now occupies Θ((1 − s ) N ) memory, and the matrix-multiply-vector operation also just costs O ((1 − s ) N + n ( in ) + n ( out ) ) .

Original: Such sparse matrix (or tensor of higher orders) structures are supported by many modern machine learning frameworks, e.g., torch.sparse in PyTorch.
Revised: Such sparse matrix (or tensor of higher orders) structures are supported by many modern machine learning frameworks, e.g., torch.sparse in PyTorch.

Original: Many of them also support automatic gradient calculation and backward propagation.
Revised: Many of them also support automatic gradient calculation and backward propagation.

Original:  With this sparse representation of θ , it can be seen that the link dropping step of Algorithm 1 (Line 9) can be done in O ((1 − s ) N log N ) time by sorting all the weight entries in their absolute values and then picking the top- K items.
Revised: Link Dropping With this sparse representation of θ , it can be seen that the link dropping step of Algorithm 1 (Line 9) can be done in O ((1 − s ) N log N ) time by sorting all the weight entries in their absolute values and then picking the top- K items.

Original:  It then suffices to figure out a way to implement the link growing step of Algorithm 1 (Line 10).
Revised: Link Growing It then suffices to figure out a way to implement the link growing step of Algorithm 1 (Line 10).

Original: Denote by L the scalar loss function of the whole neural network containing layer l .
Revised: Denote by L the scalar loss function of the whole neural network containing layer l .

Original: Assume that we have just performed a backward propagation, where layer l contributes only O ((1 − s ) N + n ( in ) + n ( out ) ) to the computation time, and g ( x ) :
Revised: Assume that we have just performed a backward propagation, where layer l contributes only O ((1 − s ) N + n ( in ) + n ( out ) ) to the computation time, and g ( x ) :

Original: =
Revised: =

Original: ∂L∂ x , g ( y ) :
Revised: ∂L∂ x , g ( y ) :

Original: =
Revised: =

Original: ∂L∂
Revised: ∂L∂

Original: y and g ( θ ) :
Revised: y and g ( θ ) :

Original: = ∂L∂θ all have been computed.
Revised: = ∂L∂θ all have been computed.

Original: Since θ is in a compact representation with (1 − s ) N elements, now g ( θ ) obtained by auto-grad also only contains (1 − s ) N items, but the growing step Line 10 basically asks to collect the top- K items in the whole dense gradient matrix with N elements.
Revised: Since θ is in a compact representation with (1 − s ) N elements, now g ( θ ) obtained by auto-grad also only contains (1 − s ) N items, but the growing step Line 10 basically asks to collect the top- K items in the whole dense gradient matrix with N elements.

Original: According to the chain rule, for a link θ ji between the i -th input neural and the j -th output neural, the partial derivative of L with respect to θ ji is given by (here, we abuse the notation a little)
Revised: According to the chain rule, for a link θ ji between the i -th input neural and the j -th output neural, the partial derivative of L with respect to θ ji is given by (here, we abuse the notation a little)

Original: [Equation]
Revised: [Equation]

Original: Hence the desired true dense n ( out ) × n ( in ) gradient matrix g ( θ ) is equal to g ( y ) x T .
Revised: Hence the desired true dense n ( out ) × n ( in ) gradient matrix g ( θ ) is equal to g ( y ) x T .

Original: Our task reduces to collect the K entries with the largest absolute values while keeping away from the locations that have just been dropped in Line 9.
Revised: Our task reduces to collect the K entries with the largest absolute values while keeping away from the locations that have just been dropped in Line 9.

Original: In fact, this procedure can be efficiently implemented by scanning via n ( out ) pointers with the help of a heap (a.k.a., a priority queue), described in the following pseudo-code Algorithm 2.
Revised: In fact, this procedure can be efficiently implemented by scanning via n ( out ) pointers with the help of a heap (a.k.a., a priority queue), described in the following pseudo-code Algorithm 2.

Original: It can be seen that Algorithm 2 consumes O ( n ( out ) + n ( in ) + | U | + (1 − s ) N ) heap operations and set operations.
Revised: It can be seen that Algorithm 2 consumes O ( n ( out ) + n ( in ) + | U | + (1 − s ) N ) heap operations and set operations.

Original: If all sets S and U are implemented using binary search trees or hash tables, the costs for each heap operation and each set operation are all within O (log N ) .
Revised: If all sets S and U are implemented using binary search trees or hash tables, the costs for each heap operation and each set operation are all within O (log N ) .

Original: Therefore the total running time of Algorithm 2 is O ((1 − s ) N log N ) .
Revised: Therefore the total running time of Algorithm 2 is O ((1 − s ) N log N ) .

Revised: Input: Input featrue vector x , output gradient vector g ( y ) , base index set S 0 , dense matrix size

Revised: N , target sparsity s , forbid index set U

Revised: Output:

Revised: An index set S such that | S | =

Revised: (1 − s ) N , S ⊇ S

Original: A. D ERIVATION OF E Q . (4) IN S ECTION 4.2.
Revised: S U = ∅ 1: Initialize S ← S 0 . 2: Sort | x | to get a permutation σ 1 , . . . σ n ( in ) , such that | x σ 1 | ≥ | x σ 2 | ≥ · · · ≥ | x σ n ( in ) | .

Revised: 3: Create a max-heap H whose elements are 3 -tuples, comparing elements according to their first components.

Revised: 4: for j = 1 . . .

Revised: n ( out ) do 5:

Revised: Append ( | g ( y )

Revised: j x σ 1 | , j, 1) into H .

Revised: 6: end for 7: while | S | < (1 − s ) N do 8:

Revised: Pop the top triple ( w, j, i ) out of H .

Revised: 9: if ( j, σ i ) / ∈ S and ( j, σ i ) / ∈ U then 10:

Revised: S ← S ∪ ( j, σ i ) 11: end if 12: if i < n ( in ) then 13:

Revised: Append ( | g ( y )

Original:   Denote p π ( s t +1 , a t +1 , · · · , s t + n , a t + n | s t , a t ) the distribution of the trajectory starting from the current state s t and action a t under policy π .
Revised: j x σ i +1 | , j, i + 1) into H . 14: end if 15: end while 16: Return S . A. D ERIVATION OF E Q . (4) IN S ECTION 4.2.  Denote p π ( s t +1 , a t +1 , · · · , s t + n , a t + n | s t , a t ) the distribution of the trajectory starting from the current state s t and action a t under policy π .

Original: For simplicity, we use E π to denote E ( s t +1 ,a t +1 , ··· ,s t + n ,a t + n ) ∼ p π ( ·| s t ,a t ) , and use E b to denote E ( s t +1 ,a t +1 , ··· ,s t + n ,a t + n ) ∼ p b ( ·| s t ,a t ) , where π and b denote the current policy and the behavior policy, respectively.
Revised: For simplicity, we use E π to denote E ( s t +1 ,a t +1 , ··· ,s t + n ,a t + n ) ∼ p π ( ·| s t ,a t ) , and use E b to denote E ( s t +1 ,a t +1 , ··· ,s t + n ,a t + n ) ∼ p b ( ·| s t ,a t ) , where π and b denote the current policy and the behavior policy, respectively.

Original: Q π ( s, a ) denotes the
Revised: Q π ( s, a ) denotes the

Original: Q function associated with policy π as defined in Eq.
Revised: Q function associated with policy π as defined in Eq.

Original: (1) in the manuscript, i.e., Q π ( s, a ) =
Revised: (1) in the manuscript, i.e., Q π ( s, a ) =

Original: E π (cid:104)(cid:80)
Revised: E π (cid:104)(cid:80)

Original: Ti = t γ i − t r
Revised: Ti = t γ i − t r

Original: ( s i , a i ) | s t =
Revised: ( s i , a i ) | s t =

Original: s, a t = a (cid:105) .
Revised: s, a t = a (cid:105) .

Original: We also use ϵ ( s, a ) to denote the network fitting error, i.e., ϵ ( s, a ) =
Revised: We also use ϵ ( s, a ) to denote the network fitting error, i.e., ϵ ( s, a ) =

Original: Q ( s, a ; θ ) − Q π ( s, a ) .
Revised: Q ( s, a ; θ ) − Q π ( s, a ) .

Original: Subsequently, we have:
Revised: Subsequently, we have:

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]

Original: ( s t + n ))

Original: [Equation]

Original: [Equation]

Original: ( s

Original: [Equation]

Original: a

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: t

Original: [Equation]

Original: n

Original: [Equation]

Original: t + n ))]

Original: [Equation]

Original: The first equality holds in the manuscript by definitions in Eq.
Revised: The first equality holds in the manuscript by definitions in Eq.

Original: (1) and of multi-step TD targets.
Revised: (1) and of multi-step TD targets.

Original: The second equality holds by firstly adding and then subtracting the term E π
Revised: The second equality holds by firstly adding and then subtracting the term E π

Original: [ γ n ( Q ( s t + n , a t + n ; θ ))] , and the last equality holds by definitions of T n ( s, a ) and ϵ ( s, a ) .
Revised: [ γ n ( Q ( s t + n , a t + n ; θ ))] , and the last equality holds by definitions of T n ( s, a ) and ϵ ( s, a ) .

Original: This decomposition shows that the expected error consists of two parts, i.e., the network fitting error and the policy inconsistency error, which are well handled by our multi-step TD targets with a dynamic-capacity buffer.
Revised: This decomposition shows that the expected error consists of two parts, i.e., the network fitting error and the policy inconsistency error, which are well handled by our multi-step TD targets with a dynamic-capacity buffer.

Original: A. A LGORITHM OF D YNAMIC - CAPACITY B UFFER IN S ECTION 4.2. 
Revised: A. A LGORITHM OF D YNAMIC - CAPACITY B UFFER IN S ECTION 4.2.

Original: Algorithm 3 presents our formal procedure for dynamically controlling the buffer capacity in Section 4.2.2.
Revised: Algorithm 3 presents our formal procedure for dynamically controlling the buffer capacity in Section 4.2.2.

Original: For each step, a new transition is inserted into the replay buffer.
Revised: For each step, a new transition is inserted into the replay buffer.

Original: To avoid the problem of policy inconsistency, we check the buffer per ∆ b steps and drop the oldest transitions if needed.
Revised: To avoid the problem of policy inconsistency, we check the buffer per ∆ b steps and drop the oldest transitions if needed.

Original: Specifically, we first set a hard lower bound B min , and a hard upper bound B max of the buffer capacity.
Revised: Specifically, we first set a hard lower bound B min , and a hard upper bound B max of the buffer capacity.

Original: (i)
Revised: (i)

Original: If the buffer size is below B min , we store all newly collected data samples.
Revised: If the buffer size is below B min , we store all newly collected data samples.

Original: (ii)
Revised: (ii)

Original: If the amount of buffered transitions has reached B max , the oldest transitions will be replaced by the latest transitions.
Revised: If the amount of buffered transitions has reached B max , the oldest transitions will be replaced by the latest transitions.

Original: (iii) When the buffer is in ( B min , B max ) , for each time, we calculate the policy distance between the oldest behavior policy and the current policy, based on the oldest transitions stored in the buffer.
Revised: (iii) When the buffer is in ( B min , B max ) , for each time, we calculate the policy distance between the oldest behavior policy and the current policy, based on the oldest transitions stored in the buffer.

Original: In addition, if the policy distance exceeds the threshold D 0 , the oldest transitions are discarded.
Revised: In addition, if the policy distance exceeds the threshold D 0 , the oldest transitions are discarded.

Original: The full algorithm is given in Algorithm 3. 
Revised: The full algorithm is given in Algorithm 3. [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original:  A. D ETAILED ANALYSIS OF DYNAMIC BUFFER
Revised: [Table] [Table] [Table] A. D ETAILED ANALYSIS OF DYNAMIC BUFFER

Original: Using a dynamic buffer can reduce the gap between target policy and behavioral policy, as we have shown empirically in Section 5.2.
Revised: Using a dynamic buffer can reduce the gap between target policy and behavioral policy, as we have shown empirically in Section 5.2.

Original: In this section, we give a more detailed analysis of the influence of the dynamic buffer.
Revised: In this section, we give a more detailed analysis of the influence of the dynamic buffer.

Original: We first define the notations used in our analysis.
Revised: We first define the notations used in our analysis.

Original: p( s ′ | s, a ) : 
Revised: p s ′ | s, a ) : Environment

Original: Environment transition probability ρ ( s ) π,t : State distribution in time t under policy π ρ ( s,a ) π,t : State-action pair distribution in time tunder policy π µ π ( τ ) : Distribution of trajectory τ
Revised: transition probability ρ ( s ) π,t : State distribution in time t under policy π ρ ( s,a ) π,t : State-action pair distribution in time t

Original:  = ( s t , a t , s t +1 , · · · , s t + n , a t + n ) under policy π d π : State-action visitation distribution under policy π , d π ( s )=
Revised: under policy π µ π ( τ ) : Distribution of trajectory τ = ( s t , a t , s t +1 , · · · , s t + n , a t + n ) under policy π d π : State-action visitation distribution under policy π , d π ( s )

Original:  (cid:80) ∞ t =0 γ t Pr ( s t = s ) , a t ∼ π ( ·| s t )
Revised: = (cid:80) ∞ t =0 γ t Pr ( s t = s ) , a t ∼ π ( ·| s t )

Original: Lemma A.1 below first shows that for two trajectory distributions generated by different policies, their KL divergence can be expressed by the KL divergence between these two policies.Lemma A.1. D KL
Revised: Lemma A.1 below first shows that for two trajectory distributions generated by different policies, their KL divergence can be expressed by the KL divergence between these two policies.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The conditional trajectory distribution can be expressed as:
Revised: The conditional trajectory distribution can be expressed as:

Original: [Equation]
Revised: [Equation]

Revised: Thus,

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Proposition A.2 below shows that the relation between the policy inconsistency error defined in equation 4 and the policy distance.
Revised: Proposition A.2 below shows that the relation between the policy inconsistency error defined in equation 4 and the policy distance.

Original: This proposition shows that multi-step TD learning can indeed be more robust with a dynamic buffer.
Revised: This proposition shows that multi-step TD learning can indeed be more robust with a dynamic buffer [Equation]

Original: Proposition A.2.
Revised: Proposition A.2.

Original: The policy inconsistency error in equation 4 can be upper bounded by
Revised: The policy inconsistency error in equation 4 can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: where r m = sup r − inf r, Q m = sup Q ( s, a ; θ ) − inf Q ( s, a ; θ ) .
Revised: where r [Equation] inf Q ( s, a ; θ ) .

Original: Proof.
Revised: Proof.

Original: Suppose the multi-step TD target is bounded, we have
Revised: Suppose the multi-step TD target is bounded, we have

Original: [Equation]
Revised: [Equation]

Original: According to Pinsker’s inequality,
Revised: According to Pinsker’s inequality,

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: using Lemma A
Revised: using Lemma A.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Our next result, Proposition A.3, below shows that the mismatch between L ( θ ) and ˆ L ( θ ) can be controlled by reducing the KL divergence between the target policy and the behavior policy (complete proof in Appendix A.4.
Revised: Our next result, Proposition A.3, below shows that the mismatch between L ( θ ) and ˆ L ( θ ) can be controlled by reducing the KL divergence between the target policy and the behavior policy (complete proof in Appendix A.4.

Original: Therefore, one can improve the value estimation by eliminating data inconsistency.
Revised: Therefore, one can improve the value estimation by eliminating data inconsistency.

Original: Proposition A
Revised: Proposition A.3.

Original: [Equation]

Original: For the target policy π and behavioural policy b 
Revised: For the target policy π and behavioural policy b , we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: where ∆ = sup | ( Q ( s, a ; θ ) − T ( s, a )) 2 | and the loss functions are defined as

Original: [Equation]
Revised: L ( θ ) =

Original: where ∆ = sup | (  Q ( s 
Revised: E ( s i ,a i ) ∼ d π [( Q ( s i , a i ; θ ) −

Original: [Equation]
Revised: [Equation]

Original: a ; θ ) − T ( s, a  )) 2 | and the loss functions are defined as
Revised: ( s i , a i )) 2 ] , ˆ L ( θ ) =

Original: [Equation]
Revised: [Equation]

Original: ( s i  , a i ; θ ) − T ( s i , a i )) 2 ] .
Revised: ( s i ,a i ) ∼ d b [( Q ( s i , a i ; θ ) − T ( s i , a i )) 2 ] .

Original: Proof.
Revised: Proof.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., the gap between the two loss functions can be bounded by the total variance distance between the two state-action visitation distributions.
Revised: e., the gap between the two loss functions can be bounded by the total variance distance between the two state-action visitation distributions.

Original: According to Achiam et al.
Revised: According to Achiam et al.

Original: (2017), we obtain
Revised: (2017), we obtain

Original: [Equation]
Revised: [Equation]

Original: Thus, the loss function gap can be bounded by the total variance distance between the two policies, i
Revised: Thus, the loss function gap can be bounded by the total variance distance between the two policies, i

Original: [Equation]
Revised: [Equation]

Original: e.,
Revised: e.,

Original: [Equation]
Revised: [Equation]

Original: With Pinsker’s inequality, we can express the upper bound with the KL divergence between the two policies:
Revised: With Pinsker’s inequality, we can express the upper bound with the KL divergence between the two policies:

Original: [Equation]
Revised: [Equation]

Original: B D ETAILS OF RL X 2 WITH TD3 AND SAC
Revised: B D ETAILS OF RL X 2 WITH TD3 AND SAC

Original: In this section, we provide the pseudo-codes of instantiations of RLx2 on TD3 and SAC in Algorithm 4 and Algorithm 5, respectively.
Revised: In this section, we provide the pseudo-codes of instantiations of RLx2 on TD3 and SAC in Algorithm 4 and Algorithm 5, respectively.

Original: We emphasize that RLx2 is a general sparse training framework for off-policy DRL and can be applied to training other DRL algorithms apart from TD3 and SAC, with sparse networks from scratch.
Revised: We emphasize that RLx2 is a general sparse training framework for off-policy DRL and can be applied to training other DRL algorithms apart from TD3 and SAC, with sparse networks from scratch.

Original: Below, we first illustrate the critical steps of RLx2 in Algorithm 4, using TD3 as the base algorithm.
Revised: Below, we first illustrate the critical steps of RLx2 in Algorithm 4, using TD3 as the base algorithm.

Original: Topology evolution is performed in Lines 15-17 and Lines 20-22 in Algorithm 1.
Revised: Topology evolution is performed in Lines 15-17 and Lines 20-22 in Algorithm 1.

Original: Specifically, we first calculate the sparsity of each layer according to the target sparsity of the total model at initialization.
Revised: Specifically, we first calculate the sparsity of each layer according to the target sparsity of the total model at initialization.

Original: The sparsity of each layer is fixed during the training.
Revised: The sparsity of each layer is fixed during the training.

Original: And we use the Erd ˝ os–R ´ enyi strategy, which is introduced in Mocanu et al.
Revised: And we use the Erd ˝ os–R ´ enyi strategy, which is introduced in Mocanu et al.

Original: (2018), to allocate the sparsity to each layer.
Revised: (2018), to allocate the sparsity to each layer.

Original: For a sparse network with L layers, this strategy utilizes the equations below:
Revised: For a sparse network with L layers, this strategy utilizes the equations below:

Original: [Equation]
Revised: [Equation]

Original: where S is the target sparsity of the model, S l is the sparsity of the l -th layer, I l is the input dimensionality of the l -th layer, O l is the output dimensionality of the l -th layer, and k is a constant.
Revised: where S is the target sparsity of the model, S l is the sparsity of the l -th layer, I l is the input dimensionality of the l -th layer, O l is the output dimensionality of the l -th layer, and k is a constant.

Original: The motivation of this strategy is that a layer with more parameters contains more redundancy.
Revised: The motivation of this strategy is that a layer with more parameters contains more redundancy.

Original: As a result, it can be compressed with a higher sparsity.
Revised: As a result, it can be compressed with a higher sparsity.

Original: The topology evolution update is performed every ∆ m time steps.
Revised: The topology evolution update is performed every ∆ m time steps.

Original: Definitions of other hyperparameters related to topology evolution are listed in Algorithm 1 in the manuscript.
Revised: Definitions of other hyperparameters related to topology evolution are listed in Algorithm 1 in the manuscript.

Original: Buffer capacity adjustment is performed in Lines 7-9.
Revised: Buffer capacity adjustment is performed in Lines 7-9.

Original: This adjustment is conducted every ∆ b step, with detailed procedure shown in Algorithm 3.
Revised: This adjustment is conducted every ∆ b step, with detailed procedure shown in Algorithm 3.

Original: Multi-step TD target is computed in Lines 10-13.
Revised: Multi-step TD target is computed in Lines 10-13.

Original: We found that using a multi-step TD target in the early stage of training may result in poor performance because the policy may evolve quickly, which results in severe policy inconsistency.
Revised: We found that using a multi-step TD target in the early stage of training may result in poor performance because the policy may evolve quickly, which results in severe policy inconsistency.

Original: Thus, we start the multi-step TD target only when the number of training steps succeeds a pre-set threshold T 0 .
Revised: Thus, we start the multi-step TD target only when the number of training steps succeeds a pre-set threshold T 0 .

Original: As mentioned in Section 4.2.1, the one-step TD target and multi-step TD target in TD3 are computed as:
Revised: As mentioned in Section 4.2.1, the one-step TD target and multi-step TD target in TD3 are computed as:

Original: [Equation]
Revised: [Equation]

Original: Note that the calculation of the multi-step TD target in SAC is slightly different from that in TD3.
Revised: Note that the calculation of the multi-step TD target in SAC is slightly different from that in TD3.

Original: Specifically, the one-step TD target for SAC is computed as:
Revised: Specifically, the one-step TD target for SAC is computed as:

Original: [Equation]
Revised: [Equation]

Original: where ˜ a t +1 ∼ π ( ·| s t +1 ) , and the n -step TD target for SAC is computed by
Revised: where ˜ a t +1 ∼ π ( ·| s t +1 ) , and the n -step TD target for SAC is computed by

Original: [Equation]
Revised: [Equation]

Original: where ˜ a t + k +1 ∼ π ( ·| s t + k +1 ) , k = 0 , 1 , · · · , n − 1 .
Revised: where ˜ a t + k +1 ∼ π ( ·| s t + k +1 ) , k = 0 , 1 , · · · , n − 1 .

Original: Due to this difference, we will see later in
Revised: Due to this difference, we will see later in

Original: Section C.3 that the resulting FLOPs are slightly different for TD3 and SAC.
Revised: Section C.3 that the resulting FLOPs are slightly different for TD3 and SAC.

Original: Besides, Algorithm 5 also gives the detailed implementation of RLx2 with SAC, where topology evolution is performed in Lines 15-17 and Lines 20-22, buffer capacity adjustment is performed in
Revised: Besides, Algorithm 5 also gives the detailed implementation of RLx2 with SAC, where topology evolution is performed in Lines 15-17 and Lines 20-22, buffer capacity adjustment is performed in

Original: Lines 7-9, and the multi-step TD target is computed in Lines 10-13.
Revised: Lines 7-9, and the multi-step TD target is computed in Lines 10-13.

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table]

Original: if //
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: periodically
Revised: if

Original: [Table]
Revised: [Table]

Original: Set N =
Revised: 9: end if // Check the

Original: [Table]
Revised: [Table]

Original: temporarily if t
Revised: periodically 10: Set t

Original: [Table]
Revised: [Table]

Original: T
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Calculate
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: Calculate multi

Original: [Table]
Revised: [Table]

Original: if
Revised: target 14:

Original: [Table]
Revised: [Table]

Revised: 15: if

Original: [Table]
Revised: [Table]

Original: Topology Evolution (
Revised: Topology Evolution (

Original: [Table]
Revised: [Table]

Original: ) for
Revised: ) for

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 17: end if //Update the [Table] of critic periodically 18: if t
Revised: 17: end if // Update the mask of critic periodically 18: if t

Original: [Table]
Revised: [Table]

Original: d
Revised: d

Original: [Table]
Revised: [Table]

Original: then 19:
Revised: then 19:

Original: [Table]
Revised: [Table]

Original: actor
Revised: actor

Original: [Table]
Revised: [Table]

Original: 20: if
Revised: 20: if

Original: [Table] [Table]
Revised: [Table] [Table]

Original: then 
Revised: then 21:

Original: [Table] [Table] end if //
Revised: Topology

Original: [Table]
Revised: [Table]

Original: the

Original: [Table] of [Table] periodically 23:
Revised: ) 22: end if // Update the mask of actor periodically 23:

Original: Update target networks: θ′ i ←
Revised: Update target networks: θ

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] θ ′ i ←
Revised: ←

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] ′ ⊙
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: ϕ //Target networks are also sparsified with the same mask 24: end if 25: end for
Revised: / Target networks are also sparsified with the same mask 24: end if 25: end for

Original: [Table]
Revised: [Table]

Original: //Update
Revised: [Table] [Table] [Table] [Table]

Original:  the mask of critic periodically 18:
Revised: [Table] [Table] [Table] end if // Update the mask of critic periodically 18:

Original: Update actor network ϕ ← ϕ − λ ∇ ϕ ( − 1 B (cid:80) min j =1 , 2 Q θ j ( s i , a i )) 19: if t/d mod ∆ m = 

Revised: Update actor network ϕ ← ϕ − λ ∇ ϕ ( − 1 B (cid:80) min j =1 , 2 Q θ j ( s i , a i )) 19: if t/d mod ∆ m =

Original: then 20:
Revised: then 20:

Original: Topology Evolution ( π ϕ ) 21: end if //Update the mask of actor periodically 22:
Revised: Topology Evolution ( π ϕ ) 21: end if // Update the mask of actor periodically 22:

Original: Automating Entropy Adjustment: α ← α − λ ∇ α 1 B (cid:80) ( − α log π ( a i | s i ) − α H ) 23:
Revised: Automating Entropy Adjustment: α ← α − λ ∇ α 1 B (cid:80) ( − α log π ( a i | s i ) − α H ) 23:

Original: Update target networks: θ ′
Revised: Update target networks: θ ′

Original: i ←
Revised: i ←

Original: τθ i + (1 − τ ) θ ′ i , θ ′ i ←
Revised: τθ i + (1 − τ ) θ ′ i , θ ′ i ←

Original: θ ′
Revised: θ ′

Original: i ⊙ M
Revised: i ⊙ M

Original: θ i //Target networks are also sparsified with the same mask 24: end for
Revised: θ i // Target networks are also sparsified with the same mask 24: end for

Original: C E XPERIMENTAL D ETAILS
Revised: C E XPERIMENTAL D ETAILS

Original: We provide more experimental details in this section, including the detailed experimental setup, the calculations of model size and FLOPs, and supplementary experiment results.
Revised: We provide more experimental details in this section, including the detailed experimental setup, the calculations of model size and FLOPs, and supplementary experiment results.

Original: C. H ARDWARE S ETUP
Revised: C. H ARDWARE S ETUP

Original: Our experiments are implemented with PyTorch (Paszke et al., 2017) and run on 8x P100 GPUs.
Revised: Our experiments are implemented with PyTorch (Paszke et al., 2017) and run on 8x P100 GPUs.

Original: Each run needs 12 hours for TD3 and 2 days for SAC for three million steps.
Revised: Each run needs 12 hours for TD3 and 2 days for SAC for three million steps.

Original: The code will be open-sourced upon publication of the paper.
Revised: The code will be open-sourced upon publication of the paper.

Original: C. H YPERPARAMETER S ETTINGS FOR R EPRODUCTION
Revised: C. H YPERPARAMETER S ETTINGS FOR R EPRODUCTION

Original: Table 4 presents detailed hyperparameters of RLx2-TD3 and RLx2-SAC in our experiments
Revised: Table 4 presents detailed hyperparameters of RLx2-TD3 and RLx2-SAC in our experiments

Original: . C. C ALCULATION OF M ODEL S IZE AND FLOP S
Revised: . C. C ALCULATION OF M ODEL S IZE AND FLOP S

Original: We present the details of calculating model sizes and FLOPs in this subsection, where focus on fully-connected layers since the networks used in our experiments are all Multilayer Perceptrons (MLPs).
Revised: We present the details of calculating model sizes and FLOPs in this subsection, where focus on fully-connected layers since the networks used in our experiments are all Multilayer Perceptrons (MLPs).

Original: These calculations can be easily extended to convolutional layers or other architectures.
Revised: These calculations can be easily extended to convolutional layers or other architectures.

Original: Besides, we omit the offset term in fully-connected layers in our calculations.
Revised: Besides, we omit the offset term in fully-connected layers in our calculations.

Original: C.3. M ODEL S IZE
Revised: C.3. M ODEL S IZE

Original: We first illustrate the calculation of model sizes, i.e., the total number of parameters in the model.
Revised: We first illustrate the calculation of model sizes, i.e., the total number of parameters in the model.

Original: Initially, for a sparse network with L fully-connected layers, we calculate the model size as:
Revised: Initially, for a sparse network with L fully-connected layers, we calculate the model size as:

Original: [Equation]
Revised: [Equation]

Original: l = (1 − S l ) I l O l ,
Revised: l = ( [Equation] I l O l ,

Original: [Table]
Revised: [Table]

Original: where S l is the sparsity
Revised: where S l is the sparsity

Original: [Table]
Revised: [Table]

Original: I l is the input dimensionality
Revised: I l is the input dimensionality

Original: [Table]
Revised: [Table]

Original: and O l is the output dimensionality of the l -th layer.
Revised: and O l is the output dimensionality of the l -th layer.

Original: Specifically, the “Total Size” column in Table 2 in the manuscript refers the model size including both actor and critic networks during training.
Revised: Specifically, the “Total Size” column in Table 2 in the manuscript refers the model size including both actor and critic networks during training.

Original: For both TD3 and SAC, double Q-learning is adopted, i
Revised: For both TD3 and SAC, double Q-learning is adopted, i

Original: [Table]
Revised: [Table]

Original: e., train two value networks concurrently.
Revised: e., train two value networks concurrently.

Original: Besides
Revised: Besides

Original: [Table]
Revised: [Table]

Original: we also use target networks in our implementations as target critics for both TD3 and SAC, yet a target actor only for TD3.
Revised: we also use target networks in our implementations as target critics for both TD3 and SAC, yet a target actor only for TD3.

Original: Thus, if we denote M Actor and M Critic as model sizes of actor and critic, respectively, the detailed calculation of model sizes can be obtained as shown in the second column of Table
Revised: Thus, if we denote M Actor and M Critic as model sizes of actor and critic, respectively, the detailed calculation of model sizes can be obtained as shown in the second column of Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: C.3. FLOP S
Revised: C.3. FLOP S

Original: Initially, for a sparse network with L fully-connected layer, the required FLOPs for a forward pass is competed as follows (also adopted in Evci et al.
Revised: Initially, for a sparse network with L fully-connected layer, the required FLOPs for a forward pass is competed as follows (also adopted in Evci et al.

Original: (2020) and Molchanov et al.
Revised: (2020) and Molchanov et al.

Original: (2019a)):
Revised: (2019a)):

Original: [Equation]
Revised: [Equation]

Original: where S l is the sparsity, I l is the input dimensionality, and O l is the output dimensionality of the l -th layer.
Revised: where S l is the sparsity, I l is the input dimensionality, and O l is the output dimensionality of the l -th layer.

Original: Denote FLOPs Actor and FLOPs Critic as the FLOPs required in a forward pass of a single actor and critic network, respectively.
Revised: Denote FLOPs Actor and FLOPs Critic as the FLOPs required in a forward pass of a single actor and critic network, respectively.

Original: The inference FLOPs is exactly FLOPs Actor as shown in the last column of Table 5.
Revised: The inference FLOPs is exactly FLOPs Actor as shown in the last column of Table 5.

Original: As for the training FLOPs, the calculation consisted of multiple forward and backward passes in several networks, which will be detailed below.
Revised: As for the training FLOPs, the calculation consisted of multiple forward and backward passes in several networks, which will be detailed below.

Original: In particular, we compute the FLOPs needed for each training iteration.
Revised: In particular, we compute the FLOPs needed for each training iteration.

Original: Besides, we omit the FLOPs of the following processes since they have little influence on the final result.
Revised: Besides, we omit the FLOPs of the following processes since they have little influence on the final result.

Original: (i) Interaction with the environment.
Revised: (i) Interaction with the environment.

Original: Each time the agent decides an action to interact with the environment takes FLOPs as FLOPs Actor , which is much smaller than the FLOPs need for updating networks as shown in Table 5 since Batch Size ≫ 1 .
Revised: Each time the agent decides an action to interact with the environment takes FLOPs as FLOPs Actor , which is much smaller than the FLOPs need for updating networks as shown in Table 5 since Batch Size ≫ 1 .

Original: (ii) Updating target networks.
Revised: (ii) Updating target networks.

Original: Every parameter in the networks is updated as θ ′ ← τθ
Revised: Every parameter in the networks is updated as θ ′ ← τθ

Original: +
Revised: +

Original: (1 − τ ) θ ′ .
Revised: (1 − τ ) θ ′ .

Original: Thus, the number of FLOPs here is the same as the model size, which is also negligible.
Revised: Thus, the number of FLOPs here is the same as the model size, which is also negligible.

Original: (iii) Topology evolution and buffer capacity adjustment.
Revised: (iii) Topology evolution and buffer capacity adjustment.

Original: These two components are performed every 10000 steps.
Revised: These two components are performed every 10000 steps.

Original: Formally speaking, the average FLOPs of topology evolution is give by Batch Size ×
Revised: Formally speaking, the average FLOPs of topology evolution is give by Batch Size ×

Original: [Equation]
Revised: [Equation]

Original: Actor (
Revised: Actor (

Original: [Equation]
Revised: [Equation]

Original: S ( a ) )
Revised: S ( a ) )

Original: [Equation]
Revised: [Equation]

Original: m for the actor, and Batch Size ×
Revised: m for the actor, and Batch Size ×

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: S (
Revised: S (

Original: [Equation]
Revised: [Equation]

Original: ∆ m for the critic, where S ( a ) and S ( c ) are the sparsity of actor and critic, respectively.
Revised: ∆ m for the critic, where S ( a ) and S ( c ) are the sparsity of actor and critic, respectively.

Original: The FLOPs of buffer capacity adjustment is 8 Batch Size × FLOPs Actor ∆ b , where 8 Batch Size is because that we use the oldest 8 Batch Size transitions to compute the policy distance.
Revised: The FLOPs of buffer capacity adjustment is 8 Batch Size × FLOPs Actor ∆ b , where 8 Batch Size is because that we use the oldest 8 Batch Size transitions to compute the policy distance.

Original: Thus, they are both negligible.
Revised: Thus, they are both negligible.

Original: Therefore, we focus on the FLOPs of updating actor and critic.
Revised: Therefore, we focus on the FLOPs of updating actor and critic.

Original: The average FLOPs of updating actor and critic can be given as:
Revised: The average FLOPs of updating actor and critic can be given as:

Original: [Equation]
Revised: [Equation]

Original: where d is the actor update interval ( 2 for TD3 and 1 for SAC in our implementations).
Revised: where d is the actor update interval ( 2 for TD3 and 1 for SAC in our implementations).

Original: Next we calculate the FLOPs of updating actor and critic, i.e. FLOPs update critic and FLOPs update actor .
Revised: Next we calculate the FLOPs of updating actor and critic, i.e. FLOPs update critic and FLOPs update actor .

Original: We first focus on TD3, while that for SAC is similar.
Revised: We first focus on TD3, while that for SAC is similar.

Original: Training FLOPs Calculation in TD 
Revised: Training FLOPs Calculation in TD

Original: (i) Critic FLOPs: Recall the way to update critic (two critics, θ 1 and θ 2 ) in TD3 is given by
Revised: (i) Critic FLOPs: Recall the way to update critic (two critics, θ 1 and θ 2 ) in TD3 is given by

Original: [Equation]
Revised: [Equation]

Original: for
Revised: for

Original: j = 1 , 2 , where B is the batch size, n -step TD target
Revised: j = 1 , 2 , where B is the batch size, n -step TD target

Original: [Equation]
Revised: [Equation]

Original: and θ ′ j refers to the target network.
Revised: and θ ′ j refers to the target network.

Original: Subsequently, we can compute the FLOPs of updating critic as:
Revised: Subsequently, we can compute the FLOPs of updating critic as:

Original: [Equation]
Revised: [Equation]

Original: where FLOPs TD target , FLOPs compute loss , and FLOPs backward pass refer to the numbers of FLOPs in computing the TD targets in forward pass, loss function in forward pass, and gradients in backward pass (backward-propagation), respectively.
Revised: where FLOPs TD target , FLOPs compute loss , and FLOPs backward pass refer to the numbers of FLOPs in computing the TD targets in forward pass, loss function in forward pass, and gradients in backward pass (backward-propagation), respectively.

Original: By Eq.
Revised: By Eq.

Original: (12) we have:
Revised: (12) we have:

Original: [Equation]
Revised: [Equation]

Original: For the FLOPs of gradients backward propagation, FLOPs backward pass , we compute it as two times the computational expense of the forward pass, widely adopted in existing literature (Evci et al., 2020), i.e.,
Revised: For the FLOPs of gradients backward propagation, FLOPs backward pass , we compute it as two times the computational expense of the forward pass, widely adopted in existing literature (Evci et al., 2020), i.e.,

Original: [Equation]
Revised: [Equation]

Original: where the extra factor 2 comes from the cost of double Q-learning.
Revised: where the extra factor 2 comes from the cost of double Q-learning.

Original: Combining Eq.
Revised: Combining Eq.

Original: (13), Eq.
Revised: (13), Eq.

Original: (14), and Eq.
Revised: (14), and Eq.

Original: (15), the FLOPs of updating critic in TD3 is:
Revised: (15), the FLOPs of updating critic in TD3 is:

Original: [Equation]
Revised: [Equation]

Original: (ii) Actor FLOPs: Recall the way to update actor (parameterized by ϕ ) in TD3 is given by
Revised: (ii) Actor FLOPs: Recall the way to update actor (parameterized by ϕ ) in TD3 is given by

Original: [Equation]
Revised: [Equation]

Original: where θ 1 refers to a critic network.
Revised: where θ 1 refers to a critic network.

Original: Subsequently, we compute the FLOPs of updating actor as:
Revised: Subsequently, we compute the FLOPs of updating actor as:

Original: [Equation]
Revised: [Equation]

Original: and similar to the calculations of updating critic, we have:
Revised: and similar to the calculations of updating critic, we have:

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (17), Eq.
Revised: (17), Eq.

Original: (17) and Eq.
Revised: (17) and Eq.

Original: (18), the flops of updating actor in TD3 is:
Revised: (18), the flops of updating actor in TD3 is:

Original: [Equation]
Revised: [Equation]

Original: Training FLOPs Calculation in SAC
Revised: Training FLOPs Calculation in SAC

Original: (i) Critic FLOPs: Calculations of FLOPs for SAC are similar to that in TD3.
Revised: (i) Critic FLOPs: Calculations of FLOPs for SAC are similar to that in TD3.

Original: The way to update the critic in SAC is:
Revised: The way to update the critic in SAC is:

Original: [Equation]
Revised: [Equation]

Original: for j = 1 , 2 , where B is the batch size, n -step TD target
Revised: for j = 1 , 2 , where B is the batch size, n -step TD target

Original: T n =
Revised: T n =

Original: [Equation]
Revised: [Equation]

Original: and θ ′ j refers to the target network.
Revised: and θ ′ j refers to the target network.

Original: Note that the way to compute multi-step TD target in SAC is slightly different from which in TD3, we have:
Revised: Note that the way to compute multi-step TD target in SAC is slightly different from which in TD3, we have:

Original: [Equation]
Revised: [Equation]

Original: Other terms for updating critic are the same as those in Eq.
Revised: Other terms for updating critic are the same as those in Eq.

Original: (13) for TD3.
Revised: (13) for TD3.

Original: Thus, the FLOPs of updating critic in SAC can be computed by
Revised: Thus, the FLOPs of updating critic in SAC can be computed by

Original: [Equation]
Revised: [Equation]

Original: 21) (ii) Actor FLOPs: The way to update the actor in SAC is:
Revised: 21) (ii) Actor FLOPs: The way to update the actor in SAC is:

Original: [Equation]
Revised: [Equation]

Original: where θ j refers to a critic network.
Revised: where θ j refers to a critic network.

Original: Subsequently, we have:
Revised: Subsequently, we have:

Original: [Equation]
Revised: [Equation]

Original: The backward pass FLOPs is the same as that in TD3, i.e.,
Revised: The backward pass FLOPs is the same as that in TD3, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Thus, the FLOPs of updating the actor in SAC is:
Revised: Thus, the FLOPs of updating the actor in SAC is:

Original: [Equation]
Revised: [Equation]

Original: Table 6 shows the relative average FLOPs of each iteration for different algorithms, where the FLOPs of training a sparse network without any of these three methods is set to 1x.
Revised: Table 6 shows the relative average FLOPs of each iteration for different algorithms, where the FLOPs of training a sparse network without any of these three methods is set to 1x.

Original: The sparsity is set to the average sparsity in different environments.
Revised: The sparsity is set to the average sparsity in different environments.

Original: The additional computations induced by topology evolution and dynamic buffer are negligible.
Revised: The additional computations induced by topology evolution and dynamic buffer are negligible.

Original: Using multi-step TD learning also does not increase computations in TD3, and only introduces a small extra computation to SAC ( < 5% ), as analyzed above.
Revised: Using multi-step TD learning also does not increase computations in TD3, and only introduces a small extra computation to SAC ( < 5% ), as analyzed above.

Original: [Table]
Revised: [Table]

Original: C. C OMPARISONS BETWEEN D IFFERENT M ASKS IN O THER E NVIRONMENTS
Revised: C. C OMPARISONS BETWEEN D IFFERENT M ASKS IN O THER E NVIRONMENTS

Original: Figure 8 shows the improvement of the learned sparse network topology by robust value learning.
Revised: Figure 8 shows the improvement of the learned sparse network topology by robust value learning.

Original: We see that in addition to Ant-v3, significant improvements are also achieved in Hopper-v3 and
Revised: We see that in addition to Ant-v3, significant improvements are also achieved in Hopper-v3 and

Original: Walker2d-v3.
Revised: Walker2d-v3.

Original: The only exception is HalfCheetah-v3.
Revised: The only exception is HalfCheetah-v3.

Original: We hypothesize that this environment is easier for the agent to learn (as the performance improves much faster in the early stage than in the other three environments), and a good mask can be found comparatively easier.
Revised: We hypothesize that this environment is easier for the agent to learn (as the performance improves much faster in the early stage than in the other three environments), and a good mask can be found comparatively easier.

Original: It is an interesting direction for future work to systematically analyze this problem
Revised: It is an interesting direction for future work to systematically analyze this problem

Original: [Figure]
Revised: [Figure]

Original: C. T RAINING C URVES OF C OMPARATIVE E VALUATION IN S ECTION 5. 
Revised: C. T RAINING C URVES OF C OMPARATIVE E VALUATION IN S ECTION 5.

Original: Figure 9 and Figure 10 show the training curves of different algorithms in four MuJoCo environments.
Revised: Figure 9 and Figure 10 show the training curves of different algorithms in four MuJoCo environments.

Original: RLx2 outperforms baseline algorithms on all four environments with both TD3 and SAC.
Revised: RLx2 outperforms baseline algorithms on all four environments with both TD3 and SAC.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] Dense

Revised: Tiny Dense

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: Table 7 shows the performance of different algorithms on four MuJoCo environments with standard deviations. Each result is calculated on 8 random seeds. RLx2 does not lead to a larger variance with topology evolution.

Original: [Figure]
Revised: Table [Table]

Revised: 82.3 ± 13.

Original: [Figure]
Revised: 102.4 ± 13.

Original: C. S UPPLEMENTARY R ESULTS FOR A BLATION S TUDY
Revised: C. S UPPLEMENTARY R ESULTS FOR A BLATION S TUDY

Original: Table 7 shows the performance of four environments with different buffer capacities.
Revised: Table 8 shows the performance of four environments with different buffer capacities.

Original: Consistent with the results in Section 5.2, a buffer that is either too small or too large can result in poor performance.
Revised: Consistent with the results in Section 5.2, a buffer that is either too small or too large can result in poor performance.

Original: Our dynamic buffer outperforms buffers with a fixed capacity. 
Revised: Our dynamic buffer outperforms buffers with a fixed capacity. [Table]

Revised: C. S ENSITIVITY A NALYSIS FOR H YPERPARAMETERS

Original: Environment
Revised: In this section [Table] we provide the detailed sensitivity analysis for new hyperparameters used in RLx2, including initial mask update fraction ζ , mask update interval ∆ m , buffer adjustment interval ∆ b , buffer policy distance threshold D 0 , and multi-step delay T 0 . Initial mask update fraction Table 9 shows the performance with different initial mask update fractions (denoted as ζ ) in different environments.

Revised: We also include the special case of keeping the mask static, i

Original: Capacity
Revised: [Table] e., ζ = 0 . From Table 9, we find that the sensitivities of the initial mask update fraction among different environments are similar. Besides, RLx2 achieves better performance with a large value of the initial mask update fraction.

Revised: There is no apparent performance degradation even if ζ is set to 0 . 9 , which may be due to the update fraction annealing scheme.

Original: [Figure]
Revised: [Table]

Revised: Mask update interval Table 10 shows the performance with different mask update intervals (denoted as ∆ m ) in different environments.

Revised: We also include the special case of never updating the mask, i.e., ∆ m = ∞ .

Revised: From Table

Revised: [Table]

Revised: we find that the sensitivities of the mask update interval among different environments are similar.

Revised: Table 10 also shows that a small mask update interval reduces the performance since adjusting the mask too frequently may drop the critical connections before their weights are updated to large values by the optimizer.

Revised: On the contrary, a large mask update interval reduces the impact caused by topology evolution but degrades to training a static sparse network.

Revised: In general, a moderate value of 1 × 10 4 is favored.

Original: [Table]
Revised: [Table]

Revised: Buffer adjustment interval Table 11 shows the performance with different buffer adjustment intervals (denoted as ∆ b ) in different environments.

Revised: We also include the special case of never adjusting the buffer capacity, i.e., ∆ b = ∞ .

Revised: From Table 11, we find that the sensitivities of the buffer adjustment interval among different environments are similar.

Revised: We find the performance is not sensitive to the buffer adjustment interval ∆ b .

Original: [Figure]
Revised: We only observe an apparent performance degradation when the buffer adjustment interval is too large such that the policy distance cannot be reduced promptly [Table] [Table] Buffer policy distance threshold Table 12 shows the performance with different buffer policy distance thresholds (denoted as D 0 ) in different environments.

Revised: We also include the special case of never adjusting the buffer capacity, i.e., D 0 = ∞ .

Revised: From Table 12, we find that the sensitivities of the buffer policy distance threshold among different environments are similar.

Revised: It is shown that a very small threshold may reduce the performance, while the dynamic-capacity buffer helps improve the performance in a wide range of the threshold, especially when D 0 = 0 .

Revised: 1 or 0 . 2 .

Revised: [Table]

Revised: Multi-step delay Table 13 shows the performance with different multi-step delays (denoted as T 0 ) in different environments.

Revised: We also include special cases of no delay, i

Revised: [Table]

Revised: e., T 0 =

Revised: [Table]

Revised: and delay all the time,

Original:  [Table]
Revised: i [Table]

Revised: e., T 0 = ∞ .

Original: [Figure]
Revised: From Table [Table] we find that the sensitivities of the multi-step delay among different environments are similar. Compared to the multi-step method without delay, the performance can be improved by using a delay mechanism in the early stage of training, also outperforming the one-step scheme.

Revised: In addition, the performance is not very sensitive to the multi-step delay T 0 , as we find that performance gains with different delays are similar.

Original: [Table]
Revised: [Table]

Revised: C. A DDITIONAL R ESULTS IN H UMANOID - V

Original: [Figure]
Revised: In this subsection [Table] we investigate the effect of RLx2 in Humanoid-v3, one of the control tasks from MuJoCo.

Revised: Humanoid-v3 is considered relatively complex due to the high input dimensionality (376).

Revised: Thus, apart from the standard 256 neurons in each hidden layer (same as other environments)

Original: [Table]
Revised: [Table]

Revised: we also train a dense model with 1024 neurons in each hidden layer.

Revised: As shown in Table 14, the model with more hidden parameters (1024 hidden dimensions) does not achieve a better performance than a small model (256 hidden dimensions).

Original: S UPPLEMENTARY R ESULTS FOR I NVESTIGATION OF THE L OTTERY T ICKET H YPOTHESIS
Revised: This implies that the latter one seems to have sufficient capacity for the control task in Humanoid-v3. [Table] In addition [Table]

Revised: Table 14 shows the performance of the sparse models trained with RLx2 and other baseline algorithms, where RLx2 succeeds in training a highly sparse model in Humanoid-v3 with performance degradation less than 5% .

Original: We provide additional experiments with TD3 for investigation of the lottery ticket hypothesis (LTH) in the other three environments, including HalfCheetah-v3, Hopper-v
Revised: In particular, RLx2 can achieve sparsity of around 90% in a small model with only 256 hidden neurons, showing its robustness for complex control tasks. Also, RLx2 outperforms most of the baseline algorithms. Although SET shows comparable performance with RLx2, RLx2 shows much higher sample efficiency than SET according to Figure 11.

Original: [Table]
Revised: [Figure]

Revised: When applying RLx2 to larger models with 1024 neurons in each hidden layer, we find that it still performs well with extremely high sparsity (around 99% ).

Original:  and Walker2d-v3.
Revised: We calculate the number of parameters of the two sparse models and find they are very close, as shown in Table 14. It suggests that RLx2 is an effective way to train a sparse model with the least parameters, and is robust to the hidden width of the dense model counterpart. C. S UPPLEMENTARY R ESULTS FOR I NVESTIGATION OF THE L OTTERY T ICKET H YPOTHESIS We provide additional experiments with TD3 for investigation of the lottery ticket hypothesis (LTH) in the other three environments, including HalfCheetah-v3, Hopper-v3, and Walker2d-v3.

Original: As shown in Figure 11, winning tickets fail to achieve the same performance as the dynamic topology in the reinforcement learning setting, while they all perform well under behavior cloning.
Revised: As shown in Figure 12, winning tickets fail to achieve the same performance as the dynamic topology in the reinforcement learning setting, while they all perform well under behavior cloning.

Original: This shows the necessity for reinforcement learning to adjust the network structure during the training process
Revised: This shows the necessity for reinforcement learning to adjust the network structure during the training process

Original: [Table]
Revised: [Figure]

Original: [Figure]
Revised: HalfCheetah-v3 with actor sparsity 92% and critic sparsity 85%

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: ) RL

Original: [Figure]
Revised: [Figure]

Original: C. S TANDARD D EVIATIONS OF R ESULTS IN T ABLE
Revised: Hopper-v3 with

Original: Table 8 shows the performance of different algorithms on four MuJoCo environments with standard deviations. Each result is calculated on 8 random seeds. RLx2 does not lead to a larger variance with topology evolution
Revised: sparsity 99% and critic sparsity 95%

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: C. V ISUALIZATION OF S PARSE M ODELS
Revised: C. V ISUALIZATION OF S PARSE M ODELS

Original: In this section, we show visualizations of the sparse networks obtained under RLx2 in our experiments.
Revised: In this section, we show visualizations of the sparse networks obtained under RLx2 in our experiments.

Original: Note that each layer in the sparse network in our implementation is bound to a binary mask
Revised: Note that each layer in the sparse network in our implementation is bound to a binary mask

Original: M θ l , i.e., θ l = θ l ⊙
Revised: M θ l , i.e., θ l = θ l ⊙

Original: M θ l for the l -th layer.
Revised: M θ l for the l -th layer.

Original: In the rest of this section, we investigate the property of the binary masks with visualization.
Revised: In the rest of this section, we investigate the property of the binary masks with visualization.

Original: By visualizing the final mask after training, we find RLx2 drops redundant dimensions in raw inputs adaptively and efficiently.
Revised: By visualizing the final mask after training, we find RLx2 drops redundant dimensions in raw inputs adaptively and efficiently.

Original: Specifically, Figure 12 shows the distribution of connections to each state dimension in Ant-v3.
Revised: Specifically, Figure 13 shows the distribution of connections to each state dimension in Ant-v3.

Original: The figure shows that the resulting topology almost has no connections for input dimensions 28 − 111 , and the connections heavily concentrate on the first 27 dimensions.
Revised: The figure shows that the resulting topology almost has no connections for input dimensions 28 − 111 , and the connections heavily concentrate on the first 27 dimensions.

Original: This observation indicates that the topology evolution method drops redundant dimensions in raw inputs adaptively and efficiently.
Revised: This observation indicates that the topology evolution method drops redundant dimensions in raw inputs adaptively and efficiently.

Original: A similar observation has also been made in Vischer et al.
Revised: A similar observation has also been made in Vischer et al.

Original: (2022), where it is shown that by iterative magnitude training, input dimensions irrelevant to the task are pruned entirely to help yield a minimal task-relevant representation.
Revised: (2022), where it is shown that by iterative magnitude training, input dimensions irrelevant to the task are pruned entirely to help yield a minimal task-relevant representation.

Original: In Figure 13, we further show visualizations of the raw binary masks (i.e., matrices with items representing neuron connections) of the actor in Ant-v3, where a black dot denotes an existing connection between the corresponding neurons, and a white point means there is no connection.
Revised: In Figure 14, we further show visualizations of the raw binary masks (i.e., matrices with items representing neuron connections) of the actor in Ant-v3, where a black dot denotes an existing connection between the corresponding neurons, and a white point means there is no connection.

Original: As mentioned above, only very few connections are kept for the redundant input dimensions.
Revised: As mentioned above, only very few connections are kept for the redundant input dimensions.

Original: We also find that the connections in the hidden layers tend to concentrate on a subset of neurons, showing that different neurons can play different roles in representations.
Revised: We also find that the connections in the hidden layers tend to concentrate on a subset of neurons, showing that different neurons can play different roles in representations.

Original: [Figure]
Revised: [Figure]