Original: O N L OTTERY T ICKETS AND M INIMAL T ASK R EPRESENTATIONS IN D EEP R EINFORCEMENT L EARNING
Revised: O N L OTTERY T ICKETS AND M INIMAL T ASK R EPRESENTATIONS IN D EEP R EINFORCEMENT L EARNING

Original: A BSTRACT
Revised: A BSTRACT

Original: The lottery ticket hypothesis questions the role of overparameterization in supervised deep learning.
Revised: The lottery ticket hypothesis questions the role of overparameterization in supervised deep learning.

Original: But how is the performance of winning lottery tickets affected by the distributional shift inherent to reinforcement learning problems?
Revised: But how is the performance of winning lottery tickets affected by the distributional shift inherent to reinforcement learning problems?

Original: In this work, we address this question by comparing sparse agents who have to address the non-stationarity of the exploration-exploitation problem with supervised agents trained to imitate an expert.
Revised: In this work, we address this question by comparing sparse agents who have to address the non-stationarity of the exploration-exploitation problem with supervised agents trained to imitate an expert.

Original: We show that feed-forward networks trained with behavioural cloning compared to reinforcement learning can be pruned to higher levels of sparsity without performance degradation.
Revised: We show that feed-forward networks trained with behavioural cloning compared to reinforcement learning can be pruned to higher levels of sparsity without performance degradation.

Original: This suggests that in order to solve the RL problem agents require more degrees of freedom.
Revised: This suggests that in order to solve the RL problem agents require more degrees of freedom.

Original: Using a set of carefully designed baseline conditions, we ﬁnd that the majority of the lottery ticket effect in both learning paradigms can be attributed to the identiﬁed mask rather than the weight initialization.
Revised: Using a set of carefully designed baseline conditions, we ﬁnd that the majority of the lottery ticket effect in both learning paradigms can be attributed to the identiﬁed mask rather than the weight initialization.

Original: The input layer mask selectively prunes entire input dimensions that turn out to be irrelevant for the task at hand.
Revised: The input layer mask selectively prunes entire input dimensions that turn out to be irrelevant for the task at hand.

Original: At a moderate level of sparsity the mask identiﬁed by iterative magnitude pruning yields minimal task-relevant representations, i.e., an interpretable inductive bias.
Revised: At a moderate level of sparsity the mask identiﬁed by iterative magnitude pruning yields minimal task-relevant representations, i.e., an interpretable inductive bias.

Original: Finally, we propose a simple initialization rescaling which promotes the robust identiﬁcation of sparse task representations in low-dimensional control tasks.
Revised: Finally, we propose a simple initialization rescaling which promotes the robust identiﬁcation of sparse task representations in low-dimensional control tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent research on the lottery ticket hypothesis (LTH, Frankle & Carbin, 2019; Frankle et al., 2019) in deep learning has demonstrated the existence of very sparse neural networks that train to performance levels comparable to those of their dense counterparts.
Revised: Recent research on the lottery ticket hypothesis (LTH, Frankle & Carbin, 2019; Frankle et al., 2019) in deep learning has demonstrated the existence of very sparse neural networks that train to performance levels comparable to those of their dense counterparts.

Original: These results challenge the role of overparameterization in supervised learning and provide a new perspective on the emergence of stable learning dynamics (Frankle et al., 2020a;b).
Revised: These results challenge the role of overparameterization in supervised learning and provide a new perspective on the emergence of stable learning dynamics (Frankle et al., 2020a;b).

Original: Recently these results have been extended to various domains beyond supervised image classiﬁcation.
Revised: Recently these results have been extended to various domains beyond supervised image classiﬁcation.

Original: These include self-supervised learning (Chen et al., 2020a), natural language processing (Yu et al., 2019; Chen et al., 2020b) and semantic segmentation (Girish et al., 2020).
Revised: These include self-supervised learning (Chen et al., 2020a), natural language processing (Yu et al., 2019; Chen et al., 2020b) and semantic segmentation (Girish et al., 2020).

Original: But how does the lottery ticket ticket phenomenon transfer to reinforcement learning agents?
Revised: But how does the lottery ticket ticket phenomenon transfer to reinforcement learning agents?

Original: One key challenge may be the inherent non-stationarity of the optimization problem in deep reinforcement learning (DRL): The data-generation process is not static, but depends on the changing state of the neural network.
Revised: One key challenge may be the inherent non-stationarity of the optimization problem in deep reinforcement learning (DRL): The data-generation process is not static, but depends on the changing state of the neural network.

Original: Furthermore, a weight may serve different roles at different stages of learning (e.g. during exploration and exploitation).
Revised: Furthermore, a weight may serve different roles at different stages of learning (e.g. during exploration and exploitation).

Original: It is not obvious how a simple weight magnitude-based pruning heuristic acting on a well-performing policy shapes the learning process of the agent.
Revised: It is not obvious how a simple weight magnitude-based pruning heuristic acting on a well-performing policy shapes the learning process of the agent.

Original: In this work, we therefore investigate winning tickets in reinforcement learning and their underlying contributing factors.
Revised: In this work, we therefore investigate winning tickets in reinforcement learning and their underlying contributing factors.

Original: We compare supervised behavioral cloning with DRL, putting a special emphasis on the resulting input representations used for prediction and control.
Revised: We compare supervised behavioral cloning with DRL, putting a special emphasis on the resulting input representations used for prediction and control.

Original: Thereby, we connect the statistical perspective of sparse structure discovery (e.g. Hastie et al., 2019) with the iterative magnitude pruning (IMP, Han et al., 2015) procedure in the context of
Revised: Thereby, we connect the statistical perspective of sparse structure discovery (e.g. Hastie et al., 2019) with the iterative magnitude pruning (IMP, Han et al., 2015) procedure in the context of

Original: Markov decision processes (MDPs).
Revised: Markov decision processes (MDPs).

Original: The contributions of this work are summarized as follows:
Revised: The contributions of this work are summarized as follows:

Original: We show that winning tickets exist in both high-dimensional visual and control tasks (continuous/discrete).
Revised: We show that winning tickets exist in both high-dimensional visual and control tasks (continuous/discrete).

Original: A positive lottery ticket effect is robustly observed for both off-policy DRL algorithms, including Deep-Q-Networks (DQN, Mnih et al., 2015) and on-policy
Revised: A positive lottery ticket effect is robustly observed for both off-policy DRL algorithms, including Deep-Q-Networks (DQN, Mnih et al., 2015) and on-policy

Original: policy-gradient methods (PPO, Schulman et al., 2015; 2017), providing evidence that the lottery ticket effect is a universal phenomenon across optimization formulations in
Revised: policy-gradient methods (PPO, Schulman et al., 2015; 2017), providing evidence that the lottery ticket effect is a universal phenomenon across optimization formulations in

Original: By comparing RL to supervised behavioral cloning (BC), that networks trained with explicit supervision can be pruned to higher sparsity levels before performance starts to degrade, indicating that the RL problem requires a larger amount of parameters to address exploration, distribution shift & quality of the credit assignment signal (section 3.1).
Revised: By comparing RL to supervised behavioral cloning (BC), that networks trained with explicit supervision can be pruned to higher sparsity levels before performance starts to degrade, indicating that the RL problem requires a larger amount of parameters to address exploration, distribution shift & quality of the credit assignment signal (section 3.1).

Original: By introducing a set of lottery ticket baselines (section 3; ﬁgure 1), we disentangle the contributions of the mask, weight initialization and layer-wise pruning ratio.
Revised: By introducing a set of lottery ticket baselines (section 3; ﬁgure 1), we disentangle the contributions of the mask, weight initialization and layer-wise pruning ratio.

Original: We demonstrate that the mask explains most of the ticket effect in behavioral cloning and reinforcement learning for MLP-based agents, whereas the associated weight initialization is less important
Revised: We demonstrate that the mask explains most of the ticket effect in behavioral cloning and reinforcement learning for MLP-based agents, whereas the associated weight initialization is less important

Original: (section 3.2 and 3.3).
Revised: (section 3.2 and 3.3).

Original: For CNN-based agents the weight initialization contributes more.
Revised: For CNN-based agents the weight initialization contributes more.

Original: By visualizing the sparsiﬁed weights for each layer, we ﬁnd that early network layers are pruned more.
Revised: By visualizing the sparsiﬁed weights for each layer, we ﬁnd that early network layers are pruned more.

Original: Entire input dimensions can be rendered invisible to MLP-based agents by the pruning procedure.
Revised: Entire input dimensions can be rendered invisible to MLP-based agents by the pruning procedure.

Original: By this mechanism, IMP compresses the input representation of the MDP (e.g. ﬁgure 1, left column, bottom row) and reveals a minimal task representation for the underlying control problems (section 4).
Revised: By this mechanism, IMP compresses the input representation of the MDP (e.g. ﬁgure 1, left column, bottom row) and reveals a minimal task representation for the underlying control problems (section 4).

Original: The IMP input layer mask not only eliminates obviously redundant dimensions, but also identiﬁes complex relationships between input features and the task of the agent (section 4.1), e.g. the proximity of an enemy or the speed of an approaching object.
Revised: The IMP input layer mask not only eliminates obviously redundant dimensions, but also identiﬁes complex relationships between input features and the task of the agent (section 4.1), e.g. the proximity of an enemy or the speed of an approaching object.

Original: We show that the input masking can be transferred to train dense agents with sparse inputs at lower costs.
Revised: We show that the input masking can be transferred to train dense agents with sparse inputs at lower costs.

Original: We show that the weight initialization scheme is important for discovering minimal representations.
Revised: We show that the weight initialization scheme is important for discovering minimal representations.

Original: Depending on the input size of different layers of the network, global magnitude-based pruning can introduce a strong layer-speciﬁc pruning bias.
Revised: Depending on the input size of different layers of the network, global magnitude-based pruning can introduce a strong layer-speciﬁc pruning bias.

Original: We compare initializations and show that a suitable initialization scheme enables the removal of task-irrelevant dimensions (section 4.2).
Revised: We compare initializations and show that a suitable initialization scheme enables the removal of task-irrelevant dimensions (section 4.2).

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Iterative Magnitude Pruning.
Revised: Iterative Magnitude Pruning.

Original: We use the iterative pruning procedure outlined in Frankle & Carbin (2019) to identify winning tickets.
Revised: We use the iterative pruning procedure outlined in Frankle & Carbin (2019) to identify winning tickets.

Original: We train DRL agents for a previously calibrated number of transitions and track the best performing network checkpoint throughout.
Revised: We train DRL agents for a previously calibrated number of transitions and track the best performing network checkpoint throughout.

Original: Performance is measured by the average return on a set of evaluation episodes.
Revised: Performance is measured by the average return on a set of evaluation episodes.

Original: Afterwards, we prune 20% of the weights with smallest magnitude globally (across all layers).
Revised: Afterwards, we prune 20% of the weights with smallest magnitude globally (across all layers).

Original: The remaining weights are reset to their initial values and we iterate this procedure (train → prune → reset).
Revised: The remaining weights are reset to their initial values and we iterate this procedure (train → prune → reset).

Original: The lottery ticket effect refers to the performance gap between the sparse network obtained via IMP and a randomly initialized network with sparsity-matched random pruning mask.
Revised: The lottery ticket effect refers to the performance gap between the sparse network obtained via IMP and a randomly initialized network with sparsity-matched random pruning mask.

Original: Lottery Tickets in Deep Reinforcement Learning.
Revised: Lottery Tickets in Deep Reinforcement Learning.

Original: Yu et al.
Revised: Yu et al.

Original: (2019) previously demonstrated the existence of tickets in DRL that outperform parameter-matched random initializations.
Revised: (2019) previously demonstrated the existence of tickets in DRL that outperform parameter-matched random initializations.

Original: They obtained tickets for a distributed on-policy actor-critic agent on a subset of environments in the ALE benchmark (Bellemare et al., 2013) as well as a set of discrete control tasks.
Revised: They obtained tickets for a distributed on-policy actor-critic agent on a subset of environments in the ALE benchmark (Bellemare et al., 2013) as well as a set of discrete control tasks.

Original: While they provide empirical evidence for the existence of lottery tickets in DRL, they did not investigate the underlying mechanisms.
Revised: While they provide empirical evidence for the existence of lottery tickets in DRL, they did not investigate the underlying mechanisms.

Original: Here, we aim to unravel these mechanisms.
Revised: Here, we aim to unravel these mechanisms.

Original: To this end, we focus on a diverse set of environments and provide a detailed comparison between supervised behavioral cloning and on-/offpolicy Deep RL with a set of carefully designed ticket baselines.
Revised: To this end, we focus on a diverse set of environments and provide a detailed comparison between supervised behavioral cloning and on-/offpolicy Deep RL with a set of carefully designed ticket baselines.

Original: We analyze the resulting masked representations that the agent learns to act upon and the impact of speciﬁc weight initializations on the resulting sparse networks.
Revised: We analyze the resulting masked representations that the agent learns to act upon and the impact of speciﬁc weight initializations on the resulting sparse networks.

Original: Lottery Tickets with Non-Stationary Data Distributions.
Revised: Lottery Tickets with Non-Stationary Data Distributions.

Original: Desai et al.
Revised: Desai et al.

Original: (2019) investigated whether trained lottery tickets overﬁt the training data distribution under which they were obtained.
Revised: (2019) investigated whether trained lottery tickets overﬁt the training data distribution under which they were obtained.

Original: Using transfer learning tasks on natural language data, they showed that lottery tickets provide general inductive biases.
Revised: Using transfer learning tasks on natural language data, they showed that lottery tickets provide general inductive biases.

Original: Similar ticket transfer results were reported by Morcos et al.
Revised: Similar ticket transfer results were reported by Morcos et al.

Original: and Mehta (2019) in the context of optimizers and vision datasets.
Revised: and Mehta (2019) in the context of optimizers and vision datasets.

Original: Unlike our work, these studies do not investigate within-training covariate shift, but instead focus on transferring ticket initializations after a full IMP run.
Revised: Unlike our work, these studies do not investigate within-training covariate shift, but instead focus on transferring ticket initializations after a full IMP run.

Original: Chen et al.
Revised: Chen et al.

Original: (2021), on the other hand, investigate the ticket phenomenon in the context of lifelong learning and class-incremental image classiﬁcation.
Revised: (2021), on the other hand, investigate the ticket phenomenon in the context of lifelong learning and class-incremental image classiﬁcation.

Original: They propose new pruning strategies to overcome the sequential nature of tasks and need for increased model capacity.
Revised: They propose new pruning strategies to overcome the sequential nature of tasks and need for increased model capacity.

Original: Compared to the
Revised: Compared to the

Original: DRL setting, the covariate shift is here determined by the curriculum schedule of tasks and not the exploration behaviour of the network-parameterized agent.
Revised: DRL setting, the covariate shift is here determined by the curriculum schedule of tasks and not the exploration behaviour of the network-parameterized agent.

Original: Deep Reinforcement Learning Background.
Revised: Deep Reinforcement Learning Background.

Original: In our off-policy DRL experiments, we train DeepQ-Networks (DQN, Mnih et al., 2015) with double Q-learning loss (Van Hasselt et al., 2016) and prioritized experience replay (Schaul et al., 2015).
Revised: In our off-policy DRL experiments, we train DeepQ-Networks (DQN, Mnih et al., 2015) with double Q-learning loss (Van Hasselt et al., 2016) and prioritized experience replay (Schaul et al., 2015).

Original: As a representative on-policy algorithm, we chose
Revised: As a representative on-policy algorithm, we chose

Original: Proximal Policy Optimization (PPO, Schulman et al., 2015; 2017).
Revised: Proximal Policy Optimization (PPO, Schulman et al., 2015; 2017).

Original: PPO is a baseline-corrected policy gradient algorithm which uses a clipping strategy to approximate a computationally expensive trust-region optimization method.
Revised: PPO is a baseline-corrected policy gradient algorithm which uses a clipping strategy to approximate a computationally expensive trust-region optimization method.

Original: For illustrative purposes, we train DQN agents on a visual navigation task, in which an agent has to collect coins in a grid while avoiding poison and two patrollers that are moving in restricted parts of the grid (ﬁgure 1, left column, bottom row; SI B).
Revised: For illustrative purposes, we train DQN agents on a visual navigation task, in which an agent has to collect coins in a grid while avoiding poison and two patrollers that are moving in restricted parts of the grid (ﬁgure 1, left column, bottom row; SI B).

Original: We scale our results to four PyBullet (Ellenberger, 2018) continuous control and a subset of ALE benchmark (Bellemare et al., 2013) environments.
Revised: We scale our results to four PyBullet (Ellenberger, 2018) continuous control and a subset of ALE benchmark (Bellemare et al., 2013) environments.

Original: Due to computational considerations we limit each individual IMP iteration for the ATARI environments to 2.5 million frames.
Revised: Due to computational considerations we limit each individual IMP iteration for the ATARI environments to 2.5 million frames.

Original: All other tasks were trained for a pre-calibrated generous amount of transitions.
Revised: All other tasks were trained for a pre-calibrated generous amount of transitions.

Original: We focus on feedforward value estimators and policies (MLP & CNN) and used default hyperparameters with little tuning (SI C).
Revised: We focus on feedforward value estimators and policies (MLP & CNN) and used default hyperparameters with little tuning (SI C).

Original: Supervised Behavioral Cloning.
Revised: Supervised Behavioral Cloning.

Original: While most supervised learning relies on a stationary data distribution provided by a static dataset, reinforcement learning agents have to acquire their training data in an action-perception loop.
Revised: While most supervised learning relies on a stationary data distribution provided by a static dataset, reinforcement learning agents have to acquire their training data in an action-perception loop.

Original: Since the agent’s behavioural policy is learned over time, the data distribution used in optimization undergoes covariate shift.
Revised: Since the agent’s behavioural policy is learned over time, the data distribution used in optimization undergoes covariate shift.

Original: To study how the covariate shift, additional exploration problem and different credit assignment signal inﬂuence winning tickets, we mimic the supervised learning case by training agents via supervised policy distillation (Rusu et al., 2015; Schmitt et al., 2018).
Revised: To study how the covariate shift, additional exploration problem and different credit assignment signal inﬂuence winning tickets, we mimic the supervised learning case by training agents via supervised policy distillation (Rusu et al., 2015; Schmitt et al., 2018).

Original: We roll out a pre-trained expert policy and train the student agent by minimizing the KL divergence between the student’s and teacher’s policies.
Revised: We roll out a pre-trained expert policy and train the student agent by minimizing the KL divergence between the student’s and teacher’s policies.

Original:  D ISENTANGLING T ICKET C ONTRIBUTIONS IN BC AND D EEP RL
Revised:  D ISENTANGLING T ICKET C ONTRIBUTIONS IN BC AND D EEP RL

Original: Examined Sparsity-Generating IMP Variants
Revised: Examined Sparsity-Generating IMP Variants

Original: Retainweights
Revised: Retainweights

Original: Retainmask
Revised: Retainmask

Original: Retain layer pruning ratio mask/weights ✓ ✓ ✓ mask/permuted ✗ ✓ ✓ permuted/permuted ✗ ✗ ✓ random/re-init ✗ ✗ ✗
Revised: Retain layer pruning ratio mask/weights ✓ ✓ ✓ mask/permuted ✗ ✓ ✓ permuted/permuted ✗ ✗ ✓ random/re-init ✗ ✗ ✗

Original: There are two contributing factors to the lottery ticket effect: The IMP-identiﬁed binary mask and the preserved initialized weights that remain after pruning ( mask/weights ).
Revised: There are two contributing factors to the lottery ticket effect: The IMP-identiﬁed binary mask and the preserved initialized weights that remain after pruning ( mask/weights ).

Original: We aim to disentangle the contributions by introducing a set of counterfactual baselines, which modify the original IMP procedure (ﬁgure 1, middle column; table 1).
Revised: We aim to disentangle the contributions by introducing a set of counterfactual baselines, which modify the original IMP procedure (ﬁgure 1, middle column; table 1).

Original: A ﬁrst baseline estimates how much of the performance of the ticket can be attributed to the initial weights, by means of a layerspeciﬁc permutation of the weights that remain after masking ( mask/permuted ).
Revised: A ﬁrst baseline estimates how much of the performance of the ticket can be attributed to the initial weights, by means of a layerspeciﬁc permutation of the weights that remain after masking ( mask/permuted ).

Original: A second, weaker baseline estimates the contribution of the mask, by also permuting the layer-speciﬁc masks ( permuted/permuted ).
Revised: A second, weaker baseline estimates the contribution of the mask, by also permuting the layer-speciﬁc masks ( permuted/permuted ).

Original: Finally, we consider the standard random/re-init baseline, which samples random binary masks – discarding layer-speciﬁc pruning ratios – and re-initializes all weights at each IMP iteration.
Revised: Finally, we consider the standard random/re-init baseline, which samples random binary masks – discarding layer-speciﬁc pruning ratios – and re-initializes all weights at each IMP iteration.

Original: Throughout the next sections we use these baselines to analyze and compare the factors that give rise to the lottery ticket effect in different control settings.
Revised: Throughout the next sections we use these baselines to analyze and compare the factors that give rise to the lottery ticket effect in different control settings.

Original: 3. C OMPARING W INNING T ICKETS IN S UPERVISED B EHAVIORAL C LONING AND D EEP RL
Revised: 3. C OMPARING W INNING T ICKETS IN S UPERVISED B EHAVIORAL C LONING AND D EEP RL

Original: Does the covariate shift in DRL affect the existence and nature of lottery tickets?
Revised: Does the covariate shift in DRL affect the existence and nature of lottery tickets?

Original: Weights pruned for their small magnitude at the end of the learning process might be needed at earlier stages, e.g., during exploration.
Revised: Weights pruned for their small magnitude at the end of the learning process might be needed at earlier stages, e.g., during exploration.

Original: If this were the case, the performance of the ticket in DRL should degrade for lower levels of sparsity than in a corresponding supervised task.
Revised: If this were the case, the performance of the ticket in DRL should degrade for lower levels of sparsity than in a corresponding supervised task.

Original: To investigate this question, we turn to a behavioral cloning setting, in which a student agent is trained to imitate the stochastic policy of a pre-trained expert and apply magnitude pruning to the student network after each training run iteration.
Revised: To investigate this question, we turn to a behavioral cloning setting, in which a student agent is trained to imitate the stochastic policy of a pre-trained expert and apply magnitude pruning to the student network after each training run iteration.

Original: By collecting transitions based on the static behavioral policy of the expert, we avoid the need for exploration and the effect of an otherwise non-stationary data distribution.
Revised: By collecting transitions based on the static behavioral policy of the expert, we avoid the need for exploration and the effect of an otherwise non-stationary data distribution.

Original: For both the discrete action space and all continuous control tasks we ﬁnd that agents trained with RL and the supervised settings start to degrade in performance at different sparsity levels (ﬁgure 2).
Revised: For both the discrete action space and all continuous control tasks we ﬁnd that agents trained with RL and the supervised settings start to degrade in performance at different sparsity levels (ﬁgure 2).

Original: More speciﬁcally, the maximal return of agents that were trained with reinforcement learning starts to drop at signiﬁcantly earlier stages of the IMP procedure.
Revised: More speciﬁcally, the maximal return of agents that were trained with reinforcement learning starts to drop at signiﬁcantly earlier stages of the IMP procedure.

Original: Hence, the covariate shift inherent to the
Revised: Hence, the covariate shift inherent to the

Original: RL problem increases the minimal required size of a winning ticket in the tasks studied here: More parameters are required for an agent that has to solve the additional exploration problem.
Revised: RL problem increases the minimal required size of a winning ticket in the tasks studied here: More parameters are required for an agent that has to solve the additional exploration problem.

Original: 3. D ISENTANGLING T ICKET C ONTRIBUTIONS IN S UPERVISED B EHAVIORAL C LONING
Revised: 3. D ISENTANGLING T ICKET C ONTRIBUTIONS IN S UPERVISED B EHAVIORAL C LONING

Original: To disentangle the contributions of the initial weights and the weight mask to the ticket effect in supervised behavioral cloning, we next compared the three baselines to the full non-randomized ticket (ﬁgure 1, top, right and 3).
Revised: To disentangle the contributions of the initial weights and the weight mask to the ticket effect in supervised behavioral cloning, we next compared the three baselines to the full non-randomized ticket (ﬁgure 1, top, right and 3).

Original: For most agents, training performance does not degrade substantially when the weights are permuted but the mask is kept intact (no signiﬁcant gap between red and green curves).
Revised: For most agents, training performance does not degrade substantially when the weights are permuted but the mask is kept intact (no signiﬁcant gap between red and green curves).

Original: But we observe a strong decrease in the ability to prune the policy to high sparsity levels if one additionally permutes the IMP-derived weight mask (gap between green and blue curves).
Revised: But we observe a strong decrease in the ability to prune the policy to high sparsity levels if one additionally permutes the IMP-derived weight mask (gap between green and blue curves).

Original: This observation holds for the cloning of expert policies in both discrete and continuous control tasks.
Revised: This observation holds for the cloning of expert policies in both discrete and continuous control tasks.

Original: Hence, the ticket effect can be mainly attributed to the mask rather than the weights.
Revised: Hence, the ticket effect can be mainly attributed to the mask rather than the weights.

Original: Finally, the traditional random/re-init baseline performs worse already for moderate levels of sparsity.
Revised: Finally, the traditional random/re-init baseline performs worse already for moderate levels of sparsity.

Original: The resulting performance gap (blue and yellow curves) indicates the contribution of the layer-wise pruning ratio to the mask effect.
Revised: The resulting performance gap (blue and yellow curves) indicates the contribution of the layer-wise pruning ratio to the mask effect.

Original: These insights emphasize the importance of strong and nuanced baselines to understand the contributions to the full lottery ticket effect.
Revised: These insights emphasize the importance of strong and nuanced baselines to understand the contributions to the full lottery ticket effect.

Original: 3. D ISENTANGLING T ICKET C ONTRIBUTIONS IN D EEP R EINFORCEMENT L EARNING
Revised: 3. D ISENTANGLING T ICKET C ONTRIBUTIONS IN D EEP R EINFORCEMENT L EARNING

Original: The observation that ticket information is mostly carried by the mask rather than the initial weights carries over to the full DRL setups (ﬁgure 1, bottom, right and ﬁgure 4).
Revised: The observation that ticket information is mostly carried by the mask rather than the initial weights carries over to the full DRL setups (ﬁgure 1, bottom, right and ﬁgure 4).

Original: When preserving the information contained in the mask, agents can be pruned to much higher levels of sparsity as compared to the case in which the mask information is distorted (permuted or resampled).
Revised: When preserving the information contained in the mask, agents can be pruned to much higher levels of sparsity as compared to the case in which the mask information is distorted (permuted or resampled).

Original: This effect holds for both PPO- and DQN-based agents and across a wide range of qualitatively different tasks.
Revised: This effect holds for both PPO- and DQN-based agents and across a wide range of qualitatively different tasks.

Original: 2 Interestingly, ticket weights are more important for CNN-based agents (e.g., see ATARI plots).
Revised: 2 Interestingly, ticket weights are more important for CNN-based agents (e.g., see ATARI plots).

Original: We provide further experimental evidence for this observation on four MinAtar games (Young & Tian, 2019) in the supplementary material (SI ﬁgure 10), where we compare
Revised: We provide further experimental evidence for this observation on four MinAtar games (Young & Tian, 2019) in the supplementary material (SI ﬁgure 10), where we compare

Original: MLP- and CNN-based agents.
Revised: MLP- and CNN-based agents.

Original: Unlike the behavioral cloning experiments, for some tasks the permuted/permuted baseline performs worse than the random/reinit conﬁguration.
Revised: Unlike the behavioral cloning experiments, for some tasks the permuted/permuted baseline performs worse than the random/reinit conﬁguration.

Original: This observation only occurs for the PyBullet control tasks in which both the separate critic and actor networks were pruned (ACP).
Revised: This observation only occurs for the PyBullet control tasks in which both the separate critic and actor networks were pruned (ACP).

Original: If instead, one only prunes the actor network (AP), as was done for the PPO agents trained on Cart-Pole, Acrobot and
Revised: If instead, one only prunes the actor network (AP), as was done for the PPO agents trained on Cart-Pole, Acrobot and

Original: ATARI, the permuted/permuted baseline performs stronger (see SI ﬁgure 15).
Revised: ATARI, the permuted/permuted baseline performs stronger (see SI ﬁgure 15).

Original: This result highlights the importance of differences in IMP-derived pruning ratios between agent modules, i.e. separate actor and critic networks.
Revised: This result highlights the importance of differences in IMP-derived pruning ratios between agent modules, i.e. separate actor and critic networks.

Original:  M INIMALLY T ASK -S UF F ICIENT R EPRESENTATIONS VIA IMP
Revised:  M INIMALLY T ASK -S UF F ICIENT R EPRESENTATIONS VIA IMP

Original: The previous results establish the importance of the information contained in the mask for the success of winning tickets.
Revised: The previous results establish the importance of the information contained in the mask for the success of winning tickets.

Original: To better understand this phenomenon and the resulting inductive biases, we analyzed the sparsiﬁed weight matrices, speciﬁcally between the input and the units in layer one.
Revised: To better understand this phenomenon and the resulting inductive biases, we analyzed the sparsiﬁed weight matrices, speciﬁcally between the input and the units in layer one.

Original: The next section sets out to answer the following questions: 1) The derived mask can be thought of as a pair of goggles which guide the processing of state information.
Revised: The next section sets out to answer the following questions: 1) The derived mask can be thought of as a pair of goggles which guide the processing of state information.

Original: What do the masked MLPbased agents observe? 2)
Revised: What do the masked MLPbased agents observe? 2)

Original: Can we transfer the input layer mask and re-use it as an inductive bias for otherwise dense agents?
Revised: Can we transfer the input layer mask and re-use it as an inductive bias for otherwise dense agents?

Original: 3) How much inﬂuence do weight initialization and input dimensionality have on the discovered input layer mask?
Revised: 3) How much inﬂuence do weight initialization and input dimensionality have on the discovered input layer mask?

Original: 4. M INIMAL T ASK R EPRESENTATIONS IN H IGH -D IMENSIONAL V ISUAL T ASKS
Revised: 4. M INIMAL T ASK R EPRESENTATIONS IN H IGH -D IMENSIONAL V ISUAL T ASKS

Original: In the visual navigation experiments, the pruning primarily affects the input layer (ﬁgure 5, top right column).
Revised: In the visual navigation experiments, the pruning primarily affects the input layer (ﬁgure 5, top right column).

Original: When visualizing the cumulative absolute weights of the input layer for an IMPderived DQN agent, we ﬁnd that IMP deletes entire dimensions from the observation vector (ﬁgure 5, left column) by removing all connections between those dimensions and the ﬁrst hidden layer.
Revised: When visualizing the cumulative absolute weights of the input layer for an IMPderived DQN agent, we ﬁnd that IMP deletes entire dimensions from the observation vector (ﬁgure 5, left column) by removing all connections between those dimensions and the ﬁrst hidden layer.

Original: These eliminated input dimensions are not task-relevant.
Revised: These eliminated input dimensions are not task-relevant.

Original: A fully-connected network trained on the remaining dimensions learns the task as quickly as when trained on the complete set of input dimensions (ﬁgure 5, bottom right column).
Revised: A fully-connected network trained on the remaining dimensions learns the task as quickly as when trained on the complete set of input dimensions (ﬁgure 5, bottom right column).

Original: Hence, the IMP-derived mask provides a compressed representation of a high-dimensional observation space, enabling successful training even at high levels of sparsity.
Revised: Hence, the IMP-derived mask provides a compressed representation of a high-dimensional observation space, enabling successful training even at high levels of sparsity.

Original: This result extends to masked input layers of MLP-agents trained on the MinAtar environments (ﬁgure 6 and SI).
Revised: This result extends to masked input layers of MLP-agents trained on the MinAtar environments (ﬁgure 6 and SI).

Original: For the Freeway task we ﬁnd that the IMP mask encodes the notion of velocity of moving cars.
Revised: For the Freeway task we ﬁnd that the IMP mask encodes the notion of velocity of moving cars.

Original: The input layer weights corresponding to the binary observation channel which encodes the slowest moving car are pruned the most, maintaining only the pixels needed to react in time.
Revised: The input layer weights corresponding to the binary observation channel which encodes the slowest moving car are pruned the most, maintaining only the pixels needed to react in time.

Original: The same principle applies to the enemy bullet channel in the SpaceInvaders task.
Revised: The same principle applies to the enemy bullet channel in the SpaceInvaders task.

Original: The agent only needs to know about a potential collision at the next time step in order to avoid being hit.
Revised: The agent only needs to know about a potential collision at the next time step in order to avoid being hit.

Original: Therefore, IMP prunes all information about the enemy bullet that is more than one step away from the agent.
Revised: Therefore, IMP prunes all information about the enemy bullet that is more than one step away from the agent.

Original: For all visualized MinAtar tasks, only the actionable row/column of the agent channel is preserved.
Revised: For all visualized MinAtar tasks, only the actionable row/column of the agent channel is preserved.

Original: In Breakout, additionally the bottom row of the ball and trail channel is pruned since the game terminates once the ball or trail reach it.
Revised: In Breakout, additionally the bottom row of the ball and trail channel is pruned since the game terminates once the ball or trail reach it.

Original: There is no action-relevant information encoded and it can be discarded.
Revised: There is no action-relevant information encoded and it can be discarded.

Original: In summary, we ﬁnd that IMP-derived input layer mask provides a visually interpretable and physically meaningful compression of the observation space.
Revised: In summary, we ﬁnd that IMP-derived input layer mask provides a visually interpretable and physically meaningful compression of the observation space.

Original: 4. M INIMAL R EPRESENTATIONS IN L OW -D IMENSIONAL C ONTROL T ASKS AND THE I SSUE OF I NITIALIZATION S CALES
Revised: 4. M INIMAL R EPRESENTATIONS IN L OW -D IMENSIONAL C ONTROL T ASKS AND THE I SSUE OF I NITIALIZATION S CALES

Original: The continuous and discrete control tasks, on the other hand, rely on low-dimensional state representations.
Revised: The continuous and discrete control tasks, on the other hand, rely on low-dimensional state representations.

Original: At ﬁrst sight, they do not contain obviously uninformative input dimensions that lend themselves to be pruned.
Revised: At ﬁrst sight, they do not contain obviously uninformative input dimensions that lend themselves to be pruned.

Original: We would expect the cumulative weight strength of each input dimension to decrease at equal speed over the course of iterative pruning.
Revised: We would expect the cumulative weight strength of each input dimension to decrease at equal speed over the course of iterative pruning.

Original: Contrary to this initial intuition,
Revised: Contrary to this initial intuition,

Original: IMP still aggressively prunes core dimensions while yielding trainable agents (ﬁgure 7, 8).
Revised: IMP still aggressively prunes core dimensions while yielding trainable agents (ﬁgure 7, 8).

Original: For a subset of continuous control tasks, we ﬁnd that one can prune 20 and up to 50 percent of the input dimensions, while still being able to train the agent to the performance level of a dense counterpart (ﬁgure 7).
Revised: For a subset of continuous control tasks, we ﬁnd that one can prune 20 and up to 50 percent of the input dimensions, while still being able to train the agent to the performance level of a dense counterpart (ﬁgure 7).

Original: The exact amount depends on the considered environment indicating a varying degree of observation over-speciﬁcation across task formulations.
Revised: The exact amount depends on the considered environment indicating a varying degree of observation over-speciﬁcation across task formulations.

Original: For example, the Walker2D environment can train to full performance with only 11 out of 22 observation dimensions, while the Ant environment requires 22 out of 28 dimensions.
Revised: For example, the Walker2D environment can train to full performance with only 11 out of 22 observation dimensions, while the Ant environment requires 22 out of 28 dimensions.

Original: We note an initial positive effect of the weight pruning on the performance of all agents indicating the effectiveness of the implied regularization.
Revised: We note an initial positive effect of the weight pruning on the performance of all agents indicating the effectiveness of the implied regularization.

Original: In the Cart-Pole task the cart position and its velocity are pruned (ﬁgure 8, top).
Revised: In the Cart-Pole task the cart position and its velocity are pruned (ﬁgure 8, top).

Original: Newton’s laws are invariant with regard to the choice of the inertial frame of reference, so the input channels encoding the cart position and velocity do not provide additional information over the pole’s angle and velocity.
Revised: Newton’s laws are invariant with regard to the choice of the inertial frame of reference, so the input channels encoding the cart position and velocity do not provide additional information over the pole’s angle and velocity.

Original: The Acrobot task requires the agent to swing-up a two-link pendulum across a line as fast as possible.
Revised: The Acrobot task requires the agent to swing-up a two-link pendulum across a line as fast as possible.

Original: Here, IMP eliminates all sine and cosine transformations of the rotational angles of the links (ﬁgure 8, middle row, right column).
Revised: Here, IMP eliminates all sine and cosine transformations of the rotational angles of the links (ﬁgure 8, middle row, right column).

Original: Since the swing-up can be achieved by coordinating and increasing the angular velocities of the two links, only these two dimensions have to be preserved from pruning.
Revised: Since the swing-up can be achieved by coordinating and increasing the angular velocities of the two links, only these two dimensions have to be preserved from pruning.

Original: These experiments demonstrate that IMP can yield fundamental insights into what physical information is sufﬁcient to solve a task and leverages them to prune the input representation without impairing performance.
Revised: These experiments demonstrate that IMP can yield fundamental insights into what physical information is sufﬁcient to solve a task and leverages them to prune the input representation without impairing performance.

Original: During our analysis we discovered that the core dimension discovery of IMP crucially depends on the initialization of the input layer weights.
Revised: During our analysis we discovered that the core dimension discovery of IMP crucially depends on the initialization of the input layer weights.

Original: Popular weight initialization heuristics such as the Kaiming family (He et al., 2015) aim to preserve activation magnitudes across network layers.
Revised: Popular weight initialization heuristics such as the Kaiming family (He et al., 2015) aim to preserve activation magnitudes across network layers.

Original: As a result, layers that receive high-dimensional input are initialized at lower values and are therefore prone to more aggressive pruning by IMP.
Revised: As a result, layers that receive high-dimensional input are initialized at lower values and are therefore prone to more aggressive pruning by IMP.

Original: This bias does not occur for Xavier initializations (Glorot & Bengio, 2010, ﬁgure 8, top rows, right column).
Revised: This bias does not occur for Xavier initializations (Glorot & Bengio, 2010, ﬁgure 8, top rows, right column).

Original: Furthermore, we experimented with different approaches to combat this initialization bias.
Revised: Furthermore, we experimented with different approaches to combat this initialization bias.

Original: We ﬁnd that downscaling the Kaiming initialization of the input layer by a factor of ten leads to a robust discovery after few IMP iterations (ﬁgure 8, top rows, middle column).
Revised: We ﬁnd that downscaling the Kaiming initialization of the input layer by a factor of ten leads to a robust discovery after few IMP iterations (ﬁgure 8, top rows, middle column).

Original: While smaller rescaling factors lead to a slower separation of dimensions, larger factors result in overly aggressive pruning of the input layer.
Revised: While smaller rescaling factors lead to a slower separation of dimensions, larger factors result in overly aggressive pruning of the input layer.

Original: In summary, we have shown that the input layer initialization can shape the resulting lottery ticket mask and provide a ﬁrst initialization heuristic promoting efﬁcient minimal task representation discovery.
Revised: In summary, we have shown that the input layer initialization can shape the resulting lottery ticket mask and provide a ﬁrst initialization heuristic promoting efﬁcient minimal task representation discovery.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Summary.
Revised: Summary.

Original: This work has investigated the mechanisms underlying the lottery ticket effect in behavioral cloning and DRL.
Revised: This work has investigated the mechanisms underlying the lottery ticket effect in behavioral cloning and DRL.

Original: We provide evidence that agents which are trained on a stationary data distribution and do not face the exploration problem, can be pruned to higher levels of sparsity while successfully performing their task.
Revised: We provide evidence that agents which are trained on a stationary data distribution and do not face the exploration problem, can be pruned to higher levels of sparsity while successfully performing their task.

Original: For both setups we ﬁnd that the effect can be attributed to the identiﬁed mask rather than to the corresponding weight initialization.
Revised: For both setups we ﬁnd that the effect can be attributed to the identiﬁed mask rather than to the corresponding weight initialization.

Original: We have shown that the mask compresses the input representation by removing task-irrelevant information in the form of visual object pixels or entire control observation dimensions.
Revised: We have shown that the mask compresses the input representation by removing task-irrelevant information in the form of visual object pixels or entire control observation dimensions.

Original: Finally, we found that the pruning process is sensitive to the layer-speciﬁc scale of the weight initialization.
Revised: Finally, we found that the pruning process is sensitive to the layer-speciﬁc scale of the weight initialization.

Original: The interpretability of the resulting minimal representations can be enhanced by heuristically re-scaling the weights in different layers.
Revised: The interpretability of the resulting minimal representations can be enhanced by heuristically re-scaling the weights in different layers.

Original: Limitations.
Revised: Limitations.

Original: While our observations are qualitatively robust, they quantitatively vary across learning paradigms, algorithms, architectures and different tasks.
Revised: While our observations are qualitatively robust, they quantitatively vary across learning paradigms, algorithms, architectures and different tasks.

Original: Compared to our results, weight values have been claimed to play a more central role in the context of lottery tickets in supervised image classiﬁcation (Frankle et al., 2020a).
Revised: Compared to our results, weight values have been claimed to play a more central role in the context of lottery tickets in supervised image classiﬁcation (Frankle et al., 2020a).

Original: Future work will need to investigate under which conditions the initialization of the weight values signiﬁcantly contributes to the ticket effect in RL.
Revised: Future work will need to investigate under which conditions the initialization of the weight values signiﬁcantly contributes to the ticket effect in RL.

Original: Furthermore, most of our experimental results have been obtained for on-policy PPO and policy-based BC agents.
Revised: Furthermore, most of our experimental results have been obtained for on-policy PPO and policy-based BC agents.

Original: Whether the interpretation of the mask effect as a representation regularizer also holds for intermediate and higher layer masks is hard to assess and remains an open question.
Revised: Whether the interpretation of the mask effect as a representation regularizer also holds for intermediate and higher layer masks is hard to assess and remains an open question.

Original: Finally, this study is empirical in its nature and will require further theoretical guidance and foundations.
Revised: Finally, this study is empirical in its nature and will require further theoretical guidance and foundations.

Original: Future Work.
Revised: Future Work.

Original: In future work we want to investigate layer-wise effects, e.g., by studying layerspeciﬁc pruning ratios, normalization schemes and ﬁne-grained baseline analyses, in the hope of identifying sparser, more efﬁcient and interpretable winning tickets.
Revised: In future work we want to investigate layer-wise effects, e.g., by studying layerspeciﬁc pruning ratios, normalization schemes and ﬁne-grained baseline analyses, in the hope of identifying sparser, more efﬁcient and interpretable winning tickets.

Original: Further work needs to be done to disentangle the contributions of the additional exploration problem, distribution shift and credit assignment signal in RL.
Revised: Further work needs to be done to disentangle the contributions of the additional exploration problem, distribution shift and credit assignment signal in RL.

Original: Our proposed baseline analysis can be applied more generally to all studies considering the lottery ticket procedure.
Revised: Our proposed baseline analysis can be applied more generally to all studies considering the lottery ticket procedure.

Original: Finally, we believe that there are many opportunities to modify the original IMP procedure to simplify the discovery of minimal representations.
Revised: Finally, we believe that there are many opportunities to modify the original IMP procedure to simplify the discovery of minimal representations.

Original: For example, pruning entire units of intermediate layers may improve the interpretability of hidden representations.
Revised: For example, pruning entire units of intermediate layers may improve the interpretability of hidden representations.

Revised: A CKNOWLEDGMENTS

Revised: We thank Jonathan Frankle for initial discussions and feedback on the ﬁrst manuscript draft.

Revised: Furthermore, we thank Joram Keijser for reviewing the manuscript.

Revised: This work is funded by the Deutsche

Revised: Forschungsgemeinschaft (DFG, German Research Foundation) under Germanys Excellence Strategy - EXC 2002/1 Science of Intelligence - project number 390523135.

Original: S UPPLEMENTARY I NFORMATION : O N L OTTERY T ICKETS AND M INIMAL T ASK R EPRESENTATIONS IN D EEP R EINFORCEMENT L EARNING A A DDITIONAL R ESULTS A. M IN A TAR D ISTILLATION AND DQN R ESULTS - MLP AND CNN- BASED A GENTS
Revised: S UPPLEMENTARY I NFORMATION : O N L OTTERY T ICKETS AND M INIMAL T ASK R EPRESENTATIONS IN D EEP R EINFORCEMENT L EARNING A A DDITIONAL R ESULTS A. M IN A TAR D ISTILLATION AND DQN R ESULTS - MLP AND CNN- BASED A GENTS

Original: To test the robustness of the lottery ticket phenomenon to different architectures and diverse tasks, we repeat the baseline comparison distillation experiments for the MinAtar environments (see ﬁgure 9).
Revised: To test the robustness of the lottery ticket phenomenon to different architectures and diverse tasks, we repeat the baseline comparison distillation experiments for the MinAtar environments (see ﬁgure 9).

Original: We trained MLP- and CNN-based agents to distill experts’ value estimators and using the same architecture and hyperparameters for all considered games.
Revised: We trained MLP- and CNN-based agents to distill experts’ value estimators and using the same architecture and hyperparameters for all considered games.

Original: For MLP agents the mask consistently contributes most to the the ticket.
Revised: For MLP agents the mask consistently contributes most to the the ticket.

Original: For CNN-based agents and selected games (Asterix and Space
Revised: For CNN-based agents and selected games (Asterix and Space

Original: Invaders) the weight initialization contributes more.
Revised: Invaders) the weight initialization contributes more.

Original: Strengthening an observation in Yu et al.
Revised: Strengthening an observation in Yu et al.

Original: (2019), we observe that the performance deteriorates at different levels of network sparsity depending on the considered game.
Revised: (2019), we observe that the performance deteriorates at different levels of network sparsity depending on the considered game.

Original: Freeway agents keep per- forming well even for high levels of sparsity, while agents trained on Breakout and Space Invaders continually get worse as the sparsity level increases.
Revised: Freeway agents keep per- forming well even for high levels of sparsity, while agents trained on Breakout and Space Invaders continually get worse as the sparsity level increases.

Original: In general we ﬁnd that the qualitative results obtained for MLP agents generalize well to CNN-based agents.
Revised: In general we ﬁnd that the qualitative results obtained for MLP agents generalize well to CNN-based agents.

Original: The only major difference is that unlike the Asterix CNN agent, the MLP agent improves their performance at moderate levels of sparsity.
Revised: The only major difference is that unlike the Asterix CNN agent, the MLP agent improves their performance at moderate levels of sparsity.

Original: In summary, we provide further consistent evidence for the strong contribution of the mask to the lottery ticket effect in DRL (both on-policy and off-policy algorithms).
Revised: In summary, we provide further consistent evidence for the strong contribution of the mask to the lottery ticket effect in DRL (both on-policy and off-policy algorithms).

Original: The results generalize between different architectures indicating that the strength of the overall lottery ticket effect is mainly dictated by the combination of the task-speciﬁcation of the environment and the DRL algorithm.
Revised: The results generalize between different architectures indicating that the strength of the overall lottery ticket effect is mainly dictated by the combination of the task-speciﬁcation of the environment and the DRL algorithm.

Original: A. T HE E F FECT OF N ETWORK C APACITY ON THE A BSOLUTE S IZE OF W INNING T ICKETS
Revised: A. T HE E F FECT OF N ETWORK C APACITY ON THE A BSOLUTE S IZE OF W INNING T ICKETS

Original: The lottery ticket hypothesis suggests that using a larger original network size increases the number of sub-networks which may turn out to be winning tickets (Frankle & Carbin, 2019).
Revised: The lottery ticket hypothesis suggests that using a larger original network size increases the number of sub-networks which may turn out to be winning tickets (Frankle & Carbin, 2019).

Original: To investigate this hypothesis for the case of policy distillation, we analyzed the effect of the initial network size on the lottery ticket effect (ﬁgure 2, right column).
Revised: To investigate this hypothesis for the case of policy distillation, we analyzed the effect of the initial network size on the lottery ticket effect (ﬁgure 2, right column).

Original: Against this initial intuition, we observe that smaller dense networks are capable of maintaining strong performance at higher levels of absolute sparsity as compared to their larger counterparts.
Revised: Against this initial intuition, we observe that smaller dense networks are capable of maintaining strong performance at higher levels of absolute sparsity as compared to their larger counterparts.

Original: Furthermore, the initial network size does not have a strong effect on the relative performance gap between the ticket conﬁguration ( mask/weights ) and the baseline ( permuted/permuted ).
Revised: Furthermore, the initial network size does not have a strong effect on the relative performance gap between the ticket conﬁguration ( mask/weights ) and the baseline ( permuted/permuted ).

Original: We suspect that larger networks can not realize their combinatorial potential due to a an unfavorable layer-wise pruning bias introduced by initialization schemes such as the Kaiming family (He et al., 2015).
Revised: We suspect that larger networks can not realize their combinatorial potential due to a an unfavorable layer-wise pruning bias introduced by initialization schemes such as the Kaiming family (He et al., 2015).

Original: An imbalance between input size and hidden layer size can have strong impact on which weights are targeted by IMP.
Revised: An imbalance between input size and hidden layer size can have strong impact on which weights are targeted by IMP.

Original: We further investigate this relationship in section 4.2.
Revised: We further investigate this relationship in section 4.2.

Original: A. T HE E F
Revised: A. T HE E F

Original: OF D IF FERENT
Revised: OF D IF FERENT

Original: ON
Revised: ON

Original: A. T HE E F FECT OF R EGULARIZATION AND L ATE R EWINDING ON L OTTERY T ICKETS IN DRL
Revised: A. T HE E F FECT OF R EGULARIZATION AND L ATE R EWINDING ON L OTTERY T ICKETS IN DRL

Original: [Figure]
Revised: [Figure]

Original: B M AZE G RID E NVIRONMENT
Revised: B M AZE G RID E NVIRONMENT

Original: The MazeGrid is a visual navigation task: The agent navigates a grid environment, which is ten pixels high and twenty pixels wide.
Revised: The MazeGrid is a visual navigation task: The agent navigates a grid environment, which is ten pixels high and twenty pixels wide.

Original: Each location holds a single unique object.
Revised: Each location holds a single unique object.

Original: There are six types of objects: empty background (black), walls (grey), the agent (cyan), two moving enemies (magenta and green), as well as 42 coins (yellow) and twelve poisons (brown) (visualized in ﬁgure 1, bottom row, left column).
Revised: There are six types of objects: empty background (black), walls (grey), the agent (cyan), two moving enemies (magenta and green), as well as 42 coins (yellow) and twelve poisons (brown) (visualized in ﬁgure 1, bottom row, left column).

Original: The layout of the game is the same in every episode.
Revised: The layout of the game is the same in every episode.

Original: The agent can walk in four directions (up, down, left, right).
Revised: The agent can walk in four directions (up, down, left, right).

Original: The enemies patrol horizontally inside the gaps in the wall.
Revised: The enemies patrol horizontally inside the gaps in the wall.

Original: A full game in motion can be watched in the project repository, which will be released after publication.
Revised: A full game in motion can be watched in the project repository, which will be released after publication.

Original: A game terminates after 200 timesteps, or if the agent collects (walks over) all coins, or if the agent is in the same location as an enemy.
Revised: A game terminates after 200 timesteps, or if the agent collects (walks over) all coins, or if the agent is in the same location as an enemy.

Original: Each collected coin yields a reward of plus one, each collected poison yields minus one, presented immediately to the agent.
Revised: Each collected coin yields a reward of plus one, each collected poison yields minus one, presented immediately to the agent.

Original: To ensure our results do not depend on the speciﬁc encoding of the environment, we compared three different representation of the MazeGrid environment (section A.3):
Revised: To ensure our results do not depend on the speciﬁc encoding of the environment, we compared three different representation of the MazeGrid environment (section A.3):

Original: • The object-map encoding consists of separate one-hot maps for each type of object.
Revised: • The object-map encoding consists of separate one-hot maps for each type of object.

Original: Empty space is encoded explicitly by a separate map, walls however do not have their own map and are only implicitly represented by the lack of any other object.
Revised: Empty space is encoded explicitly by a separate map, walls however do not have their own map and are only implicitly represented by the lack of any other object.

Original: The representation thus contains six maps, ten by twenty pixels each for a total of 1200 binary values for each state.
Revised: The representation thus contains six maps, ten by twenty pixels each for a total of 1200 binary values for each state.

Original: • The RGB encoding consists of the canonical three color channels for each location, resulting in 600 integer values in range [0 , 255] .
Revised: • The RGB encoding consists of the canonical three color channels for each location, resulting in 600 integer values in range [0 , 255] .

Original: Losing information due to occlusion is not an issue in this environment since any location can only hold a single object at a time.
Revised: Losing information due to occlusion is not an issue in this environment since any location can only hold a single object at a time.

Original: • The entangled encoding is derived from the object-map encoding by ﬂattening all values into a vector and multiplying it with a pseudo-random 1200 by 1200 matrix.
Revised: • The entangled encoding is derived from the object-map encoding by ﬂattening all values into a vector and multiplying it with a pseudo-random 1200 by 1200 matrix.

Original: The matrix’s values are sampled independently from U ( − 1 , 1) .
Revised: The matrix’s values are sampled independently from U ( − 1 , 1) .

Original: The matrix remains constant across all IMP iterations, every seed has its own matrix.
Revised: The matrix remains constant across all IMP iterations, every seed has its own matrix.

Original: All experiments in the main text were conducted on the object-map environment.
Revised: All experiments in the main text were conducted on the object-map environment.

Original: Figure 13 provides a comprehensive overview over the baselines to demonstrate that tickets do rely on one speciﬁc encoding.
Revised: Figure 13 provides a comprehensive overview over the baselines to demonstrate that tickets do rely on one speciﬁc encoding.

Original: The importance of the mask is even further highlighted by our results on RGB speciﬁcally.
Revised: The importance of the mask is even further highlighted by our results on RGB speciﬁcally.

Original: C H YPERPARAMETER S ETTINGS FOR R EPRODUCTION
Revised: C H YPERPARAMETER S ETTINGS FOR R EPRODUCTION

Original: All simulations were implemented in Python using the rlpyt DRL training package (Stooke &
Revised: All simulations were implemented in Python using the rlpyt DRL training package (Stooke &

Original: Abbeel, 2019, MIT License) and PyTorch pruning utilities (Paszke et al., 2017).
Revised: Abbeel, 2019, MIT License) and PyTorch pruning utilities (Paszke et al., 2017).

Original: The environments were implented by the OpenAI gym (Brockman et al., 2016, MIT License), MinAtar (Young &
Revised: The environments were implented by the OpenAI gym (Brockman et al., 2016, MIT License), MinAtar (Young &

Original: Tian, 2019, GPL-3.0 License), PyBullet gym (Ellenberger, 2018) packages and the ALE benchmark environment (Bellemare et al., 2013).
Revised: Tian, 2019, GPL-3.0 License), PyBullet gym (Ellenberger, 2018) packages and the ALE benchmark environment (Bellemare et al., 2013).

Original: Furthermore, all visualizations were done using Matplotlib (Hunter, 2007) and Seaborn (Waskom, 2021, BSD-3-Clause License).
Revised: Furthermore, all visualizations were done using Matplotlib (Hunter, 2007) and Seaborn (Waskom, 2021, BSD-3-Clause License).

Original: Finally, the numerical analysis was supported by NumPy (Harris et al., 2020, BSD-3-Clause License).
Revised: Finally, the numerical analysis was supported by NumPy (Harris et al., 2020, BSD-3-Clause License).

Original: We will release the code after the publication of the paper.
Revised: We will release the code after the publication of the paper.

Original: The simulations were conducted on a CPU cluster and no GPUs were used.
Revised: The simulations were conducted on a CPU cluster and no GPUs were used.

Original: Each individual IMP run required between 8 (Cart-Pole and Acrobot), 10 (MazeGrid,
Revised: Each individual IMP run required between 8 (Cart-Pole and Acrobot), 10 (MazeGrid,

Original: MinAtar) and 20 cores (PyBullet and ATARI environments).
Revised: MinAtar) and 20 cores (PyBullet and ATARI environments).

Original: Depending on the setting, a for lottery ticket experiment of 20 to 30 iterations lasts between 2 hours (Cart-Pole) and 5 (ATARI games) days of training time.
Revised: Depending on the setting, a for lottery ticket experiment of 20 to 30 iterations lasts between 2 hours (Cart-Pole) and 5 (ATARI games) days of training time.

Original: C. C ART -
Revised: C. C ART -

Original: [Table]
Revised: [Table]

Original: OLE
Revised: OLE

Original: [Table]
Revised: [Table]

Original: PPO
Revised: PPO

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original:  Distillation Loss
Revised:  Distillation Loss

Original: [Table] [Table]
Revised: [Table] [Table]