The lottery ticket hypothesis questions the role of overparameterization in supervised deep learning. But how does the distributional shift inherent to the reinforcement learning problem affect the performance of winning lottery tickets ? In this work, we  show that feed-forward networks trained via supervised policy distillation and reinforcement learning can be pruned to the same level of sparsity . Furthermore, we establish the existence of winning tickets for both on- and off-policy methods in a visual navigation and classic control task . Using a set of carefully designed baseline conditions, we find that the majority of the lottery ticket effect in reinforcement learning can be attributed to the identified mask . The resulting masked observation space eliminates redundant information and yields minimal task-relevant representations. The mask identified by iterative magnitude pruning provides an interpretable inductive bias. Its costly generation can be amortized by training dense agents with low-dimensional input and thereby at lower computational cost .
The lottery ticket hypothesis questions the role of overparameterization in supervised deep learning. But how is the performance of winning lottery tickets affected by the distributional shift inherent to reinforcement learning problems ? In this work, we address this question by comparing sparse agents who have to address the non-stationarity of the exploration-exploitation problem with supervised agents trained to imitate an expert. We show that feed-forward networks trained via reinforcement learning and imitation learning can be pruned to the same level of sparsity , suggesting that the distributional shift has a limited impact on the size of winning tickets . Using a set of carefully designed baseline conditions, we find that the majority of the lottery ticket effect in both learning paradigms can be attributed to the identified mask rather than the weight initialization. The input layer mask selectively prunes entire input dimensions that turn out to be irrelevant for the task at hand. At a moderate level of sparsity the mask identified by iterative magnitude pruning yields minimal task-relevant representations, i.e., an interpretable inductive bias. Finally, we propose a simple initialization rescaling which promotes the robust identification of sparse task representations in low-dimensional control tasks .
R
does the
is the performance of winning lottery tickets affected by the
R
the reinforcement learning problem affect the performance of winning lottery tickets
reinforcement learning problems
A
address this question by comparing sparse agents who have to address the non-stationarity of the exploration-exploitation problem with supervised agents trained to imitate an expert. We
R
supervised policy distillation and reinforcement learning
reinforcement learning and imitation learning
R
. Furthermore, we establish the existence
, suggesting that the distributional shift has a limited impact on the size
D
for both on- and off-policy methods in a visual navigation and classic control task
R
reinforcement learning
both learning paradigms
R
. The resulting masked observation space eliminates redundant information and yields minimal task-relevant representations. The
rather than the weight initialization. The input layer mask selectively prunes entire input dimensions that turn out to be irrelevant for the task at hand. At a moderate level of sparsity the
R
provides
yields minimal task-relevant representations, i.e.,
R
Its costly generation can be amortized by training dense agents with
Finally, we propose a simple initialization rescaling which promotes the robust identification of sparse task representations in
R
input and thereby at lower computational cost
control tasks