Original: Understanding the Evolution of Linear Regions in Deep Reinforcement Learning
Revised: Understanding the Evolution of Linear Regions in Deep Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Policies produced by deep reinforcement learning are typically characterised bytheir learning curves, but they remain poorly understood in many other respects.
Revised: Policies produced by deep reinforcement learning are typically characterised bytheir learning curves, but they remain poorly understood in many other respects.

Original: ReLU-based policies result in a partitioning of the input space into piecewise linearregions.
Revised: ReLU-based policies result in a partitioning of the input space into piecewise linearregions.

Original: We seek to understand how observed region counts and their densitiesevolve during deep reinforcement learning using empirical results that span arange of continuous control tasks and policy network dimensions.
Revised: We seek to understand how observed region counts and their densitiesevolve during deep reinforcement learning using empirical results that span arange of continuous control tasks and policy network dimensions.

Original: Intuitively, wemay expect that during training, the region density increases in the areas that arefrequently visited by the policy, thereby affording fine-grained control.
Revised: Intuitively, wemay expect that during training, the region density increases in the areas that arefrequently visited by the policy, thereby affording fine-grained control.

Original: We userecent theoretical and empirical results for the linear regions induced by neuralnetworks in supervised learning settings for grounding and comparison of ourresults.
Revised: We userecent theoretical and empirical results for the linear regions induced by neuralnetworks in supervised learning settings for grounding and comparison of ourresults.

Original: Empirically, we find that the region density increases only moderatelythroughout training, as measured along fixed trajectories coming from the finalpolicy.
Revised: Empirically, we find that the region density increases only moderatelythroughout training, as measured along fixed trajectories coming from the finalpolicy.

Original: However, the trajectories themselves also increase in length during training,and thus the region densities decrease as seen from the perspective of the currenttrajectory.
Revised: However, the trajectories themselves also increase in length during training,and thus the region densities decrease as seen from the perspective of the currenttrajectory.

Original: We further provide other empirically-driven observations regardingthe number of visited regions with respect to the number of neurons, changingtrajectory lengths during training, and the impact of network depth on regiondensity.
Revised: We further provide other empirically-driven observations regardingthe number of visited regions with respect to the number of neurons, changingtrajectory lengths during training, and the impact of network depth on regiondensity.

Original:  Introduction
Revised:  Introduction

Original: Deep reinforcement learning (RL) utilizes neural networks to represent the policy and to train thisnetwork to optimize an objective, typically the expected value of time-discounted future rewards.
Revised: Deep reinforcement learning (RL) utilizes neural networks to represent the policy and to train thisnetwork to optimize an objective, typically the expected value of time-discounted future rewards.

Original: Deep RL algorithms have been successfully applied to diverse applications including robotics,challenging games, and an increasing number of real-world decision-and-control problems [FrançoisLavet et al., 2018].
Revised: Deep RL algorithms have been successfully applied to diverse applications including robotics,challenging games, and an increasing number of real-world decision-and-control problems [FrançoisLavet et al., 2018].

Original: For a given choice of task, RL algorithm and policy network configuration, theperformance is commonly characterised via the learning curves which provide insight into both thefinal performance and the efficiency of the learning.
Revised: For a given choice of task, RL algorithm and policy network configuration,the performance is commonly characterised via the learning curves, which provide insight into thelearning efficiency and the final performance.

Original: However, little has been done to understand,in detail, the structure of the state-to-action mappings induced by the control policies and how thisevolves over time.
Revised: However, little has been done to understand the detailedstructure of the state-to-action mappings induced by the control policies and how these evolve overtime.

Original: In this work we aim to further understand deep feed-forward neural network policies that use rectifiedlinear activation functions (rectifier linear units or ReLUs).
Revised: In this work we aim to further understand deep feed-forward neural network policies that use rectifiedlinear activation functions (rectifier linear units or ReLUs).

Original: ReLUs [Nair and Hinton, 2010] areamong the most popular choices of activation functions due to their practical successes [Montúfaret al., 2014].
Revised: ReLUs [Nair and Hinton, 2010] areamong the most popular choices of activation functions due to their practical successes [Montúfaret al., 2014].

Original: For RL, these activations induce a piecewise linear mapping from states to actions,where the input space, i.e, the state space, is divided into distinct linear regions , where within eachregion, the actions are a linear function of the states.
Revised: For RL, these activations induce a piecewise linear mapping from states to actions,where the input space, i.e, the state space, is divided into distinct linear regions , where within eachregion, the actions are a linear function of the states.

Original: Note that the regions are formally an affinefunction of the input, due to the constant-valued bias terms.
Revised: Note that the regions are formally an affinefunction of the input, due to the constant-valued bias terms.

Original: For simplicity and convenience, these are more commonly described as linear regions.
Revised: For simplicity and convenience, these are more commonly described as linear regions.

Original: Figure 1 provides a schematic illustration of theseregions, along with a policy trajectory.
Revised: Figure 1 provides a schematic illustration of theseregions, along with a policy trajectory.

Original: The number of distinct regions into which the input space is divided is a natural measure of networkexpressivity.
Revised: The number of distinct regions into which the input space is divided is a natural measure of networkexpressivity.

Original: Learned functions with many linear regions have the capacity to build complex andflexible decision boundaries.
Revised: Learned functions with many linear regions have the capacity to build complex andflexible decision boundaries.

Original: Thus, the problem of counting the number of linear regions has beenextensively studied in recent literature [Montúfar et al., 2014, Raghu et al., 2017, Hanin and Rolnick,2019a, Serra et al., 2018].
Revised: Thus, the problem of counting the number of linear regions has beenextensively studied in recent literature [Montúfar et al., 2014, Raghu et al., 2017, Hanin and Rolnick,2019a, Serra et al., 2018].

Original: While the maximum number of regions is exponential with respect tonetwork depth [Montúfar et al., 2014], recent work has demonstrated that the number of regions isinstead typically proportional to the number neurons [Hanin and Rolnick, 2019a].
Revised: While the maximum number of regions is exponential with respect tonetwork depth [Montúfar et al., 2014], recent work has demonstrated that the number of regions isinstead typically proportional to the number neurons [Hanin and Rolnick, 2019a].

Original: For RL, we are interested in the local granularity (density) of linear regions along trajectories arisingfrom the policy.
Revised: For RL, we are interested in the local granularity (density) of linear regions along trajectories arisingfrom the policy.

Original: Fine-grained regions afford fine-grained control, and thus we may hypothesize thatregion density increases in regions frequently visited by the policy, in order to afford better control.
Revised: Fine-grained regions afford fine-grained control, and thus we may hypothesize thatregion density increases in regions frequently visited by the policy, in order to afford better control.

Original: Recent work in supervised-learning of image classification is inconsistent with regard to findingsabout the region density seen in the vicinity of data points, with some reporting a decrease [Novaket al., 2018] to provide better generalization and robustness to perturbation, and others not
Revised: Recent work in supervised-learning of image classification is inconsistent with regard to findingsabout the region density seen in the vicinity of data points, with some reporting a decrease [Novaket al., 2018] to provide better generalization and robustness to perturbation, and others not

Original: [Hanin and
Revised: [Hanin and

Original: Rolnick, 2019a].
Revised: Rolnick, 2019a].

Original: For the RL setting, we note that counting regions visited along an episode trajectoryarguably provides a meaningful and task-grounded measurement in contrast to line-segments andellipses that pass through randomly sampled points sampled from training data, which have beenused in the prior works mentioned above.
Revised: For the RL setting, we note that counting regions visited along an episode trajectoryarguably provides a meaningful and task-grounded measurement in contrast to line-segments andellipses that pass through randomly sampled points sampled from training data, which have beenused in the prior works mentioned above.

Revised: We further note that piecewise-affine control strategies arecommonly designed into control systems, e.g., via gain scheduling.

Revised: Understanding how these regionsare designed and distributed by deep RL thus helps establish bridges with these existing methods.

Original: To the best of our knowledge, our work is the first to investigate the structure and evolution oflinear regions of ReLU-based deep RL policies in detail.
Revised: To the best of our knowledge, our work is the first to investigate the structure and evolution oflinear regions of ReLU-based deep RL policies in detail.

Original: We seek to answer several basic empiricalquestions :
Revised: We seek to answer several basic empiricalquestions :

Original: Q1 Do findings for network expressivity, originally developed in supervised learning settings,apply to RL policies?
Revised: Q1 Do findings for network expressivity, originally developed in supervised learning settings,apply to RL policies?

Original: How are the region densities affected by the policy network configuration?
Revised: How are the region densities affected by the policy network configuration?

Original: Do deeper policy networks result in finer-grained regions and hence an [-increasedexpressivity?Q
-]
Revised: Do deeper policy networks result in finer-grained regions and hence an [-increasedexpressivity?Q
-]increasedexpressivity?

Original:  How do the linear regions of a policy evolve during training?
Revised: Q2 How do the linear regions of a policy evolve during training?

Original: Do we see a significantlygreater density of regions emerge along the areas of the state space frequently visited bythe episodic trajectories, thereby allowing for finer-grained control?
Revised: Do we see a significantlygreater density of regions emerge along the areas of the state space frequently visited bythe episodic trajectories, thereby allowing for finer-grained control?

Original: Do random-actiontrajectories see different densities?
Revised: Do random-actiontrajectories see different densities?

Original: The key results can be summarized as follows, for policies trained using proximal policy optimization [Schulman et al., 2017], and evaluated on four different continuous control tasks.
Revised: The key results can be summarized as follows, for policies trained using proximal policy optimization [Schulman et al., 2017], and evaluated on four different continuous control tasks.

Original: Q1:
Revised: Q1:

Original: Thereis a general alignment with recent theoretical and empirical results for supervised learning settings.
Revised: Thereis a general alignment with recent theoretical and empirical results for supervised learning settings.

Original: Region density is principally proportional to the number of neurons, with an additional small observedincrease in density for deeper networks.
Revised: Region density is principally proportional to the number of neurons, with an additional small observedincrease in density for deeper networks.

Original: Q2:
Revised: Q2:

Original: Only a moderate increase of density is observed duringtraining, as measured along fixed final-policy trajectories.
Revised: Only a moderate increase of density is observed duringtraining, as measured along fixed final-policy trajectories.

Original: Therefore, the complexity of a final learnedpolicy does not come principally from increased density on-and-around the optimal trajectories,which is a potentially surprising result.
Revised: Therefore, the complexity of a final learnedpolicy does not come principally from increased density on-and-around the optimal trajectories,which is a potentially surprising result.

Original: In contrast, as measured along the evolving current-policytrajectories, a decrease in region density is observed during training.
Revised: In contrast, as measured along the evolving current-policytrajectories, a decrease in region density is observed during training.

Original: Across all settings, we alsoobserve that the region-transition count, as observed during fixed time-duration episodes, growsduring training before converging to a plateau.
Revised: Across all settings, we also observe that the region-transition count, as observed during fixed time-duration episodes, growsduring training before converging to a plateau.

Original: However, the trajectory length, as measured in the input space, also grows towards a plateau, although not at the same rate, and this leads to variationsin the mean region densities as observed along current trajectories during training.
Revised: However, the trajectory length, as measured in theinput space, also grows towards a plateau, although not at the same rate, and this leads to variationsin the mean region densities as observed along current trajectories during training.

Original:  Related Work
Revised:  Related Work

Original: Understanding the expressivity of a neural network is fundamental to better understanding itsoperation.
Revised: Understanding the expressivity of a neural network is fundamental to better understanding itsoperation.

Original: Several works study exprressivity of deep neural networks with piecewise linear activationsby counting their linear regions [Arora et al., 2016, Bianchini and Scarselli, 2014].
Revised: Several works study exprressivity of deep neural networks with piecewise linear activationsby counting their linear regions [Arora et al., 2016, Bianchini and Scarselli, 2014].

Original: On the theoreticalside, Pascanu et al.
Revised: On the theoreticalside, Pascanu et al.

Original: [2013] show that in the asymptotic limit of many hidden layers, deep ReLUnetworks are capable of separating their input space into exponentially more linear regions comparedwith shallow networks, despite using the same number of computational units.
Revised: [2013] show that in the asymptotic limit of many hidden layers, deep ReLUnetworks are capable of separating their input space into exponentially more linear regions comparedwith shallow networks, despite using the same number of computational units.

Original: Following this,
Revised: Following this,

Original: Montúfar et al.
Revised: Montúfar et al.

Original: [2014] also explore the complexity of functions computable by deep feed-forwardneural networks with ReLU and maxout activations, and provide tighter upper and lower bounds forthe maximal number of linear regions.
Revised: [2014] also explore the complexity of functions computable by deep feed-forwardneural networks with ReLU and maxout activations, and provide tighter upper and lower bounds forthe maximal number of linear regions.

Original: They show that the number of linear regions is polynomialin the network width and exponential with respect to the network depth.
Revised: They show that the number of linear regions is polynomialin the network width and exponential with respect to the network depth.

Original: Furthermore, Raghu et al.[2017] improve the upper bound for ReLU networks by introducing a new set of expressivity measuresand showing that expressivity grows exponentially with network depth for ReLU and tanh activatedneural networks.
Revised: Furthermore, Raghu et al.[2017] improve the upper bound for ReLU networks by introducing a new set of expressivity measuresand showing that expressivity grows exponentially with network depth for ReLU and tanh activatedneural networks.

Original: Serra et al.
Revised: Serra et al.

Original: [2018] generalizes these results by providing even tighter upper andlower bounds for the number of regions for ReLU networks and show that the maximal number ofregions grows exponentially with depth when input dimension is sufficiently large.
Revised: [2018] generalizes these results by providing even tighter upper andlower bounds for the number of regions for ReLU networks and show that the maximal number ofregions grows exponentially with depth when input dimension is sufficiently large.

Original: More recent work that touches on expressivity of depth shows that the effect of depth on theexpressivity of neural networks is likely far below that of the theoretical maximum proposed byprior literature.
Revised: More recent work that touches on expressivity of depth shows that the effect of depth on theexpressivity of neural networks is likely far below that of the theoretical maximum proposed byprior literature.

Original: Hanin and Rolnick [2019b] study the importance of depth on expressivity of neuralnetworks in practice.
Revised: Hanin and Rolnick [2019b] study the importance of depth on expressivity of neuralnetworks in practice.

Original: They show that the average number of linear regions for ReLU networks atinitialization is bounded by the number of neurons raised to the input dimension, and is independentof network depth.
Revised: They show that the average number of linear regions for ReLU networks atinitialization is bounded by the number of neurons raised to the input dimension, and is independentof network depth.

Original: They also empirically show that this bound remains tight during training.
Revised: They also empirically show that this bound remains tight during training.

Original: Similarly,
Revised: Similarly,

Original: Hanin and Rolnick [2019a] find that the average distance to the boundary of the linear regionsdepends only on the number of neurons and not on the network depth – both at initialization andduring training for supervised-learning tasks on ReLU networks.
Revised: Hanin and Rolnick [2019a] find that the average distance to the boundary of the linear regionsdepends only on the number of neurons and not on the network depth – both at initialization andduring training for supervised-learning tasks on ReLU networks.

Original: This strongly suggests that deepernetworks do not necessarily learn more complex functions in comparison to shallow networks.
Revised: This strongly suggests that deepernetworks do not necessarily learn more complex functions in comparison to shallow networks.

Original: Priorto this, a number of works have shown that the strength of deep learning may arise in part from agood match between deep architectures and current training procedures [Mhaskar and Poggio, 2016,
Revised: Priorto this, a number of works have shown that the strength of deep learning may arise in part from agood match between deep architectures and current training procedures [Mhaskar and Poggio, 2016,

Original: Mhaskar et al., 2016, Zhang et al., 2021].
Revised: Mhaskar et al., 2016, Zhang et al., 2021].

Original: Notably, Ba and Caruana [2013] show that, once deepnetworks are trained to perform a task successfully, their behavior can often be replicated by shallownetworks, suggesting that the advantages of depth may be linked to easier learning.
Revised: Notably, Ba and Caruana [2013] show that, once deepnetworks are trained to perform a task successfully, their behavior can often be replicated by shallownetworks, suggesting that the advantages of depth may be linked to easier learning.

Original: Another line of work studies function complexity in terms of robustness to perturbations to the input.
Revised: Another line of work studies function complexity in terms of robustness to perturbations to the input.

Original: Sokoli ´ c et al.
Revised: Sokoli ´ c et al.

Original: [2017] theoretically studies the input-output Jacobian, a measure of robustness, relates togeneralization.
Revised: [2017] theoretically studies the input-output Jacobian, a measure of robustness, relates togeneralization.

Original: Similarly, Zahavy et al.
Revised: Similarly, Zahavy et al.

Original: [2016b] propose a sensitivity measure in terms of adversarialrobustness, and provides theoretical and experimental insights on how it relates to generalization.
Revised: [2016b] propose a sensitivity measure in terms of adversarialrobustness, and provides theoretical and experimental insights on how it relates to generalization.

Original: Novak et al.
Revised: Novak et al.

Original: [2018] also study robustness and sensitivity using the input-output Jacobian and numberof transitions along trajectories in the input space as measures of robustness.
Revised: [2018] also study robustness and sensitivity using the input-output Jacobian and numberof transitions along trajectories in the input space as measures of robustness.

Original: They show that neuralnetworks trained for image classification tasks are more robust to input perturbations in the vicinityof the training data manifold, due to training points lying in regions of lower density.
Revised: They show that neuralnetworks trained for image classification tasks are more robust to input perturbations in the vicinityof the training data manifold, due to training points lying in regions of lower density.

Original: Several otherrecent works have also focused on proposing tight generalization bounds for neural networks [Bartlettet al., 2017, Dziugaite and Roy, 2017, Neyshabur et al., 2017].
Revised: Several otherrecent works have also focused on proposing tight generalization bounds for neural networks [Bartlettet al., 2017, Dziugaite and Roy, 2017, Neyshabur et al., 2017].

Original: There are a number of works that touch on understanding deep neural networks by finding generalprinciples and patterns during training.
Revised: There are a number of works that touch on understanding deep neural networks by finding generalprinciples and patterns during training.

Original: Arpit et al.
Revised: Arpit et al.

Original: [2017] empirically show that deep networksprioritize learning simple patterns of the data during training.
Revised: [2017] empirically show that deep networksprioritize learning simple patterns of the data during training.

Original: Xu et al.
Revised: Xu et al.

Original: [2019] find a similarphenomenon in the case of 2 -layer networks with Sigmoid activations.
Revised: [2019] find a similarphenomenon in the case of 2 -layer networks with Sigmoid activations.

Original: Rahaman et al.
Revised: Rahaman et al.

Original: [2019] studydeep ReLU activated networks through the lens of Fourier analysis and show that while deep neuralnetworks can approximate arbitrary functions, they favour low frequency ones and thus, they exhibit abias towards smooth functions.
Revised: [2019] studydeep ReLU activated networks through the lens of Fourier analysis and show that while deep neuralnetworks can approximate arbitrary functions, they favour low frequency ones and thus, they exhibit abias towards smooth functions.

Original: Samek et al.
Revised: Samek et al.

Original: [2017] present two approaches in explaining predictionsof deep learning models in a classification task, with the first method computing the sensitivity of theprediction with respect to input perturbations and the second method that meaningfully decomposesthe decision in terms of input variables.
Revised: [2017] present two approaches in explaining predictionsof deep learning models in a classification task, with the first method computing the sensitivity of theprediction with respect to input perturbations and the second method that meaningfully decomposesthe decision in terms of input variables.

Original: While deep RL methods are widely used and extensively studied, few works focus on understandingthe policy structure in detail.
Revised: While deep RL methods are widely used and extensively studied, few works focus on understandingthe policy structure in detail.

Original: Zahavy et al.
Revised: Zahavy et al.

Original: [2016a] propose a visualization method to interpret theagent’s actions by describing the Markov Decision Process as a directed graph on a t-SNE map.
Revised: [2016a] propose a visualization method to interpret theagent’s actions by describing the Markov Decision Process as a directed graph on a t-SNE map.

Original: Theythen suggest ways to interpret, debug and optimize deep neural network policies using the proposedvisualization maps.
Revised: Theythen suggest ways to interpret, debug and optimize deep neural network policies using the proposedvisualization maps.

Original: Rupprecht et al.
Revised: Rupprecht et al.

Original: [2019] train a generative model over the state space of Atarigames to visualize states which minimize or maximize given action probabilities.
Revised: [2019] train a generative model over the state space of Atarigames to visualize states which minimize or maximize given action probabilities.

Original: Luo et al.
Revised: Luo et al.

Original: [2018]adapt three visualization techniques to the domain of image-based RL using convolutional neuralnetworks in order to understand the decision making process of the RL agent.
Revised: [2018]adapt three visualization techniques to the domain of image-based RL using convolutional neuralnetworks in order to understand the decision making process of the RL agent.

Original:  Piecewise Linear Regions
Revised:  Piecewise Linear Regions

Original: Throughout this work, we consider RL policies based on ReLU networks .
Revised: Throughout this work, we consider RL policies based on ReLU networks .

Original: A ReLU network isa ReLU-activated feed-forward neural network, or a multi-layer perceptron (MLP) which can beformulated as a scalar function f : R d → R o defined by a neural network with L hidden layers ofwidth d 1 , ..., d L and a o -dimensional output.
Revised: A ReLU network isa ReLU-activated feed-forward neural network, or a multi-layer perceptron (MLP) which can beformulated as a scalar function f : R d → R o defined by a neural network with L hidden layers ofwidth d 1 , ..., d L and a o -dimensional output.

Original: Assuming the output to be 1-dimensional, and followingthe notation of Rahaman et al.
Revised: Assuming the output to be 1-dimensional, and followingthe notation of Rahaman et al.

Original: [2019], we have:
Revised: [2019], we have:

Original: [Equation]
Revised: [Equation]

Original: where T ( k ) : R d k − 1 → R d k computes the weighted sum T ( k ) ( x )
Revised: where T ( k ) : R d k − 1 → R d k computes the weighted sum T ( k ) ( x )

Original: =
Revised: =

Original: W ( k ) x
Revised: W ( k ) x

Original: + b ( k ) for some weightmatrix W ( k ) and bias vector b ( k ) .
Revised: + b ( k ) for some weightmatrix W ( k ) and bias vector b ( k ) .

Original: Here, σ ( u ) = max(0 , u ) denotes the ReLU activation functionacting element-wise ona vector u
Revised: Here, σ ( u ) = max(0 , u ) denotes the ReLU activation functionacting element-wise on vector u

Original: = ( u 1 , ..., u n ) .
Revised: = ( u 1 , ..., u n ) .

Original: Given the ReLU network f from Equation 1, again following Rahaman et al.
Revised: Given the ReLU network f from Equation 1, again following Rahaman et al.

Original: [2019], piecewiselinearity can be explicit written by
Revised: [2019], piecewiselinearity can be explicit written by

Original: [Equation]
Revised: [Equation]

Original: where r is an index for the linear regions P r and 1 P r is the indicator function on P r .
Revised: where r is an index for the linear regions P r and 1 P r is the indicator function on P r .

Original: The 1 × d matrix
Revised: The 1 × d matrix

Original: W r is given by:
Revised: W r is given by:

Original: [Equation]
Revised: [Equation]

Original: where W ( k ) r is obtained from the original weight W ( k ) by setting its j th column to zero wheneverneuron j of layer k − 1 is inactive.
Revised: where W ( k ) r is obtained from the original weight W ( k ) by setting its j th column to zero wheneverneuron j of layer k − 1 is inactive.

Original:  Counting Linear Regions in RL Policies
Revised:  Counting Linear Regions in RL Policies

Original: To answer the key questions posed in the introduction, we need a method for counting the regionsencountered during the episodic trajectories taken during RL.
Revised: To answer the key questions posed in the introduction, we need a method for counting the regionsencountered during the episodic trajectories taken during RL.

Original: For each input x , we encode eachneuron of the policy network with a binary code of 0 if its pre-activation is negative, and with a binarycode 1 otherwise.
Revised: For each input x , we encode eachneuron of the policy network with a binary code of 0 if its pre-activation is negative, and with a binarycode 1 otherwise.

Original: The linear region of the input x can thus be uniquely identified by a concatenationof binary codes of all the neurons in the network called an activation pattern .
Revised: The linear region of the input x can thus be uniquely identified by a concatenationof binary codes of all the neurons in the network called an activation pattern .

Original: Figure 2 illustrates theactivation pattern construction for a 2D input space.
Revised: Figure 2 illustrates theactivation pattern construction for a 2D input space.

Original: Figure 1 provides a schematic illustration of the linear regions for a 2D state space, together withan example trajectory that consists of sequential states encountered by the policy, connected withstraight-line segments.
Revised: Figure 1 provides a schematic illustration of the linear regions for a 2D state space, together withan example trajectory that consists of sequential states encountered by the policy, connected withstraight-line segments.

Original: As seen in the figure, regions can be revisited, which leads to a distinctionbetween the number of transitions between regions and the number of unique regions visited along atrajectory.
Revised: As seen in the figure, regions can be revisited, which leads to a distinctionbetween the number of transitions between regions and the number of unique regions visited along atrajectory.

Original: We compute both of these metrics along episodic trajectories, as enabled by keeping a listof the regions already visited at any point in time when processing a given episode trajectory.
Revised: We compute both of these metrics along episodic trajectories, as enabled by keeping a listof the regions already visited at any point in time when processing a given episode trajectory.

Original: Ourregion-counting method also includes regions that are crossed by the straight-line segments betweensuccessive trajectory states, although the policy itself does not explicitly encounter these regions dueto the discrete-time nature of typical RL environments.
Revised: Ourregion-counting method also includes regions that are crossed by the straight-line segments betweensuccessive trajectory states, although the policy itself does not explicitly encounter these regions dueto the discrete-time nature of typical RL environments.

Original: [Figure]
Revised: [Figure]

Original: blue dot and the goal is to move the agent to reach a target state located at x = 100 .

Original: States are the position x (horizontal axis) and velocity ˙ x (vertical axis) of the agent, and action is the acceleration.

Original: To represent the k -th line segment of the episodic trajectories in the input space, we develop aparameterized line segment given by s ( u )
Revised: To represent the k -th line segment of the episodic trajectories in the input space, we develop aparameterized line segment given by s ( u )

Original: =
Revised: =

Original: (1 − u )
Revised: (1 − u )

Original: s k − 1 + us k , where u ∈ [0 , 1] and s k − 1 , s k ∈ R ddenote the endpoints of the line segment.
Revised: s k − 1 + us k , where u ∈ [0 , 1] and s k − 1 , s k ∈ R ddenote the endpoints of the line segment.

Original: We then calculate the exact number of the linear regions byconsidering the hidden layers of the policy network one at a time, starting from input towards theoutput, and observe how each region can be split by the neurons.
Revised: We then calculate the exact number of the linear regionsover this line segment by considering the hidden layers of the policy network one at a time, startingfrom input towards the output, and observe how each region can be split by the neurons.

Original: Starting with the first layer, weconsider neurons one by one, and identify the point in the domain of u , if any, that induces a changein the binary labeling for that neuron.
Revised: Startingwith the first layer, we consider neurons one by one, and identify the point in the domain of u ,if any, that induces a change in the binary labeling for that neuron.

Original: Each such point subdivides the domain of u into two newregions where the pre-activation will be zero for one of them in the next layer.
Revised: Each such point subdividesthe domain of u into two new regions where the pre-activation will be zero for one of them inthe next layer.

Original: By maintaining a listof these regions and the linear functions defined over them, and whether their pre-activation vanishesin the next layer, we proceed to the neurons of the next layer.
Revised: By maintaining a list of these regions and the linear functions defined over them,and whether their pre-activation vanishes in the next layer, we proceed to the neurons of the nextlayer.

Original: This process repeats for each of theregions resulting from the previous layer.
Revised: This process repeats for each of the regions resulting from the previous layer.

Original: We record the activation patterns of all the final regions,to track state visits during all segments of episode trajectories.
Revised: We recordthe activation patterns of all the final regions, to track state visits during all segments of episodetrajectories.

Original: In the end, for each trajectory τ , wecompute the total region transitions, R T ( τ ) , as well as the number of unique visited regions, R U ( τ ) ,where R T ( τ ) + 1 ≥ R U ( τ ) .
Revised: In the end, for each trajectory τ , we compute the total region transitions, R T ( τ ) , as wellas the number of unique visited regions, R U ( τ ) , where R T ( τ ) + 1 ≥ R U ( τ ) .

Original: We further compute the trajectory length in the input space, L ( τ ) .
Revised: We further computethe trajectory length in the input space, L ( τ ) .

Original: Thisallows us to compute normalized region densities according to ρ ( τ ) =
Revised: This allows us to compute normalized region densitiesaccording to ρ ( τ ) =

Original: R T ( τ ) /
Revised: R T ( τ ) /

Original: ( NL ( τ )) where N isthe total number of neurons in the policy network, in accordance with Hanin and Rolnick [2019a].
Revised: ( NL ( τ )) where N is the total number of neurons in the policy network,in accordance with Hanin and Rolnick [2019a].

Original: We use the above metrics along two types of trajectories.
Revised: We use the above metrics along two types of trajectories.

Original: First, we consider a fixed trajectory,τ ∗ as sampled from the final fully-trained policy.
Revised: First, we consider a fixed trajectory,τ ∗ as sampled from the final fully-trained policy.

Original: Second, we consider current trajectories, τ , assampled from the current policy during training.
Revised: Second, we consider current trajectories, τ , assampled from the current policy during training.

Original: Both of these trajectories offer informative viewsof the evolution of the policy.
Revised: Both of these trajectories offer informative views of the evolution of the policy.

Original: The former offers a direct picture of the linear-region density alonga meaningful, fixed region of the state space, i.e., that of the final optimized policy.
Revised: The former offers a direct picture of the linear-region density alonga meaningful, fixed region of the state space, i.e., that of the final optimized policy.

Original: The latteroffers a view of what the policy trajectories actually encounter during training.
Revised: The latteroffers a view of what the policy trajectories actually encounter during training.

Original: In order to better understand the evolution of τ , we also track its length, L ( τ ) , given that the density is determinedby the region-transition count as well as the length of the trajectory.
Revised: In order to betterunderstand the evolution of τ , we also track its length, L ( τ ) , given that the density is determinedby the region-transition count as well as the length of the trajectory.

Original: Figure 3 shows the evolutionof linear regions and the types of trajectories τ ∗ and τ for a simple 2D toy environment, and
Revised: Figure 3 shows the evolutionof linear regions and the types of trajectories τ ∗ and τ for a simple 2D toy environment, and

Original: a
Revised: a

Original: ReLU-activated policy network of depth 2 and width 8.
Revised: ReLU-activated policy network of depth 2 and width 8.

Original:  Experimental Results
Revised:  Experimental Results

Original: We conduct our experiments on four continuous control tasks including HalfCheetah-v2, Walker-v2,
Revised: We conduct our experiments on four continuous control tasks including HalfCheetah-v2, Walker-v2,

Original: Ant-v2, and Swimmer-v2 environments from the OpenAI gym benchmark suits [Brockman et al.
Revised: Ant-v2, and Swimmer-v2 environments from the OpenAI gym benchmark suits [Brockman et al.

Original: We use the Stable-Baselines3 implementations of the PPO algorithm
Revised: We use the Stable-Baselines3 implementations of the PPO algorithm

Original: [Schulman et al., 2017]in all of our experiments throughout this work.
Revised: [Schulman et al., 2017]in all of our experiments throughout this work.

Original: We run each experiment with 5 different randomseeds and the results show the mean and the standard deviation across the random seeds.
Revised: We run each experiment with 5 different randomseeds and the results show the mean and the standard deviation across the random seeds.

Original: The policynetwork architecture used is an MLP with ReLU activations in all hidden layers.
Revised: The policynetwork architecture used is an MLP with ReLU activations in all hidden layers.

Original: We train 18 policy network configurations, as listed in supplementary material, and having N ∈{ 32 , 48 , 64 , 96 , 128 , 192 } neurons, widths w ∈ { 8 , 16 , 32 , 64 } , and depths d ∈ { 1 , 2 , 3 , 4 } .
Revised: We train 18 policy network configurations, as listed in supplementary material, and having N ∈{ 32 , 48 , 64 , 96 , 128 , 192 } neurons, widths w ∈ { 8 , 16 , 32 , 64 } , and depths d ∈ { 1 , 2 , 3 , 4 } .

Original: Weuse a fixed value function network structure of (64 , 64) in all of our experiments.
Revised: Weuse a fixed value function network structure of (64 , 64) in all of our experiments.

Original: These networkconfigurations are chosen such that the network is fully-capable of learning the task and achievesnear state-of-the-art cumulative reward on the particular task it is trained on.
Revised: These networkconfigurations are chosen such that the network is fully-capable of learning the task and achievesnear state-of-the-art cumulative reward on the particular task it is trained on.

Original: This allows us to studythe evolution of linear regions disentangled from properties such as trainability, which is outside thescope of this work.
Revised: This allows us to studythe evolution of linear regions disentangled from properties such as trainability, which is outside thescope of this work.

Original: The table showing the range of the mean cumulative reward per task, across allnetwork configurations can be found in the supplementary material.
Revised: The table showing the range of the mean cumulative reward per task, across allnetwork configurations can be found in the supplementary material.

Original: We adopt the network initialization and hyperparameters of PPO from StableBaselines3 [Raffinet al., 2021] and train our policy networks on 2M samples (i.e. 2M timesteps in the environment).
Revised: We adopt the network initialization and hyperparameters of PPO from StableBaselines3 [Raffinet al., 2021] and train our policy networks on 2M samples (i.e. 2M timesteps in the environment).

Original: This initialization scheme also matches the best-practice recommendations for on-policy RL,e.g., [Andrychowicz et al., 2020], although it differs from that used in Hanin and Rolnick [2019a] fortheoretical and empirical linear region analysis.
Revised: This initialization scheme also matches the best-practice recommendations for on-policy RL,e.g., [Andrychowicz et al., 2020], although it differs from that used in Hanin and Rolnick [2019a] fortheoretical and empirical linear region analysis.

Revised: A detailed explanation of the initialization and choiceof hyperparameters is available in the supplementary material.

Original: An anonymized version of our code is available insupplemental material and will be made public upon acceptance, under an MIT license.
Revised: An anonymized version of our code isavailable in supplemental material and will be made public upon acceptance, under an MIT license.

Original: In what follows, we focus on the results for the HalfCheetah-v2 environment as a representativeexample.
Revised: In what follows, we focus on the results for the HalfCheetah-v2 environment as a representativeexample.

Original: Similar results for the other environments are provided in the supplemental material.
Revised: Similar results for the other environments are provided in the supplemental material.

Original: 5.1 Evolution of the Density of Linear Regions during Training
Revised: 5.1 Evolution of the Density of Linear Regions during Training

Original: We begin by examining the number of transitions encountered as we sweep along the fixed final-policytrajectory, R T ( τ ∗ ) , as shown in Figure 4(a).
Revised: We begin by examining the number of transitions encountered as we sweep along the fixed final-policytrajectory, R T ( τ ∗ ) , as shown in Figure 4(a).

Original: For clarity, we present results for only three policynetwork configurations.
Revised: For clarity, we present results for only three policynetwork configurations.

Original: The complete set of results is available in the supplementary material.
Revised: The complete set of results is available in the supplementary material.

Original: Thisplot shows a moderate and gradual increase of 50% in the number of region transitions observedin fixed-duration RL episodes during training, with larger policy networks having more regions.
Revised: Thisplot shows a moderate and gradual increase of 50% in the number of region transitions observedin fixed-duration RL episodes during training, with larger policy networks having more regions.

Original: Computing the region density requires dividing by number of neurons, N , and the trajectory length,
Revised: Computing the region density requires dividing by number of neurons, N , and the trajectory length,

Original: L ( τ ∗ ) , which is constant for the fixed final trajectory, τ ∗ .
Revised: L ( τ ∗ ) , which is constant for the fixed final trajectory, τ ∗ .

Original: The result is given in Figure 4(b) whichshows the mean and standard deviation of the normalized region densities across policy networkswith the same depth.
Revised: The result is given in Figure 4(b) whichshows the mean and standard deviation of the normalized region densities across policy networkswith the same depth.

Original: This shows a result that is consistent across all policy network dimensions towithin ± 30 %, and reflects an identical moderate-and-gradual increase during training, regardlessof the policy network configuration.
Revised: This shows a result that is consistent across all policy network dimensions towithin ± 30 %, and reflects an identical moderate-and-gradual increase during training, regardlessof the policy network configuration.

Original: Thus, in response to Q2 , RL policies trained with PPO see amoderate increase in region density along the parts of the state space that are frequently visited bythe policy.
Revised: Thus, in response to Q2 , RL policies trained with PPO see amoderate increase in region density along the parts of the state space that are frequently visited bythe policy.

Original: We later discuss the observed impact of policy-network depth (§5.2).
Revised: We later discuss the observed impact of policy-network depth (§5.2).

Original: We next examine the number of transitions encountered as we sweep along the evolving set of currentpolicy trajectories, R T ( τ ) , shown in Figure 4(c).
Revised: We next examine the number of transitions encountered as we sweep along the evolving set of currentpolicy trajectories, R T ( τ ) , shown in Figure 4(c).

Original: These begin near zero and then rapidly increaseas the policy succeeds at exploration.
Revised: These begin near zero and then rapidly increaseas the policy succeeds at exploration.

Original: Again, for clarity, we show a representative sample of threepolicy network dimensions here and the complete set of results are available in the supplementarymaterial.
Revised: Again, for clarity, we show a representative sample of threepolicy network dimensions here and the complete set of results are available in the supplementarymaterial.

Original: The corresponding region densities are shown in Figure 4(d), together with the fixed-regiondensities which are repeated from Figure 4(b) (see the bottom dashed curve).
Revised: The corresponding region densities are shown in Figure 4(d), together with the fixed-regiondensities which are repeated from Figure 4(b) (see the bottom dashed curve).

Original: We speculate that theobserved densities are higher early on due to the unstructured nature of early exploration andtheform of network initialization, and then decrease over time as the trajectory lengths begin to increase,as shown in Figure 4(e). We compute thenumber of repeat visits to regions as R T − R U , i.e., the region transition count minusthe unique region count. In Figure 4(f), we track this for the current trajectory during training. We
Revised: We speculate that theobserved densities are higher early on due to the unstructured nature of early exploration and the

Original: [Figure]
Revised: [Figure]

Original: N u m b e r o f t r a n s i t i o n s
Revised: N u m b e r o f t r a n s i t i o n s

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: N
Revised: N

Original: [Figure]
Revised: [Figure]

Original: m
Revised: m

Original: [Figure]
Revised: [Figure]

Original: e r o f t r a n s i t i o n s
Revised: e r o f t r a n s i t i o n s

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: T r a j ec t o r y l e n g t h
Revised: T r a j ec t o r y l e n g t h

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: i

Original: [Figure]
Revised: [Figure]

Revised: form of network initialization, and then decrease over time as the trajectory lengths begin to increase,as shown in Figure 4(e).

Revised: We compute the number of repeat visits to regions as R T − R U , i.e., the region transition count minusthe unique region count.

Revised: In Figure 4(f), we track this for the current trajectory during training.

Original: speculate that repeat visits are high early in training because of limited exploration.
Revised: Wespeculate that repeat visits are high early in training because of limited exploration.

Original: Repeat visitsremain later in training because of the cyclic nature of the learned locomotion for HalfCheetah.
Revised: Repeat visitsremain later in training because of the cyclic nature of the learned locomotion for HalfCheetah.

Original: 5.2 Impact of Policy Network Structure on Expressivity
Revised: 5.2 Impact of Policy Network Structure on Expressivity

Original: Our normalized density computation assumes that the number of transitions scales linearly with thenumber of neurons, N .
Revised: Our normalized density computation assumes that the number of transitions scales linearly with thenumber of neurons, N .

Original: However, we empirically observe that there remains a moderate dependenceon the network depth, as seen in Figure 4(b), with deeper networks resulting in moderately denserregions in the learned policies.
Revised: However, we empirically observe that there remains a moderate dependenceon the network depth, as seen in Figure 4(b), with deeper networks resulting in moderately denserregions in the learned policies.

Original: We further examine the policy network expressivity using randominfinite lines.
Revised: We further examine the policy network expressivity using randominfinite lines.

Original: Following Hanin and Rolnick [2019a], for each policy network, we sample 100 randomlines by randomly sampling 100 points from the final trajectory τ ∗ , and connecting each randomlysampled point to the origin of the policy input space.
Revised: Following Hanin and Rolnick [2019a], for each policy network, we sample 100 randomlines by randomly sampling 100 points from the final trajectory τ ∗ , and connecting each randomlysampled point to the origin of the policy input space.

Original: Figure 4(g) shows the mean and standarddeviation of the normalized number of transitions, R T /N , as computed along these infinite lines inrandom directions passing through the origin, across the 100 random lines, over the 5 random seedsof each network configuration.
Revised: Figure 4(g) shows the mean and standarddeviation of the normalized number of transitions, R T /N , as computed along these infinite lines inrandom directions passing through the origin, across the 100 random lines, over the 5 random seedsof each network configuration.

Original: These produce approximately uniform results throughout training,showing no overall changes in mean density in various directions.
Revised: These produce approximately uniform results throughout training,showing no overall changes in mean density in various directions.

Original: These results are also consistentwith the findings of Hanin and Rolnick [2019a] as they show normalized region densities are closeto 1 at initialization and remain roughly constant during training.
Revised: These results are also consistentwith the findings of Hanin and Rolnick [2019a] as they show normalized region densities are closeto 1 at initialization and remain roughly constant during training.

Original: We repeated the same experimentwith 100 random lines through the mean point instead of the origin, by connecting each randomlysampled point to the mean of the 100 sampled points.
Revised: We repeated the same experimentwith 100 random lines through the mean point instead of the origin, by connecting each randomlysampled point to the mean of the 100 sampled points.

Original: The difference between the density over linesthrough the origin and through the mean point is negligible.
Revised: The difference between the density over linesthrough the origin and through the mean point is negligible.

Original: To further examine the behavior of the policy network on areas that it does not focus on exploringduring training, we compute the region density along random trajectories created by taking randomactions in the environment.
Revised: To further examine the behavior of the policy network on areas that it does not focus on exploringduring training, we compute the region density along random trajectories created by taking randomactions in the environment.

Original: For each policy network, we sample 10 such random trajectories τ
Revised: For each policy network, we sample 10 such random trajectories τ

Original: Rand compute the normalized region density over these.
Revised: Rand compute the normalized region density over these.

Original: Figure 4(h) shows the mean and standarddeviation of this metric across the 10 random trajectories, over the 5 random seeds of each networkconfiguration.
Revised: Figure 4(h) shows the mean and standarddeviation of this metric across the 10 random trajectories, over the 5 random seeds of each networkconfiguration.

Original: This shows a very moderate decrease during training, with fairly consistent densitiesfor all policy network configurations.
Revised: This shows a very moderate decrease during training, with fairly consistent densitiesfor all policy network configurations.

Original: We speculate that the slight decrease in density over time arisesfrom the trained policy no longer visiting the same regions as the random-action policy.
Revised: We speculate that the slight decrease in density over time arisesfrom the trained policy no longer visiting the same regions as the random-action policy.

Original: Comparingthe results of random trajectories with fixed and current (Figure 4(b) and Figure 4(d)), we can see thatthe normalized region density over random-action trajectories and a fixed final trajectory are withinthe same range.
Revised: Comparingthe results of random trajectories with fixed and current (Figure 4(b) and Figure 4(d)), we can seethat the normalized region density over random-action trajectories and a fixed final trajectory arewithin the same range.

Original: Also, although this metric is higher over current trajectories early on during training,it eventually decreases and converges to the same range of region densities as the fixed trajectory.
Revised: Also, although this metric is higher over current trajectories early on duringtraining, it eventually decreases and converges to the same range of region densities as the fixedtrajectory.

Original: In Figure 5, we  visualize the regions of a policy network with trained on HalfCheetah in the 17dimensional input space through threedata points randomly sampled from the final fixed trajectoryas well as through three points sampled from a random-action trajectory. 
Revised: In the supplemental material we further directly visualize the regions of a policy networkfor HalfCheetah, as viewed via an embedding plane that either passes through three points sampledfrom the final fixed trajectory, or three points sampled from a random-action trajectory. We do notobserve significant differences between these two cases. 5.3 Additional Experiments

Original: Similarly, we do notobserve any significant difference between the granularity of regions on a plane over points of thefixed trajectory compared with the granularity of regions on a plane over points of a random-actiontrajectory.
Revised: We further study the evolution of linear-region densities for the stochastic actor-critic (SAC) RLalgorithm, to see how it may differ from PPO. The resulting comparison between PPO and SAC resultsis documented in detail in the supplementary material. Region densities observed early on duringtraining are quite different for SAC than they are for PPO, exhibiting high observed densities whichthen rapidly drop, before rising like the PPO case.

Revised: We hypothesize this is due to a combination of (i)the entropy bonus for SAC, which is then annealed away, and (ii) the different network initializationused for the baseline SAC and PPO implementations.

Revised: We examine whether a non-cyclic task, such as LunarLander, exhibits different linear-region evolutionbehavior than our four default tasks, which are naturally biased towards cyclic locomotion.

Revised: In thesupplemental material we show that the transition counts and densities along the fixed trajectory aresimilar in structure to those of the cyclic locomotion tasks, showing that a gradual increase in densityappears to be a general property for the PPO setting, even for non-cyclic tasks.

Revised: Lastly, we also examine the behavior of linear regions in the value function, for the case of

Revised: HalfCheetah and PPO.

Revised: We find that linear-regions in the value functions largely evolve in a verysimilar way to those in the policy.

Original:  Conclusions
Revised:  Conclusions

Original: The structure of ReLU-based reinforcement learning policies remains poorly understood.
Revised: The structure of ReLU-based reinforcement learning policies remains poorly understood.

Original: In thiswork, we have explored a key property of such policies, the density of linear regions observed alongpolicy trajectories, as a proxy for the complexity of functions they can learn.
Revised: In thiswork, we have explored a key property of such policies, the density of linear regions observed alongpolicy trajectories, as a proxy for the complexity of functions they can learn.

Original: We performed a setof experiments to see how this evolves during training and whether and how changes in depth, andwidth of the policy network affect it.
Revised: We performed a setof experiments to see how this evolves during training and whether and how changes in depth, andwidth of the policy network affect it.

Original: The hypothesis that the final policy trajectory sees an increase inregion density is supported to a limited degree, as we observe a moderate increase in density duringtraining, i.e., 50%.
Revised: The hypothesis that the final policy trajectory sees an increase inregion density is supported to a limited degree, as we observe a moderate increase in density duringtraining, i.e., 50%.

Original: This result is consistent across tasks and across policy networks of differentconfigurations.
Revised: This result is consistent across tasks and across policy networks of differentconfigurations.

Original: When observed from the vantage point of the evolving current trajectory, the regiondensity is high to begin with, due to the limited length of early trajectories, and then decreasesasymptotically towards the observed final density.
Revised: When observed from the vantage point of the evolving current trajectory, the regiondensity is high to begin with, due to the limited length of early trajectories, and then decreasesasymptotically towards the observed final density.

Original: For the tasks we consider, trajectory lengths areobserved to grow and asymptotically approach a plateau during training.
Revised: For the tasks we consider, trajectory lengths areobserved to grow and asymptotically approach a plateau during training.

Original: Overall, the surprisingimplication, for our experimental setting, is that the complexity of RL policies appears to be capturedby fine-tuning during learning, rather than a significant growth in the complexity of the functionsobserved along trajectories of the final policy.
Revised: Overall, the surprisingimplication, for our experimental setting, is that the complexity of RL policies appears to be capturedby fine-tuning during learning, rather than a significant growth in the complexity of the functionsobserved along trajectories of the final policy.

Original: The key properties of linear regions as demonstrated for supervised learning of image classifiers, e.g.,[Hanin and Rolnick, 2019a], are shown to generally also hold true for reinforcement learning policies.
Revised: The key properties of linear regions as demonstrated for supervised learning of image classifiers, e.g.,[Hanin and Rolnick, 2019a], are shown to generally also hold true for reinforcement learning policies.

Original: In particular, their expressivity is well modeled as being proportional to the number of neurons, andlargely independent of their width and depth, for ranges of these parameters where good policies can Epoch = Epoch = Epoch =
Revised: In particular, their expressivity is well modeled as being proportional to the number of neurons, andlargely independent of their width and depth, for ranges of these parameters where good policies canstill be learned.

Original: Epoch = [Figure] still be learned. The number of regions observed for randomly oriented line segments through the origin of the input space is also in line with earlier theoretical and empirical results, and does notchange significantly during training.
Revised: The number of regions observed for randomly oriented line segments through theorigin of the input space is also in line with earlier theoretical and empirical results, and does notchange significantly during training.

Revised: We envisage various future directions related to this work, including: (i) understanding the impact ofnon-IID data on decision regions more generally; (ii) consideration of RL algorithms that have less“learning path dependence”, e.g., via frequent (re)distillation onto randomly reinitialized policies;(iii) using visited regions to efficiently distill more interpretable non-parametric policies; and (iv) inreal-world control settings, using knowledge of decision regions, their sizes, and their orderings tobetter understand exploration and safety issues.

Original: Our work comes with a number of limitations.
Revised: Our work comes with a number of limitations.

Original: All the tasks are locomotion-based and thus eventuallyproduce cyclic trajectories.
Revised: All the tasks are locomotion-based and thus eventuallyproduce cyclic trajectories.

Original: This could mean that the results may not generalize to other types oftasks.
Revised: This could mean that the results may not generalize to other types oftasks.

Original: For example, some tasks may not result in transition counts that grow, as generally observedin our experiments.
Revised: For example, some tasks may not result in transition counts that grow, as generally observedin our experiments.

Original: Currently, we compute only the mean region density for a trajectory; it may beuseful to further understand the distribution of observed densities across trajectories.
Revised: Currently, we compute only the mean region density for a trajectory; it may beuseful to further understand the distribution of observed densities across trajectories.

Original: We note that itis common practice in RL to apply input (observation) normalization based on a rolling average ofobservations seen so far [Andrychowicz et al., 2020], and thus our results also employ this strategy.
Revised: We note that itis common practice in RL to apply input (observation) normalization based on a rolling average ofobservations seen so far [Andrychowicz et al., 2020], and thus our results also employ this strategy.

Original: Applying RL without this normalization may impact both the performance of the RL algorithm andthe resulting evolution of transition counts and region-density.
Revised: Applying RL without this normalization may impact both the performance of the RL algorithm andthe resulting evolution of transition counts and region-density.

Original: We leave a number of questions to future work, including understanding the behavior of observedlinear regions in the contexts of other RL algorithms, behavior cloning, and more.
Revised: We leave a number of questions to future work, including understanding the behavior of observedlinear regions in the contexts of other RL algorithms, behavior cloning, and more.

Original: Lastly, we believethat exploring possible connections with the lottery-ticket hypothesis [Frankle and Carbin, 2018] willbe worthwhile.
Revised: Lastly, we believethat exploring possible connections with the lottery-ticket hypothesis [Frankle and Carbin, 2018] willbe worthwhile.

Original: As foundational research related to reinforcement learning, this work could be used to developimproved reinforcement learning policy structures and algorithms, with many possible applications,both  and  terms 
Revised: As foundational research related to reinforcement learning, this work could be used to developimproved reinforcement learning policy structures and algorithms, with many possible applications,both good and bad in terms of societal impact.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 6.
Revised: [Yes] See Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] SeeSection 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes] Code isprovided in the supplemental material.
Revised: [Yes] Code is provided in the supplemental material.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5.(c)
Revised: [Yes] See Section 5.

Original:  Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] We show min/max bars.
Revised: [Yes] We show min/max bars.

Original: See Section 5.(d)
Revised: See Section 5.

Original:  Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [No] Our experiments require modestcompute resources.
Revised: [No] Our experiments require modest compute resources.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]