Original: Understanding the Evolution of Linear Regions in Deep Reinforcement Learning
Revised: Understanding the Evolution of Linear Regions in Deep Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Policies produced by deep reinforcement learning are typically characterised by their learning curves, but they remain poorly understood in many other respects.
Revised: Policies produced by deep reinforcement learning are typically characterised by their learning curves, but they remain poorly understood in many other respects.

Original: ReLU-based policies result in a partitioning of the input space into piecewise linear regions.
Revised: ReLU-based policies result in a partitioning of the input space into piecewise linear regions.

Original: We seek to understand how observed region counts and their densities evolve during deep reinforcement learning using empirical results that span a range of continuous control tasks and policy network dimensions.
Revised: We seek to understand how observed region counts and their densities evolve during deep reinforcement learning using empirical results that span a range of continuous control tasks and policy network dimensions.

Original: Intuitively, we may expect that during training, the region density increases in the areas that are frequently visited by the policy, thereby affording fine-grained control.
Revised: Intuitively, we may expect that during training, the region density increases in the areas that are frequently visited by the policy, thereby affording fine-grained control.

Original: We use recent theoretical and empirical results for the linear regions induced by neural networks in supervised learning settings for grounding and comparison of our results.
Revised: We use recent theoretical and empirical results for the linear regions induced by neural networks in supervised learning settings for grounding and comparison of our results.

Original: Empirically, we find that the region density increases only moderately throughout training, as measured along fixed trajectories coming from the final policy.
Revised: Empirically, we find that the region density increases only moderately throughout training, as measured along fixed trajectories coming from the final policy.

Original: However, the trajectories themselves also increase in length during training, and thus the region densities decrease as seen from the perspective of the current trajectory.
Revised: However, the trajectories themselves also increase in length during training, and thus the region densities decrease as seen from the perspective of the current trajectory.

Original: Our findings suggest that the complexity of deep reinforcement learning policies does not principally emerge from a significant growth in the complexity of functions observed on-and-around trajectories of the policy.
Revised: Our findings suggest that the complexity of deep reinforcement learning policies does not principally emerge from a significant growth in the complexity of functions observed on-and-around trajectories of the policy.

Original:  Introduction
Revised:  Introduction

Original: Deep reinforcement learning (RL) utilizes neural networks to represent the policy and to train this network to optimize an objective, typically the expected value of time-discounted future rewards.
Revised: Deep reinforcement learning (RL) utilizes neural networks to represent the policy and to train this network to optimize an objective, typically the expected value of time-discounted future rewards.

Original: Deep RL algorithms have been successfully applied to diverse applications including robotics, challenging games, and an increasing number of real-world decision-and-control problems [FrançoisLavet et al., 2018].
Revised: Deep RL algorithms have been successfully applied to diverse applications including robotics, challenging games, and an increasing number of real-world decision-and-control problems [FrançoisLavet et al., 2018].

Original: For a given choice of task, RL algorithm, and policy network configuration, the performance is commonly characterised via the learning curves, which provide insight into the learning efficiency and the final performance.
Revised: For a given choice of task, RL algorithm, and policy network configuration, the performance is commonly characterised via the learning curves, which provide insight into the learning efficiency and the final performance.

Original: However, little has been done to understand the detailed structure of the state-to-action mappings induced by the control policies and how these evolve over time.
Revised: However, little has been done to understand the detailed structure of the state-to-action mappings induced by the control policies and how these evolve over time.

Original: In this work, we aim to further understand deep feed-forward neural network policies that use rectified linear activation functions (rectifier linear units or ReLUs).
Revised: In this work, we aim to further understand deep feed-forward neural network policies that use rectified linear activation functions (rectifier linear units or ReLUs).

Original: ReLUs [Nair and Hinton, 2010] are among the most popular choices of activation functions due to their practical successes [Montúfar et al., 2014].
Revised: ReLUs [Nair and Hinton, 2010] are among the most popular choices of activation functions due to their practical successes [Montúfar et al., 2014].

Original: For RL, these activations induce a piecewise linear mapping from states to actions, where the input space, i.e, the state space, is divided into distinct linear regions , where within each region, the actions are a linear function of the states.
Revised: For RL, these activations induce a piecewise linear mapping from states to actions, where the input space, i.e, the state space, is divided into distinct linear regions , where within each region, the actions are a linear function of the states.

Original: Note that the regions are formally an affine function of the input, due to the constant-valued bias terms.
Revised: Note that the regions are formally an affine function of the input, due to the constant-valued bias terms.

Original: For simplicity and convenience, these are more commonly described as linear regions.
Revised: For simplicity and convenience, these are more commonly described as linear regions.

Original: Figure 1 provides a schematic illustration of these regions, along with a policy trajectory.
Revised: Figure 1 provides a schematic illustration of these regions, along with a policy trajectory.

Original: The number of distinct regions into which the input space is divided is a natural measure of network expressivity.
Revised: The number of distinct regions into which the input space is divided is a natural measure of network expressivity.

Original: Learned functions with many linear regions have the capacity to build complex and flexible decision boundaries.
Revised: Learned functions with many linear regions have the capacity to build complex and flexible decision boundaries.

Original: Thus, the problem of counting the number of linear regions has been extensively studied in recent literature [Montúfar et al., 2014, Raghu et al., 2017, Hanin and Rolnick, 2019a, Serra et al., 2018].
Revised: Thus, the problem of counting the number of linear regions has been extensively studied in recent literature [Montúfar et al., 2014, Raghu et al., 2017, Hanin and Rolnick, 2019a, Serra et al., 2018].

Original: While the maximum number of regions is exponential with respect to network depth [Montúfar et al., 2014], recent work has demonstrated that the number of regions is instead typically proportional to the number neurons [Hanin and Rolnick, 2019a].
Revised: While the maximum number of regions is exponential with respect to network depth [Montúfar et al., 2014], recent work has demonstrated that the number of regions is instead typically proportional to the number neurons [Hanin and Rolnick, 2019a].

Original: For RL, we are interested in the local granularity (density) of linear regions along trajectories arising from the policy.
Revised: For RL, we are interested in the local granularity (density) of linear regions along trajectories arising from the policy.

Original: Fine-grained regions afford fine-grained control, and thus we may hypothesize that region density increases in regions frequently visited by the policy, in order to afford better control.
Revised: Fine-grained regions afford fine-grained control, and thus we may hypothesize that region density increases in regions frequently visited by the policy, in order to afford better control.

Original: Recent work in supervised-learning of image classification is inconsistent with regard to findings about the region density seen in the vicinity of data points, with some reporting a decrease [Novak et al., 2018] to provide better generalization and robustness to perturbation, and others not
Revised: Recent work in supervised-learning of image classification is inconsistent with regard to findings about the region density seen in the vicinity of data points, with some reporting a decrease [Novak et al., 2018] to provide better generalization and robustness to perturbation, and others not

Original: [Hanin and
Revised: [Hanin and

Original: Rolnick, 2019a].
Revised: Rolnick, 2019a].

Original: For the RL setting, we note that counting regions visited along an episode trajectory arguably provides a meaningful and task-grounded measurement in contrast to line-segments and ellipses that pass through randomly sampled points sampled from training data, which have been used in the prior works mentioned above.
Revised: For the RL setting, we note that counting regions visited along an episode trajectory arguably provides a meaningful and task-grounded measurement in contrast to line-segments and ellipses that pass through randomly sampled points sampled from training data, which have been used in the prior works mentioned above.

Original: We further note that piecewise-affine control strategies are commonly designed into control systems, e.g., via gain scheduling.
Revised: We further note that piecewise-affine control strategies are commonly designed into control systems, e.g., via gain scheduling.

Original: Understanding how these regions are designed and distributed by deep RL thus helps establish bridges with these existing methods.
Revised: Understanding how these regions are designed and distributed by deep RL thus helps establish bridges with these existing methods.

Original: To the best of our knowledge, our work is the first to investigate the structure and evolution of linear regions of ReLU-based deep RL policies in detail.
Revised: To the best of our knowledge, our work is the first to investigate the structure and evolution of linear regions of ReLU-based deep RL policies in detail.

Original: We seek to answer several basic empirical questions :
Revised: We seek to answer several basic empirical questions :

Original: Q1 Do findings for network expressivity, originally developed in supervised learning settings, apply to RL policies?
Revised: Q1 Do findings for network expressivity, originally developed in supervised learning settings, apply to RL policies?

Original: How are the region densities affected by the policy network configuration?
Revised: How are the region densities affected by the policy network configuration?

Original: Do deeper policy networks result in finer-grained regions and hence an increased expressivity?
Revised: Do deeper policy networks result in finer-grained regions and hence an increased expressivity?

Original: Q2 How do the linear regions of a policy evolve during training?
Revised: Q2 How do the linear regions of a policy evolve during training?

Original: Do we see a significantly greater density of regions emerge along the areas of the state space frequently visited by the episodic trajectories, thereby allowing for finer-grained control?
Revised: Do we see a significantly greater density of regions emerge along the areas of the state space frequently visited by the episodic trajectories, thereby allowing for finer-grained control?

Original: Do random-action trajectories see different densities?
Revised: Do random-action trajectories see different densities?

Original: The key results can be summarized as follows, for policies trained using proximal policy optimization (PPO)
Revised: The key results can be summarized as follows, for policies trained using proximal policy optimization (PPO)

Original: [Schulman et al., 2017], and evaluated on four different continuous control tasks.
Revised: [Schulman et al., 2017], and evaluated on four different continuous control tasks.

Original: Q1:
Revised: Q1:

Original: There is a general alignment with recent theoretical and empirical results for supervised learning settings.
Revised: There is a general alignment with recent theoretical and empirical results for supervised learning settings.

Original: Region density is principally proportional to the number of neurons, with an additional small observed increase in density for deeper networks.
Revised: Region density is principally proportional to the number of neurons, with an additional small observed increase in density for deeper networks.

Original: Q2:
Revised: Q2:

Original: Only a moderate increase of density is observed during training, as measured along fixed final-policy trajectories.
Revised: Only a moderate increase of density is observed during training, as measured along fixed final-policy trajectories.

Original: Therefore, the complexity of a final learned policy does not come principally from increased density on-and-around the optimal trajectories, which is a potentially surprising result.
Revised: Therefore, the complexity of a final learned policy does not come principally from increased density on-and-around the optimal trajectories, which is a potentially surprising result.

Original: In contrast, as measured along the evolving current-policy trajectories, a decrease in region density is observed during training.
Revised: In contrast, as measured along the evolving current-policy trajectories, a decrease in region density is observed during training.

Original: Across all settings, we also observe that the region-transition count, as observed during fixed time-duration episodes, grows during training before converging to a plateau.
Revised: Across all settings, we also observe that the region-transition count, as observed during fixed time-duration episodes, grows during training before converging to a plateau.

Original: However, the trajectory length, as measured in the input space, also grows towards a plateau, although not at the same rate, and this leads to variations in the mean region densities as observed along current trajectories during training.
Revised: However, the trajectory length, as measured in the input space, also grows towards a plateau, although not at the same rate, and this leads to variations in the mean region densities as observed along current trajectories during training.

Original:  Related Work
Revised:  Related Work

Original: Understanding the expressivity of a neural network is fundamental to better understanding its operation.
Revised: Understanding the expressivity of a neural network is fundamental to better understanding its operation.

Original: Several works study exprressivity of deep neural networks with piecewise linear activations by counting their linear regions [Arora et al., 2016, Bianchini and Scarselli, 2014].
Revised: Several works study exprressivity of deep neural networks with piecewise linear activations by counting their linear regions [Arora et al., 2016, Bianchini and Scarselli, 2014].

Original: On the theoretical side, Pascanu et al.
Revised: On the theoretical side, Pascanu et al.

Original: [2013] show that in the asymptotic limit of many hidden layers, deep ReLU networks are capable of separating their input space into exponentially more linear regions compared with shallow networks, despite using the same number of computational units.
Revised: [2013] show that in the asymptotic limit of many hidden layers, deep ReLU networks are capable of separating their input space into exponentially more linear regions compared with shallow networks, despite using the same number of computational units.

Original: Following this,
Revised: Following this,

Original: Montúfar et al.
Revised: Montúfar et al.

Original: [2014] also explore the complexity of functions computable by deep feed-forward neural networks with ReLU and maxout activations, and provide tighter upper and lower bounds for the maximal number of linear regions.
Revised: [2014] also explore the complexity of functions computable by deep feed-forward neural networks with ReLU and maxout activations, and provide tighter upper and lower bounds for the maximal number of linear regions.

Original: They show that the number of linear regions is polynomial in the network width and exponential with respect to the network depth.
Revised: They show that the number of linear regions is polynomial in the network width and exponential with respect to the network depth.

Original: Furthermore, Raghu et al.
Revised: Furthermore, Raghu et al.

Original: [2017] improve the upper bound for ReLU networks by introducing a new set of expressivity measures and showing that expressivity grows exponentially with network depth for ReLU and tanh activated neural networks.
Revised: [2017] improve the upper bound for ReLU networks by introducing a new set of expressivity measures and showing that expressivity grows exponentially with network depth for ReLU and tanh activated neural networks.

Original: Serra et al.
Revised: Serra et al.

Original: [2018] generalizes these results by providing even tighter upper and lower bounds for the number of regions for ReLU networks and show that the maximal number of regions grows exponentially with depth when input dimension is sufficiently large.
Revised: [2018] generalizes these results by providing even tighter upper and lower bounds for the number of regions for ReLU networks and show that the maximal number of regions grows exponentially with depth when input dimension is sufficiently large.

Original: More recent works that touch on expressivity of depth show that the effect of depth on the expressivity of neural networks is likely far below that of the theoretical maximum proposed by prior literature.
Revised: More recent works that touch on expressivity of depth show that the effect of depth on the expressivity of neural networks is likely far below that of the theoretical maximum proposed by prior literature.

Original: Hanin and Rolnick [2019b] study the importance of depth on expressivity of neural networks in practice.
Revised: Hanin and Rolnick [2019b] study the importance of depth on expressivity of neural networks in practice.

Original: They show that the average number of linear regions for ReLU networks at initialization is bounded by the number of neurons raised to the input dimension, and is independent of network depth.
Revised: They show that the average number of linear regions for ReLU networks at initialization is bounded by the number of neurons raised to the input dimension, and is independent of network depth.

Original: They also empirically show that this bound remains tight during training.
Revised: They also empirically show that this bound remains tight during training.

Original: Similarly, Hanin and
Revised: Similarly, Hanin and

Original: Rolnick [2019a] find that the average distance to the boundary of the linear regions depends only on the number of neurons and not on the network depth – both at initialization and during training for supervised-learning tasks on ReLU networks.
Revised: Rolnick [2019a] find that the average distance to the boundary of the linear regions depends only on the number of neurons and not on the network depth – both at initialization and during training for supervised-learning tasks on ReLU networks.

Original: This strongly suggests that deeper networks do not necessarily learn more complex functions in comparison to shallow networks.
Revised: This strongly suggests that deeper networks do not necessarily learn more complex functions in comparison to shallow networks.

Original: Prior to this, a number of works have shown that the strength of deep learning may arise in part from a good match between deep architectures and current training procedures [Mhaskar and Poggio, 2016, Mhaskar et al., 2016,
Revised: Prior to this, a number of works have shown that the strength of deep learning may arise in part from a good match between deep architectures and current training procedures [Mhaskar and Poggio, 2016, Mhaskar et al., 2016,

Original: Zhang et al., 2021].
Revised: Zhang et al., 2021].

Original: Notably, Ba and Caruana [2014] show that, once deep networks are trained to perform a task successfully, their behavior can often be replicated by shallow networks, suggesting that the advantages of depth may be linked to easier learning.
Revised: Notably, Ba and Caruana [2014] show that, once deep networks are trained to perform a task successfully, their behavior can often be replicated by shallow networks, suggesting that the advantages of depth may be linked to easier learning.

Original: Another line of work studies function complexity in terms of robustness to perturbations to the input.
Revised: Another line of work studies function complexity in terms of robustness to perturbations to the input.

Original: Sokoli´c et al.
Revised: Sokoli´c et al.

Original: [2017] theoretically studies the input-output Jacobian, which is a measure of robustness and also relates to generalization.
Revised: [2017] theoretically studies the input-output Jacobian, which is a measure of robustness and also relates to generalization.

Original: Similarly, Zahavy et al.
Revised: Similarly, Zahavy et al.

Original: [2016b] propose a sensitivity measure in terms of adversarial robustness, and provides theoretical and experimental insights on how it relates to generalization.
Revised: [2016b] propose a sensitivity measure in terms of adversarial robustness, and provides theoretical and experimental insights on how it relates to generalization.

Original: Novak et al.
Revised: Novak et al.

Original: [2018] also study robustness and sensitivity using the input-output Jacobian and number of transitions along trajectories in the input space as measures of robustness.
Revised: [2018] also study robustness and sensitivity using the input-output Jacobian and number of transitions along trajectories in the input space as measures of robustness.

Original: They show that neural networks trained for image classification tasks are more robust to input perturbations in the vicinity of the training data manifold, due to training points lying in regions of lower density.
Revised: They show that neural networks trained for image classification tasks are more robust to input perturbations in the vicinity of the training data manifold, due to training points lying in regions of lower density.

Original: Several other recent works have also focused on proposing tight generalization bounds for neural networks [Bartlett et al., 2017, Dziugaite and Roy, 2017, Neyshabur et al., 2017].
Revised: Several other recent works have also focused on proposing tight generalization bounds for neural networks [Bartlett et al., 2017, Dziugaite and Roy, 2017, Neyshabur et al., 2017].

Original: There are a number of works that touch on understanding deep neural networks by finding general principles and patterns during training.
Revised: There are a number of works that touch on understanding deep neural networks by finding general principles and patterns during training.

Original: Arpit et al.
Revised: Arpit et al.

Original: [2017] empirically show that deep networks prioritize learning simple patterns of the data during training.
Revised: [2017] empirically show that deep networks prioritize learning simple patterns of the data during training.

Original: Xu et al.
Revised: Xu et al.

Original: [2019] find a similar phenomenon in the case of 2 -layer networks with Sigmoid activations.
Revised: [2019] find a similar phenomenon in the case of 2 -layer networks with Sigmoid activations.

Original: Rahaman et al.
Revised: Rahaman et al.

Original: [2019] study deep ReLU activated networks through the lens of Fourier analysis and show that while deep neural networks can approximate arbitrary functions, they favour low frequency ones and thus, they exhibit a bias towards smooth functions.
Revised: [2019] study deep ReLU activated networks through the lens of Fourier analysis and show that while deep neural networks can approximate arbitrary functions, they favour low frequency ones and thus, they exhibit a bias towards smooth functions.

Original: Samek et al.
Revised: Samek et al.

Original: [2017] present two approaches in explaining predictions of deep learning models in a classification task, with the first method computing the sensitivity of the prediction with respect to input perturbations and the second method that meaningfully decomposes the decision in terms of input variables.
Revised: [2017] present two approaches in explaining predictions of deep learning models in a classification task, with the first method computing the sensitivity of the prediction with respect to input perturbations and the second method that meaningfully decomposes the decision in terms of input variables.

Original: While deep RL methods are widely used and extensively studied, few works focus on understanding the policy structure in detail.
Revised: While deep RL methods are widely used and extensively studied, few works focus on understanding the policy structure in detail.

Original: Zahavy et al.
Revised: Zahavy et al.

Original: [2016a] propose a visualization method to interpret the agent’s actions by describing the Markov Decision Process as a directed graph on a t-SNE map.
Revised: [2016a] propose a visualization method to interpret the agent’s actions by describing the Markov Decision Process as a directed graph on a t-SNE map.

Original: They then suggest ways to interpret, debug and optimize deep neural network policies using the proposed visualization maps.
Revised: They then suggest ways to interpret, debug and optimize deep neural network policies using the proposed visualization maps.

Original: Rupprecht et al.
Revised: Rupprecht et al.

Original: [2019] train a generative model over the state space of Atari games to visualize states which minimize or maximize given action probabilities.
Revised: [2019] train a generative model over the state space of Atari games to visualize states which minimize or maximize given action probabilities.

Original: Luo et al.
Revised: Luo et al.

Original: [2018] adapt three visualization techniques to the domain of image-based RL using convolutional neural networks in order to understand the decision making process of the RL agent.
Revised: [2018] adapt three visualization techniques to the domain of image-based RL using convolutional neural networks in order to understand the decision making process of the RL agent.

Original:  Piecewise Linear Regions
Revised:  Piecewise Linear Regions

Original: Throughout this work, we consider RL policies based on ReLU networks .
Revised: Throughout this work, we consider RL policies based on ReLU networks .

Original: A ReLU network is a ReLU-activated feed-forward neural network, or a multi-layer perceptron (MLP) which can be formulated as a scalar function f : R d → R o defined by a neural network with L hidden layers of width d 1 , ..., d L and a o -dimensional output.
Revised: A ReLU network is a ReLU-activated feed-forward neural network, or a multi-layer perceptron (MLP) which can be formulated as a scalar function f : R d → R o defined by a neural network with L hidden layers of width d 1 , ..., d L and a o -dimensional output.

Original: Assuming the output to be 1-dimensional, and following the notation of Rahaman et al.
Revised: Assuming the output to be 1-dimensional, and following the notation of Rahaman et al.

Original: [2019], we have:
Revised: [2019], we have:

Original: [Equation]
Revised: [Equation]

Original: where T ( k ) : R d k − 1 → R d k computes the weighted sum T ( k ) ( x )
Revised: where T ( k ) : R d k − 1 → R d k computes the weighted sum T ( k ) ( x )

Original: =
Revised: =

Original: W ( k ) x
Revised: W ( k ) x

Original: + b ( k ) for some weight matrix W ( k ) and bias vector b ( k ) .
Revised: + b ( k ) for some weight matrix W ( k ) and bias vector b ( k ) .

Original: Here, σ ( u ) = max(0 , u ) denotes the ReLU activation function acting element-wise on vector u
Revised: Here, σ ( u ) = max(0 , u ) denotes the ReLU activation function acting element-wise on vector u

Original: = ( u 1 , ..., u n ) .
Revised: = ( u 1 , ..., u n ) .

Original: Given the ReLU network f from Equation 1, again following Rahaman et al.
Revised: Given the ReLU network f from Equation 1, again following Rahaman et al.

Original: [2019], piecewise linearity can be explicit written by
Revised: [2019], piecewise linearity can be explicit written by

Original: [Equation]
Revised: [Equation]

Original: where r is the index for the linear region P r and 1 P r is the indicator function on P r .
Revised: where r is the index for the linear region P r and 1 P r is the indicator function on P r .

Original: The 1 × d matrix
Revised: The 1 × d matrix

Original: W r is given by:
Revised: W r is given by:

Original: [Equation]
Revised: [Equation]

Original: where W ( k ) r is obtained from the original weight W ( k ) by setting its j th column to zero whenever neuron j of layer k − 1 is inactive for k > 1 .
Revised: where W ( k ) r is obtained from the original weight W ( k ) by setting its j th column to zero whenever neuron j of layer k − 1 is inactive for k > 1 .

Original:  Counting Linear Regions in RL Policies
Revised:  Counting Linear Regions in RL Policies

Original: To answer the key questions posed in the introduction, we need a method for counting the linear regions encountered during the episodic trajectories taken during RL.
Revised: To answer the key questions posed in the introduction, we need a method for counting the linear regions encountered during the episodic trajectories taken during RL.

Original: For each input x , we encode each neuron of the policy network with a binary code of 0 if its pre-activation is negative, and with a binary code 1 otherwise.
Revised: For each input x , we encode each neuron of the policy network with a binary code of 0 if its pre-activation is negative, and with a binary code 1 otherwise.

Original: The linear region of the input x can thus be uniquely identified by the concatenation of binary codes of all the neurons in the network called an activation pattern .
Revised: The linear region of the input x can thus be uniquely identified by the concatenation of binary codes of all the neurons in the network called an activation pattern .

Original: Figureillustrates the activation pattern construction for a 2D input space.
Revised: Figureillustrates the activation pattern construction for a 2D input space.

Original: Figure 1 provides a schematic illustration of the linear regions for a 2D state space, together with an example trajectory that consists of sequential states encountered by the policy, connected with straight-line segments.
Revised: Figure 1 provides a schematic illustration of the linear regions for a 2D state space, together with an example trajectory that consists of sequential states encountered by the policy, connected with straight-line segments.

Original: As seen in the figure, regions can be revisited, which leads to a distinction between the number of transitions between regions and the number of unique regions visited along a trajectory.
Revised: As seen in the figure, regions can be revisited, which leads to a distinction between the number of transitions between regions and the number of unique regions visited along a trajectory.

Original: We compute both of these metrics along episodic trajectories, as enabled by keeping a list of the regions already visited at any point in time when processing a given episode trajectory.
Revised: We compute both of these metrics along episodic trajectories, as enabled by keeping a list of the regions already visited at any point in time when processing a given episode trajectory.

Original: Our region-counting method also includes regions that are crossed by the straight-line segments between successive trajectory states, although the policy itself does not explicitly encounter these regions due to the discrete-time nature of typical RL environments.
Revised: Our region-counting method also includes regions that are crossed by the straight-line segments between successive trajectory states, although the policy itself does not explicitly encounter these regions due to the discrete-time nature of typical RL environments.

Original: To represent the k -th line segment of the episodic trajectories in the input space, we develop a parameterized line segment given by s ( u )
Revised: To represent the k -th line segment of the episodic trajectories in the input space, we develop a parameterized line segment given by s ( u )

Original: =
Revised: =

Original: (1 − u )
Revised: (1 − u )

Original: s k − 1 + us k , where u ∈ [0 , 1] and s k − 1 , s
Revised: s k − 1 + us k , where u ∈ [0 , 1] and s k − 1 , s

Original: k ∈ R d denote the endpoints of the line segment.
Revised: k ∈ R d denote the endpoints of the line segment.

Original: We then calculate the exact number of the linear regions over this line segment by considering the hidden layers of the policy network one at a time, starting from input towards the output, and observe how each region can be split by the neurons.
Revised: We then calculate the exact number of the linear regions over this line segment by considering the hidden layers of the policy network one at a time, starting from input towards the output, and observe how each region can be split by the neurons.

Original: Starting with the first layer, we consider neurons one by one, and identify the point in the domain of u , if any, that induces a change in the binary labeling for that neuron.
Revised: Starting with the first layer, we consider neurons one by one, and identify the point in the domain of u , if any, that induces a change in the binary labeling for that neuron.

Original: Each such point subdivides the domain of u into two new regions where the pre-activation will be zero for one of them in the next layer.
Revised: Each such point subdivides the domain of u into two new regions where the pre-activation will be zero for one of them in the next layer.

Original: By maintaining a list of these regions and the linear functions defined over them, and whether their pre-activation vanishes in the next layer, we proceed to the neurons of the next layer.
Revised: By maintaining a list of these regions and the linear functions defined over them, and whether their pre-activation vanishes in the next layer, we proceed to the neurons of the next layer.

Original: This process repeats for each of the regions resulting from the previous layer.
Revised: This process repeats for each of the regions resulting from the previous layer.

Original: We record the activation patterns of all the final regions, to track state visits during all segments of episode trajectories.
Revised: We record the activation patterns of all the final regions, to track state visits during all segments of episode trajectories.

Original: In the end, for each trajectory τ , we compute the total region transitions, R T ( τ ) , as well as the number of unique visited regions, R U ( τ ) , where R T ( τ ) + 1 ≥ R U ( τ ) .
Revised: In the end, for each trajectory τ , we compute the total region transitions, R T ( τ ) , as well as the number of unique visited regions, R U ( τ ) , where R T ( τ ) + 1 ≥ R U ( τ ) .

Original: We further compute the trajectory length in the input space, L ( τ ) .
Revised: We further compute the trajectory length in the input space, L ( τ ) .

Original: This allows us to compute normalized region densities according to ρ ( τ ) =
Revised: This allows us to compute normalized region densities according to ρ ( τ ) =

Original: R T ( τ ) /
Revised: R T ( τ ) /

Original: ( NL ( τ )) where N is the total number of neurons in the policy network, in accordance with Hanin and Rolnick [2019a].
Revised: ( NL ( τ )) where N is the total number of neurons in the policy network, in accordance with Hanin and Rolnick [2019a].

Original: We use the above metrics along two types of trajectories.
Revised: We use the above metrics along two types of trajectories.

Original: First, we consider fixed trajectories, τ ∗ as sampled from the final fully-trained policy.
Revised: First, we consider fixed trajectories, τ ∗ as sampled from the final fully-trained policy.

Original: Second, we consider current trajectories, τ , as sampled from the current policy during training.
Revised: Second, we consider current trajectories, τ , as sampled from the current policy during training.

Original: Both of these trajectories offer informative views of the evolution of the policy.
Revised: Both of these trajectories offer informative views of the evolution of the policy.

Original: The former offers a direct picture of the linear-region density along a meaningful, fixed region of the state space, i.e., that of the final optimized policy.
Revised: The former offers a direct picture of the linear-region density along a meaningful, fixed region of the state space, i.e., that of the final optimized policy.

Original: The latter offers a view of what the policy trajectories actually encounter during training.
Revised: The latter offers a view of what the policy trajectories actually encounter during training.

Original: In order to better understand the evolution of τ , we also track its length, L ( τ ) , given that the density is determined by the region-transition count as well as the length of the trajectory.
Revised: In order to better understand the evolution of τ , we also track its length, L ( τ ) , given that the density is determined by the region-transition count as well as the length of the trajectory.

Original: Figure 3 shows the evolution of linear regions and the types of trajectories τ ∗ and τ for a simple 2D toy environment, and a
Revised: Figure 3 shows the evolution of linear regions and the types of trajectories τ ∗ and τ for a simple 2D toy environment, and a

Original: ReLU-activated policy network of depth 2 and width 8.
Revised: ReLU-activated policy network of depth 2 and width 8.

Original:  Experimental Results
Revised:  Experimental Results

Original: We conduct our experiments on four continuous control tasks including HalfCheetah-v2, Walker-v2,
Revised: We conduct our experiments on four continuous control tasks including HalfCheetah-v2, Walker-v2,

Original: Ant-v2, and Swimmer-v2 environments from the OpenAI gym benchmark suits [Brockman et al., 2016].
Revised: Ant-v2, and Swimmer-v2 environments from the OpenAI gym benchmark suits [Brockman et al., 2016].

Original: We use the Stable-Baselines3 implementations of the PPO algorithm
Revised: We use the Stable-Baselines3 implementations of the PPO algorithm

Original: [Schulman et al., 2017] in all of our experiments throughout this work 1 .
Revised: [Schulman et al., 2017] in all of our experiments throughout this work 1 .

Original: We run each experiment with 5 different random seeds and the results show the mean and the standard deviation across the random seeds.
Revised: We run each experiment with 5 different random seeds and the results show the mean and the standard deviation across the random seeds.

Original: The policy network architecture is an MLP with ReLU activations in all hidden layers in our experiments.
Revised: The policy network architecture is an MLP with ReLU activations in all hidden layers in our experiments.

Original: Epoch =
Revised: Epoch =

Original: Epoch = Epoch =
Revised: Epoch = Epoch =

Original: Epoch =τ ∗ − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons τ − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons
Revised: Epoch =τ ∗ − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons τ − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons − 50 − 25 0 25 50 75 100 125 150 − 20 −− 10 − 55 1020 StateSpaceandATrajectoryVisualizedwith96Neurons

Original: We train 18 policy network configurations with N ∈ { 32 , 48 , 64 , 96 , 128 , 192 } neurons, widths w ∈ { 8 , 16 , 32 , 64 } , and depths d ∈ { 1 , 2 , 3 , 4 } .
Revised: We train 18 policy network configurations with N ∈ { 32 , 48 , 64 , 96 , 128 , 192 } neurons, widths w ∈ { 8 , 16 , 32 , 64 } , and depths d ∈ { 1 , 2 , 3 , 4 } .

Original: We use a fixed value function network structure of (64 , 64) in all of our experiments.
Revised: We use a fixed value function network structure of (64 , 64) in all of our experiments.

Original: Details of the network configurations used for the policy network are available in Appendix C. These network configurations are chosen such that the network is fully-capable of learning the task and achieves near state-of-the-art cumulative reward on the particular task it is trained on.
Revised: Details of the network configurations used for the policy network are available in Appendix C. These network configurations are chosen such that the network is fully-capable of learning the task and achieves near state-of-the-art cumulative reward on the particular task it is trained on.

Original: This allows us to study the evolution of linear regions disentangled from properties such as trainability, which is outside the scope of this work.
Revised: This allows us to study the evolution of linear regions disentangled from properties such as trainability, which is outside the scope of this work.

Original: The table showing the range of the mean cumulative reward per task, across all network configurations can be found in
Revised: The table showing the range of the mean cumulative reward per task, across all network configurations can be found in

Original: Appendix B.
Revised: Appendix B.

Original: We adopt the network initialization and hyperparameters of PPO from StableBaselines3 [Raffin et al., 2021] and train our policy networks on 2M samples (i.e. 2M timesteps in the environment).
Revised: We adopt the network initialization and hyperparameters of PPO from StableBaselines3 [Raffin et al., 2021] and train our policy networks on 2M samples (i.e. 2M timesteps in the environment).

Original: This initialization scheme also matches the best-practice recommendations for on-policy RL, e.g., [Andrychowicz et al., 2020], although it differs from that used in Hanin and Rolnick [2019a] for theoretical and empirical linear region analysis.
Revised: This initialization scheme also matches the best-practice recommendations for on-policy RL, e.g., [Andrychowicz et al., 2020], although it differs from that used in Hanin and Rolnick [2019a] for theoretical and empirical linear region analysis.

Original: A detailed explanation of the initialization and choice of hyperparameters is available in Appendix E.
Revised: A detailed explanation of the initialization and choice of hyperparameters is available in Appendix E.

Original: In what follows, we focus on the results for the HalfCheetah-v2 environment as a representative example.
Revised: In what follows, we focus on the results for the HalfCheetah-v2 environment as a representative example.

Original: Similar results for the other environments are provided in Appendix K.
Revised: Similar results for the other environments are provided in Appendix K.

Original: 5.1 Evolution of the Density of Linear Regions during Training
Revised: 5.1 Evolution of the Density of Linear Regions during Training

Original: We begin by examining the number of transitions encountered as we sweep along the fixed final-policy trajectory, R T ( τ ∗ ) , as shown in Figure 4(a).
Revised: We begin by examining the number of transitions encountered as we sweep along the fixed final-policy trajectory, R T ( τ ∗ ) , as shown in Figure 4(a).

Original: For clarity, we present results for only three policy network configurations.
Revised: For clarity, we present results for only three policy network configurations.

Original: This plot shows a moderate and gradual increase of 50% in the number of region transitions observed in fixed-duration RL episodes during training, with larger policy networks having more transitions.
Revised: This plot shows a moderate and gradual increase of 50% in the number of region transitions observed in fixed-duration RL episodes during training, with larger policy networks having more transitions.

Original: Computing the region density requires dividing by number of neurons, N , and the trajectory length, L ( τ ∗ ) , which is constant for the fixed final trajectory, τ ∗ .
Revised: Computing the region density requires dividing by number of neurons, N , and the trajectory length, L ( τ ∗ ) , which is constant for the fixed final trajectory, τ ∗ .

Original: The result is given in Figure 4(b) which shows the mean and standard deviation of the normalized region densities across policy networks with the same depth.
Revised: The result is given in Figure 4(b) which shows the mean and standard deviation of the normalized region densities across policy networks with the same depth.

Original: This shows a result that is consistent across all policy network dimensions to within ± 30 %, and reflects an identical moderate-and-gradual increase during training, regardless of the policy network configuration.
Revised: This shows a result that is consistent across all policy network dimensions to within ± 30 %, and reflects an identical moderate-and-gradual increase during training, regardless of the policy network configuration.

Original: Thus, in response to Q2 , RL policies trained with
Revised: Thus, in response to Q2 , RL policies trained with

Original: PPO see a moderate increase in region density along the parts of the state space that are frequently visited by the policy.
Revised: PPO see a moderate increase in region density along the parts of the state space that are frequently visited by the policy.

Original: We later discuss the observed impact of policy-network depth (§5.2).
Revised: We later discuss the observed impact of policy-network depth (§5.2).

Original: We next examine the number of transitions encountered as we sweep along the evolving set of current-policy trajectories, R T ( τ ) , shown in Figure 4(c).
Revised: We next examine the number of transitions encountered as we sweep along the evolving set of current-policy trajectories, R T ( τ ) , shown in Figure 4(c).

Original: These begin near zero and then rapidly increase as the policy succeeds at exploration.
Revised: These begin near zero and then rapidly increase as the policy succeeds at exploration.

Original: Again, for clarity, we show a representative sample of
Revised: Again, for clarity, we show a representative sample of

Original: [Figure]
Revised: [Figure]

Original: N u m b e r o f t r a n s i t i o n s
Revised: N u m b e r o f t r a n s i t i o n s

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: N
Revised: N

Original: [Figure]
Revised: [Figure]

Original: m
Revised: m

Original: [Figure]
Revised: [Figure]

Original: e r o f t r a n s i t i o n s
Revised: e r o f t r a n s i t i o n s

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: T r a j ec t o r y l e n g t h
Revised: T r a j ec t o r y l e n g t h

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: three policy network dimensions here.
Revised: three policy network dimensions here.

Original: The corresponding region densities are shown in Figure 4(d), together with the fixed-region densities which are repeated from Figure 4(b) (see the bottom dashed curve).
Revised: The corresponding region densities are shown in Figure 4(d), together with the fixed-region densities which are repeated from Figure 4(b) (see the bottom dashed curve).

Original: We speculate that the observed densities are higher early on due to the unstructured nature of early exploration and the form of network initialization, and then decrease over time as the trajectory lengths begin to increase, as shown in Figure 4(e).
Revised: We speculate that the observed densities are higher early on due to the unstructured nature of early exploration and the form of network initialization, and then decrease over time as the trajectory lengths begin to increase, as shown in Figure 4(e).

Original: We compute the number of repeat visits to regions as R T − R U , i.e., the region transition count minus the unique region count.
Revised: We compute the number of repeat visits to regions as R T − R U , i.e., the region transition count minus the unique region count.

Original: In Figure 4(f), we track this for the current trajectory during training.
Revised: In Figure 4(f), we track this for the current trajectory during training.

Original: We speculate that repeat visits are high early in training because of limited exploration.
Revised: We speculate that repeat visits are high early in training because of limited exploration.

Original: Repeat visits remain later in training because of the cyclic nature of the learned locomotion for HalfCheetah.
Revised: Repeat visits remain later in training because of the cyclic nature of the learned locomotion for HalfCheetah.

Original: 5.2 Impact of Policy Network Structure on Expressivity
Revised: 5.2 Impact of Policy Network Structure on Expressivity

Original: Our normalized density computation assumes that the number of transitions scales linearly with the number of neurons, N .
Revised: Our normalized density computation assumes that the number of transitions scales linearly with the number of neurons, N .

Original: However, we empirically observe that there remains a moderate dependence on the network depth, as seen in Figure 4(b), with deeper networks resulting in moderately denser regions in the learned policies.
Revised: However, we empirically observe that there remains a moderate dependence on the network depth, as seen in Figure 4(b), with deeper networks resulting in moderately denser regions in the learned policies.

Original: We further examine the policy network expressivity using random infinite lines.
Revised: We further examine the policy network expressivity using random infinite lines.

Original: Following Hanin and Rolnick [2019a], for each policy network, we sample 100 random lines by randomly sampling 100 points from the final trajectory τ ∗ , and connecting each randomly sampled point to the origin of the policy input space.
Revised: Following Hanin and Rolnick [2019a], for each policy network, we sample 100 random lines by randomly sampling 100 points from the final trajectory τ ∗ , and connecting each randomly sampled point to the origin of the policy input space.

Original: Figure 4(g) shows the mean and standard deviation of the normalized number of transitions, R T /N , as computed along these infinite lines in random directions passing through the origin, over the 5 random seeds of each network configuration.
Revised: Figure 4(g) shows the mean and standard deviation of the normalized number of transitions, R T /N , as computed along these infinite lines in random directions passing through the origin, over the 5 random seeds of each network configuration.

Original: These produce approximately uniform results throughout training, showing no overall changes in mean density in various directions.
Revised: These produce approximately uniform results throughout training, showing no overall changes in mean density in various directions.

Original: These results are also consistent with the findings of Hanin and
Revised: These results are also consistent with the findings of Hanin and

Original: Rolnick [2019a] as they show normalized region densities are close to 1 at initialization and remain roughly constant during training.
Revised: Rolnick [2019a] as they show normalized region densities are close to 1 at initialization and remain roughly constant during training.

Original: We repeated the same experiment with 100 random lines through the mean point instead of the origin, by connecting each randomly sampled point to the mean of the 100 sampled points.
Revised: We repeated the same experiment with 100 random lines through the mean point instead of the origin, by connecting each randomly sampled point to the mean of the 100 sampled points.

Original: The difference between the density over lines through the origin and through the mean point is negligible.
Revised: The difference between the density over lines through the origin and through the mean point is negligible.

Original: To further examine the behavior of the policy network on areas that it does not focus on exploring during training, we compute the region density along random trajectories created by taking random actions in the environment.
Revised: To further examine the behavior of the policy network on areas that it does not focus on exploring during training, we compute the region density along random trajectories created by taking random actions in the environment.

Original: For each policy network, we sample a set of 10 such random trajectories τ R and compute the normalized region density over these.
Revised: For each policy network, we sample a set of 10 such random trajectories τ R and compute the normalized region density over these.

Original: Figure 4(h) shows the mean and standard deviation of this metric across this set of 10 random trajectories, over the 5 random seeds of each network configuration.
Revised: Figure 4(h) shows the mean and standard deviation of this metric across this set of 10 random trajectories, over the 5 random seeds of each network configuration.

Original: This shows a very moderate decrease during training, with fairly consistent densities for all policy network configurations.
Revised: This shows a very moderate decrease during training, with fairly consistent densities for all policy network configurations.

Original: We speculate that the slight decrease in density over time arises from the trained policy no longer visiting the same regions as the random-action policy.
Revised: We speculate that the slight decrease in density over time arises from the trained policy no longer visiting the same regions as the random-action policy.

Original: Comparing the results of random trajectories with fixed and current (Figure 4(b) and Figure 4(d)), we can see that the normalized region density over random-action trajectories and a fixed final trajectory are within the same range.
Revised: Comparing the results of random trajectories with fixed and current (Figure 4(b) and Figure 4(d)), we can see that the normalized region density over random-action trajectories and a fixed final trajectory are within the same range.

Original: Also, although this metric is higher over current trajectories early on during training, it eventually decreases and converges to the same range of region densities as the fixed trajectory.
Revised: Also, although this metric is higher over current trajectories early on during training, it eventually decreases and converges to the same range of region densities as the fixed trajectory.

Original: In Appendix G we further directly visualize the regions of a policy network for
Revised: In Appendix G we further directly visualize the regions of a policy network for

Original: HalfCheetah, as viewed via an embedding plane that either passes through three points sampled from the final fixed trajectory, or three points sampled from a random-action trajectory.
Revised: HalfCheetah, as viewed via an embedding plane that either passes through three points sampled from the final fixed trajectory, or three points sampled from a random-action trajectory.

Original: We do not observe significant differences between these two visualizations.
Revised: We do not observe significant differences between these two visualizations.

Original: 5.3 Additional Experiments
Revised: 5.3 Additional Experiments

Original: To further study the generalization of our observations, we perform four additional experiments.
Revised: To further study the generalization of our observations, we perform four additional experiments.

Original: 5.3.1 RL Algorithm
Revised: 5.3.1 RL Algorithm

Original: To test how our findings generalize to RL algorithms other than PPO, we repeat our experiments by training deep RL policies with stochastic actor-critic (SAC) [Haarnoja et al., 2018] algorithm.
Revised: To test how our findings generalize to RL algorithms other than PPO, we repeat our experiments by training deep RL policies with stochastic actor-critic (SAC) [Haarnoja et al., 2018] algorithm.

Original: The experimental setup for SAC and its results are documented in detail in Appendix F. Region densities observed early on during training are quite different for SAC than they are for PPO, exhibiting high observed densities which then rapidly drop, before rising again like the PPO case.
Revised: The experimental setup for SAC and its results are documented in detail in Appendix F. Region densities observed early on during training are quite different for SAC than they are for PPO, exhibiting high observed densities which then rapidly drop, before rising again like the PPO case.

Original: We hypothesize this is due to a combination of (i) the entropy bonus for SAC, which is then annealed away, and (ii) the different network initialization used for the baseline SAC and PPO implementations.
Revised: We hypothesize this is due to a combination of (i) the entropy bonus for SAC, which is then annealed away, and (ii) the different network initialization used for the baseline SAC and PPO implementations.

Original: Despite this initial difference, the evolution pattern of densities are consistent with the PPO results later on during the course of training.
Revised: Despite this initial difference, the evolution pattern of densities are consistent with the PPO results later on during the course of training.

Original: Another observation is that similar to PPO results, densities over fixed, current and random-action trajectories are within the same range.
Revised: Another observation is that similar to PPO results, densities over fixed, current and random-action trajectories are within the same range.

Original: This supports our surprising finding that the complexity of RL policies is not principally captured by increased density on-and-around the optimal trajectories.
Revised: This supports our surprising finding that the complexity of RL policies is not principally captured by increased density on-and-around the optimal trajectories.

Original: 5.3.2 Environment
Revised: 5.3.2 Environment

Original: We examine whether a non-cyclic task, such as LunarLander, exhibits different linear-region evolution behavior compared to our four default tasks, which are naturally biased towards cyclic locomotion.
Revised: We examine whether a non-cyclic task, such as LunarLander, exhibits different linear-region evolution behavior compared to our four default tasks, which are naturally biased towards cyclic locomotion.

Original: From the results which are available in Appendix H, we can see that the transition counts and densities along the fixed trajectory are similar in structure to those of the cyclic locomotion tasks, showing that a gradual increase in density appears to be a general property for the PPO setting, even for non-cyclic tasks.
Revised: From the results which are available in Appendix H, we can see that the transition counts and densities along the fixed trajectory are similar in structure to those of the cyclic locomotion tasks, showing that a gradual increase in density appears to be a general property for the PPO setting, even for non-cyclic tasks.

Original: 5.3.3 Value Network Analysis
Revised: 5.3.3 Value Network Analysis

Original: To study the difference between the value space and policy space, we repeat our experiments on the value networks trained on HalfCheetah with PPO.
Revised: To study the difference between the value space and policy space, we repeat our experiments on the value networks trained on HalfCheetah with PPO.

Original: Full set of results for this experiment are available in
Revised: Full set of results for this experiment are available in

Original: Appendix I.
Revised: Appendix I.

Original: Our results show that linear regions in the value functions largely evolve in a very similar way to those in the policy.
Revised: Our results show that linear regions in the value functions largely evolve in a very similar way to those in the policy.

Original: 5.3.4 Behavior Cloning
Revised: 5.3.4 Behavior Cloning

Original: Deep RL provides us with a grounded setting to explore the impact of non-IID data on decision regions.
Revised: Deep RL provides us with a grounded setting to explore the impact of non-IID data on decision regions.

Original: To test if non-IID setting makes a difference, we repeat our experiments by training networks with behavior cloning (BC) using expert data from previously trained policies with PPO on HalfCheetah.
Revised: To test if non-IID setting makes a difference, we repeat our experiments by training networks with behavior cloning (BC) using expert data from previously trained policies with PPO on HalfCheetah.

Original: The experimental setup for BC and the full results are documented in detail in Appendix J. These results show that the evolution of region densities is different for BC than they are for PPO as the general increase in densities is not observed.
Revised: The experimental setup for BC and the full results are documented in detail in Appendix J. These results show that the evolution of region densities is different for BC than they are for PPO as the general increase in densities is not observed.

Original: In addition, number of transitions, the density values and length of current trajectories are significantly smaller for BC-trained policies.
Revised: In addition, number of transitions, the density values and length of current trajectories are significantly smaller for BC-trained policies.

Original: We have two hypotheses for these differences: (i) Policy’s learning history plays a significant role in the evolution of regions, as early trajectories inform the template cell divisions that later evolve with further training.
Revised: We have two hypotheses for these differences: (i) Policy’s learning history plays a significant role in the evolution of regions, as early trajectories inform the template cell divisions that later evolve with further training.

Original: Since BC policies observe the entire state space from the moment training starts, they may be able to better divide their state space and avoid adding too much granularity to certain areas.
Revised: Since BC policies observe the entire state space from the moment training starts, they may be able to better divide their state space and avoid adding too much granularity to certain areas.

Original: (ii) Network initialization largely affects the resulting linear regions and their evolution during training.
Revised: (ii) Network initialization largely affects the resulting linear regions and their evolution during training.

Original:  Conclusions
Revised:  Conclusions

Original: The structure of ReLU-based reinforcement learning policies remains poorly understood.
Revised: The structure of ReLU-based reinforcement learning policies remains poorly understood.

Original: In this work, we explored a key property of such policies, the density of linear regions observed along policy trajectories, as a proxy for the complexity of functions they can learn.
Revised: In this work, we explored a key property of such policies, the density of linear regions observed along policy trajectories, as a proxy for the complexity of functions they can learn.

Original: We performed a set of experiments to see how this evolves during training and whether and how changes in depth, and width of the policy network affect it.
Revised: We performed a set of experiments to see how this evolves during training and whether and how changes in depth, and width of the policy network affect it.

Original: The hypothesis that the final policy trajectory sees an increase in region density is supported to a limited degree, as we observe a moderate increase in density during training, i.e., 50%.
Revised: The hypothesis that the final policy trajectory sees an increase in region density is supported to a limited degree, as we observe a moderate increase in density during training, i.e., 50%.

Original: This result is consistent across tasks and across policy networks of different configurations.
Revised: This result is consistent across tasks and across policy networks of different configurations.

Original: When observed from the vantage point of the evolving current trajectory, the region density is high to begin with, due to the limited length of early trajectories, and then decreases asymptotically towards the observed final density.
Revised: When observed from the vantage point of the evolving current trajectory, the region density is high to begin with, due to the limited length of early trajectories, and then decreases asymptotically towards the observed final density.

Original: For the tasks we consider, trajectory lengths are observed to grow and asymptotically approach a plateau during training.
Revised: For the tasks we consider, trajectory lengths are observed to grow and asymptotically approach a plateau during training.

Original: Overall, the surprising implication, for our experimental setting, is that the complexity of RL policies appears to be captured by fine-tuning during learning, rather than a significant growth in the complexity of the functions observed along trajectories of the final policy.
Revised: Overall, the surprising implication, for our experimental setting, is that the complexity of RL policies appears to be captured by fine-tuning during learning, rather than a significant growth in the complexity of the functions observed along trajectories of the final policy.

Original: The key properties of linear regions as demonstrated for supervised learning of image classifiers, e.g., [Hanin and Rolnick, 2019a], are shown to generally also hold true for reinforcement learning policies.
Revised: The key properties of linear regions as demonstrated for supervised learning of image classifiers, e.g., [Hanin and Rolnick, 2019a], are shown to generally also hold true for reinforcement learning policies.

Original: In particular, their expressivity is well modeled as being proportional to the number of neurons, and largely independent of their width and depth, for ranges of these parameters where good policies can still be learned.
Revised: In particular, their expressivity is well modeled as being proportional to the number of neurons, and largely independent of their width and depth, for ranges of these parameters where good policies can still be learned.

Original: The number of regions observed for randomly oriented line segments through the origin of the input space is also in line with earlier theoretical and empirical results, and does not change significantly during training.
Revised: The number of regions observed for randomly oriented line segments through the origin of the input space is also in line with earlier theoretical and empirical results, and does not change significantly during training.

Original: On a different note, properties of linear regions as measured along policy trajectories observed for policies trained with RL, do not hold for policies trained with
Revised: On a different note, properties of linear regions as measured along policy trajectories observed for policies trained with RL, do not hold for policies trained with

Original: BC.
Revised: BC.

Original: This emphasizes the unique behavior of RL compared to its supervised learning counterpart.
Revised: This emphasizes the unique behavior of RL compared to its supervised learning counterpart.

Original: Our work comes with a number of limitations.
Revised: Our work comes with a number of limitations.

Original: Our experiments are mainly focused on locomotionbased tasks which produce cyclic trajectories.
Revised: Our experiments are mainly focused on locomotionbased tasks which produce cyclic trajectories.

Original: This could mean that the results may not generalize to other types of tasks.
Revised: This could mean that the results may not generalize to other types of tasks.

Original: For example, some tasks may not result in transition counts that grow, as generally observed in our experiments.
Revised: For example, some tasks may not result in transition counts that grow, as generally observed in our experiments.

Original: In addition, our work studies deep RL policies trained with
Revised: In addition, our work studies deep RL policies trained with

Original: PPO and SAC.
Revised: PPO and SAC.

Original: Thus, a broader analysis on more RL algorithms would shed light on the generality of the observations in this work.
Revised: Thus, a broader analysis on more RL algorithms would shed light on the generality of the observations in this work.

Original: Since network initialization largely affects the linear regions, it would be ideal to use the same initialization when comparing different RL algorithms.
Revised: Since network initialization largely affects the linear regions, it would be ideal to use the same initialization when comparing different RL algorithms.

Original: However,
Revised: However,

Original: RL algorithms are typically sensitive to network initialization, and changes in common practices in their initialization results in instabilities during training.
Revised: RL algorithms are typically sensitive to network initialization, and changes in common practices in their initialization results in instabilities during training.

Original: Currently, we compute only the mean region density for a trajectory; it may be useful to further understand the distribution of observed densities across trajectories.
Revised: Currently, we compute only the mean region density for a trajectory; it may be useful to further understand the distribution of observed densities across trajectories.

Original: We note that it is common practice in RL to apply input (observation) normalization based on a rolling average of observations seen so far [Andrychowicz et al., 2020], and thus our results also employ this strategy.
Revised: We note that it is common practice in RL to apply input (observation) normalization based on a rolling average of observations seen so far [Andrychowicz et al., 2020], and thus our results also employ this strategy.

Original: Applying RL without this normalization may impact both the performance of the RL algorithm and the resulting evolution of transition counts and region-density.
Revised: Applying RL without this normalization may impact both the performance of the RL algorithm and the resulting evolution of transition counts and region-density.

Original: Our work suggests opportunities for improving our understanding of deep reinforcement learning through studying how states are mapped to actions and how these mappings evolve during training.
Revised: Our work suggests opportunities for improving our understanding of deep reinforcement learning through studying how states are mapped to actions and how these mappings evolve during training.

Original: For instance, a number of questions for future work includes understanding the behavior of observed linear regions in the contexts of other RL algorithms, and behavior cloning.
Revised: For instance, a number of questions for future work includes understanding the behavior of observed linear regions in the contexts of other RL algorithms, and behavior cloning.

Original: Moreover, we envisage various future directions related to this work, including: (i) understanding the impact of non-IID data on decision regions more generally; (ii) consideration of RL algorithms that have less “learning path dependence”, e.g., via frequent (re)distillation onto randomly reinitialized policies; (iii) using visited regions to efficiently distill more interpretable non-parametric policies; and (iv) in real-world control settings, using knowledge of decision regions, their sizes, and their orderings to better understand exploration and safety issues.
Revised: Moreover, we envisage various future directions related to this work, including: (i) understanding the impact of non-IID data on decision regions more generally; (ii) consideration of RL algorithms that have less “learning path dependence”, e.g., via frequent (re)distillation onto randomly reinitialized policies; (iii) using visited regions to efficiently distill more interpretable non-parametric policies; and (iv) in real-world control settings, using knowledge of decision regions, their sizes, and their orderings to better understand exploration and safety issues.

Original: Lastly, we believe that exploring possible connections with the lottery-ticket hypothesis [Frankle and Carbin, 2018] will be worthwhile.
Revised: Lastly, we believe that exploring possible connections with the lottery-ticket hypothesis [Frankle and Carbin, 2018] will be worthwhile.

Original: As foundational research related to reinforcement learning, this work could be used to develop improved reinforcement learning policy structures and algorithms, with many possible applications, both good and bad in terms of societal impact.
Revised: As foundational research related to reinforcement learning, this work could be used to develop improved reinforcement learning policy structures and algorithms, with many possible applications, both good and bad in terms of societal impact.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work was supported by the NSERC grant RGPIN-2020-05929.
Revised: This work was supported by the NSERC grant RGPIN-2020-05929.

Original: David Rolnick acknowledges support from the Canada CIFAR AI Chairs program.
Revised: David Rolnick acknowledges support from the Canada CIFAR AI Chairs program.

Original: This research was enabled in part by technical support and computational resources provided by Compute Canada ( www.computecanada.ca ).
Revised: This research was enabled in part by technical support and computational resources provided by Compute Canada ( www.computecanada.ca ).

Original: Appendix A Non-Linearity Definitions
Revised: Appendix A Non-Linearity Definitions

Original: The following activation functions are discussed or used in this work:
Revised: The following activation functions are discussed or used in this work:

Original: ReLU [Nair and Hinton, 2010]: max ( x, 0) 2.
Revised: ReLU [Nair and Hinton, 2010]: max ( x, 0) 2.

Original: Tanh (Hyperbolic tangent): ( e x − e − x ) / ( e x + e − x )
Revised: Tanh (Hyperbolic tangent): ( e x − e − x ) / ( e x + e − x )

Original: B Experimental Setup
Revised: B Experimental Setup

Original: All experiments are implemented in Pytorch [Paszke et al., 2019], using the RL codebase from Stable
Revised: All experiments are implemented in Pytorch [Paszke et al., 2019], using the RL codebase from Stable

Original: Baselines

Revised: Baselines

Original: [Raffin et al., 2021].
Revised: [Raffin et al., 2021].

Original: We conduct our analysis on four continuous control tasks from the
Revised: We conduct our analysis on four continuous control tasks from the

Original: OpenAI gym benchmark suits [Brockman et al., 2016] including HalfCheetah-v2, Walker-v2, Ant-v2, and Swimmer-v2 environments, using PPO algorithm
Revised: OpenAI gym benchmark suits [Brockman et al., 2016] including HalfCheetah-v2, Walker-v2, Ant-v2, and Swimmer-v2 environments, using PPO algorithm

Original: [Schulman et al., 2017].
Revised: [Schulman et al., 2017].

Original: All of our experiments are performed over 5 trials with distinct random seeds and the metric plots are shown as the mean and standard deviation across these seeds.
Revised: All of our experiments are performed over 5 trials with distinct random seeds and the metric plots are shown as the mean and standard deviation across these seeds.

Original: All policy networks configurations are chosen such that the network is fully-capable of learning the task and achieve near state-of-the-art cumulative rewards on the particular task it is trained on.
Revised: All policy networks configurations are chosen such that the network is fully-capable of learning the task and achieve near state-of-the-art cumulative rewards on the particular task it is trained on.

Original: Except where specified, network initialization and hyperparameters are set to the defaults of the
Revised: Except where specified, network initialization and hyperparameters are set to the defaults of the

Original: PPO implementation of Stable Baselines3.
Revised: PPO implementation of Stable Baselines3.

Original: We train our policy networks on 2M samples (i.e. 2M timesteps in the environment).
Revised: We train our policy networks on 2M samples (i.e. 2M timesteps in the environment).

Original: Table 1 shows the range of the return per environment, across all network configurations and random seeds.
Revised: Table 1 shows the range of the return per environment, across all network configurations and random seeds.

Original: Return is measured as the mean trajectory cumulative reward across the batch collected at each epoch.
Revised: Return is measured as the mean trajectory cumulative reward across the batch collected at each epoch.

Original: [Table]
Revised: [Table]

Original: C Network Dimensions
Revised: C Network Dimensions

Original: In all experiments
Revised: In all experiments

Original: [Table]
Revised: [Table]

Original: the policy and value function network architectures used are MLPs with ReLU activations in all hidden layers.
Revised: the policy and value function network architectures used are MLPs with ReLU activations in all hidden layers.

Original: Except where specified, the value function has N = 128 neurons with layer widths 64 , 64 while the policy network has 18 structures listed in Table
Revised: Except where specified, the value function has N = 128 neurons with layer widths 64 , 64 while the policy network has 18 structures listed in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D Plots and Error Bars
Revised: D Plots and Error Bars

Original: Each policy network architecture is trained  with 5 random seeds.
Revised: Each policy network architecture is trained 5 times with 5 random seeds.

Original: All metric plots  are shown as the mean and standard deviation across these seeds.
Revised: All metric plots in this work are shown as the mean and standard deviation across these seeds.

Original: For the state space visualization of Figure 3, the sampling grid in input space is obtained by sampling a 300 × 300 grid with equally distanced points on a window of size 40 × 200 where x ∈ [ − 50 , 150] and ˙ x ∈ [ − 20 , 20] .
Revised: For the state space visualization of Figure 3, the sampling grid in input space is obtained by sampling a 300 × 300 grid with equally distanced points on a window of size 40 × 200 where x ∈ [ − 50 , 150] and ˙ x ∈ [ − 20 , 20] .

Original: Next, we feed each point of the sampling grid into the policy network to compute its activation pattern.
Revised: Next, we feed each point of the sampling grid into the policy network to compute its activation pattern.

Original: Then, we group all points with the same activation pattern into a single linear region uniquely colored according to its activation pattern.
Revised: Then, we group all points with the same activation pattern into a single linear region uniquely colored according to its activation pattern.

Original: Note that the way of encoding a linear region of a point as described in Section 4 guarantees that each region will be uniquely identified by an activation pattern.
Revised: Note that the way of encoding a linear region of a point as described in Section 4 guarantees that each region will be uniquely identified by an activation pattern.

Original: We create a color map for regions based on their activation patterns.
Revised: We create a color map for regions based on their activation patterns.

Original: For the state space visualization of Figure 21, we plot regions within a 2D plane that intersects the 17-dimensional input space of HalfCheetah-v2.
Revised: For the state space visualization of Figure 21, we plot regions within a 2D plane that intersects the 17-dimensional input space of HalfCheetah-v2.

Original: To obtain the 2D plane, we sample three random points from a trajectory (final trajectory τ ∗ or random-action trajectory τ R ) and find the plane that passes through these points.
Revised: To obtain the 2D plane, we sample three random points from a trajectory (final trajectory τ ∗ or random-action trajectory τ R ) and find the plane that passes through these points.

Original: We consider all the points within a square centered at the circumcenter of the three sampled points.
Revised: We consider all the points within a square centered at the circumcenter of the three sampled points.

Original: Square is scaled such that it is slightly bigger than the circle.
Revised: Square is scaled such that it is slightly bigger than the circle.

Original: In order to avoid missing small regions, which are dominant in higher dimensional spaces, we do not use the subsampling method used for the visualization of the 2D environment.
Revised: In order to avoid missing small regions, which are dominant in higher dimensional spaces, we do not use the subsampling method used for the visualization of the 2D environment.

Original: Instead, we compute the exact locations of the boundaries of the regions using our counting method describe in Section 4.
Revised: Instead, we compute the exact locations of the boundaries of the regions using our counting method describe in Section 4.

Original: Each region is then colored using a randomly sampled color.
Revised: Each region is then colored using a randomly sampled color.

Original: E Initialization and Hyperparameters
Revised: E Initialization and Hyperparameters

Original: The initialization of the policy network has a high impact on the training performance with PPO [Andrychowicz et al., 2020].
Revised: The initialization of the policy network has a high impact on the training performance with PPO [Andrychowicz et al., 2020].

Original: To be able to focus on the evolution of linear regions disentangled from properties such as trainability, which is outside the scope of this work, we use the default initialization of PPO from Stable Baselines3.
Revised: To be able to focus on the evolution of linear regions disentangled from properties such as trainability, which is outside the scope of this work, we use the default initialization of PPO from Stable Baselines3.

Original: This initialization scheme is consistent with the bestpractice recommendations for on-policy RL [Andrychowicz et al., 2020], but it differs from that used in Hanin and Rolnick [2019a].
Revised: This initialization scheme is consistent with the bestpractice recommendations for on-policy RL [Andrychowicz et al., 2020], but it differs from that used in Hanin and Rolnick [2019a].

Original: In this initialization, for both the policy and value function networks, weights of the hidden layers use orthogonal initialization with scaling √ 2 , and the biases are set to 0 .
Revised: In this initialization, for both the policy and value function networks, weights of the hidden layers use orthogonal initialization with scaling √ 2 , and the biases are set to 0 .

Original: The weights of the policy network’s output layer are initialized with the scale of 0 . 01 , and the weights of the value function network’s output layer are initialized with the scale of 1 .
Revised: The weights of the policy network’s output layer are initialized with the scale of 0 . 01 , and the weights of the value function network’s output layer are initialized with the scale of 1 .

Original: In Hanin and
Revised: In Hanin and

Original: Rolnick [2019a], networks are initialized with He normalization and biases are drawn i.i.d from a normal distribution centered at 0 with variance 10 − 6 .
Revised: Rolnick [2019a], networks are initialized with He normalization and biases are drawn i [Table] i.d from a normal distribution centered at 0 with variance 10 − 6 .

Original: We mainly adopt the default hyperparameters of PPO from the implementation of Stable Baselines in all of our experiments on PPO.
Revised: We mainly adopt the default hyperparameters of PPO from the implementation of Stable Baselines in all of our experiments on PPO.

Original: We present the details of the choices of hyperparameters in Table3.
Revised: We present the details of the choices of hyperparameters in Table

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: F Comparison of PPO and SAC
Revised: F Comparison of PPO and SAC

Original: To shed light on the effect of the choice of RL algorithm, we repeat our experiments with the Soft
Revised: To shed light on the effect of the choice of RL algorithm [Table] we repeat our experiments with the Soft

Original: Actor Critic (SAC) algorithm [Haarnoja et al., 2018].
Revised: Actor Critic (SAC) algorithm [Haarnoja et al., 2018].

Original: Similar to our main experimental setup for PPO, we use the Stable-Baselines3 implementation of the SAC algorithm and conduct our experiments on the four continuous control tasks of HalfCheetah-v2, Walker-v2, Ant-v2, and Swimmer-v2 from the
Revised: Similar to our main experimental setup for PPO, we use the Stable-Baselines3 implementation of the SAC algorithm and conduct our experiments on the four continuous control tasks of HalfCheetah-v2, Walker-v2, Ant-v2, and Swimmer-v2 from the

Original: OpenAI gym benchmark suits.
Revised: OpenAI gym benchmark suits.

Original: Again, we run each experiment with 5 different random seeds and the results show the mean and the standard deviation across the random seeds.
Revised: Again, we run each experiment with 5 different random seeds and the results show the mean and the standard deviation across the random seeds.

Original: The policy network architecture used is an MLP with ReLU activations in all hidden layers.
Revised: The policy network architecture used is an MLP with ReLU activations in all hidden layers.

Original: To ensure full training capacity, we choose more parameterized networks in this setup.
Revised: To ensure full training capacity, we choose more parameterized networks in this setup.

Original: The value function has two hidden layers with 256 neurons in each layer, while the policy network has 12 structures as listed in Table 4.
Revised: The value function has two hidden layers with 256 neurons in each layer, while the policy network has 12 structures as listed in Table 4.

Original: We adopt the network initialization and hyperparameters of SAC from Stable-Baselines3 and train our policy network on 1M samples.
Revised: We adopt the network initialization and hyperparameters of SAC from Stable-Baselines3 and train our policy network on 1M samples.

Original: Table 5 shows the range of the random return per environment, across all network configurations and random seeds.
Revised: Table 5 shows the range of the random return per environment, across all network configurations and random seeds.

Original: Return is measured as the mean trajectory return of ten trajectories collected by running the deterministic policy without action noise once everysteps.
Revised: Return is measured as the mean trajectory return of ten trajectories collected by running the deterministic policy without action noise once everysteps.

Original: We present the details of the choices of hyperparameters in Table 6.
Revised: We present the details of the choices of hyperparameters in Table 6.

Original: Figures 5-12 of this section, show the side-by-side comparison between the results of SAC trained on
Revised: Figures 5-12 of this section, show the side-by-side comparison between the results of SAC trained on

Original: HalfCheetah and results of PPO trained on HalfCheetah previously shown in Figure 4.
Revised: HalfCheetah and results of PPO trained on HalfCheetah previously shown in Figure 4.

Original: We can see that the observed region densities during training are quite different for SAC than they are for PPO, early on in training.
Revised: We can see that the observed region densities during training are quite different for SAC than they are for PPO, early on in training.

Original: We hypothesize that the many transitions (and therefore corresponding high densities) observed early on in training for SAC is due to a combination of (i) the entropy bonus for
Revised: We hypothesize that the many transitions (and therefore corresponding high densities) observed early on in training for SAC is due to a combination of (i) the entropy bonus for

Original: SAC, which is then annealed away, and (ii) the different network initialization used for the baseline
Revised: SAC, which is then annealed away, and (ii) the different network initialization used for the baseline

Original: SAC and PPO implementations.
Revised: SAC and PPO implementations.

Original: Figures 13-20 of this section, show the same set of plots for all four environments, now grouped by metric type to enable easier comparison between environments.
Revised: Figures 13-20 of this section, show the same set of plots for all four environments, now grouped by metric type to enable easier comparison between environments.

Original: As can be observed the results for normalized region densities on the fixed final trajectory (Figure 13) are broadly consistent across environments (tasks), with Swimmer being an exception.
Revised: As can be observed the results for normalized region densities on the fixed final trajectory (Figure 13) are broadly consistent across environments (tasks), with Swimmer being an exception.

Original: [Table]
Revised: [Table]

Original: N= - 16, -
Revised: N= - 16, -

Original: N= - 32, -N= - - 32,32, N=
Revised: N= - 32, -

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: f t r a n s i t i o n
Revised: f t r a n s i t i o n

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: l
Revised: l

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: h
Revised: h

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: compute the density of transitions as we sweep along these lines.
Revised: compute the density of transitions as we sweep along these lines.

Original: We then report the mean density observed over these 100 lines.
Revised: We then report the mean density observed over these 100 lines.

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: n s i t y
Revised: n s i t y

Original: [Figure]
Revised: [Figure]

Original: and compute
Revised: and compute

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: r a
Revised: r a

Original: [Figure]
Revised: [Figure]

Original: i t i
Revised: i t i

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: r o f r e p e a t
Revised: r o f r e p e a t

Original: v i s i t
Revised: v i s i t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: i s
Revised: i s

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: For each policy network configuration, we sample 100 random lines and compute the density of transitions as we sweep along these lines.
Revised: For each policy network configuration, we sample 100 random lines and compute the density of transitions as we sweep along these lines.

Original: We then report the mean density observed over these 100 lines.
Revised: We then report the mean density observed over these 100 lines.

Original: [Figure]
Revised: [Figure]

Original: G Decision Regions viewed via Embedding Planes
Revised: G Decision Regions viewed via Embedding Planes

Original: To gain insights on what the linear regions may look like in our high dimensional setting, similar to Novak et al.
Revised: To gain insights on what the linear regions may look like in our high dimensional setting, similar to Novak et al.

Original: [2018] and Hanin and Rolnick [2019a], we visualize the linear regions over a 2-dimensional slice through the input space defined by three points sampled from a trajectory emerged from the policy.
Revised: [2018] and Hanin and Rolnick [2019a], we visualize the linear regions over a 2-dimensional slice through the input space defined by three points sampled from a trajectory emerged from the policy.

Original: Figure 21 shows the linear regions of a (32 , 32) policy network trained on
Revised: Figure 21 shows the linear regions of a (32 , 32) policy network trained on

Original: HafCheetah in the 17-dimensional input space, over a 2-dimensional plane crossing three projection points sampled from the final trajectory ( τ ∗ ) in the first row, and sampled from the random-action trajectory ( τ R ) in the second row.
Revised: HafCheetah in the 17-dimensional input space, over a 2-dimensional plane crossing three projection points sampled from the final trajectory ( τ ∗ ) in the first row, and sampled from the random-action trajectory ( τ R ) in the second row.

Original: Comparing the visualizations of projected on points from τ ∗ and τ R (first and second row), we do not observe a significant difference between the granularity of regions over points of these two trajectories.
Revised: Comparing the visualizations of projected on points from τ ∗ and τ R (first and second row), we do not observe a significant difference between the granularity of regions over points of these two trajectories.

Original: Moreover, surprisingly, our visualizations are more consistent with the findings of Novak et al.
Revised: Moreover, surprisingly, our visualizations are more consistent with the findings of Novak et al.

Original: [2018] with the projection points lying in regions of lower density.
Revised: [2018] with the projection points lying in regions of lower density.

Original: [Equation]
Revised: [Equation]

Original: Epoch
Revised: Epoch

Original: [Equation]
Revised: [Equation]

Original: Epoch = Epoch =
Revised: Epoch = Epoch =

Original: [Figure]
Revised: [Figure]

Original: H Decision Regions for a Non-Cyclic Task: LunarLander
Revised: H Decision Regions for a Non-Cyclic Task: LunarLander

Original: In this section, we aim to shed light on the possible effects of cyclic nature of locomotion tasks on our previous PPO results.
Revised: In this section, we aim to shed light on the possible effects of cyclic nature of locomotion tasks on our previous PPO results.

Original: Therefore, we repeat our experiments by training PPO on
Revised: Therefore, we repeat our experiments by training PPO on

Original: LunarLanderContinuous-v2 from OpenAI gym.
Revised: LunarLanderContinuous-v2 from OpenAI gym.

Original: Figure 22 shows the plots from this experiment.
Revised: Figure 22 shows the plots from this experiment.

Original: The linear-region evolution behavior is in fact still similar with the only discrepancy being the evolution pattern of observed density over current trajectories.
Revised: The linear-region evolution behavior is in fact still similar with the only discrepancy being the evolution pattern of observed density over current trajectories.

Original: For LunarLander, observed density over current trajectories does not decrease during training while having the same range of values as the density over fixed and random-action trajectories.
Revised: For LunarLander, observed density over current trajectories does not decrease during training while having the same range of values as the density over fixed and random-action trajectories.

Original: This is due to the difference between the nature of non-cyclic and cyclic tasks.
Revised: This is due to the difference between the nature of non-cyclic and cyclic tasks.

Original: For locomotion tasks, the length of the cyclic trajectories increased during training, while for LunarLander, the trajectory length initially increases, then plateaus
Revised: For locomotion tasks, the length of the cyclic trajectories increased during training, while for LunarLander, the trajectory length initially increases, then plateaus

Original: once the the agent converges.
Revised: once the the agent converges.

Original: [Figure]
Revised: [Figure]

Original: r a j ec t o r y l e n g t h
Revised: r a j ec t o r y l e n g t h

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: I Value Network Analysis
Revised: I Value Network Analysis

Original: We also explore decision regions defined by the value-function network.
Revised: We also explore decision regions defined by the value-function network.

Original: We repeat our main experiments on the value network of PPO agents.
Revised: We repeat our main experiments on the value network of PPO agents.

Original: More particularly, we train a policy network with a fixed network structure of (64 , 64) with 18 different value network configurations on HalfCheetah using the PPO algorithm.
Revised: More particularly, we train a policy network with a fixed network structure of (64 , 64) with 18 different value network configurations on HalfCheetah using the PPO algorithm.

Original: The value network configurations used here are the same as the ones used for the policy network and are listed in Table 2, having N ∈ { 32 , 48 , 64 , 96 , 128 , 192 } neurons, widths w ∈ { 8 , 16 , 32 , 64 } , and depths d ∈ { 1 , 2 , 3 , 4 } .
Revised: The value network configurations used here are the same as the ones used for the policy network and are listed in Table 2, having N ∈ { 32 , 48 , 64 , 96 , 128 , 192 } neurons, widths w ∈ { 8 , 16 , 32 , 64 } , and depths d ∈ { 1 , 2 , 3 , 4 } .

Original: These network configurations are chosen such that the network is fully-capable of learning the task and achieves near state-of-the-art cumulative reward on the particular task it is trained on.
Revised: These network configurations are chosen such that the network is fully-capable of learning the task and achieves near state-of-the-art cumulative reward on the particular task it is trained on.

Original: To compute the metrics, we now use the weights and biases of the value-function network, instead of the policy network.
Revised: To compute the metrics, we now use the weights and biases of the value-function network, instead of the policy network.

Original: Figure 23 shows the resulting plots.
Revised: Figure 23 shows the resulting plots.

Original: In comparing Figure 23 and Figure 4, we see that the evolution of the region densities are remarkably similar in structure, for both fixed and current trajectories.
Revised: In comparing Figure 23 and Figure 4, we see that the evolution of the region densities are remarkably similar in structure, for both fixed and current trajectories.

Original: [Figure]
Revised: [Figure]

Original: N u m b e r o f t r a n s i t i o n s
Revised: N u m b e r o f t r a n s i t i o n s

Original: [Figure]
Revised: [Figure]

Original: N u m b e r o f t r a n s i t i o n s
Revised: N u m b e r o f t r a n s i t i o n s

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: T r a j ec t o r y l e n g t h
Revised: T r a j ec t o r y l e n g t h

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: origin
Revised: origin

Original: [Figure]
Revised: [Figure]

Original: J Behavior Cloning
Revised: J Behavior Cloning

Original: We perform an additional experiment to study whether linear regions emerging from policies trained with deep RL on non-IID data are different from those emerging from policies trained with BC on
Revised: We perform an additional experiment to study whether linear regions emerging from policies trained with deep RL on non-IID data are different from those emerging from policies trained with BC on

Original: IID data.
Revised: IID data.

Original: For a direct comparison, we repeat our experiments by first collecting expert data from each of the 18 network architectures previously trained with PPO on HalfCheetah.
Revised: For a direct comparison, we repeat our experiments by first collecting expert data from each of the 18 network architectures previously trained with PPO on HalfCheetah.

Original: For each expert PPO policy, we initialize a neural network policy with the same architecture and train it using BC on the collected expert data.
Revised: For each expert PPO policy, we initialize a neural network policy with the same architecture and train it using BC on the collected expert data.

Original: We then evaluate this new set of trained policies using the same evaluation method.
Revised: We then evaluate this new set of trained policies using the same evaluation method.

Original: For policy architecture A previously trained with PPO on HalfCheetah, we randomly pick one of theexperts (from the random seeds), and collect 500 episodes of expert data by letting the policy interact with the environment.
Revised: For policy architecture A previously trained with PPO on HalfCheetah, we randomly pick one of theexperts (from the random seeds), and collect 500 episodes of expert data by letting the policy interact with the environment.

Original: Actions are sampled from the stochastic policy.
Revised: Actions are sampled from the stochastic policy.

Original: We then train a new policy network with the same architect A using BC on the collected dataset.
Revised: We then train a new policy network with the same architect A using BC on the collected dataset.

Original: We repeat each experiment with 5 random seeds.
Revised: We repeat each experiment with 5 random seeds.

Original: To ensure BC-trained policies are fully trained, we choose hyperparameters such that the policy achieves similar average episodic return to its expert PPO-trained policy.
Revised: To ensure BC-trained policies are fully trained, we choose hyperparameters such that the policy achieves similar average episodic return to its expert PPO-trained policy.

Original: We present the details of the choices of hyperparameters in Table 7.
Revised: We present the details of the choices of hyperparameters in Table 7.

Original: We use the default initialization of linear layers used by PyTorch where weights are He normal with gain= √ 5 , and biases are IID normal with variance 1 / fan-in .
Revised: We use the default initialization of linear layers used by PyTorch where weights are He normal with gain= √ 5 , and biases are IID normal with variance 1 / fan-in .

Original: [Table]
Revised: [Table]

Original: Figures 24-31 of this section, show the side-by-side comparison between the results of BC trained on
Revised: Figures 24-31 of this section, show the side-by-side comparison between the results of BC trained on

Original: HalfCheetah and results of PPO trained on HalfCheetah previously shown in Figure 4.
Revised: HalfCheetah and results of PPO trained on HalfCheetah previously shown in Figure 4.

Original: We can see that the observed region densities during training are much smaller for BC than they are for PPO and that the general trend of increased density is not visible for BC.
Revised: We can see that the observed region densities during training are much smaller for BC than they are for PPO and that the general trend of increased density is not visible for BC.

Original: We hypothesize that these differences are due to a combination of (i) different evolution of linear regions for networks trained with RL and supervised learning due to their inherent difference in levels of information about the state space (ii) the different network initializations used for the BC and PPO implementations
Revised: We hypothesize that these differences are due to a combination of (i) different evolution of linear regions for networks trained with RL and supervised learning due to their inherent difference in levels of information about the state space (ii) the different network initializations used for the BC and PPO implementations

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 120 Epoch 0 .0 .0 .1 .1 .
Revised: 120 Epoch 0 .0 .0 .1 .1 .

Original: R e g i o n d e n s i t y [32][64] [128][16,16] [32,32][64,64] [8,8,8,8][16,16,16,16] [32,32,32,32]
Revised: R e g i o n d e n s i t y [32][64] [128][16,16] [32,32][64,64] [8,8,8,8][16,16,16,16] [32,32,32,32]

Original: [Figure]
Revised: [Figure]

Original: 120 Epoch 0 .0 .0 .0 .0 .0 .0 .0 .
Revised: 120 Epoch 0 .0 .0 .0 .0 .0 .0 .0 .

Original: R e g i o n d e n s i t y [32][64] [128][16,16] [32,32][64,64] [8,8,8,8][16,16,16,16] [32,32,32,32]
Revised: R e g i o n d e n s i t y [32][64] [128][16,16] [32,32][64,64] [8,8,8,8][16,16,16,16] [32,32,32,32]

Original: [Figure]
Revised: [Figure]

Original: 120 Epoch 0 .0 .0 .0 .0 .0 .
Revised: 120 Epoch 0 .0 .0 .0 .0 .0 .

Original: R e g i o n d e n s i t y [32][64] [128][16,16] [32,32][64,64] [8,8,8,8][16,16,16,16] [32,32,32,32]
Revised: R e g i o n d e n s i t y [32][64] [128][16,16] [32,32][64,64] [8,8,8,8][16,16,16,16] [32,32,32,32]

Original: [Figure]
Revised: [Figure]

Original: K Additional Results
Revised: K Additional Results

Original: We previously provided the results for the HalfCheetah environment in Figure 4.
Revised: We previously provided the results for the HalfCheetah environment in Figure 4.

Original: Figures 32of
Revised: Figures 32of

Original: this section, show the same set of plots for the three remaining environments trained with PPO.
Revised: this section, show the same set of plots for the three remaining environments trained with PPO.

Original: Plots are now grouped by metric type instead of environment, to enable easier comparison between environments.
Revised: Plots are now grouped by metric type instead of environment, to enable easier comparison between environments.

Original: Figure 40 shows the full set of results of Figures 4(a) and 4(c) for all policy network architectures trained on HalfCheetah.
Revised: Figure 40 shows the full set of results of Figures 4(a) and 4(c) for all policy network architectures trained on HalfCheetah

Original: We previously provided the results for the HalfCheetah environment in Figure

Original: [Figure]
Revised: [Figure]

Original: policy networks having more transitions.
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: , d corresponds to the depth of the policy network. These plots show that the density over fixed and current trajectories converges to the same values over training, while it increases for former and decreases for the latter.
Revised: same values over training, while it increases for former and decreases for the latter.

Original: We speculate that the density is higher for current trajectories earlier during training due to early exploration and the form of network initialization.[Figure] [Figure] i
Revised: We speculate that the density is higher for current trajectories earlier during training due to early exploration and the form of network initialization.

Original: [Figure]
Revised: [Figure]

Original: r

Original: [Figure]
Revised: [Figure]

Original: i o n

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: i t
Revised: i t

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]