In Support of Over-Parametrization in Deep Reinforcement Learning: an Empirical Study
Abstract
There is signiﬁcant recent evidence in supervised learning that, in the over-parametrized setting, wider networks achieve better test error.
In other words, the bias-variance tradeoff is not directly observable when increasing network width arbitrarily.
We investigate whether a corresponding phenomenon is present in reinforcement learning.
We experiment on four OpenAI Gym environments, increasing the width of the policy and value networks beyond their prescribed values.
Our empirical results lend support to this hypothesis.
However, tuning the hyperparameters of each network width separately remains as important future work in environments/algorithms where the optimal hyperparameters vary noticably across widths, confounding the results when the same hyperparameters are used for all widths.
1. Introduction
A longstanding notion in supervised learning is that, as model complexity increases, test error decreases initially and, eventually, increases again.
Intuitively, the idea is that as the size of your hypothesis class grows, the closer you can approximate the ground-truth function with some function in your hypothesis class.
At the same time, the larger amount of functions to choose from in your hypothesis class leads to higher estimation error (overﬁtting) from ﬁtting the ﬁnite data sample too closely.
This is the essential bias-variance tradeoff in supervised learning.
We discuss these tradeoffs in more depth in Section 2.2.
However, Neyshabur et al.
(2015) found that increasing the width of a single hidden layer neural network leads to decreasing test error on MNIST and CIFAR-10.
Since then, there has been a large amount of evidence that wider networks generalize better in a variety of different architectures and hyperparameter settings (Zagoruyko & Komodakis, 2016; Novak et al., 2018; Lee et al., 2018; Neal et al., 2018; Belkin et al., 2018; Spigler et al., 2018; Liang et al., 2017), once in the over-parametrized setting (Spigler et al., 2018; Belkin et al., 2018).
In other words, the biasvariance tradeoff is not observed in this over-parametrized setting, as network width grows (Neal et al., 2018).
How far can we inductively infer from this?
Is this phenomenon also present in deep reinforcement learning or do we eventually see a degradation in performance as we increase network width?
In this paper, we present preliminary evidence that this phenomenon is also present in reinforcement learning.
For example, using default hyperparameters, we can already see performance increase well past the default width that is commonly used (64) in Fig.
We test the hypothesis that wider networks (both policy and value) perform monotonically better than their smaller counterparts in policy gradients methods.
Of course, we will hit diminishing returns as the network width gets very large, but this is very different from the competing hypothesis that larger networks will overﬁt more.
2. Preliminaries 2.1. Supervised Learning Setting
We are given a training set
S = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x m , y m ) } of m training examples, where x i ∈ X and y i ∈ Y .
Furthermore,
Z = X
× Y ,
so S ∈
Z m .
D denotes a distribution over Z , so we have ( x i , y i ) ∼ D and S ∼ D m .
We use lowercase x and y to denote random variables because of convention in this ﬁeld.
We learn a hypothesis h ∈ H via a learning algorithm A :
Z m → H .
We denote a hypothesis learned from training set S as h S =
A ( S ) .
Given a loss function, (cid:96) :
Y×Y → R , the goal is to minimize the expected risk :
[Equation]
2.2. Tradeoffs in Model Complexity
We present a discussion on tradeoffs in model complexity because it does not appear to be much of a focus in the reinforcement learning community.
A common way of thinking about the generalization performance of a learner is through the lens of a tradeoff.
For example, when h S is chosen from a hypothesis class H , R ( h S ) can be decomposed into approximation error and estimation error
[Equation]
where E app = min h ∈H R ( h ) and E est
= R ( h S )
− E app .
Shalev-Shwartz & Ben-David (2014, Section 5.2) present this decomposition and frame it as a tradeoff.
Bottou &
Bousquet (2008) describe this as the “well known tradeoff between approximation error and estimation error” and present it in a slightly more lucid way as a decomposition of the excess risk :
[Equation]
)] where R ( h ∗ ) is the Bayes error and h ∗H = arg min h ∈H R ( h ) is the best hypothesis in H .
The approximation error can then be interpreted as the distance of the best hypothesis in H from the Bayes classiﬁer, and the estimation error can be interpreted as the average distance of the learned hypothesis from the best hypothesis in H .
It is common to associate larger H with smaller approximation error and larger estimation error.
The commonly cited universal approximation property of neural networks (Cybenko, 1989; Hornik, 1991; Leshno &
Schocken, 1993) means that the approximation error goes to 0 as the network width increases; these results do not say anything about estimation error.
A similar tradeoff in model complexity is known as the bias-variance tradeoff (Geman et al., 1992).
Bias is analogous to the approximation error while variance is analogous to the estimation error.
This tradeoff is probably even more pervasive (Bishop (2006, Chapter 3.2), Geman et al.
(1992), Hastie et al.
(2001, Chapter 2.9), Goodfellow et al.
(2016, Chapter 5.4.4)).
It is common to view the problem of designing a good learning algorithm as choosing a good H that optimizes this tradeoff.
2.3. Comparison of Supervised Learning with Reinforcement Learning
Statistical learning theory for supervised learning is given in the i.i.d.
setting.
That is, examples are independent and identically distributed.
This also means the training distribution is the same as the test distribution.
In reinforcement learning, training examples are not independent because examples within the same episode depend on each other through the current behavior policy and through the environment’s transition dynamics.
Training examples are not identically distributed because the policy produces training examples, and the policy changes over time.
For the same reason, the training distribution and the test distribution are not completely the same.
These differences make it nonobvious that the phenomenon seen in supervised learning would extend to reinforcement learning.
3. Experiments 3.1. Experimental Details
We run experiments, with a variety of combinations of environments and learning algorithms, where we vary the width of the shared policy and value network.
We use four different environments from OpenAI Gym (Brockman et al., 2016): CartPole, Acrobot, MountainCar, and Pendulum.
We use four different learning algorithms: PPO (Schulman et al., 2017), A2C
(Mnih et al., 2016), ACER (Wang et al., 2017), and ACKTR (Wu et al., 2017).
We make use of the existing implementations of these algorithms in the Stable Baselines library (Hill et al., 2018), an improved fork of OpenAI Baselines (Dhariwal et al., 2017).
We were only able to train ACKTR up to widthbecause it is an approximate second-order method.
Experiments with ACKTR are in Appendix B.
We get hyperparameters that were tuned on networks of width 64 from the RL Baselines Zoo that was built on top of Stable Baselines.
One hyperparameter is how many time steps the learners are trained for.
It is different for different environment/learner pairs, but always on the order of 1 million.
It is always the same across widths within an environment/learner pair.
In some of the plots, learners see fewer episodes because their episodes are, on average, longer.
We choose relatively simple tasks for these experiments partially because they are faster to train on, but more importantly, we choose them because their simplicity lends itself to more ease of overﬁtting.
In other words, on these tasks, we will see diminishing returns with much smaller networks, so we can test the “very wide networks will not see degraded performance” hypothesis with a much smaller range of networks.
We run each experiment with 5 different random seeds.
The policy and value networks are shared.
The architecture consists of 2 hidden fully connected layers followed by separate linear transformations: one to yield the policy and one to yield the value.
We use 2 hidden layers, rather than just 1, because 2 hidden layers are more common in reinforcement learning.
3.2. CartPole, Acrobot, and Pendulum Environments
In CartPole (Fig. 2), we see a lot of evidence for the hypothesis.
In the both the PPO and A2C experiments, peak performance is reached by width 64, and that level of performance is maintained through width 2048.
In the ACER experiment, near peak performance is reached by width 128, and through width 2048, we see peak performance.
Similarly, in Acrobot (Fig. 3), we see even more evidence for the hypothesis.
We see peak performance as early as width 16 in PPO, ACER, and A2C.
This means that Acrobot is simple enough to only require a network of width 16 (compared to 64 for CartPole).
Still, we see peak performance through width 2048 in all 3 learners.
In Pendulum (Appendix A), we see more evidence for the hypothesis.
The default width (64) network, performs distinctly worse than the wider networks.
We do not see any degradation of performance through width 2048.
We only run PPO with the Pendulum environment because RL Baselines Zoo did not have tuned hyperparameters for the other algorithms.
3.3. MountainCar Environment
In the MountainCar environment, we see the ﬁrst hint of what looks like could be evidence against the hypothesis (Fig. 4).
PPO (left) performance begins to degrade at width 2048, ACER (center) performance begins to degrade at width 512, and we see a sharp drop in performance from width 1024 to width 2048 in A2C (right).
RL algorithms are known to be highly sensitive to hyperparameter settings (Henderson et al., 2018a;
Islam et al., 2017), especially learning rate (Henderson et al., 2018b).
We believe this performance degradation is due to more variability across widths of the optimal hyperparameters on MountainCar (compared CartPole, Acrobot, and Pendulum).
Automatic Learning Rate Scaling
In order to fairly compare all the widths, we would like the hyperparameters for each of them to be optimal.
(Geiger et al., 2019) study test error when scaling network width in supervised learning, and they scale the learning rate as h − 1 .
5 , where h is the network width.
This scaling is motivated by making the number of steps to convergence independent of width, but it does not necessarily make the learning rate for each network optimal.
Because learning rate is such an important and sensitive hyperparameter in reinforcement learning (Henderson et al., 2018b), we try scaling the learning rate α with both of the following schemes: α ← min( α ∗ 64 , ( h/ 64) − 1 ) and α ← min( α ∗ 64 , ( h/ 64) − 1 . 5 ) , where α ∗ 64 is the learning rate that was tuned to network width 64 (pulled from RL Baselines
Zoo).
We see that scaling the learning rate as h − 1 (Fig. 5) and h − 1 .
5 (Appendix C, Fig.
8) actually make the largest networks perform worse, indicating that this scaling is not useful for comparing networks with optimal hyperparameters.
We present these scalings on MountainCar because it was the environment that did not look like the others, but the scalings on CartPole and Acrobot are in Appendix D.
4. Conclusion and Future Work
The phenomenon in supervised learning that motivated this work is that, in the over-parametrized setting, increasing network width leads to monotonically lower test error (no
U curve).
We ﬁnd a fair amount of evidence of this phenomenon extending to reinforcement learning in our preliminary experiments (namely CartPole, Acrobot, and Pendulum).
However, we also saw that performance did consistently degrade in the MountainCar experiments.
We believe this to be because that environment is more sensitive to hyperparameters; since the hyperparameters were chosen using width 64 and then used for all of the other widths, the hyperparameters are likely not optimal for the other widths like they are for width 64.
The MountainCar environment exaggerates this lack suboptimality more than the otherenvironments.
The main next experiments we plan to run will use an automated tuning procedure that chooses the hyperparameters for each width individually.
We believe this protocol will yield MountainCar results that look much more like the CartPole and Acrobot results.
We then plan to replicate these ﬁndings across more learning algorithms and more environments.
A. Pendulum
B. Experiments with ACKTR
C. Learning Rate Scaling on MountainCar
D. Learning Rate Scaling Experiments on CartPole and Acrobot