O BSERVATIONAL O VERFITTING IN R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
A major component of overﬁtting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP).
We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP.
When an agent overﬁts to different observation spaces even if the underlying MDP dynamics is ﬁxed, we term this observational overﬁtting .
Our experiments expose intriguing properties especially with regards to implicit regularization , and also corroborate results from previous works in RL generalization and supervised learning (SL).
I NTRODUCTION
Generalization for RL has recently grown to be an important topic for agents to perform well in unseen environments.
Complication arises when the dynamics of the environments entangle with the observation, which is often a high-dimensional projection of the true latent state.
One particular frame work, which we denote the zero-shot supervised framework (Zhang et al., 2018a;c; Nichol et al., 2018; Justesen et al., 2018) used to study RL generalization is to treat it analogous to a classical supervised learning (SL) problem – i.e. assume there exists a distribution of MDPs, train jointly on a ﬁnite “training set” sampled from this distribution, and check expected performance on the entire distribution, with the ﬁxed trained policy.
In this framework, there is a spectrum of analysis, ranging from almost purely theoretical analysis (Wang et al., 2019; Asadi et al., 2018) to full empirical results on diverse environments (Zhang et al., 2018c; Packer et al., 2018).
However, there is a lack of analysis in the middle of this spectrum.
On the theoretical side, previous works do not provide analysis for the case when the underlying MDP is relatively complex and requires the policy to be a non-linear function approximator such as a neural network.
On the empirical side, there is no common ground between recently proposed empirical benchmarks.
This is partially caused by multiple confounding factors for RL generalization that can be hard to identify and separate.
For instance, an agent can overﬁt to the MDP dynamics of the training set, such as for control in Mujoco (Pinto et al., 2017; Rajeswaran et al., 2017).
In other cases, an RNN-based policy can overﬁt to maze-like tasks in exploration (Zhang et al., 2018c), or even exploit determinism and avoid using observations (Bellemare et al., 2012; Machado et al., 2018).
Furthermore, various hyperparameters such as the batch-size in SGD (Smith et al., 2018), choice of optimizer (Kingma &
Ba, 2014), discount factor γ (Jiang et al., 2015) and regularizations such as entropy (Ahmed et al., 2018) and weight norms (Cobbe et al., 2018) can also affect generalization.
Due to these confounding factors, it can be unclear what parts of the MDP or policy are actually contributing to overﬁtting or generalization in a principled manner, especially in empirical works with newly proposed benchmarks.
In order to isolate these factors, we study one broad factor affecting generalization that is most correlated with themes in SL, speciﬁcally observational overﬁtting , where an agent overﬁts due to properties of the observation which are irrelevant to the latent dynamics of the MDP family.
To study this factor, we ﬁx a single underlying MDP’s dynamics and generate a distribution of MDPs by only modifying the observational outputs.
Our contributions in this paper are the following:
We discuss realistic instances where observational overﬁtting may occur and its difference from other confounding factors, and design a parametric theoretical framework to induce observational overﬁtting that can be applied to any underlying MDP.
We study observational overﬁtting with linear quadratic regulators (LQR) in a synthetic environment and neural networks such as multi-layer perceptrons (MLPs) and convolutions in classic Gym environments.
A primary novel result we demonstrate for all cases is that implicit regularization occurs in this setting in RL.
We further test the implicit regularization hypothesis on the benchmark CoinRun from using MLPs, even when the underlying MDP dynamics are changing per level.
The structure of this paper is outlined as follows: Section 2 discusses the motivation behind this work and introduces the theoretical model we use to study the effects of observations on overﬁtting.
Section 3 demonstrates numerous experiments using this synthetic construction that suggest implicit regularization is at work.
Finally, Section 4 tests the implicit regularization hypothesis and discusses ablations on various modern image classiﬁcation architectures and margin metrics found in the
Appendix.
M OTIVATION AND R ELATED W ORKS
Currently most architectures used in model-free RL are simple (with fewer than one million parameters) compared to the much larger and more complex ImageNet architectures used for classiﬁcation.
This is due to the fact that most RL environments studied either have relatively simple and highly structured images (e.g. Atari) compared to real world images, or conveniently do not directly force the agent to observe highly detailed images.
For instance in large scale RL such as DOTA2 (OpenAI, 2018) or Starcraft 2 (Vinyals et al., 2017), the agent observations are internal minimaps pertaining to object xy-locations, rather than human-rendered observations.
Figure 1 highlights the issues surrounding MDPs with rich, textured observations - speciﬁcally, the agent can use any features that are correlated with progress, even those which may not generalize across levels.
This is an important issue for vision-based policies, as many times it is not obvious what part of the observation causes an agent to act or generalize.
The concept of “changing background” is not new, as shown by artiﬁcial benchmarks (Zhang et al., 2018b; Gamrian & Goldberg, 2019) which augment the background of 2D observations with colors, shapes and videos.
However, a key difference in our work is that we explicitly require the “background” to be correlated with the progress rather than loosely correlated (e.g. through determinism between the background and the game avatar) or not at all.
This setting makes a more explicit connection to causal inference (Arjovsky et al., 2019; Heinze-Deml & Meinshausen, 2019;
Heinze-Deml et al., 2019) where spurious correlations between ungeneralizable features and progress may make training easy, but are detrimental to test performance because they induce false attributions.
Previously, many works interpret the decision-making of an agent through saliency and other network visualizations (Greydanus et al., 2018; Such et al., 2018) on common benchmarks such as Atari.
Our work is motivated by learning theoretic frameworks to capture this phenomena, as there is vast literature on understanding the generalization properties of SL classiﬁers (Vapnik & Chervonenkis, 1971; McAllester, 1999; Bartlett & Mendelson, 2002) and in particular neural networks (Neyshabur et al., 2015; Dziugaite & Roy, 2017; Neyshabur et al., 2017; Bartlett et al., 2017; Arora et al., 2018c).
For an RL policy with high-dimensional observations, we hypothesize its overﬁtting can come from more theoretically principled reasons, as opposed to purely good inductive biases on game images.
Hence, our unifying framework is also able to analyze the 1D case using linear projections.
As an example of what may happen in high dimensional observation space, consider linear least squares regression task where given the set X ∈ R m × d and Y ∈ R m , we want to ﬁnd w ∈ R d that minimizes (cid:96) X,Y ( w )
= (cid:107) Y −
Xw
(cid:107) 2 where m is the number of samples and d is the input dimension.
We know that if X (cid:62) X is full rank (hence d ≤ m ), (cid:96) X,Y ( . ) has a unique global minimum w ∗ =
( X (cid:62) X ) − 1 X (cid:62) Y .
On the other hand if X (cid:62) X is not full rank (eg.
when m < d ), then there are many global minima w ∗ such that Y =
Xw ∗ 1 .
Luckily, if we use any gradient based optimization to minimize the loss and initialize with w = 0 , the solution will only span column spaces of X and converges to minimum (cid:96) 2 norm solution among all global minima due to implicit regularization (Gunasekar et al., 2017).
Thus a high dimensional observation space with a low dimensional state space can induce multiple solutions, some of which are not generalizable to other functions or MDPs but one could hope that implicit regularization would help avoiding this issue.
2. N OTATION
In the zero-shot framework for RL generalization, we assume there exists a distribution D over MDPs
M for which there exists a ﬁxed policy π opt that can achieve maximal return on expectation over
MDPs generated from the distribution.
An appropriate ﬁnite training set (cid:99) M train =
{M 1 , . . .
, M n } can then be created by repeatedly randomly sampling M ∼ D .
Thus for a MDP M and any policy π , expected episodic reward is deﬁned as R M ( π ) .
In many empirical cases, the support of the distribution D is made by parametrized MDPs where some process, given a parameter θ , creates a mapping θ → M θ (e.g. through procedural generation), and thus we may simplify notation and instead deﬁne a distribution Θ that induces D , which implies a set of samples (cid:98) Θ train = { θ 1 , . . .
, θ n } also induces a (cid:99) M train =
{M 1 , . .
, M n } , and we may redeﬁne reward as R M θ ( π ) =
R θ ( π ) .
As a simpliﬁed model of the observational problem from Sonic, we can construct a mapping θ → M θ by ﬁrst ﬁxing a base MDP M = ( S , A , r, T ) , which corresponds to state space, action space, reward, and transition.
The only effect of θ is to introduce an additional observation function w θ : S → O , where the agent receives input from the high dimensional observation space O rather than from the state space S .
Thus, for our setting, θ actually parameterizes a POMDP family which can be thought of as simply a combination of a base MDP M and an observational function w θ , hence
M θ =
( M , w θ ) .
Let (cid:98) Θ train = { θ 1 , . . .
, θ n } be a set of n i.i.d.
samples from Θ , and suppose we train π to optimize reward against {M θ : θ ∼ (cid:98) Θ train } .
The objective J (cid:98) Θ ( π ) =
[Equation]
) is the average reward over this empirical sample.
We want to generalize to the distribution Θ , which can be expressed as the average episode reward R over the full distribution, i.e. J Θ ( π ) =
E θ ∼ Θ
[ R θ ( π )] .
Thus we deﬁne the generalization gap as J (cid:98) Θ ( π ) − J Θ ( π ) .
2. S ETUP
We can model the effects of Figure 1 more generally, not speciﬁc to sidescroller games.
We assume that there is an underlying state s (e.g. xy-locations of objects in a game), whose features may be very well structured, but that this state has been projected to a high dimensional observation space by
[Figure]
w θ .
To abstract the notion of generalizable and non-generalizable features, we construct a simple and natural candidate class of functions, where
[Equation]
[Figure]
In this setup, f ( · ) is a function invariant for the entire MDP population Θ , while g θ ( · ) is a function dependent on the sampled parameter θ .
h is a ”combination” function which combines the two outputs of f and g to produce a ﬁnal observation.
While f projects this latent data into salient and important, invariant features such as the avatar, monsters, and items, g θ projects the latent data to unimportant features that do not contribute to extra generalizable information, and can cause overﬁtting, such as the changing background or textures.
A visual representation is shown in Figure 2.
This is a simpliﬁed but still insightful model relevant in more realistic settings.
An important distinction we note is that while g θ can cause overﬁtting in the zero-shot setting, it can be useful in the few-shot setting as it provides more information in identifying the MDP, such as in task-identiﬁcation (Yu et al., 2017; Peng et al., 2018) and fast adaptation in meta-learning (Finn et al., 2017).
From now on, we denote this setup as the ( f, g ) -scheme .
This setting also leads to more interpretable generalization bounds - Lemma 2 of (Wang et al., 2019) provides a high probability (1 − δ ) bound for the “intrinsic” generalization gap for a set of policies Π when m levels are sampled: gap ≤ Rad m ( R Π )
+ O (cid:18)(cid:113)
log(1 /δ )
m (cid:19) , where
Rad m
[Equation]
=
[Equation]
E
[Equation]
} (cid:2) sup π ∈ Π 1 m (cid:80) mi =1 σ i R θ i ( π ) (cid:3) (cid:105) is the Rademacher Complexity under the MDP, where θ i are the ζ i parameters used in the original work, and the transition T and initialization I are ﬁxed, therefore omitted, to accommodate our setting.
The Rademacher Complexity term captures how invariant policies in the set Π with respect to θ .
For most RL benchmarks, this is not interpretable due to multiple confounding factors such as the varying level dynamics.
For instance, it is difﬁcult to imagine what behaviors or network weights a policy would possess in order to produce the exact same total rewards, regardless of changing dynamics.
However, in our case, because the environment parameters θ are only from g θ , the Rademacher
Complexity is directly based on how much the policy “looks at” g θ .
More formally, let Π ∗ be the set of policies π ∗ which are not be affected by changes in g θ ; i.e. ∇ θ π ∗ ( w θ ( s ))
= 0 ∀ s and thus
R θ ( π ∗ ) =
R const ∀ θ , which implies that the environment parameter θ has no effect on the reward; hence Rad m
[Equation]
= 0 .
2. A RCHITECTURE AND I MPLICIT R EGULARIZATION
Normally in a MDP such as a game, the concatenation operation may be dependent on time (e.g. textures move around in the frame).
In the scope of this work, we simplify the concatenation effect and assume h ( · ) is a static concatenation, but still are able to demonstrate insightful properties.
We note that this inductive bias on h allows explicit regularization to trivially solve this problem, by penalizing a policy’s ﬁrst layer that is used to “view” g θ ( s ) (Appendix A3), hence we only focus on implicit regularizations.
This setting is naturally attractive to analyzing architectural differences, as it is more closely related in spirit to image classiﬁers and SL.
One particular line of work to explain the effects of certain architectural modiﬁcations in SL such as overparametrization and residual connections is implicit regularization (Neyshabur, 2017; Neyshabur et al., 2018b), as overparametrization through more layer depth and wider layers has proven to have no (cid:96) p -regularization equivalent (Arora et al., 2019), but rather precondition the dynamics during training.
Thus, in order to fairly experimentally measure this effect, we always use ﬁxed hyperparameters and only vary based on architecture.
In this work, we only refer to architectural implicit regularization techniques, which do not have a explicit regularization equivalent.
Some techniques e.g. coordinate descent (Bradley et al., 2011) are equivalent to explicit (cid:96) 1 -regularization.
E XPERIMENTS 3. O VERPARAMTERIZED LQR
We ﬁrst analyze the case of the LQR as a surrogate for what may occur in deep RL, which has been done before for various topics such as sample complexity (Dean et al., 2017) and model-based RL (Tu & Recht, 2019).
This is analogous to analyzing linear/logistic regression (Kakade et al., 2008;
McAllester, 2003) as a surrogate to understanding extensions to deep SL techniques (Neyshabur et al., 2018a; Bartlett et al., 2017).
In particular, this has numerous beneﬁts - the cost (negative of reward) function is deterministic, and allows exact gradient descent (i.e. the policy can differentiate through the cost function) as opposed to necessarily using stochastic gradients in normal RL, and thus can cleanly provide evidence of implicit regularization.
Furthermore, in terms of gradient dynamics and optimization, LQR readily possesses nontrivial qualities compared to linear regression, as the LQR cost is a non-convex function but all of its minima are global minima (Fazel et al., 2018).
To show that overparametrization alone is an important implicit regularizer in RL, LQR allows the use of linear policies (and consequently also allows stacking linear layers) without requiring a stochastic output such as discrete Gumbel-softmax or for the continuous case, a parametrized Gaussian.
This is setting able to show that overparametrization alone can affect gradient dynamics, and is not a consequence of extra representation power due to additional non-linearities in the policy.
There have been multiple recent works on this linear-layer stacking in SL and other theoretical problems such as matrix factorization and matrix completion (Arora et al., 2018b;a; Neyshabur, 2017), but to our knowledge, we are the ﬁrst to analyze this case in the context of RL generalization.
We explicitly describe setup as follows: for a given θ , we let f ( s ) =
W constant · s , while g θ ( s ) =
W θ · s where W constant , W θ are semi-orthogonal matrices, to prevent information loss relevant to outputting the optimal action, as the state is transformed into the observation.
Hence, if s t is the underlying state at time t , then the observation is o t = (cid:20) W constant
W θ (cid:21) s t and thus the action is a t =
Ko t , where K is the policy matrix.
We sample W θ randomly, using the ”level ID” integer θ as the seed for random generation.
In terms of dimensions, if s is of shape
d state , then f also projects to a shape of d state , while g θ projects to a much larger shape d noise , implying that the observation to the agent is of dimension d signal + d noise .
In our experiments, we set as default ( d signal , d noise ) =
A key insight is that that a policy K acting on high dimensional observation Ws is equivalent to a policy K state =
KW acting on low dimensional state s .
We begin with a theorem which implies that a high dimensional observational space directly contributes to overﬁtting:
Theorem 3.1 For LQRs whose observation consists of ( d signal , d noise ) -dimensional vectors constructed with the fg -scheme and ﬁxed number of training levels m , the generalization gap upper bound scales with O ( √ d noise ) with high probability.
We empirically verify that this bound is tight in Figure 3 and defer the detailed proof to Appendix
A.4.3.
Denote (cid:107)·(cid:107) , (cid:107)·(cid:107) 1 , (cid:107)·(cid:107) F as the spectral, (cid:96) 1 , and Frobenius norms respectively of a matrix.
Experimentally, we added more (100 × 100) linear layers K = K 0 K 1 , ..., K j and increased widths for a 2-layer case (Figure 3), and observe that both settings reduce the generalization gap, and also reduce the norms (spectral, nuclear, Frobenius) of the ﬁnal end-to-end policy K , without changing its expressiveness.
This suggests that gradient descent under overparametrization implicitly biases the policy towards a “simpler” model in the LQR case.
As a surrogate model for deep RL, one may ask if the generalization gap of the ﬁnal end-to-end policy K can be predicted by functions of the layers K 0 , ..., K j .
This is an important question as it is a required base case for predicting generalization when using stochastic policy gradient with nonlinear activations such as ReLU or Tanh.
From examining the distribution of singular values on
K (Appendix A1), we ﬁnd that more layers does not bias the policy towards a low rank solution in the nonconvex LQR case, unlike (Arora et al., 2018b) which shows this does occur for matrix completion, and in general, convex losses.
Ultimately, we answer in the negative: intriguingly, SL bounds have very little predictive power in the RL domain case.
To understand why SL bounds may be candidates for the LQR case, we note that as a basic smoothness bound C ( K ) − C ( K (cid:48) )
≤ O ( (cid:107) K − K (cid:48)
(cid:107) 3 ) (Appendix A.4.3) can lead to very similar reasoning with
SL bounds.
Since our setup is similar to SL in that “LQR levels” which may be interpreted as a dataset, we use bounds of the form ∆ · Φ , where ∆ is a “macro” product term ∆ =
(cid:81) ji =0 (cid:107) K i (cid:107) ≥ (cid:13)(cid:13)(cid:13)(cid:81)
ji =
K i (cid:13)(cid:13)(cid:13) derivable from the fact that (cid:107) AB (cid:107) ≤ (cid:107) A (cid:107) (cid:107) B (cid:107) in the linear case, and Φ is a weight-counting term which deals with the overparametrized case, such as Φ = (cid:80) j i =
(cid:107) K
i (cid:107) 2 F (cid:107) K i (cid:107) 2 (Neyshabur et al., 2018a) or Φ = (cid:18)(cid:80) ji =0 (cid:16) (cid:107) K i (cid:107) 1 (cid:107) K i (cid:107) (cid:17) 2 / 3 (cid:19)
(Bartlett et al., 2017).
While gap ≤ (cid:113) O
( (cid:107) K raw (cid:107) 3 ) ≤
O ( (cid:81) ji =0 (cid:107) K i (cid:107) 32 ) =
∆ 3 / 2 , we may replace any SL perturbation bounds | f w ( x ) − f w (cid:48) ( x ) | with
C ( K ) − C
( K (cid:48) )
≤ O ( (cid:107) K − K (cid:48)
(cid:107) 3 ) , which can grant us expressions similar to Φ , but with different exponents.
However, the Φ terms increase too rapidly as shown in Figure 3.
Terms such as Frobenius product (Golowich et al., 2018) and Fischer-Rao (Liang et al., 2019) are effective for the SL depth case, but are both ineffective in the LQR depth case.
For width, the only product which is effective is the nuclear norm product.
Figure
We show that the vs noise dimension is tight as the noise dimension increases, showing that this bound is accurate
Generalization Gap vs
Number of Intermediate Layers.
We plotted different Φ = (cid:80) ji =0 (cid:107) A (cid:107) ∗ (cid:107) A (cid:107) terms without exponents, as powers of those terms are monotonic transforms since (cid:107) A (cid:107) ∗ (cid:107) A (cid:107) ≥ 1 ∀ A and (cid:107) A (cid:107) ∗ =
(cid:107) A (cid:107) F , (cid:107) A (cid:107) 1 .
We see that the naive spectral bound diverges at 2 layers, and the weight-counting sums are too loose
[Figure]
3. P ROJECTED G YM E NVIRONMENTS
In Section 3.1, we ﬁnd that observational overﬁtting exists and overparametrization potentially helps in the linear setting.
In order to analyze the case when the underlying dynamics are nonlinear, we let M be a classic Gym environment and we generate a M θ =
( M , w θ ) by performing the exact same ( f, g ) -scheme as the LQR case, i.e. sampling θ to produce an observation function w θ ( s ) =
(cid:20) W constant
W θ (cid:21) s .
We again can produce training/test sets of MDPs by repeatedly sampling θ , and for policy optimization, we use Proximal Policy Gradient (Schulman et al., 2017).
Although bounds on the smoothness term R θ ( π ) − R θ ( π (cid:48) ) affects upper bounds on Rademacher Complexity (and thus generalization bounds), we have no such theoretical guarantees in the Mujoco case as it is difﬁcult to analyze the smoothness term for complicated transitions such as Mujoco’s physics simulator.
However, we can observe empirically that the underlying state dynamics has a signiﬁcant effect on generalization performance as the policy nontrivially increased test performance such as in CartPole-v1 and Swimmer-v2, while it could not for others.
This suggests that the Rademacher complexity and smoothness on the reward function vary highly for different environments.
Even though it is common practice to use basic (2-layer) MLPs in these classic benchmarks, there are highly nontrivial generalization effects from modifying on this class of architectures.
Our results show increasing width and depth for basic MLPs can increase generalization and is signiﬁcantly dependent on the choice of activation, and other implicit regularizations such as using residual layers can also improve generalization.
Speciﬁcally, switching between ReLU and Tanh activations produces different results during overparametrization.
For instance, increasing Tanh layers improves generalization on CartPole-v1, and width increase with ReLU helps on Swimmer-v2.
Tanh is noted to consistently improve generalization performance.
However, stacking Tanh layers comes at a cost of also producing vanishing gradients which can produce subpar training performance, for e.g. HalfCheetah.
To allow larger depths, we use ReLU residual layers, which also improves generalization and stabilizes training.
Previous work (Zhang et al., 2018c) did not ﬁnd such an architectural pattern for GridWorld environments, suggesting that this effect may exist primarily for observational overﬁtting cases.
While there have been numerous works which avoid overparametrization on simplifying policies (Rajeswaran et al., 2017; Mania et al., 2018) or compactifying networks (Choromanski et al., 2018; Gaier &
Ha, 2019), we instead ﬁnd that there are generalization beneﬁts to overparametrization even in the nonlinear control case.
3. D ECONVOLUTIONAL P ROJECTIONS
From the above results with MLPs, one may wonder if similar results may carry to convolutional networks, as they are widely used for vision-based RL tasks.
As a ground truth reference for our experiment, we the canonical networks proven to generalize well in the dataset CoinRun, which are from worst to best, NatureCNN Mnih et al.
(2013), IMPALA Espeholt et al.
(2018), and IMPALALARGE (IMPALA with more residual blocks and higher convolution depths), which have respective parameter numbers (600K, 622K, 823K).
We setup a similar ( f, g ) -scheme appropriate for the inductive bias of convolutions, by passing the vanilla Gym 1D state corresponding to joint locations and velocities, through multiple deconvolutions.
We do so rather than using the RGB image from env.render() to enforce that the actual state is indeed low dimensional and minimize complications in experimentation, as e.g. inference of velocity information would require frame-stacking.
Speciﬁcally in our setup, we project the actual state to a ﬁxed length, reshaping it into a square, and replacing f and g θ both with the same orthogonally-initialized deconvolution architecture to each produce a 84 × 84 image (but g θ ’s network weights are still generated by θ 1 , ..., θ m similar to before).
We combine the two outputs by using one half of the ”image” from f , and one half from g θ , as shown back in Figure 2.
Figure 7 shows that the same ranking between the three architectures exists as well on the GymDeconv dataset.
We show that generalization ranking among NatureCNN/IMPALA/IMPALALARGE remains the same regardless of whether we use our synthetic constructions or CoinRun.
This suggests that the RL generalization quality of a convolutional architecture is not limited to real world data , as our test purely uses numeric observations, which are not based on a human-prior.
From these ﬁndings, one may conjecture that these RL generalization performances are highly correlated and may be due to common factors.
One of these factors we suggest is due to implicit regularization.
In order to support this claim, we perform a memorization test by only showing g θ ’s output to the policy.
This makes the dataset impossible to generalize to, as the policy network cannot invert every single observation function { g θ 1 ( · ) , g θ 2 ( · ) , ..., g θ n ( · ) } simultaneously.
Zhang et al.
(2018c) also constructs a memorization test for mazes and grid-worlds, and showed that more parameters increased the memorization ability of the policy.
While it is intuitive that more parameters would incur more memorization, we show in
Figure 7 (bottom) that this is perhaps not a complete picture when implicit regularization is involved.
Using the underlying MDP as a Swimmer-v2 environment, we see that NatureCNN, IMPALA,
IMPALA-LARGE have reduced memorization performances.
IMPALA-LARGE, which has more depth parameters and more residual layers (and thus technically has more capacity), memorizes less than IMPALA due its inherent inductive bias.
Another memorization test where an LQR is used as underlying MDP is shown in Appendix A.1.2, with similar results, supporting the hypothesis that these extra residual blocks may be implicitly regularizing the network.
This is corroborated by the fact that residual layers are also explained as an implicit regularization technique (Neyshabur, 2017) for SL.
O VERPARAMETRIZATION IN C OIN R UN
We test our hypothesis from the above to the CoinRun benchmark, using unlimited levels for training.
For MLP networks, we downsized CoinRun from native 64 × 64 to 32 × 32 , and ﬂattened the 32 × 32 × 3 image for input to an MLP.
Two signiﬁcant differences from previous cases are that 1.
inherent dynamics are changing per level in CoinRun, and 2.
the relevant and irrelevant CoinRun features change locations across the 1-D input vector.
Regardless, we show that overparametrization can still improve generalization in this more realistic RL benchmark, much akin to Neyshabur et al.
(2018b) which showed that overparametrization for MLPs improved generalization on 32 × 32 ×
CIFAR-10.
While we also extend the case of large-parameter convolutional networks using ImageNet networks in Appendix A.2.1, an important question is how to predict the generalization gap only from the training phase.
A particular set of metrics, popular in the SL community are margin distributions (Jiang et al., 2018; Bartlett et al., 2017), as they deal with the case for softmax outputs which do not explicitly penalize the weight norm of a network, by normalizing the ”conﬁdence” margin of the logit outputs.
While using margins on state-action pairs (from an on-policy replay buffer) is not technically rigorous, one may be curious to see if they have predictive power, especially as MLPs are relatively simple to norm-bound.
We plotted these margin distributions in Appendix A.2.2, but found that the weight norm bounds used in SL are simply too dominant for this RL case.
This, with the bound results found earlier for the LQR case, suggests that current norm bounds are simply too loose for the RL case even though we have shown overparametrization helps generalization in RL, and hopefully this motivates more of the study of such theory.
C ONCLUSION
We have identiﬁed and isolated a key component of overﬁtting in RL as the particular case of “observational overﬁtting”, which is particularly attractive for studying architectural implicit regularizations.
We have analyzed this setting extensively, by examining 3 main components:
The analytical case of LQR and linear policies under exact gradient descent, which lays the foundation for understanding theoretical properties of networks in RL generalization.
The empirical but principled Projected-Gym case for both MLP and convolutional networks which demonstrates the effects of neural network policies under nonlinear environments.
moving
empirically, MLP overparametrization also improves generalization
We noted that current network policy bounds using ideas from SL are unable to explain overparametrization effects in RL, which is an important further direction.
In some sense, this area of RL generalization is an extension of static SL classiﬁcation from adding extra RL components.
For instance, adding a nontrivial “combination function” between f and g θ that is dependent on time (to simulate how object pixels move in a real game) is both an RL generalization issue and potentially video classiﬁcation issue, and extending results to the memory-based RNN case will also be highly beneﬁcial.
Extending the analysis to off-policy methods such as Q-learning and also
ES-based methods is also important.
We believe that this work provides an important initial step towards solving these future problems
A. F ULL P LOTS FOR LQR, FG -G YM -MLP, FG -G YM -D ECONV A.1. LQR
[Figure]
[Figure] [Figure]
We further verify that explicit regularization (norm based penalties) also reduces generalization gaps.
However, explicit regularization may be explained due to the bias of the synthetic tasks, since the ﬁrst layer’s matrix may be regularized to only ”view” the output of f
[Figure]
especially as regularizing the ﬁrst layer’s weights substantially improves generalization
[Figure]
[Figure]
A. E XTENDED L ARGE RL R ESULTS A.2. L ARGE I MAGE N ET M ODELS FOR C OIN R UN
We experimentally verify in table (1) that large ImageNet models perform very differently in RL than during SL.
We note that default network with the highest test reward was IMPALA-LARGE-BN (IMPALA-LARGE, with Batchnorm) at ≈ 5 .
5 test score.
In order to verify that this is inherently a feature learning problem rather than a combinatorial problem involving objects, such as in (Santoro et al., 2018), we show that state-of-the-art attention mechanisms for RL such as Relational Memory Core (RMC) using pure attention on raw 32 ×pixels does not perform well here, showing that a large portion of generalization and transfer must be based on correct convolutional setups.
Architecture
Coinrun
[Table]
(Train,Test)
AlexNet
[Table]
v (
[Table] [Table]
CifarNet (
[Table] [Table]
IMPALA
[Table]
LARGE
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
MobileNet-
[Table] [Table]
NASNet
[Table]
CIFAR (
[Table]
NASNet
[Table]
Mobile (
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
RMC32x (
[Table]
ShakeShake (
[Table] [Table]
VGG
[Table] [Table]
VGG
[Table] [Table]
Images scaled to default image sizes ( 32 × 32 or 224 × 224 ) depending on network input requirement.
See Appendix
A5 for training curves.
We provide the training/testing curves for the ImageNet/large convolutional models used.
Note the following:
RMC32x32 projects the native image from CoinRun from 64 × 64 to 32 × 32 , and uses all pixels as components for attention, after adding the coordinate embedding found in (Santoro et al., 2018).
Optimal parameters were (mem slots = 4, head size = 32, num heads = 4, num blocks = 2, gate style = ’memory’).
Auxiliary Loss in ShakeShake was not used during training, only the pure network.
VGG-A is a similar but slightly smaller version of VGG-16.
A.2. D O S TATE -A CTION M ARGIN D ISTRIBUTIONS P REDICT G ENERALIZATION IN RL?
A conceptual difference between CoinRun and our other tasks is due to the discrete action space of CoinRun.
We verify in Figure A6, that indeed, simply measuring the raw norms of the policy network is a poor way to predict generalization, as it generally increases even as training begins to plateau.
This is inherently because the softmax on the logit output does not penalize arbitrarily high logit values, and hence proper normalization is needed.
We are curious in measuring the margin distribution of action logits, as this has been used extensively to empirically predict the generalization properties of classiﬁers (Jiang et al., 2018; Bartlett et al., 2017).
For a policy, the the margin distribution will be deﬁned as ( s, a ) →
[Equation]
, where F π ( s ) a is the logit value of action a given input s , before the softmax, and S is the matrix of states in the replay buffer, and R π is the norm-based Lipschitz bound on the policy network logits.
We used the Spectral, Sharpness and Bartlett bounds, for R π , and we replace the classical supervised learning pair ( x, y )
=
( s, a ) with the state action pairs found on-policy.
We used the following metrics R π (after removing irrelevant constants)
Bartlett Bound: (cid:16)(cid:81) di
=1 (cid:107) W
i (cid:107) (cid:17) (cid:18)(cid:80) di =1 (cid:107) W i (cid:107) 2 / 3 1 (cid:107) W i (cid:107) 2 / 3 (cid:19)
[Equation]
(Bartlett et al.
[Equation]
Sharpness Metric: (cid:114) (cid:80) di
=1 (cid:107)
[Equation]
m
[Equation]
Keskar et al
[Equation] [Equation]
3. Spectral Bound:
(cid:115)
[Table]
+
[Table]
( 6 mδ ) m (Neyshabur et al., 2018a)
Unlike the other metrics mentioned, the margin distribution converges to a ﬁxed distribution even long after training has plateaued.
However, unlike SL, the margin distribution is conceptually not fully correlated with RL generalization on the total reward, as a policy overconﬁdent in some state-action pairs does not imply bad testing performance.
This correlation is stronger if there are Lipschitz assumptions on state-action transitions, as noted in (Wang et al., 2019).
For empirical datasets such as CoinRun, a metric-distance between transitioned states is ill-deﬁned however.
Nevertheless, the distribution over the on-policy replay buffer at each policy gradient iteration is a rough measure of overall conﬁdence.
We note that there are two forms of modiﬁcations, network dependent (explicit modiﬁcations to the policy - norm regularization, dropout, etc.) and data dependent (modiﬁcations only to the data in the replay buffer - action stochasticity, data augmentation, etc.).
Ultimately however, we ﬁnd that current norm bounds R π become too dominant in the fraction, leading to the monotonic decreases in the means of the distributions as we increase parametrization.
A.2. G YM -R ETRO (S ONIC )
In the Gym-Retro benchmark (Sonic), the agent is given 47 training levels with rewards corresponding to increases in horizontal location.
The policy is trained until 5k reward.
At test time, 11 unseen levels are partitioned into starting positions, and the rewards are measured and averaged.
We brieﬂy mention that the agent strongly overﬁts to the scoreboard (i.e. an artifact correlated with progress in the level), which may be interpreted as part of the output of g θ ( · ) .
In fact, the agent is still able to train to 5k reward from purely observing the timer as the observation.
By blacking out this scoreboard with a black rectangle, we see an increase in test performance.
[Table]
A. H YPERPARAMETERS AND E XACT S ETUPS A.3. E XACT INFINITE LQR
For inﬁnite horizon case, see (Fazel et al., 2018) for the the full solution and notations.
Using the same notation ( A, B, Q, R ) , denote C ( K ) =
(cid:80) x 0 ∼D x T
P K x 0 as the cost and u t = − Kx t as the policy, where P K satisiﬁes the inﬁnite Algebraic-Ricatti equation:
[Equation]
We may calculate the precise LQR cost by vectorizing (i.e. ﬂattening) both sides’ matrices and using the Kroncker product ⊗ , which leads to a linear regression problem on P K , which has a precise solution, implementable in TensorFlow:
[Equation]
[Table]
A Uniform Random from set of orthogonal matrices on n × n , scaled 0 .
[Equation]
P ROJECTION M ETHOD
The basis for producing f, g θ outputs is due to using batch matrix multiplication operations, or ”BMV”, where the same network architecture uses different network weights for each batch dimension, and thus each entry in a batchsize of B will be processed by different network weights.
This is to simulate the effect of g θ i - The numeric ID i of the environment is used as an index to collect a specﬁcic set of network weights θ i from a global memory of network weights (e.g. using tensorflow.gather ).
We did not use nonlinear activations for the BMV architectures, as they did not change the outcome of the results.
Architecture
[Table]
A.3. I MAGE N ET M ODELS
For the networks used in the supervised learning tasks, we direct the reader to the following repository: https://github.com/tensorflow/models/blob/master/research/ slim/nets/nets_factory.py .
We also used the RMC: deepmind/sonnet/blob/ master/sonnet/python/modules/relational_memory.py
A.3. PPO P ARAMETERS
For the projected gym tasks, we used for PPO2 Hyperparameters:
[Table]
See (Cobbe et al., 2018) for the default parameters used for CoinRun.
We only varied nminibatches in order to ﬁt memory onto GPU.
We also did not use RNN additions, in order to measure performance only from the feedforward network - the framestacking/temporal aspect is replaced by the option to present the agent velocity in the image.
A. T HEORETICAL R ESULTS (LQR)
See (Fazel et al., 2018) for more extensive LQR notation and statements that we will use.
Below proofs of certain overﬁtting properties in the LQR case, which give more rigorous bounds.
A.4. N OTATION AND S ETTING
Let (cid:107)·(cid:107) be the spectral norm of a matrix (i.e. largest singular value).
Suppose C ( K ) was the inﬁnite horizon cost for an ( A, B, Q, R ) -LQR where action a t = − K
x t , x t is the state at time t , state transition is x t +1 =
A · x t + B · a t , and timestep cost is x Tt Qx t + a Tt Ra t .
C ( K ) for an inﬁnite horizon LQR, while known to be non-convex, still possess the property that when ∇ C ( K ∗ ) = 0 , K is a global minimizer, or the problem statement is rank deﬁcient.
By varying the observation projections, θ generates a population of C θ ( K ) cost functions with all of the population having the same minimizer K ∗ .
A.4.1. S MOOTHNESS B OUNDS
As described in Lemma 16 of (Fazel et al., 2018), we deﬁne
[Equation] [Equation]
and (cid:107) T K (cid:107) =
sup X T K ( X ) (cid:107) X (cid:107) over all non-zero symmetric matrices X .
Lemma 27 of (Fazel et al., 2018) provides a bound on the difference C ( K (cid:48) )
− C ( K ) for two different policies K, K (cid:48) when LQR parameters A, B, Q, R are ﬁxed.
During the derivation, it states that when
[Equation]
then
[Equation]
where P K =
(cid:13)(cid:13) T K ( Q + K T RK ) (cid:13)(cid:13) , and (cid:107) P K (cid:48)
− P K (cid:107) is bounded by the sum of two terms, which are
[Equation]
K
[Equation]
Thus we have the bound
[Equation]
Lemma 17 also states that:
[Equation]
8) where
[Equation]
Assuming that in our problem setup, x 0 , Q, R, A, B were ﬁxed, this means many of the parameters in the bounds are constant, and thus we conclude:
[Equation]
Since (cid:107) A − BK (cid:107) ≤ 1 or else T K ( X ) is inﬁnite and in this scheme
O ( (cid:107) K (cid:48) − K (cid:107) )
=
O ( (cid:107) K (cid:107) )
, we thus ﬁnally collect the terms to get the bound we will use in the next sections:
[Equation]
[Equation]
A.4. O BSERVATIONAL P ROJECTIONS
Let C ( · ) be the cost function S → R for a policy acting on the state space, and without loss of generality, normalize the constants in (11) so that C ( K (cid:48) ) − C ( K ) ≤ (cid:107) K (cid:48) − K (cid:107) 3 .
In the observational projection case, we note that an observation of Ws with policy K is exactly the same as the case when the observation is s and the policy is KW .
In our experiments, a semi-orthogonal W is sampled from combining two samples from f and g and normalizing.
The experiments allowed
W obs to be a matrix of size ( d obs , d state ) , and K size ( d action , d obs ) .
Using (11), we see that
[Equation]
The dominant term in this expression is (cid:107) K (cid:107) .
We can examine how much it scales as a function of the dimension d obs , since for any random K since for any K , we can form an LQR with K as optimal policy, which means random matrix theory can provide insights into the distribution of (cid:107) KW (cid:107) .
It is established (Rudelson & Vershynin, 2010) that for random matrices of such dimensions d obs (cid:29) d action , (cid:107) K (cid:107) scales with the dimension, with growth bound of O ( d 1 / 3 obs ) .
A.4. G ENERALIZATION B OUNDS FOR LQR LINEAR CASE
We can think of the semi-orthogonal samples W 1 , W 2 , ... as the “randomly sampled datapoints” from a distribution D obs analogous to supervised learning.
If we ﬁx A, B, Q, R , then we may write our cost function in LQR as C i ( K ) =
C ( KW ) .
Note that C ( · ) ∈ [0 , M ] for some maximal value M based on A, B, Q, R, x 0 .
Then, for a ﬁxed K , and drawing inﬁnite samples of W i , we deﬁne the following:
[Equation]
Optimizing this inﬁnite sample case is optimizing the ”true cost function”.
However, if we have ﬁnite samples S m = { W 1 , ..., W m } , we can deﬁne the average sample cost as
[Equation]
We apply the standard proof of generalization gap: From deﬁnition of supremum, for a ﬁxed K ∈ K denote
[Equation]
Denote random variable ψ ( S ) =
sup K max ∈K (cid:16) C D obs ( K max ) − (cid:98) C S m ( K max ) (cid:17) .
We need to understand how much ψ changes as a result of changing the samples W i .
A common approach to forming generalization gap bounds comes from the McDiarmid inequality:
Suppose K is ﬁxed.
If ψ ( S ) satisﬁes:
[Equation]
Then
[Equation]
Suppose that we only changed one of the samples W i .
Then we can use our main equation (11) above, to get
[Equation]
Plugging this in, we then get with probability at least 1 − δ ,
[Equation]
Note that this equation (A.4.3) essentially is the main term found in Theorem 3.1.
To bound E S [ ψ ( S )] from equation A.4.3, we use the standard deﬁnition of Rademacher complexity:
[Equation]
where Rademacher complexity is deﬁned for our case as:
[Equation]
To ease on notation
[Equation]
assume sup K = sup K ∈K .
We use the following technique:
[Equation]
which implies that after unrolling the induction step m times,
[Equation] [Equation]
where the previous equality holds since K is a convex set, and thus the set of all possible differences
K − K = K .
This is upper bounded by:
[Equation]
where the last equality follows since (cid:107) W i (cid:107)
= 1 .
We note that if (cid:80) mi =1 σ i < 0 , then the optimum K satisﬁes (cid:107) K (cid:107) = 0 , and otherwise (cid:107) K (cid:107) 3 is maximized - abusing notation slightly, let sup K (cid:107) K (cid:107)
= (cid:107)K(cid:107) 3 .
Hence the previous term from (27) is upper bounded by:
[Equation] [Equation]
where the last equation follows from a well known property of Rademacher variables, which then follows that R m ( K ) ≤ O ( √ m )
m
=
O
[Equation]
Hence it follows that gathering all terms, we have ﬁnally:
[Equation]
Since from A.4.2, (cid:107) K (cid:107) ∼ O ( d 1 / 3 obs ) , Theorem 3.1 presented from the main section follows.