L OCAL F EATURE S WAPPING FOR G ENERALIZATION IN R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Over the past few years, the acceleration of computing resources and research in deep learning has led to signiﬁcant practical successes in a range of tasks, including in particular in computer vision.
Building on these advances, reinforcement learning has also seen a leap forward with the emergence of agents capable of making decisions directly from visual observations.
Despite these successes, the over-parametrization of neural architectures leads to memorization of the data used during training and thus to a lack of generalization.
Reinforcement learning agents based on visual inputs also suffer from this phenomenon by erroneously correlating rewards with unrelated visual features such as background elements.
To alleviate this problem, we introduce a new regularization technique consisting of channel-consistent local permutations (CLOP) of the feature maps.
The proposed permutations induce robustness to spatial correlations and help prevent overﬁtting behaviors in RL.
We demonstrate, on the OpenAI Procgen Benchmark, that RL agents trained with the CLOP method exhibit robustness to visual changes and better generalization properties than agents trained using other state-of-the-art regularization techniques.
We also demonstrate the effectiveness of CLOP as a general regularization technique in supervised learning.
I NTRODUCTION
Advances made in deep learning have opened the way to many applications in computer vision such as classiﬁcation, object recognition, or image segmentation.
The powerful representation capabilities of deep neural networks paved the way for many successes in deep reinforcement learning with the design of agents able to take decisions directly from pixels (Mnih et al., 2013; 2015).
However, the sensitivity of neural networks to the distribution of training data strongly affects their generalization abilities.
Neural networks are intrinsically designed to memorize the data they are trained upon, since their ﬁtting implies empirical risk minimization (Vapnik, 1992) (they minimize the empirical average prediction error over a large training dataset).
Therefore, they are prone to prediction errors on unseen samples.
RL agents also suffer from this handicap and tend to memorize training trajectories, rather than general skills and features leading to transferable policies.
This phenomenon, usually known as overﬁtting, takes a double sense in RL.
Generalization in RL implies the ability to generalize across states (as in supervised learning), but also across environments.
It is only recently that several environments with different conﬁgurations for training and testing have emerged and received a lot of attention (Nichol et al., 2018; Justesen et al., 2018;
Zhang et al., 2018a; Cobbe et al., 2019; 2020), shedding light on the generalization issue which remained mostly overlooked, and conﬁrming the poor generalization ability of current algorithms.
Strategies to achieve good generalization and avoid overﬁtting in deep learning fall into three categories of regularization: explicit regularization (e.g., loss penalization, weight decay), implicit regularization via the architecture and optimization (e.g., dropout, batch-normalization, batch size selection, momentum, early stopping), or implicit regularization by enhancement of the input data (data augmentation).
Direct application of these strategies to deep RL agents has demonstrated some improvements in agent generalization abilities in some environments, but much progress remains to be made to integrate RL systems in real-world applications.
In this work, we address the observational overﬁtting issue introduced in Song et al.
(2019) which considers a zero-shot generalization
RL setting.
An agent is trained on a speciﬁc distribution of environments (for example some levels of a platform video game) and tested on similar environments sharing the same high-level goal and dynamics but with different layouts and visual attributes (background or assets).
We argue that a structured stochastic permutation of features of an RL agent during training leads to state-of-theart generalization performance for vision-based policies.
We introduce an efﬁcient regularization technique based on Channel-consistent LOcal Permutations (CLOP) of the feature maps that mitigates overﬁtting.
We implement it as an intermediate layer in feed-forward neural networks, and demonstrate its effectiveness on several reinforcement and supervised learning problems.
This paper is organized as follows.
Section 2 presents the necessary background on generalization in supervised learning and reinforcement learning.
Section 3 reviews recent work in the literature that allows for a critical look at our contribution and put it in perspective.
Section 4 introduces the CLOP technique and the corresponding layer.
Section 5 empirically evaluates agents using
CLOP against state-of-the-art generalization methods and discusses their strengths, weaknesses, and variants.
Section 6 summarizes and concludes this paper.
W HAT IS GENERALIZATION ?
Generalization in supervised learning (SL).
Let X be an input space of descriptors and Y an output space of labels.
A SL problem is deﬁned by a distribution p ( x, y ) of elements of X × Y , and a loss
L (ˆ y, y ) which measures how different ˆ y ∈ Y and y ∈ Y are.
Then, for a given function f intended to capture the mapping from X to Y underlying the p distribution, one deﬁnes the (expected) risk as R ( f ) =
E p
[ L ( f ( x ) , y )] .
Since the true p is generally unknown, one cannot directly minimize the risk in search for the optimal f .
Given a training set S
= { ( x i , y i ) } ni =1 of n items in X × Y drawn i.i.d.
according to p ( x, y ) , empirical risk minimization (Vapnik, 1992) seeks to minimize
R S ( f ) =
1 /n (cid:80) ni
=
[ L ( f ( x i ) , y i )] .
The ability of f to generalize to unseen samples is then deﬁned by the generalization gap R ( f ) − R S ( f ) , which is often evaluated by approaching R ( f ) as the empirical risk R T ( f ) over a test set T = { ( x i , y i ) } n (cid:48)
i =1 also drawn i.i.d.
from p ( x, y ) .
Closing the generalization gap can be attempted through structural risk minimization, which modiﬁes R S so as to include a regularization penalty into the optimization, or more generally by the introduction of inductive biases (Mitchell, 1980).
Reinforcement learning (RL).
RL (Sutton & Barto, 2018) considers the problem of learning a decision making policy for an agent interacting over multiple time steps with a dynamic environment.
At each time step, the agent and environment are described through a state s ∈ S , and an action a ∈ A is performed; then the system transitions to a new state s (cid:48)
according to probability T ( s (cid:48) | s, a ) , while receiving reward R ( s, a ) .
The tuple M = ( S , A , T, R ) forms a Markov Decision Process (Puterman, 2014, MDP), which is often complemented with the knowledge of an initial state distribution p 0 ( s ) .
A decision making policy parameterized by θ is a function π θ ( a | s ) mapping states to distributions over actions.
Training a reinforcement learning agent consists in ﬁnding the policy that maximizes the discounted expected return: J ( π θ ) =
E [ (cid:80) ∞ t =0 γ t R ( s t , a t )] .
Generalization in RL.
Departing from the rather intuitive deﬁnition of generalization in SL, the idea of generalization in RL may lead to misconceptions.
One could expect a policy that generalizes well to perform well across environments .
It seems important to disambiguate this notion and stress out the difference between generalization in the SL sense (which is a single MDP problem), and domain generalization or robustness.
For instance, one could expect a policy that learned to play a certain level of a platform game to be able to play on another level, provided the key game features (e.g. platforms, enemies, treasures) remain similar enough and the game dynamics are the same.
This type of generalization benchmark is typically captured by procedurally generated environments, such as Procgen (Cobbe et al., 2020).
Finding a policy that yields a guaranteed minimal performance among a set of MDPs (sharing common state and action spaces) is the problem of solving Robust MDPs (Iyengar, 2005).
Similarly, that of policy optimization over a distribution over MDPs (sharing common states and actions too) is that of domain generalization (Tobin et al., 2017).
We argue that these problems are unrelated and much harder than vanilla generalization in
RL.
Robust MDPs induce a max π min T,R problem, domain generalization is a max π E T,R one, while vanilla generalization as understood in SL remains a max π problem that we try to solve for all states within a single MDP (and not only the ones in the training set).
Consequently, the ability to generalize in RL is a broader notion than structural risk minimization, which, unfortunately, uses the same name of generalization.
In a given game, ﬁnding a policy that plays well on all levels is still solving the very same MDP.
The underlying transition and reward models are the same and only the parts of the state space explored in each level are different.
Note that these explored subsets of the state space may have non-empty intersections (e.g. the ﬁnal part of two different levels might be common) and the optimal distribution over actions for a given state is unique.
In this work we focus on generalization as the problem of preventing overﬁtting (and thus reducing the generalization gap) within a single MDP, which we call observational overﬁtting (Song et al., 2019).
Observational overﬁtting (OO) in RL.
The inability to generalize might be caused by overﬁtting to the partially explored environment dynamics (Rajeswaran et al., 2017), or to some misleading signal that is correlated with progress but does not generalize to new levels (Machado et al., 2018;
Song et al., 2019).
Therefore, preventing OO in RL remains the ability to generalize across states, within the same MDP.
Beyond the ability of RL agents to memorize good actions in explored states, it boils down to their ability to capture rules that extend to unencountered states, just as in structural risk minimization.
As in SL, many policies might ﬁt the observed data, but few might generalize to the true mechanisms of the underlying MDP.
Song et al.
(2019) propose to capture OO through a framework where a unique latent state space is transformed into a variety of observation spaces.
Observation functions are built by combining useful features with purely decorative and unimportant ones which vary from one observation function to the next.
They suppose the observation functions are drawn from a certain distribution (over procedural generation parameters for instance) and deﬁne the corresponding distribution over MDPs.
In turn, they deﬁne the risk and the generalization gap with respect to this distribution over MDPs.
We slightly depart from their derivation and argue this distinction is unnecessary: what is being solved is the unique MDP deﬁned by the underlying dynamics and the projection of latent states into observations.
The risk is thus deﬁned with respect to the distribution over observations induced by the distributions over initial states and observation functions.
Overall, this allows capturing the problem of generalization gap minimization in RL and underpins the developments proposed in the remainder of this paper.
R ELATED WORK
In the following paragraphs, we cover essential works which aim to improve the generalization abilities of neural networks in both supervised learning and reinforcement learning.
In supervised learning, the process of modifying a learning algorithm with the objective to reduce its test error while preserving its train error is known as regularization (Goodfellow et al., 2016).
Direct, or explicit, regularization can be achieved by adding a regularizer term into the loss function, such as an L2 penalty on networks parameters (Plaut et al., 1986; Krogh & Hertz, 1992).
A second, implicit, regularization strategy consists in feature-level manipulations like dropout (Srivastava et al., 2014), drop-connect (Wan et al., 2013) or batch-normalization (Ioffe & Szegedy, 2015).
Regularization can also be achieved implicitly by directly augmenting the training data with perturbations such as adding Gaussian noise (Bishop, 1995), or, in the case of visual inputs, random cropping and ﬂipping (Krizhevsky et al., 2012; Szegedy et al., 2017) or removing structured parts of the image (DeVries & Taylor, 2017).
Another efﬁcient augmentation strategy, called label smoothing (Szegedy et al., 2016), consists in penalizing overconﬁdent predictions of neural networks by perturbing the labels.
Combining perturbation of both inputs and outputs, Zhang et al.
(2018c)
; Yun et al.
(2019), produce synthetic data and labels using two different samples and their corresponding labels.
Many studies have highlighted the limited ability of RL agents to generalize to new scenarios (Farebrother et al., 2018; Packer et al., 2018;
Zhang et al., 2018b; Song et al., 2019; Cobbe et al., 2019).
Using saliency maps, Song et al.
(2019) exhibit that RL agents trained from pixels in environments with rich and textured observations, such as platform video games (e.g. Sonic (Nichol et al., 2018)), focus on elements of the scenery correlated with in-game progress but which lead to poor generalization in later situations.
One of the identiﬁed counter-measures to overﬁtting in RL consists in applying standard methods in supervised learning.
Cobbe et al.
(2019; 2020) demonstrated the contribution of classical supervised learning regularization techniques like weight decay, dropout, or batch-normalization to generalization in procedurally generated environments.
Similar to dataaugmentation in supervised learning, Raileanu et al.
Laskin et al.
Yarats et al.
(2020) apply visual-data-augmentation on the observations provided by the environment to train robust agents.
Igl et al.
(2019) use a selective noise injection and information bottleneck to regularize their agent.
Wang et al.
(2020) propose mixreg, a direct application of mixup (Zhang et al., 2018c) in
RL, combining two randomly sampled observations, and training the RL agent using their interpo- lated supervision signal.
Lee et al.
(2020) use a random convolution layer ahead of the network architecture, to modify the color and texture of the visual observations during training.
Tobin et al.
(2017) tackle the sim-to-real problem, using domain randomization on visual inputs to bridge the gap between simulation and reality in robotics.
Raileanu & Fergus (2021) dissociate the optimization process of the policy and value function represented by separate networks and introduce an auxiliary loss that encourages the representation to be invariant to task-irrelevant properties of the environment.
Another recent strategy consists in learning representations that are invariant to visual changes.
Higgins et al.
(2017) use a two-stage learning process: ﬁrst they extract disentangled representations from random observation and then they exploit these representations to train an RL agent.
Zhang et al.
(2020) use bisimulation metrics to quantify behavioral similarity between states and learn robust task-relevant representations.
Wang et al.
(2021b) extract, without supervision, the visual foreground to provide background invariant inputs to the policy learner.
Our work focuses on the OO situations, where an agent overﬁts visual observations features that are irrelevant to the latent dynamics of the MDP.
Our method approaches the augmentation of data by noise injection at the feature level, thus avoiding the computationally costly operations of highdimensional image transformation in regular data augmentation.
By directly modifying the encountered features’ spatial localization, the CLOP layer aims to remove the correlations between spurious features and rewards.
C HANNEL - CONSISTENT L OCAL PERMUTATION LAYER
Three key intuitions underpin the proposal of the channel-consistent local permutation (CLOP) layer.
1) In many image-based decision making problems, the important information is often spatially scarce and very localized .
The example of platform games is typical: most of the pixels are decorrelated from the MDP’s objective.
But this also applies to other games (e.g., driving, shooting) or environments such as image-based object manipulation.
Consequently, most pixels don’t convey useful information for an optimal policy and data augmentation should help disambiguate between informative and uninformative pixels.
Speciﬁcally, we can expect that an image and its altered counterpart share the same reward value for the same action.
Even if perturbing an image might mistakenly associate the wrong action to it, in most cases we expect noise injection in images to be beneﬁcial for generalization since it creates synthetic images where most of the perturbed features were unimportant in the ﬁrst place.
We refer to the
process of
modifying
a
C
× H ×
W image as
data augmentation
, whether this image
is
an input image or a set of feature
maps produced by the
C
ﬁlters of a convolutional layer
We argue that data augmentation in the latent space is more efﬁcient for generalization in decision making than in the input space .
For example, it seems more useful for generalization to learn a rule stating that an enemy sprite’s position is important for the action choice, than to learn this same rule based on independent pixels, which might lead to OO.
The successive convolutional layers provide greater levels of abstraction on each image: the deepest layer is the highest level of abstraction and forms a latent description space of the input images, with abstract features, like whole-object positions.
Consequently, to achieve generalization in RL, data augmentation should probably be done in the latent space, on the feature maps after the deepest convolutional layer, rather than in the input space.
3) Good actions in a state probably remain good actions in closeby states in the latent space .
For example, if one has determined that jumping is a good action in a speciﬁc image, then it is likely that if the image’s objects move very locally, jumping remains a good action in the corresponding new images.
This idea is related to that of Rachelson & Lagoudakis (2010) who study the locality of action domination in the state space.
Consequently, we conjecture that local shifts of features in the deepest feature maps (hence representing high-level concepts and objects, rather than raw pixels) might help generate synthetic samples that help the optimization process disambiguate between useful and unimportant information.
In turn, we expect this data augmentation to help prevent the bare memorization of good trajectories and instead generalize to good decision making rules.
The CLOP technique transposes these intuitions into convolutional networks.
We implement it as a regularization layer that we introduce after the deepest convolutional layer’s feature maps.
During training, it swaps the position of pixels in these feature maps, while preserving the consistency across channels, so as to preserve the positional consistency between feature maps.
This swapping is performed sequentially in a random order among pixels, locally (each pixel is swapped with one input: X of shape ( N, C, H, W ) if training mode then
P = shufﬂe ( { ( h, w ) } h ∈ [1 ,H ] ,w ∈ [1 ,W ] ) for ( h, w ) in P ( h (cid:48) , w (cid:48) )
← draw a direct neighbor of ( h, w )
With proba α , swap ( X [: , : , h, w ] , X [: , : , h (cid:48) , w (cid:48) ]) return X α =α = 0 .α
= 0 .α = 0 .α
=
[Figure]
.α
= neighbors), and with probability α .
At evaluation time, the CLOP layer behaves like the identity function.
Figure 1 illustrates this process and presents the pseudo-code.
Since pixels are taken in a random order, a given pixel value might be “transported” far from its original position, with a small probability.
Figure 2 illustrates the effect of varying values of α on the output of the layer.
Since CLOP is applied after the deepest convolutional layer, each of the pixels in this image should be interpreted as a salient high-level feature in the input image, and the channels correspond to different ﬁlters.
CLOP shufﬂes these descriptors while preserving channel consistency (no new colors are created in Figure 2) and neighborhoods (objects remain close to each other).
E XPERIMENTAL RESULTS AND DISCUSSION
This section ﬁrst exhibits the CLOP method’s effect on regularization in supervised learning and then demonstrates its efﬁciency in reinforcement learning.
Hyperparameters, network architectures, and implementation choices are summarized in Appendix A and B.
5. S UPERVISED LEARNING
To assess the contribution of the CLOP layer in supervised learning, we ﬁrst train a simple network (three convolutional layers followed by three linear layers) on the MNIST dataset (LeCun et al., 1998) and evaluate its generalization performance on the USPS (Hull, 1994) dataset, a slightly different digit classiﬁcation dataset.
Then, to conﬁrm this performance evaluation on large-scale images, we train a VGG11 network (Simonyan & Zisserman, 2015) using the Imagenette dataset,a subset of ten classes taken from Imagenet (Deng et al., 2009).
In both experiments, the networks are trained using four conﬁgurations: with dropout in the dense part of the network, with batch-normalization between convolutions, with a CLOP layer after the last convolutional layer, and without any regularization.
Results reported in Table 1 show that the CLOP layer allows the networks to generalize considerably better on USPS data than the unregularized network or when using dropout or batch-normalization.
Note that testing on USPS implies both generalization as deﬁned in
Section 2, and domain generalization (Wang et al., 2021a), that is the ability to generalize to other input data distributions at test time.
Likewise, applying the CLOP layer to a VGG11 improves generalization performance compared to the other methods, thus conﬁrming the intuition that localized permutations at the feature level during training may improve generalization at testing.
Appendix D features an extended discussion on the application of CLOP to other datasets (including Imagenet) and a comparison with mixup (Zhang et al., 2018c).
[Table] [Table]
5. R EINFORCEMENT LEARNING
We assess the regularization capability of CLOP on the Procgen benchmark, commonly used to test the generalization of RL agents.
Procgen is a set of 16 visual games, each allowing procedural generation of game levels.
All environments use a discrete set of 15 actions and provide 3 × 64 × 64 image observations to the agent.
Since all levels are procedurally generated, Procgen training and testing levels differ in many visual aspects like backgrounds, assets, or intrinsic level design (Figure 3).
CLOP within PPO.
In the following experiments, we train all agents with PPO (Schulman et al., 2017) and the hyperparameters recommended by Cobbe et al.
As in many PPO implementations, our actor and critic share a common stem of ﬁrst convolutional layers, extracting shared features.
Since the critic’s role is only to help estimate the advantage in states that are present in the replay buffer (in other words: since the critic won’t ever be used to predict values elsewhere than on the states it has been trained on), overﬁtting of the critic is not a critical issue in PPO.
An overﬁtting critic will provide an oracle-like supervision signal, which is beneﬁcial to the actor optimization process without inducing overﬁtting in the policy.
Therefore, we choose to append a CLOP layer only to the actor part of the agent’s network, located immediately after the shared feature extractor part (Appendix J discusses the application of CLOP of both actor and critic).
Comparison with classical supervised learning regularization methods.
We ﬁrst compare the performance of the data augmentation in the latent space performed by the CLOP layer with classical data augmentation methods that directly enhance inputs.
Following the setup of Cobbe et al.
(2020), we evaluate the zero-shot generalization of RL agents using the CLOP layer on three Procgen environments, setting the difﬁculty parameter to hard.
All agents were trained during 200M steps on a set of 500 training levels and evaluated on 1000 different test levels.
Figure 4 shows the average sum of rewards on the test levels obtained along training for all agents: the CLOP agents outperform classical data augmentation techniques (full results and discussion in Appendix G).
CLOP offers three signiﬁcant advantages: asymptotic generalization performance is better, sample complexity is smaller (the agent needs fewer interactions steps to reach high performance levels) and computa-
[Figure]
[Table]
tional complexity is negligible (see Appendix C).
Interestingly, other regularization methods based on pixel-block disturbance, such as cutout or cutout-color, despite being somehow related to the
CLOP layer but applied at the input image level, often even decrease the baseline performance.
The
CLOP layer also outperforms dropout, another feature-level manipulation method.
Compared to dropout, the CLOP layer randomly shifts features from their original position instead of completely removing them.
The persistence of these features in the feature map but in other positions appears to help to prevent spurious correlations with the reward signal.
Overall, empirical validation conﬁrms that the CLOP layer has a signiﬁcant impact on generalization, helping prevent OO by decorrelating features that are irrelevant to the task (such as background elements) from the policy’s output.
Comparison with RL regularization methods.
We also compare the CLOP layer with current state-of-the-art regularization methods speciﬁcally designed for RL: mixreg (Wang et al., 2020),
Rand+FM (Lee et al., 2020), UCB-DraC (Raileanu et al., 2020), IBAC-SNI
(Igl et al., 2019), RAD (Laskin et al., 2020), and IDAAC (Raileanu & Fergus, 2021).
Following the protocol proposed by Raileanu et al.
(2020), we conduct a set of experiments on the easy setting of all environments available in Procgen.
All agents were trained during 25M steps on 200 training levels and their performance is compared on the entire distribution of levels (Appendix G reports results on the hard setting).
We report the average sum of rewards for all environments in Table 2.
The CLOP layer outperforms other state-of-the-art methods on 7 out of the 16 Procgen games, sometimes by a large margin.
Compared to these methods, the CLOP layer offers a direct and easy way to augment the RL agent’s ability to generalize to unseen environments, with the beneﬁt of being entirely complementary with each of them (Appendix E reports results on the combination with IDAAC).
CLOP improves both performance and generalization gap.
Table 3 reports the gains obtained by the CLOP layer in terms of performance on training levels, testing levels and generalization gap, compared to PPO, after 25M steps of training on all Procgen games.
Complete results are reported in Appendix F.
The CLOP layer systematically improves the generalization gap (of 30% on average and up to 50% on StarPilot).
It also has a notable effect on both the training and the generalization absolute performance.
On all games but three (CaveFlyer, Ninja, and Plunder), the training performance is at least equivalent (10 games), and sometimes much better (BigFish, Chaser,
Leaper) with the CLOP layer, both in training speed and asymptotical performance.
The CLOP layer’s generalization absolute performance is also better on all but the same three games, 2 with a signiﬁcantly greater advantage in number of games.
It reaches signiﬁcantly better generalization performance on 11 games and equivalent performance on 2 games (Coinrun, Jumper).
Inﬂuence of the permutation rate.
The CLOP layer has a single tunable hyperparameter α , corresponding to the probability of accepting a candidate permutation between pixels during the forward pass.
Figure 5 reports the effect of varying this hyperparameter on three environments where the
CLOP layer improved generalization.
Interestingly, large values of α do not prevent the agent from learning.
In some cases (Figure 5c), it might even lead to improved performance.
Since permutations occur locally, a high probability of swapping does not imply the risk of generating latent states
[Table]
[Figure]
that are too far off from the original latent state (and would thus harm overall training performance).
Figures 6c and 6d (see next paragraph) illustrate this risk.
Another desirable property of this parameter is that performance does not seem to be too sensitive to it.
Any non-zero value seems to permit generalization.
Consequently, using the CLOP layer does not require ﬁne hyperparameter tuning.
Locality and channel-consistency are crucial features for generalization.
Figure 6 illustrates the effect of a forward pass through the CLOP layer without these components on a handcrafted, 3channel (RGB) feature map on the Leaper game (Figure 6b).
Although there is a non-zero probability that CLOP sends a pixel’s content arbitrarily far from its original position, the output of the layer retains an interpretable shape from a human perspective (Figure 6c).
In contrast, Figure 6d illustrates the output if the permutations are not local anymore and can send any pixel’s content anywhere in the image in a single permutation.
Similarly, Figure 6e shows how loosing the channel-consistency feature creates new colors in the image by recombining unrelated features together.
Both these behaviors seem undesirable for proper generalization.
We performed an ablation study to assess the inﬂuence of these two components.
Figure 7 shows that both the locality and channel-consistency properties of CLOP are beneﬁcial to generalization (full results in Appendix H).
Interestingly, nonlocal permutations and channel-inconsistent permutations still provide better generalization than plain PPO.
Such image modiﬁcations are agnostic to the feature maps spatial structure.
But since irrelevant features (e.g., background) can be swapped without consequences, and since these features concern a majority of pixels, with high probability these modiﬁcations still generate samples that preserve the useful semantic information.
However, whenever useful feature pixels are swapped without locality, this semantic information is made ambiguous between actual experience samples and augmented data, that might induce different action choices.
This conﬁrms the general idea that data augmentation helps generalization in RL, but that improved performance comes from preserving spatial knowledge (length, width, and channel-wise) in the latent space.
We also used Grad-Cam (Selvaraju et al., 2017) to produce visual explanations of the importance of these two factors (Figure 8).
An agent trained using a CLOP layer stripped of the local permutation feature focuses on a spread-out portion of the image, while the full CLOP agent displays very focused saliency areas.
The policies of the PPO agents, or those of the CLOP agents with local but channel-inconsistent per-
[Figure]
[Figure] [Figure]
mutations, display much more focused saliency maps too, despite having lower performance than the full CLOP agents.
This underlines a limit of saliency map interpretation: focussing on speciﬁc pixels is important but it is the combination of these pixel values that makes a good decision and this process is more intricate than solely concentrating on important parts of the image.
CLOP layer’s position in the network.
Figure 7 also displays the loss in generalization performance when applying CLOP to earlier layers in the IMPALA architecture used, conﬁrming that
CLOP is most beneﬁcial on the deepest feature maps (more results in Appendix I).
It also reports the effect of applying CLOP of both the actor and critic of PPO (discussion in Appendix J).
C ONCLUSION
In this work, we introduce the Channel-consistent LOcal Permutations (CLOP) technique, intended to improve generalization properties of convolutional neural networks.
This stems from an original perspective on observational overﬁtting in image-based reinforcement learning.
We discuss why observational overﬁtting is a structural risk minimization problem that should be tackled at the level of abstract state representation features.
To prevent RL agents from overﬁtting their decisions to ingame progress indicators and, instead, push informative, causal variables into the policy features, we implement CLOP as a feed-forward layer and apply it to the deepest feature maps.
This corresponds to performing data augmentation at the feature level while preserving the spatial structure of these features.
The CLOP layer implies very simple operations that induce a negligible overhead cost.
Agents equipped with a CLOP layer set a new state-of-the-art reference on the Procgen generalization benchmark.
Although generalization in RL is a broad topic that reaches out well beyond observational overﬁtting, the approach taken here endeavors to shed new light and proposes a simple and low-cost original solution to this challenge, hopefully contributing a useful tool to the community.
A N ETWORK ARCHITECTURE AND TRAINING HYPER - PARAMETERS A. S UPERVISED LEARNING
We used a simple network for the MNIST-USPS experiment, composed of a convolutional feature extractor:
• Conv2D(ﬁlters=32, kernel= 5 × 5 , stride=1, padding=2, activation=ReLU) • Max Pooling(ﬁlters= 2 × 2 , stride=2) • Conv2D(ﬁlters=64, kernel= 5 × 5 , stride=1, padding=2, activation=ReLU) • Max Pooling(ﬁlters= 2 × 2 , stride=2) • Conv2D(ﬁlters=128, kernel= 3 × 3 , stride=1, padding=2, activation=ReLU)
followed by a two hidden layer MLP:
• Dense(nb neurons=512, activation=ReLU) • Dense(nb neurons=100, activation=ReLU) • Dense(nb neurons=10, activation=Linear)
The dropout and the CLOP layers where added between the feature extractor and the classiﬁer.
For the batch-normalization conﬁguration, we added batch-normalization layers after each convolutional layer.
We trained all networks for 30 epochs with Adam, a learning rate lr = 5 · 10 − 4 , and a cosine annealing schedule.
For the Imagenette experiment (and Imagewoof experiment of Appendix D), we used Pytorch’s implementation of VGG11, replacing the output layer to match the ten classes.
For the STLexperiment of Appendix D, since the images are already downscaled, we used Pytorch’s implementation of a VGG9 network, replacing the output layer to match the ten classes.
For the Imagenet experiment of Appendix D we used Pytorch’s implementation of a VGG16 network, replacing the output layer to match the ten classes.
When testing our method, we removed the dropout layers from the fully connected part of the network and added a CLOP layer between the last convolutional layer and the ﬁrst fully connected one.
As is common on these benchmarks, we applied a horizontal stochastic ﬂipping function to preprocess the input as basic data augmentation.
We trained all networks on 90 epochs with Adam, a learning rate lr = 5 · 10 − 4 , and a cosine annealing schedule.
A. R EINFORCEMENT LEARNING
We based our implementation of PPO on the one proposed by Hojoon Lee available at: https:// github.com/joonleesky/train-procgen-pytorch .
We used an IMPALA (Espeholt et al., 2018) architecture on all agents, with a CLOP layer added to the actor part only for our CLOP agents (except for the ablation study reported in Figure 7 and Appendix J where it is also included in the critic).
Figure 9 illustrates CLOP agent’s architecture.
Values of α used for each environment are reported in Table 4.
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
B R EPRODUCIBILITY
All the experiments from Section 5 were run on a desktop machine (Intel i9, 10th generation processor, 32GB RAM) with a single NVIDIA RTX 3080 GPU.
Details about each experiments are reported in Table 5.
In a general context of data efﬁcient machine learning research
[Table]
it seems relevant to underline that each subﬁgure in Figure 4 evaluated 7 regularization methods, over 3 repetitions.
Overall, this required an order of 7 × 3 × 22 = 462 computing hours (around 20 days) for a single subﬁgure on a high-end desktop machine.
The experimental protocol used to select the three games reported in
Figure 4 relies on the results obtained on the easy mode of Procgen (Figure 10 and Table 2), on a random sampling of hard mode games, and on limited training length results on hard mode games.
Besides this information, we provide the full source code of our implementation and experiments, along with the data ﬁles of experimental results we obtained.
C C OMPUTATIONAL COST OF THE CLOP LAYER
The CLOP layer has to process each element of the intermediate feature map, but only width and height-wise.
This results in a O
( H (cid:48) W (cid:48) ) cost, where H (cid:48) and W (cid:48) are the height and width of the feature maps, which can be much smaller than that of the input image.
An important feature is the independence on the channel depth, which can grow signiﬁcantly in deep layers.
Comparatively, methods that perform data augmentation in the input space generally suffer from complexities at least in O ( CHW ) (for input images of size C × H × W ), and sometimes more for costly augmentation methods like rotations.
Table 6 reports two examples of the computational overhead of introducing the CLOP layer in a PPO training.
Table 7 performs a similar experiment comparing the running times of various standard data augmentation techniques with CLOP on MNIST and
Imagenette.
This conﬁrms the negligible overhead induced by the CLOP layer.
[Table] [Table]
D A DDITIONAL EXPERIMENTS IN SUPERVISED LEARNING
Following the experiments of Section 5.1, we evaluated the beneﬁt of the CLOP layer when training a VGG16 network on the full Imagenet database over 90 epochs, and compared with the reported improvement brought by mixup (Zhang et al., 2018c) in the same setting.
Mixup improves the generalization performance by 0.2 to 0.6% across network architectures (generalization accuracies without mixup between 76 and 80% depending on the network used).
Similarly, CLOP brings a negligible 0.15% improvement (71.6% top-1 generalization accuracy without CLOP, versus 71.75% with CLOP).
We conjecture the sample variety and diversity in Imagenet actually cancels the need for data augmentation altogether.
To conﬁrm this conjecture and reﬁne the understanding of when
CLOP is beneﬁcial or not, we ran the same experiment on several datasets with less variety in the images and less data.
STL-10 (Coates et al., 2011) is a subset of Imagenet inpired by CIFAR-
(Krizhevsky, 2009), where each class has even fewer labeled training examples than in CIFAR10 (500 training images, 800 test images per class).
Imagewoof is the subset of 10 breed classes from ImageNet.
Training on Imagewoof or on Imagenette does not beneﬁt from the diversity of image features present in the whole Imagenet dataset.
In these cases, CLOP was the method that contributed most to close the generalization gap, as shown in Table 8.
These results, along with the ones on the Imagenette dataset indicate that the CLOP layer becomes really beneﬁcial when data is scarce and one needs to avoid overﬁtting.
E C OMBINATION OF CLOP AND IDAAC
One advantage of CLOP is its simplicity and the possibility to combine it with other regularization methods.
Since, at the time of writing, IDAAC is one of the most efﬁcient algorithms on Procgen, it is tempting to evaluate its combination with CLOP.
Table 9 reports on this, on all Procgen games, easy setting, after 25M training steps.
CLOP was included in the IDAAC code provided in https://github.com/rraileanu/idaac.
[Table] [Table]
Overall, the combination neither seems beneﬁcial or detrimental and the effects are rather game dependent.
Being able to combine methods together has a “cocktail effect” that does not necessarily imply that the combination will be systematically more efﬁcient than the base elements.
IDAAC+CLOP substantially improves on IDAAC only in the Bigﬁsh game.
Conversely, it seems to degrade its performance on Plunder.
On all other games, the improvement or degradation with respect to IDAAC seems more marginal.
IDAAC’s auxiliary tasks on the policy network aim at predicting the advantage and not being able to predict the ordering between frames (which is strongly connected with predicting the value).
This focuses the features on essential information for the policy, thus creating an inductive bias and, maybe removing the need for data augmentation.
Depending on the game, swapping pixels that contain policy-relevant information might thus have a positive or negative impact on the generalization performance.
On the other hand, our implementation of CLOP is based on a standard PPO, where the policy and value networks are not separated and thus the feature maps contain both policy and value-relevant features, and thus the improvement brought by
CLOP is more important than on IDAAC.
This discussion also sheds light on the cases when the
IDAAC features seemed insufﬁcient to provide a good policy, i.e. when PPO+CLOP was already better than IDAAC: in these cases, adding CLOP to IDAAC cannot help recover the performance of
PPO+CLOP (e.g. Chaser, Dodgeball, Fruitbot, Heist, Leaper, StarPilot).
F F ULL RESULTS ON P ROCGEN ’ S EASY MODE
Table 3 reported the gain in generalization gap between plain PPO and PPO+CLOP in percentages, for better readability.
Speciﬁcally, it reported the gain in training accuracy, testing accuracy and generalization gap with respect those of the PPO baseline.
We emphasize it is a gain in generalization gap with respect to the PPO baseline, not the generalization gap itself.
In order to assess the generalization gap itself, we calculate it as the difference between training and testing levels, as a percentage of the training performance.
Table 11 reports the performance on training levels of all methods, borrowing the values mentioned in the work of Raileanu et al.
Raileanu & Fergus (2021).
This information is then combined with the performance on testing levels in order to assess the generalization gap of all methods, reported in Table 12.
This generalization gap should be read with the absolute generalization performance in mind: a small generalization gap alone is not sufﬁcient to characterize a method.
Figure 10 illustrates the evolution of both the performance and generalization gap along training, for
PPO and PPO+CLOP.
In order to follow the protocol of (Cobbe et al., 2020) and for a comparison between CLOP and
IDAAC, we also report normalized scores in Table 13, both on training and testing levels.
Note that
[Table] [Table]
these scores should be taken with a grain of salt since the normalizing R max can be quite far from the actual top performance among algorithms (e.g. Starpilot, CaveFlyer, Dodgeball, BigFish).
This has an effect of shrinking the apparent performance of some methods on average since it returns a (somehow artiﬁcially) low normalized score on these games. These normalized scores are multiplied by 100 for better readability. Figure 11 displays the histograms of the difference between CLOP and IDAAC on training and testing levels.
The speciﬁc case of the Plunder game seems to be an outlier where IDAAC vastly outperforms CLOP (and all other methods from the literature) on testing levels (by 70 . 2 normalized points). For all other games, on testing levels, IDAAC dominates by 0 to 15 .normalized points on 5 games, and CLOP dominates by 0 to 22 . 9 normalized points on the remaining 10 games.
The average over these 15 games shows CLOP being marginally better ( +3 . 7 ).
Results on training levels also show an advantage for CLOP ( +13 . 5 on average, without speciﬁc outliers).
Again, we warn against over-interpretation of these ﬁgures, that depend on the normalization factors.
We would also like to emphasize that these results compare the two top-performing methods (which both bring signiﬁcant improvements with respect to other baselines).
Given these results, it also seems relevant to note that both IDAAC and CLOP bring different insights on what is important for regularization in RL, therefore, beyond the raw comparison, these ﬁgures should mainly open new questions as to what makes a method better in each case.
Tables 14, 15 and 16 follow the outline of Table 3 and report the improvement of each method from the literature, with respect to the PPO baseline in terms of (respectively) training levels performance, testing levels performance, and generalization gap.
We report them mainly for the sake of completeness.
[Figure]
[Table]
[Table]
G F ULL RESULTS ON P ROCGEN ’ S HARD MODE
CLOP’s direct competitor, IDAAC, does not report results on Procgen’s hard mode.
As indicated in Appendix B, each subﬁgure in Figure 4 required around 20 days of computation on a high-end machine.
The experimental protocol in the Procgen paper implies testing on the full distribution of levels, including those used during training.
Our choice of using only the testing levels makes the benchmark even harder since no level seen during training is among our testing distribution.
We argue our protocol has two beneﬁts.
We test only the generalization property, without any inﬂuence from the performance on training levels.
Also, the levels selected for testing along the training process are precisely the same across algorithms.
Figure 12 shows the training curves along with the testing performance for CLOP and PPO.
Overall, the main conclusions remain: CLOP dominates in a majority of games and often improves the convergence speed, both in terms of training and testing performance.
Although it was not possible to repeat the comparison of the 7 regularization methods of Figure(or of other methods) on all Procgen games, mixreg (Wang et al., 2020) does report results on the hard setting.
Wang et al.
(2020) follow the protocol of Cobbe et al.
(2020) that evaluate the agents [Table] [Table] on the full distribution of levels.
Table 17 reports the compared performance on the full distribution of testing levels of PPO, mixreg, and CLOP in the hard setting, after 200M training steps.
Here again, CLOP dominates (by a variable margin) mixreg in 10 out of the 16 Procgen games.
H F ULL RESULTS ON THE ABLATION STUDY
Figure 13 reports the effect of turning off the channel consistency constraint, or the locality constraint, on all Procgen games.
On a majority of games (10 out of 16: Bigﬁsh, Chaser, Climber,
Dodgeball, Fruitbot, Jumper, Leaper, Maze, Miner, Starpilot), the trend shown in Section 5.2 is conﬁrmed: both locality and channel-consistency are important for generalization performance.
Three games stand-out as outliers (Bossﬁght, Coinrun, Heist), where turning off one of these constraints seemed to sometimes improve generalization performance.
First of all, we note this improvement is small compared to the improvement of CLOP upon PPO.
So this does not question the interest of using CLOP overall.
Secondly, we conjecture these improvements are rather game or trainingdependent.
Finally, it seems illegitimate to draw conclusions on the three last games (CaveFlyer,
Ninja, Plunder) since PPO outperforms CLOP on them, with or without these constraints.
Interestingly, IDAAC also performs rather poorly on two games among these three (CaveFlyer and Ninja, where IBAC-SNI is the leading algorithm), while it dominates on Plunder (Table 2).
I A PPLYING CLOP AT VARIOUS DEPTHS WITHIN THE NETWORK
The CLOP layer can be applied after any convolutional or pooling layer of a neural network.
Figure 7 shows the impact of the CLOP layer’s position, after different Resnet blocks, in the IMPALA architecture, evaluated on three Procgen games.
We can observe that the CLOP layer has a more [Table] [Table] substantial effect when applied on the most deepest feature map.
To verify whether this is also conﬁrmed in supervised learning [Table] we trained two networks (architectures in Appendix A) with a CLOP layer at various depths, on MNIST and STL-10.
The results, reported in Figure 14 show that the position of the CLOP layer also has an impact on supervised learning.
Similarly to the phenomenon observed in RL, in supervised learning, applying the CLOP layer on the deepest features maps increases the generalization capabilities of the networks in both experiments.
This goes to conﬁrm that data augmentation over abstract features (deep convolutional feature maps) is a key to efﬁcient generalization [Table]
J A PPLYING CLOP ON BOTH THE ACTOR AND THE CRITIC OF PPO
The results reported in Section 5.2 made use of CLOP only within the actor of PPO.
It is legitimate to wonder whether there is any interest in applying the CLOP layer also to the critic.
It is useful to note that PPO is a “true” policy gradient method, contrarily to SAC or TD3 that are approximate value iteration methods which include gradient ascent on the policy to solve the greediness search (the arg max operator in the Bellman equation).
Because of this, it collects samples at every round to estimate the critic and cannot rely on a persistent replay buffer.
Intrinsically, the critic is only used to compute advantage estimates in states that have been encountered during recent exploration.
Consequently, there is no need for the critic to actually generalize to unseen states [Figure] and letting the critic overﬁt the value function is actually beneﬁcial.
One could conjecture that this phenomenon of overﬁtted critic beneﬁting PPO, is also one reason for the success of DAAC (and
IDAAC), since their critic is not constrained to share a common stem of convolutional layers with the actor and is thus free to better ﬁt the values.
Figure 7 reports the difference in performance between the two versions of PPO, conﬁrming this analysis on a limited set of Procgen games.
K W HAT WOULD BE THE EQUIVALENT OF CLOP IN NON - IMAGE - BASED RL?
This short discussion was motivated by comments from the ICLR reviewers.
CLOP is essentially an image-based method that exploits the spatial structure of images.
Local permutations are a way of performing local “fuzzying”.
When such fuzzying is done in the input space it aims at invariance with respect to measurement noise.
The rationale behind the CLOP layer is that performing such a fuzzying in feature map space is a principled way to aim at object positions invariance.
Consequently, one could see the encoding of the feature maps as isomorphic to a latent space of positional variables.
In this latent space, the equivalent of our permutations would correspond (at least in the intention) to the application of a small, centered noise on positional variables, so as to reach positional invariance.
From a theoretical perspective, one could interpret this process as a margin maximization problem (the margin being deﬁned in the feature space rather than in the input space).
This is an interesting avenue for future research and this short discussion applies more generally to any noise addition process in data augmentation.