Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning
Abstract
Learning generalizable policies that can adapt to unseen environments remains challenging in visual Reinforcement Learning (RL).
Existing approaches try to acquire a robust representation via diversifying the appearances of in-domain observations for better generalization.
Limited by the specific observations of the environment, these methods ignore the possibility of exploring diverse realworld image datasets.
In this paper, we investigate how a visual RL agent would benefit from the off-the-shelf visual representations.
Surprisingly, we find that the early layers in an ImageNet pre-trained ResNet model could provide rather generalizable representations for visual RL.
Hence, we propose P re-trained I mage
E ncoder for G eneralizable visual reinforcement learning (PIE-G), a simple yet effective framework that can generalize to the unseen visual scenarios in a zeroshot manner.
Extensive experiments are conducted on DMControl Generalization
Benchmark, DMControl Manipulation Tasks, Drawer World, and CARLA to verify the effectiveness of PIE-G. Empirical evidence suggests PIE-G improves sample efficiency and significantly outperforms previous state-of-the-art methods in terms of generalization performance.
In particular, PIE-G boasts a 55 % generalization performance gain on average in the challenging video background setting.
Project sites google com view pie-g/
Introduction
Visual Reinforcement Learning (RL) has achieved significant success in learning complex behaviors directly from image observations [48, 35, 37].
Despite the progress, RL agents are often plagued by the overfitting problem [67], especially in high-dimensional observation space.
Previous studies show that it is difficult for the visual agents to generalize to unseen scenarios [8, 40], which severely limits their deployment in real-world applications.
In general, visual RL methods rely on their encoders to learn a visual representation to perceive the world.
Recent studies have found that data augmentation [65] leads to more generalizable representations so that the agents can adapt to the unseen environments with different visual appearances [58, 74].
However, most of those approaches only augment the observations of the training environments [37, 39, 68], which is unable to provide enough diversity for generalization over large domain gaps.
Furthermore, naively applying data augmentation may damage the robustness of learned representations and decrease training sample efficiency [28, 83].
Observation
Replay Buffer
GENET IM
[Figure]
To overcome these drawbacks, what we require is a universal representation that can generalize to a variety of unseen scenarios.
Recent works in representation learning demonstrate promising results in enabling pre-trained models to provide strong priors for downstream tasks [31, 11].
The pre-trained models contain representations obtained from a wide range of existing real-world image datasets.
These representations are proved to be robust to noises and capable of distinguishing salient features despite the diversity and the inconsistency [12].
Based on the observations, we would like to ask the following question: is it possible to train a visual RL agent that is augmented with pre-trained visual representations so that it can better generalize to novel tasks?
Towards answering the question, the main contribution of this paper is a surprising discovery that the off-the-shelf features of frozen models trained with ImageNet can be used as universal representations for visual RL.
Based on such findings, we present P retrained I mage E ncoder for G eneralizable visual reinforcement learning (PIE-G), a visual RL framework that allows agents to obtain enhanced training efficiency and generalization ability via integrating the extracted representations from a pre-trained
ResNet [30] encoder into RL training.
Straightforward as the framework appears, PIE-G enjoys thoughtful details and nuanced design choices to acquire representations that are suitable for control and generalizable to novel scenarios.
Specifically, we show that the choice of early layer features and the ever-updating Batch Normalization (BatchNorm) [33] are crucial for the performance gain.
To validate the effectiveness of our framework, we conduct a series of experiments on 4 benchmarks:
DMControl Generalization Benchmark (DMC-GB) [26], DMControl Manipulation Tasks
Drawer World [75] that is modified from Meta World [82], and CARLA, a realistic autonomous driving simulator.
Empirical studies have demonstrated that PIE-G achieves better or competitive results in training sample efficiency, and significantly outperforms previous state-of-the-art methods in generalization capability without bells and whistles.
Our main contributions are summarized as follows: (i)
We find that pre-trained encoders from off-the-shelf image datasets with the early layer features and ever-updating BatchNorm provide generalizable representations in visual RL.
(ii)
We propose PIE-G, a simple yet effective framework with a pre-trained encoder that can boost the sample efficiency and generalization ability in visual
RL.
(iii) PIE-G outperforms state-of-the-art methods in 4 visual generalization benchmarks by a large margin, with a 55 % boost on average on the hardest setting in DMC-GB.
Related Work
Representation learning in RL.
A large corpus of literature has sought to leverage representation learning in the setting of RL [61, 22, 68, 60, 81, 69].
More recently, there are a branch of methods that perform unsupervised pre-training to encourage exploration for better sample efficiency [53, 44, 17, 43, 38, 29].
A typical approach is to use a contrastive learning method to jointly incentivize exploration and acquire useful representations [15].
Liu et al.
[44] introduce a new type of pretraining techniques via entropy maximization in embedding space for better exploration.
In particular, inspired by the new representation algorithm in computer vision, Yarats et al.
[79] propose a SwAVlike architecture [4] for pre-training with an exploration scheme that maximizes the entropy of the state visitation distribution.
However, all these methods require the data collected in the target environments, resulting in additional sample cost.
We instead propose a cross-domain pre-training way to improve visual RL agents’ performance without any in-domain interaction during the pretraining time.
Pre-trained visual encoders for RL.
Applying the pre-trained vision model from other domains to the control tasks has gradually attracted researchers’ attention [80, 66, 36, 62, 16, 51, 76].
For example, Shah et al.
[63] and Parisi et al.
[52] suggest that with the help of experts’ demonstrations, the pre-trained ResNet [30] representations can achieve competitive performance with state-based inputs.
Moreover, human video datasets are introduced to pre-train a visual representation for downstream policy learning [49].
The pre-trained model has also been proposed for goal-specification via behavior cloning [9].
However, few works explore the effectiveness of pre-trained models for generalization.
In contrast to prior approaches, PIE-G enables agents to generalize well to the unseen visual scenarios with a large distributional shift in a zero-shot manner while achieving high sample efficiency in a standard RL training paradigm.
Generalization in visual RL.
Researchers have investigated to improve visual RL agents’ generalization ability from various aspects [1, 27, 40, 73, 2, 50, 77].
Data augmentation [40, 74, 28, 83, 18, 47] and domain randomization [71, 56, 54, 59, 5] are effective ways for generalization in visually different environments.
Notably, Hansen et al.
[26] employ a BYOL-like[24] architecture to decouple the augmentation from policy learning for better generalization.
In order to control the high variance when implementing data augmentation, Hansen et al.
[28] add a regularization term of the Q function between un-agumented and augmented data as an implicit variance reduction technique.
Meanwhile,
Yuan et al.
[83] propose a task-aware data augmentation method with the Lipschitz constant [19] for maintaining training stability.
Fan et al.
[18] apply data augmentation in the imitation learning paradigm.
Prior works rely on data augmentation to gain a robust representation for each task.
In this work, we tackle this challenge from a different perspective, utilizing a single pre-trained encoder with the universal visual representations for all the tasks.
Preliminaries
Reinforcement learning.
Due to partial state observability from images in visual RL [34], we consider learning in a Partially Observable Markov Decision Process (POMDP) [3] formulated by the tuple ⟨S , O , A , r, P , γ ⟩ where S is the state space, O is the observation space,
A is the action space, r :
S × A
(cid:55)→ R is a reward function, P ( s t +1 | s t , a t ) is the state transition function, and γ ∈ [0 , 1) is the discount factor.
The goal is to find a policy π ∗ to maximize the expected cumulative return π ∗ = arg max π E a t ∼ π ( ·| s t ) , s t ∼P (cid:104) (cid:80)
Tt =1 γ t r ( s t , a t ) (cid:105) , starting from an initial state s 0 ∈ S and obtained by following the policy π θ ( · | s t ) which is parameterized by learnable parameters θ .
Generalization.
In terms of generalization, we consider a set of POMDPs: M = {M 1 , M 2 , ..., M n } that shares the same dynamics and structures.
The only difference among them is the observation space O .
This setting is more formally described as “Block MDPs” [14].
During the training process, we only have the access to a fixed POMDP denoted M i .
Our purpose is to train an agent on a specific scenario M i to learn a policy π ∗ G which can maximize the expected cumulative return over the whole set of POMDPs in a zero-shot generalization manner.
Method
In this section, we introduce PIE-G, a simple yet effective framework for visual RL which benefits from the pre-trained encoders on other domains to facilitate sample efficiency and generalization ability.
4.1 Pre-trained Encoder
PIE-G explicitly leverages the pre-trained models as the representation extractor without any modification.
The pre-trained encoder projects high-dimensional image observations into compact, low-dimensional embeddings that are later used by RL policies.
Note that PIE-G is as simple as importing a pre-trained ResNet model from the torchvision [46] library.
This avoids the design of any auxiliary tasks to acquire useful representations.
For all the training tasks on different benchmarks, the encoder’s parameters are frozen to obtain universal visual representations.
Since the pre-trained model contains the priors from a wide range of real-world images, we hypothesize that the inherited power from a pre-trained model may help to capture and distinguish the main components of different tasks’ observations regardless of the changes of visual appearances or deformed shapes, and will further improve the sample efficiency and generalization abilities of RL agents.
To validate our hypothesis, we first encode each observation independently to obtain embeddings.
Then, the embeddings from the second layer of the pre-trained model are fused as input features to the policy networks [64, 51].
Moreover, we enable BatchNorm [33] to keep updating the running mean and running standard deviation during the policy training.
The key findings are: 1) early layers of a neural network would provide better representations for visual RL generalization, which resonates with prior works in imitation learning [52]; 2) the always updating statistics in BatchNorm helps better adapt to the shift in observation space and thus improve the generalization ability.
More detailed discussion can be found in Section 5.4.
4.2 Reinforcement Learning Backbone
We implement DrQ-v2 [78] as the base visual reinforcement learning algorithm.
DrQ-v2 is the state-of-the-art method for visual continuous control tasks, which adopts DDPG [41] coupling with clipped Double Q-learning [20] to alleviate the overestimation bias of target Q-value.
The agents are trained with two Q θ k value functions and their corresponding target network Q ¯ θ k .
The critic loss function is as follows, and the mini-batch of transitions
τ = ( s t , a t , r t : t + n − 1 , s t + n ) is sampled from the replay buffer D :
[Equation]
with n-step TD target y :
[Equation]
The actor π ϕ is trained with the following objective:
[Equation]
where s t is augmented by random shift, a t = π ϕ ( s t )
+ ϵ , ϵ is sampled from clip (cid:0) N (cid:0) 0 , σ 2 (cid:1) , − c, c (cid:1) with a decaying exploration noise σ .
Thanks to the efficiency of DrQ-v2, PIE-G enjoys faster wall-clock training time and fewer computational footprints.
We emphasize that PIE-G does not need any other proprioceptive states and sensory information as the inputs besides the representations extracted from original image observations.
In previous works [37, 78, 58, 28], different schemes of data augmentation are proposed to improve sample efficiency and generalization performance.
In practice, weak augmentation methods (e.g., random shift) of DrQ [37] and DrQ-v2 [78] are found to be beneficial for sample efficiency.
In the setting of generalization, we follow the way of using strong augmentation methods (e.g., mixup) of
SVEA [28] and DrAC [58] to further boost the performance.
It is worth mentioning that since the gradient is stopped before it reaches the encoder, all the data augmentation techniques discussed here do not affect the pre-trained visual representation.
Meanwhile, unlike Rutav et al.
[63] and Simone et al.
[52], we purely train the agent in a standard RL paradigm without any expert’s demonstration.
[Table]
Experiments
In this section
[Table]
we investigate the following questions: (1) Can PIE-G improve the agent’s generalization ability?
Specifically, how well does PIE-G deal with jittered color, moving video background, and deformed shapes of robots?
(2) Can PIE-G improve training sample efficiency?
(3) How do the choice of layers in the encoder and the use of BatchNorm [33] affect the performance?
(4) Can further finetuned RL agents outperform those with frozen visual encoders?
5.1 Setup
We evaluate our method on a wide range of tasks, including DMControl Generalization Benchmark (DMC-GB) [26], DeepMind Manipulation tasks [72], and Drawer World [75].
PIE-G is trained for 500k interaction steps with 2 action repeat and evaluated with 100 episodes for every task on the testing benchmarks.
All the generalization evaluations are in a zero-shot manner.
By default, the encoder uses the ResNet18 architecture [30].
To be more specific, the feature maps of the second layer are flattened and passed through an additional fully connected layer to serve as the representations of the observations.
More training details and environment descriptions are in Appendix B.
5.2 Evaluation on Generalization Ability
We compare the generalization ability of PIE-G with state-of-the-art methods and strong baselines:
SAC [25]: a widely used off-policy RL algorithm; DrQ [37]: a SAC-based visual RL algorithm with augmented observations; DrQ-v2 [78]: the prior state-of-the-art model-free visual RL algorithm in terms of sample efficiency; SVEA [28]: the prior state-of-the-art method in terms of generalization via reducing the Q-variance through an auxiliary loss; TLDA [83]: another state-of-the-art method in generalization by using task-aware data augmentation.
Generalization on color-jittered observations.
The agent’s generalization ability is evaluated on
DMC-GB with randomly jittered color.
For the manipulation tasks, the colors of different objects (e.g., floors, arms) are modified.
As shown in Table 1, PIE-G obtains better or competitive performance in 9 out of 10 instances.
These results suggest that the visual representation from the pre-trained model is more robust to the color-changing than the one trained by standard RL algorithms.
[Table]
Generalization on unseen and/or moving backgrounds.
We then evaluate PIE-G on the more challenging settings: video easy and video hard in DMC-GB.
The video hard setting consists of more complicated and fast-switching video backgrounds that are drastically different from the training environments.
Notably, even the reference plane of the ground is removed in this setting.
The comparison results are shown in Table 2.
PIE-G achieves better or comparable performance with the prior state-of-the-art methods in 9 out of 12 instances.
In particular, PIE-G gains significant improvement in the video hard setting over all the previous methods with +55% improvement on average.
For example, in the Finger Spin , Cup Catch , and Walker Walk tasks, PIE-G outperforms the best of the other methods by substantial margins 127.0% , 95.0% , and 59.2% respectively.
[Figure] [Figure]
Attempting to explain the success
[Figure]
we visualize the difference of the normalized feature maps extracted from the encoder whose inputs are two Walkers of the same pose but with different backgrounds, as is shown in Figure 2.
Ideally, a well-generalizable encoder would map the observations of the two Walkers to exactly the same embedding, and therefore the difference should be zero.
In practice, as shown in Figure 2, the encoder of PIE-G produces a difference much closer to zero than that of
SVEA.
Numerically, we calculate the average pixel intensity in the difference of normalized feature maps, and the intensity is decreased by 50.9% with PIE-G than that with SVEA.
[Figure]
Then, we evaluate PIE-G on the CARLA [13] autonomous driving system which contains realistic observations and complex driving scenarios.
The default setting is adopted from Zhang et al.
PIE-G and other algorithms are benchmarked on 4 diverse weather environments.
As shown in Figure 3, all the prior state-of-the-art methods cannot adapt to the new unseen weather with different lighting, humidity, road conditions etc.
The behind reason is that compared to DMC-GB whose images merely consist of a single control agent and the background, the observations of CARLA contain more distracting objects and factors; therefore, only depending on data augmentation to provide diverse data cannot tackle this complicated visual driving task.
The experimental results in Figureexhibits that thanks to the ImageNet pre-trained encoder,
PIE-G can generalize well on the complicated scenes without large performance drop.
[Table]
Furthermore, we conduct experiments on the Drawer World benchmark to test the agent’s generalization ability in manipulation tasks with different background textures.
The visualized observations are shown in
Figure 4a.
Success Rate is adopted as the evaluation metric for its goal-conditioned nature.
Table 3 illustrates that PIE-G can achieve better or comparable generalization performance in all the settings with +24% boost on average while other approaches may suffer from the CNN’s sensitivity in the face of various textures [21].
Generalization on deformed shapes.
To verify agent’s robustness in terms of the deformed shapes, we modify the shapes of the jaco arm and the target objects in the manipulation tasks, as shown in
Figure 5a.
Figure 5b demonstrates that PIE-G also improves the agents’ generalization ability with various shapes while other methods could barely generalize to these changes.
We attribute this to the lack of shape changing in previous data augmentation techniques.
Conversely, our pre-trained encoder is learned from a multitude of real-world images with various poses and shapes, thus enhancing its generalization ability on deformed shapes.
[Figure]
5.3 Evaluation on Sample Efficiency
We evaluate the sample efficiency of PIE-G on 8 relatively challenging tasks on the DeepMind Control
Suite and the Manipulation task ( Reach Duplo ).
Figure 6 demonstrates that PIE-G achieves better or comparable sample efficiency and asymptotic performance than DrQ-v2 in 7 out of 8 tasks.
To eliminate the effect of varied network size, we also include random enc , a baseline that has the same network architecture with frozen random initialized parameters.
In addition, Figure 4b shows that
PIE-G is also superior over the baselines on the Drawer world benchmark.
The results demonstrate that the pre-trained encoder inherits the powerful feature extraction ability trained from ImageNet and acquires better training sample efficiency in control tasks.
[Figure]
i s od e
[Figure]
r d
[Figure]
5.4 Ablation study
To verify the necessity of the design choices in PIE-G
[Figure]
we conduct a series of ablation studies to take a closer look at the proposed method.
More results are shown in Appendix C
Task
Layer Layer
Layer Layer
[Table]
Choice of layers.
In convolutional neural networks, the later layers capture high-level semantic features, while the early layers are responsible for extracting low-level information [45, 84, 42].
Figure 7 and Table 4 investigate how much control tasks can benefit from the features extracted from different layers.
As shown in Figure 7, the early layers preserve rich details of edges and corners, while the later layers only provide very abstract information.
Intuitively, for control tasks, a trade-off is required between low-level details and high-level semantics.
Table 4 and Figure 8 in
Appendix show that the Layer 2 gains better generalization and sample efficiency performance than the other layers.
[Figure]
Batch normalization.
Batch Normalization (BatchNorm) [33] is a popular technique in computer vision.
However, it is not widely adopted in RL algorithms.
In contrast to conventional wisdom, BatchNorm is found to be useful and important in PIE-G. Specifically, we find that calculating the mean and variance of the observations during evaluation rather than using the statistics from training data would boost the performance.
Figure 9 demonstrates that, in the most challenging settings, PIE-G with the use of BatchNorm can further improve the generalization performance.
This is largely because the distribution of observations is determined by the agent, violating the assumption of independent and identical distribution (i.i.d.).
This use of BatchNorm also reassures the recommendation from Ioffe et al.
[33] that recomputation of the statistical means and variances allows the BatchNorm layer to generalize to new data distributions.
[Table]
Adopting other datasets for pretraining.
Besides ImageNet [10], we also implement pre-trained visual encoders with other novel and popular datasets: CLIP [57] and Ego4D [23].
CLIP (Contrastive Language–Image
Pre-training) trained a large number of (image, text) pairs collected from
Internet to jointly acquire visual and text representations.
The Ego4D is a egocentric human video dataset which contains massive daily-life activity videos in hundreds of scenarios.
Table 5 shows that the agents pre-trained with CLIP achieves comparable performance with those pre-trained with ImageNet.
Since the Ego4D collects the videos with the first-person view, the view difference between the tasks and the dataset leads to a decrease in performance; nevertheless, the Ego4D pre-trained agents still obtain comparable results with the prior state-of-the-art methods.
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Finetuning the pre-trained model.
We also conduct research to finetune the encoder’s parameters instead of keeping it frozen.
Previous works [32, 55] have found that finetuning pretrained models is challenging.
Consistent with these studies, Table 6 suggests that compared with the frozen representations from pre-trained models, the finetuned representations suffer from the out-of-distribution problem [6] and lead to a performance drop in terms of the generalization ability.
[Table]
Adopting other pre-trained models
Additionally, we investigate the efficacy of other visual representations.
MoCo-v2 [7] is a pre-trained model optimized via contrastive learning to learn representations.
We find that
PIE-G with the pre-trained representations of MoCo-v2 can also obtain a comparable performance in terms of both the sample efficiency and generalization ability.
More results are shown in Appendix C.
Conclusion
In this work, we propose PIE-G, a simple yet effective framework that leverages off-the-shelf features of ImageNet pre-trained ResNet models for better generalization in visual RL.
Extensive experiments on a variety of tasks in four RL environments confirm the merits of universal visual representations, which endow the agents with improved sample efficiency and better generalization performance.
In addition, we show that the choice of layers and the use of BatchNorm are crucial for the performance gain.
Our exploration may inspire more researchers to dig into the great potential of utilizing pre-trained representations in visual RL.
Limitations.
We study generalization in the simulated environments.
However, there might be new challenges in the real-world applications.
In the future, we would like to establish and test on benchmarks of real-world scenarios.
Acknowledgements: We would like to thank Boyuan Chen, Kaizhe Hu and Guozheng Ma for giving helpful suggestions.
We are sincerely grateful to the anonymous reviewers for taking the time to review this work and giving insightful advice.
Yang Gao is supported by the Ministry of Science and
Technology of the People ´ s Republic of China, the 2030 Innovation Megaprojects "Program on New
Generation Artificial Intelligence" (Grant No. 2021AAA0150000).
Yang Gao is also supported by a grant from the Guoqiang Institute, Tsinghua University.
A Environment Details
DeepMind control suite.
DMControl Suite [70] is a widely used benchmark, which contains a variety of continuous control tasks.
For generalization evaluation, we test methods on the DMControl
Generalization Benchmark (DMC-GB) [26] that is developed based on DMControl Suite.
DMC-GB provides different levels of difficulty in terms of generalization performance for visual RL.
Visualized observations are in the Setting column of Table 1 ( Top ) and Table 2.
DeepMind control manipulation tasks.
DeepMind Control [72] contains dexterous manipulation tasks with a multi-joint Jaco arm and snap-together bricks.
In this paper, we modify the colors and shapes of the arms and the bricks in the task of Reach Duplo to test the agents’ generalization ability.
Visualized observations are shown in the Setting column of Table 1 ( Bottom ).
Drawer world benchmarks.
Meta-world [82] contains a series of vision-based robotic manipulation tasks.
Wang et al.
[75] propose a variant of Meta-world, Drawer World, with a variety of realistic textures to evaluate the generalization ability of the agent.
These tasks require a Sawyer robot arm to open or close a drawer, respectively.
The visualizations of the environment are shown in Figure 4a.
CARLA autonomous driving.
CARLA [13] is a realistic simulator for autonomous driving.
Many recent works utilize this challenging benchmark in visual RL setting.
The trained agents are evaluated on different weather and road conditions.
B Implementation Details
In this section, we provide PIE-G’s detailed settings.
As shown in Table 8, we set up our hyperparmeters and environmental details in three benchmarks.
Our method is trained for 500k interaction steps (1000k environment steps with 2 action repeat).
All experiments are run with a single GeForce
GTX 3090 GPU and AMD EPYC 7H12 64-Core Processor CPU.
All code assets used for this project came with MIT licenses.
Code: https://anonymous.4open.science/r/PIE-G-EF75/
[Table]
84 ×
Discount factor γ 0.
Action repeat 8 (cartpole) 2 (otherwise)
Frame stack
Learning rate 1e5e- 1e1e-
Random shifting padding
Training step 500k 250k 500k 500k
Evaluation episodes
Optimizer
Adam Adam
Adam Adam
Vision models.
In this paper, we use the ready-made models from the following link: ResNet: https://github.com/pytorch/vision ; Moco: https://github.com/ facebookresearch/moco (v2 version trained with 800 epochs); CLIP: https://github.com/
OpenAI/CLIP ; R3M: https://github.com/facebookresearch/r3m .
In terms of implementing data augmentation, We choose random overlay (integrate a distract image
I with the observation o linearly, o ′ =
αo + (1 − α ) I ) as our augmentation method.
Similar to the previous works [58, 28] we add a regularization term R θ to the critic objective F θ without introducing extra hyperparameters and other techniques.
Our critic loss J θ is as follows, where D is the replay buffer, s t
aug is the augmented observation, (cid:98) Q ( s t , a t ) =
r ( s t , a t ) + γ E s t + 1 ∼P
[ V ( s t + 1 )] :
J Q ( θ ) =
F Q ( θ ) + R Q ( θ ) , (3) with
F
[Equation]
(cid:20)
[Equation] [Equation]
R
[Equation]
(cid:20)
[Equation]
[Figure] [Figure]
Drawer World.
For the Drawer World task, we use a small learning rate in order to maintain the training stability.
The episode lengths in Drawer World tasks are 200 steps with 4 action repeat.
Random Conv is applied as the data augmentation method.
Meanwhile, the original reward setting in the Drawer World is prone to the Q-value divergence.
Therefore, we scale the reward by 0.01.
Manipulation tasks.
The episode lengths in Manipulation tasks are 1000 steps with 2 action repeat.
Since for generalization the data augmentation will degrade the training efficiency, we do not apply random overlay on this benchmark.
We change the physical parameters of geom size to deform shape.
All methods are evaluated with 100 episodes on different settings.
CARLA.
We adopt the setting from Zhang et al.
[85] (e.g., the reward function and training weather conditions).
The maximum episode length in CARLA tasks is 1000 steps with 2 action repeat.
C Additional Results
In this section, we provide additional experimental results about PIE-G in various aspects.
C.1 Comparison with RRL
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Table]
generalize to the new environments in the DMC-GB.
RRL [63] is another ResNet pre-trained algorithm that can achieve comparable sample efficiency with the state-based algorithms and be robust to the visual distractors.
Here we compare the generalization ability of PIE-G with RRL.
To compare fairly , we re-implement RRL with DrQ-v2 [78] as the base algorithm which is the state-of-the-art methods in DMControl Suite.
Table 9 shows that RRL cannot adapt to the environments with distributional shifts while PIE-G exhibits considerable generalization ability when facing new visual scenarios.
We suggest that the choice of layers and ever-updating BatchNorm are the crucial factors for bridging domain gaps and boosting agents’ generalization performance.
C.2 Other Pre-trained Models
As shown in Figure 10, PIE-G with the MoCo-v2 pre-trained model also gains a competitive sample efficiency with the help of the off-the-shelf visual representations
. C.3 Choice of Architectures
[Table]
We further explore the impact of different network architectures.
Since
Layer 2 shows better performance, here we choose this layer to extract features.
As shown in Table 10, three kinds of network architectures show comparable generalization performance .
Since ResNet18 is less
[Figure]
a r d
Walker Walk
[Figure]
computationally demanding and with faster wall-clock time than the other two architectures, we choose it as the network backbone
[Figure]
r d
[Figure]
C.4 Finetune Models
As shown in Figure
[Figure]
finetuning encoders will significantly reduce sample efficiency.
We suggest that during the finetuning process, the encoders have to adapt to the new data distribution and unable to inherit the useful representations learned from the ImageNet, thus severely hindering the improvement in sample efficiency.
Additionally, Table 6 and Figure 8 indicate that finetuning the model will make the agents overfit to the training environment.