RRL: R ESNET AS REPRESENTATION FOR R EINFORCEMENT L EARNING
A BSTRACT
The ability to autonomously learn behaviors via direct interactions in uninstrumented environments can lead to generalist robots capable of enhancing productivity or providing care in unstructured settings like homes.
Such uninstrumented settings warrant operations only using the robot’s proprioceptive sensor such as onboard cameras, joint encoders, etc which can be challenging for policy learning owing to the high dimensionality and partial observability issues.
We propose RRL: Resnet as a representation for Reinforcement Learning – a straightforward yet effective approach that can learn complex behaviors directly from proprioceptive inputs.
RRL fuses features extracted from pre-trained Resnet into the standard reinforcement learning pipeline and deliver results comparable to learning directly from the state.
In a simulated dexterous manipulation benchmark, where the state of the art methods fails to make signiﬁcant progress, RRL delivers contact rich behaviors.
Its effectiveness in learning behaviors directly from visual inputs with performance and sample efﬁciency matching learning directly from the state, even in complex high dimensional domains, is far from obvious.
I NTRODUCTION
Recently, Reinforcement learning (RL) has seen tremendous momentum and progress Mnih et al.
(2015); Silver et al.
; Jaderberg et al.
; Espeholt et al.
(2018) in learning complex behaviors from states Schulman et al.
(2017); Haarnoja et al.
; Rajeswaran et al.
Most success stories, however, are limited to simulations or instrumented laboratory conditions as real world doesn’t provide direct access to its underlying state.
Not only learning with state-space, but visual observation spaces have also found reasonable success Kalashnikov et al.
; OpenAI et al.
However, the sample complexity for methods that can learn directly with visual inputs is far too extreme for direct real-world training Duan et al.
; Kaiser et al.
In order to deliver the promise presented by data-driven techniques, we need efﬁcient techniques that can learn unobtrusively without environment instrumentation.
Learning without environment instrumentation, especially in unstructured settings like homes, can be quite challenging Zhu et al.
Dulac-Arnold et al.
Ahn et al.
Challenges include – (a) Decision making with incomplete information owing to partial observability as the agents must rely only on proprioceptive on-board sensors (vision, touch, joint position encoders, etc) to perceive and act.
(b) The inﬂux of sensory information makes the input space quite high dimensional.
(c) Information contamination due to sensory noise.
(d) Most importantly, the scene being ﬂushed with information irrelevant to the task (background, clutter, etc).
Agents learning under these constraints are forced to take a large number of samples simply to untangle these taskirrelevant details before it makes any progress on the true task objective.
A common approach to handle these high dimensionality issues is to learn representations that distill information into low dimensional features and use them as inputs to the policy (Figure 2).
While such ideas have found reasonable success Qin et al.
; Manuelli et al. (2019a), designing such representations require a deep understanding of the problem and domain expertise.
An alternative approach is to leverage unsupervised representation learning to autonomously acquire representations based on either reconstruction Higgins et al.
; Zhu et al.
Yarats et al.
(2020) or contrastive Srinivas et al.
; Stooke et al.
(2020) objective.
These methods are quite brittle as the representations are acquired from narrow task-speciﬁc distributions and hence, do not generalize well across different
[Figure]
This indicates that Resnet is indeed looking for the right features in our task images (right) in spite of such high distributional shift. tasks.
Additionally, they acquire task-speciﬁc representations, often needing additional samples from the environment or domain speciﬁc data-augmentations for training representations, leading to poor sample efﬁciency.
The key idea behind our method stems from an intuitive observation over the desiderata of a good representation i.e. (a) it should be low dimensional (b) it should be able to capture silent features encapsulating the diversity and the variability present in our task (c) it should be robust to irrelevant information like noise, lighting, viewpoints, etc (d) it should provide effective representation in the entire distribution that a policy can induce.
These requirements are quite harsh usually needing extreme domain expertise to manually design and an abundance of samples to automatically acquire.
Can we acquire this representation without any additional effort?
Our work takes a very small step in this direction.
The key insight behind our method is embarrassingly simple – representations do not necessarily have to be trained on the exact task distribution; a representation trained on a sufﬁciently wide distribution of real-world scenarios, will remain effective on any distribution a policy optimizing a task in the real world might induce.
While training over such wide distribution is demanding, this is precisely what the success of large image classiﬁcation models He et al.
; Simonyan & Zisserman (2015);
Tan & Le (2020); Szegedy et al.
(2015) in Computer Vision delivers – representations learned over a large family of real-world scenarios, i.e, trained over hundreds of classes.
Our Contributions : We list the major contributions of our work below-
We present a surprisingly simple method (RRL) at the intersection of representation learning, imitation leaning (IL) and reinforcement learning (RL) that uses features from pre-trained image classiﬁcation models (Resnet34) as representations in standard RL pipeline.
Our method is quite general and can be incorporated with minimal changes to state based RL/IL algorithms.
We demonstrate that features learned by image classiﬁcation models are general towards different tasks, robust to visual distractors, and when used in conjunction with standard IL and RL pipelines can efﬁciently acquire policies directly from proprioceptive inputs.
While competing methods have restricted results primarily to planar tasks devoid of depth perspectives, on a rich collection of simulated high dimensional dexterous manipulation tasks, where state-of-the-art methods struggle, we demonstrate that RRL can learn rich behaviors directly from visual inputs with performance & sample efﬁciency approaching state-based methods.
R ELATED W ORK
RRL rests on recent developments from the ﬁelds of Representation Learning, Imitation Learning and Reinforcement Learning.
In this section, we outline related works leveraging representation learning for visual reinforcement and imitation learning.
2. L EARNING WITHOUT EXPLICIT REPRESENTATION
A common approach is to learn behaviors in an end to end fashion – from pixels to actions – without explicit distinction between feature representation and policy representations.
Success stories in this categories range from seminal work Mnih et al.
mastering Atari 2600 computer games using only raw pixels as input, to Levine et al.
(2016) which learns trajectory-centric local policies using Guided Policy Search Levine & Koltun (2013) for diverse continuous control manipulation tasks in the real world learned directly from camera inputs.
More recently, Haarnoja et al.
has demonstrated success in acquiring multi-ﬁnger dexterous manipulation Zhu et al.
and agile locomotion behaviors using off-policy action critic methods Haarnoja et al.
While learning directly from pixels has found reasonable success, it requires training large networks with high input dimensionality.
Agents require a prohibitively large number of samples to untangle task-relevant information in order to acquire behaviors, limiting their application to simulations or constrained lab settings.
RRL maintains an explicit representation network to extract low dimensional features.
Decoupling representation learning from policy learning delivery results in large gains in efﬁciency.
Next, we outline related works that use explicit representations.
2. L EARNING WITH SUPERVISED REPRESENTATIONS
Another approach is to ﬁrst acquire representations using expert supervision, and use features extracted from representation as inputs in standard policy learning pipelines.
A predominant idea is to learn representative keypoints encapsulating task details from the input images and using the extracted keypoints as a replacement of the state information.
Using these techniques, Qin et al.
; Manuelli et al.
(2019b) demonstrated tool manipulation behaviors in rich scenes ﬂushed with task-irrelevant details.
Nagabandi et al.
(2019) demonstrated simultaneous manipulation of multiple objects in the task of Baoding ball tasks on a high dimensional dexterous manipulation hand.
Along with the inbuilt proprioceptive sensing at each joint, they use an RGB stereo image pair that is fed into a separately pre-trained tracker to produce 3D position estimates You et al.
(2020) for the two Baoding balls.
These methods, while powerful, learn task-speciﬁc features and requires expert supervision, making it harder to (a) translate to variation in tasks/environments, (b) scale with increasing task diversity.
RRL, on the other hand, uses single task-agnostic representations making it easy to scale.
2. L EARNING WITH UNSUPERVISED REPRESENTATIONS
With the ambition of being scalable, this group of methods intends to acquire representation via unsupervised techniques.
Sermanet et al.
(2018) uses contrastive learning to time-align visual features across different embodiment demonstration behavior transfer from human to a Fetch robot.
Burgess et al.
(2018), Finn et al.
; Zhu et al.
(2020) use variational inference Kingma & Welling (2014);
Burgess et al.
(2018) to learn compressed latent representations and use it as input to standard
RL pipeline to demonstrate rich manipulation behaviors.
Hafner et al.
(2020) additionally learns dynamics models directly in the latent space and used model-based RL to acquire behaviors on simulated tasks.
On similar tasks, Hafner et al.
(2019) uses multi-step variational inference to learn world dynamic as well as rewards models for off-policy RL.
Srinivas et al.
(2020) uses image augmentation with variations inference to construct features to be used in standard RL pipeline and demonstrated performance at par with learning directly from the state.
Laskin et al.
Kostrikov et al.
(2020) demonstrate comparable results by assimilating updates over features acquired only via image augmentation.
Similar to supervised methods, unsupervised methods often learns task-speciﬁc brittle representations and often suffers challenges from non-stationarity arising from the mismatch between the distribution representations are learned on and the distribution policy induces.
To induce stability, RRL uses pre-trained stationary representations trained on distribution with wider support than what policy can induce.
Additionally, representations learned over a wide distribution of real-world samples are robust to noise and irrelevant information like lighting, illumination, etc.
Unlike prior works Hafner et al.
(2020; 2019); Srinivas et al.
Laskin et al.
Kostrikov et al.
(2020), which demonstrate results on simple planner task devoid of depth perspective, we demonstrate RRL on a suite of high dimensional contact rich dexterous manipulation tasks.
2. L EARNING WITH REPRESENTATIONS AND DEMONSTRATIONS
Learning from demonstrations has a rich history.
We focus our discussion on DAPG Rajeswaran et al.
(2017), a state-based Natural policy gradient Rajeswaran et al.
(2018) method which optimized for the natural gradient of a joint loss with imitation as well as reinforcement objective.
DAPG has been demonstrated to outperform competing methods Gupta et al.
; Hester et al.
(2017) on the high dimensional ADROIT dexterous manipulation task suite we test on.
RRL extends DAPG to solve the task suite directly from proprioceptive signals with performance and sample efﬁciency comparable to state-DAPG.
Unlike DAPG which is on-policy, FERM Zhan et al.
(2020) is a closely related off-policy actor-critic methods combining learning from demonstrations with RL.
FERM builds on RAD Laskin et al.
and inherits its challenges.
We demonstrate via experiments that
RRL is more stable, more robust to various distractors, and convincingly outperforms FERM.
B ACKGROUND
RRL solves a standard Markov decision process (Section 3.1) by combining three fundamental building blocks.
(a) Policy gradient algorithm (Section 3.2), (b) Demonstration bootstrapping (Section 3.3), and (c) Representation learning (Section 3.4).
We brieﬂy outline these fundamentals before detailing with our method in Section 4.
3. P RELIMINARIES : MDP
We model the control problem as a Markov decision process (MDP), which is deﬁned using the tuple:
M = ( S , A , R , T , ρ 0 , γ ) .
S ∈ R n and A ∈ R m represent the state and actions.
R : S ×
A → R is the reward function.
In the ideal case, this function is simply an indicator for task completion ( sparse reward setting ).
T : S ×
A → S is the transition dynamics, which can be stochastic.
In model-free
RL, we do not assume any knowledge about the transition function and require only sampling access to this function.
ρ 0 is the probability distribution over initial states and γ ∈ [0 , 1) is the discount factor.
We wish to solve for a stochastic policy of the form π : S ×
A → R which optimizes the expected sum of rewards:
[Equation]
3. P OLICY G RADIENT
The goal of the RL agent is to maximise the expected discounted return η ( π ) [Equation 1] under the distribution induced by the current policy π .
Policy Gradient algorithms optimize the policy ( π θ ( a | s ) ) directly, where θ is the function parameter by estimating ∇ η ( π ) .
First we introduce a few standard notations, Value function : V π ( s ) , Q function : Q π ( s, a ) and the advantage function : A π ( s, a ) .
The advantage function can be considered as another version of Q-value with lower variance by taking the state-value off as the baseline.
[Equation]
The gradient can be estimated using the Likelihood ratio approach and Markov property of the problem Williams (1992) and using a sampling based strategy, [Equation]
[Equation]
Amongst the wide collection of policy gradient algorithms, we build upon Natural Policy Gradient (NPG) Rajeswaran et al.
(2018) to solve our MDP formulation owing to its stability and effectiveness in solving complex problems, especially in high dimensional domains Rajeswaran et al.
We refer to Weng for a detailed background on different policy gradient approaches.
Next, we describe how human demonstrations can be effectively used along with NPG to aid policy optimization.
3. D EMO A UGMENTED P OLICY G RADIENT
Policy Gradients with appropriately shaped rewards can solve arbitrarily complex tasks.
However, real-world environments seldom provide a shaped reward, and must be manually speciﬁed by domain experts.
Learning with sparse signals, such as task completion indicator functions can relax domain expertise in reward shaping but it results in extremely high sample complexity due to exploration challenges.
DAPG (Rajeswaran et al.) combines policy gradients with few demonstrations to curb these challenges.
We represent the demonstration dataset using ρ D = (cid:110)(cid:16) s ( i ) t , a ( i ) t , s ( i ) t +1 , r ( i ) t (cid:17)(cid:111) where t indexes time and i indexes different trajectories.
DAPG incorporates demonstrations in two way to efﬁciently learn from the them :
(1) Warm up the policy using few demonstrations (25 in our setting) using a simple Mean Squared Error(MSE) loss, i.e, initialize the policy using behavior cloning [5].
This provides an informed policy initialization that helps in resolving the exploration issue and reduces the sample complexity.
[Equation]
where, θ are the agent parameters and a ( i ) H t represents the action taken by the human.
(2) DAPG builds upon on-policy NPG algorithm Rajeswaran et
al. 2018) which uses a normalized gradient ascent procedure where the normalization is under the Fischer metric.
[Equation]
where ˆ F θ k is the Fischer Information Metric at the current iterate θ k ,
[Equation]
and g is the sample based estimate of the policy gradient [4].
To make the best use of available demonstrations, DAPG proposes a joint loss g aug with task as well as imitation objective.
The imitation objective asymptotically decays over time allowing the agent to learn behaviors surpassing the expert.
[Equation]
where, ρ π is the dataset obtained by executing the current policy, ρ D is the demonstration data and w ( s, a ) is the heuristic weighting function deﬁned as : [Equation]
[Equation]
Though this work builds upon Natural Policy Gradients, in principle, any state based RL algorithm, can be integrated with our approach.
In the next section, we outline our method that scales DAPG towards visual information.
3. R EPRESENTATION L EARNING
Until now in the above sections, we have discussed assuming direct access to the low-level state information.
In a real-world scenario, access to such state information is infeasible, instead, we have access to high dimensional observations like images.
DAPG has thus far only been demonstrated to be effective with low-level state.
DAPG is based on NPG which works well but faces issues with input dimensionality and hence, cannot be directly used with the images.
Representation learning
Bengio et al.
(2014) is learning representations of input data typically by transforming it or extracting features from it, which makes it easier to perform the task (in our case it can be used in place of the exact state of the environment).
These representation learning techniques can be used to tackle the curse of dimensionality in an intuitive way Wulfmeier et al.
The feature extractor can be (a)
Learned along with RL objective Srinivas et al.
; Yarats et al.
(b) Decoupled from
RL by pre-training itStooke et al.
The former lead to task-speciﬁc features that can get quite narrow and speciﬁc to the domain.
This can hurt the generalization performance especially when it has to perform in diverse environmental settings.
Thus, we pre-train the feature extractor initially and use it across all the tasks that we have considered.
Next we describe RRL in detail.
RRL: R ESNET AS R EPRESENTATION FOR RL
In an ideal RL setting, the agent interacts with the environment based on the current state, and in return, the environment outputs the next state and the reward obtained.
Refer to the left side of
Figure 2.
This works well in a simulated environment but in a real-world scenario, we do not have access to this low-level state information.
Instead we get the information from cameras ( I t ) and other sensors like encoders ( Sensor Data t ).
To overcome the challenges associated with learning with high dimensional spaces (highlighted in section 3.4), we use representations that project information into a lower-dimensional manifolds.
RRL uses the standard Resnet-34 model as our feature extractor.
Resnet is trained over a wide variety of real-world classes [Described in detail 7.2].
We demonstrate that the diversity of the Resnet feature extractor allows us to use it across all tasks we considered.
This further simpliﬁes our algorithm as our representations are frozen and do-not face the non-stationarity issues encountered while learning policy and representations in tandem.
The features ( h t ) obtained from the above feature extractor are appended with the information obtained from the internal joint encoders of the Adroit Hand (Sensor Data t ).
We show that this can be used as a substitute for the exact state( s t ) of the agent in the RL algorithm.
[Equation]
here h t = f
( I t ) ∈
R k , I t ∈ R n represents the raw image obtained from the camera
where k << n , and f is the feature extractor (Resnet).
[Pseudo-code - 1].
E XPERIMENTAL E VALUATIONS
Our experimental evaluations aims to address the following questions: (1) Can RRL learn complex tasks directly from proprioceptive signals (camera inputs and joint encoders) , (2) How various representational choices inﬂuence the generality and versatility of the resulting behaviors, (3) How does RRL’s performance and efﬁciency compare against other state-of-the-art methods, (4) What are the effects of various design decisions on our method?
5. T ASKS
While the applicability of prior proprioception based RL methods Laskin et al.
Kostrikov et al.
; Hafner et al.
(2020) have been limited to simple low dimensional tasks (Cartpole swing-up,
Cheetah run, Reacher, Finger spin, Walker walk, Ball in cup catch, etc), we demonstrate performance on ADROIT manipulation suite Rajeswaran et al.
(2017) which consists of high dimensional dexterous manipulation tasks (Figure8).
Furthermore, unlike prior task sets, which are fundamentally planner tasks devoid of depth perspective, the ADROIT manipulation suite consists of visually rich physically realistic tasks demanding representations untangling depth information as well.
5. R ESULTS
In Figure 3, we contrast the performance of RRL against the state of the art baselines.
We observed that NPG Kakade (2001) struggles to solve the suite even with full state information, which establishes [Figure] the difﬁculty of our task suite.
DAPG state Rajeswaran et al.
(2017) uses privileged state information from the environment to solve the tasks and pose as the best case oracle.
RRL demonstrates good performance on most task, relocate being the hardest, and often approaches performance comparable to DAPG state .
A competing baseline FERM 1 Zhan et al.
(2020) is quite unstable in these tasks.
It starts strong for hammer and door tasks but saturates in performance.
It makes slow progress in pen, and completely fails for relocate.
[Figure]
In Figure 4 we compare the computational footprint of various methods we consider.
We note that our method is approximately three times more compute-efﬁcient than FERM (and other variations we consider.
Evaluation details in 7.6).
In Figure 4 we probe the robustness of the ﬁnal policies by injecting various distractors in the environment during inference.
We note that the resilience of the resnet features induces robustness to RRL’s policies.
Task-speciﬁc features learned by FERM are brittle leading to larger degradation in performance
[Evaluation Details in 7.5]. To summarize, we demonstrate that RRL often exhibits state comparable performance, outperforming FERM in such complex high dimensional tasks, is computationally much more efﬁcient (almost 3x of SOTA) and is also quite stable to various visual distractions.
5. E FFECTS OF R EPRESENTATION
In ﬁgure 5, we study the effects various representational choices.
We note that RRL using resnet features signiﬁcantly outperforms a variant RRL(VAE)
(see appendix 7.4 for details) using features learned via variational inference Yarats et al.
; Ha & Schmidhuber (2018a;b); Higgins et al.
; Nair et al.
(2018) techniques.
This indicates that pre-trained Resnet provides superior features compared to methods that explicitly learn task-speciﬁc features using additional samples from the [Figure] [Figure]
RRL environment. Unlike features learned from the narrow distribution of task environments, Resnet features are trained on diverse real-world categories [7.2].
In addition to improved sample complexity resulting from the use of pre-trained features, the resilience, robustness, and versatility of Resnet features lead to policies that are also robust to visual distractors and clutter in the scene.
In Figure 4 we observe that RRL is quite resilient to visual distractions (light, colors, redundant objects) encountered during test cases.
5. E FFECTS OF PROPRIOCEPTION CHOICES AND SENSOR NOISE
While it’s hard to envision a robot without proprioceptive joint sensing, harsh conditions of the real-world can lead to noisy sensing, even sensor failures.
In Figure 6, we subjected RRL to (a) signals with 2% noise in the information received from the joint encoders, and (b) only visual inputs are used as proprioceptive signals.
In both these cases, our methods remained performant with slight to no degradation in performance.
5. A BLATIONS AND A NALYSIS OF D ESIGN D ECISIONS
[Figure]
In our next set of experiments, we evaluate the effect of various design decisions on our method.
In Figure 5, we study the effect of different
Resnet features as our representation.
Resnet34, though computationally more demanding (Figure 4) than Resnet18, delivers better performance owing to its improved representational capacity and feature expressivity.
A further boost in capacity (Resnet50) degrades performance, likely due to the incorporation of less useful features and an increase in samples required to train the resulting larger policy network.
Reward design, especially for complex high dimensional tasks, requires signiﬁcant domain expertise.
RRL replaces the needs of well-shaped rewards by using a few demonstrations (to curb the exploration challenges in high dimensional space) and sparse rewards (indicating task completion).
This signiﬁcantly lowers the domain expertise required for our methods.
In Figure 7, we observe that RRL (using sparse rewards) delivers competitive performance to a variant of our methods that uses well-shaped dense rewards.
In ﬁgure,
D ISCUSSION : S TRENGTHS , L IMITATIONS & O PPORTUNITIES
This paper presents an intuitive idea bringing together advancements from the ﬁelds of representation learning, imitation learning, and reinforcement learning.
A very simple method named RRL that leverages Resnet features as representation to learn complex behaviors directly from proprioceptive signals.
The resulting algorithm approaches the performance of state-based methods in complex
ADROIT dexterous manipulation suite.
Strengths : The strength of our insight lies in its simplicity, and applicability to almost any reinforcement or imitation learning algorithms that intends to learn directly from high dimensional proprioceptive signals.
We present RRL , an instantiating of this insight on top of imitation + (onpolicy) reinforcement learning method called DAPG, to showcase its strength.
It presents yet another demonstration that features learned by Resnet are quite general and are broadly applicable.
Resnet features trained over 1000s of real-world image classes are more robust and resilient in comparison to the features learned by methods that learn representation and policies in tandem using only samples from the task distribution.
The use of such general but frozen representations in conjunction with RL pipelines additionally avoids the non-stationary issues faced by competing methods, that simultaneously optimizes reinforcement and representation objectives, leading to more stable algorithms.
Additionally, not having to train your own features extractors results in a signiﬁcant sample and compute efﬁciency gains, refer to ﬁgure 4. Limitations : While this work demonstrates promises of using pre-trained features, it doesn’t investigate the data mismatch problem that might exist.
Real-world datasets used to train resnet features are from human-centric environments.
While we desire robots to operate in similar settings, there are still differences in their morphology and mode of operations.
Additionally, resent (and similar models) acquire features from data primarily comprised of static scenes.
In contrast, embodied agents desire rich features of dynamic and interactive movements.
Opportunities : RRL uses a single pre-trained representation for solving four complex and very different tasks.
Unlike the domains of vision and language, there is a non-trivial cost associated with data in robotics.
The possibility of having a standard shared representational space opens up avenues for (a) leveraging
A PPENDIX 7. RRL(O URS )
Same parameters are used across all the tasks (pen-v0, door-v0, hammer-v0 and relocate-v0) unless explicity mentioned.
Sparse reward setting is used in all the environments as proposed by Rajeswaran et al..
We have directly used the parameters provided by DAPG without any additional ﬁne tuning except for the policy size (same across all tasks).
[Table]
7. R ESNET A RCHITECTURAL D ETAILS
We use standard Resnet-34 model as RRL’s feature extractor.
The model is pre-trained on the
ImageNet dataset which consists of 1000 classes. It is trained on 1.28 million images on the classiﬁcation task of ImageNet.
The last layer (fc layer) of the model is removed and all the parameters are frozen throughout the training of the RL agent.
During inference, the observations obtained from the environment are of size 256 × 256 , a center crop of size 224 × 224 is fed into the model. We also evaluate our model using different Resnet sizes (Figure 5). Algorithm
Input 3: repeat 4: do 5:
1 to horizon do 6:
Take action a t = π θ ([ Encoder ( I t ) , SensorData t ]) and receive I t +1 , SensorData t +1 , r t +1 from the environment.
7: end for 8: end for 9:
Compute ∇ θ log π θ ( a t | s t ) for each ( s, a ) ∈ ρ π , ρ D 10:
Compute A π ( s, a ) for each ( s, a ) ∈ ρ π and w ( s, a ) for each ( s, a ) ∈
ρ D according to
Equations 2,11:
Calculate policy gradient according to12:
Compute Fisher matrix13:
Take the gradient ascent step according to 6.
Update the parameters of the value function in order to approximate(2) :
[Equation] n )
[Equation]
15: until Satisfactory performance
7. P ERFORMANCE BOOST IN R ELOCATE - V
[Figure] We were able to boost the performance of relocate signiﬁcantly simply by using three cameras instead of one.
Relocate is the toughest task among all the four settings as it severely faces issues like occlusion, precisely extracting 3-D information (location of object and target).
Both of them can be countered using multiple cameras, so we report the performance with three camera setting too. Refer to Figure
7. RRL(VAE)
For training, we collected a dataset of 1 million images of size 64 x 64 .
Out of the 1 million images collected, 25% of the images are collected using an optimal course of actions (expert policy), 25% with a little noise (expert policy + small noise), 25% with even higher level of noise (expert policy + large noise) and remaining portion by randomly sampling actions (random actions).
This is to ensure that the images collected sufﬁciently represents the distribution faced by policy during the training of the agent.
We observed that this signiﬁcantly helps compared to collecting data only from the expert policy.
The variational auto-encoder(VAE) is trained using a reconstruction objective Kingma & Welling (2014) for 10epochs.
Figure 10 showcases the reconstructed images.
We used a latent size of 512 for a fair comparison with Resnet.
The weights of the encoder are freezed and used as feature extractors in place of Resnet in RRL.
RRL(VAE) also uses the inputs from the pro-prioceptive sensors along with the encoded features.
VAE implementation courtesy Subramanian (2020).
7. V ISUAL D ISTRACTOR E VALUATION DETAILS
In order to test the generalisation performance of RRL and FERM Zhan et al.
(2020), we subject the environment to various kinds of visual distractions during inference (Figure 11).
Note all parameters are freezed during this evaluation, an average performance over 75 rollouts is reported.
Following distractors were used during inference to test robustness of the ﬁnal policy -
• Random change in light position.
• Random change in light direction.
• Random object color.
(Handle, door color for Door-v0; Different hammer parts and nail for Hammer-v0) •
Introducing a new object in scene - random color, position, size and geometry (Sphere, Capsule, Ellipsoid, Cylinder, Box).
7. C OMPUTE C OST CALCULATION
We calculate the actual compute cost involved for all the methods (RRL(Ours), FERM, RRL(Resnet50), RRL(Resnet-18)) that we have considered.
Since in a real-world scenario there is no simulation of the environment we do not include the cost of simulation into the calculation.
For fair comparison we show the compute cost with same sample complexity (4 million steps) for all the methods.
FERM is quite compute intensive (almost 3x RRL(Ours)) because (a) Data augmentation is applied at every step (b) The parameters of Actor and Critic are updated once/twice at every step (Compute results