V ISION -B ASED M ANIPULATORS N EED TO A LSO S EE FROM T HEIR H ANDS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study how the choice of visual perspective affects learning and generalization in the context of physical manipulation from raw sensor observations.
Compared with the more commonly used global third-person perspective, a hand-centric (eye-in-hand) perspective affords reduced observability, but we ﬁnd that it consistently improves training efﬁciency and out-of-distribution generalization.
These beneﬁts require no algorithmic changes other than the perspective and hold across a variety of learning algorithms, experimental settings, and distribution shifts, albeit only when hand-centric observability is sufﬁcient.
When this is not the case, including a third-person perspective is necessary for learning, but also harms out-of-distribution generalization.
To mitigate this, we propose to regularize the third-person information stream via a variational information bottleneck.
On six representative manipulation tasks with varying hand-centric observability adapted from the Meta-World benchmark, this results in a state-of-the-art reinforcement learning agent operating from both perspectives improving its out-of-distribution generalization on every task.
While some practitioners have long put cameras in the hands of robots, our work systematically analyzes the beneﬁts of doing so and provides simple and broadly applicable insights for improving vision-based robotic manipulation.
I NTRODUCTION
Physical manipulation is so fundamental a skill for natural agents that it has been described as a “Rosetta Stone for cognition” (Ritter & Haschke, 2015).
How can we endow machines with similar mastery over their physical environment?
One promising avenue is to use a data-driven approach, in which the mapping from raw sensor observations of the environment (and other readily available signals, e.g. via proprioception) to actions is acquired inductively.
Helpful inductive biases in modern machine learning techniques such as over-parameterized models and stochastic gradient descent have enabled surprising (and poorly understood) generalization capabilities in some applications (Neyshabur et al., 2014;
Belkin et al., 2019; Zhang et al., 2021).
Despite this, visuomotor policies learned end-to-end remain brittle relative to many common real-world distribution shifts: subtle changes in lighting, texture, and geometry that would not faze a human cause drastic performance drops (Julian et al., 2020).
While a wide variety of algorithms have been proposed to improve the learning and generalization of object manipulation skills, in this paper we instead consider the design of the agent’s observation space, a facet of the learning pipeline that has been underexplored (Section 5).
Indeed, in some applications of machine learning, e.g., image classiﬁcation or text summarization, the disembodied nature of the task affords relatively little ﬂexibility in this regard.
Yet, even in these settings, simple data processing techniques such as normalization and data augmentation can have noticeable effects on learning and generalization (Perez & Wang, 2017).
The role of data can only be more profound in an embodied setting: any sensors capable of being practically instrumented will only provide a partial observation of the underlying world state.
While partial observability is typically regarded as a challenge that only exacerbates the difﬁculty of a learning problem (Kaelbling et al., 1998), we may also consider how partial observations can facilitate the acquisition of useful symmetries.
The natural world gives clear examples of this.
For instance, because cutaneous touch is inherently restricted to sensing portions of the environment in direct contact with the agent, tactile sensing by construction exhibits invariances to many common transformations on the underlying world state; grasping an apple from the checkout counter (without looking at it) is largely the same as doing so from one’s kitchen table.
Due in part to the nascent state of tactile sensing hardware (Yuan et al., 2017) and simulation (Agarwal et al., 2020), in this work we investigate the above insight in vision, the ubiquitous robotic sensory modality.
In particular, we focus on the role of perspective as induced from the placement of cameras.
To roughly imitate the locality of cutaneous touch, we consider the hand-centric (eye-in-hand) perspective arising from mounting a camera on a robotic manipulator’s wrist.
We also consider the more commonly used third-person perspective afforded by a ﬁxed camera in the world frame.
The main contribution of this work is an empirical study of the role of visual perspective in learning and generalization in the context of physical manipulation.
We ﬁrst perform a head-to-head comparison between hand-centric and third-person perspectives in a custom simulated grasping environment that features three kinds of distribution shifts.
We ﬁnd that using the hand-centric perspective, with no other algorithmic modiﬁcations, reduces aggregate out-of-distribution failure rate by 92% , 99% , and 100% (relative) in the imitation learning, reinforcement learning, and adversarial imitation learning settings, respectively.
We also observe appreciable gains in training efﬁciency in the latter two settings.
Despite their apparent superiority, hand-centric perspectives cannot be used alone for tasks in which their limited observability is a liability during training.
To realize the beneﬁts of hand-centric perspectives more generally, we propose using both hand-centric and third-person perspectives in conjunction for full observability while regularizing the latter with a variational information bottleneck (Alemi et al., 2016) to mitigate the latter’s detrimental effects on out-of-distribution generalization.
We instantiate this simple and broadly applicable principle in DrQ-v2 (Yarats et al., 2021), a state-of-the-art vision-based reinforcement learning algorithm, and ﬁnd that it reduces the aggregate out-of-distribution failure rate compared to using both perspectives naively by 64% (relative) across six representative manipulation tasks with varying levels of hand-centric observability adapted from the Meta-World benchmark (Yu et al., 2020).
P ROBLEM S ETUP
Preliminaries: MDPs and POMDPs.
We frame the physical manipulation tasks considered in this work as discrete-time inﬁnite-horizon Markov decision processes (MDPs).
An MDP M is a 6-tuple ( S , A , P, R, γ, µ ) , where S is a set of states, A is a set of actions, P : S ×
A → Π( S ) is a state- transition (or dynamics) function, R :
S ×
A → R is a reward function, γ ∈ (0 , 1) is a discount factor, and µ ∈ Π( S ) is an initial state distribution.
An MDP whose state cannot be directly observed can be formalized as a partially observable MDP (POMDP), an 8-tuple ( S , A , P, R, γ, µ, Ω , O ) that extends the underlying MDP with two ingredients: a set of observations Ω and an observation function O : S ×
A → Π(Ω) .
We consider only a restricted class of POMDPs in which the observation function is limited to be O :
S → Ω .
To solve a POMDP, we optimize a policy π : Ω → Π( A ) to maximize the expected return R ( M , π ◦ O ) =
E µ,P,π [ (cid:80) ∞ t =
γ t R ( s t , a t )] , where π ◦ O maps a state to an action distribution via composing the policy and observation function.
Observation functions.
In this work, we denote the observation functions corresponding to the hand-centric and third-person perspectives as O h and O 3 , respectively.
We also consider proprioception via forward kinematics, denoting this as O p .
Often, multiple observation functions are used together; for example, we denote using both the hand-centric and proprioceptive observations as
O h + p .
Invariances and generalization.
We say that a function f : X ×Y → Z is invariant in domain sub- space
X to a transformation T : X → X iff ∀ x ∈ X , y ∈ Y .
f ( T ( x ) , y ) =
f ( x, y ) .
We formalize the notion of generalization by saying that π ◦ O generalizes in M to a distribution shift caused by transformation T iff R (
M , π ◦ O ) is invariant in M to T .
We consider two kinds of generalization: in-distribution and out-of-distribution generalization, also referred to as interpolation and extrapolation.
The latter corresponds to the agent generalizing in M to some speciﬁed transformation, and the former is a special case when the transformation is identity.
In this work, we limit the scope of the transformations on M we consider to those acting on the initial state distribution µ through the state set S .
A few concrete examples of such transformations are illustrated in Figure 1.
H AND -C ENTRIC VS . T HIRD -P ERSON P ERSPECTIVES
The ﬁrst hypothesis we investigate is that using the hand-centric perspective O h instead of the thirdperson perspective
O 3 can signiﬁcantly improve the learning and generalization of the agent π ◦ O .
In this section, we probe this hypothesis in a setting where the hand-centric perspective gives sufﬁcient observability of the scene (we consider when this does not hold in Section 4).
See project website https://sites.google.com/view/seeing-from-hands for animated visualizations.
3. E XPERIMENTAL S ETUP
We consider a visuomotor grasping task instantiated in PyBullet (Coumans & Bai, 2016–2021).
A
Franka Emika Panda manipulator is tasked with picking up a speciﬁc cube that initially rests on a table.
The action space is 4-DoF, consisting of 3-DoF end-effector position control and 1-DoF gripper control.
Observation functions include O h and O 3 , which output 84 × 84 RGB images, and O p , which outputs 3D end-effector position relative to the robot base, 1D gripper width, and a
Boolean contact ﬂag for each of two gripper “ﬁngers“.
We use three learning algorithms:
imitation learning with dataset aggregation (DAgger) (Ross et al., 2011), reinforcement learning using data-regularized Q-functions (DrQ) (Kostrikov et al., 2020), and adversarial imitation learning using discriminator-actor-critic (DAC)
(Kostrikov et al., 2018).
We defer exposition on these algorithms to Appendix A.2.
We run DAgger and DrQ on three experiment variants each targeting a different distribution shift (detailed and visualized in Appendix A.1).
With
DAC, we assess in-distribution generalization in the training environment and out-of-distribution generalization between demonstration (demo) collection and the training environment.
Details on the model architectures and hyperparameters used can be found in Appendices A.3 and A.4.
3. E XPERIMENTAL R ESULTS AND D ISCUSSION
DAgger and DrQ results are reported in Figure 2 and aggregated in Table 1.
DAC results are reported in Figure 3, with experiment variant descriptions in the caption.
For DAgger (left two columns of Figure 2), we ﬁnd that the hand-centric perspective leads to clear improvements in out-of-distribution generalization (test) across all three experiment variants despite in-distribution generalization progress (train) being essentially identical between π ◦ O h + p and π ◦
O 3+ p .
The only exceptions to π ◦
O h
+ p generalizing better are in some instances of the distractor objects variant.
Here, seeing the red, green, and blue distractor objects during training was sufﬁcient for both π ◦ O h
+ p and π ◦
O 3+ p to learn to ignore these object colors, even under distractor distri- bution shift.
Generalization to white distractors was likely facilitated by the RGB representation of white as the “sum” of red, green, and blue.
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
For DrQ (right two columns of Figure 2), the differences between π ◦ O h + p and π ◦ O 3+ p extend into training time.
In the table height variant, π ◦ O h + p exhibits increased sample efﬁciency for training as well as similar out-of-distribution generalization beneﬁts as seen for DAgger.
For the distractor objects variant, π ◦
O h
+ p converges before π ◦ O 3+ p makes any signiﬁcant progress on success rate (though we did observe increasing returns).
Since DrQ trained π ◦ O 3+ p to convergence for the other variants within the same interaction budget, it follows that the presence of the distractors rendered the training task too hard for π ◦ O 3+ p , but not for π ◦
O h
+ p .
In the table textures variant, the generalization improvement of π ◦
O h
+ p over π ◦ O 3+ p is less extreme.
We attribute this to invariances to image-space transformations learned via the data augmentation built into DrQ. In
Appendix A.5, an ablation in which this augmentation is removed further shows its importance.
[Figure] For DAC
[Figure]
we ﬁnd stark improvements in the generalization of π ◦ O h over that of π ◦ O 3 . In the ﬁrst DAC-speciﬁc experiment variant (left plot of Figure 3), π ◦ O h fully generalizes in-distribution with as few as 5 demos, whereas π ◦ O 3 achieves signiﬁcantly lower success, even with 25 demos and much more online interaction.
In the second variant (center plot of Figure 3), the distribution shift between demo collection and training barely affects π ◦ O h , but severely compromises the training of π ◦ O 3 .
In the third variant (right plot of Figure 3), despite the presence of distractor objects giving the discriminator strong predictive power in distinguishing between demos and agent behavior, π ◦
O h still achieves a signiﬁcant measure of in-distribution generalization, whereas π ◦ Omakes little progress even with eight times the number of demos.
We remark that, in the context of adversarial imitation learning, π ◦ O h achieves its sample efﬁciency and robustness without any special requirements on the training data (Zolna et al., 2020) or modiﬁed training objectives (Xu &
Denil, 2020).
C OMBINING H AND -C ENTRIC AND T HIRD -P ERSON P ERSPECTIVES
The previous set of experiments demonstrate how hand-centric perspectives can lead to clear improvements in learning and generalization over third-person perspectives.
Unfortunately, this does not mean that the use of hand-centric perspectives is a panacea.
The limited observability of handcentric perspectives is a double-edged sword: depending on the environment and task, it can enable π ◦ O h to establish useful invariances, or confuse π ◦ O h by enforcing harmful ones.
In this section, we focus on evaluating across tasks of varying hand-centric observability, including those in which insufﬁcient observability severely undermines π ◦ O h .
How can we realize the beneﬁts of hand-centric perspectives even in such scenarios?
4. R EGULARIZING THE T HIRD -P ERSON I NFORMATION S TREAM
Insufﬁcient observability arising from using O h alone necessitates the inclusion of O 3 .
While using both perspectives should effectively resolve the issue of insufﬁcient observability and enable the agent to train, we know from Section 3.2 that the use of the third-person perspective can hamper out-of-distribution generalization by allowing the agent to “overﬁt” to particularities of the training distribution.
To mitigate this, we propose to regularize the third-person perspective’s representation.
While multiple regularization techniques could conceivably be suitable to this end, we choose the variational information bottleneck (VIB) to use in our experiments due to its simplicity, theoretical justiﬁcation, and empirical performance (Alemi et al., 2016).
For our subsequent experiments, we build on top of the state-of-the-art vision-based actor-critic reinforcement learning algorithm DrQ-v2 (Yarats et al., 2021)
(see Appendix B.3 for a detailed description).
When we use both hand-centric and third-person observations o h and o 3 , we instantiate two separate image encoders f ξ h and f ξ 3 .
We denote the corresponding representations as z h and z 3 .
These are concatenated before being fed to the actor π φ and critic networks Q θ 1 , Q θ 2 .
We apply a VIB to the third-person information stream to regularize the DrQ-v2 critic.
This amounts to a variational approximation to maximizing the mutual information between the third-person ob- servations and the critic’s predictions of the temporal difference targets while minimizing the mutual information between the third-person observations and their representations.
We implement this by replacing the deterministic third-person encoder f ξ 3 with a stochastic encoder p ξ 3 ( z 3 | o 3 ) , specifying a prior p ( z 3 ) , and adding a weighted KL divergence term to the critic loss.
The VIB-regularized
DrQ-v2 critic objective is
[Equation]
where D is the replay buffer.
We specify p ξ 3 ( z 3 | o 3 ) as a diagonal Gaussian and p ( z 3 ) as a standard
Gaussian, which enables analytical computation of the KL divergence.
We use the reparameterization trick to enable optimization of the ﬁrst term via pathwise derivatives.
We do not need to modify the actor objective as only gradients from the critic are used to update the encoder(s) in DrQ-v2.
We remark that a (variational) information bottleneck can be applied to many imitation learning and reinforcement learning algorithms (Peng et al., 2018;
Goyal et al., 2019; Igl et al., 2019; Kumar et al., 2021).
4. E XPERIMENTAL SETUP
We evaluate the learning and generalization performance of seven DrQ-v2 agents: π ◦ O h + p (hand-centric perspective), π ◦ O 3+ p (third-person perspective), π ◦
O h
+3+ p (both perspectives), π ◦
O h
+
+ p + VIB ( z 3 ) (both perspectives with a VIB on the third-person information stream), and three ablation agents introduced later.
We evaluate the agents on six tasks adapted from the Meta-World benchmark (Yu et al., 2020).
We design the task set to exhibit three levels of handcentric observability (high, moderate, and low) with two tasks per level.
In each task, a simulated Sawyer robot manipulates objects resting on a table.
The action space is 4-DoF, consisting of 3-DoF end-effector position control and 1-DoF gripper control.
We do not use the original Meta-World observation space as it contains low-dimensional pose information about task-pertinent objects instead of images.
Rather, we conﬁgure the observations so that O h and O 3 output 84 × 84 RGB images, and O p outputs 3D end-effector position and 1D gripper width.
See Figure 4 for a visualization of each task through the lens of O h and O 3 .
Experiments in Appendix B.5 establish that proprioception alone is not sufﬁcient to reliably solve any of the tasks.
Experiments in Appendix B.6 consider variations of peg-insert-side that require an additional 1-DoF end-effector orientation control.
While the distribution shifts in the experiments of the previous section arise from transformations on the table height, distractor objects, and table textures, in this section we focus on distribution shifts arising from transformations on the initial object positions.
All object positions have disjoint initial train and test distributions such that the latter’s support “surrounds” that of the former (see Tablein Appendix B.2 for details).
[Figure]
cce [Figure] r a t e
[Figure]
Aside from adapting the DrQ-v2 algorithm to our setting as described above
[Figure]
we use the original
DrQ-v2 model and hyperparameters with some minor exceptions (see Appendix B.7 for details).
Hyperparameters that are common to all agents are shared for a given task.
With agents that include regularization, we tune the regularization weight(s) on a validation sample from the test distribution.
Test success rate is computed on a separate sample of 20 environments from the test distribution.
4. E XPERIMENTAL RESULTS AND DISCUSSION
Main experimental results in Meta-World are presented in Figure 5.
Results for ablation studies on π ◦
O h
+
+ p + VIB ( z 3 ) are presented in Figure 12 in Appendix B.4.
Table 2 summarizes both ﬁgures.
When using both perspectives, regularizing the third-person perspective’s representation via a VIB reduces the interquartile mean of the out-of-distribution failure rate across all six tasks by 64% (relative).
We also note that this method achieves the best performance in each individual task, albeit sometimes with less sample efﬁciency.
To properly explain these phenomena, we now embark on a more stratiﬁed analysis and discussion of the results.
Characterization of hand-centric observability via training performance.
When the world state is sufﬁciently observable via the hand-centric perspective, we expect the convergence during training of π ◦ O h + p to match or surpass that of π ◦ O 3+ p .
We ﬁnd that this is indeed the case for handle-pressside, button-press, soccer, and peg-insert-side (high and moderate hand-centric observability), and not the case for reach-hard or peg-insert-side-hard (low hand-centric observability).
This validates our selection and framing of the tasks at different levels of hand-centric observability.
Interestingly, we observe that in peg-insert-side-hard, π ◦ O h + p eventually achieves some success during training by “zooming out” to improve its observability.
[Table]
Hand-centric perspective vs. third-person perspective.
When hand-centric observability is high or moderate, π ◦ O h
+ p generalizes better out-of-distribution than π ◦ O 3+ p , corroborating results from Section 3.2 with another form of distribution shift.
When hand-centric observability is low, π ◦
O h
+ p both trains and generalizes worse than π ◦ O 3+ p .
This supports our motivation for considering using both perspectives in conjunction.
Effect of combining the hand-centric and third-person perspectives.
When hand-centric observability is high or moderate, including the third-person perspective can harm generalization.
We see that for button-press, peg-insert-side, handle-press-side, and soccer, π ◦
O h
+3+ p is sandwiched between π ◦ O h + p and π ◦
O 3+ p on the test distribution.
The drop from π ◦ O h + p to π ◦
O h +3+ p is signiﬁcant for the former two tasks, and marginal for the latter two.
This validates our hypothesis that including O 3 enables the agent to “overﬁt” to training conditions.
When hand-centric observability is low, combining both perspectives results in π ◦ O h +3+ p matching or surpassing the training performance of π ◦
O h
+ p and π ◦
O 3+ p , and greatly outperforming both at test time.
This validates our hypothesis that, when necessary, including third-person observations helps resolve training difﬁculties arising from insufﬁcient hand-centric observability.
Effect of regularizing the third-person information stream via a VIB.
π ◦ O h +3+ p + VIB ( z 3 ) consistently improves upon π ◦
O h +3+ p in out-of-distribution generalization for all tasks except handle-press-side, in which the two are about equal.
This directly indicates the beneﬁt of the VIB regularization.
These gains come at the cost of slightly delaying the convergence of training.
However, it is arguable that this is inevitable and even desirable.
A known phenomenon in neural network training is that spurious correlations or “shortcuts” in the data are sometimes easier to learn than causal relationships (Sagawa et al., 2019).
Slower training and higher generalization may indicate the avoidance of such behavior.
Additionally, in button-press, π ◦ O h
+
+ p + VIB ( z 3 ) recovers the out-of-distribution generalization exhibited by π ◦ O h
+ p , and when hand-centric observability is moderate,
π ◦
O h
+
+ p + VIB ( z 3 ) improves upon π ◦
O h
+ p .
To better understand what makes π ◦
O h +3+ p + VIB ( z 3 ) work the best, we also conduct three ablation studies.
Curves for the ablation studies are presented in Figure 12 in the Appendix.
What if both perspectives are regularized?
The ablation agent π ◦
O h
+
+ p + VIB ( z h )
+
VIB ( z 3 ) adds a separate VIB to the hand-centric information stream in an analogous manner to how the third-person perspective’s representation is regularized (detailed in Section 4.1).
We use the same β 3 for both and tune β h .
Note that setting β h = 0 for π ◦ O h +3+ p + VIB ( z h ) + VIB ( z 3 ) recovers π ◦
O h +3+ p + VIB ( z 3 ) modulo stochasticity in z h , so we limit the lowest value β h can take to 0 .
We ﬁnd that in no task does π ◦
O h +3+ p + VIB
( z h ) + VIB ( z 3 ) outperform
π ◦ O h +3+ p + VIB ( z 3 ) , validating our choice of only regularizing the third-person perspective’s representation.
Assessing the importance of the hand-centric perspective.
π ◦ O 3 (cid:48)
+3+ p + VIB ( z 3 ) uses a second third-person perspective O 3 (cid:48) instead of the hand-centric perspective O h .
Visualizations from this additional third-person perspective are shown in Figure 13 in Appendix B.4.
We re-tune β 3 for this agent.
We ﬁnd that π ◦ O 3 (cid:48) +3+ p + VIB ( z 3 ) performs signiﬁcantly worse than
π ◦
O h +3+ p + VIB ( z 3 ) , afﬁrming the beneﬁt of using the hand-centric perspective in the multi-perspective setting.
z h -dependent regularization of z 3 .
VIB ( z 3 ) reduces the information contained in z 3 without directly considering z h .
With the ablation agent π ◦ O h
+
+ p
+ (cid:96) 2 ( z 3 ) , we consider a simple form of z h -dependent regularization of z 3 in which we push z 3 towards z h by adding a weighted regularization term α 3 (cid:107)
z 3 − stopgrad( z h ) (cid:107) 22 to the DrQ-v2 critic objective.
This approach seems promising given that π ◦
O h
+
+ p consistently outperforms π ◦ O 3+ p across all six tasks, suggesting that even in the midst of substantial partial observability, z h may represent information in a useful and generalizable way.
We tune α 3 .
We ﬁnd that π ◦
O h +3+ p
+ (cid:96) 2 ( z 3 ) marginally improves over vanilla
π ◦ O h +3+ p
but still comes far short of π ◦
O h
+
+ p + VIB ( z 3 ) , suggesting that the two perspectives contain important complementary information that is better represented separately.
R ELATED W ORK
Learning for vision-based object manipulation.
A wide range of works have focused on algorithmic development for end-to-end learning of vision-based object manipulation skills (Levine et al., 2016;
Agrawal et al., 2016;
Finn et al., 2016; 2017; Kalashnikov et al., 2018; Srinivas et al., 2018;
Ebert et al., 2018; Zhu et al., 2018; Jayaraman et al., 2018; Rafailov et al., 2021).
Some works on learned visuomotor control use eye-in-hand cameras for tasks such as grasping (Song et al., 2020) and insertion (Zhao et al., 2020; Puang et al., 2020; Luo et al., 2021; Valassakis et al., 2021), and others which pre-date end-to-end visuomotor learning use both eye-in-hand and third-person cameras for visual servoing (Flandin et al., 2000;
Lippiello et al., 2005).
Very few works consider the design of camera placements (Zaky et al., 2020) or conduct any controlled comparisons on different combinations of visual perspectives (Zhan et al., 2020; Mandlekar et al., 2021; Wu et al., 2021).
Unlike all of these works, we propose speciﬁc hypotheses regarding the beneﬁts of different choices of visual perspective and perform a systematic empirical validation of these hypotheses with evaluation on multiple families of learning algorithms, manipulation tasks, and distribution shifts.
The role of perspective on generalization.
Hill et al.
(2019) assess an agent learning to execute language instructions in simulated environments using high-level actions and ﬁnd that using an egocentric observation space results in better systematic generalization to new instruction nounverb combinations.
Szot et al.
ﬁnd that an agent tasked to pick up a certain object (using abstracted grasping) in a cluttered room generalizes better to unseen objects and room layouts when using wrist- and head-mounted cameras in conjunction.
Our work provides complementary evidence for the effect of perspective on the generalization of learned agents in a markedly different setting: we consider vision-based physical manipulation.
Also, the aforementioned works rely on memoryaugmented agents to resolve partial observability as is common in navigation tasks, whereas we use third-person observations as is standard in tabletop manipulation and demonstrate the importance of regularizing their representation.
Invariances through data augmentation in reinforcement learning.
Several works have investigated ways to apply standard data augmentation techniques from computer vision in the reinforcement learning setting (Laskin et al., 2020;
Kostrikov et al., 2020; Yarats et al., 2021).
These works consider data augmentation as a means to prescribe invariances to image-space transformations, whereas we are concerned with how different observation functions facilitate generalization to environmental transformations.
To emphasize that these directions are orthogonal, we use
DrQ (Kostrikov et al., 2020) and DrQ-v2 (Yarats et al., 2021) in our experiments.
C ONCLUSION
In this work, we focus on empirically studying an underexplored design choice in the embodied learning pipeline: the observation function.
While hand-centric robotic perception is more traditionally instrumented with tactile sensing, our ﬁndings using vision afﬁrm that perspective, even when controlling for modality, can play an important role in learning and generalization.
This insight may very well apply to robotic systems that leverage tactile sensing.
Overall, in the context of end-to-end learning for visuomotor manipulation policies, our ﬁndings lead us to recommend using hand-centric perspectives when their limited observability is sufﬁcient, and otherwise defaulting to using both hand-centric and third-person perspectives while regularizing the representation of the latter.
The breadth of the learning algorithms, manipulation tasks, and distribution shifts that we base these conclusions on, coupled with their simplicity and lack of restrictive assumptions, suggests that these recommendations should be broadly applicable, even to more complex, longer-horizon tasks that feature sub-tasks analogous to those we experiment with.
R EPRODUCIBILITY S TATEMENT
Sections 3.1 and 4.2 give overviews of the experimental setup, and Appendices A and B ﬂesh out the full experimental protocol in stringent detail.
We expect this to be sufﬁcient for independent replication of our main ﬁndings.
Separately, for the sake of reproduction, we will also release code for all experiments upon publication
A C UBE G RASPING E XPERIMENT D ETAILS A. E NVIRONMENT D ETAILS
For the cube grasping experiments in Section
[Table]
we investigate three types of distribution shifts.
The experiment variants for DAgger and DrQ are summarized in Table 3.
The DAC experiments featured a subset of these conditions explained in the caption of Figure 3.
Figures 6,
[Table]
and 8 visualize each type of distribution shift.
[Table]
A. A LGORITHMS
The dataset aggregation (DAgger) algorithm proposed by Ross et al.
(2011) is an iterative online algorithm for training an imitation learning policy.
In each iteration i (which we call a “DAgger round”), the current policy π i is run to sample a set of trajectories, and an expert policy π ∗ is used to label each of the visited states with an optimal action.
These labeled trajectories are aggregated into a dataset D that grows in size over the DAgger rounds, and the imitation learning policy ˆ π i is trained on the entire D for some number of epochs before repeating the above procedure in the next iteration.
The trajectory-generating policy π i is often modiﬁed such that in earlier DAgger rounds the expert policy π ∗ is utilized more heavily than the imitation learning policy ˆ π i when collecting new trajectories, i.e. π i = β i π ∗ +(1 − β i )
ˆ π i , where β i is typically annealed over time (e.g., linearly from 1 to 0 over the DAgger rounds).
The Data-regularized Q (DrQ) algorithm proposed by Kostrikov et al.
(2020) is a model-free, offpolicy, actor-critic reinforcement learning algorithm that applies image augmentation techniques commonly used in computer vision (primarily random shifts) to input images, along with regularizations of the Q target and function, such that deep neural network-based agents can be trained effectively from pixels.
The original DrQ paper uses soft actor-critic (Haarnoja et al., 2018) and
DQN (Mnih et al., 2013) as backbones; we use the soft actor-critic version in our experiments because the cube grasping action space is continuous.
The discriminator actor-critic algorithm (DAC) was proposed in Kostrikov et al.
and is an offpolicy version of the generative adversarial imitation learning (GAIL) method (Ho & Ermon, 2016).
Unlike Kostrikov et al.
we use a deterministic reinforcement learning algorithm similar to that of Fujimoto et al.
(2018), as we ﬁnd this helps stability.
To scale the method to image observations, we apply similar augmentation techniques as in Kostrikov et al.
A. M ODEL A RCHITECTURES
For DAgger in the cube grasping experiments discussed in Section 3, we feed the 84 × 84 images into a ResNet-18 convolutional image encoder (He et al., 2016) trained from scratch, with the ﬁnal classiﬁcation layer replaced by a linear layer that outputs a 64-dimensional representation.
We concatenate proprioceptive information (3D end-effector position relative to the robot base, 1D gripper width, and a Boolean contact ﬂag for each of two gripper “ﬁngers”) to the image representation, and the result is passed into feedforward policy and value networks with two hidden layers of 32 units each.
For DrQ, we use the original actor-critic DrQ model proposed by Kostrikov et al.
(2020), except for one modiﬁcation: we concatenate proprioceptive information (3D end-effector position relative to the robot base, 1D gripper width, and a Boolean contact ﬂag for each of two gripper “ﬁngers”) to the ﬂattened image representation before feeding it into the actor and critic networks.
For the DAC algorithm we use the same convolutional architectures as Kostrikov et al.
The convolutional encoder is shared between the discriminator, actor and critic.
We use additional MLP heads with capacities 128, 256 and 256 respectively for those components, as we empirically found that lower-capacity networks decrease the likelihood of overﬁtting to spurious features.
A. H YPERPARAMETERS
The DAgger, DrQ, and DAC hyperparameters used in the cube grasping experiments are listed in
Tables 4, 5, and 6, respectively.
[Table] [Table]
[Table]
A. A BLATION S TUDY : R EMOVING THE D ATA A UGMENTATION IN D R Q
In this experiment
[Figure]
we investigate the effect of the data augmentation component of the DrQ algorithm by ablating it.
The motivation is to see whether data augmentation is still necessary for a policy using the hand-centric perspective, which already leads to lower overﬁtting and better generalization.
The results in Figure 9 reveal that the augmentation is indeed still crucial because without it, training does not converge even with much more environment interaction.
However, the handcentric perspective does still enable the agent to make greater progress
[Figure]
seeds.
A. M INOR D ISCREPANCIES BETWEEN A LGORITHMS
Due to implementation idiosyncrasies, there are minor discrepancies in how each algorithm processes environment observations.
Following Kostrikov et al.
(2020), for DrQ and DAC-DrQ observations are “frame-stacked” with three time steps’ observations.
This was not done for DAgger.
Proprioceptives are used for DAgger and DrQ but not for DAC-DrQ. We take the position that these differences increase the generalizability of the trends we observe.
We emphasize that the target effect under consideration is O h vs. O 3 in each setting.
B M ETA -W ORLD E XPERIMENT D ETAILS B. I NDIVIDUAL T ASK D ESCRIPTIONS
In this section we explain the tasks that the agents must learn to accomplish in the six Meta-World environments discussed in Section 4.2 and visualized in Figure 4.
We also explain why each task falls under a certain level of hand-centric observability.
For details regarding the train and test distributions, see Appendix B.2.
• handle-press-side: The goal is to press the handle fully downwards.
Hand-centric observability is high because the handle is well aligned with the hand-centric camera’s ﬁeld of view.
• button-press: The goal is to push the button fully inwards.
Hand-centric observability is high because the button is well in view of the hand-centric camera, and the button remains largely in view as the gripper approaches and presses it.
• soccer:
The goal is to push or pick-and-place the ball into the center of the goal net.
Handcentric observability is moderate because when the gripper approaches the ball, the observability of the goal net is appreciably reduced.
• peg-insert-side: The goal is to lift the peg and insert it into the hole in the target box.
Hand-centric observability is moderate because when the gripper approaches the peg, the observability of the target box is appreciably reduced.
• reach-hard: The goal is to move the gripper to the green goal site, which is initialized either to the left or right side of the gripper with equal probability (see Figure 10).
Hand-centric observability is low because the gripper is initialized at the same height as the goal, and we restrain the gripper from moving vertically.
Effectively, if given just the hand-centric perspective’s observations, the agent does not know in which direction to move the gripper in the beginning of an episode.
• peg-insert-side-hard: The goal is the same as in peg-insert-side, but like the green goal site in reach-hard, the
peg in this environment is initialized either to the left or right side of the gripper with equal probability (see Figure 10).
Hand-centric observability is low because the gripper is initialized at the same height as the peg such that the peg is not initially visible to the hand-centric view (though we do not prohibit vertical movement of the gripper as in reach-hard, since this would make the peg insertion part of the task impossible), and also because the peg and target box are initialized much farther apart than they are in peg-insertside (thus, the target box is completely out of view as the agent approaches and grasps the peg).
B. T RAIN AND T EST D ISTRIBUTIONS
At training time, initial positions of the objects in the Meta-World tasks are uniformly sampled within some support.
At test time, initial positions are sampled from a uniform distribution that is completely disjoint from the training distribution, such that we test on out-of-distribution initial object positions.
To implement this, at test time we resample the set of initial object positions if any of the positions overlaps with its train-time distribution.
The full set of train-time and test-time initial object positions is shown in Table 7.
For visualizations, see Figure 11.
[Table]
[Table]
if goal is on left of gripper: if goal is on left of gripper:
[Table]
if goal is on right of gripper: if goal is on right of gripper:
[Table]
if peg
[Table]
on left of gripper
[Table]
on
[Table]
gripper
[Table]
if peg starts on right of gripper: if peg starts on right of gripper:
[Table]
B.
D R Q- V
DrQ-v2 (Yarats et al., 2021) is a state-of-the-art vision-based actor-critic reinforcement learning algorithm that uses deep deterministic policy gradients (DDPG) (Lillicrap et al., 2015) as a backbone (whereas DrQ-v1 by Kostrikov et al.
(2020) uses soft actor-critic).
The DrQ-v2 model includes:
• a convolutional image encoder f ξ that outputs representation z = f ξ (aug( o )) given framestacked image observations o and a data augmentation function aug , • two critic networks Q θ k that output Q-values Q θ k
( z , a ) , k = 1 , 2 , `a la clipped double Qlearning (Fujimoto et al., 2018), • and an actor network π φ that outputs action a = π φ ( z )+ (cid:15), (cid:15) ∼ N (0 , σ 2 ) , with σ 2 annealed over the course of training
The individual critic losses are given by
[Equation]
(2) where τ = ( o t , a t , r t : t + n − 1 , o t + n ) is a sample from replay buffer D and y is the temporal difference target estimated via n -step returns:
[Equation]
(3) for slow-moving critic weights ¯ θ 1 , ¯ θ 2 .
We omit presentation of the actor loss as we do not need to modify it; in DrQ-v2, only gradients from the critic loss are used to update the weights of the encoder(s).
In terms of the model architecture used in the experiments discussed in Section 4, we use the original
DrQ-v2 architecture, except for two modiﬁcations: ﬁrst, we concatenate proprioceptive information (3D end-effector position and 1D gripper width) to the ﬂattened image representation before feeding it into the actor and critic networks.
Second, when using two perspectives at the same time (e.g., hand-centric and third-person), we use two separate image encoders that do not share weights.
The two representations are concatenated together (along with the proprioceptive information) and fed into the actor and critic networks.
The dimensionality of each encoder’s output representation is preserved, thereby doubling the dimensionality of the ﬁnal combined image representation.
B. A BLATIONS ON π ◦ O h +3+ p + VIB ( z 3 )
Figure 12 presents the train and test curves of the ablation experiments presented in Section 4.3.
Figure 13 contains illustrations of the second third-person perspective used in π ◦ O 3 (cid:48)
+3+ p + VIB ( z 3 ) .
[Figure]
B. P ROPRIOCEPTION - ONLY A BLATION
In this ablation experiment
[Figure]
we demonstrate that visual observations are a necessary component of the observation space, i.e. the tasks we experiment with cannot be consistently solved with proprioceptive observations alone.
We run DrQ-v2 on all six Meta-World tasks introduced in Section 4.2 without image observations and show the results in Figure 14.
Unlike policies that are afforded vision, these proprioception-only policies do not approach 100% success rate on the training distributions.
[Figure]
cce ss r a
[Figure]
B. E XPERIMENTS WITH E ND -E FFECTOR O RIENTATION C ONTROL
The experiments in Section 4 involved a 4-DoF action space consisting of 3-DoF end-effector position control and 1-DoF gripper control, which was sufﬁcient for solving all of the Meta-World tasks.
In this section, we add one more degree of freedom for end-effector orientation control (allowing the parallel-jaw gripper to swivel) and then construct and experiment on two modiﬁed versions of the peg-insert-side task that cannot be solved without end-effector rotations.
The train and test distributions of initial object center-of-masses are the same as those in the original peg-insert-side task.
In the ﬁrst modiﬁed version, the end-effector is initially rotated 90 degrees from its original orientation, forcing the agent to rotate the end-effector before grasping the peg (see the center column of Figure 15 for a visualization).
The second modiﬁed version of the task includes the following changes: (1) the proprioceptive observations also include the end-effector’s orientation (as a quaternion), and (2) the peg—not the end-effector—is initially rotated by 90 degrees (see the rightmost column of Figure 15 for a visualization).
Not only does (2) force the agent to rotate the end-effector before grasping the peg, but it also requires the agent to re-orient the peg correctly before inserting it into the box.
The experimental results for DrQ-v2 in these two new environments are shown in
Figure 16.
Figure 16)
[Figure]
B. H YPERPARAMETERS
We present the DrQ-v2 hyperparameters used in the Meta-World experiments in Table 8.
The conﬁguration is largely identical to the one used in the original DrQ-v2 algorithm (Yarats et al., 2021)
[Table] [Table]
weight β 3 of KL div.
term in 500 for { soccer, peg-insert-side-hard } ; π ◦ O h +3+ p + VIB ( z 3 ) 50 for peg-insert-side w/ rotated initial gripper (for Appendix B.6) 1 for peg-insert-side w/ rotated initial peg (for Appendix B.6) 10 for all other environments weights β h , β 3 of KL div.
terms in β h : π ◦
O h
+
+ p + VIB ( z h )
+ VIB ( z 3 )
[Table]
.1 for { button-press, reach-hard, soccer } ;
[Table]
for all other environments β 3 : same as β 3 of KL div.
term in π ◦
O h +3+ p + VIB ( z 3 ) weight β 3 of KL div.
term in 0 for handle-press-side; π ◦ O 3 (cid:48) +3+
[Table]
[Table]
weight α 3 of
[Table]
reg
[Table]
term in 10 8 for
[Table]
environments
[Table]
C M ISCELLANEOUS D ETAILS
We applied an exponentially weighted moving average ﬁlter on the data for DrQ in Figure 2 ( α = 0 . 6 ), for DAC in Figure 3 ( α = 0 . 3 ), and for DrQ-v2 in Figures 5 and 12 ( α = 0 . 5 ) to smoothen the train and test curves for increased readability.
The smoothing factor α lies in the range [0 , 1] , where values closer to 0 correspond to more smoothing.