Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation
[Figure]
E go ce n t r i c
Th i r d - P e r s o n
[Figure]
We
Abstract
Learning to solve precision-based manipulation tasks from visual feedback using
Reinforcement Learning (RL) could drastically reduce the engineering efforts required by traditional robot systems.
However, performing ﬁne-grained motor control from visual inputs alone is challenging, especially with a static thirdperson camera as often used in previous work.
We propose a setting for robotic manipulation in which the agent receives visual feedback from both a third-person camera and an egocentric camera mounted on the robot’s wrist.
While the thirdperson camera is static, the egocentric camera enables the robot to actively control its vision to aid in precise manipulation.
To fuse visual information from both cameras effectively, we additionally propose to use Transformers with a crossview attention mechanism that models spatial attention from one view to another (and vice-versa), and use the learned features as input to an RL policy.
Our method improves learning over strong single-view and multi-view baselines, and successfully transfers to a set of challenging manipulation tasks on a real robot with uncalibrated cameras, no access to state information, and a high degree of task variability.
In a hammer manipulation task, our method succeeds in 75% of trials versus 38% and 13% for multi-view and single-view baselines, respectively.
Introduction
Most solutions for robotic manipulation today rely on highly structured setups that allow for full state information, ﬁne-grained robot calibration, and predeﬁned action sequences.
This requires substantial engineering effort, and the resulting system is intolerant to changes in the environment.
Visual feedback from a mounted camera has gained popularity as an inexpensive tool for relaxing the 35th Conference on Neural Information Processing Systems (NeurIPS 2021). assumption of full state information [28, 30, 44, 48], and Reinforcement Learning (RL) has emerged as a promising technique for learning ﬂexible control policies without the need for detailed human engineering [43, 39, 31, 14].
Together, vision-based RL could enable use of robots in unstructured environments through ﬂexible policies operating directly from visual feedback, without assuming access to any state information [29, 35, 32, 49].
However, solving complex precision-based manipulation tasks in an end-to-end fashion remains very challenging, and current methods often rely on important state information that is difﬁcult to obtain in the real world [13, 30, 22, 2, 33], and/or rely on known camera intrinsics and meticulous calibration [44, 36] in order to transfer from simulation to the real world.
We argue that for a system to be truly robust, it should be able to operate from visual feedback, succeed without the need for camera calibration, and be ﬂexible enough to tolerate task variations, much like humans.
In particular,
Land et al.
[27] ﬁnd that when humans perform a complex motor task, the oculo-motor system keeps the centre of gaze very close to the point at which information is extracted, also known as visual ﬁxation .
However, previous work in visual RL often limit themselves to a single, static third-person monocular camera, which makes extraction of local information from areas of interest challenging, and uncalibrated cameras only exacerbate the problem.
In this work, we propose a setting for vision-based robotic manipulation in which the agent receives visual feedback from two complementary views: (i) a third-person view ( global information), and (ii) an egocentric view ( local information), as shown in Figure 1.
While the third-person view is static, the egocentric camera moves with the robot gripper, providing the robot with active vision capabilities analogous to the oculo-motor system in humans.
Because the agent has control over its egocentric vision, it can learn to actively position its camera such that it provides additional information at regions of interest, e.g. accurate localization of points of contact in ﬁne-grained manipulation tasks.
To fuse visual feedback from the two views effectively, we propose to integrate an explicit modeling of visual ﬁxation through a network architecture with soft attention mechanisms.
Speciﬁcally, we propose to use Transformers [40] with a cross-view attention module that explicitly models spatial attention from one view to another.
Each view is encoded using separate ConvNet encoders, and we fuse their corresponding feature maps by applying our cross-view attention module bidirectionally.
In this way, we let every spatial region in the egocentric view highlight the corresponding regions of interest in the third-person view, and vice-versa.
We use the learned features as input for an RL policy and optimize the network end-to-end using a reward signal.
This encourages the agent to actively control the egocentric camera in a way that maximizes success.
To demonstrate the effectiveness of our method, we conduct extensive empirical evaluation and compare to a set of strong baselines on four precision-based manipulation tasks: (1) Reach , a task in which the robot reaches for a goal marked by a red disc placed on the table, (2) Push , a task in which the robot pushes a cube to a goal marked by a red disc, both placed on the table,
(3) Peg in Box , a task in which the objective is to insert a peg tied to the robot’s end-effector into a box placed on the table, (4) Hammer , a task in which the objective is to hammer in an out-of-position peg.
Each of the four tasks are shown in Figure 1.
We ﬁnd that our multi-view setting and cross-view attention modules each improve learning over single-view baselines, and we further show that our method successfully transfers from simulation to a real robot setup (shown in Figure 2) with uncalibrated cameras, no access to state information, and a high degree of task conﬁguration variability.
In the challenging Hammer task, our method is signiﬁcantly more successful during transfer than any of our baselines, achieving a success rate of 75 % versus 38% for a multi-view baseline without cross-view attention, and only 13% and 0 .
0% for our single-view baselines.
Finally, we qualitatively observe that (i) multi-view methods appear less prone to error due to lack of camera calibration, and (ii) our proposed cross-view attention mechanism improves precision in tasks that require ﬁne-grained motor control.
Related Work
Vision-Based Robotic Manipulation.
Learning RL policies for end-to-end vision-based robotic manipulation via task supervision has been widely explored [29, 31, 35, 32, 14, 25, 45, 49].
For example, Lillicrap et al.
[31] solve simulated continuous control tasks directly from images with no access to state information, and Zhan et al.
[49] shows that vision-based RL can solve real-world manipulation tasks efﬁciently.
Despite this progress it is still very challenging – especially for vision-based policies – to transfer learned skills to other environments, e.g. Sim2Real [34, 9, 16].
To facilitate transfer, related work also leverage important state information such as object pose, which is readily available in simulation but difﬁcult to obtain in the real world [13, 30, 22, 2, 36, 33].
In comparison, our approach learns directly from raw images and transfers to a real robot with uncalibrated cameras.
Multi-View Robotic Manipulation.
Using multiple views in robotic manipulation remains a relatively unexplored topic that has only recently gained interest [1, 33, 6].
For example, Akinola et al.
[1] show that third-person views from multiple cameras improves policy learning in precision-based manipulation tasks.
In addition to camera inputs, they assume access to state information from the gripper.
OpenAI et al.
[33] explore a multi-view robotic manipulation setting similar to ours, but they require privileged state information that is not easily available in the real world.
Concurrent to our work, Chen et al.
[6] propose a framework for learning 3D keypoints for control from multiple third-person views.
All of the aforementioned works only consider manipulation tasks in simulation .
In this work, we develop a real robotic system that operates strictly from uncalibrated egocentric and third-person cameras.
Active Vision for Manipulation.
Robots operating only from egocentric images suffer from incomplete state information (occlusion, small ﬁeld-of-view), while when operating only from third-person vision fail to capture more accurate interactions.
A natural solution to this problem is active vision.
An active vision system is one that can manipulate the viewpoint of the camera(s) in order to investigate the environment and get better information from it [5, 42].
Traditional methods for active vision [8] either use hand-crafted utility functions [26] or are uncertainty or reconstruction based [12], whereas learning based [18, 23] approaches explicitly learn these utility functions for selecting the next-best view using ground truth labels.
In this work, we use the task-based reward to guide active vision.
Our approach employs an egocentric camera tied to the wrist of the robot for active vision.
Attention Mechanisms in RL.
Our multi-view fusion builds upon the soft attention mechanism which has been widely adopted in natural language processing [40, 10] and computer vision [41, 11], and has recently been popularized in the context of RL [21, 7, 20, 15, 46, 19].
For example, Jiang et al.
[21] propose an attention-based communication model that scales to cooperative decision-making between a large amount of agents, and Yang et al.
[46] use a cross-modal Transformer to fuse state information with a depth input in simulated locomotion tasks.
James et al.
[19] is a parallel work that is closest to our approach.
However, like most other works on image-based RL, they restrict their study to a single, ﬁxed third-person view.
We propose a cross-view attention mechanism that fuses information from multiple camera views, and we are – to the best of our knowledge – the ﬁrst to demonstrate the effectiveness of attention-based policies in transfer from simulation to a real robot setup.
Background
Vision-Based Reinforcement Learning.
We formulate interaction between the agent’s vision-based control policy and environment as an inﬁnite-horizon Partially Observable Markov Decision Process (POMDP) [24] described by a tuple ( S , A , P , p 0 , r, γ ) , where S is the state space,
A is the action space, P : S × A (cid:55)→ S is a transition function that deﬁnes a conditional probability distribution
P ( ·| s t , a t ) over possible next states given current state s t ∈ S and action a t ∈ A taken at time t , P 0 is a probability distribution over initial states, r :
S × A
(cid:55)→ R is a scalar reward function, and γ ∈ [0 , 1) is a discount factor.
In a POMDP, the underlying state s of the system is assumed unavailable, and the agent must therefore learn to implicitly model states from raw image observations.
Our goal is to learn a policy π : S (cid:55)→ A that maximizes return E Γ ∼ π [ (cid:80) ∞ t =
γ t r ( s t , a t )] along a trajectory Γ = ( s 0 , s 1 , . .
, s ∞ ) where s 0 ∼ P 0 , a t ∼ π ( ·| s t ) , and s t +1 ∼ P ( ·| s t , a t ) , and we use the Soft
Actor-Critic [14] RL algorithm for policy search.
Soft Actor-Critic (SAC) [14] is an off-policy actor-critic algorithm that learns a (soft) state-action value function Q θ :
S × A (cid:55)→ R , a stochastic policy π θ as previously deﬁned, and optionally a temperature parameter τ θ .
Q θ is optimized to minimize the (soft) Bellman residual [38], π θ is optimized using a maximum entropy objective [4, 50], and τ θ is optimized to maintain a desired expected entropy; see [14] for further details.
For brevity, we generically refer to learnable parameterization by a subscript θ throughout this work. [Figure] [Figure]
Robot Setup.
Our real-world setup is shown in Figure 2.
We use a Ufactory xArm 7 robot with an xArm Gripper as the robot platform in both our real-world and simulation experiments, although our method is agnostic to the speciﬁc robot hardware.
The camera poses and the shape and size of objects in the simulation roughly mimic that of the real-world.
We apply the same pre-processing to both the simulated and real images.
Image samples from the simulation and real-world robot setup can be seen in Figure 3.
Two cameras are mounted to overlook the robot workspace where the task is being performed.
With respect to the robot, one camera is directly placed in front of the whole setup (third-person view) with a large ﬁeld of view covering the robot and the workspace, and the other camera (egocentric view) is attached to the robot at its wrist, i.e. right above where the gripper meets the robot.
We emphasize that the cameras are uncalibrated , that the policy operates from RGB images ( 84 × 84 pixels) captured by the two cameras, and that the policy has no access state information
Method
Inspired by visual ﬁxation in the human eye, we propose to fuse egocentric and third-person views in a multi-camera robotic manipulation setting, leveraging Transformers for explicit modeling of ﬁxation through its attention mechanism.
Our method is a general framework for multi-view robotic manipulation that is simple to both implement and deploy in a real-world setting.
For simplicity, we describe our method using SAC [14] as backbone learning algorithm as used in our experiments, but we emphasize that our method is agnostic to the particular choice of learning algorithm.
We introduce each component of our method in the following.
La y e r N o r m
L i nea r A c t i v a t i on A tt en t i on M odu l e
Query
Key
Value aug aug
E go ce n t r i c
Th i r d - P e r s on
A c t o r
C r i t i c
SA C
Lea r ne r
Softmax
Scale
Attention Module Cross-View Attention
La y e r N o r m
L i nea r A c t i v a t i on L i nea r A tt en t i on M odu l e
L i nea r
[Figure]
We propose to learn vision-based RL policies for robotic manipulation tasks using raw, uncalibrated RGB inputs from two complementary camera views: (i) an egocentric view O 1 from a camera mounted on the wrist of the robot and (ii) a ﬁxed third-person view O 2 .
The third-person view provides global information about the scene and robot-object relative conﬁgurations, while the egocentric view serves to provide local information for ﬁne-grained manipulation; see Figure 3 for samples.
Because the egocentric camera is attached to the robot, it provides the robot with active vision capabilities – control over one of the two views.
Our method learns a joint latent representation from the two camera views, encouraging the model to relate global and local information, and actively position itself such that the egocentric view provides valuable local information for ﬁne-grained manipulation.
4.2 Network Architecture
Figure 4 provides an overview of our method and architecture.
Our proposed architecture takes two raw RGB images O 1 , O 2 ∈ R C × H × W from the egocentric and third-person cameras, respectively (s.t.
the system state s is approximated by { O 1 , O 2 } ), and encodes them separately using ConvNet encoders f θ 1 and f θ 2 to produce latent feature maps Z 1 , Z 2 ∈ R C (cid:48)
× H (cid:48) × W (cid:48) .
We propose to fuse the two feature maps using a Transformer [40] with a cross-view attention module that takes Z 1 , Zas input and produces a single feature map H =
T θ ( Z 1 , Z 2 ) .
The aggregated features H are ﬂattened and fed into the policy (actor) π θ and state-action value function (critic) Q θ s.t.
a ∼ π θ ( ·| T θ ( f θ 1 ( O 1 ) , f θ 2 ( O 2 ))) and similarly for the critic.
In practice, we optimize all components end-to-end using the actor and critic losses of SAC, but only back-propagate gradients from the critic to encoder components T θ , f θ 1 , f θ 2 , as is common practice in image-based RL [47, 37, 17, 25].
In the following section, we describe our cross-view attention mechanism in more detail.
4.3 Cross-View Attention
Our Transformer T θ learns to perform cross-view attention by associating spatial information between intermediate feature maps Z 1 , Z 2 from the egocentric view O 1 and third-person view O 2 , respectively.
T θ takes as input Z 1 , Z 2 and produces a joint embedding
[Equation]
where g θ 1 , g θ 2 are Multi-Layer Perceptrons (MLP), LN is a LayerNorm [3] normalization, and
A 12 , A 21 are normalized (soft) scaled dot-product attention [40] weights
[Equation]
) for Q i , K j , V j ∈ R C (cid:48)
× H (cid:48) × W (cid:48) denoted as the queries, keys, and values, respectively, for the crossview attention between Z i and Z j , and σ is a Softmax normalization.
Q i , K j , V j are view-dependent embeddings
[Equation]
where ψ Qθi , ψ Kθj , ψ Vθj are 1 × 1 convolutional layers.
Intuitively, each of the two cross-view attention mechanisms between feature maps Z i , Z j can be interpreted as a differential (spatial) lookup operation, where Z i is used as query to retrieve information in Z j .
By performing cross-view attention bidirectionally as shown in Equation 2, T θ enables ﬂow of spatial information both from the egocentric view to the third-person view and vice-versa.
With spatial attention, the policy can learn to associate concepts present in both views, e.g. objects or the gripper as shown in Figure 5, and enhance both object-centric features as well as the overall 3D geometry of the scene.
We note that, while we consider two views in this work, our method can trivially be extended to n views by letting T θ compute cross-view attention bidirectionally between all pairs of views.
4.4 Data Augmentation
To aid both learning in simulation and transfer to the real world, we apply data augmentation to image observations, both in our method and across all baselines.
During training, we sequentially apply stochastic image shift [25] and color jitter augmentations to each view independently after sampling from the replay buffer, i.e. O aug1 =
aug( O 1 , ζ 1 ) , O aug2 =
aug( O 2 , ζ 2 ) , ζ 1 , ζ 2 ∼ Ω where ζ 1 , ζparameterize the augmentations and Ω is a uniform distribution over the joint augmentation space.
Experiments
We investigate the effects of our proposed multi-view setting as well as our Transformer’s cross-view attention mechanism on a set of precision-based robotic manipulation tasks from visual feedback.
Both our method and baselines are trained entirely in simulation using dense rewards and randomized initial conﬁgurations of the robot, goal, and objects across the workspace.
We evaluate methods both in the simulation used for training, and a real robot setup as described in Section 3.
During evaluation, agents have no access to reward signals and are expected to generalize without any trials nor prior knowledge about the test conditions.
To ensure consistent results, we report success rate over a set of pre-deﬁned goal and object locations both in simulation and the real world.
Goal locations vary between tasks, and the robot is reset after each trial.
We ﬁrst detail our experimental setup and then discuss our ﬁndings.
Tasks.
Figure 1 provides samples for each task and view considered.
Using the simulation setup as described in Section 3, we consider the following tasks: (1) Reach , a task in which the robot reaches for a goal marked by a red disc placed on the table.
Success is considered when the robot gripper reaches within 5 cm of the goal.
(2) Push , a task in which the robot pushes a cube to a goal marked by a red disc, both placed on the table.
Success is considered when the cube reaches within 10 cm of the goal.
(3) Peg in Box , a task in which the objective is to insert a peg tied to the robot’s end-effector
[Figure]
[Table]
into a box placed on the table.
Success is considered when the peg reaches within 5 cm of the goal and as a result gets inserted into the box.
(4) Hammer , a task in which the objective is to hammer in an out-of-position peg.
At each episode, one peg is randomly selected from 4 differently colored pegs.
Success is considered when the out-of-position peg is hammered back within 1 cm of the box.
Both the Reach and Push tasks use an XY action space, with movement along Z constrained.
Peg in Box and Hammer use an XYZ action space.
Episodes are terminated when the policy succeeds, collides with the table, or a maximum number of time steps is reached.
An episode is counted as a success if the success criteria deﬁned above is met for any time step in the episode.
Setup.
We implement our method and all baselines using Soft Actor-Critic (SAC) [14] as learning algorithm and, whenever applicable, we similarly use the same network architecture and hyperparameters adopted from previous work on image-based RL [17, 15].
As in previous work, f θ consists ofconvolutional layers.
Observations are RGB images of size 84 × 84 from either one or two cameras depending on the method.
Although related works commonly approximate the system state s using a stack of frames, we empirically ﬁnd a single frame sufﬁcient for both learning and transfer.
All methods are trained for 500 k frames and evaluated for 30 trials ( 24 in Hammer : 6 for each peg across 3 object locations), both in simulation and on the real robot.
In simulation, all results are averaged over 3 model seeds; in real world experiments, we report transfer results for the best seed of each method due to real world constraints.
Baselines.
We compare our method to a set of strong baselines, all using Soft Actor-Critic (SAC) as learning algorithm and the same choice of network architecture, hyperparameters, and data augmentations as our method, whenever applicable.
Speciﬁcally, we compare our method to the following baselines: (1) Third-Person View ( 3rd ) using visual feedback from a ﬁxed third-person camera, (2) Egocentric View ( Ego ) using only the egocentric camera mounted on the robot’s wrist, (3) Multi-View ( Multi ) that uses both views as input, encodes each view using separate encoders, and aggregates extracted features using addition.
We emphasize that all baselines are implemented using the same image augmentations (image shift and color jitter) as our method, which makes them largely equivalent to DrQ [25], a state-of-the-art algorithm for image-based RL that builds on SAC.
Lastly, we note that (3) implements our method without cross-view attention and is therefore analogous to the method proposed by Akinola et al.
[1] but for a combination of third-person and egocentric views, without state information from the gripper, and including recent advances in image-based RL such as augmentations.
In addition to these baselines, we also ablate design choices in our cross-view attention mechanism.
[Table]
u r s
M u l t iV i e w
T h
i r d - P e r s on
[Table]
5.1 Robotic Manipulation in Simulation
Before discussing our real robot experiments, we ﬁrst consider methods in simulation.
Training performance (in terms of episode return) of our method and baselines is shown in Figure 6.
Our method using both multi-view inputs and a Transformer performs on par or better than baselines on all tasks, and we particularly observe improvements in sample efﬁciency on the Peg in Box task that requires 3D geometric understanding.
All methods except the egocentric-only baselines trivially solve the Reach task, but we include it to better ground our results.
We conjecture that the egocentric baseline performs signiﬁcantly worse than other methods due to its lack of global scene information.
The corresponding success rates for each task and method after training for 500k frames are shown in
Table 1.
Interestingly, while a third-person view is sufﬁcient for learning to solve 3 out of 4 tasks, we observe substantial improvements in success rate in both our method ( + 56 % ) and multi-view ( +20%) , which is not immediately obvious from the episode returns.
Qualitatively, we ﬁnd that the third-person baseline often approaches the peg but fails to hit it, whereas our method is comparably better at precise manipulation.
5.2 Sim2Real Transfer
We now consider deployment of the learned policies in a Sim2Real setting, i.e. the policies trained in simulation are transferred to a real robot setup as shown in Figure 3.
Success rates on the real robot are shown in Table 2.
[Table]
We ﬁnd that the success of the single-view baselines drops considerably when transferring to the real world.
For example, third-person baseline ( 3rd ) achieves success rates of 75% and 80% for the Push and Peg in Box tasks, respectively, in simulation, while merely 10% and 23% in the real world.
We conjecture that this drop in performance is due to the reality gap – and the lack of camera calibration in particular.
With the addition of an egocentric view, the multi-view baseline improves transfer to 23% and 50% on the two Push and Peg in Box tasks.
Finally, we observe no drop in success for our method on the two tasks, achieving a success rate of 80% in both tasks, and only a small drop in success rate on the challenging Hammer task that requires 3D understanding and a high level of precision.
Speciﬁcally, our method succeeds in 75% of trials in the Hammer task versus only 38% and 13% for the multi-view and single-view baselines, respectively.
As such, view aggregation using
Transformers appear to be a promising research direction.
Similar to our simulation experiments, we also study the qualitative behavior of policies when transferring to the real world.
Figure 7 shows sample trajectories for our method, the multi-view baseline, and the third-person baseline on the Hammer task.
In the Hammer task, we observe that the multi-view baseline frequently misses its target by a small margin (presumably due to the reality gap), and the third-person baseline systematically fails to reach the peg, which we conjecture is due to error in 3D perception from the uncalibrated camera.
Using our method, we observe that the robot frequently positions its gripper above the peg, such that it is visible from the egocentric view during the hammering motion.
Finally, we also evaluate the qualitative behavior of the cross-view attention module.
Attention maps for a set of spatial queries are shown in Figure 5.
We ﬁnd that the agent often attends to regions of interest, such as objects or the gripper.
Transformer-based policies could therefore also be a promising technique for explainability in RL.
5.3 Ablations
Our experiments discussed in Section 5.1 and 5.2 ablate the choice of camera views.
Our method learns cross-view attention between the egocentric view and the third-person view and models each of the directions individually, i.e. an attention map A 12 is computed for egocentric → third-person direction, and A 21 is likewise computed for the opposite direction.
We ablate each of these two modules such that cross-view attention is only learned unidirectionally.
Our ablations indicate that unidirectional cross-view attention achieves similar success rates to that of the multi-view baselines, only improving marginally in the Hammer task for the egocentric → third-person direction.
In comparison, our proposed bidirectional formulation of cross-view attention succeeds more often than either ablation.
We conjecture that letting both views attend to each other improves ﬂow of information, which in turn improves the expressiveness of the learned joint representation.
Conclusion
Precise robotic manipulation from visual feedback is challenging.
Through experiments in both simulation and the real world, we observe that both our multi-view setting as well as our proposed cross-view attention mechanism improves learning and in particular improves transfer to the real world, even in the challenging setting of Sim2Real with uncalibrated cameras, no state information, and a high degree of task variability.
Therefore, we believe our proposed problem setting and method is a promising direction for future research in robotic manipulation from visual feedback.