Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds
Abstract: We study how choices of input point cloud coordinate frames affect learning of manipulation skills from 3D point clouds.
There exist a variety of coordinate frame choices to normalize captured robot-object-interaction point clouds.
We ﬁnd that different frames have a profound effect on agent learning performance, and the trend is similar across 3D backbone networks.
In particular, the end-effector frame and the target-part frame achieve higher training efﬁciency than the commonly used world frame and robot-base frame in many tasks, intuitively because they provide helpful alignments among point clouds across time steps and thus can simplify visual module learning.
Moreover, the well-performing frames vary across tasks, and some tasks may beneﬁt from multiple frame candidates.
We thus propose FrameMiners to adaptively select candidate frames and fuse their merits in a task-agnostic manner.
Experimentally, FrameMiners achieves onpar or signiﬁcantly higher performance than the best single-frame version on ﬁve fully physical manipulation tasks adapted from ManiSkill and OCRTOC.
Without changing existing camera placements or adding extra cameras, point cloud frame mining can serve as a free lunch to improve 3D manipulation learning.
Introduction
With the rapid development and proliferation of low-cost 3D sensors, point clouds have become more accessible and affordable in robotics tasks [1].
Also, the tremendous progress in building neural networks with 3D point clouds [2, 3, 4, 5, 6, 7] has enabled powerful and ﬂexible frameworks for 3D visual understanding tasks such as 3D object detection [5, 8, 9] 6D pose estimation [10, 11] and instance segmentation [12, 13].
Very recently, point cloud started to be used as the input to deep reinforcement learning (RL) for object manipulation [14, 15, 16], which aims at learning mappings directly from raw 3D sensor observations of unstructured environments to robot action commands.
These end-to-end learning methods avoid highly structured pipelines and laborious human engineering required by conventional robot manipulation systems.
When building an agent with point cloud input, existing works [14, 15, 16] typically incorporate offthe-shelf point cloud backbone networks (e.g., PointNet [2]) into the pipeline as a feature extractor of the scene.
However, some facets in constructing point cloud representations have been overlooked.
For example, in the literature of 3D deep learning, the choice of coordinate frame signiﬁcantly affects task performance [2, 17, 18, 19, 20, 21].
On 3D instance segmentation benchmarks for autonomous driving, previous work such as [5] showed a pipeline to process input point clouds in the camera frame, frustum frame, and object frame subsequently, leading to a large performance boost in comparison to using the camera frame alone.
For our goal of manipulation skill learning, point clouds describe dynamic interactions between robots and objects, including frequent contacts and occlusions.
This is a novel and more complex setting that differs from well-explored scenarios in 3D supervised learning (e.g., single objects, outdoor scenes for autonomous driving).
Under this setting, choices of coordinate frames are more ﬂexible and diverse as multiple entities (e.g., robot and manipulated object) and dynamic movements are involved.
In this work, we ﬁrst examine whether and how different coordinate frames may affect the performance and sample efﬁciency of point cloud-based RL for object manipulation tasks.
We study four candidate coordinate frames: world frame, robot-base frame, end-effector frame, and targetpart frame.
These frames differ in positions of origin and orientations of axes, and canonicalize inputs in different manners (e.g., a ﬁxed third-view, ego-centric, hand-centric, object-centric).
The comparison and analysis are performed on ﬁve distinct physical manipulation tasks adapted from
ManiSkill [22] and OCRTOC [23], covering various numbers of arms, robot mobilities, and camera settings.
Results show that the choice of frames has profound effects.
In particular, the end-effector frame and the target-part frames, rarely considered in previous works, lead to signiﬁcantly better sample efﬁciency and ﬁnal convergence than the widely used world frame and robot-base frame on many tasks.
Visualization and analysis indicate that, by using different coordinate frames to represent input point clouds, we are actually performing various alignments of input scenes through SE (3) transformations, which may simplify the learning of visual modules.
However, the well-performing single coordinate frame may vary from task to task, and in many cases, we may need coordination between decisions made according to multiple coordinate frames.
For example, tasks equipped with dual-arm robots may beneﬁt from both left-hand and right-hand frames.
For mobile manipulation tasks involving both navigation and manipulation, different frames could favor different skills (e.g., robot-base frame for navigation skills, end-effector frame for manipulation skills).
We thus propose three task-agnostic strategies to adaptively select from multiple candidate coordinate frames and fuse their merits, leading to more efﬁcient and effective object manipulation policy learning.
Because we do not need to capture additional camera views or rely on task-speciﬁc frame selections, our frame mining strategies can be used as a free lunch to improve existing methods on point cloud-based policy learning.
We call these fusion approaches as
FrameMiners .
Experimentally, we ﬁnd that it matters to fuse information from multiple frames, but the speciﬁc FrameMiner to choose does not create much performance difference.
In particular , we use one of the FrameMiners, MixAction, to interpret the importance of different frames in the policy execution process, and the interpretation agrees with our intuitions.
In summary, the main contributions of this work are as follows:
• We ﬁnd that the choice of coordinate frame has a profound impact on point cloud-based RL for object manipulation.
In particular, the end-effector frame and the target-part frame lead to much better sample efﬁciency than the widely-used world frame and robot-base frame on many tasks; • We ﬁnd that well-performing frames differ task by task, necessitating task-agnostic ways to select and fuse frames.
This observation is consistent across 3D backbone networks; • We propose FrameMiners, a collection of methods to fuse information from multiple candiate frames.
FrameMiners provide a free lunch to improve existing point cloud-based manipulation learning methods without changing camera placements or requiring additional camera views.
Related Work
Manipulation Learning with Point Clouds Visual representation learning for object manipulation has been extensively studied [24, 25, 26, 27, 28, 29, 30, 31].
With the ﬂourishment of 3D deep learning [2, 3, 4, 5, 6, 7], a major line of work learns representations from 3D point clouds for object manipulation [32, 33, 34, 35, 36, 37, 38].
Recently, people have also started to incorporate point clouds into deep reinforcement learning (RL) pipelines for manipulation learning [14, 15, 16].
However, existing point cloud-based manipulation learning methods have not paid enough attention to coordinate frame selections of input point clouds, which is fundamental in 3D visual learning.
Some very recent work [39, 40] explored placement and selection of camera views and fusion of multi-view images.
We differ from them in that we focus on the preprocessing of captured input point clouds without modifying existing camera conﬁgurations or adding additional cameras.
Normalization and View Fusion in Point Cloud Learning Normalizing input point clouds is a common practice in 3D deep learning literature.
For example, in single object analysis (e.g., classiﬁcation and part segmentation), people often normalize input point clouds into a categorical canonical pose with unit scale [2, 3], simplifying network training.
Prior works ﬁnd that existing point cloud networks [2, 3, 7, 41, 42] are very sensitive to input normalization [21, 43, 44], and many recent attempts explore rotation invariant [45, 46, 47] and equivariant methods [21, 48, 49] for 3D deep learning.
Compared to well-studied scenarios (e.g., single object and autonomous driving), normalizations of point clouds under robot-object interactions are under-explored.
[Figure] In LiDAR point cloud learning for autonomous driving, many work focuses on the fusion of multiple views [17, 18, 19, 20].
Unlike fusing multiple camera scans, there is only one point cloud.
They propose to process the point cloud from different views (e.g., perspective view and birds-eye view) to combine their merits, which has proven to be helpful.
Our work shares a similar idea, but we focus on robotic object manipulation settings, and the choice of coordinate systems is more diverse.
Point Cloud Coordinate Frame Selection Matters 3.1 Problem Setup
[Figure]
We aim to learn agents with point cloud input for object manipulation tasks via RL.
A task is formally deﬁned as a Partially-Observable
Markov Decision Process (POMDP), which is represented by a tuple M = ( S, A, µ, T, R, γ, Ω , O ) .
Here S and
A are the environment state space and the action space.
µ ( s ) , T ( s (cid:48) | s, a ) , R ( s, a ) , and γ are the initial state distribution, state transition probability, reward function, and discount factor, respectively.
O ( s ) :
S → Ω is the observation function that maps environment states to the observation space Ω .
Our agent is represented by a policy π : Ω → A , which aims to maximize the expected accumulated return given by J ( π ◦ O ) =
E µ,T,π [ (cid:80) ∞ t =0 γ t r ( s t , a t )] .
Note that π does not have access to the environment state s and only has access to the observation
O ( s ) .
In this work, O ( s ) consists of two parts: (1) a 3D point cloud captured by depth cameras; (2) proprioceptive states for the robot, such as joint positions and joint velocities.
For the ﬁrst part, if there are multiple cameras, we fuse all point clouds into a single one by transforming them into the same coordinate frame and concatenating the points together.
Fig.
2 shows the architecture of a 3D point cloud-based agent, which we use to discuss in this section.
It ﬁrst exploits a 3D backbone (e.g., PointNet [2]) to extract visual features from a 3D (fused) point cloud.
The extracted features are then concatenated with proprioceptive robot states and fed into separate multi-layer perceptrons (MLP) for action and value prediction.
The input (fused) point cloud can be represented in different coordinate frames before being fed into the 3D backbone network, and the choice of coordinate frame is independent of camera views .
For example, a point cloud captured by a camera mounted on the robot’s head can be transformed into the end-effector frame.
In this work, we study how point cloud coordinate frames affect sample efﬁciency and ﬁnal convergence of object manipulation learning.
Unlike prior works [39, 40], we do not change robot camera conﬁgurations (e.g., camera placement, inclusion of additional cameras).
As shown in Fig.
1, we exemplify the frame selection problem on ﬁve fully-physical manipulation tasks, covering various numbers of robot arms, mobilities, and camera settings.
Among them, PickObject is adapted from OCRTOC [23], and the other four tasks are adapted from ManiSkill [22].
On PickObject, a ﬁxed-base single-arm robot learns to physically grasp an object from the table, lift it up to a target height, and keep it static for a while.
Point clouds are captured from a 3rd-view camera.
On ManiSkill tasks, agents learn generalizable physical manipulation skills (i.e., opening cabinet doors / drawers, pushing chairs / moving buckets to target positions) across objects with diverse topology, geometry, and appearance.
We utilize mobile robots with one or two arms.
Point clouds come from a panoramic camera mounted on the robot’s head.
Action space includes joint velocities of the arm(s) and the mobile robot base, along with joint positions of the gripper(s).
3.2 Choices of Point Cloud Coordinate Frame
For 3D supervised learning tasks such as object classiﬁcation and detection, it’s a common practice to normalize input point clouds, and the choice of coordinate frames signiﬁcantly affects task performance [2, 21, 17, 18, 19, 20].
In point cloud-based manipulation learning, we are faced with an underexplored, yet more challenging, setting.
First, point clouds describe more complex robotobject interactions, possibly including frequent contacts and occlusions.
Furthermore, compared to supervised learning, 3D visual modules receive weaker supervision signals during RL training.
Therefore, it may become even more important to lessen the burden of visual module learning by properly normalizing input point clouds.
Unlike previous well-studied point cloud learning scenarios (e.g., single-object point clouds, LiDAR point clouds for autonomous driving), there exist more diverse choices of coordinate frames.
In this paper, we compare and analyze four candidates:
• A world frame is attached to a ﬁxed point in the world (e.g., the start point of a trajectory).
• A robot-base frame is attached to the robot base, offering an egocentric perspective on a mobile robot.
For a ﬁxed-base robot, world frame and robot-base frame could be equivalent.
In many object manipulation tasks, movements of robot end-effector(s) play important roles, and we can attach an end-effector frame to each of them.
Note that for dual-arm robots, there are two end-effectors and thus two end-effector frames.
• A target-part frame is attached to the object part the robot intends to interact with (e.g., target door handle for the OpenCabinetDoor task).
When we transform captured point clouds into the world frame, the robot-base frame, and the endeffector frame, we may need proprioceptive robot states and potential robot movement tracking, which is typically accessible in modern robots.
When we transform point clouds into the target-part frame, we may need to leverage off-the-shelf 3D object detection and pose estimation techniques.
However, in this paper, we mainly focus on the choices of coordinate frames themselves.
In our simulated experiments, we use ground truth object poses for the target-part frame. [Figure]
In Fig.
3, we visualize an example trajectory under four coordinate frames.
As shown in the ﬁgure, different coordinate frames canonicalize inputs in different manners (e.g., a ﬁxed third-view, egocentric, hand-centric, and object-centric), which is essentially performing various alignments among point clouds across multiple time steps.
For example, in the end-effector frame, the end-effector is always aligned at the origin throughout a trajectory.
Such alignments may simplify the learning of visual modules in distinct ways.
With the end-effector frame, the network does not need to locate the end-effector in point clouds (always at the origin).
Similarly, with the target-part frame, it can be easier to determine the relative position between the target part and the robot end-effector.
The robot-base frame naturally aligns its frame axes with the moving directions of the robot’s base.
3.3 Single-Frame Comparison on Manipulation Tasks
We compare the four coordinate frames on the ﬁve manipulation tasks by training PPO [50] agents using PointNet [2] as the 3D visual backbone.
In this section, the (fused) point cloud is transformed into a single coordinate frame.
For PushChair and MoveBucket tasks that use a dual-arm robot, we use the right hand frame as the end-effector frame (we observe almost identical performance using the left hand frame).
For the target-part frame, we choose the handle frame for OpenCabinetDoor and OpenCabinetDrawer tasks, chair seat frame for the PushChair task, bucket for the MoveBucket task, and the target object for the PickObject task.
Further details are presented in the supplementary.
Fig.
4 shows the results.
We observe that distinct coordinate frames lead to very different agent training performances.
Overall, the world frame is the least effective, especially in PushChair and MoveBucket that involve more pronounced movement of the robot base.
This suggests that the alignment of a static point in the world-frame is less helpful for the tasks.
Compared to the commonly-used world-frame and robot-base frame, the end-effector frame has much higher sample efﬁciency on all single-arm tasks (i.e., OpenCabinetDoor, OpenCabinetDrawer, and PickObject), demonstrating the beneﬁts of end-effector alignment.
However, it shows similar or worse performance on PushChair and MoveBucket, intuitively because these tasks rely on dual-arm coordination, but our point cloud is normalized to a single end-effector frame (i.e., right hand frame).
In addition, the target-part frame achieves the best sample efﬁciency on most tasks, suggesting that the target-part alignment across time could be of great help for point cloud-based visual manipulation learning.
3.4 Further Analysis
In robot manipulation tasks, the network needs to infer binary relations between subjects (e.g., relative pose between the end-effector and the cabinet handle).
By aligning point clouds under certain frames (e.g., end-effector frame), these tasks may be reduced to single-subject location tasks (e.g., simply copying the handle pose), which become much easier to solve.
To conﬁrm this hypothesis, we perform a diagnosis experiment on OpenCabinetDoor, where we intentionally remove all robot points (i.e., blue points in Fig. 3) and see its effect on different coordinate frames.
As shown in Fig.
5, after the robot points are removed, the end-effector frame performs the same, while the robot-base frame performs worse (the task is still solvable since the end-effector position is also provided in the proprioceptive robot state).
This suggests that the end-effector frame allows an agent to focus on the target object, along with its interaction with the robot hand, which veriﬁes
[Figure]
[Figure] We utilized PointNet [2] as our 3D visual backbone for its fast speed and general good performance.
However, it’s unclear whether point cloud frame selection is also crucial for other 3D backbones, especially those more complex and powerful.
Therefore, we conduct the same experiments as Sec 3.3 using
SparseConvNet [51], a heavier 3D backbone network, on the four ManiSkill tasks (further details in the supplementary).
As shown in Fig.
6, we observe similar relative performance between frames as before (e.g., the world frame performs poorly; the end-effector frame outperforms the world frame and the robot-base frame on OpenCabinetDoor and OpenCabinetDrawer).
Interestingly, using SparseConvNet doesn’t improve the overall performance over PointNet.
Mining Multiple Coordinate Frames
We have shown that different point cloud coordinate frames lead to distinct sample efﬁciencies and ﬁnal performances in manipulation learning.
However, a single frame can perform well on some tasks but poorly on others, and we wish to ﬁnd good frames in a task-agnostic manner.
Moreover, for complex manipulation tasks, a single frame could be insufﬁcient, and synergistic coordination between multiple frames could provide unparalleled advantages.
For example, when robots are equipped with multiple arms, each arm may have its preferred coordinate frame (e.g., left-hand frame and right-hand frame).
In addition, in tasks that involve simultaneous navigation and manipulation (e.g., on PushChair and MoveBucket, an agent needs to move towards the target while manipulating chairs or buckets), different frames could beneﬁt different skills (e.g., robot-base frame for navigation skills, and end-effector frame for manipulation skills).
Therefore, it is of great help to propose a prior-agnostic method that can automatically select the best frame from multiple candidates or combine the merits of them.
Again, we are not talking about fusing multiple camera views.
Point clouds from multiple camera views are ﬁrst fused together into a single point cloud, before being transformed to each coordinate frame.
In this section, we will present a collection of three strategies to adaptively select and fuse multiple candidate coordinate frames, and we call them FrameMiners .
In particular, we will ﬁrst introduce
FrameMiner-MixAction in Section 4.1 in detail to interpret the importance of different frames in the policy execution process.
We will then brieﬂy introduce the other two FrameMiners and compare different approaches with single-frame baselines.
4.1 FrameMiner-MixAction
Inspired by the idea of mixture of experts [52], we propose a general and interpretable framework,
FrameMiner-MixAction (FM-MA).
As shown in Fig.
7, FM-MA takes a (fused) point cloud and n candidate coordinate frames as input, and ﬁrst transforms the point cloud into n coordinate frames.
For each transformed point cloud, FM-MA employs an expert network, consisting of a 3D visual backbone (e.g., PointNet [2]) and an MLP, to propose full robot actions (e.g. target velocity of m joints).
Since different experts use different coordinate frames, they are encouraged to specialize different skills and controls of different joints.
Finally, we combine actions from the n experts through input-dependent weights.
Speciﬁcally, we concatenate extracted visual features from all n frames with the proprioceptive robot state, feed it into an MLP, and predict a weight [Figure]
[Figure] proposal for each pair of expert and joint (there are n × m weights in total). For each joint, we normalize the weights over n experts and fuse the
FM-MA fuses actions by predicting joint-speciﬁc weights, since we believe that, for different joints, we need to extract information from different coordinate frames.
Furthermore, the weights are inputdependent, potentially allowing the model to capture dynamic joint-frame relations at different stages of a task.
Fig.
8 conﬁrms these hypotheses.
On MoveBucket trajectories, we observe distinct frame preferences between different robot joints.
The left and right-hand frames contribute signiﬁcantly to their respective joint actions.
In addition, the weight distribution changes greatly over different trajectory stages.
Initially, when the robot is moving towards the bucket but not interacting with it, the base frame contributes more.
However, when hands start to manipulate the bucket, the weights of the hand frames increase.
In particular, when the robot places the bucket onto the platform, we need careful coordination among all joints, and thus similar weights from each frame.
4.2 Comparison of Different FrameMiners
To study how network architectures inﬂuence coordinate frame fusion, we also propose other two strategies: FrameMiner-FeatureConcat (FM-FC) and FrameMiner-TransformerGroup (FM-TG).
For each transformed point cloud, FM-FC uses an individual PointNet to extract visual feature.
All visual features are then concatenated and fed into an MLP to predict robot action.
FM-TG decomposes our robot action into three groups: base joint actions, left-hand joint actions, and righthand joint actions (only two groups for single-arm tasks).
After visual features are extracted from
PointNets, they are fused through a Transformer [53] to produce a feature for each action group, which passes through an MLP to predict its respective joint actions (see supplementary for details).
[Table]
We compare our three FrameMiners with single-frame baselines.
Speciﬁcally, in this section, we focus on frame mining among the robot-base frame and end-effector frame(s).
For the dual-arm tasks (i.e., PushChair and MoveBucket), the endeffector frames include both the left-hand frame and the righthand frame.
We will discuss the inclusion of target-part frame in Section 4.3.
Fig.
9 and Tab. 1 show the comparison results.
On single-arm tasks (i.e., OpenCabinetDoor/Drawer), our FrameMiners perform on par with the end-effector frame, which suggests that FrameMiners can automatically select the best single frame. On dual-arm tasks, our FrameMiners signiﬁcantly outperform single-frame base-
[Figure]
lines, demonstrating the advantage of coordination between multiple coordinate frames. While it matters to fuse information from multiple frames, the speciﬁc FrameMiner to choose does not create much performance difference.
Empirically, we ﬁnd FM-MA less sensitive to training parameters, and FM-TG more computationally expensive.
4.3 Target-Part Frame
[Figure]
could further boost the performance.
In Section 4.2, we focus on the fusion of robot-base frame and end-effector frames, since the target-part frame relies on pose estimation of target objects, which requires extra efforts in real-world settings.
As shown in Fig.
10, if object poses are estimated, we sometimes observe further performance boost of FrameMiners from the targetpart frame.
For example, on PushChair, by incorporating the target-part frame, the success rate of FM-MA increases from 364% to 533%.
On PickObject, FM-MA already achieves good performance without the target-part frame (942%); incorporating it slightly improves the success rate to 971%.
Real World Experiments
To further verify that our learned policies can be deployed on realworld robots without introducing extra domain gaps, we test on PickObject with a Kinova Jaco2 Spherical 7-DoF robot, an Intel RealSense [54] D435 camera for uncolored point cloud capture, and a
Rubik’s cube from YCB objects [55, 56]
(see Fig. 11).
We use a 3-DoF end-effector position controller and a 1-DoF gripper position controller.
We train the FM-MA policy (Section 4.1) by fusing the robot-base frame and the end-effector frame.
At test time, we build a digital twin in the simulator over 25 sampled initializations of the real environment with a vision-based pipeline like Jiang et al.
By following trajectories from policy rollout, we obtain a 84% success rate with FM-MA, compared to an 80% success rate with the end-effector frame.
The robot-base frame is unable to achieve successful picks under our training budget.
Note that the performance differences in the real world are very similar to the simulation environment, indicating that point cloud frame selection or mining does not affect original domain gap.
More details are presented in supplementary.
Conclusion and Limitations
We ﬁnd that choices of point cloud coordinate frames have a profound impact on learning manipulation skills.
Our proposed FrameMiners can adaptively select and fuse multiple candidate frames, serving as a free lunch for 3D point cloud-based manipulation learning.
Currently, our FrameMiners need to process each frame separately, leading to more network computation.
In the future, we would like to explore more advanced fusion strategies to further improve network efﬁciency as well as performance.
In addition, the target-part frame requires human judgment to determine the target part candidates and 6D pose estimation of the target parts, although we have shown our method can also achieve great improvements without the target-part frame (Section 4.2).
Acknowledgments
This work is supported in part by gifts from Qualcomm.
We would like to thank Jiayuan Gu and
Zhiao Huang for their helpful discussion and manuscript proofreading.