Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control
Abstract: Collocated tactile sensing is a fundamental enabling technology for dexterous manipulation.
However, deformable sensors introduce complex dynamics between the robot, grasped object, and environment that must be considered for ﬁne manipulation.
Here, we propose a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment.
Our method combines the perceived 3D geometry of the membrane with proprioceptive reaction wrenches to predict future deformations conditioned on robot action.
Grasped object poses are recovered from membrane geometry and reaction wrenches, decoupling interaction dynamics from the tactile observation model.
We benchmark our approach on two real-world contact-rich tasks: drawing with a grasped marker and in-hand pivoting.
Our results suggest that explicitly modeling membrane dynamics achieves better task performance and generalization to unseen objects than baselines.
Introduction
[Figure]
Tactile sensing collocated at the contact interface between the robot and environment is a key enabling technology for dexterous manipulation [1].
The resulting information-rich and highly discriminative tactile cues are far more informative than proprioceptive external joint torques.
To date, a number of high-resolution collocated tactile sensors have gained traction in the robotics community including Soft Bubbles [2], GelSlim [3], DIGIT [4], and FingerVision [5].
However, there are two coupled challenges in using this class of sensors: the high-dimensionality of the sensor signal (signature) and the sensor dynamics introduced between the robot and what it is contacting.
Recent progress in exploiting these high-dimensional signatures has resulted in sensor speciﬁc algorithms for state-estimation [6, 7, 8, 9, 10, 11] and controls [12, 13, 14, 15].
However, most progress has addressed low-deformation sensors with negligible dynamics (e.g., GelSlim, DIGIT, and FingerVision).
These dynamics cannot be ignored for high-deformation tactile sensors (e.g.,
Soft Bubbles).
This is because large sensor deformations lead to both non-negligible relative motion of the grasped object with respect to the end-effector and incipient slip as the object is brought into contact with the environment.
When used effectively, this high compliance is desirable for contact rich interactions as it allows for large and stable contact patches as well as gradual force build up.
In this paper, we ﬁrst illustrate deformation and force transmission differences between hard and soft tactile sensors by showing the relative behavior of the Soft Bubbles and GelSlim 3.0.
The mechanical interface of the former is an inﬂated latex membrane while the latter uses a dense polymer.
This exercise motivates our main contribution: a method to learn soft tactile sensor membrane dynamics that accounts for the sensor deformations caused by the physical interaction between the grasped object and environment.
Our approach integrates the perceived 3D geometry of the sensor membrane with proprioceptive reaction wrenches and predicts future membrane deformations conditioned on robot actions.
A key feature of our method is the decoupling of the sensor membrane dynamics from the tactile observation model.
Tactile observation models compute features such as the object pose and extrinsic contact location from the sensor state (3D geometry and reaction wrenches).
This decoupling exploits the fact that sensor membrane dynamics are shareable across tasks because they are inherent to the sensor mechanics while the observation model can be taskspeciﬁc.
We demonstrate how our method enables reasoning over the joint dynamics of the robot, grasped object, and environment to enable precise control of object pose and force transmission and empirically compare our method against 5 baselines on two real-world contact rich robotic tasks.
Related Work
There are two major types of tactile sensing: localized and distributed.
Localized sensing, here referring to the use of a single force-torque sensor often at the robot wrist, summarizes external contact information as a wrench composed by six numbers, 3 linear and 3 torque terms.
Localized sensing can be provided by the robot through joint torques or using F/T sensors [16, 17, 18].
In contrast to localized, distributed tactile sensing collocated at the contact interface can provide information dense feedback in the form of images [2, 3, 5] or pressure distributions [19].
The survey by Yamaguchi and Atkeson [20] summarizes recent advances in these sensors.
With the promise of information dense sensing; however, comes challenges including novel dynamics at the contact interface, and tactile signature representation for state-estimation, prediction, and controls.
Finding good tactile sensing representations is challenging because it typically requires either expert knowledge or considerable amounts of data.
Approaches such as TACTO [21] and Narang et al. [11] have reduced data requirements by simulating DIGIT [4] and SynTouch Biotac pressure-based sensors [19], respectively.
Building on this, Kelestemur et al.
[22] propose an object pose estimator trained in simulation that transfers to the real-world.
However, current tactile simulators only support a small set of relatively rigid sensors and simulation of soft sensor, much like other deformables, remains a signiﬁcant challenge.
Therefore, our data is collected from real-world interactions.
Early work exploited dense tactile signals in closed-loop controllers for contact servoing [12] or active exploration [13].
More recently, tactile control approaches have been used in robotic bimanual object manipulation [23], deformable object manipulation [24], and peg-in-hole insertion [25].
These later approaches use the GelSlim tactile sensor [3] which exhibits small and approximately linear deformations.
This simpliﬁed model is not effective for the Soft Bubbles and assuming sticking external contact is restrictive in many applications (see Sec.
4.1 and 5).
Closely related to our work, previous work has explored explicitly modelling the tactile sensor dynamics.
Van Hoof et al.
[15] projected the Biotak tactile signals into a learned low-dimensional representation where dynamics are linear.
Other approaches such Tian et al.
[14] have modelled the dynamics of a vision-based tactile sensor such as GelSight.
Their entire control pipeline is formulated in the sensor space, requiring the goal states to be speciﬁed as tactile states.
Instead, we decouple the sensor dynamics from the task dynamics, allowing us to specify tasks in terms of object poses and forces transmitted to the environment.
Lambeta et al.
[4] extended their control formulation to a multi-ﬁnger control setting by adding structure to the tactile dynamics predictions.
This approach conditions the dynamics on the estimated object pose, which creates a dependency on the pose estimation and may limit the model’s ability to generalize to different objects.
Problem Formulation
The goal of the robot is to control the grasped object pose and the force transmitted to the environment as the object is brought into contact.
Our central idea is to represent these “task dynamics” by modeling them using the tactile sensor membrane dynamics.
We are motivated by the fact that the grasped object pose and transmitted forces are resolved simultaneously with the membrane deformation and end-effector reaction wrenches.
This means the robot can plan for desired task states by reasoning through the membrane dynamics – i.e., planning actions that result in membrane deformations that in turn result in the desired poses and transmitted forces.
[Figure]
Let p t denote the 3D pose of any points on the surface of the membrane, then the dynamics model p t +
=
f
( p t , w t , a t , z t ) represents how each point deforms conditioned on the current membrane geometry, end-effector reaction force w t , robot action a t , and the grasped object z t .
Next, let q t denote the object-environment conﬁguration, then the observation model q t =
g ( p t , w t , z t ) maps the current membrane geometry and reaction force to conﬁgurations.
We deﬁne the task state as x t = ( q t , w t ) and goal as x g .
Planning and control reduces to solving the optimization problem:
[Equation]
where Q and R are positive deﬁnite matrices weighing the relative importance of reaching goal states with effort.
The key to solving this problem is learning a sufﬁcientlyaccurate estimate of f , which is the focus of this paper.
Methods
In this section, we ﬁrst illustrate characteristic deformation and force transmission differences between soft and hard sensors (Soft Bubbles vs. GelSlim 3.0) to motivate the need for our membrane dynamics.
Next, we present our main contribution: our approach to learning a soft tactile sensor membrane dynamics model that accounts for the sensor deformations caused by the physical interaction between the grasped object and environment as well as an observation model used to extract the task state.
Finally, we describe the controller which leverages these dynamics for task execution.
4.1 Sensor Deformation Evaluation
[Figure]
Controlling both in-hand object pose and transmitted force is key for dexterous manipulation.
In this section, we ask just how signiﬁcant is the relative object displacement w.r.t.
to the end-effector for soft vs hard sensors and how is transmitted force affected?
Fig.
3 shows an illustrative comparison of the Soft Bubbles and Gelslim 3.0 for two in-contact motions along the sensor’s main axes.
The results show that the Soft Bubble membrane deforms an order of magnitude more than the GelSlim.
We also observe that the object orientation varies signiﬁcantly more (approx.
25  ) during tangential motion.
Moreover, we observe signiﬁcant slip between the object and GelSlim, while the Bubbles maintain sticking contact due to large contact patches resulting from the signiﬁcant deformation.
The GelSlim deformations are less than 1 mm and the resulting force transmission proﬁle is relatively sharp, similar to rigid-body interactions.
We also observe force plateauing and relative slip much earlier in the vertical contact task.
In contrast, the soft sensors’ large compliance allows for a more gradual force transmission without slip for a larger range of motion.
While this compliance is desirable for many contact-rich tasks, it must be accounted for during ﬁne manipulation.
The large contact patches result in increased surface area which provides two useful features: increased perception of the object shape at contact and larger distribution of friction through a concave contact surface.
The former point can improve in-hand object pose estimation because it provides more features for inference, and the latter can improve grasp stability owing to larger and more distributed frictional forces where the membrane “hugs” the object.
The gradual build up of force is effectively a form of passive compliance.
This compliance prevents damage to position controlled robots by mitigating rigid-on-rigid body contact.
Further, it eases the burden on impedance control by providing local to contact compliance that can be perceived and controlled through robot motion.
Thus, the remainder of this paper investigates how to learn the membrane dynamics of Soft Bubble sensors in order to realize the above beneﬁts for dexterous manipulation.
4.2 Membrane Dynamics Model
The goal of the membrane dynamics model is to predict the tactile sensor membrane deformations p t +1 and reaction forces w t +1 as a function of their current geometry p t , reaction force w t , the robot action a t , and the tool geometry z t .
The model has access to the current membrane geometry measured by a time-of-ﬂight depth sensor mounted within the gripper.
This membrane geometry p can be represented as either a depth map image or a 3D point cloud using the camera intrinsics.
Here, we use the depth map encoding as it is a more structured and simpliﬁes pixel-wise correspondences.
The dynamics model predicts the pixel-wise deformation of the sensor membrane and the reaction force sensed at the end-effector using the architecture depicted in Fig.
In order to enable multistep planning, the model also predicts the grasp frame pose reached by the robot r t +1 as a consequence of taking the action a t (accounting for the robot impedance).
Similar to [15], the model
[Figure]
encodes tactile images into a lower-dimensional latent representation.
Intuitively, the model learns to exploit structure in the deformed membrane geometry to obtain a more compact representation.
The object geometry is also encoded into a lower-dimensional representation to help the membrane dynamics model combine these multi-modal inputs.
Since the object geometry is provided as a point cloud, our model uses a PointNet [26] inspired network pre-trained on ModelNet40 [27] and ﬁne-tuned on our task data to obtain a lower-dimensional latent representation of the grasped object.
We train our dynamics model in two-steps.
First, we learn the tactile image latent representation in an Autoencoder [28] fashion with only the tactile images from our training dataset.
Our model is able to exploit tactile data from different tasks by training the tactile encoder-decoder on the combined dataset, subsequently sharing this latent tactile representation across tasks.
We also pretrain the object geometry model embedding on ModelNet40 and freeze all weights but the last layer.
Second, we freeze the tactile encoder-decoder and train the dynamics model end-to-end.
We train the dynamics model f with data composed by state-action-state transitions conditioned on the object geometry embedding ( s t , z t , a t , s t +1 ) .
In our case s t =
( p t , w t , r t ) .
We employ a supervised reconstruction loss L dyn on the predicted state composed of 3 mean squared error (MSE) terms, one for each of the three sensory modalities our model predicts: tactile, wrenches, and poses.
The losses are aggregated based on weights ↵ i to compensate for the discrepancies in units and scale of each sensory contribution (see Appendix A.2 for implementation details).
[Equation]
where ˆ
[Equation]
r t , z t , a t )
4.3 Observation Model
[Figure]
(a) Observation Model The deformed depth maps are projected using the camera intrinsics and extrinsics to obtain the joint pointcloud of the deformed sensors.
Then this is compared with a reference pointcloud of the undeformed sensor state to extract the contact points.
Finally, ICP ﬁts an object model to the detected contact points to estimates the corre- sponding object pose.
[Figure]
(b) Control Pipeline Given a measured state, our controller queries the membrane dynamics model with sampled actions to obtain the predicted mem- brane states.
The object pose is estimated from the predicted membrane states and it is compared with the desired poses to compute the costs associated to the sample actions.
The costs are aggregated and the resultant optimal action is executed by the robot.
The observation model g maps the current membrane geometry p t and end-effector reaction wrench w t to an estimate of the current grasped object pose q t .
It has access to the object geometry z t , robot proprioception r t , and the location of the environment contact surface.
We decouple pose estimation from membrane dynamics, enabling ﬂexibility in observation model choice and improving generalization across objects and tasks.
Fig.
5a illustrates the observation model.
First, we use the camera intrinsics and extrinsics to project the depth maps into the end-effector frame.
Next, we extract contact points by comparing the current tactile images to their undeformed reference states.
We extract contact points by computing pixel-wise deformation and selecting the top 10% that deform at least 3 mm.
See
Appendix A.3 for more details.
Next, we use Iterative Closest Points (ICP) to estimate the object pose from the extracted contact points.
When the grasped object is in contact with the environment, we project the predicted object pose to the objectenvironment contact manifold [29].
4.4 Controller
The goal of the controller is to drive the task states x t = ( q t , w t ) , composed by object-environment conﬁgurations q and transmitted forces w , to their goal values x g =
( q g , w g ) by solving the optimization problem posed in Sec. 3.
To this end, we use model-predictive control; speciﬁcally, the Model-Predictive Path Integral (MPPI) controller [30].
MPPI is effective in handling continuous action spaces and can be parallelizable for an efﬁcient implementation with neural network models.
Fig.
5b illustrates the control pipeline.
The controller iteratively optimizes a nominal control sequence by sampling action sequences and rolling out their dynamics using our learned dynamics model and observation model.
These trajectories are sequences of task states x t , . . . , x t + N , and they are obtained by ﬁrst using our learned dynamics model to obtain dynamic states s i , and then using the observation model to obtain the correspondent task states x i .
Finally, the trajectory costs are computed by comparing the predicted task states, with the desired values under a quadratic cost and the nominal control sequence is updated.
For pivoting, the goal object orientations are deﬁned as the desired tool orientation with respect to the robot.
For drawing, the goal object orientations are perpendicular to the whiteboard along the drawing outline. In both cases the desired forces are perpendicular to the environment surface to encourage contact. See Appendix A.4 for implementation details.
Experiments and Results
We demonstrate our proposed approach to tactile control in two real-world tasks: drawing with a grasped marker and in-hand pivoting of objects.
Both of these tasks require the robot to apply forces through the grasped object to the environment while controlling its pose with respect to the end-effector.
Since the grasp is not rigid, the object may move w.r.t.
the end-effector due to deformations or slip.
Effective task execution must account for relative motion and prevent failure due to the tool slipping out of the grasp.
Our method uses the same dynamics model architecture (Fig. 4) and learned latent membrane geometry representation across tasks.
One key beneﬁt of learning membrane deformations as opposed to object poses is that although each task has different dynamics, the membrane deformation dynamics are inherent to the sensor.
This allows us to combine data from multiple tasks to learn the membrane latent representation.
However, since each task has particular dynamics and action spaces, the membrane dynamics function (Fig. 4 gray block) is task-speciﬁc.
We evaluate our two tasks on multiple tools, 8 for drawing and 8 for pivoting, Fig. 7.
For each task, we split our tools into two sets: we collect training data with 5 tools (train object set), and use the remaining 3 to test how well our learned model generalizes to unseen objects (test object set).
Using the 5 training tools, we collect 800 state-action-state triplets per tool, obtaining a total ofsamples to train the models.
Our data collection combines random samples with epsilon-greedy samples from a Jacobian controller (see Appendix B.1 for details).
5.1 Baselines
We benchmark our proposed dynamics model against 5 baselines inspired by related approaches.
The original baselines only consider tactile signatures as inputs.
Hence, we extend each to also incorporate reaction wrenches, robot poses, and object geometries.
We also modify the baselines such that all evaluated methods use the same observation model.
This way, all methods have access to the same information for a fair comparison.
Our ﬁrst baseline, Bubble Linear Dynamics is inspired by [15] where the dynamics are constrained to be linear in the tactile latent space.
Our second baseline,
Object Pose Dynamics , is inspired by [4] where the object in-hand pose dynamics are modeled as opposed to membrane dynamics.
We adapt this baseline by replacing the object pose estimator with our observation model for consistency across all methods.
Our third baseline, Fixed Model , is only used in the drawing task.
It assumes that there are no dynamics and that the membrane state remains constant.
This baseline illustrates the importance of modeling membrane deformations.
Our
[Figure]
fourth baseline, Jacobian , is only used in the pivoting task and is inspired by [12].
This baseline assumes that the grasped object is ﬁxed w.r.t.
the environment during contact.
Our ﬁnal baseline does not use a dynamics model.
Instead, the controller takes pseudo-random actions.
For drawing, pseudo-random actions move the robot along the drawing path while uniformly randomly selecting the end-effector orientation, grasp width, and height from the action space.
For pivoting, these actions bring the object into contact with the table while randomly selecting subsequent end-effector motion and grasp width.
This baseline evaluates the complexity of each task.
5.2 Drawing with a Marker
The goal of this task is to control the marker pose and force transmitted through its tip to draw a desired shape while preventing the marker from slipping out of the grasp.
For simplicity, we choose a continuous 0.6 meter line.
Fig.
6 shows the robot action space composed of Cartesian displacement in the plane of the line, rotation about an axis perpendicular to this plane, and the gripper width.
We evaluate the drawing performance using a camera to extract a binary mask of the white board and comparing it to the desired binary image (see Appendix B.2 for more details).
The drawing score is a value between 0 and 1, representing the percent of the line the robot has successfully drawn.
The goal states provided to the controller are a sequence of poses where the marker tip is perpendicular to and in contact with the board along the desired drawing path.
Fig.
8 shows the drawing score distributions.
Table 1 summarizes the drawing evaluation scores.
We perform 10 trials per marker.
The trials end when the ﬁnal desired pose is reached, if the maker slips out of the hand, or if the robot can no longer correct the marker pose because of joint limits or collisions.
Our results show that nonlinear membrane model (ours) outperforms the baselines both in training and test objects.
For train objects, our model successfully draws on average a 10.0% more of the desired line than the second best baseline ( Bubble Linear Dynamics ), and a signiﬁcant 32.6% more than Object Pose
Dynamics .
For test objects, the difference is 7.4% and 25.8% respectively.
We believe this is due to better robustness to uncertainty, generalization, and expressivity (see Sec. 6).
Our results also show that Object Pose Dynamics performs fairly similar to Fixed Model .
Moreover, in general test objects perform slightly better on average than the training objects.
A possible explanation is that their geometry and tip properties make the test objects easier to control, something we could not know a priori .
The “magnetic” brand marker is particularly challenging likely due to its ﬁne tip.
Removing this marker from the evaluation metrics eliminates the discrepancy between test and training sets, suggesting that the model has effectively learned important salient features of the task that it can indeed generalize to novel markers.
5.3 In-Hand Pivoting
The goal of this task is to drive the grasped object to a desired in-hand conﬁguration solely using the environment to modify its conﬁguration w.r.t the end-effector.
Fig.
6 shows the action space of the pivoting task which is composed of Cartesian motion in the plane of the gripper jaws, rotation about an axis perpendicular to this plane, and the gripper width.
Our metric for pivoting performance is error in desired vs. realized tool orientation.
We use our observation model to estimate the ﬁnal pose.
The desired tool pose is sampled at random within the graspable range.
The controller optimizes the pivoting action sequence to achieve the ﬁnal pose by exploiting the dynamics model.
Table 1 summarizes the pivoting evaluation scores (see Appendix C.1 for the evaluation distributions).
We perform 10 pivoting executions per tool.
The trials end when reaching the ﬁnal desired pose
[Figure]
D r a w i n g
S c
o r e
Representation
Drawing Scores
Pivoting Scores [deg]
Train Objects
Test Objects
Train Objects
Test Objects
Mean "
Std #
Mean "
Std #
Mean #
Std #
Mean #
Std #
Bubble Dynamics 0.
Bubble Linear Dyn.
Object Dynamics 0.
Fixed / Jacobian 0.
Pseudo-Random 0.
Discussion and Limitations
In this paper, our experiments show that decoupling membrane dynamics from object dynamics and characterizing the sensor dynamics improves task performance as well as the ability to use novel unseen tools.
We believe that modelling sensor dynamics performs better than object dynamics because: i) sensor dynamics explicitly considers the combined compliance of the robot and the sensor, ii) are more robust against observation noise, and iii) can share tactile signatures across tasks and objects.
Our observation model has some stochasticity, and its variance can highly impact trajectories that roll out from its observations.
Instead, our approach rolls out dynamic trajectories agnostic to the observation model.
As a result, the ﬁnal pose prediction is less effected by the observation variability.
See Appendix A.6 for a detailed comparison.
A limitation of our approach is that it assumes rigid objects with known geometry.
This assumption simpliﬁes the observation model, planning, and controls, since it restricts object-environment interactions to rigid on rigid.
While the membrane dynamics model may not be affected significantly by the object compliance, our method will require a more sophisticated observation and controls framework.
Another limitation of our model is the quality of long-horizon predictions.
Long-horizon predictions can be poor because recursive model calls may lead to out of distribution predictions.
This is the main reason why we use a planning horizon of 2 steps.
Some methods to mitigate this effect are multi-step prediction methods [31] or ﬂow-based projections [32].
Another direction to explore is developing new membrane representations that are robust to this effect.
Finally, our model predictions are smooth.
As a consequence, our method may fail to model drastic state changes like the outcomes of hitting obstacles, or dropping the tool.
This could be addressed by leveraging the structure of non-smooth/hybrid dynamics [33] during representation learning.
Acknowledgments
This research project is supported by Toyota Research Institute under the University Research Program (URP) 2.0.
This work has been partially supported by the mobility grants program of Centre de Formaci´o Interdisciplin`aria Superior (CFIS) - Universitat Polit`ecnica de Catalunya (UPC).
We would also like to show our gratitude to the anonymous reviewers for their helpful comments in reviewing the paper.
We also thank the members of the Manipulation and Machine Intelligence (MMINT) Lab for their support and feedback.