MidasTouch: Monte-Carlo inference over distributions across sliding touch
Abstract: We present MidasTouch, a tactile perception system for online global localization of a vision-based touch sensor sliding on an object surface.
This framework takes in posed tactile images over time, and outputs an evolving distribution of sensor pose on the object’s surface, without the need for visual priors.
Our key insight is to estimate local surface geometry with tactile sensing, learn a compact representation for it, and disambiguate these signals over a long time horizon.
The backbone of MidasTouch is a Monte-Carlo particle ﬁlter, with a measurement model based on a tactile code network learned from tactile simulation.
This network, inspired by LIDAR place recognition, compactly summarizes local surface geometries.
These generated codes are efﬁciently compared against a precomputed tactile codebook per-object, to update the pose distribution.
We further release the YCB-Slide dataset of real-world and simulated forceful sliding interactions between a vision-based tactile sensor and standard YCB objects.
While single-touch localization can be inherently ambiguous, we can quickly localize our sensor by traversing salient surface geometries.
[Figure]
performs online global localization of a vision-based touch sensor on an object surface during sliding interactions. Given posed tactile images over time, this system leverages local surface geometry within a nonparametric particle ﬁlter to generate an evolving distribution of sensor pose on the object’s surface.
Introduction
Interactive perception is the Catch-22 [1] of robotics; vision can be used to track objects for downstream contact-rich interactions, but such interactions can occlude visual tracking.
In particular, end-effector-object relative positioning is crucial for contact-rich policies like sliding [2–6], in-hand manoeuvres [7–10], ﬁnger-gaiting 11], and multi-ﬁnger enclosure [12].
Additionally, vision is often affected by object transparency, specularity, and poor scene illumination.
With high-dimensional vision-based tactile sensors [13–15], we now have a window into local object interactions.
While tactile images from these sensors capture local surface geometry, they lack global context necessary for relative pose tracking.
We address this challenge with MidasTouch , an online tactile perception system that tracks the evolving pose distribution of a vision-based touch sensor sliding across known objects.
We acquire global context by integrating local observations over long horizons.
The form-factor of vision-based tactile sensors has restricted prior methods to small parts [16–18] or local tracking [19, 20].
For everyday objects, we can leverage compositionality [21]; each is made up of local geometries that are spatially ﬁxed relative to one another.
Consider a simple mug: made up of a curved body, ﬂat base, rounded handle, and sharp lip.
Without global context, a single-touch is ambiguous: a perceived sharp edge could lie anywhere along the lip of the mug.
Such a likelihood distribution is spread across the object’s surface and not unimodal, but interaction over long time horizons can disambiguate it.
This mirrors haptic apprehension , or the exploratory procedures humans perform when presented with a familiar object [22].
We approach this as an analogue to mobile robot Monte-Carlo ﬁltering [23], but instead apply it on the surface manifold of the object.
Just as a mobile robot has access to detailed ﬂoor plans, odometry, and cameras, manipulators have access to object meshes, end-effector poses, and vision-based touch.
This is applicable to environments with known object models like households, warehouses, and factories, further facilitated by large-scale scanned object datasets [24, 25].
While priors from vision can bootstrap our system [26], they are not a prerequisite for global estimation.
In addition to open-sourcing MidasTouch, we release a comprehensive real-world and simulated dataset of sliding DIGIT [15] interactions across standard YCB objects [24] with ground-truth.
This serves as both an evaluation of MidasTouch, and as a benchmark currently lacking in the tactile sensing community.
Both are accessed from our project website.
Speciﬁcally, our contributions are:
Online particle ﬁltering over posed tactile images for a distribution of ﬁnger-object poses, 2.
Learned embeddings for vision-based touch using local surface geometry, 3.
The YCB-Slide dataset of DIGIT and YCB object interactions for evaluation and benchmarking.
Our framework relies on recent developments in tactile sensing, rendering, and robotics: (i) visionbased tactile sensors [13–15, 27–31] like the GelSight and DIGIT have the requisite spatial acuity to discern local geometric features, (ii) tactile simulation [32–34] with realistic rendering enables learning tactile observation models and precomputing object-speciﬁc interactions with sizeable data, which would be infeasible in the real-world, and (iii) learned models for 3D place recognition [35– 37] spawned by ubiquitous LIDAR and RGB-D data that can be extended to tactile sensing.
Related work
Tactile pose inference: Binary contact sensors have been used in conjunction with particle ﬁlters for global estimation of robot hands relative to simple geometries [38–43].
These methods require a large amount of touches, but serve as a touchstone for Monte-Carlo methods that have seen great success in mobile robotics [23, 44].
With tactile arrays, local patch measurements are integrated for planar estimation [45, 46] or for 3D alignment [47].
The synergy of vision and touch gives much needed global context [48, 49] but is outside our current work’s scope.
With vision-based touch, Li et al.
[16] show planar small-part localization by directly computing the homography on GelSight heightmaps.
Relative pose-tracking has also been learnt directly from tactile images either via recurrent [50], or auto-encoder [15, 19] networks.
Additionally, local tracking has been explored with online factor-graph optimization [19, 20, 51].
The aforementioned approaches are unimodal and rely on good pose initializations; MidasTouch is nonparametric and requires no such knowledge.
Additionally, Kelestemur et al.
[52] show category-level object pose estimation with a parallel jaw gripper.
Closely related is Tac2Pose [17, 18], which performs pose estimation of small-parts with the GelSlim [14].
It produces a distribution of object poses learned in tactile simulation from contact shapes.
Similarly, Gao et al.
[53] estimates contact location on objects through vision-based touch and audio with a small, discrete set of measurements.
We differentiate our work in both context and approach:
(i) we ﬁlter over long time horizons rather than single/multi touch predictions, (ii) tactile embeddings are learned from local surface geometry rather than images, and (iii) we consider objects considerably larger than the robot ﬁnger.
Place recognition for touch: A compact representation for tactile images enables easy frame-toframe or frame-to-model tracking.
Inspired by the computer vision community, a popular interme- diate is a learned embedding from either RGB [15, 19] or binary contact masks [17, 18].
Realistic tactile simulators, like TACTO [32], enable training these models to the scale and generality of arbitrary real-world interactions.
For example, Bauza et al.
[18] learn object-speciﬁc embeddings to match 2D contact shapes.
For contact-rich manipulation, local 3D geometry is a natural candidate, and iterative closest point (ICP) has shown promise for frame-to-frame tracking [20, 54].
For vision-based touch, 3D geometry is obtained either via photometric stereo [13, 55, 56], or image-to-heightmap models [20, 31, 57– 59].
However, ICP is only suitable for local tracking and not global localization: it is sensitive to initialization and is intractable to scale as a sampling-based measurement model.
Succinctly, what is an accurate and efﬁcient similarity metric for tactile geometry?
With the ubiquity of LIDAR/RGB-D data, matching unordered point sets is vital for place recognition in the SLAM community [36, 37].
Following the seminal PointNet [60], Choy et al.
[61] later developed efﬁcient and expressive sparse 3D convolution.
This backbone was used to aggregate local point features to global point-cloud embeddings for LIDAR place-recognition [37].
We show tactile geometries can be compacted into codes with the same architecture, for easy frame-to-model queries.
Problem formulation
For a vision-based tactile sensor dynamically sliding along a known object’s surface, our goal is to track the distribution of 6D sensor pose x t 2 SE (3) in the object-centric frame.
Such pose context is useful for downstream planning and control, for instance manipulating an object in-hand [7].
The sensor is afﬁxed on a robot ﬁnger and we have access to the end-effector pose, but its relative position and orientation on the object’s surface is unknown.
At a each timestep t , our measurements are a tactile image, and noisy sensor pose in the robot’s reference-frame, z t =
I t 2 R 240 ⇥ 320 ⇥ 3 , p t
SE (3) .
For simplicity, we assume the object is stationary and sliding is a single continuous contact interaction.
The dimensions of the objects are much larger than the sensor’s contact area.
We assume an uninformative initial prior for x t , but later show the beneﬁt of visual priors.
Global localization during sliding touch
MidasTouch comprises three distinct modules, as illustrated in Figure 1: a tactile depth network (TDN: Section 4.1), tactile code network (TCN: Section 4.2), and particle ﬁlter (Section 4.3).
At a high-level, the TDN ﬁrst converts a tactile image I t into its local 3D geometry P t via a learned observation model.
The 3D information is then condensed into a tactile code E t by the TCN through a sparse 3D convolution network.
Finally, the downstream particle ﬁlter uses these learned codes in its measurement model, and outputs a sensor pose distribution that evolves over time.
Throughout this work, we use 10 YCB objects with diverse geometries in our tests: sugar_box , tomato_soup_can , mustard_bottle , bleach_cleanser , mug , power_drill , scissors , adjustable_wrench , hammer , and baseball .
In Appendix G, we further demonstrate the method on small parts.
4.1 Tactile depth network (TDN)
The tactile depth network learns the inverse sensor model to recover local 3D geometry from a tactile image.
We adapt a fully-convolutional residual network [59, 62], and train it in a supervised fashion to predict local heightmaps from tactile images.
Network and training: This network uses a ResNet-50 backbone, with a series of upprojection blocks.
With the optical tactile simulator TACTO [32], we render a large collection of DIGIT images with ground-truth heightmaps.
The images are from simulated interaction with 40 YCB [24] object meshes, while holding out all test objects.
The images are rendered at 5000 different poses per object: randomizing for contact point, orientation, and indentation depth.
For sim2real transfer, we calibrate TACTO with real-world data from multiple independent DIGITs, and apply random lighting augmentations (refer Appendix A).
Per object, the split of train-validation-test is 4000:500:500 .
Image to 3D:
At each timestep, we pass the 240 ⇥ 320 RGB tactile image I t through the network to get heightmap H t .
To remove non-contact areas we compute a mask C t by depth thresholding, and get ˆ H t =
H t  C t .
Finally, the heightmap is reprojected to 3D via the camera’s known perspective projection model ˆ H t 7!
P t .
Evaluations on the test set show heightmap RMSE of 0 .
135 mm with respect to ground-truth.
Figure 2 shows examples of the TDN on real-world DIGIT interactions.
4.2 Tactile code network (TCN)
This network summarizes large, unordered point clouds of local geometry into a low-dimensional embedding space, or code.
If two sensor measurements are nearby in pose-space, they observe similar geometries and therefore, their codes will also be nearby in embedding-space.
However, the inverse is not necessarily true: measurements from two opposite corners of a cube may have similar codes, but their corresponding sensor poses are dissimilar.
This is an inherent challenge of tactile localization, the so-called contact non-uniqueness highlighted by Bauza et al.
In our work getting the most-likely modes of this distribution is sufﬁcient since the downstream particle ﬁlter can then disambiguate them temporally.
An analog in the SLAM community is the loop closure problem from 3D data.
LIDAR place recognition modules, such as PointNetVLAD [36] and MinkLoc3D [37], use metric learning to learn point cloud similarity.
While tactile images differ from natural images, the geometries from LIDAR and tactile sensors (normalized for scale) are similar.
The MinkLoc3D architecture, based on
MinkowskiNet [61], performs state-of-the-art point cloud retrieval through sparse 3D convolutions.
Network and training: The architecture comprises of three components: (i) Voxelization of P t into a quantized sparse tensor ˆ P t = { h ˆ x i , ˆ y i , ˆ z
i , 0 / 1 i } , (ii) Feature pyramid network [63] for per-voxel local features
ˆ P ft = { h ˆ x i , ˆ y i , ˆ z i , ˆ f i 2 R 256 i } , and (iii) Generalized-mean pooling for point cloud embedding vector E t 2 R 256 .
We use the pre-trained weights, learned from four comprehensive
LIDAR datasets [37], and ﬁne-tune them with TACTO data collected over the 40 YCB objects.
This is trained with a contrastive triplet loss, and we accumulate positive and negative local 3D geometries through pose-supervision.
Tactile codebook: Once we have a code E t , we would like to efﬁciently compare this with a dense set of contacts.
Inspired by prior work [18, 26], we build a tactile codebook comprising of M = {+
 k randomized+} sensor poses on each object’s mesh.
We evenly sample these points and normals on the mesh with random orientations and indentations (refer Appendix A).
We feed these poses into TACTO to generate a dense set of tactile images for each object.
We pass the generated images through the TDN + TCN to get a codebook for the speciﬁc object o :
C o = { h p [ m ] , E [ m ] i } Mm =1 , where p [ m ] are SE (3) sensor poses in codebook and E [ m ] are corresponding tactile codes.
Figure 3 shows a t-SNE visualization of the codebooks, a colorspace representation of local geometric similarity.
We observe regions with similar geometries have identical hues and the traversal of our sensor between these geometries will give us valuable measurement signals.
We build a KD-Tree with 6-element vectors: { [ p [ m ] trans , ↵ log ( p [ m ] rot )] } Mm =1 for nearest-neighbor search. Here, log ( · ) is the SO (3) logarithm map obtained via Theseus [65], and ↵ = 0 .
01 is the rotation scaling factor.
Thus, given a candidate pose x [ i ] t we do not have to render and obtain its corresponding code, but instead just perform a pose-space lookup C o ( x [ i ] t ) .
With memoization of code generation, we make getting codes for thousands of arbitrary pose particles tractable.
[Table]
[Figure]
p
[Figure]
Single-touch localization: To understand the effectiveness of codes as a proxy for pose prediction, we conduct a set of single-touch experiments in simulation.
This also provides an idea of which objects are salient, and which are adversarial, as highlighted in Figure 4.
We observe that we perform signiﬁcantly better than random touches for all objects, and those with symmetric, regular structures exhibit long-tail errors.
This shows that while a single-touch has meaningful signal, it can be valuable to disambiguate these readings temporally with a ﬁltering framework.
We compare against embeddings from tactile images in Appendix B.
4.3 Filtering over posed touch
The particle ﬁlter approximates the posterior distribution of the sensor pose as a set X t
= { h x [ i ] t , w [ i ] t i } N t i =1 , where x [ i ] t are SE (3) sensor pose particles, w [ i ] t are the predicted particle weights, and N t is number of particles.
Rather than a unimodal Gaussian, this arbitrary distribution captures the multi-modality of global localization.
We propagate the posterior distribution based on the stream of tactile images and noisy sensor poses, emulating a conventional particle ﬁlter but with a learned measurement model.
Additional implementation details can be found in Appendix C.
Initialization: The initial distribution X 0 is sampled from a coarse prior, which can (optionally) be obtained from vision [26].
Estimating contact location from vision is inherently noisy and can be ambiguous due to object symmetries.
In our experiments, we ﬁnd that even with very uninformative priors, our multi-modal ﬁlter can prune out hypotheses and converge to the correct one.
We sample particles from a 6D prior about the ground-truth pose x gt 0 as x 0 ⇠ N ( x gt 0 ,   trans   rot  ) , such that 3  trans = M diag and 3  rot = 180  .
This weak prior scatters the particles across the object surface, and allows comparison across objects as a function of their mesh diagonal length M diag .
In our results, we also show particle ﬁlter ablations with tighter pose priors.
After sampling, we project the particles back onto the surface by querying their nearest neighbors from the tactile codebook.
Motion model: We typically have access to noisy global estimates of end-effector pose, p t , via robot kinematics.
Our sensor odometry comes from the relative pose predictions  p t = ( p t  1 )  1 · p t .
The motion model propagates particles X t  1 forward by sampling from a state transition probability:
[Equation]
Additionally, odometry can also be from marker ﬂow [66] or image-to-image tracking [19, 20].
Measurement update: In this step, we update the particle weights based on how their tactile codes match the current measurement E t , as a function of cosine distance.
At each timestep, we lookup the codebook for particle codes (Section 4.2) and perform a matrix-vector multiplication:
[Equation]
These weights are scaled [0 , 1] and represent how well the candidate particle poses match with the current measurement.
These weights feed into the subsequent resampling step.
Particle resampling: We sample a new set of particles X t from the proposal distribution with a probability proportional to their weights.
Low-variance resampling [67] is a popular solution that covers the sample set in a systematic manner.
Hypothesis clustering: Alongside a distribution of poses, downstream tasks may need distinct pose hypotheses.
To achieve this, we hierarchically cluster particles in R 3 using DB-SCAN [68].
We average the cluster positions and quaternions [69] to get a hypothesis set h t :  x 1 t . . .
x Ht .
The YCB-Slide dataset
To evaluate MidasTouch, and enable further research in tactile sensing, we introduce our YCB-Slide dataset (more in Appendix D).
It comprises of DIGIT sliding interactions on the 10 YCB objects from Section 4.
We envision this can contribute towards efforts in tactile localization, mapping, object understanding, and learning dynamics models.
We provide access to DIGIT images, sensor poses, ground-truth mesh models, and ground-truth heightmaps + contact masks (simulation only).
Simulated interactions: We simulate sliding across objects using TACTO, mimicking realistic interaction sequences.
The pose sequences are geodesic-paths of ﬁxed length L = 0 .
5 m, connecting random waypoints on the mesh.
We corrupt sensor poses with zero-mean Gaussian noise  trans = 0 .
5 mm,  rot = 1  .
We record ﬁve trajectories per-object; 50 interactions in total.
A representative sample of the trajectories, along with their accrued tactile geometries are shown in Figure 5.
Real-world interactions:
In the real-world, we perform sliding experiments through handheld operation of the DIGIT.
We keep each YCB object stationary with a heavyduty bench vise, and slide along the surface and record 60 s trajectories at 30 Hz.
We use an OptiTrack system for timesynced sensor poses, with 8 cameras tracking the reﬂective markers.
We afﬁx six markers on the DIGIT and four on each test object.
The canonical object pose is adjusted to agree with the groundtruth mesh models.
Minor misalignment of sensor poses from human error are rectiﬁed by postprocessing the trajectories to lie on the object surface.
We record ﬁve logs perobject, for a total of 50 sliding interactions.
Our experimental setup and dataset are illustrated in Figures 5 and 6.
Experimental results
Simulation: We evaluate MidasTouch over the 50 trajectories collected in Section 5.
As the ﬁlter is non-deterministic, each trajectory is run 10 times for averaged results over 500 trials.
Figureshows qualitative results for three representative trajectories.
At each timestep, we visualize the pose distribution and plot the average particle RMSE with respect to the ground-truth pose.
We see convergence to the most-likely pose hypothesis over the sliding interactions.
Upon convergence, the hypothesis clustering step sets an averaged pose for each cluster: visualized as the pose axes with uncertainty ellipses.
We also highlight the current tactile image, surface geometry, and comparison with the tactile codebook.
These trials are initialized with pose prior [  trans ,  rot ] from Section 4.3.
Figure 8 [left] accumulates quantitative metrics over the 500 simulated trials.
First, we show the ﬁnal pose errors across all trials compared against the initial error.
Overall, the averaged ﬁnal pose errors are 0 .
74 cm and 9 .
43  ; the per-object errors roughly correlate to the singletouch errors from Figure 4.
While the error drops signiﬁcantly for most trials, it ﬂuctuates depending on each trajectory’s salient geometries (or lack thereof).
Larger pose-errors can be at- mustard _ bottle tomato
[Table]
baseball
[Table]
[Figure]
[Table] [Figure] [Figure]
[Table]
[Figure]
tributed to tracking multiple modes that equally explain the same sliding sequence (please refer to supplementary video).
We consider this a beneﬁt of our multi-modal framework, and is especially prevalent for symmetric objects like the sugar_box , mustard_bottle , and mug .
[Figure]
We perform ablations over pose prior (Figure[right]), initializing each trial with uncertainty  ⇥ [  trans ,  rot ] .
This serves as a proxy for visual-priors; better initializations lead to fewer candidate modes.
The full-stack operates, on average, at the rate of 10Hz. Real-world: We run 500 similar trials for the realworld data from Section 5 with  = 0 .
5 , which serves as a bound for a reasonable prior estimate available from vision.
The real-world tactile heightmaps are noisier than simulation, thus a tighter initialization prevents particle depletion.
In Figure 10 we present three representative trajectories that show the hypotheses converge to the ground-truth pose.
The accumulated [Table] bleach [Table]
cleanser [Table] [Figure] statistics in Figure 9 shows reduced ﬁnal error across all objects, except baseball .
The averaged ﬁnal pose errors are higher than the simulation results: 1 . 97 cm and 21 .
48  .
Once again, we see tools and intricate objects are the easiest to localize on while symmetric objects are the hardest.
An anomaly is the baseball , on which we fail to localize in all trials.
We attribute this to a lack of distinct geometry detected from real tactile images, effectively meaning we are trying to localize on a featureless sphere.
These failure modes are highlighted in Figure 11.
We present best-hypothesis error metrics and further qualitative results in Appendix E
[Figure]
Conclusion and limitations
[Figure]
In this work [Figure] we demonstrate ﬁnger-object global localization from posed tactile images.
This online method outputs a pose-distribution on the object’s surface that converges over time as the sensor traverses salient surface geometries.
Speciﬁcally, our system is the ﬁrst to learn tactile embeddings from local 3D geometry, and disambiguate them with a nonparametric particle ﬁlter.
Our experiments demonstrate the surprising effectiveness of pairing learned tactile perception with Monte-Carlo methods to resolve distribution ambiguities.
Limitations:
The current formulation is limited to a moving sensor relative to a ﬁxed-pose object (or vice-versa).
In future work, we wish to incorporate a dynamic object in our motion model through (i) visual measurements [26], and/or (ii) local in-hand tracking [20].
Our method currently does not work in scenarios where we lack ground-truth object models.
Along with reconstructing objects [70], we would need to build the tactile codebook on-the-ﬂy.
For future in-hand manipulation tasks, it is also necessary to scale MidasTouch to multi-contact conﬁgurations.
Our on-surface assumption leads to poor behavior when we break contact with the object, and the effects of shear on the tactile image sequence have been ignored [71].
Object compositionality doesn’t hold for deformable or articulated objects, so modeling these properties is an interesting future direction [72].
With a differentiable ﬁlter [73] we can ﬁne-tune the system end-to-end for more robust real-world performance. Acknowledgments
We thank Wenzhen Yuan, Ming-Fang Chang, Wei Dong, Maria Bauza Villalonga and Antonia Bronars for insightful discussions.
We are grateful towards the CMU AI Maker Space and Greg Armstrong for facilitating the collection of the YCB-Slide dataset.
The authors acknowledge funding from Meta AI, and this work was partially carried out while Sudharshan Suresh interned at Meta AI.