Paper: The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning (ICLR-2024)

Reviewer comment (Experiments):
A prevalent challenge in contemporary research within this domain is tactile simulation. The tactile simulation employed in this study lacks the desired accuracy. Moreover, the introduced method hasn't been empirically tested or combined with tactile sensors in real-world experiments, which is a missing link for comprehensive validation.

Author response:
MuJoCo’s contact force optimization is equivalent to a penalty-based approach, which is essentially the same used to compute force maps in [A] and [B], with those works showing zero-shot transfer to the real-world for the specific tasks considered. Additionally, there are other remarkable examples of sim-to-real transfer using MuJoCo [C], [D]. Based on such evidence, we would like to highlight that MuJoCo’s touch grid is currently the most realistic general-purpose simulator of tactile maps, especially given the wide range of possibilities for proper system identification. We expect this very important clarification to change the reviewer’s evaluation of the contribution.

While the main focus of our work was on studying how to extract stronger representations when visual and tactile data are available, we also showed some promising direction (e.g., improvement of vision-only policies with tactile-enabled representation training) for real-world deployment, which we leave as future work.

Reviewer comment (Experiments):
The ViT model, as utilized, is computationally intensive and has a propensity to overfit to specific problems. The potential benefits of ViT's self-supervised pre-training techniques, such as MAE, ought to be realized across a diverse range of datasets or environments. Given the constraints observed in the study—limited tasks, a restricted set of objects, and minimal variations in background textures—it's challenging to fully endorse the promise of the proposed approach.

Author response:
Thanks for suggesting this comparison! We indeed compared to MVP [E], which is basically relying on ViT representations pretrained across datasets, and on an additional baseline using CLIP [F] visual encoder. As shown in Figure 10b of the updated appendix, both M3L and its vision-only MAE variation outperform MVP and CLIP representations on the in-domain learning task (tactile insertion). We believe this is due to the fact that we let the representations further adapt during task learning. Note that training such MVP and CLIP baselines was even more computationally intensive than training our all-in-one approach, due to the inference of the much larger MVP (22M) and CLIP (88M) encoders, compared to the small ViT used in M3L (5M).

We omit generalization results on the new baselines due to the low success rates on the in-domain task.

Reviewer comment (Experiments):
The simulation results are interesting in terms of the same algorithm being able to learn policies for very different tasks (as mentioned in the strengths) but it is unclear why the experiments use only three tasks. Most reinforcement learning in simulation papers show results on a large number of environments, and this is important to understand the capabilities of the proposed approach without over-indexing on a particular environment.

Author response:
**#2: Is it possible to have more than three simulation tasks?**

We agree that evaluation on a large variety of tasks would be meaningful and thus, we are currently working on further extending the number of tasks. 

We want to emphasize that our three environments are the **first** RL environments both with visual and **high-resolution tactile** sensing on a general-purpose physics engine like MuJoCo. This required carefully designing and adapting each of the tasks for this work, as opposed to other fields in RL that have the possibility of benchmarking on existing environments. To give an idea of the process involved, to increase the number of contact points on the Shadow Hand, we needed to split each collision mesh in a much higher number of meshes making sure that each of these was still convex (as required by MuJoCo), which is a non-trivial process for each robot embodiment. Ours is an effort in this direction, and open-sourcing such environments will hopefully give the opportunity to fellow researchers to further extend the benchmarking opportunities in the area. As the reviewer pointed out, however, we purposely focused on tasks that are very different amon each other to cover a wider spectrum of applications.

Reviewer comment (Experiments):
There are no external baselines compared against. All the comparisons are with variants of the proposed M3L algorithm. The related works in representation learning that use vision-only observations, including papers that do pre-training for representation learning, are relevant for empirical comparisons (for example [4,5]).

Author response:
**#3: There are no external baselines compared against. All the comparisons are with variants of the proposed M3L algorithm.**

As suggested by the reviewer, we have included comparisons to other approaches in Figure 10 of the updated version: contact location-based tactile representations [A] and vision-only pre-trained representations (MVP [D], CLIP [E]). 

While [A] is contemporaneous work, we found it interesting to compare with a similar transformer architecture, trained in a single stage from vision and touch. In particular, we also investigate the tactile representation used in [A], that is, discretized contact locations. In contrast, our approach uses intensity and directions of spatially distributed force vectors (force maps). The results are shown in Figure 10a of the updated appendix, with the discretized contact location approach outperformed by both our M3L and end-to-end baselines even for the in-domain learning task (tactile insertion).

We also added comparisons to baselines based on vision-only representations extracted through MVP [D] and CLIP [E] visual encoders, kept frozen during PPO training. Note that both encoders are much larger than the one used in M3L (5M for M3L, 22M for MVP, and 88M for CLIP), leading to considerably slower training. The results are shown in Figure 10b of the updated appendix, with both M3L and its vision-only MAE variation outperforming MVP and CLIP representation on the in-domain learning task (tactile insertion).

We omit generalization results on the new baselines due to the low success rates on the in-domain task.

Reviewer comment (Evaluation):
The introduction mentions "M3L demonstrates better zero-shot generalization to unseen objects and variations of the task scene"; however, the experiments show only very weak generalization to minor changes in the object (in particular mass, damping, friction variations). Hence the claim is not really substantiated. It is important to show generalization to actually different objects (for example with geometric variations like shape and size, in addition to the current variations). Also, the results should evaluate this for a number of different objects and not just a single new object, in order for the conclusions to be statistically significant.

Author response:
**#4: The experiments show only very weak generalization to minor changes in the object (in particular mass, damping, friction variations). It is important to show generalization to actually different objects.**

In the tactile insertion task, we indeed test the generalization capability to **different objects**, as illustrated in Figure 4 and Figure 8.

Moreover, we would argue that other perturbations in the objects are not minor, i.e., 10x friction and damping, 2x weight. In addition, we test on different object poses in Door Opening and camera poses in In-Hand Rotation.

Reviewer comment (Experiments):
Manipulation with tactile sensor and visual input has been evaluated in the real world for different tasks[1]. To actually validate the performance of the proposed method, the real world evaluation is required. Especially for noisy tactile sensory information.

Author response:
**#1: Manipulation with tactile sensor and visual input has been evaluated in the real world for different tasks [A]. To actually validate the performance of the proposed method, the real world evaluation is required. Especially for noisy tactile sensory information.**

We would like to highlight that [A] is a contemporaneous work according to ICLR guidelines, having been arxived only 10 days prior to the submission of this paper. We are however happy to cite and compare to [A] (see also point #2). 

[A] trained an end-to-end baseline (in two privileged learning stages) and achieved sim-to-real transfer through large-scale GPU parallelization and domain randomization. The focus of our work was instead on studying how to extract stronger representations when visual and tactile data are available. We also showed some promising direction (e.g., improvement of vision-only policies with tactile-enabled representation training) for real-world deployment, which we leave as future work.

Regarding the robustness to noise, please see our response to #7.

Reviewer comment (Experiments):
To validate the effectiveness of MAE architecture and self-supervised objective in this application, additional baselines (not limited to the baselines i mentioned) need to be compared. Qi, et al utilize a visuotactile transformer to model sequential observations. Hansen, et al show promising result in applying self-supervised learning in visual RL.

Author response:
**#2: Additional baselines need to be compared.**

Thank you for your suggestion. We have compared our method with additional baselines: contact location-based tactile representations [A] and vision-only pre-trained representations (MVP [B], CLIP [C]).

Figure 10a of the updated appendix shows that both our M3L and end-to-end baselines outperform the contemporaneous work that uses discretized contact location approach [A] for the in-domain learning task (tactile insertion), confirming the advantages of using high-resolution force maps as opposed to low dimensional contact quantities.

The pre-trained representations, MVP [B] and CLIP [C], also show slower training curves in Figure 10b. Note that the encoders used in MVP (22M parameters) and CLIP (88M parameters) are much larger than our M3L (5M parameters).

We omit generalization results on the new baselines due to the low success rates on the in-domain task.

Reviewer comment (Experiments):
Missing tactile-only baseline, to fully address the essence of the multimodal representation.

Author response:
**#3: Missing tactile-only baseline.**

This is a great point! We have run additional experiments with the tactile-only baseline. Interestingly, although the Door Opening task can be fully solved without vision, M3L (vision + tactile) achieves much faster and better learning results than the tactile-only baseline. The benefit of M3L could come from (1) our multimodal representation learning and (2) the use of visual observations. We included this result in Figure 11 of the updated paper.

Note that the insertion and hand tasks are misspecified when vision is missing, because of missing target and incomplete state information. We ran such a baseline on the insertion task to confirm such a claim (see Figure 11).

Reviewer comment (Evaluation):
Out of three tasks, the proposed framework only outperforms vision-only baselines in In-hand rotation tasks. It seems tactile insertion and door-opening tasks are not quite tasks to evaluate the additional usage of tactile information.

Author response:
**#5: Out of three tasks, the proposed framework only outperforms vision-only baselines in In-hand rotation tasks.**

Our results show that 1) vision+touch policies perform best when investigating zero-shot generalization, 2) training representations with vision+touch also improve vision policies, 3) representation learning approaches outperform end-to-end approaches when learning in-domain. 1) and 2) are shown in Figure 5 in the paper, while 3) is shown in Figure 6 in the paper. These results are consistent across all tasks, so we disagree with the reviewer comment. Please let us know if you have any other concerns regarding this.

Reviewer comment (Experiments):
The result in Figure 10, shows the learned MAE could reconstruct almost the full observation with quite limited observation. I’m wondering whether the learned model can show the same level of performance if there are noise in visual observation or tactile reading. Otherwise it looks like the model is overfitting to the training distribution.

Author response:
**#7: Can the learned model show the same level of performance if there is noise in visual observation or tactile reading?**

Thanks for the great suggestion! We added a noise robustness analysis in Figure 12 of the update appendix, where we add zero-mean Gaussian noise to both visual and tactile data at inference time. The figure shows the performance of M3L vs the end-to-end for increasing noise variance, highlighting how M3L success rates do not degrade up to a reasonable degree of disturbances and also how M3L consistently outperforms the end-to-end approach in terms of robustness.