Paper: Learning to Jointly Understand Visual and Tactile Signals (ICLR-2024)

Reviewer comment (Reproducibility):
The cross-modality query necessitates an optimization process. Therefore, it's crucial to provide information regarding the time cost of these experiments. For instance, how much time would be required to employ the neural field in this inverse manner?

Author response:
We thank the reviewer for bringing this to our attention. We have included section A6 for more detail on the time for test time optimization.

Specifically, for test-time optimization of the latent code of our proposed approach, we use Adam optimizer to run 1000 time steps with a batch size of 32. For 1000 time steps, we use 56 seconds in total, 17.87 iteration/second, on average every example uses 1.75 seconds. With better computational resources or faster optimizers, the inference speed can be faster.

Reviewer comment (Presentation):
The absence of videos in the paper is a notable limitation. Including video content could significantly enhance the understanding of the tasks and experiments conducted in the study.

Author response:
Thank you for this constructive feedback. We have updated our website to include videos on our [supplementary website](https://sites.google.com/view/iclr-submission-force-vision)

Reviewer comment (Reproducibility):
Do you want to claim the dataset as one of the contribution? In another word, would you open source the dataset once the paper is accepted?

Author response:
Thank you for the question, and yes, we will open-source the dataset once the paper is accepted. We have updated our details on the dataset that will be open-sourced for public use.

Reviewer comment (Writing):
Could you clarify the symbol $\gamma$ used in Equation 2? I was unable to locate a definition for it within the text.

Author response:
Thank you for bringing this up. This is a typo. We have updated our draft with the correct formulation. Specifically, $L_{Rec} = | \phi_{c} (z_{c, i}) - c_i|^2 + | \phi_{h} (z_{h, i}) - h_i|^2 + | \phi_{x} (z_{x, i}) - x_i|^2 $

Reviewer comment (Theory):
Your elaboration on $I_i$ and $I_j$ would be appreciated, specifically in relation to the following sentence. How is the distance within the space of $I$ quantified, and how is the subtraction operation defined in Eq3 for $I_i$ and $I_j$, given that $I$ is a three-modality tuple? While I recognize that the manifold isometry loss is a standard loss in the manifold learning field, I would like to confirm if the subtraction operation is a simple reduction operation in the raw data format.  
> any two samples sampled from the signal agnostic manifold $\{z_i, z_j \} ⊆ M$ respects the distance between the samples $I_i, I_j$

Author response:
Thank you for this question. We have updated our draft for more clarity on the manifold isotropy loss. Specifically, we allow the distance between two sampled latent codes to be equal to the distance between the explicit signals expressed by the latent codes, using $d_z(i, j)$ to denote the latent distance between two latent codes $z_i,z_j$, and use $d_\mathcal{I}(i, j),\mathcal{I} \in \{x, c, h\}.$ to denote distance between 2 signals:  $\| c_i - c_j \|$, $\| x_i - x_j \|$, or $\| h_i - h_j \|$. In our formulation: 
    
$$L_{Iso} = d_z(i, j) - d_{\mathcal{I} (i, j)} \\ =  \left(d_z(z_{c,i}, z_{c, j}) - d_{\mathcal{I}} (c_i, c_j)\right) +\left(d_z(z_{h,i}, z_{h, j}) - d_{\mathcal{I}} (h_i, h_j)\right)+ \left(d_z(z_{x,i}, z_{x, j}) - d_{\mathcal{I}} (x_i, x_j)\right)$$
$$= \left( \| \|(z_{c, i}) - (z_{c, j})\| - \| c_i - c_j \|   \| \right)+  \left(\| \|(z_{x, i}) - (z_{x, j})\| - \| x_i - x_j \|   \| \right) + \left(\| \|(z_{h, i}) - (z_{h, j})\| - \| h_i - h_j \|  \| \right) $$
 $$= \left( \| \|\Gamma_c({z_i}) - \Gamma_c({z_j})\| - \| c_i - c_j \|   \| \right)+  \left(\| \|\Gamma_x({z_i}) - \Gamma_x({z_j})\| - \| c_i - c_j \|   \| \right) + \left(\| \|\Gamma_h({z_i}) - \Gamma_h({z_j})\| - \| h_i - h_j \|  \| \right) $$
    
We have updated our manuscript to reflect this in equations 4 and 5 in Section 4.2.

Reviewer comment (Experiments):
My major concern is since the training set : testing set is aroud 12:1, and only include 4 categoreis, can this method really generalize to unseen objects? How is the diversity of the training and testing set?

Author response:
Thank you for raising this concern, we have revised our manuscript for increased clarity on the dataset statistics, shown in Section 3.  we provide more details and more description on the details of our dataset and train-test split. Our training and test set is 6:1, with 124,000 frames of data for training, (10,000 + 6,000 + 5,000) frames of data for testing. The details are as follows

- Our training set contains 81 objects spanning 4 different categories, with paired tactile and video recordings of manipulation, containing 124000 frames of data.
- Our test set is constructed with 3 different subsets:
    - in-distribution test-set: constructed with withheld sequences of the same 81 objects seen during training, containing 10,000 frames of data.
    - in-category test-set: contains withheld 4 objects unseen during training, containing around 6,000 frames of data.
    - out-of-category test set: we collect data on 4 objects of unseen categories, containing 5,000 frames of data.

We test our proposed method on novel instances of a seen category as well as a novel category as shown in Section 5.3 in our paper. We do want to note that with more data, our proposed method can achieve better generalization.

Reviewer comment (Presentation):
Presentation with Figure1: It would be better to also draw the process of getting shared latent space in the Figure1 for better understanding, it's quiet hard to undertand how to get a shared latent space from signals that are different dimension, could the author illustrate more about this? Also projection layer of x seems missing in Figure 1.

Author response:
Thank you for the constructive feedback. We have updated the figure and writing for more clarity, reflected in Section 4.1 and 4.2 in blue. 

With regard to the shared latent space, Existing methods, such as TouchandGo, ObjectFolder, VisGel, etc., employ encoder-decoder techniques where the latent codes are constructed and obtained from encoding a specific modality of information. However, we use an auto-decoder technique, using different branches of decoders to construct different modalities of information (tactile/vision). In this way, we allow gradients from different modalities of information to simultaneously update and optimize the latent embedding. In our implementation, we follow the DeepSDF implementation and randomly initialize our shared latent embedding to be trainable. During training, we sample a latent code $z_i$ to decode to different through different neural fields (coordinate networks), tactile signal $I_h = \phi_h (\Gamma_h(z_i))$ and video signal $I_x = \phi_x(\Gamma_x(z_i))$. 

In order to handle information of different dimensions, we use neural fields or coordinate networks. As described in Section 3.1 in our paper, neural fields are different from traditional convolution networks whose construction is significantly bound by the dimension of signals. Neural fields take in coordinates $u$ and some latent code $z_{k}$ that contains information about the signal $k \in \{ c, x, h\}$  to decode value at that coordinate $u$.  Formally, the value of the $i$-th sample for signal $k$ at coordinate $c$ is obtained: 
$I_{k, i, u} = \phi_k ( z_{k, i}, u )$. 
In our implementation, we use MLP that takes in input of 
$d_z + d_u$ 
dimensional to decode signal values 
$R^{d_{I}}$
 at coordinate 
$u$, $\phi_j$ 
where 
$\phi_j: R^{d_z + d_u} \rightarrow R^{d_I}$. 
For more information and interest in neural fields, please refer to [3]

We have updated our text and visual presentation on the construction of the video projection layer in our manuscript to provide more detail and clarity. 
The video projection layer $\Gamma_x$ is constructed by concatenating the image projection layer and tactile projection layer, For the details of each of our projection layers:

$z_{c, i} = \Gamma_c(z_i), \Gamma_c \in \mathbb{R}^{256 \times 256}$ 

$z_{h, i} = \Gamma_h(z_i), \Gamma_h \in \mathbb{R}^{256 \times 16}$ 

$z_{x, i} = \{\Gamma_c; \Gamma_h\}(z_i) = \{\Gamma_c(z_i) ; \Gamma_h(z_i)\} = \{z_{c, i} ; z_{h, i}\}$   
or equivalently, $\Gamma_x = \{\Gamma_c; \Gamma_h\} \in \mathbb{R}^{256 \times 272}$

[1] Gao et al. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations. CoRL, 2021.
[2] Yang et al. Touch and go: Learning from human-collected vision and touch. NeuIPS
[3] Li et al. Connecting touch and vision via cross-modal prediction. CVPR June 2019
[4] Xie, Y., Neural fields in visual computing and beyond. In *Computer Graphics Forum*.

Reviewer comment (Reproducibility):
How is the robustness of the tactile glove, will it need to calibrate a lot to make sure the tactile data is accurate?

Author response:
Thank you for the question. we have included more details on the background of the gloves and glove calibration in the dataset section in Appendix section A.1. 
Specifically, the glove only needs to be calibrated once. The glove is made using piezoresistive film. The working principle of this material is when the material stretches, it causes the deformation of carbon nanotubes, which then induces the relative electrical resistance changes and thus difference in voltage readouts. The material in general is sturdy and the material property does not change much with wear and tear. Therefore, the glove only needs to be calibrated once to convert the sensor readouts in voltages to force magnitudes in Newton before use.

Reviewer comment (Presentation):
The dataset would benefit from enhanced visualization and in-depth details, possibly within the appendix.

Author response:
Thank you for this feedback. We have updated our manuscript to include more details about our dataset both in Section A2 in the Appendix and on our [supplementary website](https://sites.google.com/view/iclr-submission-force-vision/). We added a demo figure of video sequences of our tactile and RGB signal from 12 objects sampled from our dataset, as well as some video sequences of experiments. We have also added more details about our dataset collection and dataset statistics in Section 3 of our paper. We also added a Section to cover more background on the tactile glove and sensor calibration. We hope these modifications improve the clarity of our paper about our dataset.

Reviewer comment (Writing):
There's a typographical error on page 5 after equation 2; "Additioanlly" should be corrected to "Additionally."

Author response:
We have corrected this typo. Thank you for bringing this to our attention.

Reviewer comment (Experiments):
Based on observations from figures 3 and 4, the sequences appear to have minimal variation across different frames. Displaying greater variation would add value. Additionally, considering a baseline that utilizes only the initial frame, as opposed to the entire sequence data, could provide intriguing insights.

Author response:
Thank you for the feedback. We hope that the video demo of our dataset on our supplementary webpage helps provide a better illustration of the signal variation across different frames. 
    
We added a test that only the first frame as the test-time optimization energy signal and added it to our ablation experiments. Empirically, we observe that it only using first frame decreases cross-modal retrieval accuracy. We believe that this difference is caused by the increase of variational inference, or in other words, with less optimization signal, there are now more local minima during test-time optimization that fit the optimization objective. More details can be found in Sec. A4 in our appendix. We would appreciate more details on the kind of insights or the kind of first frame setup you may be looking for.