Paper: Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (ICLR-2024)

Reviewer comment (Experiments):
The method proposed in this paper may not be directly applicable to scenarios where the training set has missing modalities.

Author response:
This is a good point. Indeed if a modality is completely missing from the training set, then our method (and almost all existing methods) cannot compensate for that. However, if the training set has some samples missing for some modalities, we can pretrain a network with the available modalities and then perform parameter-efficient network adaptation to compensate for the missing modalities at the test time. Since our approach is mainly focused on adapting a pretrained model to different missing modality situations, we can use the same adaptation procedure using a different base network. The performance in this case will probably depend on the quality of the base model pretrained with missing samples from different modalities.

Reviewer comment (Experiments):
The method proposed in this paper lacks extensive comparisons with other methods~(for example, [1,2,3]) in the context of multimodal classification tasks, which significantly undermines the persuasiveness of Table 5.

Author response:
Our work focuses on parameter efficient adaptation techniques for existing multimodal models. The main goal is to adapt already pretrained models to different missing modality scenarios. While we cited the first two papers (\[1, 2]) in our work, we did not compare with them for the following reasons:\
\
**Paper \[1]** suggested multitask learning and search for optimal fusion strategy to enhance robustness (mainly for two modalities). Extending the methods for masking out attention for more than two modalities and for tasks that rely on all the tokens is not obvious. Furthermore, the paper did not provide a codebase or results for the datasets we used in our experiments; therefore, we could not compare with them.\
\
**Paper \[2]** suggested Shared Specific Feature Modelling (ShaSpec), which is based on modeling and fusing shared and specific features. The paper focused on medical imaging (brain tumor segmentation) and classification. They did not provide any baseline for the datasets and tasks that we used in our work. That is why we did not compare our results with the paper.\
\
**Paper \[3] -** This paper was submitted to arxiv 12 days after the ICLR submission deadline. For this reason, we could not cite or compare with this paper. We will be happy to cite the paper in the revised version. The paper did not provide code or baseline results for the datasets and tasks that we used in our paper. For these reasons, we are unable to provide a comparison.

Reviewer comment (Novelty):
Although the idea is simple and effective for achieving parameter-efficient adaptation of models, the application of existing techniques (i.e. low rank adaptation, scale and shift transformations, and BitFit) to adapt a pre-trained model on a subset of modalities seems low from novelty aspect.

Author response:
**Although the idea is simple and effective for achieving parameter-efficient adaptation of models, the application of existing techniques (i.e. low rank adaptation, scale and shift transformations, and BitFit) to adapt a pre-trained model on a subset of modalities seems low from a novelty aspect.**\
We agree with you that the idea is simple and effective. Low rank adaptation, BitFit and Scale and Shift are originally designed for efficient model finetuning and transfer learning. We show that they can also be applied to adapt models to different missing modality scenarios.\
   1\) To the best of our knowledge, we are the first to show that parameter efficient adaptation can be used to enhance performance in different missing modality scenarios. While it may be simple, we hope this will be informative and useful for the broader community.\
   2\) We provide a generic framework for missing modality adaptation that can be applied to a wide range of multimodal datasets, tasks, and models.\
   3\) The adapted model can easily switch to different network states based on the available modalities with minimal latency, computational, or memory overhead.

Reviewer comment (Evaluation):
There is no insight and/or analyses into how the proposed paramater-efficient is helpful towards making the model robust to missing modalities. It is important to develop the grounding of the proposed method since only showing improvements over the baseline with empirical results doesn't help much towards this.

Author response:
**There is no insight and/or analyses into how the proposed parameter-efficient is helpful towards making the model robust to missing modalities. It is important to develop the grounding of the proposed method since only showing improvements over the baseline with empirical results doesn't help much towards this.**\
Our main motivation and insight for the proposed approach is to adapt the functional representation of the network according to the available modality combination. For instance, if one of the modalities is missing, we want to modify the feature representations and fusion of other modalities. We could do this by changing the entire network for every modality combination, which is infeasible due to computational and storage cost. In this paper, we perform parameter-efficient adaptation to achieve the same goal. A rigorous analysis of how the network adaptation changes the functional representation of the network will certainly be great, but it is a nontrivial task and beyond the scope of one paper.

Reviewer comment (Experiments):
The paper doesn’t mention any clear justification on why SSA was chosen over the LoRA and BitFit to report the results in the paper.

Author response:
**The paper doesn’t mention any clear justification on why SSA was chosen over the LoRA and BitFit to report the results in the paper.**\
We have discussed this in the second paragraph of Section 3.3. We primarily selected the SSF technique because of its simplicity and effectiveness. SSF has several benefits over other adaptation methods:\
   1\) The parameters (γ, β) are independent of the input features, which makes SSF applicable to diverse tasks and input modality combinations. \
   2\) We can easily insert SSF layers in the existing model without changing the model architecture.\
   3\) We can easily switch/select the corresponding SSF parameters for a given input modality combination.\
   4\) SSF works relatively better than LoRa, BitFit and Norm in most of the scenarios as shown in Table 4.

Reviewer comment (Experiments):
Comparison is missing with any of the existing method for the task of multimodal sentiment analyses.

Author response:
**Comparison is missing with any of the existing methods for the task of multimodal sentiment analysis.**\
Thank you very much for pointing this out. We are including the comparison with existing methods for multimodal sentiment analysis in the table below. All the results are generated using the codebase available at: [**https://github.com/thuiar/MMSA**](https://github.com/thuiar/MMSA) with the same settings that we used for our experiments.

   |             |                |           |           |           |           |              |           |           |           |
   | :--------- | :------------ | :-------: | :-------: | :-------: | :-------: | :----------: | :-------: | :-------: | :-------: |
   |   Datasets  |     Methods    |    Audio  ||   Visual             || Audio-Visual          ||  Average         ||
   |             |                |    ACC    |     F1    |    ACC    |     F1    |      ACC     |     F1    |    ACC    |     F1    |
   |  CMU -MOSI  |    MulT \[1]   |   48.31   |   40.98   |   52.44   |   51.77   |     48.93    |   41.95   |   49.89   |   44.90   |
   |       |    MFN \[2]    |   56.86   |   44.81   |   55.95   |   42.94   |     56.86    |   51.07   |   56.56   |   46.27   |
   |       |    TFN \[3]    |   42.23   |   25.07   |   42.38   |   25.40   |     42.23    |   25.07   |   42.28   |   25.18   |
   |       | BERT\_MAG \[4] | **57.77** |   42.31   | **57.77** |   42.31   |   **57.77**  |   42.31   | **57.77** |   42.31   |
   |       |    LMF \[5]    |   42.23   |   25.07   |   43.14   |   27.54   |     43.29    |   27.61   |   42.89   |   26.74   |
   |       |      Ours      |   50.00   | **46.71** |   54.88   | **54.39** |     55.49    | **53.96** |   53.46   | **51.69** |
   | CMU - MOSEI |    MulT \[1]   |   37.15   |   20.12   |   38.28   |   23.70   |     41.91    |   32.78   |   39.11   |   25.53   |
   |       |    MFN \[2]    |   58.48   | **58.31** |   60.35   |   59.48   |     59.74    |   60.37   |   59.52   | **59.39** |
   |       |    TFN \[3]    |   37.15   |   20.12   |   37.15   |   20.12   |     37.15    |   20.12   |   37.15   |   20.12   |
   |       | BERT\_MAG \[4] |   62.83   |   48.50   |   61.39   |   49.70   |     62.83    |   48.51   |   62.35   |   48.90   |
   |       |    LMF \[5]    |   42.38   |   34.48   |   57.15   |   57.85   |     55.94    |   56.63   |   51.82   |   49.65   |
   |       |      Ours      | **62.85** |   55.55   | **62.49** | **60.00** |   **63.32**  | **60.69** | **62.89** |   58.75   |

   From the table above, we can see that for the CMU-MOSI dataset, BERT\_MAG works better in terms of accuracy but our method works better in terms of F1 score. One thing to mention is that BERT\_MAG uses a pre-trained BERT model and finetunes it on the dataset but we are not using any pre-training. For CMU-MOSEI, our adaptation method works better for most of the cases.

Reviewer comment (Experiments):
Why CMU-MOSI and CMU-MSEI datasets were chosen to evaluate and report the performance of the method?

Author response:
**Why were CMU-MOSI and CMU-MSEI datasets chosen to evaluate and report the performance of the method?**\
We wanted to evaluate our method for diverse modality combinations. We have shown results for RGB, depth and thermal modalities for multimodal semantic segmentation. Multimodal material segmentation uses RGB, thermal, angle of linear polarization and degree of linear polarization. Apart from these modalities, we also wanted to test our approach on other modalities including video, audio and text. CMU-MOSI and CMU-MSEI datasets have those modalities which corroborate the fact that our approach can be applied to a diverse number of modalities.

Reviewer comment (Evaluation):
What are the possible reasons for relatively lower performance of other adaptation strategies, compared to SSA, in Table 4? Some explanation would be helpful to better understand their comparison in missing modalities scenario.

Author response:
**What are the possible reasons for relatively lower performance of other adaptation strategies, compared to SSA, in Table 4? Some explanation would be helpful to better understand their comparison in the missing modalities scenario.**\
SSF learns parameters ($\gamma$, $\beta$) for scaling and shifting the intermediate features. We can consider BitFit as a subset of SSF which only shifts the intermediate features. LoRA is only applied to the attention layers ($W_q$ and $W_v$) following the original paper and the number of parameters for LoRA is less than SSF. Norm only optimizes the norm layers while SSF is applied after every linear, convolutional and norm layer. Which gives SSF more advantage over other methods to modulate the intermediate features and perform better on downstream tasks.

Reviewer comment (Experiments):
The method only uses zero to replace missing modalities, lacking other compensation methods such as duplicating available modal data or utilizing cross-modal generation techniques to fully showcase the impact of missing modalities.

Author response:
**The method only uses zero to replace missing modalities, lacking other compensation methods such as duplicating available modal data or utilizing cross-modal generation techniques to fully showcase the impact of missing modalities.**\
   We used zeros to replace missing modalities because that seems to be a common practice in recent papers \[1, 2]. We compared the performance of duplicating available modalities and replacing zeros for missing modalities in the table below (same settings as Table 1 in the paper). The results show that duplicating available modalities is inferior to the adapted model (and in some cases worse than replacing zeros for missing modalities). For MFNet and NYUDv2 datasets, we replaced the missing modality with a copy of the available modality. For the MCubeS dataset, we replaced the missing modalities with a copy of RGB modality. In all the combinations, the adapted model works much better than the modality duplication method.

   |         |             |         |                                |                                    |           |
   | :-----: | :---------: | :-----: | :----------------------------: | :--------------------------------: | :-------: |
   | Dataset |    Input    | Missing | Pretrained (Zeros for missing) | Pretrained (Duplicate for missing) |  Adapted  |
   |  MFNet  | RGB-Thermal |    -    |              60.10             |                  -                 |     -     |
   |         |     RGB     | Thermal |              53.71             |              52.33             | **55.22** |
   |         |   Thermal   |   RGB   |              35.48             |                44.43               | **50.89** |
   |  NYUDv2 |  RGB-Depth  |    -    |              56.30             |                                    |           |
   |         |     RGB     |  Depth  |              51.19             |              46.19             | **52.82** |
   |         |    Depth    |   RGB   |              5.26              |                13.94               | **36.72** |
   |  MCubeS |  RGB-A-D-N  |    -    |              51.54             |                  -                 |     -     |
   |         |   RGB-A-D   |    N    |              49.06             |                49.93               | **51.11** |
   |         |    RGB-A    |   D-N   |              48.81             |                49.23               | **50.62** |
   |         |     RGB     |  A-D-N  |              42.32             |                48.96               | **50.43** | 

   Utilizing cross-modal generation is also a very good idea for missing modality compensation. Table 3 in our paper provides a comparison with one such method, TokenFusion, which dynamically detects uninformative tokens and substitutes these tokens with projected and aggregated tokens from available modalities. The CRM method in Table 2 uses self-distillation between the clean and masked modalities to learn complementary and non-local representations for better performance on missing modality scenarios. Our method performs better than both these methods in most of the scenarios.

Reviewer comment (Evaluation):
The method's SSF strategy is an input-independent adaptation approach, it struggles to maintain reliable performance in scenarios with significant distribution differences between the training and testing sets.

Author response:
**The method's SSF strategy is an input-independent adaptation approach; it struggles to maintain reliable performance in scenarios with significant distribution differences between the training and testing sets.**\
   This is a good point and SSF may perform poorly in some cases, but in our experience SSF provides quite robust adaptation. According to \[3], parameters learnt by SSF are input-independent that makes it more suitable for representing the distribution of the different downstream dataset by modulating the intermediate feature. We could learn the modulation parameters by conditioning them on the input features, but that would require some additional modules (e.g. MLP and activation functions) at the expense of additional parameters.

Reviewer comment (Evaluation):
Table 9 shows that there is still a gap compared to other adaptation method (BitFit), which makes the advantages of the proposed SSF strategy not obvious enough.

Author response:
**Table 9 shows that there is still a gap compared to other adaptation methods (BitFit), which makes the advantages of the proposed SSF strategy not obvious enough.**\
The gap between BitFit and SSF is really small (0.1-0.2%). If we analyze all the results in Tables 4, 8, 9, and 10, the advantage of SSF would become obvious. BitFit provides better mIoU in two scenarios in Table 9, whereas SSF provides better mean accuracy and F1 scores. If we compare overall performance on all the three datasets (NYUDv2, MFNet and MCubeS), SSF works better than other methods for most of the scenarios as shown in Tables 4, 8, 9 and 10. For this reason, we preferred SSF as it can be applied to different datasets with diverse modalities.