Paper: Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning (ICLR-2024)

Reviewer comment (Theory):
As the authors already point out in Section 5, the proposed approach requires a predefined order of modalities, which could be a problem when there are > 3 modalities and we do not have prior information about the multimodal task.

Author response:
**Algorithms for defining the order of modalities:** Thank you for pointing out your concern on defining the order of the modalities when there are > 3 modalities and we do not have prior information about the multimodal task. The sequence of modalities indeed influences the performance of our model. When the number of modalities increases and there is a lack of relevant prior knowledge, determining the richness of information in each modality and the sequence among them becomes crucial. To address this problem, we have developed practical algorithms to define the order of modalities: 1. We employ Sample Entropy (SampEn) to assess the information richness within different modalities, which serves as the basis for ranking these modalities. 2. For specific downstream tasks, we implement a greedy algorithm in conjunction with a submodular function to more comprehensively determine the order of the modalities. Detailed explanations and algorithms for these methods are provided in the appendix J (page 26-27), as also mentioned in the 'Response to all reviewers' comments above.

Reviewer comment (Experiments):
Would the proposed hierarchical architecture introduce some inference latency compared with the standard non-hierarchical approaches (Take Figure 3 as an example, say compared to concatenation of $X_0$, $X_1$ and $X_2$ and then passed to an MLP head)?

Author response:
**Inference latency:** Thank you for your valuable suggestion to test the inference latency, which has enabled a more thorough assessment of our model's performance. In response to your concern about the inference latency of our proposed hierarchical architecture as compared to standard non-hierarchical approaches, we conducted a comparative analysis using a simple MLP head. This involved concatenating modalities M0, M1, and M2 and processing them through an MLP head, representative of the predictor component in our ITHP model. The inference latency for this simple configuration was approximately 0.00221 ms per sample. In contrast, our full ITHP model, incorporating a hierarchical structure, recorded an inference latency of 0.0120 ms per sample using the same experiment setting. It's noteworthy that during inference, the detectors in our model are not active, contributing to its overall efficiency. Despite the anticipated increase in latency compared to an MLP head due to its hierarchical nature, our model remains competitive in terms of efficiency. We appreciate your suggestion and have included the above-mentioned discussion in Appendix G (page 24).

Reviewer comment (Experiments):
It seems in the current framework, feature extraction from raw video, audio & text & the proposed ITHP are separate (first feature extraction then use the encoded features in the hierarchical information flow). The temporal nature of these data is not utilized. I wonder if the authors could briefly comment on the potential of extending the ITHP model to handle sequence data (that naturally comes with a temporal order).

Author response:
**The potential of extending the ITHP model to handle sequence data:**

Thank you for your insightful comment and question on the potential extension of the ITHP model to handle sequence data and the non-stationary properties of temporal data. 

Currently, our ITHP framework separates feature extraction from the hierarchical information flow. As per your suggestion, we see the value in integrating sequence handling directly into the ITHP model. Because the ITHP design is compatible with existing neural network architectures, it could be merged with models such as LSTM [1], Transformers [2], etc which excel in capturing temporal dynamics crucial in multimodal learning scenarios. In this setup, we could leverage such a neural network to process the sequence inputs in the prime modality while the ITHP model integrates this temporal information with other modalities hierarchically. This approach would allow us to effectively utilize the temporal nature of the data. 

Simultaneously, we acknowledge the non-stationary nature [3] of temporal data where the statistical properties of the data can change over time. In response to this challenge, we could consider each distinct statistical state in the non-stationary temporal data as a separate modality. By treating these states as separate modalities, we could apply the ITHP model to discover the shared underlying causal relationships between these states, allowing us to effectively uncover dynamic interactions and dependencies between different temporal states. 

This extension of the ITHP model would involve identifying changes in the statistical properties of the temporal data, segmenting the data based on these changes, and then applying the ITHP model to this segmented data. In doing so, we could potentially capture the non-stationary characteristics of the temporal data and improve the model's performance in tasks that involve such data.

Your comment has opened an exciting avenue for future work. We appreciate your thoughtful feedback and will consider it as we continue to refine and extend the ITHP model. We are eager to explore how the ITHP model can be adapted to handle the unique challenges and opportunities presented by non-stationary temporal data.

[1] Cheng, Fei, and Yusuke Miyao. "Classifying temporal relations by bidirectional LSTM over dependency paths." Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2017.

[2] Zhou, Haoyi, et al. "Informer: Beyond efficient transformer for long sequence time-series forecasting." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 12. 2021.

[3] Huang, Biwei, Kun Zhang, Jiji Zhang, Joseph Ramsey, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. "Causal discovery from heterogeneous/nonstationary data." The Journal of Machine Learning Research 21, no. 1 (2020): 3482-3534.

Reviewer comment (Novelty):
The inspiration from neuroscience research seems simple, and unnecessary to the main processing pipeline of the main work, making the main work distracting.

Author response:
**Regarding to the inspiration from neuroscience research:** Our model is inspired by neuroscience research, highlighting how the brain hierarchically processes multimodal information. We have integrated an information bottleneck framework into our design to emulate the hierarchical process, which stands as an innovation in our work and sets our model apart from traditional approaches. Rather than initiating with a fusion structure as is common in conventional models, our model adopts a comprehensive, information theoretic approach.

Reviewer comment (Experiments):
The paper should include some recent SOTA works, at lease works in 2022, on comparing the sentiment analysis.

Author response:
**Comparison on recent related works** Thank you for your suggestions on including more recent state-out-of-art works. Taking into account the suggestions from you and other reviewers, we have included UniMSE [2], MIB [4] and BBFN [5] for CMU-MOSI dataset and DynMM [1] for CMU-MOSEI dataset. The comparative results for the CMU-MOSI dataset have been updated in the revised version in Table 2 (page 8) and we also provide it as follows:

| **Task Metric** | **BA↑** | **F1↑** | **MAE↓** | **Corr↑** |
|-----------------|---------|---------|-----------|-----------|
| UniMSE [5] | 85.9 | 85.8 | 0.691 | 0.809 |
| MIB [7] | 85.3 | 85.3 | 0.711 | 0.798 |
| BBFN [8] | 84.3 | 84.3 | 0.776 | 0.755 |
| **ITHP** | **88.7** | **88.6** | **0.643** | **0.852** |
Table 2. Model Performance on CMU-MOSI dataset.

We also compare the performance of our ITHP model with DynMM on the CMU-MOSEI dataset. DynMM achieved an accuracy of 79.8% and a Mean Absolute Error (MAE) of 0.60. In contrast, our model achieved an accuracy of 87.3% and an MAE of 0.56, surpassing DynMM by over 9.4% in accuracy and exhibiting a 6.7% lower MAE. These results further underscore the superiority of our model in terms of accuracy. We have also discussed all these models in our related work in Appendix B (page 14) as follows:

“MIB focuses on isolating essential data features, filtering out the extraneous for purer multimodal fusion. BBFN reimagines the fusion process by merging and differentiating between modal inputs. The current SOTA model presents novel insights, UniMSE innovatively integrates multimodal sentiment analysis and emotion recognition in conversations into a unified framework, enhancing prediction accuracy by leveraging the synergies and complementary aspects of sentiments and emotions. The SPECTRA [3] is pioneering in spoken dialog understanding by pre-training on speech and text, featuring a unique temporal position prediction task for precise speech-text alignment. The DynMM creates the possibility of reducing computing resource consumption. By introducing dynamic multi-modal fusion, which fuses input data from multiple modalities adaptively, leading to reduced computation, improved representation power, and enhanced robustness. ”

Reviewer comment (Theory):
Why sequential processing is good? Except for inspiration from neural inspirations. This operation introduces the extra problem of the order of modality.

Author response:
1.	Regarding your concern about the sequential pipeline versus fusion of all modalities once together, we wish to highlight that, while other state-of-the-art baseline models adopt strategies to fuse all modalities at once, the experiments have demonstrated the superiority of our hierarchical approach (refer to Table 1,2 and 3 in the main body of the article, Table 4 in Appendix E, Table 5 in Appendix F.1 and Table 6 in Appendix F.3).

2.	Your point about the potential loss of complementary information in the third modality is well-taken. Our method, however, ensures that no modality is neglected. Specifically, the information bottleneck is applied not only between the state $B_0$ and the second modality $X_1$ but also between the state $B_1$ and the third modality $X_2$. The entire process is data-driven, and the final information flow considers all modalities, representing the optimal solution. This guarantees that all modalities are considered, utilizing the information of the third modality (please refer to Appendix C (Pages 14-17) for the detailed design of our model).

3.	Our sequential pipeline design is not arbitrary but is grounded in neuroscience principles. We believe that the hierarchical design allows for a more natural and robust integration of modalities, which is evidenced in our experiment results compared with the baseline fusion models. We also provide the detailed explanation and theoretical basis for designing our hierarchical model based on the information bottleneck in Appendix C (Pages 14-17).

Reviewer comment (Experiments):
Does the best beta and gamma share across different tasks? Or it is needed to balance for different tasks, making this method quite cumbersome.

Author response:
**Regarding the adjustment of model parameters across different tasks:** Thank you for raising the question about whether our model requires excessive adjustments of hyperparameters based on the task. Our answer is that such extensive adjustments are not required. In our experiments, we have tested different values of $\beta$ and $\gamma$. We found that for different experiments, these two parameters do not require much adjustment. Referring to Figure 4 (page 8), for instance, when both $\beta$ and $\gamma$ values are all smaller than 4, the model cannot retain the relevant information from the secondary modalities while losing information from the input modality with a low Lagrange multiplier, leading to worse performance. However, the results remain almost unchanged when beta and gamma are chosen within a wide range when $\beta>=8$. This implies that we have a substantial margin for parameter selection, eliminating the need for excessive adjustments. Moreover, in the experiments with the CMU-MOSI/CMU-MOSEI datasets, we keep the hyper parameters the same with ($\beta=8$ and $\gamma=32$) without tuning these hyper parameters, demonstrating the robustness of our parameter choices.

Reviewer comment (Experiments):
Even though this paper shows interesting modeling design and experimental results, there is a gap between state-of-the-art methods such as UniMSE and SPECTRA, for example, with respect to performances. What kinds of criteria to choose the model configuration and comparative methods in the paper?

Author response:
**Regarding the comparison between our method, ITHP, against SOTA methods such as UniMSE and SPECTRA:**  Thank you for sharing these related works [1,2], we have first added them into the references and discussed them in the related work section in Appendix B (page 14) as follows:
“The current state-of-the-art model presents novel insights, UniMSE [1]  innovatively integrates multimodal sentiment analysis and emotion recognition in conversations into a unified framework, enhancing prediction accuracy by leveraging the synergies and complementary aspects of sentiments and emotions. SPECTRA [2] is pioneering in spoken dialog understanding by pre-training on speech and text, featuring a unique temporal position prediction task for precise speech-text alignment .”

We have also expanded our baselines to include more recent works such as UniMSE [1], MIB [3] and BBFN [4] for a more comprehensive comparison, as detailed in the updated Table 2 (page 8) in the paper. The results show that our model continues to surpass the baselines.

Regarding the comparison with the models you mentioned, particularly UniMSE [1] and SPECTRA [2]: First, regarding performance, we outperform UniMSE on CMU-MOSEI and CMU-MOSI, two common evaluation benchmarks shared by both of the works, by around 1% and 3% points in terms of F1 score. Meanwhile, SPECTRA is surpassed by 1% in MOSI datasets, demonstrating our model’s superiority. Here we would like to point out that our data preprocessing is based on the previous paper MAG [5], which only provides off-the-shelf representations of the images and audio. However, both UniMSE and SPECTRA, adopted a different pipeline (e.g., in UniMSE, they use more powerful tools such as effecientNet to extract features from the video, coupled with additional A/V-sLSTM to further distill the multimodal signals). In this sense, the comparison with Self-MM, MMIM, and MAG are considered as more fair baselines given the same pre-processing pipelines, where ITHP has dominated in performance against all of the above.
Second, our ITHP is an information-theoretic with guaranteed proof of aggregating the most useful information across different modalities. Previous works, such as UniMSE and SPECTRA, either *empirically* instead of *theoretically* adopts contrastive loss to force the latent space to be as distinctive as possible, or *empirically* aligns sequential audio-text features which are not generalizable to images.

[1] UniMSE : G. Hu, et al., UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition, EMNLP 2022

[2] SPECTRA : T. Yu, et al., Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment, ACL 2023

[3] MIB: Mai S, Zeng Y, Hu H. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. IEEE Transactions on Multimedia. 2022.

[4] Han, Wei, et al. "Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis." Proceedings of the 2021 International Conference on Multimodal Interaction. 2021.

[5] Rahman, Wasifur, et al. "Integrating multimodal information in large pretrained transformers." Proceedings of the conference. Association for Computational Linguistics. Meeting. Vol. 2020. NIH Public Access, 2020.

Reviewer comment (Reproducibility):
This paper seems not to present enough information on experimental setting and implementation to achieve reproducibility.

Author response:
**Clarity on reproducibility of experimental setup:** We understand your concern for details of our experimental setup, which is critical for reproducibility. It should be noted that we have indeed provided the source code link for our model and experiments in Appendix A "CODE AVAILABILITY" (page 14). At the same time, we have listed the selection of model parameters in the experimental part in detail in the Appendix E "MODEL ARCHITECTURE AND HYPERPARAMETERS" (page 17-18). We also summarize the description here for convenience:

“Here we provide a detailed description of our model architecture and the hyperparameter selection in different tasks for reproducibility purposes. For the ITHP, to get the latent state's distribution of $B_0$, we directly use two fully connected linear layers to build the encoder, where the first layer extracts the feature vectors of input $X_0$ and the second layer outputs parameters for a latent Gaussian distribution - a mean vector and the logarithm of the variance vector. By sampling from this distribution, we obtain a latent state that represents the input state further used to (1) feed into the multilayer perceptron (MLP) layer to predict $X_1$; (2) feed into the next hierarchical layer for another modality information. Note that this is a serial structure, and the remaining layers share the same design. The predictor is chosen specifically by the downstream tasks. For the task of sarcasm detection, we elect to use an MLP with two layers as the predictor. For the sentiment analysis task, the model ITHP-DeBERTa is trained in an end-to-end manner. The raw features of acoustic and visual modalities are pre-extracted from CMU-MOSI [6]. The textual features are extracted by pre-trained DeBERTa [7, 8]. The textual features serve as input to our ITHP module while the pre-extracted acoustic and visual act as the detectors. For the task of sarcasm detection, unless otherwise specified, we set the  hyperparameters as follows: $\beta=32, \gamma=8, \lambda=1$. We perform a 5-fold cross-validation, and for each experiment, we train the ITHP model for 200 epochs using an Adam optimizer with a learning rate of $10^{-3}$. For the task of sentiment analysis, unless otherwise specified, we set the  hyperparameters as follows: $\beta=8, \gamma=32, \lambda=1$. We run each experiment for 40 epochs using the Adam optimizer with a learning rate of $10^{-5}$. We repeat each experiment 5 times to calculate the mean value of the metrics.”

We are committed to making our code and datasets public to ensure that our work can be replicated and verified by others in the field.

Reviewer comment (Experiments):
It lacks justification for the component choices. Which points should be clarified through experiments? Why BERT and DeBERTa are utilized as language models?

Author response:
**Rationale for component selection:** Thank you for pointing out your concerns on component choice. Your point about the rationale for our component selection, specifically the rationale for using BERT and DeBERTa, is easily accepted. Though our method ITHP is a **model-agnostic** method, we choose BERT (RoBERTa) and DeBERTa as they are the **go-to** method for text encoding and also have some benefits such as lightweight (hundred-million-scale parameter size). They are widely used in most of the text-encoding/natural language understanding related works ranging from question answering, sentiment analysis, commonsense reasoning , and so on. In fact, the adoption of DeBERTa instead of merely RoBERTa is a demonstration of generalizability across different methods for text encoding. We leave the adaptations to other text encoders, such as T5, Electra, and even more recent LLMs such as Llama, GPT3, as a future work to better understand the generalizability of our work. 

[6] Zadeh, Amir, et al. "Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos." arXiv preprint arXiv:1606.06259 (2016).

[7] He, Pengcheng, et al. "Deberta: Decoding-enhanced bert with disentangled attention." arXiv preprint arXiv:2006.03654 (2020).

[8] He, Pengcheng, Jianfeng Gao, and Weizhu Chen. "Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing." arXiv preprint arXiv:2111.09543 (2021).

Reviewer comment (Theory):
ITHP extracts commonly encoded information from different modalities; however, would this procedure be called "fusion" appropriately? Suppose the primary modality is $M_p$ and a secondary modality be $M_1$. ITHP cannot capture information encoded in $M_1$ but not $M_p$. In this sense, ITHP would be an information distillation rather than a fusion method.

Author response:
**The usage of "fusion" in our method:** Thank you for highlighting the usage of the term “fusion” in our study. We are grateful for your detailed attention to this aspect, and we have thoroughly reviewed our manuscript to ensure that this term is used accurately and appropriately. Fusion models primarily aim to integrate information from various modalities into a unified representation [1,2]. In terms of this objective, our work can certainly be viewed as a fusion model constructing a compact representation by utilizing information from various modalities. In terms of technique, we indeed adopt a distinct hierarchical approach compared to traditional fusion models, which stands as a significant innovation in our work. Although the secondary modalities serve as detectors in our model, it is worthy noting that, within the deep learning architecture, our model has certain capability to indirectly learn the complementary information from secondary modalities. As explained in Appendix H (page 24):” by harnessing the capabilities of our deep learning framework, our model has a certain ability to learn the complementary information from secondary modalities. To achieve a harmonious representation, we’ve incorporated specific terms in the loss function. Specifically, taking the 3-modalities learning problem as an example, term $L_{1} =−\beta\cdot I\left (B_{0};X_{1} \right) $ is tasked with supervising the knowledge transfer from $X_{1}$ to $B_{0}$, while term $L_{2} =−\gamma\cdot I\left (B_{1};X_{2} \right) $ is responsible for supervising the knowledge dissemination from $X_{2}$ to $B_{1}$. Through this framework, the ITHP model is primed to discern and incorporate complementary information from non-input modalities.” In summary, we prefer to use the term 'fusion' in the context of our mode, and we will make it clear in our revised version. 

[1] Gao, Jing, et al. "A survey on deep learning for multimodal data fusion." Neural Computation 32.5 (2020): 829-864.

[2] Atrey, Pradeep K., et al. "Multimodal fusion for multimedia analysis: a survey." Multimedia systems 16 (2010): 345-379.

Reviewer comment (Writing):
When talking about "fusion," one expects to integrate information of **different** types.

Author response:
**The usage of "fusion" in our method:** Thank you for highlighting the usage of the term “fusion” in our study. We are grateful for your detailed attention to this aspect, and we have thoroughly reviewed our manuscript to ensure that this term is used accurately and appropriately. Fusion models primarily aim to integrate information from various modalities into a unified representation [1,2]. In terms of this objective, our work can certainly be viewed as a fusion model constructing a compact representation by utilizing information from various modalities. In terms of technique, we indeed adopt a distinct hierarchical approach compared to traditional fusion models, which stands as a significant innovation in our work. Although the secondary modalities serve as detectors in our model, it is worthy noting that, within the deep learning architecture, our model has certain capability to indirectly learn the complementary information from secondary modalities. As explained in Appendix H (page 24):” by harnessing the capabilities of our deep learning framework, our model has a certain ability to learn the complementary information from secondary modalities. To achieve a harmonious representation, we’ve incorporated specific terms in the loss function. Specifically, taking the 3-modalities learning problem as an example, term $L_{1} =−\beta\cdot I\left (B_{0};X_{1} \right) $ is tasked with supervising the knowledge transfer from $X_{1}$ to $B_{0}$, while term $L_{2} =−\gamma\cdot I\left (B_{1};X_{2} \right) $ is responsible for supervising the knowledge dissemination from $X_{2}$ to $B_{1}$. Through this framework, the ITHP model is primed to discern and incorporate complementary information from non-input modalities.” In summary, we prefer to use the term 'fusion' in the context of our mode, and we will make it clear in our revised version. 

[1] Gao, Jing, et al. "A survey on deep learning for multimodal data fusion." Neural Computation 32.5 (2020): 829-864.

[2] Atrey, Pradeep K., et al. "Multimodal fusion for multimedia analysis: a survey." Multimedia systems 16 (2010): 345-379.

Reviewer comment (Experiments):
In addition, when both modalities contain similar information, if the one in $M_1$ is noisy, would ITHP be affected? Can ITHP correctly identify the noise in $M_1$ and mainly use the information from $M_p$ instead?

Author response:
**Model performance in the presence of noise:** Thanks for the insightful suggestions on adding noise to the modality $M_{p}$ and modality $M_{1}$. We believe these experiments will bring valuable insights to further elucidate our model. Theoretically, our model filters redundant information within its information-bottleneck hierarchical structure, and we anticipate it will have advantages over traditional fusion models in terms of reducing the impact of noise. According to your suggestions, we conducted the noise addition experiments as follows:

In the sarcasm detection dataset, we introduced noise to the normalized data of both the primary and secondary modalities separately to test the model's resistance to noise interference.  We apply a series of Gaussian noises with a mean of 0 and a varying standard deviation ranging from 0.05 to 0.3, increasing in increments of 0.05. For each noise level, we repeated the experiment 50 times and calculated the average result. We have included these experiments in the revised version (Please refer to Appendix F.6, page 23-24), and the results are presented below:


| $\sigma$ | -     | 0.05  | 0.1   | 0.15  | 0.2   | 0.25  | 0.3   |
|----------------------|-------|-------|-------|-------|-------|-------|-------|
| **MSDM Precision** | 0.719 | 0.711 (-1.11%) | 0.709 (-1.39%) | 0.707 (-1.67%) | 0.704 (-2.09%) | 0.700 (-2.64%) | 0.688 (-4.31%) |
| **MSDM Recall** | 0.714 | 0.705 (-1.26%) | 0.702 (-1.68%) | 0.700 (-1.96%) | 0.695 (-2.66%) | 0.691 (-3.22%) | 0.679 (-4.90%) |
| **ITHP Precision** | 0.753 | 0.751 (-0.27%) | 0.748 (-0.66%) | 0.747 (-0.80%) | 0.746 (-0.93%) | 0.743 (-1.33%) | 0.740 (-1.73%) |
| **ITHP Recall** | 0.752 | 0.750 (-0.27%) | 0.745 (-0.93%) | 0.743 (-1.20%) | 0.742 (-1.33%) | 0.740 (-1.60%) | 0.736 (-2.13%) |

Table 1. Evaluating Gaussian Noise Impact on Sarcasm Detection in Text Modality. 

Table 1 presents the weighted precision and recall for sarcasm detection on the MUStARD dataset, with an analysis of Gaussian noise addition to the text ($M_{1}$) modality. Results are averaged over five cross-validation folds and 50 trials for each level of noise intensity. Each column indicates the standard deviation (represented by $\sigma$) of the introduced Gaussian noise with a mean of 0. Changes in performance metrics relative to the noise-free baseline are expressed as percentages



| $\sigma$ | -      | 0.05     | 0.1      | 0.15     | 0.2      | 0.25      | 0.3       |
|----------------|--------------|------------|-----------|------------|------------|------------|-------------|
| **MSDM Precision**| 0.719         | 0.709 (-1.39%)    | 0.705 (-1.95%)    | 0.702 (-2.36%)    | 0.698 (-2.92%)    | 0.692 (-3.76%)    | 0.683 (-5.01%)    |
| **MSDM Recall**   | 0.714         | 0.703 (-1.54%)    | 0.700 (-1.96%)    | 0.694 (-2.80%)    | 0.689 (-3.50%)    | 0.678 (-5.04%)    | 0.668 (-6.44%)    |
| **ITHP Precision**    | 0.753         | 0.748 (-0.66%)    | 0.745 (-1.06%)    | 0.743 (-1.33%)    | 0.742 (-1.46%)    | 0.739 (-1.86%)    | 0.735 (-2.39%)    |
| **ITHP Recall**       | 0.752         | 0.745 (-0.93%)    | 0.741 (-1.46%)    | 0.740 (-1.60%)    | 0.738 (-1.86%)    | 0.733 (-2.53%)    | 0.730 (-2.93%)    |

Table 2. Evaluating Gaussian Noise Impact on Sarcasm Detection in Video Modality. 

This table presents the weighted precision and recall for sarcasm detection on the MUStARD dataset, with an analysis of Gaussian noise addition to the video ($M_{p}$) modality. Results are averaged over five cross-validation folds and 50 trials for each level of noise intensity. Each column indicates the standard deviation (represented by $\sigma$) of the introduced Gaussian noise with a mean of 0. Changes in performance metrics relative to the noise-free baseline are expressed as percentages

The results shown in Table 1 and Table 2 demonstrate that, compared to the baseline model, our model exhibits greater robustness to noise interference. The impact of adding noise to the primary modality is more significant than the effect of noise on the secondary modality.

Reviewer comment (Experiments):
How do you choose the primary modality? And how would different choices of primary modality affect the performance of ITHP?

Author response:
**Experiments with different primary modalities:** 
We appreciate your question regarding the selection of the primary modality and its impact on the ITHP model's performance. In fact, we have already conducted the experiments for varying primary modalities as highlighted in  Section 5 (page 9) and the results are shown in Appendix F.5 (Page 23). We also present the results below. This table illustrates the effects of changing the primary modality (modal order) on weighted precision, recall, and F-score. 

| Metric              | V→T→A | V→A→T | T→V→A | T→A→V | A→V→T | A→T→V |
|:---------------------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|
| Weighted Precision  | **75.3**  | 73.7  | 73.0  | 72.0  | 71.4  | 70.8  |
| Weighted Recall     | **75.2**  | 73.5  | 72.8  | 71.8  | 71.2  | 70.6  |
| Weighted F-Score    | **75.2**  | 73.4  | 72.9  | 71.7  | 71.5  | 70.6  |

Table 3. Varying Orders of Modalities.

The table showcases the impact of changing the order of modalities on the weighted Precision, Recall, and F-Score across both sarcastic and non-sarcastic classes, averaged across five folds. Underlined values highlight the best results for each metric. The modalities are denoted as: $T$ for text, $A$ for audio, and $V$ for video.

The results show that the order of modalities, particularly the designation of the primary one, considerably influences the performance of the model. In many cases, we can rely on prior knowledge to guide the selection of the primary modality and its sequence. However, when such knowledge is lacking, effective methods are essential for determining these aspects. In response to the feedback from reviewers, we have developed practical algorithms to define the primary modality and the order of modalities: (i) We employ Adapting Sample Entropy (SampEn) to assess the information richness within different modalities, which serves as the basis for ranking these modalities. (ii) For specific downstream tasks, we implement a greedy algorithm in conjunction with a submodular function to more comprehensively determine the order of the modalities. Detailed explanations and algorithms for these methods are provided in the appendix J (page 26-27), as also mentioned in the 'Response to all reviewers' comments above.

Based on the methods for modalities ranking we proposed, we validated the modal selection order used in our experiments. For the first method, the computed SampEn for the V/T/A modalities are 2.388/2.063/0.071, respectively. This indicates that the ranking of information richness for these modalities should be V→T→A. Additionally, by applying the algorithm (ii), we obtained the same modality order of V→T→A. Hence, we chose V as our primary modality. This aligns with the results obtained in our experiments, confirming the viability of our proposed detection criteria.

Reviewer comment (Experiments):
How would ITHP perform in the presence of noise in some of the modalities (the primary or the secondary ones)?

Author response:
**Model performance in the presence of noise:** Thanks for the insightful suggestions on adding noise to the modality $M_{p}$ and modality $M_{1}$. We believe these experiments will bring valuable insights to further elucidate our model. Theoretically, our model filters redundant information within its information-bottleneck hierarchical structure, and we anticipate it will have advantages over traditional fusion models in terms of reducing the impact of noise. According to your suggestions, we conducted the noise addition experiments as follows:

In the sarcasm detection dataset, we introduced noise to the normalized data of both the primary and secondary modalities separately to test the model's resistance to noise interference.  We apply a series of Gaussian noises with a mean of 0 and a varying standard deviation ranging from 0.05 to 0.3, increasing in increments of 0.05. For each noise level, we repeated the experiment 50 times and calculated the average result. We have included these experiments in the revised version (Please refer to Appendix F.6, page 23-24), and the results are presented below:


| $\sigma$ | -     | 0.05  | 0.1   | 0.15  | 0.2   | 0.25  | 0.3   |
|----------------------|-------|-------|-------|-------|-------|-------|-------|
| **MSDM Precision** | 0.719 | 0.711 (-1.11%) | 0.709 (-1.39%) | 0.707 (-1.67%) | 0.704 (-2.09%) | 0.700 (-2.64%) | 0.688 (-4.31%) |
| **MSDM Recall** | 0.714 | 0.705 (-1.26%) | 0.702 (-1.68%) | 0.700 (-1.96%) | 0.695 (-2.66%) | 0.691 (-3.22%) | 0.679 (-4.90%) |
| **ITHP Precision** | 0.753 | 0.751 (-0.27%) | 0.748 (-0.66%) | 0.747 (-0.80%) | 0.746 (-0.93%) | 0.743 (-1.33%) | 0.740 (-1.73%) |
| **ITHP Recall** | 0.752 | 0.750 (-0.27%) | 0.745 (-0.93%) | 0.743 (-1.20%) | 0.742 (-1.33%) | 0.740 (-1.60%) | 0.736 (-2.13%) |

Table 1. Evaluating Gaussian Noise Impact on Sarcasm Detection in Text Modality. 

Table 1 presents the weighted precision and recall for sarcasm detection on the MUStARD dataset, with an analysis of Gaussian noise addition to the text ($M_{1}$) modality. Results are averaged over five cross-validation folds and 50 trials for each level of noise intensity. Each column indicates the standard deviation (represented by $\sigma$) of the introduced Gaussian noise with a mean of 0. Changes in performance metrics relative to the noise-free baseline are expressed as percentages



| $\sigma$ | -      | 0.05     | 0.1      | 0.15     | 0.2      | 0.25      | 0.3       |
|----------------|--------------|------------|-----------|------------|------------|------------|-------------|
| **MSDM Precision**| 0.719         | 0.709 (-1.39%)    | 0.705 (-1.95%)    | 0.702 (-2.36%)    | 0.698 (-2.92%)    | 0.692 (-3.76%)    | 0.683 (-5.01%)    |
| **MSDM Recall**   | 0.714         | 0.703 (-1.54%)    | 0.700 (-1.96%)    | 0.694 (-2.80%)    | 0.689 (-3.50%)    | 0.678 (-5.04%)    | 0.668 (-6.44%)    |
| **ITHP Precision**    | 0.753         | 0.748 (-0.66%)    | 0.745 (-1.06%)    | 0.743 (-1.33%)    | 0.742 (-1.46%)    | 0.739 (-1.86%)    | 0.735 (-2.39%)    |
| **ITHP Recall**       | 0.752         | 0.745 (-0.93%)    | 0.741 (-1.46%)    | 0.740 (-1.60%)    | 0.738 (-1.86%)    | 0.733 (-2.53%)    | 0.730 (-2.93%)    |

Table 2. Evaluating Gaussian Noise Impact on Sarcasm Detection in Video Modality. 

This table presents the weighted precision and recall for sarcasm detection on the MUStARD dataset, with an analysis of Gaussian noise addition to the video ($M_{p}$) modality. Results are averaged over five cross-validation folds and 50 trials for each level of noise intensity. Each column indicates the standard deviation (represented by $\sigma$) of the introduced Gaussian noise with a mean of 0. Changes in performance metrics relative to the noise-free baseline are expressed as percentages

The results shown in Table 1 and Table 2 demonstrate that, compared to the baseline model, our model exhibits greater robustness to noise interference. The impact of adding noise to the primary modality is more significant than the effect of noise on the secondary modality.

Reviewer comment (Theory):
How can ITHP accommodate tasks that require combining complementary information from different modalities?

Author response:
**The usage of "fusion" in our method:** Thank you for highlighting the usage of the term “fusion” in our study. We are grateful for your detailed attention to this aspect, and we have thoroughly reviewed our manuscript to ensure that this term is used accurately and appropriately. Fusion models primarily aim to integrate information from various modalities into a unified representation [1,2]. In terms of this objective, our work can certainly be viewed as a fusion model constructing a compact representation by utilizing information from various modalities. In terms of technique, we indeed adopt a distinct hierarchical approach compared to traditional fusion models, which stands as a significant innovation in our work. Although the secondary modalities serve as detectors in our model, it is worthy noting that, within the deep learning architecture, our model has certain capability to indirectly learn the complementary information from secondary modalities. As explained in Appendix H (page 24):” by harnessing the capabilities of our deep learning framework, our model has a certain ability to learn the complementary information from secondary modalities. To achieve a harmonious representation, we’ve incorporated specific terms in the loss function. Specifically, taking the 3-modalities learning problem as an example, term $L_{1} =−\beta\cdot I\left (B_{0};X_{1} \right) $ is tasked with supervising the knowledge transfer from $X_{1}$ to $B_{0}$, while term $L_{2} =−\gamma\cdot I\left (B_{1};X_{2} \right) $ is responsible for supervising the knowledge dissemination from $X_{2}$ to $B_{1}$. Through this framework, the ITHP model is primed to discern and incorporate complementary information from non-input modalities.” In summary, we prefer to use the term 'fusion' in the context of our mode, and we will make it clear in our revised version. 

[1] Gao, Jing, et al. "A survey on deep learning for multimodal data fusion." Neural Computation 32.5 (2020): 829-864.

[2] Atrey, Pradeep K., et al. "Multimodal fusion for multimedia analysis: a survey." Multimedia systems 16 (2010): 345-379.