Paper: Deep Equilibrium Multimodal Fusion (ICLR-2024)

Reviewer comment (Evaluation):
Computational cost: Though recursively fusing multimodal information is quite novel and have a chance to get better performance, I wonder if the proposed method is more time-consuming that its counterparts? It seems such comparisons is lack in the main paper and supplementary. Given this point, I encourage the authors to share more explanations about this.

Author response:
We have tracked the runtime of DEQ fusion on VQA-v2, the results are 0.338 and 0.405 seconds per batch without and with DEQ fusion respectively. Since the majority of the runtime is due to the unimodal feature extractors, we believe that we can tolerate this computational overhead to achieve higher performance

Reviewer comment (Reproducibility):
Scalability: As the authors claim that DEQ fusion is readily pluggable to existing multimodal frameworks, I think it is deserving to further clarify how to combining DEQ fusion into existing multimodal fusion methods. For example, some pseudo code code will be very appreciated and make the proposed method easier to follow.

Author response:
Thanks for your advice. Please find the pseudo codes below written in PyTorch. Inserting our DEQ fusion requires only modifying three line of codes within the model. Also note that we will release our source codes.  

```diff
    class Model(nn.Module):

        def __init__(self, ...):
            ...
+           self.fusion = DEQFusion(...)

        def forward(self, x):
            ...
-           fused_feature = torch.cat([feat for feat in unimodal_feats]) # unimodal_feats contains all features extracted from unimodal encoders
+           fused_feature = self.fusion(unimdaol_feats)
            ...
```

Reviewer comment (Novelty):
Motivation: The authors claim that DEQ fusion is an unified framework that looks into three aspects simultaneously including stabilizing and aligning multimodal signals, integrating interactions across modalities from multi-level, dynamically perceiving information. In my view, many attention-based multmodal fusion methods may also achieve the aforementioned points. In section 3.3, the authors say that DEQ fusion is not depend on any unimodal feature extraction or preprocessing methods. However, there exists some attention-based fusion methods which may also independent on unimodal feature extraction methods. For example, a simple self-attention module or MMTM[1]. Could the authors give some further clarifications?

Author response:
Thanks for pointing us to MMTM. MMTM leverages the squeeze and excitation (SE) technique to recalibrate the channel-wise features from multiple CNN streams for multimodal feature alignment. This work provides additional evidence for one of the mentioned key elements: stabilizing and aligning multimodal features. We have included MMTM in our revised paper to provide further evidence to strengthen this element.

We are also aware that MMTM is also capable of capturing high-level information. Similar to other works that perform higher-level information integration, this high-level information is captured by stacking multiple fusion units. As a result, MMTM requires manual design of different architectures for different applications. Quoting from MMTM: 
>>"Although the module design is generic and could potentially be added at any level in the network hierarchy, the optimal locations and number of modules are different for each application". 

As such, how many fusion units are needed to obtain the optimal fused feature still remains a question for these types of works, whereas our approach is fully self-adaptive and requires minimal manual design.

Reviewer comment (Writing):
The phrase "every level" in the introduction implies a comprehensive integration of cross-modality interactions throughout the multi-modal fusion process. However, given the paper’s focus on fusion of features, which is traditionally associated with late fusion, there seems to be a discrepancy. The paper apparently does not delve into early or middle fusion strategies. To reconcile this, one could interpret “every level” as referring to different stages or aspects within the late fusion process itself, although this may require clarification from the authors for a precise understanding.

Author response:
As mentioned in Section 3.3, our DEQ fusion is indeed a type of late fusion. The term "every level" is thus within the late fusion process itself, where the level indicates the level of information integrated into the fused feature. When we consider the high-level information, it is mainly the product of deeper networks.

Reviewer comment (Evaluation):
The ablation studies conducted on the BRCA and CMU-MOSI datasets highlight the significance of the DEQ, a component not originally introduced by this work, overshadowing the impact of f_{\theta} and f_{fuse}. This raises concerns regarding the efficacy of the designed architecture in fully capitalizing on the potential for interaction among modalities. It suggests a need for further investigation and possibly a reevaluation of the architecture to ensure optimal performance.

Author response:
Our focus is to develop a robust and generalizable multimodal fusion framework. As shown in Tables 5 and 7, naively applying DEQ to unimodal feature extractors or the fused feature leads to suboptimal results, whereas considering intra- and inter-modality features jointly (our approach) gives the best results. We would like to emphasize that our designs of $f_\theta$ and $f_{fuse}$ are not unique. One may consider employing modality-specific alternatives to further boost performance, however, this is not the focus of our work.

Reviewer comment (Writing):
The introduction categorizes three ways ‘stabilizing and aligning..., integrating..., dynamically perceiving...’ but provides limited insights, necessitating to explain the reason behind. It should be clarified that how each way contributes to better multi-modal learning. For instance, the rationale behind the need for a multi-modal model to eliminate redundancy is not clear. A thorough analysis is essential to elucidate why previous research has concentrated on these specific approaches, helping to strengthen the foundational knowledge and context for the study.

Author response:
We have included a further explanation in our revised paper, please find our revision for the related work section (Appendix B). Please note that these findings are suggested and backed up by several previous works.

Reviewer comment (Novelty):
It is recommended that the authors provide a more comprehensive explanation regarding the advantages of this design in stabilizing intra-modal representations. A thorough exploration of the key difference compared to previous research, particularly in terms of enhancing stability, would enhance the reader's understanding.

Author response:
The principle of DEQ fusion is that the equilibrium state of the fused feature satisfies $z_{fuse}^{(i+1)}=f(z_{fuse}^{(i)})$ (slightly simplifies the notation here), which achieves stable multimodal representation while considering multi-level intra- and inter-modality feature fusion. Because it is important to ensure the *dynamic* fusion process, i.e., any modality needs to account for information from all other modalities (as suggested by [1,2,3]), we thus do not particularly emphasize unimodal feature stability.

Reviewer comment (Novelty):
While the proposed architecture aims to achieve stable intra-modal representations, its application is limited for it only targets to the fusion of features. This raises the question of whether the method could also encompass the learning of stable uni-modal encoders, fostering the acquisition of even more robust features. Exploring this avenue could potentially enhance the method’s applicability and effectiveness.

Author response:
The results for this are in Tables 5 and 7, where we evaluate DEQ applied to the unimodal encoders ($f_\theta$ with DEQ). On all three benchmarks, applying DEQ to unimodal encoders contributes to a 1-2% gain over most evaluation metrics. By jointly considering unimodal and multi-modal features, we are able to achieve optimal results.

Reviewer comment (Writing):
It is advisable to provide detailed explanations for the dimensions of each vector and matrix utilized in the study, like $z$ in Related work.

Author response:
The features $z$ are of the same dimension as the unimodal features $x$, such that we can compute the summation in equation 8. The dimensions of the features can be freely altered depending on the model output dimensions and different fusion module designs.

Reviewer comment (Experiments):
Given that Multi-bench encompasses a diverse array of fusion strategies, it is crucial for the authors to specify which particular method was employed on MM-IMDB dataset.

Author response:
Multi-bench integrates many multimodal fusion methods and benchmarks, which is readily deployable for experiments. Our experiments including compared methods were reproduced with Multi-bench on MM-IMDB.

Reviewer comment (Writing):
The vectors $\alpha_i$ ought to be represented in bold to maintain consistency with standard mathematical notation.

Author response:
This is updated in the revised paper.

Reviewer comment (Experiments):
The method in this paper is compared with the weight-tied method, which shows that the method in this paper can converge. This is obvious because the method optimizes fθ by the formula z* = fθ(z*,x), and does not impose such a constraint on the weight-tied method with a finite number of layers, and the weight-tied method certainly cannot converge.

Author response:
There may be some confusion here. As mentioned in Section 2.1 of our paper, the idea of finding the equilibrium is motivated by two interesting observations, which were both claimed in [1]. They found that the hidden layers of some very deep networks tend to converge to some fixed points, and employing the same weight in each layer (so-called weight-tied) also achieves similar results. From this perspective, by applying *any weight* a large number of (or infinitely many) times, the output would eventually satisfy $z_{i+1}=f_{\theta}(z_i)$. The advantages of applying DEQ are to accelerate the "infinite" forward process and memory-efficient analytical one-step backward pass.

With this in mind, please note that the memory efficiency and the acceleration of the forward process come from the fixed-point solvers and analytical backpropagation, hence are not our contributions. Instead, we focus on performing robust and generalizable multimodal fusion based on DEQ. As we show in Tables 5 and 7, naively applying DEQ to unimodal feature extractors or the fused feature leads to suboptimal results, whereas considering intra- and inter-modality features jointly (our approach) gives the best results.

Reviewer comment (Experiments):
In the original DEQ paper, DEQ is proposed for memory efficiency, and the effect is similar to that of weight-tied, and it would be better if the article gave a comparative experiment with the weight-tied method.

Author response:
There may be some confusion here. As mentioned in Section 2.1 of our paper, the idea of finding the equilibrium is motivated by two interesting observations, which were both claimed in [1]. They found that the hidden layers of some very deep networks tend to converge to some fixed points, and employing the same weight in each layer (so-called weight-tied) also achieves similar results. From this perspective, by applying *any weight* a large number of (or infinitely many) times, the output would eventually satisfy $z_{i+1}=f_{\theta}(z_i)$. The advantages of applying DEQ are to accelerate the "infinite" forward process and memory-efficient analytical one-step backward pass.

With this in mind, please note that the memory efficiency and the acceleration of the forward process come from the fixed-point solvers and analytical backpropagation, hence are not our contributions. Instead, we focus on performing robust and generalizable multimodal fusion based on DEQ. As we show in Tables 5 and 7, naively applying DEQ to unimodal feature extractors or the fused feature leads to suboptimal results, whereas considering intra- and inter-modality features jointly (our approach) gives the best results.

Reviewer comment (Writing):
Some of the expressions in the paper are unscientific and abstract, such as the sentence on page 3: ’Our fusion design is flexible from the standpoint that fθi(·) can be altered arbitrarily to fit multiple modalities.’ It could be better expressed as ‘Our fusion design is flexible from the standpoint that fθi(·) can be altered arbitrarily to fit multiple level features’.

Author response:
Our original sentence is trying to express that the design of $f_{\theta_i}$ is not unique and our design might not be optimal. One may consider altering the architecture of this module to be modality-specific to possibly further boost the performance, however, this is not our focus. Our goal instead is to propose a general framework that is applicable to most modalities and multimodal tasks. We will double check our writing to avoid such ambiguities.

Reviewer comment (Presentation):
The drawing is not intuitive.

Author response:
We would reconsider our figures along with our captions to improve the clarity.

Reviewer comment (Theory):
When the equilibrium state is reached, why an informative unified representation in a stable feature space for multimodal learning be obtained? What is the relationship between these two? The paper does not give proof.

Author response:
As mentioned both in Section 1 and Appendix B in our paper, various works have suggested the importance of capturing *stable* and *higher-level* representations to enforce better cross-modality for better multimodal fusion [2,3]. The principle of DEQ fusion is that the equilibrium state of the fused feature satisfies $z_{fuse}^{(i+1)}=f(z_{fuse}^{(i)})$ (slightly simplifies the notation here). By recursively applying the fusion layer, we gradually obtain higher and higher levels of feature information. The process stabilizes at the equilibrium state, as such, the unified representation at this stage captures *stable* and *multi-level* representations.

The advantages of finding the equilibrium state to extract stable and multi-level unified representation are mainly evaluated empirically in our paper, rather than theoretically. Previous methods mostly extract finite levels of features, where the multi-level feature extractions are generally achieved by stacking multiple fusion blocks. Our experiments show that DEQ fusion outperforms these finite-level methods by a considerable margin, thus demonstrating the informativeness of the extracted feature from our DEQ fusion.

Reviewer comment (Experiments):
There may be challenges in generalising the fixed-point approach to different types of multimodal data or applications.

Author response:
We would again like to highlight that our DEQ fusion was evaluated on four widely used multimodal benchmarks, involving several different modalities, thus demonstrating strong generalizability. We will consider adding more benchmarks to further strengthen this point.

Reviewer comment (Novelty):
Some related work is missing, such as "Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction" (TPAMI, 2022) and "Equivariant Multi-Modality Image Fusion" (Zhao et al, 2023).

Author response:
Thanks for your suggestions. We have referred to the mentioned works in our revised paper, please find our revision in the related work (Appendix B).