Paper: Sparse MoE as a New Treatment: Addressing Forgetting, Fitting, Learning Issues in Multi-Modal Multi-Task Learning (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the work is limited. To solve the modality forgetting problem, the authors deploy load and importance balancing loss. To solve the other two problems, the authors use standard hyperparameter tuning methods. It is unclear which part is truly originated from the authors.

Author response:
## **[Cons 1. The Novelty of The Work is Limited?]**
We respectfully disagree. We point out its novelty richness from three distinct aspects: 
1. [**SMoA Designs**] Our approach addresses the modality forgetting problem through the novel SM$^4$ structure, comprising SMoE and SMoA modules governed by modality-specific routing policies. **Note that the load and importance balancing loss is not designed for and can not solve the modality forgetting problem.** 
2. [**Automatic Expert Allocation**] We emphasize that AEA is NOT a standard hyperparameter tuning. While hyperparameters rely on human intervention, our approach automatically decides the expert number based on the model training dynamics. Therefore, in our case, the expert number is not a hyperparameter. 
3. [**Automatic Learning Pace for Different Modality**] The ALP automates the learning pace of each modality in a dynamic fashion, which is also not the standard hyperparameter tuning technique.

Reviewer comment (Evaluation):
I am confused about the results in 5. What does 32N mean? Intuitively, increasing N should lead to good performance, and 32N is indeed the largest in the table, so it is not surprising that it has the best result. What is the message to convey here?

Author response:
## **[Cons 3. Confused About The Results in 5. What Does 32N Mean?]**
For the clarification of “32N”, we've revised the notation to N=4, N=8, N=16, and N=32. 
The optimal number of experts in Mixture-of-Experts (MoEs) remains a subject of debate, with conflicting views on the ideal quantity. While several studies advocate that increased experts enhance performance, an excessive number may introduce redundancy or noise. Table A3 of [1] demonstrates that increasing the expert number brings extra performance will gradually diminish as the expert number goes up, and Table 11 in [2] shows the large expert number may decrease performance.
Therefore, we investigate this further for multi-modal multi-task learning, and our findings suggest that larger expert pools consistently yield performance improvements.

[1] Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners

[2] TUTEL: Adaptive Mixture-of-Experts at Scaler

Reviewer comment (Writing):
**ALP**: It is unclear what "learning pace" means in this context, i.e. is it the learning rate or some components that directly influence the training trajectory?

Author response:
## **[Cons 3. What does "learning pace" Mean in This Context in ALP? ]**
The learning pace in ALP refers to the learning rate (or the optimization step size) of different modalities.

Reviewer comment (Presentation):
Table 1 and 2 are quite unclear, what is the "setting" here referred to, is it the dataset size, or the model size? For example, in Table 2, SM\(^4\) Medium - AV-MNIST has 1.23M params while the same method in the large setting has 0.76M params. The results of HighMMT seem to be quite different from the original, which requires further investigations.

Author response:
## **[Cons 3. What is The "setting" in Table 1, 2 Refers to?]**
The “setting” here refers to the number of tasks that follow the HighMMT.
## **[Cons 4. Model Parameter Number of Medium Setting is More Than The Large Setting.]**
Although the large setting contains more tasks, more task number does not mean more parameters. As shown in Table 14 and Table 15 of our paper, the total number of experts in the medium setting is $32$, and we use $16$ experts in the large setting. Therefore, the number of model parameters in the medium setting is larger than in the large setting.
## **[Cons 5. The Results of HighMMT Seem to be Quite Different From The Origine.]**
Our implementation of HighMMT adheres to their official repo [HighMMT] (https://github.com/pliang279/HighMMT) and the default settings outlined in the HighMMT paper. For instance, our utilization of learning rates is as follows: 0.0005 for the small setting, 0.001 for the medium setting, and 0.0008 for the large setting.
Furthermore, our supplementary materials contain the HighMMT official code and reproduction scripts, ensuring the exact replication of all hyper-parameters specified in the HighMMT paper.
If Reviewer cbhp could kindly point out another better re-implementation of HighMMT, we would like to follow it and update the configurations.

Reviewer comment (Presentation):
Other suggestions: Figure 2 is not nice, please consider using subfigure, e.g. 2a, 2b, etc. instead of the current presentation.

Author response:
## **[Cons 6. Expression Suggestion.]**
We've made adjustments according to your invaluable suggestions, including improvements to Figure 2 for clarity.

Reviewer comment (Reproducibility):
Clarifications regarding the AEA, ALP modules, and the settings in Table 1 & 2.

Author response:
## **[Cons 1. AEA Introduces an Additional Hyper-Parameter, Additional Pre-training Phase?]**
The AEA is not working on a pre-train phase; it is executed during our multi-modal multi-task learning, which takes into account multi-task learning, and the AEA stopped tuning expert numbers by itself, not a hyper-parameter. 
The “n” in Algorithm 2 is actually the number of iterations within a single multi-modal multi-task training epoch. Therefore, we do not include additional hyper-parameters. 
As we mentioned in Section 3.3, the load and importance balancing loss we used in routing networks make sure most experts will be utilized.
Moreover, as shown in **T.1** our method doesn't increase training time significantly. 
The above concerns have been included in our revision.

**T.1** SM$^4$ and HighMMT multi-modal multi-task training time.
| Model   | Small Setting (h) | Medium Setting (h) | Large Setting (h) |
| ------- | ----------------- | ------------------ | ----------------- |
| SM$^4$  | 33                | 18                 | 8                 |
| HighMMT | 33                | 17.5               | 8                 |

## **[Cons 3. What does "learning pace" Mean in This Context in ALP? ]**
The learning pace in ALP refers to the learning rate (or the optimization step size) of different modalities.
## **[Cons 3. What is The "setting" in Table 1, 2 Refers to?]**
The “setting” here refers to the number of tasks that follow the HighMMT.
## **[Cons 4. Model Parameter Number of Medium Setting is More Than The Large Setting.]**
Although the large setting contains more tasks, more task number does not mean more parameters. As shown in Table 14 and Table 15 of our paper, the total number of experts in the medium setting is $32$, and we use $16$ experts in the large setting. Therefore, the number of model parameters in the medium setting is larger than in the large setting.
## **[Cons 5. The Results of HighMMT Seem to be Quite Different From The Origine.]**
Our implementation of HighMMT adheres to their official repo [HighMMT] (https://github.com/pliang279/HighMMT) and the default settings outlined in the HighMMT paper. For instance, our utilization of learning rates is as follows: 0.0005 for the small setting, 0.001 for the medium setting, and 0.0008 for the large setting.
Furthermore, our supplementary materials contain the HighMMT official code and reproduction scripts, ensuring the exact replication of all hyper-parameters specified in the HighMMT paper.
If Reviewer cbhp could kindly point out another better re-implementation of HighMMT, we would like to follow it and update the configurations.

Reviewer comment (Evaluation):
The authors employ MultiBench for evaluation, while the metrics of robustness and training cost are ignored. Without checking robustness, it is unclear if the trained model is robust to missing or noisy modalities, which shall be an important criterion in multimodal learning.

Author response:
## **[Cons 1. Without Checking Robustness.]**

We conduct extra experiments to examine the robustness of SM$^4$ to missing modalities. Due to the limited time in the rebuttal period, we will add the robustness evaluation of noisy modalities in our final version.
As shown in Table **T1**, we assess SM$^4$ and HighMMT with the model UR-FUNNY under three missing scenarios of missing text, video, and audio, respectively. Our results imply that SM$^4$ has a relatively better robustness towards missing modalities.

**T.1** The robustness comparison between SM$^4$ and highMMT. We show the $\delta$ value, which is defined as the value of performance drop when missing one modality. A smaller $\delta$ value indicates better robustness against the modality missing.

| UR-FUNNY| Missing text | Missing video | Missing audio |
| -------------------| ------------ | ------------- | ------------- |
| SM$^4$ | $1.16$ | $4.77$ | $0.92$ |
|HighMMT| 8.22 | 10.30 | 6.62 |

| MOSEI| Missing image | Missing audio| Missing text |
| -------------------| ------------ | ------------- | ------------- |
| SM$^4$ | $0.39$| $0.82$ | $0.92$|
|HighMMT| 10.45 | 17.38 | 12.23 |

| MIMIC | Missing table| Missing timeseries| 
| -------------------| ------------ | ------------- | 
| SM$^4$ | $8.71$| 17.99 | 
|HighMMT| 9.72 | $11.67$ | 

| AV-MNIST| Missing image| Missing audio| 
| -------------------| ------------ | ------------- | 
| SM$^4$ | 60.6| $13.60$ | 
|HighMMT| $55.85$ | 36.27 |

Reviewer comment (Evaluation):
The authors employ MultiBench for evaluation, while the metrics of robustness and training cost are ignored. The trade-off between training cost and model performance of SM^4 is unclear. In particular, deciding number of experts for each modality seems to be time-consuming. Checking the trade-off and comparing SM^4 with simple methods such as early/late can help measure the practical value of this work more precisely.

Author response:
## **[Cons 2. The Trade-off Between Training Cost and Model Performance of SM^4 is Unclear.]**

Thanks for the great point.
- The total number of training epochs for SM$^4$ and other baselines are the same. The procedure of deciding the number of experts for each modality does NOT require extra training epochs. Specifically, both HighMMT and SM$^4$ use $100, 100, 100$ training epochs for the small, medium, and large settings, respectively. 
- To further convince review mdyv, we measure the training time of SM$^4$ and HighMMT in the same device (i.e., single NVIDIA A30 GPU). Our results demonstrate that SM$^4$ uses **similar training costs** to reach consistently enhanced performance, as evidenced in Table **T.2**. It indicates an improved trade-off between training cost and model performance.

**T.2** SM$^4$ and HighMMT multi-modal multi-task training time.
| Model   | Small Setting (h) | Medium Setting (h) | Large Setting (h) |
| ------- | ----------------- | ------------------ | ----------------- |
| SM$^4$  | 33                | 18                 | 8                 |
| HighMMT | 33                | 17.5               | 8                 |

Reviewer comment (Evaluation):
The reported performance of MultiBench models in Table 2 may be overly simplified. As the complexities of the MultiBench models greatly vary, simply reporting the aggregated performance (e.g., the range) makes it difficult to position SM^4 in this regard. Also, the dependencies between efficiency and performance are ignored, similar to the issue in weakness 1.b. A candidate method could be the 2D visualization adopted by MultiBench and used for studying trade-offs.

Author response:
## **[Cons 3. The Reported Performance of MultiBench Models in Table 2 Maybe Overly Simplified.]**
To address the concern regarding the detailed performance of MultiBench models, we included a more comprehensive performance report below (Table **T.3**, **T.4**, **T.5**, and **T.6** below) and in our revision (Table 10 (page 22) in our revision). 

Also, the typo in Table 1 has been corrected in our revision.

**T3**. Detailed performance, parameter usage, and FLOPS of multi-modal learning method on MultiBench benchmark. For the ''FLOPS(G)'', ''-'' indicates the MultiBench does not provide official implementation. Notably, the empty FLOPS of the MultiBench Model (MFAS)'' is due to the FLOPS of ''MFAS'' being dynamic during training. For each dataset, we choose multi-modal models with the best/worst performance and multi-modal models with the largest/smallest parameter numbers, respectively.
| Method                             | Dataset           | Performance | \# Parameter (M) | FLOPS (G) | 
| ---------------------------------- | ----------------- | ----------- | ---------------- | --------- |
| MultiBench Models (TF-LSTM)        | PUSH $\downarrow$ | 0.574       | 23.5             | 25.11     |    
| MultiBench Models (LF-LSTM)        | PUSH $\downarrow$ | $0.290$       | $1.90$             | $14.07$     |     
| MultiBench Models (MULT)             | PUSH $\downarrow$ | 0.402       | 14.6             | 19.20     |    
| MultiBench Models (LRTF)             | V\&T              | 93.3        | $1.09 $            | $5.20$      |  
| MultiBench Models (LF)             | V\&T              | $93.6$        | 1.20             | $5.20$      |    
| MultiBench Models (RefNet)         | V\&T              | 93.5        | 135              | $-$       |     
| MultiBench Models (TF)             | ENRICO            | 46.6        | $19.3$             | 314.13    |   
| MultiBench Models (GradBlend)      | ENRICO            | $51.0$        | $19.3$             | 314.13    |  
| MultiBench Models (RefNet)         | ENRICO            | 44.4        | 25.7             | $2.67$      |
| MultiBench Models (GradBlend)      | AV-MNIST          | 68.5        | 0.29             | 0.50      |   
| MultiBench Models (MFAS)           | AV-MNIST          | $72.8$        | $0.14$             | $-$       |  
| MultiBench Models (RefNet)         | AV-MNIST          | 70.9        | 14.1             | $0.25$      |  
| MultiBench Models (EF-GRU)         | UR-FUNNY          | 60.2        | 3.58             | 3.13      |  
| MultiBench Models (MULT)           | UR-FUNNY          | $66.7$        | 2.38             | 3.37      | 
| MultiBench Models (MCTN)           | UR-FUNNY          | 63.2        | $0.19$             | $0.17$      |  
| MultiBench Models (TF)             | UR-FUNNY          | 61.2        | 12.2             | 2.67      | 
| MultiBench Models (MCTN)           | MOSEI             | 76.4        | $0.19$             | $0.15$      |   
| MultiBench Models (MULT)           | MOSEI             | $82.1$        | 4.75             | 3.35      | 
| MultiBench Models (LF-Transformer) | MOSEI             | 80.6        | 31.5             | 21.6      |   
| MultiBench Models (MI-Matrix)      | MIMIC             | 67.9        | 0.801            | $0.005 $    |   
| MultiBench Models (LF)             | MIMIC             | $68.9$       | 0.034            | $0.005$     |   
| MultiBench Models (LRTF)           | MIMIC             | 68.5        | $0.008$            | $0.005$          |  

**T4**. Performance, parameter usage, and FLOP of our model, and HighMMT in the small setting.
| Method  | Dataset           | Performance | \# Parameter (M) | FLOPS (G) |
| ------- | ----------------- | ----------- | ---------------- | --------- |
| HighMMT | PUSH $\downarrow$ | 0.445       | 0.89             | 5.14      |
| HighMMT | V\&T              | 96.10       | 0.85             | 32.48     |
| SM$^4$  | PUSH $\downarrow$ | $0.331$       |$0.27 $            | $2.59$      |
| SM$^4$  | V\&T              | $96.33$       | $0.25$             | $17.38$          |

**T5**. Performance, parameter usage, and FLOP of our model, and HighMMT in the medium setting.
| Method  | Dataset           | Performance | \# Parameter (M) | FLOPS (G) |
| ------- | ----------------- | ----------- | ---------------- | --------- |
| HighMMT | ENRICO            | 53.10       | $0.58 $            | 79.48     |
| HighMMT | PUSH $\downarrow$ | 0.600       | $0.63$             | 21.60     |
| HighMMT | AV-MNIST          | 68.48       | $0.52 $            | 0.95      |
| SM$^4$  | ENRICO            | $71.58 $      | 1.23             | $1.10$      |
| SM$^4$  | PUSH $\downarrow$ |$ 0.475$       | 1.25             | $2.33$      |
| SM$^4$  | AV-MNIST          | $71.86$       | 1.23             | $0.41$          |

**T6**. Performance, parameter usage, and FLOP of our model, and HighMMT in the large setting.
| Method  | Dataset  | Performance | \# Parameter (M) | FLOPS (G) |
| ------- | -------- | ----------- | ---------------- | --------- |
| HighMMT | UR-FUNNY | 62.00       | $0.52$             | 1.51      |
| HighMMT | MOSEI    | 78.40       | $0.52$             | 1.65      |
| HighMMT | MIMIC    | 65.60       | $0.52$             | 0.67      |
| HighMMT | AV-MNIST | 70.60       | $0.52$             | 0.95      |
| SM$^4$  | UR-FUNNY | $64.24$       | 0.76             | $0.38$      |
| SM$^4$  | MOSEI    | $79.47$       | 0.76             | $0.53$      |
| SM$^4$  | MIMIC    | $67.91$       | 0.76            | $0.15$      |
| SM$^4$  | AV-MNIST | $71.05$       | 0.76             | $0.43$          |

Reviewer comment (Writing):
Minor typo: YR-FUNNY in Table 1.

Author response:
Also, the typo in Table 1 has been corrected in our revision.

Reviewer comment (Writing):
The description of proposed method is very difficult to follow. I have no idea how the method works from just reading the paper. For example, it does not clearly state what are exactly the experts and where they come from. I cannot get much from Figure 1. It seems that experts are grouped. But I was not able to find a discussion why/how they are grouped.

Author response:
The experts are duplicated multi-layer perceptrons (MLP). Specifically, in SMoE, the experts are duplicated from the original feedforward networks. The experts in SMoA are duplicated from the query, key, and value MLPs, respectfully. 

In order to distinguish experts from different sources, we name them as different expert groups. **Experts are grouped by the nature of where they are duplicated from.**

The above clarifications have been included in our revisions.

Reviewer comment (Theory):
The motivation of ALP is not clear to me. Does unstable routing policy just mean changes in the policy across iterations? Such change does not necessarily link to the routing distribution entropy.

Author response:
1. *What does an unstable policy mean?*

An unstable routing policy means the changes in the policy across iterations.

2. *How do such changes link to routing distribution entropy?*

They are linked. We measure the routing distribution of each expert across the iterations and then calculate an averaged entropy across all the experts. Therefore, the entropy here directly measures the stability of the routing policy over the training iterations. For example, a high entropy indicates that expert routing is quite converged and stable.

3. *Motivation of ALP.*

The motivation of adaptive learning pace (ALP) is to align different learning paces between modalities, aiming to synchronize the optimization of multiple objectives. To be specific, from Figure 2(d), the stability of modality-specific routing seems to reveal the convergence of modality-specific training, serving as a good guidance to decay its learning rate.
The above clarifications have been included in our revisions.

Reviewer comment (Reproducibility):
Comparing Table 2 with Table 3 of Liang, et al., 2022, there is large difference in the performance of HighMMT on same datasets. A discussion of where the discrepancy coming from is needed.

Author response:
Thanks for pointing it out.
- We highlight that our performance of HighMMT is produced with the official implementation from HighMMT’s repository (https://github.com/pliang279/HighMMT). We strictly follow the default configurations reported in their paper, as shown in Tables 8, 9, and 10. For example, we use learning rates of 0.0005, 0.001, and 0.0008 for the small, medium, and large settings, respectively.  
- Additionally, our supplementary materials have included the HighMMT code and reproduction scripts, maintaining exact replication of their hyper-parameter settings.
- If Reviewer m8oX could kindly point out another better re-implementation of HighMMT, we would like to follow it and update the configurations.

Reviewer comment (Experiments):
What are the criteria/considerations used in selecting datasets/tasks for evaluation? There are many other tasks and modalities in MultiBench. How the proposed method works for those?

Author response:
The selection of our three multi-modal task groups follows HighMMT’s standards. 
- The small setting encompasses similar research areas with varying modality inputs;
- The medium setting spans three domains featuring different modalities;
- The large setting involves three domains incorporating diverse modalities.

Yes, our proposed method is a general multi-modal, multi-task pipeline, which can be leveraged for various combinations of tasks and modalities. **In our submission, we follow the standard multi-modal multi-task settings from the HighMMT paper**. That is, HighMMT/Multibench currently only provides the default hyper-parameter configurations for the small, medium, and large settings, which are adopted in our paper as the representative baseline. In our future works, we will examine our methods for extra combinations of tasks and modalities.