Paper: Fusing Models with Complementary Expertise (ICLR-2024)

Reviewer comment (Experiments):
The proposed fusion method relies on a validation set containing data samples from all \( K \) domains, and potentially, the distribution of the validation data is very similar to that of the test data. Traditional approaches for combining expert predictions may be ineffective and mostly use heuristic schemes such as averaging models' outputs or using the most confident model, but they do not assume access to additional validation data. If validation data is available, could a parameterized fuser also be trained for traditional methods? This raises concerns about fairness, or alternatively, is this intended as a setup for a new fusion task?

Author response:
**Major concern**: *the fusion based method relies on a validation set (+ additional assumption that the validation set should be similar to the test set).*  
**Response**: Please kindly refer to our general response.  
\
\
**Concern**: *If there is available validation data, can we train a parameterized fuser for traditional methods? Therefore, this is not fair in a sense, or can I understand that this is a setup for a new fusion task?*  

**Response**: We kindly request that the reviewer provides further clarification on their concern, as we are not entirely clear on the issue.  

When comparing our proposed FoE method to traditional methods, the key difference lies in the approach. Traditional ensembles aggregate experts' outputs through methods like simple averaging or majority voting. Thus, not like FoE, it is test sample agnostic. The MoE method, on the other hand, requires joint training of experts with the gating mechanism. In contrast, FoE allows the use of pre-developed expert models, which are then fused for enhanced performance. This is achieved by training a lightweight fuser model, which is responsible for selecting the most suitable expert for each test sample.

Reviewer comment (Evaluation):
In Table 3, the difference between "CNN DM Expert" (higher part) and "CNN DM Expert only" (lower part) is unclear. Why is there such a wide performance gap between the two?

Author response:
**Concern**: *Some more explanations about the results in Table 3.*  

**Response**: "CNN DM Expert'' refers to the accuracy of the CNN DM expert on each sentiment analysis task, serving as a baseline to indicate individual expert performance. "CNN DM Only" assesses the fuser's performance when only the embedding feature of the CNN DM expert is used for training. This demonstrates that FoE is inherently frugal on the summarization task, as relying on a single expert's embedding is sufficient for accurate expert selection. And that explains why there is a large performance gap between those results. We modified Table 3 to make this less confusing, thanks for pointing it out!

Reviewer comment (Theory):
To train using the loss in Eq 3.3, one needs to know a priori the labels, ie. which is the correct model for that input. How reasonable is that assumption? Do we also know "exactly" which model should be selected?

Author response:
**Concern**: *How reasonable is that assumption that one knows the correct model for each input? Do we also know "exactly" which model should be selected?*  

**Response**: Please kindly refer to our general response.

Reviewer comment (Theory):
I have some doubts about Equations 4.3 and 4.4 that I would like the authors to clarify. As far as I understand, to obtain the optimal (argmin), we must execute all the experts individually (in Eq. 4.3) and all the subsets in $\mathcal{F}\backslash\tilde{\mathcal{S}}$ for Eq. 4.4. This sounds quite intensive, and definitely more intensive than just running $\mathcal{S}$ experts once. I know there is something I am missing here, I kindly ask the authors to bring some clarity in this sense.

Author response:
**Concern**: *Clarifying Equations 4.1, 4.3, and 4.4.*  

**Response**: Thank you for bringing up these questions regarding our equations. We have made changes in the main text to make things clearer. Regarding Eq. 4.1, we sum over $k:f_k \in \mathcal{S}$ because, in our notation, $\mathcal{S}$ includes all experts in $\mathcal{\tilde{S}}$ (queried experts) plus some extra expert we want to query in the next step; therefore, it should not be constant when choosing a new expert. Regarding Eq. 4.3 and 4.4, we assume access to the experts’ outputs for all data points in a validation set. For example, if we have a validation set with a hundred data points and five experts, we have a total of $100\times 5 = 500$ predictions/generations in the validation set. In the first equation, we just choose the expert which had the best average performance in the validation set, while in the second we choose a new expert which is expected to reduce the loss by the biggest amount. The intensive part here is training many fuser models to combine experts in different ways. Still, that cost can be alleviated using k-nearest neighbors as explained in the paragraph “Obtaining fusers for subsets of experts” (Section 4.2).

Reviewer comment (Theory):
Why is the cost term in Equation 4.1 summed over $f_k\in \mathcal{S}$? I would have expected this sum to be over $f_k\in \tilde{\mathcal{S}}$, otherwise the term becomes constant wrt. the queried experts, right?

Author response:
**Concern**: *Clarifying Equations 4.1, 4.3, and 4.4.*  

**Response**: Thank you for bringing up these questions regarding our equations. We have made changes in the main text to make things clearer. Regarding Eq. 4.1, we sum over $k:f_k \in \mathcal{S}$ because, in our notation, $\mathcal{S}$ includes all experts in $\mathcal{\tilde{S}}$ (queried experts) plus some extra expert we want to query in the next step; therefore, it should not be constant when choosing a new expert. Regarding Eq. 4.3 and 4.4, we assume access to the experts’ outputs for all data points in a validation set. For example, if we have a validation set with a hundred data points and five experts, we have a total of $100\times 5 = 500$ predictions/generations in the validation set. In the first equation, we just choose the expert which had the best average performance in the validation set, while in the second we choose a new expert which is expected to reduce the loss by the biggest amount. The intensive part here is training many fuser models to combine experts in different ways. Still, that cost can be alleviated using k-nearest neighbors as explained in the paragraph “Obtaining fusers for subsets of experts” (Section 4.2).

Reviewer comment (Evaluation):
How can we use $c_k$ in practice? Can we use it to model aspects like energy consumption (for running an expert), flops, etc.?

Author response:
**Concern**: *Using $c_k$ in practice.*  

**Response**: The $c_k$’s can include different types of costs. It could include energy consumption if experts are run locally or API costs when a third party provides experts. We made this point more explicit in the text.

Reviewer comment (Evaluation):
Can the authors comment on Figure 2? Why is FrugalML performing so poorly? Even much worse than randomly picking the experts?

Author response:
**Concern**: *More explanation on why FrugalML performed so poorly, and adding error bars for the random expert baseline.*  

**Response**: FrugalML considers composing up to two experts. Thus, in our setting, its performance saturates after two experts (i.e., allowing FrugalML to query more experts is not going to improve its performance). However, when querying either one or two experts, FrugalML still outperforms the approach of selecting random experts. We updated Figure 2 to include error bars for the random expert baseline.

Reviewer comment (Evaluation):
Additionally, it would be interesting to add std bars for the Random Experts (selecting different random subsets of them, specially at lower values of the x axis).

Author response:
**Concern**: *More explanation on why FrugalML performed so poorly, and adding error bars for the random expert baseline.*  

**Response**: FrugalML considers composing up to two experts. Thus, in our setting, its performance saturates after two experts (i.e., allowing FrugalML to query more experts is not going to improve its performance). However, when querying either one or two experts, FrugalML still outperforms the approach of selecting random experts. We updated Figure 2 to include error bars for the random expert baseline.

Reviewer comment (Writing):
In Section 5.2, the authors claim `Though sentiment analysis is essentially a classification task, we train the fuser using the generative model strategy 3.3`. I believe this is a typo and should be "using the classification model strategy".

Author response:
**Concern**: *Though sentiment analysis is essentially a classification task, we train the fuser using the generative model strategy 3.3. I believe this is a typo and should be "using the classification model strategy".*  

**Response**: No, it’s not a typo. The key difference between our generative model strategy and the classification strategy lies in their approaches: while the generative model strategy is designed to predict "which expert to use," the classification strategy directly employs the fuser for classification tasks. In our experiment, even though sentiment analysis inherently involves prediction, we train the fuser to determine which expert should be used for the sentiment analysis task.

Reviewer comment (Writing):
Minor notation consistency comment. The set $\mathcal{S}$ is not defined when it first appears. Furthermore, one can find both $k\in\mathcal{S}$ and $f_k\in\mathcal{S}$ in the manuscript, which complicates readability.

Author response:
**Concern**: *Notation inconsistency.*  

**Response**: Thank you for pointing this out. We have corrected this issue in the main text.

Reviewer comment (Experiments):
A primary limitation is the assumption that data for training the fuser are readily available. The proposed approach requires a labeled dataset to train the fuser network. In the discriminative case, we feed the input example to different individual models, take model outputs as the inputs to the fuser network, and train the fuser to predict the label of the input example. Similarly, in the generative case, we feed the fuser network with individual model outputs and train the fuser to predict the best model index, which is obtained by feeding the labeled data to individual models and selecting the model that achieves the best performance. This prerequisite may not be realistic in practical scenarios, where there is no or only a few labeled data. If a large labeled dataset is available, it might be more efficient to fine-tune a foundational model or employ few-shot learning. A pivotal aspect of this research should be the generalization capabilities of the trained fuser across different tasks (e.g., train the fuser on some tasks and test the fuser on unseen tasks), which would significantly enhance the paper's contribution.

Author response:
Please refer to our general response to this point. 

Regarding the suggestion that “If a large labeled dataset is available, it might be more efficient to fine-tune a foundational model or employ few-shot learning,” we contend that in FoE, only a very small fuser model needs to be trained, typically a small MLP or even a kNN, where no training is required. In terms of the number of parameters, the fuser models are usually orders of magnitude smaller than the expert models. This approach is computationally much cheaper than fine-tuning a foundational model, as the reviewer suggested, and also cheaper than fine-tuning any individual expert model.

Reviewer comment (Experiments):
The paper lacks an in-depth analysis of the fuser network. Although it is described as a lightweight MLP network, there's no exploration of how different architectures (e.g., simpler networks like linear models or more complex ones like transformers) might impact performance. Additionally, details on the training configurations, such as learning rate, epochs, dataset, and stopping criteria, are missing. The influence of the $K$ parameter in the $K$-NN component of the fuser network is also unclear. What is the rationale for choosing $K=9$?

Author response:
Thanks for pointing it out! A promising aspect of the FoE is its flexibility in the architecture of the fuser model; it doesn't require any specific design. In our experiments, we explored using both simple MLPs and kNNs as fusers and found that kNNs offer certain advantages, particularly in frugal settings. Additionally, we experimented with simple linear models during the development of the FoE method, although these results are not yet included in the paper. The table below compares the performance on the CIFAR-100 dataset using both MLP and linear models as fusers, each trained on the combined training set of all experts. We observed that using models as complex as MLPs are sufficiently effective, as evidenced by our results in summarization and sentiment analysis tasks where our fuser performs comparably to the oracle models. Therefore, we believe there is no need to explore more complex models like transformers, especially since the input for the fusers is not sequence data, and doing so would compromise the FoE's advantage of having an easily trainable fuser. We will include a discussion on the specific choices of fuser architecture in the revised draft.  
  
| Fuser type      | Final Accuracy (%) |
| ----------- | ----------- |
| MLP as the fuser      | **84.23**       |
| Linear model (logistic regression) as the fuser   | 83.12        |
  
For questions regarding experimental details, we will revise the experiment section to include more comprehensive descriptions of our setups. The details of our hyper-parameters are already specified in Appendix C. The fuser training process is so simple that there aren’t many hyper-parameters to report.  

As for the different choices of $\kappa$, our approach is to select a relatively small and odd number, as is common practice in such experiments. We will update the manuscript with a plot showing results for various values of $\kappa$ shortly, as these experiments are currently in progress. For additional experimental details, we have provided the code used in our experiments, which contains all the necessary information to replicate our results.

Reviewer comment (Experiments):
The experiment sections lack in-depth descriptions (see questions). The paper would benefit from reallocating less critical sections (like the connection between FrugalFoE and the A* algorithm) to the Appendix and expanding on the experimental details.

Author response:
We will expand our experiment section in the final version.

Reviewer comment (Presentation):
Page 8, Table 2: Could you clarify the distinction between "TFN Expert Only" and "TFN Expert," and between "Poem Expert" and "Poem Expert Only"?

Author response:
"TFN Expert" refers to the accuracy of the TFN expert on each sentiment analysis task, serving as a baseline for individual expert performance. "TFN Expert Only" evaluates the fuser's performance when it is trained using only the embedding feature of the TFN expert. This indicates that FoE is inherently efficient for sentiment analysis tasks since using the embedding of a single expert is enough to accurately predict expert selection. We modified the names in the tables to make them easier to understand.

Reviewer comment (Experiments):
Page 8, Table 2: What are the results for the "confidence-based fusion" and "ensemble" baselines? The same question applies to Tables 3 and 4. The confidence-based fusion seems less intuitive in the generative case, how about simply selecting the maximum confidence at each decoding step?

Author response:
The “confidence-based fusion” is a simple baseline that uses the maximum softmax scores (MaxSoftmax) of each expert as a measure of their confidence for certain test samples. MaxSoftmax is a popular and commonly used technique in the field of uncertainty quantification, always selecting the expert that returns the highest MaxSoftmax score. For the “ensemble” baseline, we average the logits of all experts for prediction. However, we agree that both “confidence-based fusion” and “ensemble” baselines may not be intuitive or even feasible in the generative case, which is why we excluded them from Table 3 and Table 4.

Reviewer comment (Evaluation):
Page 9, Table 4: Could you explain more details of FoE (Expert 1), FoE (Expert 2), and FoE (Expert 3)? Why does adding more experts appear to degrade performance?

Author response:
The baselines FoE (Expert 1), FoE (Expert 2), and FoE (Expert 3) are similar to the “Poem Expert” and “Poem Expert Only” baselines, as previously explained. These baselines indicate the performance of FoE when using different individual experts' embeddings as features to train the fuser model, rather than the performance of using one, two, or three experts. We have not yet experimented with adding more experts. Our FrugalFoE can effectively identify and eliminate redundant experts, selecting the one that provides the most effective embedding in this weak expert setup. We modified the names in the tables to make them easier to understand.

Reviewer comment (Theory):
Page 4: The statement "As long as there is a label shift among the domains, we expect $E[f(X_k)] = E[Y_k]$" needs clarification. Why is this expected?

Author response:
The expression $E[f(X_k)] = E[Y_k]$ simply states that average predictions of an expert across the samples from its domain are approximately equal to class marginals on this domain. This is always true for any reasonable classification model. “We expect” refers to these marginals varying across domains (the last three words in this sentence) in the presence of label shift, which is true by definition of the label shift.

Reviewer comment (Writing):
Page 1: The statement "our emphasis is on generalization to test data distributions where none of the experts perform well individually". Shouldn't it be "a few" instead of "none" based on the problem formulation?

Author response:
No, "none" is correct. Recall that test distribution is a mixture of all domains. Individual experts only perform well on their respective fraction of the test data, i.e., $\frac{1}{K}$ of the test set, thus none of them perform well on the complete test data distribution. This is also evident in our experiments, i.e., in Table 3 individual experts perform poorly on the test distribution as reported in the column "Avg."

Reviewer comment (Theory):
Page 6: The condition "If ... <0 we terminate the search" should be ">0"?

Author response:
Yes, you are right. Thank you for pointing this out! We have fixed the condition.

Reviewer comment (Experiments):
The proposed algorithm requires multiple experts to be used together, unlike Sparse MoE, which means that the inference cost is multiple times that of each expert. Therefore, the correct baseline for the proposed algorithm is to compare it to an equal number of parameters with the sum of all individual experts. The authors should make this comparison in their paper.

Author response:
**Concern**: *The correct baseline for the proposed algorithm is to compare it to an equal number of parameters with the sum of all individual experts or Sparse MoE*.

**Response**: 
We would like to respectfully disagree with the reviewer’s point. We view FoE as a lightweight paradigm that allows us to easily fuse pre-trained expert models to achieve better performance on a mixture of downstream tasks and domains.  

It is not very clear how to directly adopt Sparse MoE in a scenario where $K$ pre-trained experts are provided with the data used to build them. One possible approach is to freeze all pre-trained experts and only train the gating network using the available datasets. In such a case, the gating network would need to directly process data, including images and sequences of tokens, requiring a complex architecture like a BERT, a ViT, or a convolutional neural network. This requirement significantly increases the computational difficulty and model complexity needed to build such a Sparse MoE.  

Regarding the suggestion to use a model with the combined number of parameters of all experts, we believe this approach is not quite practical. FoE allows us to leverage the resources created by the entire open-source community, such as trained and fine-tuned expert models or LLMs, to achieve better results. For example, one can easily download ten 13B expert LLMs from Hugging Face with just a few lines of code. However, obtaining and running inference on a 130B expert model, as suggested by the reviewer, is far more challenging. In most cases, such a 130B expert LLM does not even exist, and constructing one poses even greater challenges.