Paper: Knowledge Fusion by Evolving Language Models (ICLR-2024)

Reviewer comment (Presentation):
Table 1 is Confusing. What do you mean by "round" and "key step"? More explanation is in need.

Author response:
W2: Thank you for the clarification. We have added the explanation in the caption of Table 1. Round means the number of times the models are edited when implementing a certain knowledge fusion method. The key step is renamed to key technique to highlight the difference between knowledge fusion methods.

Reviewer comment (Novelty):
Missing related work. The task arithmetic should be discussed, with EDITING MODELS WITH TASK ARITHMETIC as a representative example.

Author response:
W4: Thank you for the suggestion. We have added the discussion in the manuscript on the related work [1] in Section 2 and Section 6. In addition, we have cited two important related work about robust finetuning [2] [3] in the manuscript for further reference. 

[1] Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., & Farhadi, A. (2022). Editing models with task arithmetic. arXiv preprint arXiv:2212.04089. 

[2] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020. 

[3] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959–7971, 2022b.

Reviewer comment (Novelty):
The paper suggests the direct application of existing evolutionary algorithms for knowledge fusion, which is of limited novelty, yet it lacks an explanation for why evolutionary algorithms can ensure convergence to an optimal result. Furthermore, there is a significant concern regarding the substantial search cost incurred during the evolution process.

Author response:
W1: Thank you for requesting more clarity on evolutionary algorithms (EA) and the related computing cost.  

Proving convergence in EAs is challenging due to the stochastic and heuristic nature and is usually done using empirical studies and analyses. 

Despite that, EAs are adept at exploring complex and high-dimensional search spaces, where traditional optimization or search algorithms might struggle (e.g., learning tasks with complex and non-differentiable objective functions). Hence, EAs have been successfully applied in various fields, including engineering, robotics, finance, biology, and more, solving complex real-world problems where traditional methods fall short. Moreover, EAs are complementary to other techniques and yield improved performance.  

The computation cost is discussed in Appendix C.5. We found that the computation cost is reasonable, i.e., the evolution process can be completed within half an hour using either A6000 or A800 GPU. The ability to produce higher-quality solutions in diverse NLP tasks makes evolution process valuable for research and practical applications. Moreover, we demonstrate that combining EA with existing methods yields more efficient solutions in our ablation study (Section 5.4). Hence, continued research can aim at improving EAs' efficiency, scalability, and applicability across a broader range of problem domains.

Reviewer comment (Experiments):
Absence of experiments involving natural language generation and other model architectures (e.g., encoder-decoder or decoder-only): All experiments are based on the encoder-only model for natural language understanding tasks. It would be valuable to observe experiments using encoder-decoder or decoder-only models, especially in natural language generation tasks.

Author response:
W2: Thank you very much for this insightful suggestion. We have incorporated additional architectures, T5-base and GPT2, into our experiments, as suggested by Reviewers 1 and 3. The new results have been added to Tables 2 and 4, demonstrating the consistently superior performance of our proposed method compared to existing methods. This reinforces the generalizability of our approach to virous neural architectures.

Reviewer comment (Writing):
The presentation of this paper is subpar. In addition to language issues, numerous crucial concepts are not explained clearly.

Author response:
W3: Thank you for pointing out the issue. We acknowledge the need for a clearer presentation and have revised the manuscript accordingly. Concepts, such as multi-task learning and the evolutionary algorithm for optimization, have been explained in more detail in the Introduction. We also discuss more details about compared method and experiment settings in Appendix C. Also, we provide explanation to the Reviewer’s questions in the following and revised the manuscript accordingly. We believe these suggested changes have improved the overall quality of the presentation.

Reviewer comment (Reproducibility):
What are the implementation details of combining evolver with other model merging methods?

Author response:
Q1: Thank you for the question. The implementation details of combining evolver with other model merging methods are established in the computation of the population's scores after mutation and crossover. The combining approach involves calculating an overall score by using model merging on the mutated individuals along with the non-mutated individuals in the current evolution. In contrast, the simple evolver determines the success of mutation based on the score of individual entities, while combining assesses it based on the score of the entire population after individual mutation. We added a paragraph in Appendix A and details are shown in step 3 (model inference) in Algorithm 1. For further reference, the reader could also refer to the source code.

Reviewer comment (Evaluation):
What are the details of Avg. *f*1*..*N, Best. *f*1*..*N, Domain-Specific, and MTL? Especially, what is the difference between Best. *f*1*..*N and Domain-Specific?

Author response:
Q2: Thanks for requesting more clarity on description of abbreviations. We now provide more explanation in Appendix C.2 accordingly to enhance readability. The following are the details:  

Avg. $f1*..N$ shows the average performance of all individual models. Best. $f1..N$ shows the performance of the best single individual model, as determined by using the validation set. Domain-Specific shows the performance of the individual models corresponding to the training data set for each test set. MTL shows the performance of the model trained with multi-task learning method.

Reviewer comment (Presentation):
In my perspective, the major issue is the presentation of the paper. I find the design of the diagrams, tables, and experiment setups overly similar to a paper authors cited [1], namely Figure 1, Table 2, Table 3. At first glance, I was very confused because of the similarity; until I realized that the submission indeed proposes novel ideas and presents interesting new results. I believe whether the similarity matters is subjective, as it is inevitable for follow-up studies to apply the same experiment setups. Therefore, I would like to hand over the issue to the Area Chair. At the same time, I hope to hear from authors about any plans to modify layouts of Figure 1, Table 2, Table 3 to avoid potential confusion.

Author response:
W1: Thank you for the point out the issue. We have re-designed diagrams and tables to avoid potential confusion with paper [1]. Our motivation of taking reference from the experimental setup in the paper [1] is mainly for fair and sufficient comparison with previous methods. The clear comparison could allow us to emphasize the main contribution of our paper that is the newly proposed model evolution algorithm. Furthermore, to further we have used a new experiment setup to compare the performance with coefficient search in Appendix D. This helps further improve the overall novelty of our paper. 

We have revised Fig.1. Since the parts on ensembling and model merging are not our contribution, we remove the part of ensembling and clarify that model merging is the previous method and model evolution is our proposed method. In Table 2 and Table 4, we have added the results of models with encoder-decoder architecture and decoder-only architecture. The modified figures and tables are shown in our new version of the paper. Lastly, we have modified a few minor writing issues about citations, i.e., we remove the parenthesis of citation when the author is included in the sentence. We hope that these changes have addressed the Reviewer's concerns about the presentation of our manuscript.

Reviewer comment (Writing):
There are also other minor writing issues in the paper, such as citations should not be in parenthesis when the authors are included in a sentence.

Author response:
W1: Thank you for the point out the issue. We have re-designed diagrams and tables to avoid potential confusion with paper [1]. Our motivation of taking reference from the experimental setup in the paper [1] is mainly for fair and sufficient comparison with previous methods. The clear comparison could allow us to emphasize the main contribution of our paper that is the newly proposed model evolution algorithm. Furthermore, to further we have used a new experiment setup to compare the performance with coefficient search in Appendix D. This helps further improve the overall novelty of our paper. 

We have revised Fig.1. Since the parts on ensembling and model merging are not our contribution, we remove the part of ensembling and clarify that model merging is the previous method and model evolution is our proposed method. In Table 2 and Table 4, we have added the results of models with encoder-decoder architecture and decoder-only architecture. The modified figures and tables are shown in our new version of the paper. Lastly, we have modified a few minor writing issues about citations, i.e., we remove the parenthesis of citation when the author is included in the sentence. We hope that these changes have addressed the Reviewer's concerns about the presentation of our manuscript.

Reviewer comment (Experiments):
What is the performance of merged models with simple average / regmean / fisher-weighted averaging when you apply coefficient search? Does the proposed approach improve over coefficient search?

Author response:
W2/ Q1: Thank you for the insightful suggestion of using coefficient search. We are grateful that the Reviewer raises this essential concern, which could potentially inspire the development of novel algorithm.  

We would like to answer this question from both the theory and experiment aspects. Theoretically, a coefficient search can be treated as a method with greedy rounds aiming to obtain better efficiency for different models to merge, but a vanilla coefficient search method does not consider the crossover ratio for a better search result.  

We have conducted the experiment to investigate the coefficient search. The coefficient search is a promising scheme to improve the model merging performance, by searching the optimal $\alpha$. We find that the proposed model evolution can also be integrated with the coefficient search method by searching the optimal scale factor $f$. We have performed the grid search of $alpha$ and $f$ with intervals of 0.05 from 0.1 to 0.9. We present the result in Appendix D. In the setting of Simple, Fisher and RegMean, the results show that a default version of evolver (with scale factor $f=0.5, cr=0.5$) outperforms the coefficient search results. Notably, the integration with scale factor search could further boosts the performance of the evolver, which is worth further investigation. 

We thank the Reviewer again for the useful comments. We have revised the manuscript according to the Reviewer’s suggestion and response to each comment provided in the Weakness section above. We hope that our rebuttal aligns with the reviewer’s expectations, and we hope that the Reviewer can consider possibly giving a higher rating. Thanks.