Paper: Knowledge Fusion of Large Language Models (ICLR-2024)

Reviewer comment (Reproducibility):
The implementation part is not very well written, and I cannot completely understand the specific details toward the mechanism of the alignment and fusion.

Author response:
**Q1: Regarding the specific implementation details of alignment and fusion.**

**A1:**

1）MinED works for token alignment

For an input text x, token alignment involves aligning two distribution matrices from two LLMs. Therefore, the alignment comprises two dimensions: token-wise with respect to text x and distribution-wise with respect to the vocabulary. 

In the token dimension, we utilize the dynamic programming approach to recursively minimize the total cost of editing one sequence of tokens to align with another. When the mapped tokens are identical, such as the token "now" in the given example, these tokens are effectively aligned, allowing for the corresponding distributions to align subsequently. However, when the mapped tokens exhibit differences, such as the 'get' and 'gets' tokens in the example, the previous EM method does not align these tokens, resulting in the distributions degenerating to one-hot vectors. In contrast, our proposed MinED method successfully aligns the 'gets' token with the 'get' token, as they exhibit the minimal edit distance in the vocabularies derived from the two LLMs.

Concerning the distribution dimension, the alignment is performed between two vocabularies from different tokenizers of two LLMs. Therefore, for distribution values with identical tokens, such as 'current 0.05' and 'current 0.04', they will be aligned effectively. For distribution values involving different tokens, such as 'immediate 0.04' and 'immediately 0.03', the EM method disregards this value. However, the proposed MinED method maps 'immediately' to 'immediate' due to their minimal edit distance, resulting in the successful alignment of these distribution values.

An example: https://anonymous.4open.science/r/ICLR24_Anonymous-9D08/token_and_vocab_alignment.pdf

We will include the above explanation when updating the paper.

2）MinCE/AvgCE works for knowledge fusion

Following token alignment, the next crucial problem in knowledge fusion is to merge the distribution matrices of text x from different LLMs and continually train the target LLM. We selected the cross-entropy loss between the distribution matrices and the gold labels as an indicator of the prediction quality of the LLMs. Specifically, we utilized either the minimum cross-entropy scores (MinCE) or a weighted average of the matrices based on cross-entropy scores (AvgCE) to merge the distribution matrices from different LLMs.

Reviewer comment (Reproducibility):
For the detailed implementation part, I am a little bit confused. Can you explain in detail how MinED works for token alignment, and how MinCE/AvgCE merge different matrices into P_t?

Author response:
**Q1: Regarding the specific implementation details of alignment and fusion.**

**A1:**

1）MinED works for token alignment

For an input text x, token alignment involves aligning two distribution matrices from two LLMs. Therefore, the alignment comprises two dimensions: token-wise with respect to text x and distribution-wise with respect to the vocabulary. 

In the token dimension, we utilize the dynamic programming approach to recursively minimize the total cost of editing one sequence of tokens to align with another. When the mapped tokens are identical, such as the token "now" in the given example, these tokens are effectively aligned, allowing for the corresponding distributions to align subsequently. However, when the mapped tokens exhibit differences, such as the 'get' and 'gets' tokens in the example, the previous EM method does not align these tokens, resulting in the distributions degenerating to one-hot vectors. In contrast, our proposed MinED method successfully aligns the 'gets' token with the 'get' token, as they exhibit the minimal edit distance in the vocabularies derived from the two LLMs.

Concerning the distribution dimension, the alignment is performed between two vocabularies from different tokenizers of two LLMs. Therefore, for distribution values with identical tokens, such as 'current 0.05' and 'current 0.04', they will be aligned effectively. For distribution values involving different tokens, such as 'immediate 0.04' and 'immediately 0.03', the EM method disregards this value. However, the proposed MinED method maps 'immediately' to 'immediate' due to their minimal edit distance, resulting in the successful alignment of these distribution values.

An example: https://anonymous.4open.science/r/ICLR24_Anonymous-9D08/token_and_vocab_alignment.pdf

We will include the above explanation when updating the paper.

2）MinCE/AvgCE works for knowledge fusion

Following token alignment, the next crucial problem in knowledge fusion is to merge the distribution matrices of text x from different LLMs and continually train the target LLM. We selected the cross-entropy loss between the distribution matrices and the gold labels as an indicator of the prediction quality of the LLMs. Specifically, we utilized either the minimum cross-entropy scores (MinCE) or a weighted average of the matrices based on cross-entropy scores (AvgCE) to merge the distribution matrices from different LLMs.

Reviewer comment (Experiments):
As a combined objective is used in FuseLLM (5), what is the choice of $\lambda$ in your experiment? I saw from Appendix E that for dynamic design $\lambda$ is set from 0.7 - 1.0, while 1.0 reduces the (5) to the regular CLM objective. Could you provide a more detailed elaboration on this hyperparameter?

Author response:
**Q2: Regarding the choice of λ in the experiment.**

**A2:** When introducing the training details in Section 4.1 (third paragraph), we specified that FuseLLM utilizes a fixed value of λ set at 0.9. In Appendix E, an alternative approach is discussed, wherein λ dynamically adjusts during training. As the training progresses, λ is linearly increased from 0.7 to 1.0, as further elaborated in the appendix. We will offer a more detailed explanation of this hyperparameter when revising the paper.

Reviewer comment (Evaluation):
The results (Table 1, 2 and 3) show that FuseLLM could not outperform original LLMs in some tasks (some in BBH and all the ME). Does it mean that the FuseLLM only work in some domains? Or in another word, does the FuseLLM could only inherit specific capabilities of LLMs? Have the authors studied this phenomenon? Overall, the improvements are not remarkable, will such fusion effort necessary (with a maybe large training cost)?

Author response:
**Q2: Regarding the FuseLLM could not outperform original LLMs in some tasks and only inherit specific capabilities of LLMs.** 

**A2:** Given that we assessed the proposed FuseLLM across a diverse set of 42 tasks, including reasoning, commonsense, and code generation, expecting significant improvements on every task for the fused model appears challenging and, in some cases, unrealistic. There are two primary factors for this difficulty, as discussed in Section 4.2. Firstly, certain source LLMs exhibit suboptimal performance compared to the target LLM in specific tasks, inevitably influencing the fusion results. Secondly, the relevance between the continual training dataset and downstream tasks also plays a crucial role in performance. Considering the diverse domains and tasks covered by downstream tasks, considerable effort is required to curate a corpus tailored for continual training.

In addition to the 42 tasks, we have conducted supplementary experiments on five new generative tasks (knowledge-based question-answering, reading comprehension, content analysis, machine translation, and theorem application) during this rebuttal. The experimental results further validate the efficacy of FuseLLM. We will provide a detailed explanation and incorporate these new experiments in the paper update.

| Models | TrivialQA | Drop | Lambada | IWSLT2017 | SciBench |
| :-: | :-: | :-: | :-: | :-: | :-: |
| OpenLlama | 39.96 | 22.31| 70.31 | 5.51 | 0.677 |
| MPT | 28.89 | 23.54 | 70.08 | 5.49 | 0.876 |
| Llama-2 | 52.46 | 27.25 | 73.28 | 6.48 | 0.137 |
| Llama-2 CLM | 53.14 | 28.51 | 73.45 | **6.91** | 0.937 |
| FuseLLM | **54.49** | **28.97** | **73.72** | 6.75 | **1.649** |

Reviewer comment (Experiments):
The experiments are only based on Llama-2, OpenLLama and MPT and on 7B scale. Why choosing these three types of models? Can the authors give some explanations? We know that the model will emerge powerful capabilities via scaling up. Maybe the fusion effectiveness will drop with the parameter scale grows. Authors may provide a discussion about this argue.

Author response:
**Q3: Regarding the choosing of Llama-2, OpenLlama, and MPT on a 7B scale.** 

**A3:** The motivation behind FuseLLM is to integrate the collective knowledge of multiple LLMs. This collective knowledge manifests in performance variations among source LLMs across diverse inputs and tasks, primarily stemming from disparities in their architecture and pre-training corpora. Among the three LLMs that we have selected, OpenLlama's pre-training corpus comprises Falcon refined-web data and the Starcoder dataset, which differs from Llama-2 in terms of the pre-training corpus. Meanwhile, MPT's pre-training corpus includes S2ORC and The Stack, and it replaces positional encoding with ALiBi and expands the dimension of the intermediate layer in FFN, making it distinct from Llama-2 in both pre-training corpus and architecture. 

The reasons for using LLMs at the 7B scale are twofold. Firstly, while our method is applicable to the fusion of larger source LLMs, such as 70+B LLMs, the cost associated with continually training a target large LLM is substantial unless we opt for a smaller target model. Secondly, the availability of choices for source LLMs diminishes as we scale up. Therefore, we conducted experiments on the 7B scale to demonstrate the effectiveness of the proposed fusion method and discussed the use of a larger 13B LLM to distill a smaller 7B LLM in Section 4.5.  

Currently, we are conducting experiments to fuse larger LLMs (13B), and we will update the results once the experiment is concluded.

Reviewer comment (Evaluation):
How do the Avg. values be calculated in the last lines of Table 1, 2 and 3? Directly averaging all the values does not make sense due to the disparate data scales of each task.

Author response:
**Q4: Regarding the Avg. values calculating.** 

**A4:** Following previous work, such as Llama-2 [1], we employed simple averages for tasks belonging to the same category, such as the 27 tasks in BBH. In response to your question, we supplement results using weighted averages based on the number of samples in each task, and the results are shown below. We will clarify this in the revised version.

| Models | BBH | CommonSense | MultiPL-E |
| :-: | :-: | :-: | :-: |
| OpenLlama | 33.73 | 70.24 | **17.94** |
| MPT | 33.16 | 71.85 | 17.08 |
| Llama-2 | 39.63 | 73.21 | 14.48 |
| Llama-2 CLM | 40.42 | 73.41 | 14.69 |
| FuseLLM | **41.76** | **73.91** | 15.41 |

[1] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

Reviewer comment (Writing):
Are the Q_t in Eq. 4 and the Q_t in Eq. 2 the same? I guess the Q_t in Eq. 4 should be used another notation.

Author response:
**Q5: Regarding the O_t in Eq.2 and Q_t in Eq.4.** 

**A5:** We would like to clarify that there is no problem with the notation. In Eq.2, the one-hot label matrix is denoted as O_t, while in Eq.4, the output distribution matrix of the target LLM is represented by Q_t.

Reviewer comment (Experiments):
The proposed approach relying on continual pretraining with a small pretraining corpus (MiniPile) looks limited to fusing the knowledge of the raw pretrained version of LLMs, but not applicable to "-Chat" or "-Instruct" models (e.g., Vicuna, Alpaca, MPT-instruct, ChatGPT, etc.)

Author response:
**Q1: Regarding the application to “-Chat” or “-Instruct” models.** 

**A1:** Recall that the proposed FuseLLM method involves extracting distribution matrices from multiple distinct source LLMs and continually training the target LLM. Therefore, our approach is applicable to both '-Chat' and '-Instruct' LLMs, provided that all corresponding continual-training samples adhere to the instruction-tuning format. It is also essential to mask the instruction part when calculating the training loss. To confirm this, we conducted new experiments on the fusion of '-Instruct' LLMs.

Specifically, we initiated fine-tuning on Llama-2, OpenLlama, and MPT using 20K samples from Evol-Instruct, ShareGPT, and Open-Platypus datasets, respectively. Consequently, the three source LLMs transitioned into '-Instruct' LLMs. Then, we sampled another 5K samples from each of the aforementioned datasets to create a corpus for continual training, designating Llama-2 Evol-Instruct as the target LLM for model fusion. We assessed the instruction-following performance on the Vicuna Benchmark using the GPT-4 scoring standard [1], where a higher score indicates superior performance. The results presented in the following table show that FuseLLM surpasses each individual source '-Instruct' LLM, achieving the best performance.   We will include this experiments when updating the paper.

[1] Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://vicuna.lmsys.org.

| Models | GPT-4 Score |
| :-: | :-: |
| OpenLlama ShareGPT | 7.23 |
| MPT Open-Platypus | 6.46 |
| Llama-2 Evol-Instruct | 7.88 |
| Llama-2 Evol-Instruct CLM | 8.03 |
| FuseLLM | **8.16** |

Reviewer comment (Evaluation):
The performance improvement from the proposed fusion approach is not significant enough. Specifically,\
      on reasoning tasks: 39.70 (LLaMA2) -> 40.44 (LLaMA2 + MiniPile) -> 41.78 (LLaMA2 + MiniPile  + model fusion)\
      on commonsense tasks: 63.76 (LLaMA2) -> 63.86 (LLaMA2 + MiniPile) -> 64.56 (LLaMA2 + MiniPile + model fusion)\
      on code generation tasks: 14.63 (LLaMA2) -> 14.83 (LLaMA2 + MiniPile) -> 15.56 (LLaMA2 +MiniPile + model fusion)\
Given the MiniPile continual pretraining is not a key contribution of this paper, the actual accuracy improvement brought by the proposed model fusion on commonsense and code generation is actually smaller than 1 percent.

Author response:
**Q3: Regarding the not significant performance improvement.** 

**A3:** Given that we assessed the proposed FuseLLM across a diverse set of 42 tasks, including reasoning, commonsense, and code generation, expecting significant improvements on every task for the fused model appears challenging and, in some cases, unrealistic. There are two primary factors for this difficulty, as discussed in Section 4.2. Firstly, certain source LLMs exhibit suboptimal performance compared to the target LLM in specific tasks, inevitably influencing the fusion results. Secondly, the relevance between the continual training dataset and downstream tasks also plays a crucial role in performance. Considering the diverse domains and tasks covered by downstream tasks, considerable effort is required to curate a corpus tailored for better continual training.

In addition to the 42 tasks, we have conducted supplementary experiments on five new generation tasks (knowledge-based question-answering, reading comprehension, content analysis, machine translation, and theorem application) during this rebuttal (refer to Q2). The experimental results further validate the efficacy of FuseLLM. We will provide a detailed explanation and incorporate these new experiments in the paper update.

Reviewer comment (Experiments):
The evaluation of this paper doesn't include the comparison of FuseLLM with any of the previous model fusion techniques, e.g., LLM-Blender (Jiang et al.) mentioned in the "related work" section. Notably, the LLM-blender also doesn't require heavy continual training on a 7B large model, which is not discussed.

Author response:
**Q4: Regarding the comparison to previous model fusion techniques.** 

**A4:** The motivation behind FuseLLM is to integrate the collective knowledge of multiple LLMs with diverse architectures and pre-training corpora. **Consequently, the traditional fusion method of model merging, which demands identical model architectures, is not directly applicable in this context. While the ensemble technique performs a weighted average of prediction distributions from multiple LLMs, the drawback lies in the substantial memory and time costs when maintaining multiple source LLMs during inference.** Moreover, in the context of generative models that operate through step-by-step inference, the cost associated with distribution averaging at each step is considerable. A relatively feasible ensemble approach, such as LLM-Blender, involves combining the output texts of multiple LLMs. However, this approach requires additional training for a ranker and a fuser, demanding a substantial amount of high-quality labeled data.

For a fair comparison with model merging and model ensemble methods, in Section 4.6 we simulated fusion scenarios where the same LLM underwent continual training on different corpora. To further compare our method with LLM-Blender, we performed new experiments on the BBH and MultiPL-E benchmarks using the open-source ranker and fuser models of LLM-Blender. Notably, the CommonSense benchmark, which utilizes perplexity-based evaluation, cannot accommodate LLM-Blender. The experimental results are presented below, where LLM-Blender-top-3-rank-fuse refers to using the ranker to obtain the top 3 results and then using the fuser to combine them, and LLM-Blender-top-1-rank represents simply using the ranker to obtain the top 1 result. We observed a notable performance deterioration after fusion when employing both the ranker and fuser. This could be attributed to the fuser model's training within the instruction-tuning context, potentially leading to inadequate generalization to the test tasks. Furthermore, while LLM-Blender outperforms the combined ranker and fuser approach when solely utilizing the ranker, it remains inferior to the best-performing source LLM. This suggests the ranker model's inability to efficiently discriminate the optimal responses when combining different LLMs. The new experiments will be incorporated into the revised version.

| Models | BBH | MultiPL-E |
| :-: | :-: | :-: |
| OpenLlama | 33.87 | **18.11** |
| MPT | 33.38 | 17.26 |
| Llama-2 | 39.70 | 14.63 |
| LLM-Blender-top-3-rank-fuse | 24.48 | 0.06 |
| LLM-Blender-top-1-rank | 37.17 | 17.85 |
| Llama-2 CLM | 40.44 | 14.83 |
| FuseLLM | **41.75** | 15.56 |