Paper: Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective (ICLR-2024)

Reviewer comment (Experiments):
How does your method compare with the existing distillation and pruning methods? It would have been great to see some other methods applied for comparison.

Author response:
We are grateful for your valuable review and thoughtful suggestions, and we try to answer your questions as follows.

> **Q1**: How does your method compare with the existing distillation and pruning methods? It would have been great to see some other methods applied for comparison.

Firstly, we would like to clarify that our study's focus on parametric knowledge transfer is fundamentally distinct from distillation and pruning methods, characterized by the following differences:

* **Purpose and Focus**: Rather than proposing better distillation or pruning methods, our study is focused on exploring the transferability of implicit knowledge embedded in static parameters. While prior research has explored the detectability and editability of parametric knowledge, its transferability remains less explored. Our experiments provide empirical evidence in the knowledge transfer scenario, where student models demonstrate improved performance after receiving task-specific knowledge from the teacher model, as shown in Tables 1 and 2.

* **Process and Efficiency**: Our approach differs from standard knowledge distillation, which typically requires fine-tuning or the direct involvement of the teacher model in student model training — a computationally intensive process. In contrast, our parametric knowledge transfer involves extracting task-specific parameters from the vanilla teacher model and integrating them into the student model. This method, requiring only 32 inferences from the teacher model, offers a significant efficiency advantage, especially in the context of LLMs.

Despite these differences, we recognize the relevance of distillation methods in the knowledge transfer scenario and have conducted comparative experiments using LLaMA-1 models (13B to 7B and 30B to 7B). Here KD refers to vanilla knowledge distillation, while SeqKD is sequence-level knowledge distillation [1].

| Models          | GSM    | MMLU   | Super NI | AlpacaFarm | Average |
|-----------------|--------|--------|----------|------------|---------|
| Vanilla 7B      | 4.70   | 32.10  | 5.55     | -          | -       |
| Vanilla 13B     | 4.93   | 43.50  | 7.78     | -          | -       |
| 7B-LoRA         | 17.26  | 43.43  | 40.49    | 9.07       | 27.56   |
| 13B to 7B (KD)   | 17.69  | 43.57  | 42.08    | 9.32       | 28.17   |
| 13B to 7B (SeqKD)| 17.86  | 43.33  | 41.91    | 9.36       | 28.12   |
| 13B to 7B (Ours) | **18.73**  | 44.03  | 42.37    | 9.28       | 28.60   |
| 30B to 7B (KD)   | 17.81  | 44.10  | 41.96    | 9.48       | 28.34   |
| 30B to 7B (SeqKD)| 17.99  | 43.97  | 42.40    | **9.61**       | 28.49   |
| 30B to 7B (Ours) | 18.63  | **45.20**  | **43.08**    | 9.40       | **29.08**  |

* All models are fine-tuned with LoRA, using identical training data and hyperparameters.
* In this paper, we attempt to explore the evidence that knowledge in static parameters can be transferred between different LLMs, and knowledge transfer is the scenario in which we find and provide empirical evidence. Our focus is not on proposing to find better methods in this scenario.

**References**:

[1] Kim et al.  Sequence-level knowledge distillation. EMNLP 2016.

Reviewer comment (Novelty):
Given the rapid evolution of LLMs, do you foresee the long-term viability and adaptability of the proposed knowledge transfer approach?

Author response:
> **Q2**: Given the rapid evolution of LLMs, do you foresee the long-term viability and adaptability of the proposed knowledge transfer approach?

As the landscape of LLMs continues to rapidly evolve, impacting model structures, scales, and the encoding of implicit knowledge, we believe the long-term viability and adaptability of our proposed knowledge transfer approach for several key aspects:

* **Efficiency and Practicality**: The process of transferring knowledge using sub-matrix extraction and the LoRA-based approach is efficient, requiring only a small number of inferences from the teacher model (32 in our case). As LLMs grow in size and complexity, this kind of efficiency becomes ever more crucial.

* **Scalability with Model Sizes**: Our method has demonstrated effectiveness in transferring knowledge between models of varying sizes, as shown in our experiments with different scales of LLaMA models (7B, 13B, 30B, and the newly added 70B). Moreover, as the teacher model's scale increases, our approach shows improved performance in knowledge transfer, aligning well with the ongoing trend of scaling up LLMs.

* **Architecture Flexibility**: Our approach, grounded in the concept of parametric knowledge transfer, is not tightly bound to any specific model architecture. The underlying principles are adaptable to various forms of LLMs. As we demonstrated in our experiments with LLaMA models, our approach can handle different scales, even with different numbers of layers and dimensions or architectural differences (newly added LLaMA-2 70B to 7B). This flexibility suggests the method's potential adaptability to future LLM architectures.

* **Methodological Flexibility**: Centered around the idea of intrinsic dimension and utilizing techniques like LoRA, our approach can be adjusted or expanded to incorporate other methods that refine matrix intrinsic dimensions. This adaptability makes our framework suitable for integration with future developments in LLM methodologies.

* **Potential for Further Research**: Our research presents empirical evidence of the transferability of implicit knowledge within model parameters across various LLMs. This opens up avenues for further research, especially as LLMs increase in size, making fine-tuning or updating model parameters more challenging. Exploring how to better utilize knowledge within static parameters may become increasingly practical and significant.

In conclusion, despite the rapid evolution of LLMs presenting new challenges, the core principles of our parametric knowledge transfer approach have the potential to remain applicable and adaptable in the long run.

Reviewer comment (Writing):
The technical details (Section 3) are succintly written but could be written in a more clear and understandable way (e.g. summarize how standard LoRA is modified in knowledge injection). Certain details are explained in a more accessible manner in e.g. 4.3 but should be present in the main technical spec.

Author response:
Thank you for your valuable feedback, especially regarding the clarity of our technical writing and the legend of Figure 3!

In response to your suggestions, we have revised the sections on sub-matrix extraction (Section 3.2) and knowledge injection using LoRA (Section 3.3) within the methodology part of our paper. Additionally, we have redesigned Figure 3 and updated its corresponding explanation in Section 4.3 of the analysis section. These modifications are highlighted in blue in our revised manuscript for your convenience.

Please let us know if there are still parts that you find challenging to comprehend. We are committed to enhancing our presentation to ensure it is accessible and clear to all readers.

Reviewer comment (Presentation):
The diagrams can be more clearly framed with larger font and clear colors. In particular, the legend for Figure 3 is very difficult to read.

Author response:
Thank you for your valuable feedback, especially regarding the clarity of our technical writing and the legend of Figure 3!

In response to your suggestions, we have revised the sections on sub-matrix extraction (Section 3.2) and knowledge injection using LoRA (Section 3.3) within the methodology part of our paper. Additionally, we have redesigned Figure 3 and updated its corresponding explanation in Section 4.3 of the analysis section. These modifications are highlighted in blue in our revised manuscript for your convenience.

Please let us know if there are still parts that you find challenging to comprehend. We are committed to enhancing our presentation to ensure it is accessible and clear to all readers.

Reviewer comment (Reproducibility):
Can the authors provide more detail about 3.2 - how exactly the submatrix (with highest cumulative sensitivity) is extracted from the larger-parameter teacher model layer to create a mapping matrix in the student model's parameter set? And why in particular extracting a submatrix (with arbitrary possible location in the larger 2D teacher layer matrix) works?

Author response:
For the implementation of our method, we don't employ a mapping matrix. Instead, our process is as follows:

1. We go through all sub-matrices in the teacher model's parameter matrix that align with the dimensions of the student model. During this, we identify and extract the sub-matrix with the highest total sensitivity scores.
2. This chosen submatrix is then directly decomposed and used to initialize the LoRA module for the student model.

We apologize for any confusion and have revised the descriptions in Sections 3.2 and 3.2 in the updated manuscript to clarify this process.

As for why extracting sub-matrices proves more effective than selecting individual weights or rows and columns, our approach is inspired by research in structured pruning [1-2]. Studies in this area have shown that structurally pruning layers, heads, intermediate dimensions, or blocks in weight matrices is more effective than point-wise pruning. These findings suggest that task-specific knowledge is often more concentrated in complete structures within a network, rather than being dispersed across isolated weights. In our research on parametric knowledge transfer, we've observed similar patterns, affirming the efficacy of extracting and utilizing these integral sub-structures.

**References**:

[1] Lagunas et al. Block Pruning For Faster Transformers. EMNLP 2021.

[2] Xia et al. Structured Pruning Learns Compact and Accurate Models. ACL 2022.

Reviewer comment (Experiments):
The experiments are limited. This paper only tests on a few datasets for LLMs, does not discuss which amount of parameters is the best for transferring, and does not compare with other distillation methods, and does not compare with other methods about intrinsic dimension.

Author response:
> **Q3**: Discussion about which amount of parameters is the best for transferring.

Our analysis of the origin of parameters (Section 4.3, Figure 5(b)) involves a discussion of parameter amounts. For example, transferring a combination of the embedding layer, FFN, and attention modules, which constitute 0.608% of the model's parameters, yields the best results. In contrast, transferring a single module, like the FFN alone which accounts for 0.343%  trainable parameters, leads to relatively poor performance.

To further investigate the effect of the amount of parameters, we extend the experiments at LLaMA-1 13B to 7B and 30B to 7B by introducing comparisons with different LoRA r (intrinsic rank). The results are the average scores on the four datasets.

| Transfer Module | LoRA r | Param. | 13B to 7B | 30B to 7B |
|-----------------|--------|--------|----------|----------|
| Embedding       | 128    | 0.137% | 27.33    | 27.52    |
| Attention       | 16     | 0.248% | 27.53    | 27.92    |
| FFN             | 16     | 0.343% | 27.87    | 28.23    |
| All             | 4      | 0.152% | 27.93    | 28.37    |
| All             | 8      | 0.304% | 28.17    | 28.48    |
| All             | 16     | 0.608% | 28.22    | **28.63**    |
| All             | 32     | 1.216% | 28.19    | 28.54    |
| All             | 64     | 2.432% | **28.27**    | 28.60    |

* For the transfer involving only the embedding layer, we set LoRA r at 128, ensuring that the number of trainable parameters remains comparable.
* We observe that increasing LoRA r beyond 16 does not significantly enhance the results when transferring all modules. Consequently, we maintain LoRA r at 16 for the experiments presented in this paper.
* Our analysis indicates that the origin of the parameters has a more pronounced impact on the effectiveness of parametric knowledge transfer than the amount of parameters transferred.

> **Q4**: Comparison with distillation methods.

To address your query, it's important to first emphasize that our study's focus on parametric knowledge transfer is fundamentally distinct from traditional distillation methods, characterized by the following differences:

* **Purpose and Focus**: Rather than proposing better distillation methods, our study is focused on exploring the transferability of implicit knowledge embedded in static parameters. While prior research has explored the detectability and editability of parametric knowledge, its transferability remains less explored. Our experiments provide empirical evidence in the knowledge transfer scenario, where student models demonstrate improved performance after receiving task-specific knowledge from the teacher model, as shown in Tables 1 and 2.

* **Process and Efficiency**: Our approach differs from standard knowledge distillation, which typically requires fine-tuning or the direct involvement of the teacher model in student model training — a computationally intensive process. In contrast, our parametric knowledge transfer involves extracting task-specific parameters from the vanilla teacher model and integrating them into the student model. This method, requiring only 32 inferences from the teacher model, offers a significant efficiency advantage, especially in the context of LLMs.

Despite these differences, we recognize the relevance of distillation methods in the knowledge transfer scenario and have conducted comparative experiments using LLaMA-1 models (13B to 7B and 30B to 7B). Here KD refers to vanilla knowledge distillation, while SeqKD is sequence-level knowledge distillation [5].

| Models          | GSM    | MMLU   | Super NI | AlpacaFarm | Average |
|-----------------|--------|--------|----------|------------|---------|
| Vanilla 7B      | 4.70   | 32.10  | 5.55     | -          | -       |
| Vanilla 13B     | 4.93   | 43.50  | 7.78     | -          | -       |
| 7B-LoRA         | 17.26  | 43.43  | 40.49    | 9.07       | 27.56   |
| 13B to 7B (KD)   | 17.69  | 43.57  | 42.08    | 9.32       | 28.17   |
| 13B to 7B (SeqKD)| 17.86  | 43.33  | 41.91    | 9.36       | 28.12   |
| 13B to 7B (Ours) | **18.73**  | 44.03  | 42.37    | 9.28       | 28.60   |
| 30B to 7B (KD)   | 17.81  | 44.10  | 41.96    | 9.48       | 28.34   |
| 30B to 7B (SeqKD)| 17.99  | 43.97  | 42.40    | **9.61**       | 28.49   |
| 30B to 7B (Ours) | 18.63  | **45.20**  | **43.08**    | 9.40       | **29.08**  |

* All models are fine-tuned with LoRA, using identical training data and hyperparameters.
* In this paper, we attempt to explore the evidence that knowledge in static parameters can be transferred between different LLMs, and knowledge transfer is the scenario in which we find and provide empirical evidence. Our focus is not on proposing to find better methods in this scenario.

> **Q5**:  Discussion about other methods for intrinsic dimension.

Regarding the intrinsic dimension aspect, our focus isn't to develop or compare various methods in this area. Instead, we propose using the concept of intrinsic dimension as a bridge for transferring parametric knowledge. We instantiate our framework with LoRA, a standard representative in this domain. The process involves decomposing the teacher model's parameter matrix, which then serves as the initialization for the student model's LoRA module, aiding in the knowledge injection.

Notably, the proposed workflow is designed to be adaptable, allowing any method that refines the intrinsic dimension of matrices to be directly incorporated into our existing framework.

> **Q6**: About “this paper only tests on a few datasets for LLMs”.

We would like to highlight the diversity and comprehensiveness of the 4 datasets we selected. 3 of these are collections of multiple tasks or domains: MMLU includes questions from 57 varied domains, Super NI covers 1,616 different NLP tasks, and AlpacaFarm is a compilation of open-ended conversation sources from five different origins. Given their diversity and extensive use in evaluating LLMs, we believe that the consistent improvements demonstrated by our method across these varied categories of benchmarks are a strong indication of its effectiveness.

Reviewer comment (Experiments):
The application of this method is limited. The paper does not experiment on different models. Can the knowledge transfer across architectures with this method? This paper also assumes that the small model is already fine pre-trained. Can we transfer to the random-initialized model?

Author response:
> **Q7**: The paper does not experiment on different models. Can the knowledge transfer across architectures with this method?

We answer this question from the following two perspectives:

* **Previous Research on Parametric Knowledge**: Earlier studies have primarily focused on detecting and editing parametric knowledge within a single pre-trained language model. When it comes to merging models, the common practice has been to use two identical models. As a result, the potential for transferring parametric knowledge between models with different layers, dimensions, or overall architectures remains largely unexplored.

* **Our Experimental Models**: In our experiments, we use LLaMA-1 and LLaMA-2 models, which differ in their pre-training. The LLaMA models come in various scales (7B, 13B, 30B, 70B), each with a different number of layers and dimensions. In addition, the LLaMA-2 70B model has a unique attention module. Despite these significant differences, our experiments successfully demonstrate the transferability of parametric knowledge across various models. Moreover, our supplementary experiments transferring knowledge from LLaMA-2 70B to 7B (see Q1) confirm that our method remains effective even when attention architectures differ. Given that most Transformer variants share the same architecture in terms of FFNs and embedding layers, our method appears viable for transferring parametric knowledge across these models.

> **Q8**: This paper also assumes that the small model is already fine pre-trained. Can we transfer to the random-initialized model?

Recent research on knowledge transfer in the context of PLMs and LLMs typically assumes that student models are fine pre-trained. This is primarily due to two factors:

1. From a computational perspective, the cost of pre-training from scratch is prohibitive.
2. For practical applications, the majority of task-specific models are built upon and fine-tuned using existing PLMs or LLMs, leading to a research focus on the fine-tuning stage.

However, we strongly agree that the study on the random-initialized models has promising applications, and we discuss this direction of research in our related work (Section 3.2), termed knowledge inheritance. This concept involves larger models receiving parameters from smaller models to accelerate the pre-training process. This research direction contrasts with the focus of our paper, as it emphasizes a different phase of training (pre-training rather than fine-tuning) and a reverse direction of knowledge flow (from smaller to larger models, as opposed to our large-to-small transfer).

Thank you once again for your valuable feedback and comments! They have been crucial in enhancing the comprehensiveness of our experiments and enriching our study with deeper insights and more empirical explorations in the field of parametric knowledge transfer. If there are any further questions or aspects you feel remain unaddressed, we are more than willing to provide additional information and clarifications as needed.

Reviewer comment (Experiments):
I would like to see results from llama2 70b to 7b.

Author response:
> **Q1**: I would like to see results from llama2 70b to 7b.

We add the following experiment based on your suggestion, where for the GSM and MMLU datasets we present the 0-shot results and the metric used for Super NI is ROUGE-L.

| Models      | GSM    | MMLU   | Super NI | AlpacaFarm | Average |
|-------------|--------|--------|----------|------------|---------|
| Vanilla 7B  | 3.34   | 41.70  | 4.68     | -          | -       |
| Vanilla 13B | 6.52   | 52.10  | 4.84     | -          | -       |
| 7B-LoRA     | 23.38  | 47.77  | 41.25    | 20.50      | 33.23   |
| &nbsp;&nbsp;&nbsp;&nbsp;+13B Param.  | 25.30  | 49.37  | 42.98    | **24.64**      | 35.57   |
| &nbsp;&nbsp;&nbsp;&nbsp;+70B Param. | **26.16**  | **49.60**  | **43.65**    | 24.27      | **35.92**   |

* Due to the differences in attention architectures between LLaMA-2 70B and 7B models (with the 70B employing grouped-query attention and the 7B using multi-head attention), we restrict the parametric knowledge transfer from 70B to 7B to the FFN and embedding layers, which account for 0.36% trainable parameters. In contrast, in the transfer from the 13B to 7B model, we also include the attention module, increasing the percentage of trainable parameters to 0.61%.
* Nevertheless, the transfer from the 70B to the 7B model demonstrates greater performance gains than transferring from 13B to 7B. This implies that our parametric knowledge transfer approach becomes increasingly effective as the teacher model scales up, even in the presence of architectural differences beyond the number of layers and dimensions.