Paper: Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization (ICLR-2024)

Reviewer comment (Theory):
The ability to switch different tasks efficiently would be lost in BOFT due to the fact that BOFT is based on multiplication whereas LoRA is based on addition.

Author response:
**A1**: Thanks for the question! The reviewer may have some misunderstanding on BOFT. It is true that BOFT has a totally different motivation with LoRA, and therefore, our weights are multiplied with the pretrained weight matrix, rather than being added to the pretrained weight matrix. However, from a practical perspective, BOFT still has all the benefits that LoRA possesses, including switching tasks efficiently. During finetuning, only the BOFT orthogonal matrices are updated and can be stored separately. We can also perform the “merge” and “unmerge” operations, similar to LoRA. In LoRA, the “merge” operation is to add the delta weight and the “unmerge”  operation is to subtract the delta weight. Equivalently, “merging” BOFT weight will be matrix multiplying with the original weight, while the “unmerge”  operation will be matrix multiplying with the inverse of the BOFT orthogonal matrix. We note that the inverse of an orthogonal matrix can be easily obtained by its transpose, making the task switching very efficient.

Reviewer comment (Experiments):
For the MMLU dataset, the performance of Llama 2 13B and/or Llama 2 70B should be given, because PEFT methods are designed for fine-tuning large language models. The performance comparison of Llama 2 7B seems not to be enough.

Author response:
**A2**: Thanks for the great suggestion. We agree with the reviewer that the results of finetuning larger models can better showcase our advantages, and we did exactly what the reviewer suggested. However, due to the limitation of our compute and rebuttal time, we are sorry that it is impossible for us to finetune with Llama2-70B. 

To address the reviewer’s concerns, we finetune Llama2-13B on the Stanford Alpaca dataset and evaluate it on the MMLU dataset. The general settings (including hyperparameters and # params) of Llama2-13B are exactly the same as that of Llama2-7B. The performance gain of BOFT for finetuning Llama2-13B is still very obvision, similar to the case of finetuning Llama2-7B. The results are given as follows:

**Updated MMLU results for Llama 13B**:

|     Method     |#params| MMLU  | (5-shot) |     |      |       | MMLU   |(0-shot)|     |       |       |
|----------------|-------|-------|-------|-------|-------|-------|--------|-------|-------|-------|-------|
|                |       |Humans | STEM  | Social| Other | Avg.  | Humans | STEM  | Social| Other | Avg.|
|Llama 13B   **new**    |       | 52.5 | 43.5 | 62.7 | 60.4 | 54.6 | 48.1 | 42.5 | 60.7 | 59.5 | 52.3 | 
|LoRA (r=16)  **new**    | 0.06% | 52.9 | 43.4 | 63.4 | 61.3 | 55.1 | 49.1 | **43.2** | 62.5 | 59.9 | 53.3 | 
|OFT (b=16)   **new**    | 0.06% | 52.4 | 43.3 | 63.9 | 61.8 | 55.1 | 49.0 | 42.9 | 62.8 | **61.2** | 53.4 |
|BOFT (m=2, b=8)  **new** | 0.06% | **53.2** | **44.1** | **64.1** | **62.2** | **55.7** | **50.1** | 43.0 | **63.0** | 60.8 | **53.9** |

which can be summarized as follows:

|     Method     |#params| MMLU (5-shot) | MMLU (0-shot) |
|----------------|-------|---------------|---------------|
|                |       |  Avg.  |  Avg.|
|Llama 13B   **new**     |       |  54.6 | 52.3 | 
|LoRA (r=16)  **new**    | 0.06% |  55.1 | 53.3 | 
|OFT (b=16)    **new**   | 0.06% | 55.1 | 53.4 |
|BOFT (m=2, b=8) **new**  | 0.06% | **55.7** | **53.9** |

Reviewer comment (Experiments):
The ablation study of $b$ and $m$ of BOFT seems to be necessary because all different $b$ and $m$ of BOFT are chosen in Table 2, 4, and 5, which implies that BOFT seems not to be practical compared to LoRA.

Author response:
**A3**: Thanks for the question! In fact, our hyperparameters are originally chosen to match the number of trainable parameters of the baseline methods, rather than being optimized for the performance. To demonstrate that the hyperparameters of BOFT do not need to be tuned, we add two types of experiments to address the reviewer’s concern. 

First, we follow the reviewer’s suggestion and conduct a comprehensive ablation study. We conduct an ablation study to demonstrate the impact of hyperparameters on OFT and BOFT given similar parameters. We conduct experiments in the landmark-to-face task and follow the original setting of OFT to calculate the landmark reproject error for the number of blocks (r=2 and r=16), and choose the BOFT to have a similar number of parameters. We see that BOFT consistently outperforms OFT and LoRA. **More significantly, BOFT demonstrates a huge advantage over OFT in the 20M case, showing that the improvement over OFT is not incremental.**  Please also refer to Figure 8 in the main paper for more ablation study on BOFT.

**Ablation study comparing OFT and BOFT given similar number of parameters**:

| LoRA (#params) | Error (LoRA) |  OFT (#params)   | Error (OFT) | BOFT (#params) | Error (BOFT) | 
|:----------------:|:-----------:|:--------------:|:------------:|:---------------:|:------------:|
|    2.52 M (r=16) |    8.878 |  2.71 M (r=2)    |   8.876     |     2.66 M (r=32, m=2)    |  **8.070**      |
 |  20.17 M (r=128) |   8.038   |   20.89 M (r=16)  |   6.407     |     20.76 M (r=8, m=4)    |  **5.667**   |

**More ablation study for BOFT** (with full dense BOFT orthogonal matrix): 

The following hyperparameter settings are selected such that the BOFT produces a full dense orthogonal matrix, so the only free hyperparameter is $e$. When $r=8,16,32$, $m$ has to be $4,5,6$, respectively, such that the BOFT orthogonal matrix is a dense one.

| Method         | #Params | Error |
|----------------|:-------:|:-----:|
|BOFT (r=32, m=6)|  7.69 M | 6.731 |
|BOFT (r=16, m=5)| 12.93 M | 6.387 |
|BOFT (r=8, m=4) | 20.76 M | 5.667 |

Second, we further address the reviewer’s concern with another experiment. Since we use BOFT with $b=8, m=2$ for all the NLP tasks, we adopt the same set of hyperparameter ($b=8$ and $m=2$) for all the vision experiments to see how BOFT performs. We experiment BOFT (m=2, b=8) and OFT (b=16) on both SAM and DINOv2 (VTAB-1K) to be consistent with previous experiments in NLP. From the results, we can see that BOFT outperforms OFT and LoRA across different tasks, given a similar (less) number of parameters. We compare the new results along with the results from the paper in the following table and find that BOFT with this consistent hyperparameter setting achieves even better performance in all the vision tasks.

**Updated VTAB-1K results (with BOFT (m=2, b=8) and OFT (b=16))**:

| Method  | #params |  1  | 2  |  3  |  4  |  5  |  6  |  7  |  8  |  9  | 10  | 11  | 12  | 13  | 14  | 15  | 16  | 17  | 18  | 19  | Avg  |
|------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|
|OFT (b=16, **paper**)|2.10M|77.7|91.9|80.1|**99.7**|94.7|92.9|59.3|88.4|96.4|91.5|77.2|81.0|64.7|60.5|84.0|92.2|61.1|34.8|**40.3**|77.3|
|BOFT (b=4, m=4, **paper**)|1.77M|78.2|91.4|79.6|**99.7**|94.9|92.8|59.4|88.1|96.4|**91.6**|76.2|81.9|**65.4**|60.0|84.5|92.9|61.3|**37.1**|39.3|77.4|
|BOFT (b=6, m=2, **paper**)|1.11M|**78.3**|91.5|79.9|**99.7**|**95.0**|92.0|**60.2**|88.2|96.5|91.4|77.2|80.5|64.1|**61.4**|**85.0**|91.6|60.8|34.0|38.5|77.1|
|BOFT (b=8, m=2)|1.99M|78.1|**92.5**|**80.6**|**99.7**|**95.0**|**93.0**|59.9|**88.9**|**96.6**|**91.6**|**77.3**|**84.5**|64.9|**61.4**|84.1|**93.9**|**62.0**| 36.2 | 40.0 |**77.9**|

which can be summarized as follows:

| Method  | #params |  Avg  |
|------|-----|-----|
|BOFT (b=4, m=4, **paper**)|1.77M|77.4|
|BOFT (b=6, m=2, **paper**)|1.11M|77.1|
|OFT (b=16, **paper**)|2.10M|77.3|
|BOFT (b=8, m=2)|1.99M|**77.9**|

Reviewer comment (Experiments):
I just wonder whether or not BOFT can outperform LoRA (r=64) in Table 2 and 3 as well.

Author response:
**A4**: Thanks for the question. We conducted the experiment of LoRA (r=64) in Table 2 and 3. The results are shown below. We can see that for the MMLU and the MATH benchmark, compared to LoRA (r=64) that uses 4 times the number of parameters, BOFT still achieves slightly better results.

**Updated MMLU results (with LoRA (r=64))**:

|        Method           |#params| MMLU  | (5-shot) |     |      |       | MMLU   |(0-shot)|     |       |       |
|-------------------------|-------|-------|-------|-------|-------|-------|--------|-------|-------|-------|-------|
|                         |       |Humans | STEM  | Social| Other | Avg.  | Humans | STEM  | Social| Other | Avg.|
|Llama2 7B **paper**      |       | 43.0 | 36.9 | 51.6 | 52.1 | 45.7 | 38.8 | 33.3 | 46.8 | 45.0 | 40.8 |
|LoRA (r=16) **paper**    | 0.125%| 42.9 | 38.5 | 54.5 | 53.8 | 47.0 | 42.5 | 37.1 | 51.5 | 52.3 | 45.5 |
|LoRA (r=32) **paper**    | 0.25% | 42.9 | 38.7 | 54.6 | 54.7 | 47.3 | 42.5 | 36.7 | 52.8 | 52.7 | 45.9 |
|LoRA (r=64)  **new**   | 0.5%  | 44.0 | 40.1 | 53.5 | 54.3 | 47.7 | 44.0 | 37.3 | 53.2 | 53.1 | **46.7** |
|OFT (b=16) **paper**     | 0.13% | 44.0 | 38.9 | 54.2 | 54.3 | 47.5 | 44.0 | 36.7 | 52.9 | 52.0 | 46.2 |
|BOFT (m=2, b=8) **paper**| 0.12% | 44.5 | 39.0 | 54.4 | 55.1 | **47.9** | 44.3 | 37.4 | 53.1 | 52.8 | **46.7** |

which can be summarized as follows:

|        Method           |#params| MMLU  (5-shot) | MMLU  (0-shot) |
|-------------------------|-------|----------------|----------------|
|                         |       |     Avg.       |      Avg.      |
|Llama2 7B **paper**      |       |  45.7 |  40.8 |
|LoRA (r=16) **paper**    | 0.125%|  47.0 |  45.5 |
|LoRA (r=32) **paper**    | 0.25% |  47.3 |  45.9 |
|LoRA (r=64) **new**            | 0.5%  |  47.7 |  **46.7** |
|OFT (b=16) **paper**     | 0.13% |  47.5 |  46.2 |
|BOFT (m=2, b=8) **paper**| 0.12% |  **47.9** | **46.7** |


**Updated GSM8K and MATH results (with LoRA (r=64))**:

|          Method         | # Param | GSM8K  | MATH |
|-------------------------|---------|--------|------|
|Llama2 7B **paper**      |         | 14.6   | 2.5 |
|LoRA (r=32) **paper**    |  0.25%  | 50.2   | 7.8 |
|LoRA (r=64)   **new**           |  0.5%   |**50.6**| 8.5 | 
|OFT (b=16) **paper**     |  0.13%  | 50.1   | 8.4 |
|BOFT (m=2, b=8) **paper**|  0.12%  | **50.6**   |**8.6**|

Reviewer comment (Novelty):
The information transmission framework listed as the first contribution of the paper is described as novel but I would argue this is connectivism. A similar figure can be found in every book on deep learning, for example opening Goodfellow's [Deep Learning book][goodfellow] there is a figure on page 170 that describes a bipartite connectivist diagram like this. It's possible I am missing something though.

Author response:
**A1**: Thanks for referring us to the figure! In our hard copy Deep Learning book, we didn’t find the figure on Page 170, but we searched around this page and found some connection graph for describing MLPs. We agree with the reviewer that our information transmission graph is a form of connectivism, but we would like to emphasize that our information transmission framework is distinct from such MLP connection graphs in the following aspects (please do correct us if the reviewer means a different figure in the deep learning book).

First, the purpose of our information transmission graph is to provide a graphical understanding of the factorization of a dense orthogonal matrix (such that the butterfly network can be naturally motivated in this framework), while the MLP connection graphs only serves the purpose of visualizing the connection weights. Under the information transmission framework, one can easily come up with a way to construct a new sparse matrix factorization (e.g., Figure 2 is an alternative factorization other than the butterfly structure). We are not claiming the information transmission graph itself as a novel concept, but instead, its usage to enable the study of the parameter efficiency issue in orthogonal finetuning is novel.

Second, our information transmission graph is used to underpin the study of parameter efficiency of orthogonal finetuning, while the figure in the deep learning book is used to visualize the connectivity between different layers.

Third, due to the orthogonality requirement, our information transmission framework has a few corresponding constraints, such as dense connectivity and minimal free edges (the number of edges between adjacent layers is at least the number of nodes). This is quite different from general connectivism for MLP.

Last, while the figure in the deep learning book is a general connectivity graph,  we would like to re-emphasize that we don’t claim the novelty of our information transmission perspective in a general sense. Instead, we consider the information transmission framework as novel when studying the parameter efficiency issue in orthogonal finetuning.

Reviewer comment (Presentation):
The entire premise of the paper could also be introduced early in that case by putting an illustration of the butterfly parameterisation along with Figure 1 and referencing it on the first page.

Author response:
**A2**: Great suggestion! We will follow the reviewer’s suggestion and re-arrange the structure of the paper for better clarity in a later revision.

Reviewer comment (Presentation):
Table 5 is in the wrong place, it sits next to a description of the VTAB-1K task, but presents results on the next task.

Author response:
**A3**: Great suggestion! We will follow the reviewer’s suggestion and fix this issue in a later revision.

Reviewer comment (Novelty):
The paper extends OFT but the improvement appears to be incremental, with minor performance gains on competing methods.

Author response:
**A4**: Thanks for the question. In fact, BOFT consistently provides better parameter efficiency than both OFT and LoRA in all the tasks, including vision, NLP and text-to-image generation. Such a consistent improvement is actually very important, since it demonstrates the effectiveness of BOFT as a generally useful finetuning method. Besides, the performance gain is also dependent on the task itself. Moreover, the performance gain of BOFT over OFT and LoRA is actually significant in some tasks. We kindly point the reviewer to Table 6 and Figure 8, underlining the faster convergence and better performance of BOFT over OFT when finetuning Stable Diffusion. In the NLP and Llama tasks (Figure 1, 2, 3), when comparing BOFT and OFT given a similar number of parameters (BOFT (m=2, b=8) and OFT (b=16)), BOFT consistently outperforms OFT while using fewer parameters.

In general, the importance of BOFT lies in its generalization of OFT. OFT has no principled way to be parameter-efficient, and the block diagonality to save parameters introduces additional assumptions which are shown to limit the performance in the Stable Diffusion experiment (see later). BOFT provides a smooth selection of the number of trainable parameters (see Figure 5 and Figure 19 of the updated manuscript). In contrast, changing the number of diagonal blocks in the original OFT provides a very limited number of choices for the trainable parameters. For example, for an orthogonal matrix of size $512\times 512$ in OFT, we list the number of trainable parameters by varying all the valid number of diagonal blocks $1\leq r \leq 16$:

| | r=1 |  r=2  | r=4  | r=8  | r=16  |
|---|:---:|:---:|:---:|:---:|:---:|
|#params|130.8K|65.3K|32.5K|16.1K|7.9K|

Between two adjacent $r$, the number of parameters can vary quite a lot, which limits its flexibility. BOFT can perfectly address this problem by providing a smooth interpolation of the trainable parameter number between adjacent values in $r$.

To better address the reviewer’s concern, we added additional experiments to SAM and VTAB-1K for BOFT (m=2, b=8) and OFT (b=16) such that the hyperparameters are consistent with previous experiments in NLP and Llama. This is to further highlight that the improvement of BOFT over OFT is consistent and the hyperparameter tuning is generally not needed for BOFT. We compare the new results with the results from the submission in the following tables. The result obtained by BOFT(m=2, b=8) is even better than the originally reported BOFT. It shows that our improvement is not obtained by extensive hyperparameter tuning. In fact, we originally changed the hyperparameter of BOFT in the submission simply to match the number of trainable parameters of LoRA and OFT, rather than tuning its performance.

**Updated VTAB-1K results (with BOFT (m=2, b=8) and OFT (b=16))**:
| Method  | #params |  1  | 2  |  3  |  4  |  5  |  6  |  7  |  8  |  9  | 10  | 11  | 12  | 13  | 14  | 15  | 16  | 17  | 18  | 19  | Avg  |
|------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|
|OFT (b=16, **paper**)|2.10M|77.7|91.9|80.1|**99.7**|94.7|92.9|59.3|88.4|96.4|91.5|77.2|81.0|64.7|60.5|84.0|92.2|61.1|34.8|**40.3**|77.3|
|BOFT (b=4, m=4, **paper**)|1.77M|78.2|91.4|79.6|**99.7**|94.9|92.8|59.4|88.1|96.4|**91.6**|76.2|81.9|**65.4**|60.0|84.5|92.9|61.3|**37.1**|39.3|77.4|
|BOFT (b=6, m=2, **paper**)|1.11M|**78.3**|91.5|79.9|**99.7**|**95.0**|92.0|**60.2**|88.2|96.5|91.4|77.2|80.5|64.1|**61.4**|**85.0**|91.6|60.8|34.0|38.5|77.1|
|BOFT (b=8, m=2, **new**)|1.99M|78.1|**92.5**|**80.6**|**99.7**|**95.0**|**93.0**|59.9|**88.9**|**96.6**|**91.6**|**77.3**|**84.5**|64.9|**61.4**|84.1|**93.9**|**62.0**| 36.2 | 40.0 |**77.9**|

which can be summarized as follows:

| Method  | #params |  Avg  |
|------|-----|-----|
|BOFT (b=4, m=4, **paper**)|1.77M|77.4|
|BOFT (b=6, m=2, **paper**)|1.11M|77.1|
|OFT (b=16, **paper**)|2.10M|77.3|
|BOFT (b=8, m=2, **new**)|1.99M|**77.9**|

Reviewer comment (Experiments):
The authors mention the spectral benefits of this method over LoRA but do not explore this in detail in experiments, looking only at benchmark results.

Author response:
More importantly, the **spectral property** (i.e., Orthogonal finetuning will not change the spectral norm of the pretrained weight matrices) aims to provide justifications for why BOFT leads to faster convergence and better training stability than LoRA (especially in the Stable Diffusion experiments). To further address the reviewer’s concern, we additionally conduct a backdoor attack experiment to see whether such a nice spectral property leads to a more robust model. The robustness experiment is guided by the insight that keeping the spectral norm / layer-wise Lipschitz constant unchanged during finetuning will be more resistant to attacks since it will not significantly change the model behavior. We generally follow the detailed experimental settings as [Data Poisoning Attacks Against Multimodal Encoders, ICML 2023]:

**Base Model**: CLIP "openai/clip-vit-base-patch32"
**Dataset**: MS COCO
**Training data (poisoned)**: aims to map texts in one class (original class: dog) to images in another class (target class: truck). The ratio is 0.24 (following the same settings as [Data Poisoning Attacks Against Multimodal Encoders, ICML 2023])
**Evaluation Metric**: Hit@10 - the fraction of image samples for which the target images are included in the first 10 entities of the rank list for the image retrieval task. A higher poison hit@10 indicates a more successful attack.

|Method|Hit@10|
|---|:---:|
|Full Finetuning (epoch 10)|50.8|
|LoRA (epoch 10)|54.8|
|BOFT (epoch 10)|**29.6**|

We can see that BOFT yields a much lower attack success rate, meaning that BOFT leads to a more robust model. We note that this is beyond the scope of our paper, but this will be a very interesting direction to explore in the future. We sincerely thank the reviewer for inspiring us to conduct this experiment!