Update README.md

README.md

@@ -3,142 +3,24 @@ license: apache-2.0
 datasets:
 - yahma/alpaca-cleaned
 ---
-# MixLoRA: Resource-Efficient Model with Mix-of-Experts Architecture for Enhanced LoRA Performance
+# MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts
 
 <div align="left"><img src="https://huggingface.co/scu-kdde/alpaca-mixlora-7b/resolve/main/MixLoRA.png" width=60%"></div>
 
-GitHub: https://github.com/mikecovlee/mlora
+GitHub: [mikecovlee/mlora](https://github.com/mikecovlee/mlora)
 
-The fundamental concept of MixLoRA is based on a pre-trained model with all parameters frozen, such as LLaMA-7B. It involves training multiple LoRA expert modules on top of its fully connected layer (FFN). Simultaneously, a routing layer (Gate Linear) is trained, creating a more powerful Mixture of Experts (MoE) language model. Theoretically, this approach allows achieving performance similar to existing MoE models but with fewer resources.
+Large Language Models (LLMs) have showcased exceptional performance across a wide array of Natural Language Processing (NLP) tasks. Fine-tuning techniques are commonly utilized to tailor pre-trained models to specific applications. While methods like LoRA have effectively tackled GPU memory constraints during fine-tuning, their applicability is often restricted. On the other hand, Mix-of-Expert (MoE) models, such as Mixtral 8x7B, demonstrate remarkable performance while maintaining a reduced parameter count. However, the resource requirements of these models pose challenges, particularly for consumer-grade GPUs.
 
-In addition, MixLoRA also allows simultaneous fine-tuning of the attention layer, contributing to improved fine-tuning outcomes. In experiments, when the attention layer is fine-tuned simultaneously, the MixLoRA model composed of 8 experts exhibits a faster rate of loss reduction compared to the MixLoRA model with 9 experts.
+To address this challenge, we propose MixLoRA, an innovative approach aimed at constructing a resource-efficient sparse MoE model. MixLoRA inserts multiple LoRA-based experts within the feed-forward network block of a frozen pre-trained dense model through fine-tuning, employing a top-k routing strategy. Unlike other LoRA MoE methods, MixLoRA enhances model performance by utilizing independently configurable attention layer LoRA adapters, supporting LoRA and its variants for the construction of experts, and applying auxiliary load balance loss to address the imbalance problem of the router.
 
-MixLoRA exists within m-LoRA in a specific adapter form. Consequently, m-LoRA is capable of simultaneously loading, training, and fine-tuning multiple distinct MixLoRA models. However, it's essential to note that these models must be based on the same pre-trained model.
-
-## MMLU Scores
-
-|Model|Configuration|MMLU Average|STEM|Social Sciences|Humanities|Other|
-|-----------------|---------------------------------|--------|--------|--------|--------|--------|
-|Alpaca-LoRA-7B   |LoRA Rank = 16, QKVO             |  24.2  |  24.1  |**25.0**|  25.2  |  22.7  |
-|Alpaca-MixLoRA-7B|LoRA Rank = 8, Top-2 of 8 Experts|**25.5**|**26.1**|  23.3  |**25.3**|**26.9**|
-
-## Configuration of MixLoRA
-
-Compared with LoRA, MixLoRA have some additional configurations.
-```json
-{
-  "name": "lora_0",
-  "optim": "adamw",
-  "lr": 1e-5,
-  "batch_size": 16,
-  "micro_batch_size": 2,
-  "test_batch_size": 64,
-  "num_epochs": 3,
-  "r": 8,
-  "lora_alpha": 16,
-  "lora_dropout": 0.05,
-  "target_modules": {
-      "q_proj": true,
-      "k_proj": false,
-      "v_proj": true,
-      "o_proj": false,
-      "w1_proj": false,
-      "w2_proj": false,
-      "w3_proj": false
-  },
-  "data": "yahma/alpaca-cleaned",
-  "prompt": "template/alpaca.json",
-  "group_by_length": false,
-  "expand_side": "right"
-}
-```
-This is an example of LoRA training configuration. You can find instructions at [README.md](./README.md).
-
-MixLoRA have two routing strategies: top-k routing (like *Mixtral*) and top-1 switch routing (like *Switch Transformers*), can be configured with `"routing_strategy": "mixtral"` or `"routing_strategy": "switch"`.
-
-**Top-k Routing**
-```
-{
-  ...
-  "routing_strategy": "mixtral",
-  "num_experts": 8,
-  "act_fn": "silu",
-  "top_k": 2,
-  ...
-}
-```
-
-**Top-1 Switch Routing**
-```
-{
-  ...
-  "routing_strategy": "switch",
-  "num_experts": 8,
-  "act_fn": "gelu_new",
-  "expert_capacity": 32,
-  "jitter_noise": 0.1,
-  "ffn_dropout": 0.1,
-  ...
-}
-```
-You can add these items into training configurations to enable the MixLoRA architecture.
-## Create MixLoRA model
-
-Basic command for creating a baseline model on the [Alpaca Cleaned](https://github.com/gururise/AlpacaDataCleaned) dataset:
-```bash
-CUDA_VISIBLE_DEVICES=0 python mlora.py \
-  --base_model yahma/llama-7b-hf \
-  --config ./config/alpaca_mixlora.json \
-  --load_8bit
-```
-Please note that once the MixLoRA model is created, the number of experts in the model cannot be changed.
-
-## Fine-tuning MixLoRA model
-
-The MixLoRA model can also undergo further fine-tuning.
-Basic command for finetuning a model on the [Alpaca Cleaned](https://github.com/gururise/AlpacaDataCleaned) dataset:
-```bash
-CUDA_VISIBLE_DEVICES=0 python mlora.py \
-  --base_model yahma/llama-7b-hf \
-  --config ./config/alpaca_mixlora.json \
-  --load_8bit \
-  --load_adapter
-```
-
-## Evaluate MixLoRA model
-
-Currently, MixLoRA supports evaluation only through the m-LoRA framework.
-```bash
-CUDA_VISIBLE_DEVICES=0 python mlora.py \
-  --base_model yahma/llama-7b-hf \
-  --config ./config/alpaca_mixlora.json \
-  --load_8bit \
-  --inference
-```
-This apporach allows inference multiple MixLoRA and LoRA adapters simultaneously. We also provide a WebUI and an example for inference.
-```bash
-# Run WebUI of Inference
-CUDA_VISIBLE_DEVICES=0 python inference.py \
-  --base_model yahma/llama-7b-hf \
-  --lora_weights scu-kdde/alpaca-mixlora-7b \
-  --template template/alpaca.json \
-  --load_8bit
-
-# Simply Generate
-CUDA_VISIBLE_DEVICES=0 python generate.py \
-  --base_model yahma/llama-7b-hf \
-  --lora_weights scu-kdde/alpaca-mixlora-7b \
-  --template template/alpaca.json \
-  --load_8bit \
-  --instruction "What is m-LoRA?"
-```
+In experiments, MixLoRA achieves commendable performance across all evaluation metrics in both single-task and multi-task learning scenarios. Implemented within the m-LoRA framework, MixLoRA enables parallel fine-tuning, inference, and evaluation of multiple mixture-of-experts models on a single 24GB consumer-grade GPU without quantization, thereby reducing GPU memory consumption by 41\% and latency during the training process by 17\%.
 
 ## Citation
 Please cite the repo if you use the code in this repo.
 ```bibtex
 @misc{alpaca-mixlora-7b,
-  author = {Dengchun, Li and Tingfeng, Lan and Zhengmao, Ye and Lei, Duan and Mingjie, Tang},
-  title = {MixLoRA MoE model based on AlpacaCleaned dataset and LLaMA-7B base model},
+  author = {Dengchun, Li and Yingzi, Ma and Naizheng, Wang and Lei, Duan and Jie, Zuo and Mingjie, Tang},
+  title = {MixLoRA LoRA MoE adapter based on AlpacaCleaned dataset and LLaMA-2-7B base model},
   year = {2024},
   publisher = {HuggingFace Hub},
   howpublished = {\url{https://huggingface.co/scu-kdde/alpaca-mixlora-7b}},