Mistral-7B-OpenOrca-lora-merged
This is a test.
This is a regenerated model that combines the base model Mistral-7B-v0.1 with the LoRA model Mistral-7B-OpenOrca-lora.
This LoRA model is extracted from the efficient parameter fine-tuned model (Mistral-7B-OpenOra), and now it needs to be verified whether this LoRA model can achieve comparable performance with the original model.
The final goal is to create a toolkit that can simultaneously load multiple LoRA modules, and automatically switch to the appropriate combination of LoRA modules based on user queries to generate the best answer.
The source code is here
Mistral-7B-OpenOrca
Extract lora model Mistral-7B-OpenOrca-lora from Mistral-7B-OpenOrca;
Merge the base model Mistral-7B-v0.1 with lora model to Mistral-7B-OpenOrca-lora-merged
LLM Evaluation ...
Local Test
ARC_acc_norm (25-shot) | HellaSwag_acc_norm (10-shot) | MMLU_acc (5-shot) | TruthfulQA_mc2 (0-shot) | GSM8K_acc (8-shot) | Open LLM Score | |
---|---|---|---|---|---|---|
Mistral-7B-OpenOrca | 71 | 83 | 61.42 | 45 | 40 | 65.11 |
r=256 | 68 | 84 | 64.28 | 46.953 | 41 | 65.81 |
r=64 | 67 | 84 | 64.26 | 47.32 | 41 | 65.65 |
r=16 | 65 | 83 | 62.84 | 46.95 | 38 | 64.45 |
Open LLM Leaderboard
ARC_acc_norm (25-shot) | HellaSwag_acc_norm (10-shot) | MMLU_acc (5-shot) | TruthfulQA_mc2 (0-shot) | Open LLM Score | |
---|---|---|---|---|---|
Mistral-7B-SlimOrca | 62.54 | 83.86 | 62.77 | 54.23 | 65.85 |
Mistral-7B-OpenOrca | 64.08 | 83.99 | 62.24 | 53.05 | 65.84 |
lm-evaluation-harness
Metric | Mistral-7B-OpenOrca | Mistral-7B-OpenOrca-lora | Mistral-7B-OpenOrca-lora-merged |
---|---|---|---|
ARC | 64.08 | ||
HellaSwag | 83.99 | ||
MMLU | 62.24 | ||
TruthfulQA | 53.05 | ||
Average | 65.84 |
HumanEval
Metric | Mistral-7B-OpenOrca | Mistral-7B-OpenOrca-lora | Mistral-7B-OpenOrca-lora-merged |
---|---|---|---|
humaneval-python | 35.976 |
Training procedure
The following bitsandbytes
quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
Framework versions
- PEFT 0.5.0
- Downloads last month
- 0