update training, export model and evaluation part

README.md

@@ -8,7 +8,7 @@ license: llama2
 
 # Dolphin_ORCA_PlatyPus_LLaMA_70b
 
-#### Dataset
+### Dataset
 Here is the list of datasets used:
 * Dolphin
 * Open-Platypus
@@ -21,26 +21,66 @@ Here is the list of datasets used:
 
 <br>
 
-#### license disclaimer:
+### Training FrameWork and Parameters
 
-This model is bound by the license & usage restrictions of the original Llama-2 model. And comes with no warranty or gurantees of any kind.
+#### FrameWork
+https://github.com/hiyouga/LLaMA-Efficient-Tuning
+We add flash_attention_2 and ORCA dataset support, with some minor modifications.
+
+<br>
+
+#### Parameters
+We list some training parameters here:
+| Parameter             | Value       |
+|-----------------------|-------------|
+| Finetune_Type         | QLoRA(NF4)  |
+| LoRA_Rank             | 16          |
+| LoRA_Alpha            | 16          |
+| Batch_Size            | 14          |
+| GPUs                  | 8xA100(80G) |
+| LR_Scheduler          | cosine      |
+| LR                    | 3e-4        |
+| Epoch                 | 1           |
+| DeepSpeed             | ZERO-2      |
+
+<br>
+
+### Model Export
+We tried two methods to fuse the adapter back to the base model:
+* https://github.com/hiyouga/LLaMA-Efficient-Tuning/blob/main/src/export_model.py
+* https://github.com/jondurbin/qlora/blob/main/qmerge.py
+
+Generally, the second will get better ARC(+0.15) and Truthful_QA(+0.3) scores but the other two(MMLU(-0.2) and HelloSwag(-0.2)) seems to degenerate (Just for my model).
 
 <br>
 
-#### Evaluation
+### Evaluation
 
-TODO
+| Metric                | Value |
+|-----------------------|-------|
+| ARC (25-shot)         | 72.27 |
+| HellaSwag (10-shot)   | 87.74 |
+| MMLU (5-shot)         | 70.23 |
+| TruthfulQA (0-shot)   | 63.37 |
+| Avg.                  | 73.40 |
 
 <br>
 
+### license disclaimer:
+
+This model is bound by the license & usage restrictions of the original Llama-2 model. And comes with no warranty or gurantees of any kind.
+
+<br>
 
-#### Limitations & Biases:
+
+
+
+### Limitations & Biases:
 
 Llama 2 and fine-tuned variants are a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Llama 2 and any fine-tuned varient's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Llama 2 variants, developers should perform safety testing and tuning tailored to their specific applications of the model.
 
 Please see the Responsible Use Guide available at https://ai.meta.com/llama/responsible-use-guide/
 
-
 <br>
 
 ### Citiation: