hiyouga commited on
Commit
e65fec5
1 Parent(s): d34a7d4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +35 -2
README.md CHANGED
@@ -12,9 +12,10 @@ tags:
12
  - lora
13
  ---
14
 
15
- An instruction-tuned LoRA model of https://huggingface.co/baichuan-inc/baichuan-7B
16
 
17
- This checkpoint is trained with: https://github.com/hiyouga/LLaMA-Efficient-Tuning
 
18
 
19
  Usage:
20
 
@@ -39,6 +40,38 @@ You could also alternatively launch a CLI demo by using the script in https://gi
39
  python src/cli_demo.py --model_name_or_path hiyouga/baichuan-7b-sft
40
  ```
41
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
  Loss curve on training set:
43
  ![train](training_loss.svg)
44
 
 
12
  - lora
13
  ---
14
 
15
+ A bilingual instruction-tuned LoRA model of https://huggingface.co/baichuan-inc/baichuan-7B
16
 
17
+ - Instruction-following dataset: alpaca, alpaca-zh, codealpaca
18
+ - Training framework: https://github.com/hiyouga/LLaMA-Efficient-Tuning
19
 
20
  Usage:
21
 
 
40
  python src/cli_demo.py --model_name_or_path hiyouga/baichuan-7b-sft
41
  ```
42
 
43
+ ---
44
+
45
+ You could reproduce our results with the following scripts:
46
+
47
+ ```bash
48
+ CUDA_VISIBLE_DEVICES=0 python src/train_sft.py \
49
+ --model_name_or_path baichuan-inc/baichuan-7B \
50
+ --do_train \
51
+ --dataset alpaca_gpt4_en,alpaca_gpt4_zh,codealpaca \
52
+ --finetuning_type lora \
53
+ --lora_rank 16 \
54
+ --lora_target W_pack,o_proj,gate_proj,down_proj,up_proj \
55
+ --output_dir baichuan_lora \
56
+ --overwrite_cache \
57
+ --per_device_train_batch_size 8 \
58
+ --per_device_eval_batch_size 8 \
59
+ --gradient_accumulation_steps 8 \
60
+ --preprocessing_num_workers 16 \
61
+ --lr_scheduler_type cosine \
62
+ --logging_steps 10 \
63
+ --save_steps 100 \
64
+ --eval_steps 100 \
65
+ --learning_rate 5e-5 \
66
+ --max_grad_norm 0.5 \
67
+ --num_train_epochs 2.0 \
68
+ --dev_ratio 0.01 \
69
+ --evaluation_strategy steps \
70
+ --load_best_model_at_end \
71
+ --plot_loss \
72
+ --fp16
73
+ ```
74
+
75
  Loss curve on training set:
76
  ![train](training_loss.svg)
77