Hugging Face's logo

sqllama
/
alpaca-lora-dono

Model card Files Community
chrisdono commited on
Commit
099d41e
•
1 Parent(s): a005e66

added readme

Browse files
Files changed (1) hide show
  1. README.md +134 -0
README.md ADDED
@@ -0,0 +1,134 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## Setup Notes
2
+
3
+ For this model, a VM with 2 T4 GPUs was used. When comparing to the VM with a single T4 GPU, training was around 1.5x (maybe more) faster.
4
+
5
+ To get the training to work on the 2 GPUs (utilize both GPUS simultaneously), the following command was used to initiate training.
6
+
7
+ WORLD_SIZE=2 CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 --master_port=1234 finetune.py --base_model 'decapoda-research/llama-7b-hf' --data_path 'yahma/alpaca-cleaned' --output_dir './lora-alpaca' --num_epochs 1 --micro_batch_size 8
8
+
9
+ Note 1. Micro batch size was increased from the default 4 to 8. Note that increasing it further is possible based on other training that has been performed. This was a first attempt.
10
+ Note 2. Output directory was initially lora-alpaca and then contents were moved to new folder when initializing git repository.
11
+
12
+
13
+ ## Log
14
+
15
+ (sqltest) chrisdono@deep-learning-duo-t4-3:~/alpaca-lora$ WORLD_SIZE=2 CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 --master_port=1234 finetune.py --base_model 'decapoda-research/llam
16
+ a-7b-hf' --data_path 'yahma/alpaca-cleaned' --output_dir './lora-alpaca' --num_epochs 1 --micro_batch_size 8
17
+ WARNING:torch.distributed.run:
18
+ *****************************************
19
+ Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your appli
20
+ cation as needed.
21
+ *****************************************
22
+
23
+
24
+ ===================================BUG REPORT===================================
25
+ Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
26
+ ================================================================================
27
+ ===================================BUG REPORT===================================
28
+ Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
29
+ ================================================================================
30
+ /opt/conda/envs/sqltest/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /opt/conda/envs/sqltest did not contain libcudart.so as expected! Searching further path
31
+ s...
32
+ warn(msg)
33
+ CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
34
+ CUDA SETUP: Highest compute capability among GPUs detected: 7.5
35
+ /opt/conda/envs/sqltest/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /opt/conda/envs/sqltest did not contain libcudart.so as expected! Searching further path
36
+ s...
37
+ warn(msg)
38
+ CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
39
+ CUDA SETUP: Highest compute capability among GPUs detected: 7.5
40
+ CUDA SETUP: Detected CUDA version 113
41
+ CUDA SETUP: Loading binary /opt/conda/envs/sqltest/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda113.so...
42
+ CUDA SETUP: Detected CUDA version 113
43
+ CUDA SETUP: Loading binary /opt/conda/envs/sqltest/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda113.so...
44
+ Training Alpaca-LoRA model with params:
45
+ base_model: decapoda-research/llama-7b-hf
46
+ data_path: yahma/alpaca-cleaned
47
+ output_dir: ./lora-alpaca
48
+ batch_size: 128
49
+ micro_batch_size: 8
50
+ num_epochs: 1
51
+ learning_rate: 0.0003
52
+ cutoff_len: 256
53
+ val_set_size: 2000
54
+ lora_r: 8
55
+ lora_alpha: 16
56
+ lora_dropout: 0.05
57
+ lora_target_modules: ['q_proj', 'v_proj']
58
+ train_on_inputs: True
59
+ add_eos_token: False
60
+ group_by_length: False
61
+ wandb_project:
62
+ wandb_run_name:
63
+ wandb_watch:
64
+ wandb_log_model:
65
+ resume_from_checkpoint: False
66
+ prompt template: alpaca
67
+
68
+ Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [01:24<00:00, 2.57s/it]
69
+ Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [01:24<00:00, 2.57s/it]
70
+ The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
71
+ The tokenizer class you load from this checkpoint is 'LLaMATokenizer'.
72
+ The class this function is called from is 'LlamaTokenizer'.
73
+ The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
74
+ The tokenizer class you load from this checkpoint is 'LLaMATokenizer'.
75
+ The class this function is called from is 'LlamaTokenizer'.
76
+ Found cached dataset json (/home/chrisdono/.cache/huggingface/datasets/yahma___json/yahma--alpaca-cleaned-5d24553f76c14acc/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e
77
+ 6e)
78
+ 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 13.91it/s]
79
+ trainable params: 4194304 || all params: 6742609920 || trainable%: 0.06220594176090199
80
+ Loading cached split indices for dataset at /home/chrisdono/.cache/huggingface/datasets/yahma___json/yahma--alpaca-cleaned-5d24553f76c14acc/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7
81
+ af1cf934bed8e233e6e/cache-45a7f72cdaee9ff3.arrow and /home/chrisdono/.cache/huggingface/datasets/yahma___json/yahma--alpaca-cleaned-5d24553f76c14acc/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797
82
+ e5a6b2dd7af1cf934bed8e233e6e/cache-c14794386159bdb7.arrow
83
+ Found cached dataset json (/home/chrisdono/.cache/huggingface/datasets/yahma___json/yahma--alpaca-cleaned-5d24553f76c14acc/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e
84
+ 6e)
85
+ 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 330.68it/s]
86
+ trainable params: 4194304 || all params: 6742609920 || trainable%: 0.06220594176090199
87
+ Loading cached split indices for dataset at /home/chrisdono/.cache/huggingface/datasets/yahma___json/yahma--alpaca-cleaned-5d24553f76c14acc/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7
88
+ af1cf934bed8e233e6e/cache-45a7f72cdaee9ff3.arrow and /home/chrisdono/.cache/huggingface/datasets/yahma___json/yahma--alpaca-cleaned-5d24553f76c14acc/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797
89
+ e5a6b2dd7af1cf934bed8e233e6e/cache-c14794386159bdb7.arrow
90
+ {'loss': 1.8867, 'learning_rate': 2.9999999999999997e-05, 'epoch': 0.03}
91
+ {'loss': 1.8339, 'learning_rate': 5.6999999999999996e-05, 'epoch': 0.05}
92
+ {'loss': 1.6664, 'learning_rate': 8.699999999999999e-05, 'epoch': 0.08}
93
+ {'loss': 1.3046, 'learning_rate': 0.000117, 'epoch': 0.1}
94
+ {'loss': 1.115, 'learning_rate': 0.000147, 'epoch': 0.13}
95
+ {'loss': 1.0706, 'learning_rate': 0.00017399999999999997, 'epoch': 0.15}
96
+ {'loss': 1.0269, 'learning_rate': 0.000204, 'epoch': 0.18}
97
+ {'loss': 1.0012, 'learning_rate': 0.000234, 'epoch': 0.21}
98
+ {'loss': 0.9608, 'learning_rate': 0.00026399999999999997, 'epoch': 0.23}
99
+ {'loss': 0.9563, 'learning_rate': 0.000294, 'epoch': 0.26}
100
+ {'loss': 0.9512, 'learning_rate': 0.00029166666666666664, 'epoch': 0.28}
101
+ {'loss': 0.9505, 'learning_rate': 0.00028125, 'epoch': 0.31}
102
+ {'loss': 0.9326, 'learning_rate': 0.0002708333333333333, 'epoch': 0.33}
103
+ {'loss': 0.9229, 'learning_rate': 0.00026041666666666666, 'epoch': 0.36}
104
+ 37%|███████████████████████████████████████████████████████▎ | 145/388 [1:44:04<2:54:41, 43.14s/it]
105
+ {'loss': 0.918, 'learning_rate': 0.00025, 'epoch': 0.39}
106
+ {'loss': 0.9128, 'learning_rate': 0.00023958333333333332, 'epoch': 0.41}
107
+ {'loss': 0.9021, 'learning_rate': 0.00022916666666666664, 'epoch': 0.44}
108
+ {'loss': 0.9115, 'learning_rate': 0.00021874999999999998, 'epoch': 0.46}
109
+ {'loss': 0.8915, 'learning_rate': 0.00020833333333333332, 'epoch': 0.49}
110
+ {'loss': 0.8993, 'learning_rate': 0.00019791666666666663, 'epoch': 0.51}
111
+ {'eval_loss': 0.9055714011192322, 'eval_runtime': 179.4765, 'eval_samples_per_second': 11.144, 'eval_steps_per_second': 0.696, 'epoch': 0.51}
112
+ {'loss': 0.9015, 'learning_rate': 0.00018749999999999998, 'epoch': 0.54}
113
+ {'loss': 0.9008, 'learning_rate': 0.00017708333333333332, 'epoch': 0.57}
114
+ {'loss': 0.8846, 'learning_rate': 0.00016666666666666666, 'epoch': 0.59}
115
+ {'loss': 0.8976, 'learning_rate': 0.00015625, 'epoch': 0.62}
116
+ {'loss': 0.8936, 'learning_rate': 0.00014583333333333332, 'epoch': 0.64}
117
+ {'loss': 0.8883, 'learning_rate': 0.00013541666666666666, 'epoch': 0.67}
118
+ {'loss': 0.8839, 'learning_rate': 0.000125, 'epoch': 0.69}
119
+ {'loss': 0.8922, 'learning_rate': 0.00011458333333333332, 'epoch': 0.72}
120
+ 73%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 285/388 [3:27:30<1:13:45, 42.96s/it]
121
+ {'loss': 0.8916, 'learning_rate': 0.00010416666666666666, 'epoch': 0.75}
122
+ {'loss': 0.8845, 'learning_rate': 9.374999999999999e-05, 'epoch': 0.77}
123
+ {'loss': 0.8804, 'learning_rate': 8.333333333333333e-05, 'epoch': 0.8}
124
+ {'loss': 0.8831, 'learning_rate': 7.291666666666666e-05, 'epoch': 0.82}
125
+ {'loss': 0.8753, 'learning_rate': 6.25e-05, 'epoch': 0.85}
126
+ {'loss': 0.8818, 'learning_rate': 5.208333333333333e-05, 'epoch': 0.87}
127
+ {'loss': 0.8935, 'learning_rate': 4.1666666666666665e-05, 'epoch': 0.9}
128
+ {'loss': 0.8688, 'learning_rate': 3.125e-05, 'epoch': 0.93}
129
+ {'loss': 0.8873, 'learning_rate': 2.0833333333333333e-05, 'epoch': 0.95}
130
+ {'loss': 0.8869, 'learning_rate': 1.0416666666666666e-05, 'epoch': 0.98}
131
+ 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 382/388 [4:36:54<04:16, 42.78s/it]
132
+ 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 388/388 [4:41:13<00:00, 43.06s/it]
133
+ {'train_runtime': 16873.8448, 'train_samples_per_second': 2.949, 'train_steps_per_second': 0.023, 'train_loss': 0.9972113518370795, 'epoch': 1.0}
134
+ 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 388/388 [4:41:13<00:00, 43.49s/it]