lmind_hotpot_train8000_eval7405_v1_recite_qa_meta-llama_Llama-2-7b-hf_3e-5_lora2
This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on the tyzhu/lmind_hotpot_train8000_eval7405_v1_recite_qa dataset. It achieves the following results on the evaluation set:
- Loss: 0.6450
- Accuracy: 0.7555
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 20.0
Training results
Training Loss | Epoch | Step | Validation Loss | Accuracy |
---|---|---|---|---|
1.0799 | 1.0 | 1089 | 1.0260 | 0.7169 |
1.0425 | 2.0 | 2178 | 0.9992 | 0.7192 |
1.0165 | 3.0 | 3267 | 0.9771 | 0.7211 |
0.9772 | 4.0 | 4357 | 0.9578 | 0.7230 |
0.9463 | 5.0 | 5446 | 0.9372 | 0.7247 |
0.9022 | 6.0 | 6535 | 0.9148 | 0.7269 |
0.8621 | 7.0 | 7624 | 0.8946 | 0.7289 |
0.8297 | 8.0 | 8714 | 0.8746 | 0.7310 |
0.7947 | 9.0 | 9803 | 0.8553 | 0.7329 |
0.7436 | 10.0 | 10892 | 0.8319 | 0.7350 |
0.7032 | 11.0 | 11981 | 0.8134 | 0.7371 |
0.6781 | 12.0 | 13071 | 0.7954 | 0.7388 |
0.6416 | 13.0 | 14160 | 0.7733 | 0.7411 |
0.5968 | 14.0 | 15249 | 0.7518 | 0.7434 |
0.5642 | 15.0 | 16338 | 0.7366 | 0.7453 |
0.5442 | 16.0 | 17428 | 0.7135 | 0.7475 |
0.5038 | 17.0 | 18517 | 0.6964 | 0.7494 |
0.4795 | 18.0 | 19606 | 0.6801 | 0.7515 |
0.4508 | 19.0 | 20695 | 0.6601 | 0.7537 |
0.42 | 20.0 | 21780 | 0.6450 | 0.7555 |
Framework versions
- Transformers 4.34.0
- Pytorch 2.1.0+cu121
- Datasets 2.18.0
- Tokenizers 0.14.1
Model tree for tyzhu/lmind_hotpot_train8000_eval7405_v1_recite_qa_meta-llama_Llama-2-7b-hf_3e-5_lora2
Base model
meta-llama/Llama-2-7b-hfDataset used to train tyzhu/lmind_hotpot_train8000_eval7405_v1_recite_qa_meta-llama_Llama-2-7b-hf_3e-5_lora2
Evaluation results
- Accuracy on tyzhu/lmind_hotpot_train8000_eval7405_v1_recite_qaself-reported0.756