--- library_name: peft license: llama3.1 base_model: meta-llama/Llama-3.1-8B tags: - generated_from_trainer model-index: - name: Llama3.1-8B-QA_CoT-LAW-Instruct-r64 results: [] --- # Llama3.1-8B-QA_CoT-LAW-Instruct-r64 This model is a fine-tuned version of [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.4632 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 3.6e-05 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 100 - num_epochs: 1 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:-----:|:---------------:| | 0.5851 | 0.0089 | 200 | 0.6076 | | 0.5957 | 0.0178 | 400 | 0.5905 | | 0.6387 | 0.0267 | 600 | 0.5838 | | 0.5579 | 0.0356 | 800 | 0.5772 | | 0.5691 | 0.0444 | 1000 | 0.5704 | | 0.5905 | 0.0533 | 1200 | 0.5665 | | 0.5664 | 0.0622 | 1400 | 0.5619 | | 0.5537 | 0.0711 | 1600 | 0.5583 | | 0.5937 | 0.08 | 1800 | 0.5558 | | 0.5942 | 0.0889 | 2000 | 0.5535 | | 0.4662 | 0.0978 | 2200 | 0.5509 | | 0.5291 | 0.1067 | 2400 | 0.5480 | | 0.5892 | 0.1156 | 2600 | 0.5467 | | 0.535 | 0.1244 | 2800 | 0.5433 | | 0.5375 | 0.1333 | 3000 | 0.5417 | | 0.5217 | 0.1422 | 3200 | 0.5401 | | 0.5701 | 0.1511 | 3400 | 0.5364 | | 0.5183 | 0.16 | 3600 | 0.5353 | | 0.5149 | 0.1689 | 3800 | 0.5345 | | 0.5166 | 0.1778 | 4000 | 0.5321 | | 0.5322 | 0.1867 | 4200 | 0.5304 | | 0.5882 | 0.1956 | 4400 | 0.5286 | | 0.5571 | 0.2044 | 4600 | 0.5266 | | 0.5265 | 0.2133 | 4800 | 0.5258 | | 0.477 | 0.2222 | 5000 | 0.5224 | | 0.5099 | 0.2311 | 5200 | 0.5220 | | 0.5123 | 0.24 | 5400 | 0.5206 | | 0.4467 | 0.2489 | 5600 | 0.5195 | | 0.5461 | 0.2578 | 5800 | 0.5179 | | 0.4798 | 0.2667 | 6000 | 0.5167 | | 0.5436 | 0.2756 | 6200 | 0.5150 | | 0.5304 | 0.2844 | 6400 | 0.5134 | | 0.4854 | 0.2933 | 6600 | 0.5116 | | 0.5618 | 0.3022 | 6800 | 0.5112 | | 0.537 | 0.3111 | 7000 | 0.5099 | | 0.5352 | 0.32 | 7200 | 0.5088 | | 0.4811 | 0.3289 | 7400 | 0.5082 | | 0.452 | 0.3378 | 7600 | 0.5065 | | 0.5921 | 0.3467 | 7800 | 0.5049 | | 0.5043 | 0.3556 | 8000 | 0.5046 | | 0.5269 | 0.3644 | 8200 | 0.5028 | | 0.5481 | 0.3733 | 8400 | 0.5013 | | 0.5285 | 0.3822 | 8600 | 0.5009 | | 0.5726 | 0.3911 | 8800 | 0.4998 | | 0.5535 | 0.4 | 9000 | 0.4984 | | 0.5185 | 0.4089 | 9200 | 0.4976 | | 0.5432 | 0.4178 | 9400 | 0.4963 | | 0.4337 | 0.4267 | 9600 | 0.4953 | | 0.4643 | 0.4356 | 9800 | 0.4948 | | 0.4715 | 0.4444 | 10000 | 0.4932 | | 0.4945 | 0.4533 | 10200 | 0.4925 | | 0.5382 | 0.4622 | 10400 | 0.4916 | | 0.4542 | 0.4711 | 10600 | 0.4908 | | 0.5242 | 0.48 | 10800 | 0.4900 | | 0.5305 | 0.4889 | 11000 | 0.4883 | | 0.5547 | 0.4978 | 11200 | 0.4877 | | 0.4623 | 0.5067 | 11400 | 0.4870 | | 0.501 | 0.5156 | 11600 | 0.4862 | | 0.5399 | 0.5244 | 11800 | 0.4851 | | 0.4667 | 0.5333 | 12000 | 0.4846 | | 0.5373 | 0.5422 | 12200 | 0.4836 | | 0.4633 | 0.5511 | 12400 | 0.4827 | | 0.473 | 0.56 | 12600 | 0.4820 | | 0.5364 | 0.5689 | 12800 | 0.4812 | | 0.4721 | 0.5778 | 13000 | 0.4805 | | 0.5895 | 0.5867 | 13200 | 0.4797 | | 0.5211 | 0.5956 | 13400 | 0.4787 | | 0.4942 | 0.6044 | 13600 | 0.4780 | | 0.4934 | 0.6133 | 13800 | 0.4773 | | 0.5101 | 0.6222 | 14000 | 0.4767 | | 0.4187 | 0.6311 | 14200 | 0.4761 | | 0.5371 | 0.64 | 14400 | 0.4752 | | 0.5041 | 0.6489 | 14600 | 0.4745 | | 0.4861 | 0.6578 | 14800 | 0.4740 | | 0.4723 | 0.6667 | 15000 | 0.4733 | | 0.4193 | 0.6756 | 15200 | 0.4727 | | 0.5096 | 0.6844 | 15400 | 0.4720 | | 0.4864 | 0.6933 | 15600 | 0.4715 | | 0.479 | 0.7022 | 15800 | 0.4711 | | 0.4329 | 0.7111 | 16000 | 0.4704 | | 0.565 | 0.72 | 16200 | 0.4700 | | 0.5131 | 0.7289 | 16400 | 0.4695 | | 0.4722 | 0.7378 | 16600 | 0.4692 | | 0.5044 | 0.7467 | 16800 | 0.4686 | | 0.4701 | 0.7556 | 17000 | 0.4681 | | 0.4476 | 0.7644 | 17200 | 0.4677 | | 0.4116 | 0.7733 | 17400 | 0.4672 | | 0.4345 | 0.7822 | 17600 | 0.4668 | | 0.469 | 0.7911 | 17800 | 0.4665 | | 0.5491 | 0.8 | 18000 | 0.4661 | | 0.531 | 0.8089 | 18200 | 0.4658 | | 0.4313 | 0.8178 | 18400 | 0.4655 | | 0.5244 | 0.8267 | 18600 | 0.4652 | | 0.4248 | 0.8356 | 18800 | 0.4649 | | 0.5095 | 0.8444 | 19000 | 0.4647 | | 0.4436 | 0.8533 | 19200 | 0.4645 | | 0.5154 | 0.8622 | 19400 | 0.4643 | | 0.5023 | 0.8711 | 19600 | 0.4641 | | 0.5312 | 0.88 | 19800 | 0.4639 | | 0.4934 | 0.8889 | 20000 | 0.4638 | | 0.4962 | 0.8978 | 20200 | 0.4637 | | 0.4964 | 0.9067 | 20400 | 0.4636 | | 0.4966 | 0.9156 | 20600 | 0.4635 | | 0.4303 | 0.9244 | 20800 | 0.4634 | | 0.4531 | 0.9333 | 21000 | 0.4633 | | 0.439 | 0.9422 | 21200 | 0.4633 | | 0.4537 | 0.9511 | 21400 | 0.4633 | | 0.4511 | 0.96 | 21600 | 0.4632 | | 0.4713 | 0.9689 | 21800 | 0.4632 | | 0.5265 | 0.9778 | 22000 | 0.4632 | | 0.4636 | 0.9867 | 22200 | 0.4632 | | 0.4287 | 0.9956 | 22400 | 0.4632 | ### Framework versions - PEFT 0.12.0 - Transformers 4.47.0 - Pytorch 2.5.1+cu124 - Datasets 3.0.0 - Tokenizers 0.21.0