Edit model card

llama3-8b-instruct-qlora-medium

This model is a fine-tuned version of LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.7329

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • num_epochs: 30

Training results

Training Loss Epoch Step Validation Loss
2.2884 1.0 105 1.2658
2.0727 2.0 210 1.0205
1.9709 3.0 315 0.9518
1.8768 4.0 420 0.9206
1.7711 5.0 525 0.8761
1.6379 6.0 630 0.8487
1.4834 7.0 735 0.8200
1.3144 8.0 840 0.8076
1.1514 9.0 945 0.7972
1.0148 10.0 1050 0.7865
0.8944 11.0 1155 0.7846
0.7844 12.0 1260 0.7767
0.699 13.0 1365 0.7688
0.6215 14.0 1470 0.7631
0.5602 15.0 1575 0.7584
0.503 16.0 1680 0.7548
0.4597 17.0 1785 0.7514
0.4226 18.0 1890 0.7484
0.3903 19.0 1995 0.7441
0.3646 20.0 2100 0.7390
0.3407 21.0 2205 0.7385
0.3237 22.0 2310 0.7357
0.3108 23.0 2415 0.7343
0.2999 24.0 2520 0.7337
0.2917 25.0 2625 0.7333
0.2868 26.0 2730 0.7324
0.2815 27.0 2835 0.7327
0.28 28.0 2940 0.7315
0.2785 29.0 3045 0.7322
0.2791 30.0 3150 0.7329

Framework versions

  • PEFT 0.10.0
  • Transformers 4.40.0
  • Pytorch 2.2.1+cu121
  • Datasets 2.18.0
  • Tokenizers 0.19.1
Downloads last month
0
Unable to determine this model’s pipeline type. Check the docs .

Adapter for