Edit model card

lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3_Qwen_Qwen1.5-4B_3e-5_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3 dataset. It achieves the following results on the evaluation set:

  • Loss: 2.2508
  • Accuracy: 0.5852

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 20.0

Training results

Training Loss Epoch Step Validation Loss Accuracy
1.8512 0.9973 187 1.7184 0.6032
1.7174 2.0 375 1.7018 0.6049
1.6805 2.9973 562 1.6938 0.6061
1.6412 4.0 750 1.6925 0.6067
1.5834 4.9973 937 1.7047 0.6062
1.5304 6.0 1125 1.7239 0.6056
1.452 6.9973 1312 1.7508 0.6039
1.3847 8.0 1500 1.7711 0.6028
1.3177 8.9973 1687 1.8049 0.6009
1.2747 10.0 1875 1.8298 0.5998
1.2202 10.9973 2062 1.8814 0.5981
1.1589 12.0 2250 1.9311 0.5959
1.1231 12.9973 2437 1.9429 0.5955
1.0624 14.0 2625 1.9969 0.5933
1.0185 14.9973 2812 2.0319 0.5922
0.9718 16.0 3000 2.0798 0.5903
0.9101 16.9973 3187 2.1396 0.5887
0.8606 18.0 3375 2.1882 0.5870
0.8168 18.9973 3562 2.2291 0.5863
0.777 19.9467 3740 2.2508 0.5852

Framework versions

  • PEFT 0.5.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
1
Unable to determine this model’s pipeline type. Check the docs .

Adapter for

Dataset used to train tyzhu/lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3_Qwen_Qwen1.5-4B_3e-5_lora2

Evaluation results

  • Accuracy on tyzhu/lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3
    self-reported
    0.585