Edit model card

lmind_hotpot_train8000_eval7405_v1_reciteonly_qa_Qwen_Qwen1.5-4B_3e-4_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_hotpot_train8000_eval7405_v1_reciteonly_qa dataset. It achieves the following results on the evaluation set:

  • Loss: 2.7813
  • Accuracy: 0.6609

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 20.0

Training results

Training Loss Epoch Step Validation Loss Accuracy
1.4488 1.0 250 1.4958 0.6770
1.3142 2.0 500 1.5007 0.6772
1.1176 3.0 750 1.5507 0.6756
0.9253 4.0 1000 1.6442 0.6728
0.7213 5.0 1250 1.7736 0.6701
0.5718 6.0 1500 1.8863 0.6682
0.4232 7.0 1750 2.0245 0.6660
0.3334 8.0 2000 2.1773 0.6642
0.2433 9.0 2250 2.2681 0.6632
0.2076 10.0 2500 2.3732 0.6629
0.1632 11.0 2750 2.4368 0.6623
0.1491 12.0 3000 2.5182 0.6617
0.1275 13.0 3250 2.5680 0.6619
0.1273 14.0 3500 2.6412 0.6613
0.1129 15.0 3750 2.6497 0.6617
0.1129 16.0 4000 2.6932 0.6614
0.102 17.0 4250 2.7003 0.6612
0.1109 18.0 4500 2.7033 0.6614
0.0997 19.0 4750 2.7139 0.6613
0.1012 20.0 5000 2.7813 0.6609

Framework versions

  • PEFT 0.5.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
1
Unable to determine this model’s pipeline type. Check the docs .

Adapter for

Dataset used to train tyzhu/lmind_hotpot_train8000_eval7405_v1_reciteonly_qa_Qwen_Qwen1.5-4B_3e-4_lora2

Evaluation results

  • Accuracy on tyzhu/lmind_hotpot_train8000_eval7405_v1_reciteonly_qa
    self-reported
    0.661