Edit model card

lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_3e-5_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx dataset. It achieves the following results on the evaluation set:

  • Loss: 1.3258
  • Accuracy: 0.7513

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 20.0

Training results

Training Loss Epoch Step Accuracy Validation Loss
1.6137 0.9997 839 0.7177 1.8264
1.5978 1.9994 1678 0.7190 1.8264
1.5493 2.9991 2517 0.7211 1.7810
1.5101 4.0 3357 0.7231 1.7413
1.4678 4.9997 4196 0.7247 1.7400
1.4236 5.9994 5035 0.7267 1.7019
1.3843 6.9991 5874 0.7286 1.6725
1.3481 8.0 6714 0.7304 1.6381
1.2954 8.9997 7553 0.7324 1.6103
1.2426 9.9994 8392 0.7338 1.5785
1.2169 10.9991 9231 0.7355 1.5435
1.167 12.0 10071 0.7375 1.5216
1.1276 12.9997 10910 0.7392 1.4949
1.0819 13.9994 11749 0.7406 1.4819
1.032 14.9991 12588 0.7426 1.4468
0.9981 15.9997 13424 1.4092 0.7443
0.9523 16.9994 14263 1.3949 0.7463
0.9281 17.9991 15102 1.3853 0.7477
0.8664 19.0 15942 1.3669 0.7496
0.8537 19.9985 16780 1.3258 0.7513

Framework versions

  • PEFT 0.5.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
1
Unable to determine this model’s pipeline type. Check the docs .

Adapter for

Dataset used to train tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_3e-5_lora2

Evaluation results

  • Accuracy on tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx
    self-reported
    0.751