Edit model card

lmind_nq_train6000_eval6489_v1_docidx_v3_Qwen_Qwen1.5-4B_3e-5_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_nq_train6000_eval6489_v1_docidx_v3 dataset. It achieves the following results on the evaluation set:

  • Loss: 4.3717
  • Accuracy: 0.4408

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 20.0

Training results

Training Loss Epoch Step Validation Loss Accuracy
1.9685 0.9985 341 3.0537 0.4626
1.9337 2.0 683 2.9928 0.4709
1.9029 2.9985 1024 3.0243 0.4705
1.856 4.0 1366 3.0766 0.4697
1.8019 4.9985 1707 3.1923 0.4696
1.7406 6.0 2049 3.2573 0.4684
1.6974 6.9985 2390 3.3286 0.4672
1.6249 8.0 2732 3.4775 0.4647
1.5993 8.9985 3073 3.5378 0.4636
1.5449 10.0 3415 3.6347 0.4597
1.4855 10.9985 3756 3.6955 0.4553
1.4205 12.0 4098 3.8478 0.4479
1.3757 12.9985 4439 3.9185 0.4487
1.3098 14.0 4781 3.9575 0.4455
1.2574 14.9985 5122 4.1279 0.4457
1.2049 16.0 5464 4.1540 0.4448
1.1617 16.9985 5805 4.2049 0.4454
1.1046 18.0 6147 4.2909 0.4432
1.043 18.9985 6488 4.3535 0.4385
1.0044 19.9707 6820 4.3717 0.4408

Framework versions

  • PEFT 0.5.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
1
Unable to determine this model’s pipeline type. Check the docs .

Adapter for

Dataset used to train tyzhu/lmind_nq_train6000_eval6489_v1_docidx_v3_Qwen_Qwen1.5-4B_3e-5_lora2

Evaluation results

  • Accuracy on tyzhu/lmind_nq_train6000_eval6489_v1_docidx_v3
    self-reported
    0.441