Hugging Face's logo

tyzhu
/
lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_3e-5_lora2

PEFT
Safetensors
Generated from Trainer
Eval Results
Model card Files Community
lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_3e-5_lora2 / README.md
tyzhu's picture
tyzhu
End of training
5d15588 verified
preview code
|
raw
history blame contribute delete
No virus
3.29 kB
metadata
license: other
base_model: Qwen/Qwen1.5-4B
tags:
  - generated_from_trainer
datasets:
  - tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx
metrics:
  - accuracy
model-index:
  - name: lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_3e-5_lora2
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx
          type: tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.7512829373650108
library_name: peft

lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_3e-5_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx dataset. It achieves the following results on the evaluation set:

  • Loss: 1.3258
  • Accuracy: 0.7513

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 20.0

Training results

Training Loss Epoch Step Accuracy Validation Loss
1.6137 0.9997 839 0.7177 1.8264
1.5978 1.9994 1678 0.7190 1.8264
1.5493 2.9991 2517 0.7211 1.7810
1.5101 4.0 3357 0.7231 1.7413
1.4678 4.9997 4196 0.7247 1.7400
1.4236 5.9994 5035 0.7267 1.7019
1.3843 6.9991 5874 0.7286 1.6725
1.3481 8.0 6714 0.7304 1.6381
1.2954 8.9997 7553 0.7324 1.6103
1.2426 9.9994 8392 0.7338 1.5785
1.2169 10.9991 9231 0.7355 1.5435
1.167 12.0 10071 0.7375 1.5216
1.1276 12.9997 10910 0.7392 1.4949
1.0819 13.9994 11749 0.7406 1.4819
1.032 14.9991 12588 0.7426 1.4468
0.9981 15.9997 13424 1.4092 0.7443
0.9523 16.9994 14263 1.3949 0.7463
0.9281 17.9991 15102 1.3853 0.7477
0.8664 19.0 15942 1.3669 0.7496
0.8537 19.9985 16780 1.3258 0.7513

Framework versions

  • PEFT 0.5.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1