Hugging Face's logo

tyzhu
/
lmind_hotpot_train8000_eval7405_v1_doc_qa_Qwen_Qwen1.5-4B_3e-5_lora2

PEFT
Safetensors
Generated from Trainer
Eval Results
Model card Files Community
lmind_hotpot_train8000_eval7405_v1_doc_qa_Qwen_Qwen1.5-4B_3e-5_lora2 / README.md
tyzhu's picture
tyzhu
End of training
c3a26b2 verified
preview code
|
raw
history blame contribute delete
No virus
3.29 kB
metadata
license: other
base_model: Qwen/Qwen1.5-4B
tags:
  - generated_from_trainer
datasets:
  - tyzhu/lmind_hotpot_train8000_eval7405_v1_doc_qa
metrics:
  - accuracy
model-index:
  - name: lmind_hotpot_train8000_eval7405_v1_doc_qa_Qwen_Qwen1.5-4B_3e-5_lora2
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: tyzhu/lmind_hotpot_train8000_eval7405_v1_doc_qa
          type: tyzhu/lmind_hotpot_train8000_eval7405_v1_doc_qa
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.5000634920634921
library_name: peft

lmind_hotpot_train8000_eval7405_v1_doc_qa_Qwen_Qwen1.5-4B_3e-5_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_hotpot_train8000_eval7405_v1_doc_qa dataset. It achieves the following results on the evaluation set:

  • Loss: 3.5856
  • Accuracy: 0.5001

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 20.0

Training results

Training Loss Epoch Step Validation Loss Accuracy
1.7766 0.9998 1089 2.3352 0.5155
1.7291 1.9995 2178 2.3047 0.5177
1.6609 2.9993 3267 2.3113 0.5187
1.6108 4.0 4357 2.3487 0.518
1.5489 4.9998 5446 2.3843 0.5161
1.5093 5.9995 6535 2.4637 0.5138
1.4364 6.9993 7624 2.5600 0.5108
1.4038 8.0 8714 2.6166 0.5105
1.3439 8.9998 9803 2.7111 0.5092
1.2928 9.9995 10892 2.8851 0.5062
1.2284 10.9993 11981 2.9595 0.5042
1.2142 12.0 13071 3.0894 0.5034
1.1791 12.9998 14160 3.1358 0.5024
1.1272 13.9995 15249 3.2272 0.5017
1.0934 14.9993 16338 3.3488 0.5001
1.0586 16.0 17428 3.4072 0.5015
1.0415 16.9998 18517 3.4943 0.5009
1.011 17.9995 19606 3.5465 0.5001
1.0036 18.9993 20695 3.5320 0.5011
0.9595 19.9954 21780 3.5856 0.5001

Framework versions

  • PEFT 0.5.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1