Hugging Face's logo

tyzhu
/
lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3_Qwen_Qwen1.5-4B_5e-4_lora2

PEFT
Safetensors
Generated from Trainer
Eval Results
Model card Files Community
lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3_Qwen_Qwen1.5-4B_5e-4_lora2 / README.md
tyzhu's picture
tyzhu
End of training
286cde7 verified
preview code
|
raw
history blame contribute delete
No virus
3.3 kB
metadata
license: other
base_model: Qwen/Qwen1.5-4B
tags:
  - generated_from_trainer
datasets:
  - tyzhu/lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3
metrics:
  - accuracy
model-index:
  - name: lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3_Qwen_Qwen1.5-4B_5e-4_lora2
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: tyzhu/lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3
          type: tyzhu/lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.5833811659192825
library_name: peft

lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3_Qwen_Qwen1.5-4B_5e-4_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_nq_train6000_eval6489_v1_reciteonly_qa_v3 dataset. It achieves the following results on the evaluation set:

  • Loss: 2.8927
  • Accuracy: 0.5834

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0005
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 20.0

Training results

Training Loss Epoch Step Validation Loss Accuracy
1.745 0.9973 187 1.6686 0.6096
1.4255 2.0 375 1.6989 0.6087
1.0295 2.9973 562 1.8159 0.6052
0.6982 4.0 750 1.9994 0.5996
0.4584 4.9973 937 2.2058 0.5949
0.3048 6.0 1125 2.3636 0.5928
0.2175 6.9973 1312 2.5218 0.5905
0.1719 8.0 1500 2.6292 0.5909
0.1355 8.9973 1687 2.7028 0.5892
0.1302 10.0 1875 2.7502 0.5876
0.1281 10.9973 2062 2.7715 0.5875
0.1331 12.0 2250 2.7947 0.5862
0.1435 12.9973 2437 2.7769 0.5867
0.1468 14.0 2625 2.7598 0.5862
0.1449 14.9973 2812 2.8147 0.5855
0.1404 16.0 3000 2.8564 0.5859
0.121 16.9973 3187 2.8381 0.5863
0.1198 18.0 3375 2.8844 0.5841
0.1198 18.9973 3562 2.9040 0.5834
0.1215 19.9467 3740 2.8927 0.5834

Framework versions

  • PEFT 0.5.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1