Hugging Face's logo

tyzhu
/
lmind_nq_train6000_eval6489_v1_doc_qa_v3_3e-5_lora2

Safetensors
Generated from Trainer
Eval Results
Model card Files Community
lmind_nq_train6000_eval6489_v1_doc_qa_v3_3e-5_lora2 / README.md
tyzhu's picture
tyzhu
End of training
a6c626b verified
preview code
|
raw
history blame contribute delete
No virus
5.1 kB
metadata
license: llama2
base_model: meta-llama/Llama-2-7b-hf
tags:
  - generated_from_trainer
datasets:
  - tyzhu/lmind_nq_train6000_eval6489_v1_doc_qa_v3
metrics:
  - accuracy
model-index:
  - name: lmind_nq_train6000_eval6489_v1_doc_qa_v3_3e-5_lora2
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: tyzhu/lmind_nq_train6000_eval6489_v1_doc_qa_v3
          type: tyzhu/lmind_nq_train6000_eval6489_v1_doc_qa_v3
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.5422564102564102

lmind_nq_train6000_eval6489_v1_doc_qa_v3_3e-5_lora2

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on the tyzhu/lmind_nq_train6000_eval6489_v1_doc_qa_v3 dataset. It achieves the following results on the evaluation set:

  • Loss: 2.0598
  • Accuracy: 0.5423

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 50.0

Training results

Training Loss Epoch Step Accuracy Validation Loss
1.3948 1.0 529 0.6132 1.3087
1.3789 2.0 1058 0.6146 1.2897
1.3259 3.0 1587 0.6179 1.2849
1.2853 4.0 2116 0.6159 1.3169
1.2556 5.0 2645 0.6132 1.3532
1.1972 6.0 3174 0.6126 1.4135
1.1839 7.0 3703 0.6081 1.5007
1.1334 8.0 4232 0.6074 1.5242
1.0966 9.0 4761 0.5803 1.6107
1.0485 10.0 5290 0.6049 1.6749
1.021 11.0 5819 0.6015 1.7324
0.9918 12.0 6348 0.6007 1.7632
0.947 13.0 6877 0.6011 1.8303
0.9376 14.0 7406 0.5991 1.8873
0.898 15.0 7935 0.5976 1.9688
0.8559 16.0 8464 0.5988 1.9724
0.8348 17.0 8993 0.5714 1.9815
0.8106 18.0 9522 0.598 2.0386
0.7848 19.0 10051 0.5964 2.0627
0.745 20.0 10580 0.5966 2.0825
0.7208 21.0 11109 0.5959 2.0959
0.6842 22.0 11638 0.5968 2.1534
0.6661 23.0 12167 0.5975 2.1792
0.6193 24.0 12696 0.5967 2.1530
0.6064 25.0 13225 0.5958 2.1720
0.5776 26.0 13754 0.5966 2.2162
0.5492 27.0 14283 0.5862 2.2382
0.5256 28.0 14812 0.5963 2.2273
0.5128 29.0 15341 0.5948 2.2448
0.4846 30.0 15870 0.5846 2.2697
0.4623 31.0 16399 0.5968 2.2425
0.4468 32.0 16928 0.5936 2.2654
0.4714 33.0 17457 0.5957 2.1317
0.8308 34.0 17986 0.5973 1.9392
0.6478 35.0 18515 0.5979 2.0346
0.612 36.0 19044 0.5978 2.0882
0.5928 37.0 19573 0.5970 2.1420
0.5698 38.0 20102 0.5966 2.1569
0.5444 39.0 20631 0.5956 2.1954
0.5404 40.0 21160 0.5942 2.1724
0.5124 41.0 21689 0.5939 2.2020
0.5342 42.0 22218 0.5938 2.1955
0.5385 43.0 22747 0.5946 2.1431
0.5673 44.0 23276 0.5948 2.1269
0.7034 45.0 23805 0.5917 2.0601
1.0751 46.0 24334 0.5861 1.8910
1.9072 47.0 24863 0.5518 2.1388
5.2339 48.0 25392 0.3825 4.4877
2.573 49.0 25921 0.5283 2.2255
2.1439 50.0 26450 0.5423 2.0598

Framework versions

  • Transformers 4.34.0
  • Pytorch 2.1.0+cu121
  • Datasets 2.18.0
  • Tokenizers 0.14.1