---
license: llama2
base_model: meta-llama/Llama-2-7b-hf
tags:
- generated_from_trainer
datasets:
- tyzhu/lmind_nq_train6000_eval6489_v1_doc_qa_v3
metrics:
- accuracy
model-index:
- name: lmind_nq_train6000_eval6489_v1_doc_qa_v3_3e-5_lora2
  results:
  - task:
      name: Causal Language Modeling
      type: text-generation
    dataset:
      name: tyzhu/lmind_nq_train6000_eval6489_v1_doc_qa_v3
      type: tyzhu/lmind_nq_train6000_eval6489_v1_doc_qa_v3
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.5422564102564102
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# lmind_nq_train6000_eval6489_v1_doc_qa_v3_3e-5_lora2

This model is a fine-tuned version of [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) on the tyzhu/lmind_nq_train6000_eval6489_v1_doc_qa_v3 dataset.
It achieves the following results on the evaluation set:
- Loss: 2.0598
- Accuracy: 0.5423

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 50.0

### Training results

| Training Loss | Epoch | Step  | Accuracy | Validation Loss |
|:-------------:|:-----:|:-----:|:--------:|:---------------:|
| 1.3948        | 1.0   | 529   | 0.6132   | 1.3087          |
| 1.3789        | 2.0   | 1058  | 0.6146   | 1.2897          |
| 1.3259        | 3.0   | 1587  | 0.6179   | 1.2849          |
| 1.2853        | 4.0   | 2116  | 0.6159   | 1.3169          |
| 1.2556        | 5.0   | 2645  | 0.6132   | 1.3532          |
| 1.1972        | 6.0   | 3174  | 0.6126   | 1.4135          |
| 1.1839        | 7.0   | 3703  | 0.6081   | 1.5007          |
| 1.1334        | 8.0   | 4232  | 0.6074   | 1.5242          |
| 1.0966        | 9.0   | 4761  | 0.5803   | 1.6107          |
| 1.0485        | 10.0  | 5290  | 0.6049   | 1.6749          |
| 1.021         | 11.0  | 5819  | 0.6015   | 1.7324          |
| 0.9918        | 12.0  | 6348  | 0.6007   | 1.7632          |
| 0.947         | 13.0  | 6877  | 0.6011   | 1.8303          |
| 0.9376        | 14.0  | 7406  | 0.5991   | 1.8873          |
| 0.898         | 15.0  | 7935  | 0.5976   | 1.9688          |
| 0.8559        | 16.0  | 8464  | 0.5988   | 1.9724          |
| 0.8348        | 17.0  | 8993  | 0.5714   | 1.9815          |
| 0.8106        | 18.0  | 9522  | 0.598    | 2.0386          |
| 0.7848        | 19.0  | 10051 | 0.5964   | 2.0627          |
| 0.745         | 20.0  | 10580 | 0.5966   | 2.0825          |
| 0.7208        | 21.0  | 11109 | 0.5959   | 2.0959          |
| 0.6842        | 22.0  | 11638 | 0.5968   | 2.1534          |
| 0.6661        | 23.0  | 12167 | 0.5975   | 2.1792          |
| 0.6193        | 24.0  | 12696 | 0.5967   | 2.1530          |
| 0.6064        | 25.0  | 13225 | 0.5958   | 2.1720          |
| 0.5776        | 26.0  | 13754 | 0.5966   | 2.2162          |
| 0.5492        | 27.0  | 14283 | 0.5862   | 2.2382          |
| 0.5256        | 28.0  | 14812 | 0.5963   | 2.2273          |
| 0.5128        | 29.0  | 15341 | 0.5948   | 2.2448          |
| 0.4846        | 30.0  | 15870 | 0.5846   | 2.2697          |
| 0.4623        | 31.0  | 16399 | 0.5968   | 2.2425          |
| 0.4468        | 32.0  | 16928 | 0.5936   | 2.2654          |
| 0.4714        | 33.0  | 17457 | 0.5957   | 2.1317          |
| 0.8308        | 34.0  | 17986 | 0.5973   | 1.9392          |
| 0.6478        | 35.0  | 18515 | 0.5979   | 2.0346          |
| 0.612         | 36.0  | 19044 | 0.5978   | 2.0882          |
| 0.5928        | 37.0  | 19573 | 0.5970   | 2.1420          |
| 0.5698        | 38.0  | 20102 | 0.5966   | 2.1569          |
| 0.5444        | 39.0  | 20631 | 0.5956   | 2.1954          |
| 0.5404        | 40.0  | 21160 | 0.5942   | 2.1724          |
| 0.5124        | 41.0  | 21689 | 0.5939   | 2.2020          |
| 0.5342        | 42.0  | 22218 | 0.5938   | 2.1955          |
| 0.5385        | 43.0  | 22747 | 0.5946   | 2.1431          |
| 0.5673        | 44.0  | 23276 | 0.5948   | 2.1269          |
| 0.7034        | 45.0  | 23805 | 0.5917   | 2.0601          |
| 1.0751        | 46.0  | 24334 | 0.5861   | 1.8910          |
| 1.9072        | 47.0  | 24863 | 0.5518   | 2.1388          |
| 5.2339        | 48.0  | 25392 | 0.3825   | 4.4877          |
| 2.573         | 49.0  | 25921 | 0.5283   | 2.2255          |
| 2.1439        | 50.0  | 26450 | 0.5423   | 2.0598          |


### Framework versions

- Transformers 4.34.0
- Pytorch 2.1.0+cu121
- Datasets 2.18.0
- Tokenizers 0.14.1