---
license: other
base_model: Qwen/Qwen1.5-4B
tags:
- generated_from_trainer
datasets:
- tyzhu/lmind_nq_train6000_eval6489_v1_qa
metrics:
- accuracy
model-index:
- name: lmind_nq_train6000_eval6489_v1_qa_Qwen_Qwen1.5-4B_lora2
  results:
  - task:
      name: Causal Language Modeling
      type: text-generation
    dataset:
      name: tyzhu/lmind_nq_train6000_eval6489_v1_qa
      type: tyzhu/lmind_nq_train6000_eval6489_v1_qa
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.5578974358974359
library_name: peft
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# lmind_nq_train6000_eval6489_v1_qa_Qwen_Qwen1.5-4B_lora2

This model is a fine-tuned version of [Qwen/Qwen1.5-4B](https://huggingface.co/Qwen/Qwen1.5-4B) on the tyzhu/lmind_nq_train6000_eval6489_v1_qa dataset.
It achieves the following results on the evaluation set:
- Loss: 2.4726
- Accuracy: 0.5579

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 1
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 20.0

### Training results

| Training Loss | Epoch   | Step | Accuracy | Validation Loss |
|:-------------:|:-------:|:----:|:--------:|:---------------:|
| 1.7657        | 0.9973  | 187  | 0.5738   | 1.6215          |
| 1.497         | 2.0     | 375  | 0.5742   | 1.6180          |
| 1.2345        | 2.9973  | 562  | 0.5713   | 1.6951          |
| 1.0084        | 4.0     | 750  | 0.5659   | 1.8059          |
| 0.8397        | 4.9973  | 937  | 0.5647   | 1.9245          |
| 0.7186        | 6.0     | 1125 | 0.5614   | 2.0345          |
| 0.6421        | 6.9973  | 1312 | 0.5608   | 2.1148          |
| 0.5968        | 8.0     | 1500 | 0.5585   | 2.1779          |
| 0.5417        | 8.9973  | 1687 | 0.5568   | 2.2654          |
| 0.5356        | 9.9733  | 1870 | 0.5594   | 2.2527          |
| 0.5261        | 10.9973 | 2057 | 2.3376   | 0.5585          |
| 0.5179        | 12.0    | 2245 | 2.3704   | 0.5595          |
| 0.5116        | 12.9973 | 2432 | 2.3617   | 0.5589          |
| 0.5056        | 14.0    | 2620 | 2.4022   | 0.5581          |
| 0.5063        | 14.9973 | 2807 | 2.3861   | 0.5587          |
| 0.4796        | 16.0    | 2995 | 2.3658   | 0.5585          |
| 0.4757        | 16.9973 | 3182 | 2.4195   | 0.5577          |
| 0.4779        | 18.0    | 3370 | 2.4573   | 0.5573          |
| 0.4782        | 18.9973 | 3557 | 2.4896   | 0.5589          |
| 0.4784        | 19.9733 | 3740 | 2.4726   | 0.5579          |


### Framework versions

- PEFT 0.5.0
- Transformers 4.40.2
- Pytorch 2.3.0
- Datasets 2.19.1
- Tokenizers 0.19.1