---
license: apache-2.0
datasets:
- randomani/MedicalQnA-llama2
language:
- en
library_name: adapter-transformers
pipeline_tag: question-answering
tags:
- medical
---

# Fine-Tuning LLaMA-2 Chat Model with Medical QnA Dataset using QLoRA

This repository contains the code and configuration for fine-tuning the LLaMA-2 chat model using the Medical QnA dataset with the QLoRA technique.Used only 2k data elements for training due to constrained gpu resources.

## Model and Dataset

- **Pre-trained Model**: `NousResearch/Llama-2-7b-chat-hf`
- **Dataset for Fine-Tuning**: `randomani/MedicalQnA-llama2`
- **Fine-Tuned Model Name**: `Llama-2-7b-Medchat-finetune`

## QLoRA Parameters

- **LoRA Attention Dimension** (`lora_r`): 64
- **LoRA Scaling Alpha** (`lora_alpha`): 16
- **LoRA Dropout Probability** (`lora_dropout`): 0.1

## bitsandbytes Parameters

- **Use 4-bit Precision** (`use_4bit`): True
- **4-bit Compute Dtype** (`bnb_4bit_compute_dtype`): float16
- **4-bit Quantization Type** (`bnb_4bit_quant_type`): nf4
- **Use Nested Quantization** (`use_nested_quant`): False

## Training Arguments

- **Number of Training Epochs** (`num_train_epochs`): 1
- **Use fp16** (`fp16`): False
- **Use bf16** (`bf16`): False
- **Training Batch Size per GPU** (`per_device_train_batch_size`): 4
- **Evaluation Batch Size per GPU** (`per_device_eval_batch_size`): 4
- **Gradient Accumulation Steps** (`gradient_accumulation_steps`): 1
- **Enable Gradient Checkpointing** (`gradient_checkpointing`): True
- **Maximum Gradient Norm** (`max_grad_norm`): 0.3
- **Initial Learning Rate** (`learning_rate`): 2e-4
- **Weight Decay** (`weight_decay`): 0.001
- **Optimizer** (`optim`): paged_adamw_32bit
- **Learning Rate Scheduler Type** (`lr_scheduler_type`): cosine
- **Maximum Training Steps** (`max_steps`): -1
- **Warmup Ratio** (`warmup_ratio`): 0.03
- **Group Sequences by Length** (`group_by_length`): True
- **Save Checkpoints Every X Steps** (`save_steps`): 0
- **Logging Steps** (`logging_steps`): 25

## Supervised Fine-Tuning (SFT) Parameters

- **Maximum Sequence Length** (`max_seq_length`): None
- **Packing Multiple Short Examples** (`packing`): False

## References

For more details and access to the dataset, visit the [Hugging Face Dataset Page](https://huggingface.co/datasets/randomani/MedicalQnA-llama2).