# Llama-2-7b-chat-MedFAQ2
[NousResearch/Llama-2-7b-chat-hf](https://huggingface.co/NousResearch/Llama-2-7b-chat-hf) fine-tuned on [kazcfz/MedFAQ-llama2-2k](https://huggingface.co/datasets/kazcfz/MedFAQ-llama2-2k)

<hr style="margin-top: 1.0em; margin-bottom: 1.0em;">

**QLoRA parameters**
Parameter | Value
--- | ---
lora_r | 64
lora_alpha | 16
lora_dropout | 0.1

**bitsandbytes parameters**
Parameter | Value
--- | ---
use_4bit | True
bnb_4bit_compute_dtype | "float16"
bnb_4bit_quant_type | "nf4"
use_nested_quant | False

**TrainingArguments parameters**
Parameter | Value
--- | ---
num_train_epochs | 5
fp16 | False
bf16 | False
per_device_train_batch_size | 4
gradient_accumulation_steps | 1
gradient_checkpointing | True
max_grad_norm | 0.3
learning_rate | 2e-4
weight_decay | 0.001
optim | "paged_adamw_32bit"
lr_scheduler_type | "cosine"
max_steps | -1
warmup_ratio | 0.03
group_by_length | True
save_steps | 0
logging_steps | 25

**SFT parameters**
Parameter | Value
--- | ---
max_seq_length | None
packing | False

<br>

## Prompt template: Llama-2-Chat
```
<s>[INST] {prompt} [/INST]
```

<br>

## Sample (actual) output (Max length: 200)
**Base Model**
```
Query: <s>[INST] Am I at risk for kidney disease? [/INST]
Response: I cannot provide medical advice or diagnose any health conditions, including kidney disease. everybody's health is unique, and there are many factors that can increase or decrease the risk of developing kidney disease. However, I can provide some general information on the risk factors for kidney disease and suggest ways to reduce your risk.

Risk factors for kidney disease include:

1. Family history: If you have a family history of kidney disease, you are more likely to develop the condition.
2. Diabetes: Diabetes is a common cause of kidney disease. If you have diabetes, you are at a higher risk of developing kidney disease.
3. High blood pressure: High blood pressure can damage the kidneys and increase the risk of kidney disease.
4. Obesity: Being overweight or obese can increase
```
**Fine-tuned**
```
Query: <s>[INST] Am I at risk for kidney disease? [/INST]
Response: Your risk for kidney disease depends on your age, race, and medical history. If you have any of the following risk factors, talk to your doctor about your risk for kidney disease:

Family history of kidney disease
High blood pressure
Diabetes
High cholesterol
African American or Hispanic/Latino ethnicity
History of kidney disease in your family
History of stroke or heart disease
Obesity
Certain medications, such as certain antibiotics, NSAIDs, and certain medications for depression
Certain medical conditions, such as polycystic kidney disease (PKD), lupus, and certain genetic disorders Source: FRESENIUS KIDNEY CARE. Link: https://www.freseniuskidneycare.com/fre
```
**Reload model + Merge LoRA weights?**
```
Query: <s>[INST] Am I at risk for kidney disease? [/INST]
Response: Your risk for kidney disease depends on your age, family medical history, and other factors.

Risk factors for kidney disease include:

Diabetes
High blood pressure
African American or Hispanic/Latino race
Family history of kidney disease
Obesity
Certain medical conditions, such as high cholesterol, heart disease, and polycystic kidney disease
If you have any of these risk factors, talk to your doctor about your risk for kidney disease. Source: THE UNIVERSITY OF KANSAS HEALTH SYSTEM. Link: https://www.kansashealthsystem.com/care/conditions/kidney-disease/kidney-disease-faq. Field: Nephrology. Disease: Kidney D

```