---
language: 
  - en
license: other
datasets:
- vicgalle/alpaca-gpt4
---

# phi-1_5-qlora-alpaca-instruction Model Card

## Model Description

This model is a causal language model based on the `microsoft/phi-1_5` and has been finetuned using QLORA technology on the `vicgalle/alpaca-gpt4` dataset.

## Fine-tuning Details
- **Base Model**: `microsoft/phi-1_5`
- **Fine-tuning Dataset**: `vicgalle/alpaca-gpt4`
- **Hardware**: NVIDIA 3090ti
- **Training Duration**: 14 hours
- **VRAM Consumption**: Approx. 20 GB 
- **Token Max Length**: 2048
- **Model Size**: 1.5billion + qlora weights merged

### Hyperparameters

```python
# Lora Configuration
config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["Wqkv", "out_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Training Hyperparameters
training_arguments = TrainingArguments(
        output_dir=f"{local_path}/output_dir",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=6,
        learning_rate=2e-4,
        lr_scheduler_type="cosine",
        evaluation_strategy = "steps",
        eval_steps=500,
        save_strategy="epoch",
        logging_steps=100,
        num_train_epochs=6,
        report_to = 'wandb',
        run_name = run_name
    )

```

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "nps798/phi-1_5-qlora-alpaca-instruction"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map={"": 0},
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True
)

prompt= """Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Choose three places you would like to visit and explain why.

### Response:"""
inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False)
outputs = model.generate(**inputs, max_length=500)
text = tokenizer.batch_decode(outputs)[0]
print(text)

```

## Example
| Question | Original phi 1.5 | This fine-tuned model |
|----------|----------|----------|
| I'm planning to visit Taiwan soon. Can you recommend some local festivals or cultural events that I shouldn't miss? | There are many festivals and cultural events in Taiwan that you should definitely check out. Some popular ones include the Dragon Boat Festival, the Mid-Autumn Festival, and the Dragon Boat Festival Parade. You can also visit the Taiwan Museum of Natural History to learn more about the country's rich history and culture. | 1. Taipei City: This is the capital of Taiwan and is known for its modern architecture, shopping, and nightlife.
2. Taipei Zoo: This is one of the largest zoos in the world, and it's home to a wide variety of animals, including tigers, elephants, and gorillas.
3. Taipei Gardens: This is a large botanical garden that features a wide variety of plants, including tropical flowers, shrubs, and trees.
4. Taipei Park: This is a large park that features a variety of attractions, including a zoo, a botanical garden, and a zoo.
5. Taipei Aquarium: This is one of the largest aquariums in the world, and it features a wide variety of marine animals, including dolphins, whales, and sharks.
6. Taipei International Airport: This is the largest airport in the world, and it's home to a wide variety of airlines, including American, British, and Japanese.
7. Taipei International Museum: This is one of the largest museums in the world, and it features a wide variety of exhibits, including art, history, and science.
8. Taipei International School: This is one of the largest schools in the world, and it's home to a wide variety of students, including children, teenagers, and adults. |
| Row2Col1 | Row2Col2 | Row2Col3 |
| Row3Col1 | Row3Col2 | Row3Col3 |



## License
Because the base model is microsoft phi-1.5b model, this fine-tuned model is provided under the MICROSOFT RESEARCH LICENSE and is meant for non-commercial use only.

## Author
I am a medical doctor interested in ML/NLP field. 
If you have any advice, suggestions, or opportunities, or simply want to discuss the fascinating intersection of medicine and technology, please don't hesitate to reach out.