File size: 3,031 Bytes
f0b4587 010bd27 f0b4587 010bd27 1a1008c 010bd27 3c7a39d 010bd27 acf884a 010bd27 1a1008c 010bd27 1a1008c 010bd27 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
---
license: mit
pipeline_tag: text-generation
tags:
- text-generation-inference
language:
- en
---
# Phi-3-mini-128k-instruct-int4
- Orginal model : [microsoft/Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct)
- Quantized using [intel/auto-round](https://github.com/intel/auto-round)
## Description
**Phi-3-mini-128k-instruct-int4** is an int4 model with group_size 128 of the [microsoft/Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct).
The above model was quantized using AutoRound(Advanced Weight-Only Quantization Algorithm for LLMs) released by [intel](https://github.com/intel).
you can find out more in detail through the the [GitHub Repository](https://github.com/intel/auto-round).
## Training details
### Cloning a repository(AutoRound)
```
git clone https://github.com/intel/auto-round
```
### Enter into the examples/language-modeling folder
```
cd auto-round/examples/language-modeling
pip install -r requirements.txt
```
### Install FlashAttention-2
```
pip install flash_attn==2.5.8
```
Here's an simplified code for quantization. In order to save memory in quantization, we set the batch size to 1.
```
python main.py \
--model_name "microsoft/Phi-3-mini-128k-instruct" \
--bits 4 \
--group_size 128 \
--train_bs 1 \
--gradient_accumulate_steps 8 \
--deployment_device 'gpu' \
--output_dir "./save_ckpt"
```
## Model inference
### Install the necessary packages
```
pip install auto_gptq
pip install optimum
pip install -U accelerate bitsandbytes datasets peft transformers
```
### Example codes
```
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
torch.random.manual_seed(0)
model = AutoModelForCausalLM.from_pretrained(
"ssuncheol/Phi-3-mini-128k-instruct-int4",
device_map="cuda",
torch_dtype="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("ssuncheol/Phi-3-mini-128k-instruct-int4")
messages = [
{"role": "system", "content": "You are a helpful digital assistant. Please provide safe, ethical and accurate information to the user."},
{"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"},
{"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."},
{"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"},
]
pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
)
generation_args = {
"max_new_tokens": 500,
"return_full_text": False,
"temperature": 0.0,
"do_sample": False,
}
output = pipe(messages, **generation_args)
print(output[0]['generated_text'])
```
## License
The model is licensed under the MIT license.
|