|
|
--- |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
# **Phi-3.5-mini-instruct-mlx-int4** |
|
|
|
|
|
This is a quantized INT4 model based on Apple MLX Framework Phi-3.5-mini-instruct. You can deploy it on Apple Silicon devices. |
|
|
|
|
|
## **Installation** |
|
|
|
|
|
|
|
|
```bash |
|
|
|
|
|
pip install -U mlx-lm |
|
|
|
|
|
``` |
|
|
|
|
|
## **Conversion** |
|
|
|
|
|
```bash |
|
|
|
|
|
python -m mlx_lm.convert --hf-path microsoft/Phi-3.5-mini-instruct -q |
|
|
|
|
|
``` |
|
|
|
|
|
## **Samples** |
|
|
|
|
|
```python |
|
|
|
|
|
from mlx_lm import load, generate |
|
|
|
|
|
model, tokenizer = load("Your Phi-3.5-instruct-mlx-int4 Path") |
|
|
|
|
|
prompt = tokenizer.apply_chat_template( |
|
|
[{"role": "user", "content": "能介绍一下 iOS 吗？"}], |
|
|
tokenize=False, |
|
|
add_generation_prompt=True, |
|
|
) |
|
|
|
|
|
response = generate(model, tokenizer, prompt=prompt,max_tokens=1024, verbose=True) |
|
|
|
|
|
``` |
|
|
|
