README.md

---
base_model: microsoft/phi-2
inference: true
model_type: phi
quantized_by: mgoin
tags:
- nm-vllm
- sparse
---

## phi-2-pruned50
This repo contains model files for [phi 2](https://huggingface.co/microsoft/phi-2) optimized for [NM-vLLM](https://github.com/neuralmagic/nm-vllm), a high-throughput serving engine for compressed LLMs.

This model was pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).

## Inference
Install [NM-vLLM](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage: 
```bash
pip install nm-vllm[sparse]
```
Run in a Python pipeline for local inference:
```python
from vllm import LLM, SamplingParams

# Create a sparse LLM
llm = LLM("nm-testing/phi-2-pruned50", sparsity="sparse_w16a16")

prompt = "Once upon a time, there was a little car named Beep."
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.0, max_tokens=200)

# Generate texts from the prompts. The output is a list of RequestOutput objects
# that contain the prompt, generated text, and other information.
outputs = llm.generate(prompt, sampling_params)
# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"\nGenerated text: {prompt}{generated_text}\n")

"""
Generated text: Once upon a time, there was a little car named Beep. Beep was a small car, but he was very fast and loved to go on adventures. Beep had a friend named Bop who was a big car. Bop was very slow and loved to stay at home. Beep and Bop were very different, but they were still friends.

One day, Beep and Bop decided to go on an adventure together. Beep was excited to explore new places and Bop was excited to see Beep explore. They started their adventure by driving on a bumpy road. Beep was having a great time, but Bop was having a hard time. Bop was so big that he couldn't fit in the small spaces between the bumps. Beep was having a great time, but Bop was having a hard time.

As they continued their adventure, they came across a big hill. Beep was excited to climb the hill, but Bop was scared. Bop was so big that he couldn't
"""
```

## Prompt template

```
"Instruct: <prompt>\nOutput:"
```

## Sparsification
For details on how this model was sparsified, see the `recipe.yaml` in this repo and follow the instructions below.

Install [SparseML](https://github.com/neuralmagic/sparseml):
```bash
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
```

Replace the recipe as you like and run this one-shot compression script to apply SparseGPT:
```python
import sparseml.transformers

original_model_name = microsoft/phi-2"
calibration_dataset = "open_platypus"
output_directory = "output/"

recipe = """
test_stage:
  obcq_modifiers:
    SparseGPTModifier:
      sparsity: 0.5
      sequential_update: true
      targets: ['re:model.layers.\d*$']
"""

# Apply SparseGPT to the model
sparseml.transformers.oneshot(
    model=original_model_name,
    dataset=calibration_dataset,
    recipe=recipe,
    output_dir=output_directory,
)
```

## Slack

For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)