Create README.md

README.md

@@ -0,0 +1,93 @@
+---
+base_model: microsoft/phi-2
+inference: true
+model_type: phi
+quantized_by: mgoin
+tags:
+- nm-vllm
+- sparse
+---
+
+## phi-2-pruned50
+This repo contains model files for [phi 2](https://huggingface.co/microsoft/phi-2) optimized for [NM-vLLM](https://github.com/neuralmagic/nm-vllm), a high-throughput serving engine for compressed LLMs.
+
+This model was pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).
+
+## Inference
+Install [NM-vLLM](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage: 
+```bash
+pip install nm-vllm[sparse]
+```
+Run in a Python pipeline for local inference:
+```python
+from vllm import LLM, SamplingParams
+
+# Create a sparse LLM
+llm = LLM("nm-testing/phi-2-pruned50", sparsity="sparse_w16a16")
+
+prompt = "Once upon a time, there was a little car named Beep."
+# Create a sampling params object.
+sampling_params = SamplingParams(temperature=0.0, max_tokens=200)
+
+# Generate texts from the prompts. The output is a list of RequestOutput objects
+# that contain the prompt, generated text, and other information.
+outputs = llm.generate(prompt, sampling_params)
+# Print the outputs.
+for output in outputs:
+    prompt = output.prompt
+    generated_text = output.outputs[0].text
+    print(f"\nGenerated text: {prompt}{generated_text}\n")
+
+"""
+Generated text: Once upon a time, there was a little car named Beep. Beep was a small car, but he was very fast and loved to go on adventures. Beep had a friend named Bop who was a big car. Bop was very slow and loved to stay at home. Beep and Bop were very different, but they were still friends.
+
+One day, Beep and Bop decided to go on an adventure together. Beep was excited to explore new places and Bop was excited to see Beep explore. They started their adventure by driving on a bumpy road. Beep was having a great time, but Bop was having a hard time. Bop was so big that he couldn't fit in the small spaces between the bumps. Beep was having a great time, but Bop was having a hard time.
+
+As they continued their adventure, they came across a big hill. Beep was excited to climb the hill, but Bop was scared. Bop was so big that he couldn't
+"""
+```
+
+## Prompt template
+
+```
+"Instruct: <prompt>\nOutput:"
+```
+
+## Sparsification
+For details on how this model was sparsified, see the `recipe.yaml` in this repo and follow the instructions below.
+
+Install [SparseML](https://github.com/neuralmagic/sparseml):
+```bash
+git clone https://github.com/neuralmagic/sparseml
+pip install -e "sparseml[transformers]"
+```
+
+Replace the recipe as you like and run this one-shot compression script to apply SparseGPT:
+```python
+import sparseml.transformers
+
+original_model_name = microsoft/phi-2"
+calibration_dataset = "open_platypus"
+output_directory = "output/"
+
+recipe = """
+test_stage:
+  obcq_modifiers:
+    SparseGPTModifier:
+      sparsity: 0.5
+      sequential_update: true
+      targets: ['re:model.layers.\d*$']
+"""
+
+# Apply SparseGPT to the model
+sparseml.transformers.oneshot(
+    model=original_model_name,
+    dataset=calibration_dataset,
+    recipe=recipe,
+    output_dir=output_directory,
+)
+```
+
+## Slack
+
+For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)