metadata
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
inference: false
model_type: llama
prompt_template: |
<|im_start|>user\n
{prompt}<|im_end|>\n
<|im_start|>assistant\n
quantized_by: mwitiderrick
tags:
- deepsparse
TinyLlama 1.1B Chat 1.0 - DeepSparse
This repo contains model files for TinyLlama 1.1B Chat optimized for DeepSparse, a CPU inference runtime for sparse models.
This model was quantized and pruned with SparseGPT, using SparseML.
Inference
Install DeepSparse LLM for fast inference on CPUs:
pip install deepsparse-nightly[llm]
Run in a Python pipeline:
from deepsparse import TextGeneration
prompt = "How to make banana bread?"
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
model = TextGeneration(model_path="hf:nm-testing/TinyLlama-1.1B-Chat-v1.0-pruned50-quant-ds")
print(model(formatted_prompt, max_new_tokens=200).generations[0].text)
"""
Sure, here's a recipe for making banana bread:
Ingredients:
- 1 cup of all-purpose flour
- 1 cup of unsalted butter
- 1 cup of unsalted sugar
- 1 cup of mashed bananas
- 1 cup of milk
- 1/2 cup of egg whites
- 1/4 cup of melted butter
Instructions:
1. Preheat oven to 375°F (150°C).
2. In a large mixing bowl, combine flour, sugar, mashed bananas, milk, egg whites, and butter. Mix well.
3. Add melted butter and mix again.
4. Add melted sugar and mix again.
5. Add melted milk and mix again.
6. Add egg whites and mix
"""
Prompt template
<|im_start|>user\n
{prompt}<|im_end|>\n
<|im_start|>assistant\n
Sparsification
For details on how this model was sparsified, see the recipe.yaml in this repo and follow the instructions below.
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py TinyLlama/TinyLlama-1.1B-Chat-v1.0 open_platypus --precision float16 --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
cp deployment/model.onnx deployment/model-orig.onnx
Run this kv-cache injection to speed up the model at inference by caching the Key and Value states:
import os
import onnx
from sparseml.exporters.kv_cache_injector import KeyValueCacheInjector
input_file = "deployment/model-orig.onnx"
output_file = "deployment/model.onnx"
model = onnx.load(input_file, load_external_data=False)
model = KeyValueCacheInjector(model_path=os.path.dirname(input_file)).apply(model)
onnx.save(model, output_file)
print(f"Modified model saved to: {output_file}")
Follow the instructions on our One Shot With SparseML page for a step-by-step guide for performing one-shot quantization of large language models.
Slack
For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community