DeepSparse Sparse LLMs
Collection
Useful LLMs for DeepSparse where we've pruned at least 50% of the weights!
•
10 items
•
Updated
•
4
This repo contains model files for MiniChat-1.5-3B optimized for DeepSparse, a CPU inference runtime for sparse models.
This model was quantized and pruned with SparseGPT, using SparseML.
Install DeepSparse LLM for fast inference on CPUs:
pip install deepsparse-nightly[llm]
Run in a Python pipeline:
from deepsparse import TextGeneration
prompt = "How to get in a good university?"
formatted_prompt = f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n"
model = TextGeneration(model_path="hf:neuralmagic/MiniChat-1.5-3B-pruned50-quant-ds")
print(model(formatted_prompt, max_new_tokens=200).generations[0].text)
"""
As an AI, I don't have personal experiences or opinions, but I can provide you with some general advice on how to get into a good university. Here are some tips to consider:
1. Academic performance: A good university requires good academic performance. This means you need to maintain a good GPA (grade point average) and achieve high marks in your courses. To do this, you need to put in the effort to learn and understand the course material.
2. Pursue a diverse range of courses: A good university student should not limit themselves to just one area of study. They should take courses in various fields that interest them. This will help them develop a wide range of skills and knowledge.
3. Networking: A good university student should network with others in their courses and beyond. This can be done through attending events like guest lectures, group meetings, and social events.
4. Be proactive
"""
<s> [|User|]\n
{prompt}
</s>[|Assistant|]\n
For details on how this model was sparsified, see the recipe.yaml
in this repo and follow the instructions below.
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py GeneZC/MiniChat-1.5-3B open_platypus --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
cp deployment/model.onnx deployment/model-orig.onnx
Run this kv-cache injection to speed up the model at inference by caching the Key and Value states:
import os
import onnx
from sparseml.exporters.kv_cache_injector import KeyValueCacheInjector
input_file = "deployment/model-orig.onnx"
output_file = "deployment/model.onnx"
model = onnx.load(input_file, load_external_data=False)
model = KeyValueCacheInjector(model_path=os.path.dirname(input_file)).apply(model)
onnx.save(model, output_file)
print(f"Modified model saved to: {output_file}")
Follow the instructions on our One Shot With SparseML page for a step-by-step guide for performing one-shot quantization of large language models.
For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community
Base model
GeneZC/MiniChat-1.5-3B