Edit model card

Nous-Hermes-Llama2-7b - DeepSparse

This repo contains model files for Nous-Hermes-Llama2-7b optimized for DeepSparse, a CPU inference runtime for sparse models.

This model was quantized and pruned with SparseGPT, using SparseML.

Inference

Install DeepSparse LLM for fast inference on CPUs:

pip install deepsparse-nightly[llm]

Run in a Python pipeline:

from deepsparse import TextGeneration

prompt = "How to make banana bread?"
formatted_prompt =  f"### Instruction\n{prompt}\n### Response:\n"

model = TextGeneration(model_path="hf:nm-testing/Nous-Hermes-llama-2-7b-pruned50-quant-ds")

print(model(formatted_prompt, max_new_tokens=200).generations[0].text)
"""
To make banana bread, start by preheating the oven to 350 degrees Fahrenheit.
In a bowl, mix together 1 cup of flour, 1 cup of sugar, and 1 teaspoon of baking soda.
Then, add 1 cup of milk and 1 cup of mashed banana.
Mix well and pour the mixture into a greased pan.
Bake the bread for about 45 minutes or until a toothpick inserted comes out clean.
"""

Prompt template

### Instruction:
<prompt>

### Response:
<leave a newline blank for model to respond>

Sparsification

For details on how this model was sparsified, see the recipe.yaml in this repo and follow the instructions below.

git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py NousResearch/Nous-Hermes-llama-2-7b open_platypus --precision float16 --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment 
cp deployment/model.onnx deployment/model-orig.onnx

Run this kv-cache injection to speed up the model at inference by caching the Key and Value states:

import os
import onnx
from sparseml.exporters.kv_cache_injector import KeyValueCacheInjector
input_file = "deployment/model-orig.onnx"
output_file = "deployment/model.onnx"
model = onnx.load(input_file, load_external_data=False)
model = KeyValueCacheInjector(model_path=os.path.dirname(input_file)).apply(model)
onnx.save(model, output_file)
print(f"Modified model saved to: {output_file}")

Follow the instructions on our One Shot With SparseML page for a step-by-step guide for performing one-shot quantization of large language models.

Slack

For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community

Downloads last month
12
Inference Examples
Inference API (serverless) has been turned off for this model.

Model tree for neuralmagic/Nous-Hermes-llama-2-7b-pruned50-quant-ds

Quantized
(4)
this model

Collection including neuralmagic/Nous-Hermes-llama-2-7b-pruned50-quant-ds