neuralmagic/llama2.c-stories110M-pruned2.4

llama2.c-stories110M-pruned2.4

This repo contains model files for llama2.c 110M tinystories optimized for NM-vLLM, a high-throughput serving engine for compressed LLMs.

This model was pruned with SparseGPT, using SparseML.

Inference

Install NM-vLLM for fast inference and low memory-usage:

pip install nm-vllm[sparse]

Run in a Python pipeline for local inference:

from vllm import LLM, SamplingParams

model = LLM("nm-testing/llama2.c-stories110M-pruned2.4", sparsity="semi_structured_sparse_w16a16")
prompt = "My name is "
sampling_params = SamplingParams(max_tokens=100,temperature=0)
outputs = model.generate(prompt, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

""""
3 years old. My name is Sam. I love to play with my toys. I love to play with my toys.
One day, my mom takes me to the park. She brings a big bag. She takes out a big bag. It is full of things.
At the park, Sam sees a big box. He sees it was made from paper. He sees it is made from paper. He sees it is made from paper.
Sam's mom takes outs
"""

Prompt template

N/A

Sparsification

For details on how this model was sparsified, see the recipe.yaml in this repo and follow the instructions below.

Install SparseML:

git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"

Replace the recipe as you like and run this one-shot compression script to apply SparseGPT:

import sparseml.transformers

original_model_name = "Xenova/llama2.c-stories110M"
calibration_dataset = "open_platypus"
output_directory = "output/"

recipe = """
test_stage:
  obcq_modifiers:
    SparseGPTModifier:
      sparsity: 0.5
      sequential_update: true
      quantize: false
      mask_structure: '2:4'
      targets: ['re:model.layers.\d*$']
"""

# Apply SparseGPT to the model
sparseml.transformers.oneshot(
    model=original_model_name,
    dataset=calibration_dataset,
    recipe=recipe,
    output_dir=output_directory,
)

Slack

For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community