neuralmagic
/

Sparse-Llama-3.1-8B-2of4

Text Generation

Safetensors

vllm

sparsity

Model card Files Files and versions Community

nm-research commited on 28 days ago

Commit

c28c229

verified ·

1 Parent(s): 0f371f6

Update README.md

Browse files

Files changed (1) hide show

README.md +148 -139

README.md CHANGED Viewed

@@ -1,139 +1,148 @@
----
-tags:
-- vllm
-- sparsity
-pipeline_tag: text-generation
-license: llama3.1
-base_model: meta-llama/Llama-3.1-8B
----
-# Sparse-Llama-3.1-8B-2of4
-## Model Overview
-- **Model Architecture:** Llama-3.1-8B
-  - **Input:** Text
-  - **Output:** Text
-- **Model Optimizations:**
-  - **Sparsity:** 2:4
-- **Release Date:** 11/20/2024
-- **Version:** 1.0
-- **License(s):** [llama3.1](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/blob/main/LICENSE)
-- **Model Developers:** Neural Magic
-This is the 2:4 sparse version of [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B).
-On the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), it achieves an average score of 62.16, compared to 63.19 for the dense model—demonstrating a **98.37% accuracy recovery**. On the [Mosaic Eval Gauntlet](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md) benchmark (version v0.3), it achieves an average score of 53.85, versus 55.34 for the dense model—representing a **97.3% accuracy recovery**.
-### Model Optimizations
-This model was obtained by pruning all linear operators within transformer blocks to the 2:4 sparsity pattern: in each group of four weights, two are retained while two are pruned. In addition to pruning, the sparse model was trained with knowledge distillation for 13B tokens to recover the accuracy loss incurred by pruning. For pruning, we utilize optimized version of [SparseGPT](https://arxiv.org/abs/2301.00774) through [LLM-Compressor](https://github.com/vllm-project/llm-compressor), and for sparse training with knowledge distillation we utilize [SquareHead approach](https://arxiv.org/abs/2310.06927).
-## Deployment with vLLM
-This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend. vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
-## Evaluation
-This model was evaluated on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1) with the [vLLM](https://docs.vllm.ai/en/stable/) engine for faster inference. In addition to the OpenLLM benchmark, the model was evaluated on the [Mosaic Eval Gauntlet](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md) benchmark (version v0.3). The evaluation results are summarized below.
-### Accuracy
-#### Open LLM Leaderboard evaluation scores
-<table>
-    <tr>
-        <td><strong>Benchmark</strong></td>
-        <td style="text-align: center"><strong>Llama-3.1-8B</strong></td>
-        <td style="text-align: center"><strong>Sparse-Llama-3.1-8B-2of4</strong></td>
-    </tr>
-    <tr>
-        <td>ARC-C (25-shot)</td>
-        <td style="text-align: center">58.2</td>
-        <td style="text-align: center">59.4</td>
-    </tr>
-    <tr>
-        <td>MMLU (5-shot)</td>
-        <td style="text-align: center">65.4</td>
-        <td style="text-align: center">60.6</td>
-    </tr>
-    <tr>
-        <td>HellaSwag (10-shot)</td>
-        <td style="text-align: center">82.3</td>
-        <td style="text-align: center">79.8</td>
-    </tr>
-    <tr>
-        <td>WinoGrande (5-shot)</td>
-        <td style="text-align: center">78.3</td>
-        <td style="text-align: center">75.9</td>
-    </tr>
-    <tr>
-        <td>GSM8K (5-shot)</td>
-        <td style="text-align: center">50.7</td>
-        <td style="text-align: center">56.3</td>
-    </tr>
-    <tr>
-        <td>TruthfulQA (0-shot)</td>
-        <td style="text-align: center">44.2</td>
-        <td style="text-align: center">40.9</td>
-    </tr>
-    <tr>
-        <td><strong>Average Score</strong></td>
-        <td style="text-align: center"><strong>63.19</strong></td>
-        <td style="text-align: center"><strong>62.16</strong></td>
-    </tr>
-    <tr>
-        <td><strong>Accuracy Recovery (%)</strong></td>
-        <td style="text-align: center"><strong>100</strong></td>
-        <td style="text-align: center"><strong>98.37</strong></td>
-    </tr>
-</table>
-#### Mosaic Eval Gauntlet evaluation scores
-<table>
-    <tr>
-        <td><strong>Benchmark</strong></td>
-        <td style="text-align: center"><strong>Llama-3.1-8B</strong></td>
-        <td style="text-align: center"><strong>Sparse-Llama-3.1-8B-2of4</strong></td>
-    </tr>
-    <tr>
-        <td>World Knowledge</td>
-        <td style="text-align: center">59.4</td>
-        <td style="text-align: center">55.6</td>
-    </tr>
-    <tr>
-        <td>Commonsense Reasoning</td>
-        <td style="text-align: center">49.3</td>
-        <td style="text-align: center">50.0</td>
-    </tr>
-    <tr>
-        <td>Language Understanding</td>
-        <td style="text-align: center">69.8</td>
-        <td style="text-align: center">69.0</td>
-    </tr>
-    <tr>
-        <td>Symbolic Problem Solving</td>
-        <td style="text-align: center">40.0</td>
-        <td style="text-align: center">37.1</td>
-    </tr>
-    <tr>
-        <td>Reading Comprehension</td>
-        <td style="text-align: center">58.2</td>
-        <td style="text-align: center">57.5</td>
-    </tr>
-    <tr>
-        <td><strong>Average Score</strong></td>
-        <td style="text-align: center"><strong>55.34</strong></td>
-        <td style="text-align: center"><strong>53.85</strong></td>
-    </tr>
-    <tr>
-        <td><strong>Accuracy Recovery (%)</strong></td>
-        <td style="text-align: center"><strong>100</strong></td>
-        <td style="text-align: center"><strong>97.3</strong></td>
-    </tr>
-</table>

+---
+tags:
+- vllm
+- sparsity
+pipeline_tag: text-generation
+license: llama3.1
+base_model: meta-llama/Llama-3.1-8B
+---
+# Get Started
+Sparse-Llama-3.1 models use 2:4 semi-structured sparsity to deliver 2x model size and compute reduction.
+Explore the [launch blog](https://neuralmagic.com/blog/24-sparse-llama-smaller-models-for-efficient-gpu-inference/) to learn more about Sparse-Llama-3.1 and its potential for efficient, scalable AI deployments.
+You can also find all available models in our [Neural Magic HuggingFace collection](https://huggingface.co/collections/neuralmagic/sparse-llama-31-2of4-673f6e96ae74efa213cf1cff).
+**Looking to build on top of sparse models?** Whether you aim to reduce deployment costs, improve inference performance, or create highly optimized versions for your enterprise needs, Sparse Llama provides the ideal foundation. These models offer state-of-the-art efficiency with 2:4 structured sparsity, enabling cost-effective scaling without sacrificing accuracy.
+[Connect with us](https://neuralmagic.com/book-a-demo/) to explore how we can help integrate sparsity into your AI workflows.
+# Sparse-Llama-3.1-8B-2of4
+## Model Overview
+- **Model Architecture:** Llama-3.1-8B
+  - **Input:** Text
+  - **Output:** Text
+- **Model Optimizations:**
+  - **Sparsity:** 2:4
+- **Release Date:** 11/20/2024
+- **Version:** 1.0
+- **License(s):** [llama3.1](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/blob/main/LICENSE)
+- **Model Developers:** Neural Magic
+This is the 2:4 sparse version of [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B).
+On the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), it achieves an average score of 62.16, compared to 63.19 for the dense model—demonstrating a **98.37% accuracy recovery**. On the [Mosaic Eval Gauntlet](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md) benchmark (version v0.3), it achieves an average score of 53.85, versus 55.34 for the dense model—representing a **97.3% accuracy recovery**.
+### Model Optimizations
+This model was obtained by pruning all linear operators within transformer blocks to the 2:4 sparsity pattern: in each group of four weights, two are retained while two are pruned. In addition to pruning, the sparse model was trained with knowledge distillation for 13B tokens to recover the accuracy loss incurred by pruning. For pruning, we utilize optimized version of [SparseGPT](https://arxiv.org/abs/2301.00774) through [LLM-Compressor](https://github.com/vllm-project/llm-compressor), and for sparse training with knowledge distillation we utilize [SquareHead approach](https://arxiv.org/abs/2310.06927).
+## Deployment with vLLM
+This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend. vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
+## Evaluation
+This model was evaluated on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1) with the [vLLM](https://docs.vllm.ai/en/stable/) engine for faster inference. In addition to the OpenLLM benchmark, the model was evaluated on the [Mosaic Eval Gauntlet](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md) benchmark (version v0.3). The evaluation results are summarized below.
+### Accuracy
+#### Open LLM Leaderboard evaluation scores
+<table>
+    <tr>
+        <td><strong>Benchmark</strong></td>
+        <td style="text-align: center"><strong>Llama-3.1-8B</strong></td>
+        <td style="text-align: center"><strong>Sparse-Llama-3.1-8B-2of4</strong></td>
+    </tr>
+    <tr>
+        <td>ARC-C (25-shot)</td>
+        <td style="text-align: center">58.2</td>
+        <td style="text-align: center">59.4</td>
+    </tr>
+    <tr>
+        <td>MMLU (5-shot)</td>
+        <td style="text-align: center">65.4</td>
+        <td style="text-align: center">60.6</td>
+    </tr>
+    <tr>
+        <td>HellaSwag (10-shot)</td>
+        <td style="text-align: center">82.3</td>
+        <td style="text-align: center">79.8</td>
+    </tr>
+    <tr>
+        <td>WinoGrande (5-shot)</td>
+        <td style="text-align: center">78.3</td>
+        <td style="text-align: center">75.9</td>
+    </tr>
+    <tr>
+        <td>GSM8K (5-shot)</td>
+        <td style="text-align: center">50.7</td>
+        <td style="text-align: center">56.3</td>
+    </tr>
+    <tr>
+        <td>TruthfulQA (0-shot)</td>
+        <td style="text-align: center">44.2</td>
+        <td style="text-align: center">40.9</td>
+    </tr>
+    <tr>
+        <td><strong>Average Score</strong></td>
+        <td style="text-align: center"><strong>63.19</strong></td>
+        <td style="text-align: center"><strong>62.16</strong></td>
+    </tr>
+    <tr>
+        <td><strong>Accuracy Recovery (%)</strong></td>
+        <td style="text-align: center"><strong>100</strong></td>
+        <td style="text-align: center"><strong>98.37</strong></td>
+    </tr>
+</table>
+#### Mosaic Eval Gauntlet evaluation scores
+<table>
+    <tr>
+        <td><strong>Benchmark</strong></td>
+        <td style="text-align: center"><strong>Llama-3.1-8B</strong></td>
+        <td style="text-align: center"><strong>Sparse-Llama-3.1-8B-2of4</strong></td>
+    </tr>
+    <tr>
+        <td>World Knowledge</td>
+        <td style="text-align: center">59.4</td>
+        <td style="text-align: center">55.6</td>
+    </tr>
+    <tr>
+        <td>Commonsense Reasoning</td>
+        <td style="text-align: center">49.3</td>
+        <td style="text-align: center">50.0</td>
+    </tr>
+    <tr>
+        <td>Language Understanding</td>
+        <td style="text-align: center">69.8</td>
+        <td style="text-align: center">69.0</td>
+    </tr>
+    <tr>
+        <td>Symbolic Problem Solving</td>
+        <td style="text-align: center">40.0</td>
+        <td style="text-align: center">37.1</td>
+    </tr>
+    <tr>
+        <td>Reading Comprehension</td>
+        <td style="text-align: center">58.2</td>
+        <td style="text-align: center">57.5</td>
+    </tr>
+    <tr>
+        <td><strong>Average Score</strong></td>
+        <td style="text-align: center"><strong>55.34</strong></td>
+        <td style="text-align: center"><strong>53.85</strong></td>
+    </tr>
+    <tr>
+        <td><strong>Accuracy Recovery (%)</strong></td>
+        <td style="text-align: center"><strong>100</strong></td>
+        <td style="text-align: center"><strong>97.3</strong></td>
+    </tr>
+</table>