-
RedHatAI/DeepSeek-R1-Distill-Llama-8B-FP8-dynamic
Text Generation • Updated • 327 • 2 -
RedHatAI/DeepSeek-R1-Distill-Llama-70B-FP8-dynamic
Text Generation • Updated • 104k • 7 -
RedHatAI/DeepSeek-R1-Distill-Qwen-32B-FP8-dynamic
Text Generation • Updated • 1.66k • 6 -
RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8
Text Generation • Updated • 553 • 1

Neural Magic
AI & ML interests
LLMs, optimization, compression, sparsification, quantization, pruning, distillation, NLP, CV
Recent Activity
The Future of AI is Open
If you are looking for compressed models to run with vLLM, they have been moved to the RedHatAI organization. We are looking forward to continue publishing optimized models for open source use!
Neural Magic helps developers in accelerating deep learning performance using automated model compression technologies and inference engines. Download our compression-aware inference engines and open source tools for fast model inference.
- vLLM: A high-throughput and memory-efficient inference engine for at-scale deployment of performant open-source LLMs
- LLM Compressor: HF-native library for applying quantization and sparsity algorithms to llms for optimized deployment with vLLM
In this profile we provide accurate model checkpoints compressed with SOTA methods ready to run in vLLM such as W4A16, W8A16, W8A8 (int8 and fp8), and many more! If you would like help quantizing a model or have a request for us to add a checkpoint, please open an issue in https://github.com/vllm-project/llm-compressor.
Collections
14
-
RedHatAI/granite-3.1-2b-instruct-quantized.w4a16
Text Generation • Updated • 138 -
RedHatAI/granite-3.1-2b-instruct-quantized.w8a8
Text Generation • Updated • 133 -
RedHatAI/granite-3.1-8b-instruct-quantized.w4a16
Text Generation • Updated • 251 • 1 -
RedHatAI/granite-3.1-8b-instruct-quantized.w8a8
Text Generation • Updated • 229 • 1
spaces
4
Quant Llms Text Generation
Quantized vs. Unquantized LLM: Text Generation Comparison
Llama 3 8B Chat Deepsparse
Chat with a Llama-3-8B-Instruct model efficiently using text
Llama 2 Sparse Transfer Chat Deepsparse
Sparse Llama Gsm8k
Solve math problems with chat-based guidance