stamsam/Instella-Prometheus-GGUF

Instella-Prometheus GGUF 🔥

The code-fire, packed for local inference.

This is the dedicated GGUF release of Instella-Prometheus, a merged, standalone coding model built from AMD's Instella-MoE-16B-A3B-SFT. Prometheus is a 16B-total / ~2.8B-active MoE model, distilled for direct, code-first answers under a bare user prompt:

• No system prompt required
• No thinking-tag suppression or decoder bans
• No LoRA adapter or PEFT dependency
• Designed for llama.cpp and compatible GGUF runtimes

Choose your quantization

Quantization	File	Approx. size	Use when
Q8_0	Instella-Prometheus-Q8_0.gguf	16.9 GB	You want near-full-weight quality
Q4_K_M	Instella-Prometheus-Q4_K_M.gguf	9.4 GB	You want the best quality/size balance
Q3_K_M	Instella-Prometheus-Q3_K_M.gguf	8.2 GB	You need the lowest memory footprint of these three

All files are in the repository root so the Hugging Face Hub can identify and display them as GGUF quantized variants.

llama.cpp

# Example: Q4_K_M
./llama-cli \
  -m Instella-Prometheus-Q4_K_M.gguf \
  -p "Write a Python function to merge overlapping intervals." \
  -n 1024

For chat frontends, use the model's built-in GGUF metadata and chat template when supported by the runtime. The intended contract is simply: user task in, clean answer out.

What changed from base?

On a 12-task bare-user Python suite:

Metric	Base Instella	Instella-Prometheus
Code blocks	2 / 12	12 / 12
Syntax valid	2 / 12	12 / 12
Functional passes	2 / 12	10 / 12
Think-tag leakage	12 / 12	0 / 12
Natural EOS	2 / 12	12 / 12

The full Transformers release, training details, dataset provenance, and evaluation notes are available in the canonical model repository.

License

Apache-2.0. See the canonical repository for base-model and dataset acknowledgements.

The fire belongs to them. The torch is yours.