Inference Endpoints
Edit model card


static quants of

weighted/imatrix quants are available at


If you are unsure how to use GGUF files, refer to one of TheBloke's READMEs for more details, including on how to concatenate multi-part files.

Provided Quants

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Link Type Size/GB Notes
PART 1 PART 2 Q2_K 52.2
PART 1 PART 2 IQ3_XS 58.3
PART 1 PART 2 IQ3_S 61.6 beats Q3_K*
PART 1 PART 2 Q3_K_S 61.6
PART 1 PART 2 IQ3_M 64.6
PART 1 PART 2 Q3_K_M 67.9 lower quality
PART 1 PART 2 Q3_K_L 72.7
PART 1 PART 2 IQ4_XS 76.5
PART 1 PART 2 Q4_K_S 80.6 fast, recommended
PART 1 PART 2 Q4_K_M 85.7 fast, recommended
PART 1 PART 2 Q5_K_S 97.1
PART 1 PART 2 PART 3 Q5_K_M 100.1
PART 1 PART 2 PART 3 Q6_K 115.6 very good quality
PART 1 PART 2 PART 3 PART 4 Q8_0 149.5 fast, best quality

Here is a handy graph by ikawrakow comparing some lower-quality quant types (lower is better):


And here are Artefact2's thoughts on the matter:

FAQ / Model Request

See for some answers to questions you might have and/or if you want some other model quantized.


I thank my company, nethype GmbH, for letting me use its servers and providing upgrades to my workstation to enable this work in my free time.

Downloads last month


Downloads are not tracked for this model. How to track
Unable to determine this model’s pipeline type. Check the docs .

Finetuned from

Datasets used to train mradermacher/Eurux-8x22b-nca-GGUF