Transformers
English
Inference Endpoints
Saily_220B-GGUF / README.md
mradermacher's picture
auto-patch README.md
7dedcfd verified
metadata
base_model: deepnight-research/Saily_220B
datasets:
  - tiiuae/falcon-refinedweb
  - EleutherAI/pile
  - meta-math/MetaMathQA
language:
  - en
library_name: transformers
license: llama2
no_imatrix: 'GGML_ASSERT: llama.cpp/ggml.c:16553: i != GGML_HASHTABLE_FULL'
quantized_by: mradermacher

About

static quants of https://huggingface.co/deepnight-research/Saily_220B

Usage

If you are unsure how to use GGUF files, refer to one of TheBloke's READMEs for more details, including on how to concatenate multi-part files.

Provided Quants

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Link Type Size/GB Notes
PART 1 PART 2 Q2_K 76.9
PART 1 PART 2 IQ3_XS 85.5
PART 1 PART 2 Q3_K_S 90.1
PART 1 PART 2 IQ3_S 90.4 beats Q3_K*
PART 1 PART 2 IQ3_M 93.5
PART 1 PART 2 PART 3 Q3_K_M 100.6 lower quality
PART 1 PART 2 PART 3 Q3_K_L 109.5
PART 1 PART 2 PART 3 IQ4_XS 112.7
PART 1 PART 2 PART 3 Q4_0 117.7 fast, low quality
PART 1 PART 2 PART 3 Q4_K_S 118.6 fast, recommended
PART 1 PART 2 PART 3 Q4_K_M 125.3 fast, recommended
PART 1 PART 2 PART 3 Q5_K_S 143.8
PART 1 PART 2 PART 3 PART 4 Q5_K_M 147.7
PART 1 PART 2 PART 3 PART 4 Q6_K 171.4 very good quality
P1 P2 P3 P4 P5 Q8_0 221.8 fast, best quality

Here is a handy graph by ikawrakow comparing some lower-quality quant types (lower is better):

image.png

And here are Artefact2's thoughts on the matter: https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9

FAQ / Model Request

See https://huggingface.co/mradermacher/model_requests for some answers to questions you might have and/or if you want some other model quantized.

Thanks

I thank my company, nethype GmbH, for letting me use its servers and providing upgrades to my workstation to enable this work in my free time.