Smaller GGUF for stock llama.cpp (2.5 GB)
#3
by mike18 - opened
Hi Jared, we've published GGUF builds of Kev-4B that load in stock llama-server (llama.cpp from 9 October 2026 or newer) and answer on /v1/systemone:
https://huggingface.co/turintech/Kev-4B-Artemis-GGUF
The default file is 2.5 GB against 8.4 GB for the BF16 and needs 7.6 GiB of GPU memory instead of 13.1 GiB (NVIDIA RTX 5090). It gives the same answer as the original on 93.0% of 2,000 test decisions, close to the ggml-org Q4_K_M (93.2%) while 16% smaller, with accuracy unchanged on those decisions (1.7 points lower on held-out decisions). The card has the full comparison.
These are independent conversions by TurinTech. Thanks for releasing Kev under Apache-2.0. Happy to answer questions here.