lukealonso/Kimi-K3-QSRT-K2

This is 2bpw, 8x RTX 6000 (or 6~8x DGX Spark), quant of Kimi K3, with a KLD of only 0.08 and surprisingly strong coding/benchmark performance.

You'll need a vLLM based on: https://github.com/local-inference-lab/vllm/tree/dev/infernal-invocation (docker container soon) b12x for the special kernels required: https://github.com/local-inference-lab/b12x

More details/instructions to follow, but technical information can be found here: https://github.com/local-inference-lab/qsrt/blob/master/docs/qsrt-2bpw-codec.md