GGUF kernels
Compute directly on the packed blocks of a GGUF checkpoint, so a quantized model runs without ever materializing a dense copy of its weights.
Ops
| op | signature |
|---|---|
dequantize |
(blocks, ggml_type, rows, cols, dtype) -> (rows, cols) |
mul_mat_vec |
(blocks, x, ggml_type, out_features) -> (rows, out_features) f32, fused dequant-gemv |
blocks is a GGUF weight exactly as stored: (out_features, bytes_per_row) uint8. mul_mat_vec
takes at most MAX_GEMV_ROWS rows of x; above that, unpack with dequantize and use a dense
matmul. The quant types each backend implements a gemv for are in GEMV_TYPES.
Devices
| backend | torch | targets |
|---|---|---|
| CUDA | 2.11, 2.12 | x86_64-linux, cu126/128/130/132, sm 7.5–12.0 |
| Metal | 2.12, 2.13 | aarch64-darwin |
Where the kernels come from
llama.cpp's ggml, vendored rather than reimplemented.
vendor/UPSTREAM records the revision.
CUDA compiles ggml-cuda's sources; Metal compiles ggml-metal.metal into the embedded metallib.
Only the files listed in build.toml's src are built — the rest of the tree rides along so a pin
bump cannot leave a dangling include.
Updating
python vendor.py --rev <llama.cpp commit> # re-vendor, updates vendor/UPSTREAM
nix run .#build-and-copy # rebuild every variant into build/
vendor.py copies whole trees plus one rename: upstream's mmvq.cu lands as mmvq-impl.cuh,
because gguf_cuda/ggml_dispatch.cu #includes it instead of compiling it as its own translation
unit.