medic17/koboldcpp-cuda-builds

koboldcpp builds with machine code for Ampere and Ada GPUs

Builds of koboldcpp's CUDA backend from unmodified release tags, with one change to the compiler flags.

Why

The official release compiles its GPU code with -gencode arch=compute_NN,code=compute_NN. That emits only PTX (portable intermediate code), so the NVIDIA driver compiles it the first time a model is loaded on a machine and caches the result in ~/.nv/ComputeCache. On a freshly created cloud GPU instance that cache is empty: measured on an NVIDIA A40 with a 31 GB model, the first launch spent about 55 s in "Starting model warm up", against 0.3 s on later launches. These builds add machine code, so there is nothing to compile: 11 s to ready instead of 69 s, with the same generation speed.

What is different from the release

Only the CUDA 12 flag block of the Makefile:

NVCCFLAGS += -Wno-deprecated-gpu-targets \
             -gencode arch=compute_80,code=compute_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_89,code=sm_89 \
             -DKCPP_LIMIT_CUDA_MAX_ARCH=800

• sm_86 machine code: A40, RTX A6000, RTX 30 series.
• sm_89 machine code: L40, L40S, RTX 6000 Ada, RTX 40 series.
• compute_80 PTX: fallback for any other card, which then compiles on first use as the release does.
• The release's compute_50, compute_61 and compute_75 targets are dropped.

Each tarball records the source commit (SOURCE_COMMIT) and the exact flags (BUILD_FLAGS).

Files

koboldcpp-<version>-cu12-sm86-sm89.tar.gz unpacks to koboldcpp/ with koboldcpp.py, koboldcpp_cublas.so and the files the release bundle carries. Only the CUDA backend is included. Run it with:

python3 koboldcpp/koboldcpp.py --usecuda --model model.gguf

Needs at run time: Python 3 with jinja2 and psutil, the CUDA 12 runtime and cuBLAS libraries (libcublas-12-8 on Ubuntu), and an NVIDIA driver.

Reproduce

build.sh in this repository is the complete build. It runs as a Hugging Face Job:

hf jobs run --flavor cpu-upgrade --timeout 2h --secrets HF_TOKEN \
  -e KCPP_VERSION=1.121 -e HF_REPO=<user>/<repo> \
  nvidia/cuda:12.8.1-devel-ubuntu22.04 \
  bash -c 'apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null && curl -fsSL https://huggingface.co/<user>/<repo>/resolve/main/build.sh | bash'

Licence

koboldcpp is licensed under AGPL-3.0; its bundled GGML, llama.cpp and stable-diffusion.cpp parts are MIT. Both licence texts are inside each tarball. The corresponding source is the upstream repository at the commit in SOURCE_COMMIT, plus build.sh here.