llama.cpp Presets

A small collection of llama.cpp server presets for local inference on Windows 11.

The included launcher supports both the standard CUDA build of llama.cpp and the TurboQuant fork. Models are downloaded from Hugging Face on first launch and cached in D:\models.

Target hardware

These presets are intended for:

Operating system: Windows 11
GPU: NVIDIA GeForce RTX 2080 Super, 8 GB VRAM
System memory: 24 GB RAM

The larger models will not fit entirely in 8 GB of VRAM. The launcher enables CUDA unified memory so llama.cpp can use system RAM, but performance will depend on how much of the model and KV cache can remain on the GPU. The configured context size is an upper limit; start with a shorter context if memory pressure or slowdowns occur.

Included presets

Preset	Hugging Face model	Context	Notes
`Qwen3.6-35B`	`unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_M`	131,072	MoE model with MTP draft speculation
`Qwen3.6-35B-TQ`	`unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_M`	131,072	TurboQuant KV-cache settings
`Qwen3.8-27B`	`unsloth/Qwen3.8-27B-GGUF:UD-IQ2_XXS`	131,072	27B model with MTP draft speculation
`medgemma-27b-it`	`unsloth/medgemma-27b-it-GGUF:Q4_K_M`	65,536	Image input enabled
`granite-4.0-micro-OpenMed`	`unsloth/granite-4.0-micro-OpenMed-GGUF:Q4_K_M`	65,536	General text preset

Common settings enable Jinja chat templates, memory locking, flash attention, reasoning, and a single parallel request. Model-specific settings are defined in preset.ini.

Requirements

Install these on the Windows 11 machine:

Git
CMake
Visual Studio 2022 with C++ build tools
NVIDIA driver and CUDA toolkit compatible with the current llama.cpp checkout

The build scripts expect the following directories:

D:\llama.cpp
D:\llama-cpp-turboquant
D:\models

If you use different locations, edit the paths in the .bat files before running them.

Build

Standard llama.cpp

Run build-llama.cpp.bat from Windows. It pulls the latest source, configures a CUDA build, and creates the Release binaries under:

D:\llama.cpp\build\bin\Release

TurboQuant

Run build-llama.cpp-tq.bat to build the TurboQuant version under:

D:\llama-cpp-turboquant\build\bin\Release

Run the server

Run server.bat from Windows:

server.bat or server.bat --llama — use the standard llama.cpp server
server.bat --tq — use the TurboQuant server

The server listens on all interfaces at:

http://localhost:8080

The launcher passes -hf wrenchpilot/presets, so the server loads the preset repository from Hugging Face. The required model files are downloaded automatically and stored below D:\models.

Benchmark with llama-bench

Run llama-bench.bat from Windows to compare model settings. The script uses the standard llama.cpp build by default and benchmarks the configured Qwen3.6-35B model with these n-cpu-moe values:

32, 34, 40, 48, 56, 64

Each run uses a 512-token prompt, 128 generated tokens, five repetitions, full GPU-layer offload where possible, flash attention, and memory locking. CSV results are saved in a timestamped file under bench-results.

Common examples

rem Benchmark the default Qwen3.6-35B model
llama-bench.bat

rem Compare a custom n-cpu-moe sweep
llama-bench.bat --ncmoe 24,32,40,48,56,64

rem Benchmark Qwen3.8-27B instead
llama-bench.bat unsloth/Qwen3.8-27B-GGUF:UD-IQ2_XXS --ncmoe 0,16,24,32

rem Use the TurboQuant build
llama-bench.bat --tq

rem Compare quantizations from the same Hugging Face repository
llama-bench.bat --quant Q8_0,UD-Q4_K_M

rem Full sweep: 2 model quants x 3 K types x 3 V types x all n-cpu-moe values
llama-bench.bat --quant Q8_0,UD-Q4_K_M --cache-type-k q4_0,q5_0,q8_0 --cache-type-v q4_0,q5_0,q8_0 --ncmoe 32,34,40,48,56,64

rem Compare KV-cache types; TurboQuant keeps K at q8_0 and varies V
llama-bench.bat --tq --cache-type-k q8_0 --cache-type-v turbo4,turbo3,turbo2

Use llama-bench.bat --help for the full option list. --cache-type-k and --cache-type-v each accept comma-separated lists and benchmark every K/V combination. When combined with --quant, the launcher benchmarks every quantization/K/V combination. Comma-separated values also work unquoted from PowerShell. The selected llama-bench.exe must already exist in the Release directory created by the corresponding build script.

llama-bench measures raw prompt-processing and token-generation throughput. It does not load the named section from preset.ini, apply sampling parameters, or exercise the server's MTP draft-speculation configuration. Use server.bat for an end-to-end test of those settings.

Memory and performance notes

An RTX 2080 Super has 8 GB of VRAM; do not assume the 35B or 27B models will run fully GPU-resident.
Unified memory is enabled by server.bat to allow overflow into the 24 GB system-memory pool.
Large context windows consume substantial KV-cache memory. Reduce context length if the process becomes unstable or the machine starts paging.
Use the standard preset first when troubleshooting. Use --tq only with the TurboQuant build.
Only expose port 8080 to trusted networks; the launcher binds to 0.0.0.0.

Files

preset.ini — model presets and shared inference parameters
server.bat — Windows server launcher
llama-bench.bat — llama.cpp throughput benchmark launcher
build-llama.cpp.bat — standard CUDA build script
build-llama.cpp-tq.bat — TurboQuant CUDA build script

Downloads last month: -; Downloads are not tracked for this model. How to track