squad-0.8b-GGUF

A team of models. On your device. In milliseconds. ▶ Live demo

squad-0.8b (revision 58c22ba, October 2026) for llama.cpp (CPU, Metal, CUDA, Vulkan): a new trunk, a router and four 2-bit LoRA experts, games, e2e (decisions of a test agent driving a web app), multilingual and documents: a MoQLE (Mixture of Quantized LoRA Experts). Stock llama.cpp only: no patched server. Method and limitations: squad-0.8b.

Results (squad-0.8b, measured on the NVIDIA runtime)

	squad-0.8b (previous)	squad-0.8b (this revision)	basal-omni-pico
Benchmark Heaven (public, 231)	77.1	76.6	75.8
JevExtra	71.3	71.7	64.6
JevSpatial	75.8	79.7	68.4
Rules-7	63.8	74.1	69.5
JevBench accuracy	84.9	86.4	84.0
JevBench coherence	70.6	76.4	74.2
KLEJ (Polish, 7 tasks)	70.9 (trunk only)	73.4	75.3
games_bench (9 unseen games)	11.3	10.4	7.5
e2e on 8 real web apps (61 tests × 3 runs)	10 / 183	34 / 183	–

The GGUF build gives the same answers (checked below); read the main card for what these benchmarks do and do not show.

Checked against the NVIDIA reference (qev.py of squad-0.8b, fast CUDA engine, specialists on; 333 requests: the 231 Benchmark Heaven items, 40 game states, 36 simulated web-app steps, 16 multilingual and 10 document requests; qev_gguf.py on Metal, Q8_0):

• the same route on 100% of requests (333 / 333: 243 trunk, 34 games, 36 e2e-tester, 19 multilingual, 1 documents);
• the same top answer on 99.5% of the 401 routed questions (max |Δp| 0.077, mean 0.006);
• with each expert forced on every request: 99.3% (trunk), 98.5% (games), 99.5% (e2e-tester), 99.3% (multilingual), 99.0% (documents) the same top answer.

The differences come from Q8_0 weights and 2-bit lora_A as exact Q2_K (lora_B in Q8_0) vs the bf16 CUDA engine.

With the router: qev_gguf.py

pip install llama-cpp-python==0.3.36 tokenizers decider-ai==1.9.0 huggingface_hub safetensors numpy pillow
# Apple silicon: CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python==0.3.36 ...

from qev_gguf import QEVGGUF   # qev_gguf.py in this repo

m = QEVGGUF.load("labqoat/squad-0.8b-GGUF", "qev-0.8b-Q8_0.gguf", specialists=True)   # images on by default
m.decide(state, questions); m.last_routes     # ["trunk"] / ["games"] / ["e2e-tester"] / ["multilingual"] / ["documents"]

python qev_gguf.py --serve --specialists           # POST http://127.0.0.1:8017/v1/systemone, text and images
python qev_gguf.py --serve --specialists --no-images

The router reads the text prompt through the main model's first 6 layers (a second context on the same weights, stopped there by llama.cpp's eval callback) and a 1.1 MB head (router.safetensors), top-1 of {trunk, games, e2e-tester, multilingual, documents}; the answer pass then runs with the chosen adapter. router.min_p_slots in specialists.json holds per-expert confidence thresholds: the e2e expert is used only when the router gives it at least 35%, else the trunk answers. Images load by default (mmproj-qev-0.8b-f16.gguf); a request with an image never goes to the games expert.

Speed on an Apple M4 Pro (Metal, Q8_0, router on, one request at a time, no caches): median 90 ms per request over the 333 requests above; 54 ms for a Benchmark Heaven question, 36 ms for a short multilingual request, 86 ms for a game state, 408 ms for a web-app step (pages of 2-4k tokens). Repeated pages are faster with the prefix cache (on by default).

Stock llama-server: one expert, always on

Stock llama-server (llama.cpp b11786 or newer; older builds, e.g. Homebrew's 0.5.0, have no /v1/systemone) cannot run the router, but it can serve one expert on every request:

hf download labqoat/squad-0.8b-GGUF lora-e2e-tester-2bit.gguf --local-dir .
llama-server -hf labqoat/squad-0.8b-GGUF:Q8_0 --lora lora-e2e-tester-2bit.gguf        # text and images
llama-server -hf labqoat/squad-0.8b-GGUF:text-Q8_0 --lora lora-e2e-tester-2bit.gguf   # text only, questions jointly

POST /v1/systemone then answers with the e2e expert on every request (or lora-games-2bit.gguf, lora-multilingual-2bit.gguf, lora-documents-2bit.gguf; without --lora it is the trunk alone). The default :Q8_0 file takes images and answers each question in its own prompt; :text-Q8_0 is text only and answers several questions jointly. Checked on the Mac with llama.cpp b11786: both files with the e2e adapter give the same top answer as qev_gguf.py with that expert forced on 100% of 40 questions (Benchmark Heaven and web-app steps; max |Δp| 0.043).

Files

File	What it is
qev-0.8b-Q8_0.gguf	the trunk (new in this revision), default: takes images in llama-server, one prompt per question
qev-0.8b-text-Q8_0.gguf, qev-0.8b-bf16.gguf	the trunk, text-only builds for llama-server (questions answered jointly)
qev-0.8b-vision-Q8_0.gguf	the same file as qev-0.8b-Q8_0.gguf
mmproj-qev-0.8b-f16.gguf, mmproj-qev-0.8b-Q8_0.gguf	the vision projectors (unchanged: the vision tower is the same; qev_gguf.py loads the f16 one by default)
lora-games-2bit.gguf, lora-e2e-tester-2bit.gguf, lora-multilingual-2bit.gguf, lora-documents-2bit.gguf	the experts: rank 64 on ffn gate / up / down of layers 12–23, lora_A as exact Q2_K (8.1 MB each)
router.safetensors, specialists.json	the router head and how the pieces fit (slots, min_p_slots)
qev_gguf.py, add_systemone_metadata.py, systemone.jinja, systemone_vision.jinja, qev_config.json	runner, metadata tool, prompt templates, temperatures

The trunk files were converted from squad-0.8b@58c22ba with llama.cpp b11786 (convert_hf_to_gguf.py --no-mtp) and carry the previous files' metadata (templates, decision type, temperatures, pre-tokenizer) unchanged; Q8_0 is the reference ggml quantizer (it reproduces the previous Q8_0 file from the previous bf16 one byte for byte).

Not re-evaluated for this revision: image input (the trunk changed; the vision tower and projector files are the same) and the probability temperatures (the previous version's).

License: Apache-2.0.