shahjenil/Docket-3B-GGUF

Docket-3B — GGUF Q8_0

Q8_0 conversion of Docket-3B, the BF16 release. No retraining or calibration refitting.

Docket returns choice, yes/no and score decisions with probabilities over the allowed answers. Use the Docket decision backend to construct the trained prompt and score the candidate tokens.

Serve it

With docket-serve, on a Mac, Linux or Windows machine (Python 3.10+; no compiler, Xcode, CUDA or PyTorch), a CPU, or a small GPU:

git clone https://github.com/shah-jenil/docket-serve.git docket && cd docket
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt     # prebuilt llama.cpp, Metal on Apple Silicon
.venv/bin/docket-serve                        # http://127.0.0.1:8080, downloads this repo on first start
python3 scripts/smoke_test.py                 # in a second terminal: PASS

The download includes the GGUF, tokenizer files, chat template and docket_calibration.json. To serve a local copy instead, point .venv/bin/docket-serve --backend gguf --model at this folder or the .gguf inside it, and keep all files together. For CPU inference add --n-gpu-layers 0; --max-model-len 4096 lowers memory (default context 8192 tokens). Setup details: GGUF guide. Request shapes, endpoints and limits: API reference.

Which release to use

Repo	Format	Use it for
Docket-3B	BF16 safetensors	GPU serving with vLLM; reference quality
Docket-3B-GGUF	GGUF Q8_0	Mac (Metal), CPU or a small GPU with llama.cpp

Paired core comparison

Measured on an RTX 3090 with 1326 fixed held-out/general evaluation examples and unchanged calibration. Numbers below are workflow-weighted argmax accuracy on this subset, not full-suite results. Score-level accuracy uses argmax; the API returns expected score, measured separately in core_comparison.json.

Slice	Questions	BF16 Transformers	Q8 llama.cpp
enterprise	531	90.84%	90.85%
reasoning	251	75.68%	75.28%
logical_verification	160	100.00%	100.00%
boolq	128	82.81%	83.59%
wanli	128	74.22%	73.44%
clinc150	128	76.56%	78.12%

Overall prediction agreement: 98.87%. Mean calibrated distribution total-variation distance: 0.007289.

The included comparison also contains BF16 llama.cpp results to separate runtime effects from quantization. Latency uses warm sequential uncached requests on an NVIDIA GPU; M3 latency has not been measured. The HTTP API passed choice/noul/score, 255-choice, 10-level score, Unicode, multi-question and context/error checks. CPU-only execution was smoke-tested. This does not establish accuracy on every workflow or input length.

Provenance

Pinned base revision: e459acceac81e5fe67c07d9cfc72329a332e7eb1. Adapter SHA-256: 8c3623db324aaa76cf3d942d9c7909bd68a611ced7740d94c2594c2b5ab46a19. Conversion source revisions and runtime packages: conversion_environment.json. Weights and sidecar file checksums: artifact_manifest.json.

Apache-2.0. See LICENSE and NOTICE. Docket is not affiliated with IBM or TypeSafe AI.