Dev-4B-GGUF

Dev-4B for llama.cpp. Dev-4B answers questions about a document instantly with a confidence score. When the quick answer is likely wrong, it stops and reasons step by step from the same reading of the document.

It does not run in Ollama or LM Studio. Dev-4B switches a small adapter on partway through each prompt and reads the model's internal state. Neither app exposes either, and current Ollama refuses models with adapters. It runs in llama.cpp itself, through the small server included here, which has the same API and browser playground as Dev-4B's other builds.

Files

File	What it is	Size
Qwen3-4B-Instruct-2507-Q8_0.gguf	The unchanged base model, Qwen3-4B-Instruct-2507 at revision cdbee75, quantised to 8 bits	4.3 GB
Dev-4B-adapter.gguf	Dev-4B's adapter as a llama.cpp activated LoRA, switched on at Question:	132 MB
Dev-4B-head.npz	The decision head, temperatures and router, run in numpy	0.35 MB
src/, serve/, ports/	The server (Python, no PyTorch) and a small C file for the exact mode

Run it

You need Python 3.12 and llama.cpp (brew install llama.cpp on a Mac; elsewhere see llama.cpp's install guide). Memory: about 5 GB.

hf download suhaas-teja/Dev-4B-GGUF --local-dir Dev-4B-GGUF && cd Dev-4B-GGUF
pip install -e .

There are two ways to run it. Both serve the playground at http://127.0.0.1:8008 and the API described in Dev-4B's docs.

Exact mode (tested on a Mac with Homebrew's llama.cpp). This compiles a 150-line C file once against your llama.cpp library:

sh ports/llamacpp/shim/build.sh ./libdmm_llama.dylib
python ports/llamacpp/dev_server.py --backend lib --shim ./libdmm_llama.dylib \
    --base Qwen3-4B-Instruct-2507-Q8_0.gguf --lora Dev-4B-adapter.gguf --head Dev-4B-head.npz

Easy mode (stock llama-server, any platform it runs on). Nothing to compile:

llama-server -m Qwen3-4B-Instruct-2507-Q8_0.gguf --lora Dev-4B-adapter.gguf \
    --embeddings --pooling last -c 8192 -np 1 --port 8090
python ports/llamacpp/dev_server.py --backend server --url http://127.0.0.1:8090 --head Dev-4B-head.npz

	Exact mode	Easy mode
Quick answers	the same as Dev-4B	within about 0.01 on average: stock llama-server switches the adapter on one token early
When it stops to reason	reuses its reading of the document (first words after 0.26 s on a 600-token document)	reads the document again (about 1.6 s)

How close is it to the original?

On 180 test questions on a Mac (M1 Pro), exact mode agrees with the Dev-4B MLX build on 179 decisions (the other is a near-tie). It has the same accuracy (72.35%) and flags the same 35 questions for reasoning. Easy mode agrees on 179 of 180 decisions and flags 34. Reasoning runs at about 34 tokens a second on that Mac.

Example: a refund question that needs arithmetic gets a quick "yes" at 0.64 and a router risk of 0.84, above the 0.516 threshold, so Dev-4B reasons it through and ends with "Answer: Yes" (7.8 s in either mode).

Checksums (SHA-256)

9741f6040faa0065921515143bc9042f5560fcb5eed28b138b0d20ca867f5b2a  Qwen3-4B-Instruct-2507-Q8_0.gguf
ca7c9719eacdb9abfcda97df3d9463575e915e6e4c18c4f4c6f7a2282c9d9b3a  Dev-4B-adapter.gguf

Built with llama.cpp build 11146 (7fe450e19).

Licence

• Dev-4B-adapter.gguf and Dev-4B-head.npz: CC BY-NC-SA 4.0, as Dev-4B's weights (non-commercial; see LICENSE-WEIGHTS.md for why). Attribution: "Dev-4B by Suhaas Teja Vijjagiri".
• Qwen3-4B-Instruct-2507-Q8_0.gguf: Apache-2.0, as the original Qwen model. All credit for it goes to the Qwen team.
• The code: Apache-2.0 (LICENSE).