sebika123123/Qwen3.8-27B-Fast-Coder

Qwen3.8-27B Fast Coder

Makes the Qwen3.8-27B AI model answer about twice as fast on your own PC when you use it as a coding assistant, and it still does the same work. This page is a how-to plus the proof (tested settings, which file fits which graphics card, and a small test that shows the speed-up costs nothing in quality).

This repo has no model weights. The speed comes from how the model is run (guessing several tokens ahead and copying repeated text), not from different weights. Get the GGUF from unsloth/Qwen3.8-27B-GGUF (we recommend UD-IQ3_XXS for a 16 GB card) and run it with the settings below. Everything here was measured on one RTX 5070 Ti; see the limits at the end.

Start here (the easy way, about 10 minutes, no command line)

You need a PC with an NVIDIA graphics card that has 16 GB of memory (12 GB works with a smaller file, see "Which file for which card" below) and Unsloth Studio, a free app that runs AI models on your own PC (install guide: unsloth.ai/docs).

1. Open Unsloth Studio and go to the Model hub.
2. Get the model: search unsloth/Qwen3.8-27B-GGUF and download the file called UD-IQ3_XXS (10.9 GB).
3. Turn on the speed-up before you load it, in the run settings: set Speculative Decoding to MTP+Ngram and Draft Tokens to 4, then tick Remember for this model.
4. Press Load. Chat with it in Studio, or connect a coding tool (Cline, OpenCode, Continue, ...) to the address Studio shows you.

That is all. If you want the details, the proof, other graphics cards, or you run llama.cpp yourself, keep reading.

your graphics card memory	take this file	expect
16 GB	UD-IQ3_XXS	the numbers on this page
12 GB	UD-Q2_K_XL	works, a little less careful with tools (not tested on a 12 GB card)
24 GB or more	UD-Q4_K_M	should be at least as good, but slower (not tested here)
8 GB	none	this model is too big for it

What the words mean. Speculative decoding = the model guesses the next few words and checks the guesses all at once, so you get exactly the same answer, faster. MTP = a small guessing helper that is built into this model. n-gram = copying text that already appeared (code you are editing, tool output). GGUF = the file format local AI models come in. Draft tokens = how many words it guesses ahead.

The short version

On one RTX 5070 Ti (16 GB), these settings roughly double the speed of Qwen3.8-27B on agent work (reading files, editing, running tests) and halve the time a job takes. The same jobs get solved.

13 easy agent jobs x 3 attempts	thinking on	thinking off
no guessing (plain llama-server)	50 tok/s, 23.9 s per job	50 tok/s, 12.0 s per job
these settings	98 tok/s, 12.5 s per job	113 tok/s, 5.9 s per job

--spec-type ngram-mod,draft-mtp --spec-draft-n-max 4 -ctk q8_0 -ctv q8_0 -np 1 -c 32768 -fa on -ngl 99 --jinja

• draft-mtp uses the model's own multi-token-prediction head to guess the next 4 tokens; ngram-mod copies text that already appeared (file contents, tool output). The main model checks every guess in one pass, so the output is the same, only faster.
• For a 16 GB card use unsloth's UD-IQ3_XXS (10.9 GB). Smaller files do not finish jobs sooner: with the guessing on, they save memory, not time, and they need more tokens per job.
• Things that did not help here: guessing more than 4 tokens ahead, the DFlash 2 drafter (+4-7% speed for 1.1 GB more memory), 2-bit files.

All numbers are from one machine and three attempts per job; read docs/METHOD.md for the limits and results/RESULTS.md for every table.

Quick start

1. A llama.cpp build with --spec-type draft-mtp and ngram-mod (the measurements used build 11696, the copy that ships with Unsloth Studio; a current llama.cpp release should do).
2. A Qwen3.8-27B GGUF that has an MTP layer inside, for example Qwen3.8-27B-UD-IQ3_XXS.gguf (10.9 GB). Check any file with python gguf_info.py file.gguf ("nextn_predict_layers = 1" means it has one).
3. Start it:

# Windows
set MODEL=D:\models\Qwen3.8-27B-UD-IQ3_XXS.gguf
set LLAMA_SERVER=D:\llama.cpp\llama-server.exe
launch\serve.bat

# Linux / macOS
MODEL=~/models/Qwen3.8-27B-UD-IQ3_XXS.gguf LLAMA_SERVER=~/llama.cpp/build/bin/llama-server launch/serve.sh

The server speaks the OpenAI API at http://127.0.0.1:8812/v1 (no key), with tool calling (--jinja). Point your coding tool at that address. See launch/README.md for what every flag does and the optional variables.

In Unsloth Studio

Load the file from the Model hub (unsloth/Qwen3.8-27B-GGUF, variant UD-IQ3_XXS), then set these in the Run settings panel and tick Remember for this model:

field	value
Context Length	32768 (16384 if memory is tight; Studio's estimate may warn at 32768 on a 16 GB card)
Parallel Slots	1
KV Cache Dtype	q8_0
Speculative Decoding	MTP+Ngram
Draft Tokens	4

Studio's automatic mode decides by free GPU memory. In an early test on our card it left MTP off (about 49 tok/s); in a later test with the stock UD-IQ3_XXS file and everything on Auto it kept MTP by cutting the parallel slots from 4 to 1 (about 80 tok/s on four generic prompts, 8k context). Choosing Speculative Decoding yourself does not depend on that memory estimate, and MTP+Ngram with 4 draft tokens is +25-30% over MTP alone on agent work (see the results). The labels come from Studio's own interface and may differ slightly between versions. Studio launches exactly --spec-type ngram-mod,draft-mtp --spec-draft-n-max 4 with these values (checked with Studio's own loader).

A dflash-*.gguf sidecar in the model's folder is found by Studio's Auto mode, but on a 16 GB card Auto drops it ("the drafter does not fit") and falls back to the n-gram lookup alone (50 tok/s), so this repo does not ship one.

Which file for which card

Memory is what the setup itself needs at 32k context with the 8-bit cache, measured as GPU memory in use minus the 2.1 GB the Windows desktop took on the test machine. Rows marked "not measured" are estimates from file sizes.

card	file (unsloth UD- quants)	memory needed	status
24 GB and up	Q4_K_M (16.5 GB) or Q5_K_M (19.8 GB)	about 17-21 GiB	not measured here. A third party found Q4_K_M equal to BF16 on agentic benchmarks (quesma); bigger files decode slower.
16 GB	IQ3_XXS (10.9 GB)	11.4 GiB	measured: 98 tok/s thinking on, 113 off. IQ3_S (12.0 GB, 12.5 GiB) also fits at 32k and buys nothing.
12 GB	Q2_K_XL (9.8 GB) or IQ2_S (8.4 GB + 1.4 GB MTP file)	10.8 / 10.1 GiB	measured on the 16 GB card, not on a 12 GB one. Tool-call discipline is weaker (Q2_K_XL never asked for a missing detail). Needs a light desktop, maybe CTX=16384.
8 GB	not recommended	IQ2_XXS needs 9.1 GiB	measured, does not fit, and solved fewer jobs (49/57 against 52-54) at 3x the tokens per job.

The IQ2 and IQ1 files carry no MTP layer; pass unsloth's separate MTP/mtp-Qwen3.8-27B-Q4_0.gguf as MTP_FILE.

What is in here

launch/        serve.bat, serve.sh and a README with every flag explained
bench.py       A/B speed test: starts llama-server per setting, sends the same prompts, reads llama.cpp's own timings
gguf_info.py   what is inside a GGUF (architecture, quant types, whether it has an MTP layer), header only
eval/          the agent eval: 19 jobs with hidden checks + 23 tool-call cases, self-testing harness
results/       every table (RESULTS.md) and the raw summary of each run (runs/)
docs/METHOD.md how it was measured, and where it can mislead you

The eval

eval/ gives the model six tools in a throwaway project folder and a job (rename a function everywhere, fix a crash and the wrong numbers behind it, add an option across five files, follow a project's contribution rules, find the setting that really applies when the docs are wrong, implement a spec with many edge cases, and so on). When the model stops, a hidden check decides whether the job is really done. It records solved jobs, tool-call errors, tokens and llama.cpp's own timings, so a speed setting, a quant or a fine-tune can be judged on "same results, less time".

cd eval
python agent_eval.py selftest                          # no server needed; checks the harness itself
python agent_eval.py run --url http://127.0.0.1:8812   # against any running OpenAI-compatible server
python agent_eval.py run --serve agent4 --seeds 3      # start llama-server from a bench.py config and test it
python agent_eval.py compare <run folder> <run folder> --md

Its run tool executes the model's node / python code in the throwaway folder with your user's rights; see eval/README.md. One of the 19 jobs (t05) uses a synthetic stand-in file, not the file used for some earlier private runs, so its numbers can differ slightly from the tables.

Limits

One GPU, one llama.cpp build, three attempts per job, mostly easy jobs: the eval shows that a setup still works as an agent and catches big drops, but it cannot rank near-equal files. Speculative decoding is lossless in theory and the runs agree (every guessing setting solved the same jobs), which is evidence, not proof. Other cards and operating systems were not measured.

Credits and licences

The code here is MIT (LICENSE). The model and GGUF files are not part of this repo and have their own licences: Qwen3.8 by the Qwen team, the quants by unsloth, the DFlash 2 drafter by incoai, and llama.cpp by ggml-org. The tool-call cases follow the style of the Berkeley Function-Calling Leaderboard; they contain no code or data from it.