fasolack/Qwen2.5-Coder-14B-W4A16

Qwen2.5-Coder-14B-Instruct → W4A16

int4 weights, BF16 activations. Quantized with AutoRound, calibrated on a code / tool-calling / general mix -- not the stock generic-text calibration most quantization repos ship with.

Built specifically to run on a single 12 GB card (an RTX 3080 Ti) behind a coding agent, where the full-precision model (~28 GB) and even most other current-generation Qwen builds don't fit. See the companion project's log entry for why this specific base model, not a newer one, was the right choice.

Why this base model, not something newer

Qwen's current (3.6/3.8) generation doesn't ship a dense model in the 12-14B range -- it jumps from small models straight to 27B+ dense and 30B+ MoE, none of which fit a 12 GB card at any reasonable quantization. Qwen2.5-Coder-14B-Instruct is older but genuinely well-proven, purpose-built for coding, and small enough to actually fit with real context headroom after quantization. Picked over an unverified community distillation of the newer generation for the same reason every build in this family insists on its own eval numbers: don't publish something you haven't actually checked.

Calibration

45%/40%/15%-style splits don't apply as-is here -- this build leans harder into code specifically, since it's headed for a dedicated coding box, not a general-purpose agent driver:

Source	Share	Dataset
Code	55%	ise-uiuc/Magicoder-OSS-Instruct-75K
Tool-calling	30%	NousResearch/hermes-function-calling-v1
General	15%	NeelNanda/pile-10k

512 samples, 2048 tokens each, AutoRound W4A16 (group size 128, symmetric).

BF16 keep-list

Unlike the newer hybrid-attention/MoE builds in this family, this is a plain dense transformer (Qwen2ForCausalLM) -- verified against the real architecture with inspect_layers.py, not assumed. The only layer kept BF16 is lm_head (untied output embedding, cheap insurance for logit quality). No linear-attention gates, no MoE experts, no vision tower to worry about here.

Serving

Single GPU, TP=1 -- this model is small enough that multi-GPU serving would be pure overhead.

Tool-calling needs a custom parser, included in this repo (qwen25coder_flex_tool_parser.py). The base model's own chat template asks for Hermes-style JSON wrapped in <tool_call></tool_call>, but it doesn't reliably produce that exact tag -- verified this is a base-model quirk, not something this quantization introduced, by reproducing it against the raw BF16 checkpoint directly. Across a few different prompts it used <tools> (the tag meant for the tool schema, not the call), <function-call> (invented, appears nowhere in its own template), and once no tag at all. What stayed correct every time was the JSON payload itself. The included parser strips every wrapper spelling actually observed and pulls the JSON object out regardless of what (if anything) wraps it -- run install_tool_parser.sh once against your vLLM environment (source its venv first), then:

vllm serve fasolack/Qwen2.5-Coder-14B-W4A16 \
    --served-model-name qwen2.5-coder-14b-w4a16 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --kv-cache-dtype fp8 \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --tool-call-parser qwen25coder_flex

The stock hermes parser will silently fail on this model -- tool calls land in content as raw text with finish_reason: "stop" instead of a populated tool_calls array. If you don't want to install the custom parser, at minimum know this limitation exists before relying on this model behind an agent.

A LoRA fix was attempted and, on the second try, worked -- with a real cost. See PAPER.md for the full writeup. A LoRA on attention/MLP projections alone left tag compliance at 0/120 (unchanged) despite measurably improving argument correctness and false-positive rate -- root-caused to the frozen lm_head lacking a confident weight for the rare <tool_call> token. Adding lm_head to the LoRA's target modules fixed it completely: 100% tag compliance (120/120), generalizing across every eval category. But a full regression check found real, CI-exceeding drops in HumanEval pass@1 (68.29% -> 58.54%) and GSM8K strict-match (72.10% -> 64.97%), while MMLU-Pro and GSM8K flexible-extract stayed within noise. This is not a free fix -- which build to use depends on deployment context: an agent loop with no custom parser installed may prefer the tag-fixed build despite the coding-capability cost; a deployment already running qwen25coder_flex_tool_parser.py loses little by staying on this shipped build. This build (the one described by this card) is published here. The tag-fixed build (Qwen2.5-Coder-14B-W4A16-toolfix-v2) is not -- it remains a local experimental artifact, not a published alternative.

Evaluation

Measured with EleutherAI lm-evaluation-harness against the served endpoint. No BF16 baseline was measured for this build (unlike the 35B build in this family) -- this model already has well-established public numbers from Qwen themselves and prior community quantizations; the point of measuring here was to validate this specific calibration and quantization, not to re-establish what the base model can do.

Task	This build (int4 W4A16)
GSM8K (flexible-extract)	80.21% ± 1.10%
HumanEval (pass@1)	68.29% ± 3.64%
MMLU-Pro (100/subject)	41.71% ± 1.29%

The MMLU-Pro number reads low next to this family's 27B/35B builds (both in the 80s) -- that's not a quantization problem, it's this model: smaller, older-generation, and picked specifically to fit 12 GB of VRAM, not for general-knowledge breadth. HumanEval (the number that actually matters for what this build is for) is solid.

Coverage gap, stated plainly: none of these three benchmarks exercise tool-calling. The custom parser fix described above was verified separately, by hand, against the served model (a single-tool call and a two-tool multi-call, both correctly extracted) -- not by this benchmark suite. If you're relying on tool-calling specifically, that's worth knowing wasn't covered by an automated eval here.

Limitations and intended use

• Calibrated for code and tool-calling; general-knowledge tasks may be marginally worse than a generically-calibrated W4A16 build of the same model.
• Built for a single small-VRAM GPU. If you have real multi-GPU headroom, a newer-generation model (this family's Qwen3.8-27B or Qwen3.6-35B-A3B builds) will outperform this one.
• On a 12 GB card, context headroom is tight enough that any non-model VRAM usage costs real, measurable capacity -- a GPU-accelerated desktop session sharing the card is the most common culprit. Run this class of card headless where possible; see STATUS.md for the concrete before/after this made on the deployment box (roughly 3x the usable context at otherwise-identical settings).
• Internal build, not independently safety-tuned beyond what the base model already carries.