Model Card for Qwen3.5-9B (2-bit GGUF)

A 2-bit GGUF build of Qwen3.5-9B, packaged to run entirely on device. The tokenizer, the merge table, the chat template, the vision tower and the sampling card are all embedded in the weight file, so nothing calls out to a server at inference time and there are no sibling files to keep in sync.

This repository holds the quantized weights only, not training data or the original checkpoint.

This file uses a non-standard block type and stock llama.cpp will not read it. The weights are stored in a custom 2-bit format that upstream ggml does not implement, so the file is intended for a companion on-device runtime that does. If you are looking for a build that runs in llama.cpp, Ollama or LM Studio today, use one of the standard-quant GGUF releases instead.

Files

File	Purpose
Qwen3.5-9B-Q2_E8.gguf	everything: weights, tokenizer, chat template, vision tower, sampling card

One file, nothing beside it. The sampling parameters travel inside the GGUF, so a runtime never has to guess them or borrow another model's settings.

Model Details

Model Description

Qwen3.5-9B is a dense hybrid. Its layers alternate three Gated DeltaNet linear-attention blocks to one full attention block. The linear blocks carry a small fixed-size recurrent state instead of a growing key/value cache, so memory stays flat as context grows, while the periodic attention blocks preserve exact long-range recall.

Vision is included. A 27-block vision encoder is packaged in the same file, so image input works from this build alone.

A multi-token-prediction layer is present, which a runtime that supports self-speculative decoding can use as a drafter; one that does not will ignore it.

• Developed by: Qwen, Alibaba (base model); quantization by leok7v
• Model type: Hybrid Gated DeltaNet + attention causal language model
• Language(s): English and the languages of the base model
• License: Apache 2.0, inherited from the base model
• Quantized from model: Qwen/Qwen3.5-9B

Model Sources

• Repository (this build): https://huggingface.co/leok7v/Qwen3.5-9B
• Base model: Qwen/Qwen3.5-9B

Uses

Direct Use

On-device chat, summarization, drafting, question answering over supplied text, and description of supplied images, in offline or privacy-sensitive settings where prompts must not leave the device.

Out-of-Scope Use

Anything the base model's licence and card exclude. This build adds no alignment, no safety tuning and no filtering of its own; it is the upstream model in a different numeric format. Not a basis for legal, medical, financial or safety decisions, and not intended for high-stakes automated decisions without human review.

Bias, Risks, and Limitations

The base model's biases and failure modes carry over unchanged.

Two bits is aggressive, and it shows. Outputs will not match the upstream model token for token, and the gap is widest where the model was already uncertain. In practice the structure of an answer survives better than its details: a chain of reasoning can be laid out correctly and still contain an arithmetic slip, and long multi-step derivations are where errors accumulate.

Anyone depending on this build should evaluate it on their own task rather than assume it behaves like the full-precision model or like a 4-bit one.

Recommendations

Treat generations as drafts to verify, not as fact. For anything where accuracy matters, and for arithmetic in particular, check the result.

How to Get Started with the Model

The tokenizer, the chat template and the sampling card are embedded, so a runtime reads all of them from the GGUF. The chat template documents the roles, the thinking block and the tool-call format.

The base model card recommends these sampling parameters, and the embedded sampling card carries them:

mode	temperature	top_p	top_k	presence_penalty
Thinking	1.0	0.95	20	1.5
Instruct (non-thinking)	0.7	0.80	20	1.5

reasoning_effort is supported by the template; low and medium cost fewer tokens per turn than the default.

Technical Specifications

Model Architecture and Objective

Property	Value
Objective	Autoregressive next-token prediction
Hidden size	4096
Layers	32 (Gated DeltaNet + attention, three to one) + 1 MTP
Attention heads	16 query / 4 key-value (grouped-query)
Head dimension	256 (partial rotary, 64)
Linear-attention state	128, 16 groups, inner size 4096
Feed-forward	12288
Vocabulary	248320
Context length	up to 262144 tokens
Vision encoder	27 blocks, width 1152, 768 px, patch 16

Weight format

4.13 GiB, 776 tensors, GGUF v3, 16384-byte tensor alignment.

Type	Tensors	Covers
2-bit	200	the transformer trunk
F16	112	the vision encoder
BF16	8	the multi-token-prediction layer
Q4_0	2	the embedding table and the output head
F32	454	norms, layer scalars, the recurrent-state parameters

The 2-bit type is a custom ggml block: each group of eight weights is snapped to the nearest point of an E8 lattice codebook and stored as one 16-bit index, which is exactly two bits per weight. That is why stock llama.cpp cannot read the file.

Citation

Please cite the upstream Qwen3.5-9B model. See the base model card at Qwen/Qwen3.5-9B for the canonical reference and license.

Model Card Contact

leok7v on Hugging Face.