islamassanov/Qwen3.5-2B-auto-optimized

Qwen3.5-2B, auto-optimized by Fable

This is Qwen3.5-2B auto-optimized by Claude Fable for fast single-request text generation. Fable built and tuned the included qwen35_fast inference engine while keeping Qwen's original BF16 weights unchanged.

Single H100, batch 1, 256 output tokens	Decode tokens/s
Transformers eager	50–51
vLLM 0.29	429–437
vLLM 0.29 + MTP	527–915
Fable-optimized engine	582–848

Across 12 development workloads, the Fable engine delivered 14× the decode speed of Transformers eager and 1.02× the speed of vLLM with MTP (geometric means). On 12 held-out workloads, it reached 528–866 tokens/s and 1.01× vLLM with MTP.

Setup

Use Python 3.12 and an NVIDIA CUDA GPU. Download the model and install its dependencies:

pip install huggingface_hub
hf download islamassanov/Qwen3.5-2B-auto-optimized --local-dir qwen35-fable
pip install -r qwen35-fable/requirements.txt

Run the optimized engine:

import sys
from transformers import AutoTokenizer

path = "qwen35-fable"
sys.path.insert(0, path)
from qwen35_fast import Engine

tokenizer = AutoTokenizer.from_pretrained(path)
prompt = [{"role": "user", "content": "Explain how a CPU cache hierarchy works."}]
input_ids = tokenizer.apply_chat_template(
    prompt, tokenize=True, add_generation_prompt=True, enable_thinking=False
)["input_ids"]

engine = Engine(path, spec_k=2, compile_blocks=True, fused_gdn=True)
output = engine.generate(
    input_ids,
    n_out=512,
    eos_ids={tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|im_end|>")},
)
print(tokenizer.decode(output["tokens"], skip_special_tokens=True))

The original checkpoint also works with Transformers for Qwen's standard text and vision-language workflows; the speed figures above use qwen35_fast.

What Fable changed

• Captured the decode step in a CUDA graph to reduce launch overhead.
• Fused decode operations with torch.compile and Triton Gated-DeltaNet kernels.
• Used the checkpoint's MTP head to propose two tokens per step, accepting each only when the target model makes the same greedy choice.

The optimized path runs batch-1, greedy, non-thinking text generation. The weights and tokenizer come unchanged from Qwen revision 15852e8c16360a2fea060d615a32b45270f8a8fc.

Benchmarks

Results are median decode tokens/s over five runs on one H100 SXM, with 256 generated tokens per prompt. Prompts cover prose, code, and structured output at 128, 512, 2,048, and 8,192 input tokens. Engines ran sequentially in the same session; decode timing starts after the first token.

Development prompts	Transformers eager	vLLM + MTP	Fable engine
Prose, 128–8,192 input tokens	50–51	528–576	582–618
Code, 128–8,192 input tokens	50–51	637–736	661–738
Structured, 128–8,192 input tokens	50–51	824–915	807–848

Full per-prompt results, held-out measurements, and the IFEval regression are in RESULTS.md. The project repository contains the benchmark code and development history.