neural-nova/Qwen3-4B-Instruct-2507-optimized

Optimized Transformers — Qwen/Qwen3-4B-Instruct-2507

This package contains an auto-generated optimized build of Qwen/Qwen3-4B-Instruct-2507 produced by the NeuralNova Auto-Optimization pipeline. The forward and backward passes of the model's bottleneck operations have been replaced with custom CUDA kernels, improving inference throughput over stock Transformers.

This repo does not host model weights. It ships the optimization code only; weights are still pulled from Qwen/Qwen3-4B-Instruct-2507 at load time.

Optimized ops: RMSNorm (27.9x standalone speedup), MLP (1.7x standalone speedup) Throughput improvement: 1.48x serving throughput (37.2 → 55.0 tok/s), 1.61x finetune throughput (4625 → 7451 tok/s) Output quality: PASS — 20/20 prompts passed HalluAnalyst checks; no hallucinations or factual errors detected

---

⚠️ Kernel binaries — read before using

kernels/RMSNorm and kernels/MLP ship as precompiled .so binaries only — the CUDA source (kernel.cu) is not included in this release. They will only load on a matching stack:

• Python 3.12 (cp312)
• CUDA 13.0, torch 2.11.0
• GPU compute capability sm_80 / sm_86 / sm_89 / sm_90 (A100, H100, RTX 3080–4090)

On any other stack, pip install will succeed but importing the extension will fail or crash. If you need a different environment, you'll need to rebuild from source — source is not currently published here.

---

Installation

Install in order:

Step 1 — Install Python dependencies

pip install -r requirements.txt

Step 2 — Install CUDA kernels

Pre-built binaries are included — no compiler or CUDA toolkit required (see compatibility warning above):

pip install kernels/RMSNorm
pip install kernels/MLP

Step 3 — Apply the patched Transformers file

This build modifies exactly one file in huggingface/transformers v5.8.1: modeling_qwen3.py (Qwen3RMSNorm.forward and Qwen3MLP.forward only, verified by diff against the upstream release). Install upstream transformers at that version, then drop in the patched file from patched_transformers/:

pip install transformers==5.8.1
python -c "import transformers, os, shutil; d = os.path.dirname(transformers.__file__) + '/models/qwen3'; shutil.copy('patched_transformers/modeling_qwen3.py', d)"

Step 4 — Install flash-attn

The patched Transformers uses FlashAttention-2 for the attention op. flash-attn compiles CUDA kernels from source — install build dependencies first and use MAX_JOBS for parallel compilation (otherwise the build can take 8+ hours):

# Install build dependencies (ninja enables parallel C++ compilation — required)
pip install packaging psutil ninja

# Verify ninja is working before proceeding
ninja --version && echo $?
# Must print a version string and exit code 0.
# If exit code is non-zero, re-run: pip install --force-reinstall ninja

# Install flash-attn with parallel jobs (takes 60–90 min on first install)
MAX_JOBS=8 pip install flash-attn --no-build-isolation

# Verify
python -c "import flash_attn; print('flash-attn OK, version:', flash_attn.__version__)"

---

Usage

Use patched Transformers as you would the standard transformers library — the CUDA kernels are injected transparently:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")

inputs = tokenizer("Hello, how are you?", return_tensors="pt").to("cuda")
model = model.cuda()
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Serving

To serve the model with transformers serve:

transformers serve --model Qwen/Qwen3-4B-Instruct-2507 --port 8000

---

Benchmark Results

Metric	Baseline	Optimized	Speedup
Serving throughput (tok/s)	37.22	54.96	1.48x
Serving avg latency (s)	20.9	14.3	-32%
GSM8K accuracy (50 samples)	0.94	0.92	within noise
Finetune throughput (tok/s)	4625	7451	1.61x

Configuration: 30 requests, concurrency=4, max_tokens=256, bfloat16, Qwen/Qwen3-4B-Instruct-2507

---

Notes

• This package was generated for Qwen/Qwen3-4B-Instruct-2507 — kernels are tuned for this model's specific layer shapes and dtypes (hidden_size=2048, intermediate_size=6912, 28 decoder layers).
• System requirements: Python 3.12, CUDA 13.0, GPU with sm_80 / sm_86 / sm_89 / sm_90 architecture (A100, H100, RTX 3080+, RTX 4090).
• The patched file in patched_transformers/ contains targeted modifications only to Qwen3RMSNorm.forward and Qwen3MLP.forward, based on transformers v5.8.1. modular_qwen3.py is unmodified from upstream and is not included here.
• Attention, DecoderLayer, and RotaryEmbedding custom kernels were excluded because they lack KV-cache support required for auto-regressive generation.