qwen36-vision-tower

The vision tower of Qwen3.6-35B-A3B, extracted unmodified from the base checkpoint and packaged so it can be grafted onto sprappcom/malay35b-pqm (or any other Qwen3.6-35B-A3B-derived .pqm) through the prism-engine vision bridge.

Source	Qwen/Qwen3.6-35B-A3B (Apache-2.0), model.visual.* tensors only
Parameters	446.6 M
Precision	BF16
File	vision_tower.safetensors — 893,175,200 bytes, sha256 160afeed128baaa9918c8c87da4d6d8892c179eba8d0079b2f1697ddfc42773b
Architecture	Qwen3-VL style ViT: depth 27, hidden 1152, 16 heads, patch 16, spatial merge 2, projected to out_hidden_size=2048 (the text model's n_embd)
Output	[num_image_tokens, 2048] embeddings that prism-engine splices into the prompt in place of `<

What is in this repo

File	Purpose
vision_tower.safetensors	The weights (strict-loaded, no missing/unexpected keys)
config.json	Minimal config: only the vision_config block + the vision token ids the bridge/engine read. Deliberately not a full text-model config
preprocessor_config.json	Qwen2VLImageProcessor settings (patch 16, merge 2, mean/std 0.5)
server.py	The vision bridge server (Python, Unix-socket, length-prefixed protocol). Includes the 1024 px longest-side resize cap
requirements.txt	Exact pinned Python deps that were verified working
LICENSE	Proprietary terms (same as the .pqm repos) + Apache-2.0 attribution for the base weights

Requirements

• prism-engine prism_server built with vision-bridge wiring (PRISMX_VISION_BRIDGE_SOCK support). prism-engine is proprietary and is not included here — contact sprappcom@gmail.com.
• A Qwen3.6-35B-A3B-derived text model in .pqm form, e.g. sprappcom/malay35b-pqm (malay35b.pqm + its malay35b.tok tokenizer sidecar).
• Python 3.10+ with the pinned deps below (verified on Python 3.12.3). CUDA optional (CPU mode works, see footprints).
• The Python venv must live on a filesystem that allows executing shared libraries (torch ships .so files) — not on a noexec tmpfs such as /dev/shm on many containers.

Run

Every command below was run literally, in this order, on an A100 80 GB box (2026-08-26) and produced the outputs shown — see Verified end-to-end at the bottom.

1. Bridge (Python)

# Either the HF CLI ...
hf download sprappcom/qwen36-vision-tower --local-dir ./qwen36-vision-tower
# ... or plain curl (no CLI needed):
mkdir -p qwen36-vision-tower && cd qwen36-vision-tower
T=https://huggingface.co/sprappcom/qwen36-vision-tower/resolve/main
for f in config.json preprocessor_config.json server.py requirements.txt LICENSE vision_tower.safetensors; do
  curl -fL -o "$f" "$T/$f"
done
sha256sum vision_tower.safetensors   # 160afeed128baaa9918c8c87da4d6d8892c179eba8d0079b2f1697ddfc42773b

python3 -m venv .venv && source .venv/bin/activate      # put .venv on a normal (exec) filesystem
pip install --index-url https://download.pytorch.org/whl/cu121 torch==2.5.1 torchvision==0.20.1
pip install -r requirements.txt

export PRISMX_VISION_TOWER_SD=$PWD/vision_tower.safetensors   # the weights
export PRISMX_VISION_CONFIG_DIR=$PWD                          # dir holding config.json + preprocessor_config.json
export PRISMX_VISION_DEVICE=cuda:0                            # or "cpu"
export PRISMX_VISION_BRIDGE_SOCK=/tmp/prismx_vision_bridge.sock
python3 server.py
# -> [vision_bridge] real vision tower loaded on cuda:0: 446571248 params, strict-load verified, processor=Qwen2VLImageProcessor
# -> [vision_bridge] listening on /tmp/prismx_vision_bridge.sock

Set CUDA_VISIBLE_DEVICES if you need to pin the GPU (PRISMX_VISION_DEVICE=cuda:0 then means "the first visible GPU"). On a single GPU the bridge and prism_server share the same device. The bridge is a long-lived process: supervise it (systemd/supervisord) next to prism_server.

2. prism_server (proprietary)

export PRISMX_PQM_STANDALONE=1
export PRISMX_PQM=/path/to/malay35b.pqm
export PRISMX_TOKENIZER=/path/to/malay35b.tok
export PRISMX_VISION_BRIDGE_SOCK=/tmp/prismx_vision_bridge.sock   # same path as the bridge
export PRISMX_STRIP_THINK_OPENAI=1      # hide <think>…</think> reasoning from message.content (recommended)
export PRISMX_MIN_TEMPERATURE=0.15      # greedy decoding can collapse on this stack
prism_server /path/to/malay35b.pqm 0.0.0.0:8080 --max-batch 1 --max-seq 8192
# A100 80 GB with CPU-RAM expert offload (~10.9 GB VRAM):  add  --n-cpu-moe 36 --moe-cache-experts 2048
# small GPU (8 GB):                                         add  --n-cpu-moe 38 --moe-cache-experts 512
# -> [prism_server] GPU lane ready; max_batch=1 max_seq=8192 ...
# -> [prism_server] listening on http://0.0.0.0:8080

Argument order matters: the first positional argument is the model path (in standalone mode it is only used as the served model id, the weights come from PRISMX_PQM), the second is the bind address. Passing the address first makes the server treat it as the model path and silently bind the default 0.0.0.0:8080.

--max-seq must hold the image tokens + your text + the model's reasoning + the answer: one image contributes ceil(W/32) * ceil(H/32) tokens after the 1024 px cap (640x488 → 300, 1024x768 → 768) plus <|vision_start|>/<|vision_end|>. 4096 is the practical minimum; 8192 is what the verified setup used.

3. Request shape (OpenAI content-array)

# Put the JSON in a file: a base64 photo is easily >100 KB, which overflows the shell's
# argument limit if you inline it in `curl -d "..."` ("Argument list too long").
B64=$(base64 -w0 photo.jpg)
cat > req.json <<REQ
{
  "model": "malay35b",
  "max_tokens": 2000,
  "temperature": 0.3,
  "messages": [{
    "role": "user",
    "content": [
      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,$B64"}},
      {"type": "text", "text": "Describe this image."}
    ]
  }]
}
REQ
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'content-type: application/json' -d @req.json

Content parts are spliced in array order. image_url.url must be a data:image/<fmt>;base64,<payload> URI (a bare base64 payload is also accepted); remote http(s):// image URLs are not fetched. Any format PIL decodes (JPEG/PNG/WebP) works. Max encoded image size: 20 MiB. usage.prompt_tokens in the response includes the image tokens (a 640x488 photo + a 12-token question → prompt_tokens: 312), which is how you can tell the image was actually spliced.

max_tokens must cover the model's reasoning. The engine always primes a <think> block; for image prompts the model reasons for ~450–1200 tokens before answering. With PRISMX_STRIP_THINK_OPENAI=1 that reasoning is stripped from message.content, so if max_tokens runs out inside it you get content: "" with finish_reason: "length". Measured on the NYT photo: max_tokens: 300 → empty every time; max_tokens: 1500 → finish_reason: "stop" in 7 of 9 runs and still empty in 2 of 9 (the reasoning length varies run to run at temperature 0.3). Use max_tokens >= 2000 for image requests. Streaming ("stream": true) works with images but the first content delta only arrives after the reasoning closes (12–19 s measured for a 312-token image prompt); the SSE stream itself is well-formed (data: {...} chunks, final usage, data: [DONE]). A per-request enable_thinking: false switch (closed <think> prime, first content in ~2.4 s, 5–8× fewer completion tokens) exists in prism-engine PR #11 but is not in the shipped build yet — and in the measured runs it made image answers noticeably more prone to invented headlines/dates, so keep thinking on when grounding matters.

Malay answers: a Malay question alone ("Apakah yang anda lihat dalam gambar ini?") was answered in English in every measured run, as was "Terangkan gambar ini dalam Bahasa Melayu.". Add a Malay system message — {"role":"system","content":"Anda ialah pembantu yang sentiasa menjawab dalam Bahasa Melayu."} — and the image description comes back in coherent Malay.

If the bridge is down, requests containing an image fail immediately (4 ms measured) with an explicit HTTP 400: {"error":{"message":"vision bridge unreachable: connect <sock>: No such file or directory (os error 2)","type":"invalid_request_error","code":"invalid_image"}} — the engine never silently drops the image. Text-only requests keep working while the bridge is down, and image requests resume as soon as the bridge is back (no engine restart needed).

Measured footprints

Measured 2026-08-26 on prism-a100b (A100 SXM4 80 GB, driver 595.71, the pinned venv above, malay35b.pqm standalone with --n-cpu-moe 36 --moe-cache-experts 2048 --max-batch 1 --max-seq 8192, bridge and engine on the same GPU). VRAM is the nvidia-smi delta for that GPU.

Setup	Footprint
Bridge alone, GPU (BF16, PRISMX_VISION_DEVICE=cuda:0)	1,304 MiB VRAM (nvidia-smi per-process 1.3 GB)
+ prism_server malay35b (A100, 36 CPU-MoE blocks)	+10,890 MiB VRAM, +15.2 GiB page-locked host RAM
Peak during a 780-token image request (4000x3000 photo, capped to 1024x768)	+282 MiB over idle (12,476 MiB total for bridge+engine)
Bridge embed time, 640x488 JPEG → 300 tokens	33–38 ms (3 reps)
Bridge embed time, 4000x3000 JPEG → resized 1024x768 → 768 tokens	260–290 ms (3 reps, includes decode + LANCZOS resize)
Engine prefill-only (max_tokens: 1), text 22 tokens	0.21 s
Engine prefill-only, 312 tokens (300 image)	1.20 s → image adds ≈ +1.0 s
Engine prefill-only, 780 tokens (768 image)	2.21 s → image adds ≈ +2.0 s
Decode with an image in context	~40 tok/s, same as text-only (706 completion tokens in 17.7 s; text-only 83 tokens in 2.05 s)
Tower alone, CPU (PRISMX_VISION_DEVICE=cpu)	works; noticeably slow per image, fine for single-user
malay35b .pqm with --n-cpu-moe 38 + tower on an RTX 4060 Laptop 8 GB	7.0 GB VRAM total (5.8 GB text + ~1.1 GB tower)

Verified end-to-end (2026-08-26, A100)

All via the OpenAI content-array request above, temperature 0.3, max_tokens 1500 (use 2000, see above):

Test	prompt_tokens	Result
Real NYT front-page photo (640x488), "Describe this image. What kind of document is it?"	312	"This is the front page of The New York Times … Masthead … slogan All the News That's Fit to Print … LATE CITY EDITION" — 706 completion tokens, 17.7 s, no CJK/garbage
4000x3000 solid blue JPEG, "What colour is this image?"	780	"solid blue background … royal blue or cobalt" — no OOM, 7.8 s
4000x3000 solid red JPEG	780	"solid red background" — 8.7 s
Same NYT photo, Malay system prompt + "Apakah yang anda lihat dalam gambar ini?"	333	Coherent Malay: "Ini adalah foto muka surat depan The New York Times … Nama Surat Khabar … Edisi: Late City Edition …" (29 s, 1182 tokens)
Text-only prompts before and after the image requests (EN + MS)	18–22	Unaffected, finish_reason: "stop", ~40 tok/s
Streaming image request	312	173 SSE chunks, 0 malformed, final usage, data: [DONE]

Known limitations

• Dense small text is not read reliably. The NYT photo's masthead, slogan and edition line were read correctly every time, but the body headlines were described as "placeholder / blank / pixelated" and one Malay run hallucinated a date. Treat the model as a describer, not an OCR engine.
• Flat vector-style logo images graft poorly (near-uniform colour fields, no photographic texture): the model tends to describe them vaguely or wrongly. Photographs and screenshots with real texture work markedly better. (Solid single-colour test images are identified correctly.)
• Large-photo resize cap: 1024 px longest side. server.py downscales anything larger (LANCZOS) before patching. Without this cap a 4000x3000 phone photo produced enough patch tokens to OOM/hang the bridge; the cap bounds VRAM and latency. Fine detail in very large images is lost.
• Reasoning tax. The shipped engine has no per-request "thinking off" switch; every image answer costs ~450–1200 reasoning tokens first (see max_tokens above). The enable_thinking: false engine change (prism-engine PR #11) removes the tax but measurably increases confabulation on images.
• Two crash domains. The bridge is a separate Python process talking to prism_server over a Unix socket. If either dies, image requests fail until it is restarted; supervise both (e.g. supervisord). The bridge serialises tower forward passes with a lock — one image at a time.
• Images only; no video path is wired even though temporal_patch_size=2 is in the config.
• This is the milestone-1 Python bridge, not a native Rust/CUDA port. Expect one Python/torch process (~2-3 GB host RAM) alongside the engine.

License

other / bcz-proprietary — see LICENSE. Same terms as the sprappcom/*-pqm repos. The weights themselves are an unmodified extraction from Qwen/Qwen3.6-35B-A3B (Apache-2.0); that attribution is preserved. The bridge, packaging and prism-engine integration are proprietary to BCZ Singapore Pte Ltd. Contact: sprappcom@gmail.com.