DeepSeek V4 Flash 0731 — Vision (projector-only graft)

STATUS: WORK IN PROGRESS. GRPO training is live right now. The projector uploaded here is cycle 555 of a 12,000-cycle target. Rewards and answer quality are still improving; expect this repo to be updated with later checkpoints.

This repo contains only the vision components of a projector-only vision graft onto DeepSeek V4 Flash:

frozen text backbone — deepseek-ai/DeepSeek-V4-Flash-0731 (revision 9e165c30e2704aec5d9d593cce3eebd58bbef1cb), referenced, not included (~176 GB — download it separately)
frozen MoonViT vision tower (vision_tower.safetensors, 329 tensors, 834 MB) — Kimi-K2.6 MoonViT, filtered from the exolabs/Kimi-K2.6-vision export
trainable ~40 M parameter projector (mm_projector.safetensors, 80 MB) — the only trained part: LayerNorm(1152) → 2×2 patch-merge concat (4608) → Linear 4608×4608 → GELU → Linear 4608→4096, WebBrain "patchmerger" architecture, spliced into the prompt embeddings at image positions

What does "0731" mean?

The backbone is the DeepSeek V4 Flash 0731 GA release (July 31 checkpoint of the text-only MoE): 43 layers (2 sliding-window + 21 compressed-sparse + 20 heavily-compressed attention), shared-KV MQA (1 KV head / 64 query heads / head_dim 512), MoE top-6 of 256 experts, fp8 attention + FP4 experts (expert_dtype: fp4, served through W4A16 fused MoE kernels). It is completely frozen here — this repo ships no backbone weights; the serving directory is assembled by symlinking the base-model shards next to the files in this repo (instructions below).

Attribution

WebBrain — the projector architecture, its warm-start weights, the 64-id routing palette, and the NaViT featurization all derive from webbrain-one/DeepSeek-V4-Flash-0731-Vision-NVFP4. The projector here was initialized from their mm_projector.safetensors and then trained further with GRPO.
Original method — the projector-graft recipe originates from Harry Partridge (@part_harry_, "GLM 5.2 With Vision"), via the community glm-vision reproduction lineage: 0xSero/fable-glm-vision (weights) and 0xSero/glm-vision-sft-mix (data mix).
MoonViT tower — Moonshot AI's Kimi-K2.6 vision encoder, via the exolabs/Kimi-K2.6-vision export. The kimi_k25_*.py / media_utils.py processor files are the Kimi processor code.
Base model — deepseek-ai/DeepSeek-V4-Flash-0731 (MIT license, inherited here).

Training

GRPO reinforcement learning (run grpo_wb_partial), projector-only — the optimizer holds projector parameters exclusively; backbone and tower cannot move by construction. 6 rollouts per prompt group, reward on answer correctness, thinking mode on, 512 new tokens. Data mix (web/art/OCR): multiple-choice art (WikiArt via the glm-vision mix), OCRVQA / TextVQA / IAM handwriting / rendered text (the_cauldron), and UI screenshots with a dense click-distance reward (wave-ui-25k, ShowUI-desktop). SFT previously validated that this projector grounds vision (in-distribution teacher-forced loss 1.15 ≈ val 1.17, clicks pixel-exact); RL is improving task reward from the WebBrain warm start.

At cycle 555, per-cycle mean rewards fluctuate roughly 0.1–0.5 (multiple-choice art peaking around 0.84 on good cycles, screenshot click reward ~0.4). Uploaded checkpoints:

file	keys	use
`mm_projector.safetensors`	`mm_projector.pre_norm.* / proj.0.* / proj.2.*`	drop-in for vLLM serving (this layout)
`training_checkpoint/projector_grpo_wb_partial_cycle555.safetensors`	`pre_norm.* / linear_1.* / linear_2.*`	raw trainer checkpoint (same tensors, trainer key names)

Architecture notes you must not break

Hash routing / routing palette. The first 3 MoE layers of DSV4 route by token id (tid2eid[input_ids]), not by a learned gate. The token ids placed at image positions therefore decide which experts process image tokens in layers 0–2. This checkpoint cycles the 64-id routing_palette from config.json across image positions (WebBrain scheme). Train and serve must assign identical image-position ids — the vLLM model code in this repo does this for you; do not modify it.
Image span markers: begin_of_image_token_id 128129, media_placeholder_token_id 128130, end_of_image_token_id 128131.
Featurization is NaViT-style: bicubic resize on 0–255 floats, zero-pad to 28-px multiples, Kimi normalization stats, ~2 M pixel budget (preprocessor_config.json + dsv4_vision_processing.py are the exact training-side processors).

Serving in vLLM — exact reproduction

Verified end-to-end on 4× RTX PRO 6000 Blackwell Max-Q 96 GB (SM 12.0), TP4: boots to Application startup complete, serves image+text requests including a 896×896 image (~1027 image tokens) through the MoE prefill with no kernel errors.

1. Assemble the model directory

Download the base model, then symlink its shards next to this repo's files:

export HF_HOME=/data/huggingface   # or wherever you keep blobs
BASE=$(hf download deepseek-ai/DeepSeek-V4-Flash-0731 --revision 9e165c30e2704aec5d9d593cce3eebd58bbef1cb)
THIS=$(hf download jarrelscy/deepseek-v4-flash-0731-vision)

M=/data/huggingface/dsv4flash-models/0731-vision-wb
mkdir -p "$M" && cd "$M"
ln -s "$BASE"/model-000*.safetensors .
ln -s "$BASE"/encoding "$BASE"/tokenizer.json "$BASE"/tokenizer_config.json \
      "$BASE"/generation_config.json "$BASE"/LICENSE .
cp "$THIS"/config.json "$THIS"/preprocessor_config.json \
   "$THIS"/model.safetensors.index.json "$THIS"/chat_template.jinja \
   "$THIS"/dsv4_vision_processing.py "$THIS"/kimi_k25_processor.py \
   "$THIS"/kimi_k25_vision_processing.py "$THIS"/media_utils.py \
   "$THIS"/mm_projector.safetensors "$THIS"/vision_tower.safetensors .

model.safetensors.index.json (included here) is the merged index mapping all 72,652 tensors — backbone tensors to the symlinked shards, vision tensors to the two safetensors files above.

2. vLLM code: image + overlay + patches

Serving runs the production vLLM container image voipmonitor/vllm:gilded-gnosis-v20-vllm749050e-si8110e3e-fi801d57a-cu132-20260730-r14 (vLLM base commit 749050e per the tag) with two modifications, both shipped in this repo:

Vision model overlay (vllm/overlay/) — 7 files bind-mounted read-only over the installed vLLM package (see the launch script for the exact mount targets). These add DeepseekV4VisionForConditionalGeneration (model_type: deepseek_v4_vision): MoonViT + WebBrain patchmerger projector grafted onto the DSV4 text model, text weights delegated to the native DSV4 loader, 64-id palette hash routing at image positions (with the int32/int64 palette-dtype fix for vLLM V1), and the fp8 → deepseek_v4_fp8 quant-method promotion for the composite vision config.
W4A16 MoE kernel fix (vllm/patches/sparkinfer_w4a16_tile_k.patch) — applied to the in-image sparkinfer/moe/_shared/kernels/w4a16/kernel.py at container start. Without it, request-time MoE prefill crashes with force_tile_config fc2 tile (tile_k=32, tile_n=512) does not fit problem N/K=4096/512: the FC2 "ultra-wide" tile override hard-codes tile_k=32, which is rejected by the launch-boundary _candidate_tile_fits floor (tile_k >= 64). The patch gates the override behind that same floor so it falls back to the fit-checked (fc2_tile_n=256, fc2_tile_k=64) slab — numerically identical, only the FC2 output-tile width changes. This bug is independent of the vision path (it hits pure-text MTP too).

Provenance patches: the overlay code was developed on a local vLLM fork, branch dsv4flash-vision, head commit 53963a0c5 — vllm/patches/0001…0004-*.patch are those commits (git format-patch), and 0005-dsv4flash-vision-worktree-uncommitted.patch is the final uncommitted fix set (palette dtype + quant-method promotion). The overlay files are the authoritative, boot-tested code — they were regenerated against the container's vLLM version (vllm/regenerate.sh shows how); the patches are reference material if you want to port the graft onto a different vLLM tree.

3. Launch

vllm/launch_wb_rollout.sh is the exact boot-tested launcher. Edit the paths at the top (OVERLAY, MODEL, the kernel.py.patch mount) to your locations, then:

PORT=8001 TP=4 bash vllm/launch_wb_rollout.sh

Core of what it runs (see the script for the full env — the B12X_* / SPARKINFER_* / CUTE_DSL_ARCH=sm_120a variables are tuned for SM 12.0 Blackwell workstation GPUs; adjust arch-specific ones for other hardware):

# inside the container, after applying the kernel patch (the script exits if the patch fails):
vllm serve /models/dsv4-vision-wb \
  --served-model-name deepseek/v4flash \
  --host 0.0.0.0 --port 8001 \
  --trust-remote-code \
  --kv-cache-dtype fp8 --block-size 256 --load-format safetensors \
  --tensor-parallel-size 4 --moe-backend b12x --linear-backend b12x \
  --gpu-memory-utilization 0.90 --max-model-len 16384 --max-num-seqs 16 \
  --max-num-batched-tokens 4096 --max_cudagraph_capture_size 256 --enable-chunked-prefill \
  --enable-prefix-caching \
  --attention-backend B12X_MLA_SPARSE \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
  --no-enable-flashinfer-autotune \
  --enforce-eager

Note: the palette rewrite at image positions is position-dependent — keep the launch script's prefill/batching settings as shipped (--max-num-batched-tokens 4096 covers a full ~2 M px image prompt in one prefill chunk).

4. Sending images

Standard OpenAI chat completions with image content; the chat template (chat_template.jinja) expands the image span, the processor produces the patches:

curl http://localhost:8001/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "deepseek/v4flash",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
    {"type": "text", "text": "What does the sign say?"}
  ]}],
  "max_tokens": 128
}'

Sanity check that image ingestion works: prompt token count must scale with image size (text-only ~16 → 448 px ~275 → 896 px ~1043 tokens).

What works / known limits

Works

Image+text prefill through the full MoE stack (verified up to ~1027 image tokens in one prompt)
Grounding is faithful: in a serving spot-check, 3/6 in-distribution OCRVQA answers were exact matches to gold (misses were fine-grained genre questions), matching the training stack's measured projector quality
UI click grounding ("respond with the click coordinate as (x, y)") — trained mid-run, emission and accuracy improving

Limits

Training is incomplete (cycle 555 / 12,000) — this is a research checkpoint, not a release
Fine-grained text OCR is weak / out-of-distribution (~0.25 exact-match on the OCR val set); author-name reads and dense small text often fail
Synthetic inputs far from the web/art/OCR training mix (e.g. solid-color fields) give unreliable answers
Video path untested; the config's video plumbing is inherited from the Kimi processor and unused

License

The backbone is MIT (DeepSeek). The MoonViT tower inherits Moonshot AI's Kimi-K2.6 license terms; the projector weights and serving code in this repo are released under MIT.

Downloads last month: 7

Model tree for jarrelscy/deepseek-v4-flash-0731-vision

Base model

deepseek-ai/DeepSeek-V4-Flash-0731

Adapter

(3)

this model