prometheusAIR/DeepSeek-V4-Flash-Vision-Exp-GGUF

DeepSeek-V4-Flash-Vision-Exp — GGUF (multimodal, imatrix)

Imatrix GGUFs of deepseek-ai/DeepSeek-V4-Flash-Vision-Exp that actually do vision — four rungs from 66.9 to 108.7 GiB. It also ships the importance matrix all four rungs were built with, and the patch llama.cpp currently needs to run them.

⚠️ This needs a patched llama.cpp

Stock llama.cpp cannot load these files, and upstream PR #28133 alone crashes on the first image. Both problems and the fix are described in Why a patch is needed. Build instructions are below.

Files

Four rungs, all built from the same importance matrix, which is included.

rung	size	bits/weight¹	notes
IQ1_M	66.9 GiB	~1.9	smallest; most aggressive, least validated
IQ2_XXS	78.8 GiB	~2.2	fits one 96 GB card at full 1M context
IQ2_S	95.8 GiB	~2.7
IQ3_XXS	108.7 GiB	~3.1	highest quality here

Shard naming is DeepSeek-V4-Flash-Vision-Exp-<RUNG>-0000{1..4}-of-00004.gguf.

also included	size
mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf	890 MiB	vision tower — required for images, works with any rung
DeepSeek-V4-Flash-Vision-Exp-imatrix.gguf	470 MB	the importance matrix all four were built with
dsv4-vision-routing.patch	9 KB	apply to llama.cpp + PR #28133

¹ Effective average over the whole file, not the expert type alone. The rung name is the dominant expert type; every rung mixes several — see Recipes. Note general.file_type reads Q6_K in all of them: that field records the base type passed to llama-quantize, and cannot express per-tensor overrides.

Will this run on my GPU?

Weights only — add ~13 GiB for KV/compute at a full 1M context, or far less at shorter contexts. Keep the vision tower on CPU with --no-mmproj-offload unless you have room; it is a one-off cost per image (466M params over ≤384 visual tokens).

your VRAM	rung
96 GB (1×)	IQ2_XXS at full 1M ctx (measured 96.9 GB peak), or IQ2_S/IQ3_XXS at reduced ctx
80 GB (1×)	IQ1_M, or IQ2_XXS with a reduced --ctx-size
64 GB (1×)	IQ1_M at reduced ctx
128–192 GB (2×)	IQ3_XXS comfortably
less, or CPU	any rung with --n-cpu-moe / -ngl tuning; CPU-only works, slowly

Build

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
# PR #28133 adds the vision tower and the image token-block layout
git fetch origin pull/28133/head:pr28133 && git checkout pr28133
# this repo's patch adds modality-conditional MoE routing (see below)
git apply /path/to/dsv4-vision-routing.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-mtmd-cli

Run

Substitute your chosen rung for IQ2_XXS below.

./build/bin/llama-server \
  --model DeepSeek-V4-Flash-Vision-Exp-IQ2_XXS-00001-of-00004.gguf \
  --mmproj mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf --no-mmproj-offload \
  -ngl 999 --ctx-size 1048576 -fa on -b 2048 -ub 512 -np 1 --no-mmap --jinja

-ub 512 is needed to reach 1M context on a 96 GB card. -ub 2048 roughly doubles prefill but adds ~4.2 GB of compute buffer, which will not fit alongside a speculative draft; drop the draft if you want the larger ubatch.

Quick check with llama-mtmd-cli:

./build/bin/llama-mtmd-cli \
  -m DeepSeek-V4-Flash-Vision-Exp-IQ2_XXS-00001-of-00004.gguf \
  --mmproj mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf \
  --image photo.jpg -p "What is in this image?"

Why a patch is needed

DeepSeek-V4-Flash-Vision carries a second MoE router bias, ffn.gate.bias_vl, used only for image tokens. The reference implementation selects experts with

scores + torch.where(input_ids >= vocab_size, bias_vl, bias)

and on the three hash-routed layers image tokens abandon the tid2eid lookup entirely for a learned top-k over scores + bias_vl.

Two consequences:

1. Converters that drop bias_vl produce a model that routes image tokens by text preferences. Measured across all 43 layers, the two biases are uncorrelated (mean Pearson r = −0.026): they encode genuinely different expert preferences. This repo keeps the tensor, as blk.N.exp_probs_b_vl.bias, which is why stock llama.cpp reports wrong number of tensors; expected 1371, got 1328.

2. Without the routing fix, decoding an image aborts. An image arrives as an embeddings batch, and llm_graph_input_embd::set_input only writes the token tensor when ubatch->token is set — so the hash layers gather tid2eid[t_inp_tokens] from uninitialised memory and hit GGML_ASSERT(i01 >= 0 && i01 < ne01). PR #28133 touches no file under src/, so it hits this too.

The patch adds FFN_EXP_PROBS_B_VL to the deepseek4 and dflash architectures and selects the bias per batch. Because llama.cpp delivers an image as its own homogeneous embeddings batch — every position in it is an image token, markers included — this is exact for the batches mtmd produces, not an approximation.

Recipes

All four share the same non-expert treatment: attention, shared experts and dense at Q6_K; token_embd and output at Q8_0; routers (ffn_gate_inp, exp_probs_b, exp_probs_b_vl) at BF16/F32. They differ only in the routed experts:

rung	ffn_gate_exps, ffn_up_exps	ffn_down_exps blk 0–9, 33–42	ffn_down_exps blk 10–32
IQ1_M	IQ1_M	IQ2_XS	IQ2_XXS
IQ2_XXS	IQ2_XXS	IQ3_XXS	IQ2_XS
IQ2_S	IQ2_S	IQ3_S	IQ3_XXS
IQ3_XXS	IQ3_XXS	IQ3_S	IQ3_S

ffn_down gets the higher rung on the first ten and last ten blocks because that is where quantization damage concentrates.

The importance matrix covers 614,400 tokens (75 × 8192-token chunks) with full expert coverage — no expert went uncalibrated, which matters for a 256-expert MoE.

FP8 source weights were dequantized to BF16, not Q8_0 — BF16 represents the E4M3 significand exactly once the power-of-two E8M0 block scale is applied, so this avoids a second lossy step on 365 attention and shared-expert tensors.

Known limitations

• In-span attention visibility is not implemented. The reference lets tokens inside an [IMAGE_START, IMAGE_END] span see each other bidirectionally and feeds that to the DFlash sparse-attention indexer; this build falls back to the causal window. One small image is unaffected in practice, but larger images, multiple images and long contexts are unmeasured.
• No KL-divergence ladder has been measured. The rungs are ordered by bits, which is the expected ordering, but this repo makes no measured quality claim between them. IQ1_M in particular is aggressive and unvalidated behaviourally.
• Testing so far is single-image. Multi-image and interleaved prompts are untested.
• This is an experimental upstream model, and the vision support in PR #28133 is unmerged and may change.

Verification

Built on a single RTX PRO 6000 Blackwell (96 GB). IQ2_XXS tested end to end with DeepSeek's own inference/examples/images/carrots.jpeg:

"A pile of fresh orange carrots with green tops on a white background."

profile	prefill	decode
image, -ub 512, Q2_K DSpark draft	135.8 tok/s	129.4 tok/s (63.9% draft acceptance)
text, same	177.9	140.9 (68.2%)

License

MIT, inherited from the base model.