mradermacher

/

model_requests

Like 205

Independent A/B test with llama.cpp b10218

I tested both projectors with the same Dolphin language model and llama.cpp configuration.

Test environment

• llama.cpp b10218, commit de69995
• Native Linux x86_64 CUDA 12.8 build
• NVIDIA GeForce RTX 5090, 32 GB VRAM
• llama-server with the MTMD multimodal backend

Language model:

Dolphin-Mistral-24B-Venice-Edition-heretic.Q4_K_M.gguf

Original projector

Dolphin-Mistral-24B-Venice-Edition-heretic.mmproj-Q8_0.gguf

The original projector consistently failed during initialization:

clip_init: failed to load model:
operator(): unable to find tensor v.token_embd.img_break

mtmd_init_from_file: error: Failed to load CLIP model
srv load_model: failed to load multimodal model

Patched projector

I changed only the projector and kept the same language model and llama.cpp runtime:

Dolphin-Mistral-24B-Venice-Edition-heretic.mmproj-Q8_0-img-break-F32.gguf

Download command:

hf download LS110824/text_encoders \
  Dolphin-Mistral-24B-Venice-Edition-heretic.mmproj-Q8_0-img-break-F32.gguf

Result

With the patched projector:

• llama-server loaded the model and projector successfully.
• The v.token_embd.img_break error disappeared.
• An image-conditioned inference request completed successfully.
• The server generated a non-empty 336-token response.
• Model and projector loading took approximately 3.47 seconds.
• Inference took approximately 6.44 seconds.
• No CLIP or MTMD loading errors occurred.

Relevant output:

Model loading complete | platform=linux | elapsed 3.47 s

Inference complete |
prompt 6044 tokens |
completion 336 tokens |
total 6380 tokens |
elapsed 6.44 s

This A/B test confirms that adding the missing F32 v.token_embd.img_break tensor fixes projector initialization with llama.cpp b10218 on this system.

hi, help me here please: so just mmproj broken? and how was it fixed? perhaps need to submit a pr to official llamacpp repo so that this doesnt happen in the future? since I assume not only this model might be affected

Hi Thats i Testet and change it

1. TEST TARGET

Text model:
Dolphin-Mistral-24B-Venice-Edition-Q8_0.gguf

Vision projector used as repair base:
Dolphin-Mistral-24B-Venice-Edition-heretic.mmproj-Q8_0.gguf

Repaired experimental projector:
Dolphin-Mistral-24B-Venice-Edition-heretic.mmproj-Q8_0-imgbreak-fixed.gguf

Runtime:
llama.cpp

Observed llama.cpp build during tests:
9761 (721354fbd)

Context:
131072 tokens

Open WebUI:
OpenAI-compatible llama.cpp connection

Memory integration:
MEMORIA context/retrieval path through Open WebUI

2. INITIAL FAILURE

The Q8 multimodal projector failed during CLIP/Pixtral initialization.

Relevant loader error:

clip_init: unable to find tensor v.token_embd.img_break
Failed to load CLIP

This occurred before multimodal inference.

The failure was not treated as a VRAM/Q8-performance issue.
Inspection pointed to a missing Pixtral/Mistral3 image-break embedding.

3. IMG_BREAK SOURCE INSPECTION

Official Venice tokenizer IDs observed:

9 [TOOL_CALLS]
10 [IMG]
11
12 [IMG_BREAK]
13 [IMG_END]

Source tensor:
language_model.model.embed_tokens.weight

Source row:
12

Source embedding shape:
5120 values

Original source dtype:
BF16

Repair output dtype:
F32

Reason for F32:
The extracted vector was deliberately stored as F32 for compatibility with
the projector/runtime path used in this experiment.

Extracted F32 row statistics:

finite: True
min: -0.0087890625
max: 0.008056640625
mean: -1.0783890502352733e-05

First 8 F32 values:

0.0057373
0.00020409
0.00270081
0.0025177
-0.00041199
-0.00147247
-0.00017357
-0.00095749

F32 [IMG_BREAK] vector SHA-256:

cc2f1288d643281c55bcccbb8b9142a1b4893c20e1d0f6de2330aa65314cfc46

4. PROJECTOR PATCH

Patch scope:

Exactly one missing tensor was added:

v.token_embd.img_break

Tensor type:
F32

Length:
5120

The existing quantized projector tensors were preserved rather than
re-quantized as part of this repair.

Original Q8 mmproj SHA-256:

1d71e5d4c757c157c6c0ed71e0393b548ebafadd1ad32d70081f49139bae3ce1

Repaired experimental mmproj SHA-256:

e04a12c4d4e031542a33d8457fac28612aed570ccf7cffcabd0b6c3b6075968a

5. LOADER RESULT AFTER PATCH

Result:
GREEN for loading.

Observed:

• multimodal model loaded
• projector type: pixtral
• vision available
• video modality reported available
• audio unavailable
• context configured at 131072
• dual-GPU runtime operational

The repair therefore resolved the specific missing-tensor loader failure.
If you need my mmproj write me .
Have a great Day DokMeMoRiA