ifire/Qwen3-VL-Embedding-2B-GGUF

Qwen3-VL-Embedding-2B — GGUF, with vision tower

Embeddings from Qwen/Qwen3-VL-Embedding-2B, converted for llama.cpp. The mmproj vision tower is included and complete.

Text embeddings reach parity with PyTorch. Image embeddings do not — they land at cosine 0.967, not 0.9999. Usable, but measurably not the same vector. Details below.

Files

file	size	sha256
Qwen3-VL-Embedding-2B-f16.gguf	3.21 GB	9ea1b7d96afdb3820d3dce1947386fe6a96200a8f8ef5fd6dde05e58d5d2ab20
mmproj-Qwen3-VL-Embedding-2B-f16.gguf	0.76 GB	e04ee32f0bc8b15399b0732525b8244c9279734c5bf0a80df4356af11e660c70

Verification

Measured on an Apple M2 Pro, 32 GB, macOS, against the float32 PyTorch reference on identical input — ten short strings of the kind this was built for, run through both paths.

	cosine vs torch	p50 latency
this GGUF, llama.cpp Metal f16	0.999997 mean, 0.999995 min	29.9 ms
PyTorch float16 on MPS	1.0 by definition	51.2 ms

1.71x faster than PyTorch on the same machine and the same inputs, at cosine 0.999997. Output is 2048-dimensional and already L2-normalised.

A latency figure is a claim about a machine, not about a model. Re-measure on yours.

The vision path: works, but not at parity

Ten licence-clean COCO images through both paths, against the PyTorch reference.

input	cosine vs torch
text	0.999997 mean, 0.999995 min
image	0.967420 mean, 0.951388 min

Images produce genuinely distinct, meaningful embeddings — two different images score 0.146 against each other, so the vision tower is being used. But 0.967 is not 0.9999, and for retrieval that difference is real.

The gap tracks image size: corr(pixels, cosine) = +0.730 across the sample. Larger images agree better. That is the signature of a preprocessing difference — resize and interpolation between Qwen3VLImageProcessor and llama.cpp's clip path — rather than of the weights or the vision/text fusion. The mmproj carries all 18 deepstack tensors (v.deepstack.{5,11,17}.*, matching deepstack_visual_indexes: [5, 11, 17]) and clip.cpp implements deepstack.

Two API footguns worth knowing

Use multimodal_data, not image_data.

{"input": {"prompt_string": "…<__media__>…", "multimodal_data": ["<base64>"]}}

image_data is a /completion field. On /v1/embeddings it is accepted without error and silently ignored — every image then returns the identical text-only vector. That failure is invisible without a reference to compare against.

The OpenAI-style input: [{"type": "image_url", …}] form returns HTTP 500. PR #18665 adds that schema.

Use

llama-server -m Qwen3-VL-Embedding-2B-f16.gguf \
  --mmproj mmproj-Qwen3-VL-Embedding-2B-f16.gguf \
  --embedding --pooling last -ngl 99

--pooling last is not optional. This model pools the last token (1_Pooling/config.json: pooling_mode_lasttoken), and mean pooling silently produces a different vector rather than an error.

How it was converted

llama.cpp 925e1179947ea0c0ebfb0032df18af3a729822be, convert_hf_to_gguf.py:

python convert_hf_to_gguf.py <model> --outtype f16 --sentence-transformers-dense-modules
python convert_hf_to_gguf.py <model> --mmproj --outtype f16

--sentence-transformers-dense-modules carries the dense layers that follow pooling. Without it the vectors are wrong in a way nothing reports.

Licence

Apache-2.0, inherited from the base model. Weights are Qwen's; this repository contributes only the conversion and the measurements above.