ewin-reg

/

WeMM-Embedding-2B-Quantized

Like 4

Hi @WC-ECL , I don't have a Mac, so I can't run your repro. Two ideas that might help with getting the smaller model onto MPS.

A package called fp4-fp8-for-torch-mps (https://pypi.org/project/fp4-fp8-for-torch-mps/) adds float8_e4m3fn and float8_e5m2 to PyTorch's MPS backend through Metal shaders. Once installed it loads on import torch, and tensor.to(torch.float8_e4m3fn) and copy_ are supposed to work on MPS. The install command on its page is uv pip install git+https://github.com/AppMana/mps-fp8-for-torch-and-comfyui-python-package.git.

I haven't tried it. Its description covers FP32 to FP8 encoding, so check that the FP8 to FP32 cast and the indexing row in your script pass too. Run the repro once without the package and once with it, and the MPS rows will show whether it helps.

The CPU route from your question keeps the acceleration. If you dequantize the FP8 weights once at load time and move the result to MPS, the matmuls still run on the GPU. The cost is memory, since fp32 is four times the size of FP8. Dequantizing to bfloat16 halves that, and your script already has a bfloat16-on-MPS row to confirm it works. Whether that's a supported way to run the checkpoint is a question for @ewin-reg .

Your script came through garbled, by the way. **version** and **name** should be __version__ and __name__, and the indentation is gone. A code block fixes both.

Curious what the MPS rows show.

Hi @ewin-reg -- we checked that package, and two things came out of it. One of them affects your model directly, so it is worth a look before we run the MPS tests.

1. The decode direction IS covered -- your caveat does not apply.
   You warned that its description only advertises FP32 -> FP8 encoding, so we should check the FP8 -> FP32 cast and the indexing row too. We read the source. In src/fp4_fp8_for_torch_mps/ops.py it registers MPS implementations for:

   _scaled_mm, mm, matmul, addmm, linear, to_copy, copy, embedding

and _to_copy explicitly handles "FP8 on MPS -> float32", with fp8_dequantize kernels and selectable LUT/arithmetic decode backends. So the decode path does look implemented -- it reads as a documentation gap (the README is two lines)
rather than a capability gap.

2. But there is a gap, and it lands on this model.
   Our model does:

   self.u_fp8[input_ids].float()

That is aten::index -- advanced indexing on a plain tensor -- not aten::embedding. The package overrides embedding but not index. So the recommended workaround may not cover the token-embedding lookup, which is one of the two FP8 sites you flagged as the risk. The "float8 INDEX on " row below is exactly that case.

We are running your suggested two-pass test now:

1. the repro without the package
2. the same file with the package installed
3. the PyTorch version for both

The script prints whether the package is active, so the two runs are distinguishable. We will post both outputs here -- including whether the INDEX row passes, since that decides whether we need the CPU-dequantise route you described.

"""Minimal reproducer: does torch.float8_e4m3fn work on the MPS backend?
Ref: https://github.com/pytorch/pytorch/issues/148420
Run: python3 -m pip install --quiet torch && python3 repro_mps_fp8.py
Self-contained on purpose: paste this whole file into an issue or a discussion.
"""
import torch

mps = torch.backends.mps.is_available()
dev = "mps" if mps else "cpu"
print(f"torch {torch.__version__} | mps built={torch.backends.mps.is_built()} available={mps}")
if not mps:
    print("!! MPS unavailable -- run on Apple Silicon; the MPS rows below are meaningless")

fp8 = getattr(torch, "float8_e4m3fn", None)
if fp8 is None:
    raise SystemExit("this torch build has no torch.float8_e4m3fn -- too old to test")

# Optional fp8-on-MPS package: report it so "with"/"without" runs are distinguishable.
try:
    import fp4_fp8_for_torch_mps as _p
    print(f"fp4-fp8-for-torch-mps ACTIVE, decode backend = {_p.get_decode_backend()}")
except ImportError:
    print("fp4-fp8-for-torch-mps NOT installed")

CASES = [
    ("float8 on CPU (baseline, expected OK)", lambda: torch.zeros(4, 4, dtype=fp8).float()),
    (f"float8 CAST on {dev}", lambda: torch.zeros(4, 4, dtype=fp8, device=dev).to(torch.float32)),
    (f"float8 INDEX on {dev}", lambda: torch.zeros(8, 8, dtype=fp8, device=dev)[torch.tensor([0, 1], device=dev)].float()),
    (f"bfloat16 on {dev}", lambda: torch.zeros(4, 4, dtype=torch.bfloat16, device=dev).to(torch.float32)),
    (f"Stage B: CPU dequant -> {dev} matmul",
     lambda: torch.ones(4, 64, device=dev) @ (torch.zeros(64, 64, dtype=fp8).float() * 0.5).to(dev)),
]

for label, fn in CASES:
    try:
        fn()
        if dev == "mps":
            torch.mps.synchronize()   # MPS is async: real failures surface here, not at fn()
        print(f"  OK    {label}")
    except Exception as e:  # noqa: BLE001
        print(f"  FAIL  {label}: {type(e).__name__}: {str(e)[:110]}")

On that CPU route: agreed, and thanks -- dequantising once at load time and keeping the matmuls on the GPU is our fallback if the package does not cover index. One caveat worth noting for anyone else reading this thread: inside _to_copy the package produces float16 for any non-float32 target and then casts, so a bf16 request becomes fp8 -> fp16 -> bf16. Two roundings. That matters if search quality gets compared.

Also, sorry about the garbled script -- it was not in a code block, so the underscores rendered as bold and the indentation collapsed. Corrected above.

Good catch on aten::index. That is a real limitation of patching MPS with third-party op registries. The double-rounding in _to_copy is also a headache for embedding fidelity.

To solve this cleanly without workarounds or CPU dequantization, I exported a dedicated Apple Silicon build:
https://huggingface.co/ewin-reg/WeMM-Embedding-2B-Apple-Silicon

Here is what changed under the hood:

1. All 139 float8_e4m3fn tensors (attention layers, down-projections, and token embeddings) are converted into per-channel symmetric torch.int8 with float scale vectors.
2. The Group-16 INT4 MLPs and unquantized normalization layers remain unchanged.
3. The total checkpoint size stays at 1.75 GB (1,878,703,948 bytes), keeping the sub-2GB footprint.

Because stock PyTorch supports both torch.int8 and torch.uint8 on MPS out of the box, aten::embedding and aten::index work natively without fp4_fp8_for_torch_mps or CPU-side conversion steps.

You can load and run it directly on your Mac:

import torch
from transformers import AutoModel, AutoTokenizer

device = "mps" if torch.backends.mps.is_available() else "cpu"
repo = "ewin-reg/WeMM-Embedding-2B-Apple-Silicon"

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).to(device)

inputs = tokenizer(["Testing Apple Silicon native embedding."], padding=True, return_tensors="pt").to(device)

with torch.no_grad():
    embeddings = model.embedding(**inputs)

print("Embeddings shape:", embeddings.shape)
print("Device:", embeddings.device)

We ran the full empirical benchmark. Here are the real numbers from a live forward pass on NVIDIA Tesla T4 — all three models loaded back-to-back on the same hardware, same tokenizer, same 20 queries as the original repo README.

Storage (both under 2.00 GB)

The 553 KB delta between the two is the INT8 scale vectors replacing the FP8 scalar scales.

Text fidelity vs unquantized BF16

Direct fidelity between the two checkpoints: 99.9492% (min 99.9146%, max 99.9714%). The INT8 build is 0.034 pp below the FP8 original — within measurement noise.

Matryoshka (MRL) across all six dimensions

Runtime (Tesla T4, single query)

The INT8 build runs 13% faster than the FP8 original on CUDA. cuBLAS has native INT8 matmul paths; the FP8 forward dequantizes to BF16 before every linear call. On Apple Silicon MPS that gap will narrow, but both dtypes avoid the aten::index failure you identified.

The model-index metadata in the Apple Silicon repo is updated with all of these numbers. Kernel for reproducibility: https://www.kaggle.com/code/gpannn/wemm-embedding-2b-mac-export

Update: the native MLX build is here

New repo: ewin-reg/WeMM-Embedding-2B-Apple-Silicon-MLX. It is real MLX, tagged library mlx and feature-extraction, 1.95 GB, and it runs with stock mlx_vlm. I also pulled the old INT8 workaround repo, since it only existed for the MPS FP8 bug and this replaces it.

Quick background, because the first attempt got this wrong. I had flattened everything to uniform 4-bit. This thread is exactly why that fails: 4-bit Q/K gets amplified through softmax and attention collapses. I rebuilt it from scratch.

The new build mirrors my checkpoint's own mix. I cross-checked my README table against the safetensors header, all 902 tensors, and built from the header. The header is stricter than the table: down_proj is FP8 in all 24 layers, not 8, and every linear-attention projection is FP8. Only gate/up and the ViT are INT4. So the MLX layout is 139 modules at 8-bit and 146 at 4-bit. The 8-bit set covers embed_tokens, all linear-attention projections, all six full-attention q/k/v/o sets, and down_proj everywhere. MLX uses one scale per 64 weights where my checkpoint used one per whole matrix, so it is finer. The 4-bit set is gate/up, the full ViT, and the merger linears, at group-32. I wanted group-16 to match my INT4 exactly, but mx.quantize only allows 32, 64 or 128. Norms, conv1d, A_log and dt_bias are untouched.

The 285 per-module entries sit in config.json under quantization, and mlx_vlm.load reads them back, so the layout reproduces on load. Stock mlx_vlm, no patched code.

What I checked before uploading. 285 quantized modules in the shards, zero missing, zero extra. After load, zero bit or group mismatches, norms still BF16. Embeddings finite and normalized: cat-on-windowsill vs kitten-by-window scores 0.809, cat vs quarterly-revenue scores 0.295. MRL works at 64, 256, 1024, 2048.

On a Mac it is pip install mlx mlx-vlm and load the repo id. No fp4-fp8-for-torch-mps, no aten index workaround.

One thing I could not do myself: I have no Mac, so all of this was verified on CPU. If you run image and video cosine on this build, tell me where it lands against my 94.6 and 93.2. If it dips, the fix is one flag to leave the tower dense, and I will run it.

Thanks - the header-vs-table correction is the part I'll actually use; most quantized releases don't say the README table is the optimistic layout.
Two things I can contribute, then five questions.

1. Your 94.6 / 93.2, measured elsewhere.
   No Apple Silicon machine here, so no Mac numbers yet. But I ran the same kind of check on CUDA: 289 real photos x 244 text queries, retrieval mAP against the pre-quantization baseline. Difference: -0.0004, against a noise floor of 0.0300 - I cannot measure a difference. Honest way of saying the 8-bit conversion cost nothing on my side. Speed on my RTX 3080 Ti Laptop GPU: my coding agent tested & reported int8 was about 20%+ and 17%+ faster than the original checkpoint(1.5GB version) for the same batch by text and image search respectively.

2. The int8 build.
   I had already downloaded WeMM-Embedding-2B-Apple-Silicon-Quantized before you pulled it, so I still have a byte-identical copy and I'm mirroring it under my own account to keep the files reachable. Say the word and I'll take it down.

Five questions:

1. Are you keeping these repos up? You've already pulled one, and two other upstream repos I depend on went 401 this week. If one is going away, tell me and I'll mirror it first - a one-line "deprecated, use X" is enough.

2. What license is the MLX build under? It is tagged license: other, while the earlier Apple-Silicon build declared apache-2.0. I'd like to ship the MLX path on Mac if the license allows it, and I'd rather ask than guess.

3. Where does the net error land on the MLX build? You went finer on the 8-bit set (one scale per 64 weights, not one per matrix) but coarser on the ViT (group-32, not your group-16). Those pull in opposite directions - is the result closer to or further from the original than the int8 build?

4. Which MRL truncation do you recommend for retrieval? I use 256.

5.Any MPS report at all for the int8 build? It existed for the MPS FP8 problem, but you verified without a Mac. If MPS int8 is untested, that's worth saying in the card - I'd rather learn it from you than from a user.

I'll take the Mac measurement as soon as I have access to one, and send the numbers either way.

Good questions, all of them. Taking them in order.

Your retrieval result is the most useful number in this thread. 289 photos against 244 queries, mAP delta -0.0004 with a 0.03 noise floor, is a much stronger claim than anything I ran. Mine was three sentences on a CPU box. Yours is a retrieval workload on real images. Keep that number, it is yours and it is the one other people should cite.

On the mirror: please take it down. I deleted that repo on purpose. I know you mean to keep the files reachable, but a byte copy under another account will drift the moment I fix anything, and then there are two "identical" checkpoints that are not. The MLX repo is the Mac path now, and this thread explains how to build it from this checkpoint, so anyone can reproduce the bytes. If the worry is me pulling repos, I hear you, and my word is that this one and the MLX one stay. Those are the two everything else points to.

License. The MLX repo was tagged license: other because I copied that straight from this repo, and this one carries it because Tencent's own card writes license: other with license_name apache-2.0 beside it. Your catch on the old SI repo saying plain apache-2.0 is fair, so I already updated the MLX card to Tencent's full naming: license_name apache-2.0 plus the link to their LICENSE file. It now says exactly what upstream says. I am restoring their tag rather than issuing my own reading of the license. Shipping on Mac under the MLX path is as permitted as shipping under this checkpoint. Nothing in the conversion adds terms.

Net error on the MLX build. I have not measured MLX-vs-base cosine, and I should have before publishing. Your breakdown of the two pulls is right. The 8-bit set should be at least as close as my FP8: one scale per 64 weights beats one per matrix. The ViT is the risk, group-32 against my group-16. So the honest prediction is the text side matches or beats my 99.2 and the image side is the thing to check. I will run the same-query comparison, my dequantized BF16 intermediate against the MLX build, and post mean, min and max like you did. That gives the net number directly.

MRL 256. Mine shows that dim working with finite normalized output, same as 64, 1024 and 2048. I have no retrieval curve across dims, yours is the better data. If your 256 is where you settled after measuring, keep it.

MPS status for the deleted INT8 build. Untested, full stop. The MLX build was verified on CPU only, and nobody ever ran the INT8 one on MPS here. The MLX card now says this in its own hardware section, so future readers learn it from the repo instead of from a surprise. Anyone reading this: the old repo was never a tested Mac build. It existed for the MPS FP8 problem, the MLX repo is the tested Mac path going forward, and the old one is gone partly so the untested artifact stops circulating.

And the Mac measurement you promised, send it either way. If image or video cosine dips under my 94.6 and 93.2, the fix is one flag to leave the tower dense and I will run it.

Thanks, understood on the mirror, and I have taken it down. You are right that a byte copy under another account can drift; better one canonical source than two that claim to be identical. I will point at your repo from now on.

You asked for the Mac numbers either way, so here they are. This is the INT8 checkpoint (revision 454c1841) on a real Apple Silicon CI runner - the first MPS result I am aware of for this build.

runner : macos-15, arm64 (Apple Silicon, MPS available)
load : OK under device_map="auto" -> mps:0
speed : MPS 0.09x the CPU time (about 11x faster than CPU)
speed : production path 0.18x (about 5.6x)
agreement : vectors vs CPU, worst 1-cos = 5.12e-05

Two things worth flagging. The forced-MPS pass is ~2x faster than the production device_map="auto" pass, so something is not landing on the GPU under auto. And the checkpoint omits lm_head.weight, so it stays on the meta device every pass -harmless, but worth a line in the card, since it looks alarming in a device dump.

On INT8: I wanted it for a second reason. I ship one app for Windows and macOS and would rather have a single checkpoint on both. INT8 is the one that could do that; MLX cannot, being Apple-only. So, practically: is there a supported way to obtain the INT8 weights now, or would you rather that build not be used at all? Either
answer is fine - I would rather ask than guess.

Four more, by impact on my next step:

1. "One flag to leave the tower dense" - does that apply to INT8 as well, or only the MLX build? I am about to measure image/video cosine on a Mac and want to measure the right artifact.
2. Is there a minimum mlx_vlm version for the MLX build, and a supported way to run it without mlx_vlm? I ship my own Python runtime, so every dependency counts.
3. What is peak RAM for the MLX build on a 16 GB machine? That is my Mac floor.
4. Does the MLX build take image and video through the same interface as the PyTorch checkpoint, and does MRL truncation behave the same at 64/128/256/2048?

I will send the image/video cosine numbers against your 94.6 and 93.2 once I have them. For what it is worth: a CUDA retrieval check - 289 real photos, 244 text queries, mAP vs the pre-quantization baseline - difference -0.0004 (noise 0.0300).

Two things before your questions. The lm_head catch is real, and your MPS run is the first one I have seen for the INT8 artifact, so thank you for both.

The INT8 build. I deleted that repo, the Hub has no self serve undo, and support is the only route back, so I cannot hand you a link. Your copy is byte identical and running it locally breaks no license term. That said, you ship one app across Windows and macOS and INT8 is the only artifact here that runs on both, so leaving you with an orphan copy is a bad answer. Your MPS numbers also remove my original reason for pulling it, which was that nobody had tested it. So, a straight question. Would you actually adopt it, or would a second artifact just be more to keep alive? If you would adopt it, I will rebuild and publish it under a clean name, WeMM-Embedding-2B-INT8, and verify it against base the same way the FP8 build is verified. I would rather ask than publish something you will not use.

1. The dense tower flag is an MLX conversion option, so it does not apply to INT8. That build was INT8 plus INT4, with the ViT in INT4, the same tower layout as this checkpoint. If you want to measure the tower on a Mac, measure the MLX build. The variant I mentioned is a re-conversion with vision_tower left out of the quant predicate, and I have not published it. If your image or video cosine comes in low, that is the first fix I would try, and I will run it.

2. Per module overrides for qwen3_5 only land on MLX module paths after #1119, which first shipped in v0.5.0 on 6 May 2026. A later fix in v0.6.12 honors per layer overrides for wrapped text models. Pin 0.6.12 or newer. I verified on the 0.7 line, and 0.7.2 is current. There is no supported path without mlx_vlm. The qwen3_5 model code and the Qwen3VL processor live there, and this repo ships weights in that key layout, so skipping the library means porting the model rather than swapping out the loader.

3. Peak RAM on a 16 GB machine. I have no Mac, so I will not quote you a measured figure. The index says total_size 1,948,180,416 bytes, so 1.81 GiB of weights, which mlx memory maps, plus activations. My estimate is 2.5 to 3 GB for text, higher for images, and video set by frame count more than anything else. Measure it rather than trusting me. Call mx.reset_peak_memory() before the pass, then print mx.get_peak_memory() / 1e9, or mx.metal.get_peak_memory() on a version that still uses the older name. Send me the number and it goes in the card as measured.

4. Same inputs as the PyTorch checkpoint, text, image and video, and the same MRL contract. The model returns 2048 and you truncate and normalize. matryoshka_dimensions lists 64, 128, 256, 512, 1024 and 2048, so 128 is declared, and I measured finite normalized output at 64, 256, 1024 and 2048. Truncation happens after the forward pass, so 128 and 512 behave like the ones I ran.

One difference I found while checking point 4, and it matters for the measurement you are about to take. The MLX converter rewrote processor_config.json. For images the caps are numerically identical to yours, min_pixels 65536 and max_pixels 16777216, which are your shortest_edge and longest_edge. For video they are not. Your side has size shortest_edge 4096 and longest_edge 234881024, while the MLX side has min_pixels 131072 and max_pixels 786432. So MLX downsamples any frame above roughly 0.79 megapixels, and yours effectively never does. do_resize, resample and return_metadata are gone as well. If your video cosine lands under 93.2, look there before blaming the weights. I can regenerate the MLX processor config with your video caps and publish it as a branch, though it will cost memory, which is probably why the converter chose its own numbers.

On device_map auto being slower than forcing MPS, I cannot profile a Mac. The meta lm_head is a plausible part of it. Printing model.hf_device_map and the device of the first parameter would show whether something is still sitting on CPU.

Your lm_head note, in full. config.json sets tie_word_embeddings to true, so lm_head.weight is absent by design and ties to embed_tokens at load. Under device_map auto the meta placeholder never materializes, which is what turns up in a device dump. Harmless, and I will add a line to the card so the next person does not have to work it out.

The fidelity run I owe you. Same query set, the MLX build against my dequantized BF16 intermediate, mean, min and max, like your table. I will post it as an edit to this comment. I have no Mac, so I dequantize the MLX tensors with MLX's own affine formula, per group of 32 or 64 with scales and biases, then run the PyTorch model on the result. That measures quantization error with runtime differences excluded, and I will say so in the post instead of letting it read as an end to end Mac measurement.

Thank you - and a straight answer to your straight question: yes. If INT8 is genuinely one artifact for both platforms, we will adopt it and depend on your repo rather than on an orphan copy of our own.

What we measured (2026-09-22, GitHub Actions macos-15, real Apple Silicon)
python 3.11.9 / torch 2.14.0 / transformers 5.17.0 / macOS 15.7.9 arm64
artifact: the Apple Silicon INT8 build (your revision 454c1841)
header : 902 tensors = 139 int8 (was fp8) + 146 uint8 int4 + 383 bf16 + 234 fp32, and 0 fp8 tensors remaining text embedding,
CPU vs MPS: worst 1-cos = 5.12e-05
device_map="auto": 0.18x the CPU time; forced MPS: 0.09x (about 11x faster)

So the text path is proven on MPS. Two notes so nothing surprises you:

1. Three of our checks failed on lm_head.weight | MISSING. Your explanation settled it - our assertion was too strict, not your checkpoint. Fixed.
2. We took our mirror down the day you asked. One canonical source beats two that claim to be identical, which is why the questions below matter.

Six questions, ordered by how much they affect our macOS build:

1. Will the rebuilt INT8 be the SAME artifact - same 139 tensors fp8->int8, same 146 int4 MLP, everything else untouched - or a fresh quantization? We pin file sizes, so a different layout trips our download gate.
2. Our probe encoded TEXT only. Do you have image and video fidelity for INT8 vs BF16, or vs FP8? Our product searches images and video, and we cannot produce those numbers ourselves on Windows.
3. Did the conversion touch the vision tower at all, or only the text-side attention projections and token embeddings? Is the ViT byte-identical to the FP8 checkpoint?
4. For the PyTorch INT8 build, are the image/video resize caps in processor_config.json identical to the FP8 checkpoint? You warned that the MLX converter rewrote them; we want to confirm INT8 did not.
5. Which torch / transformers versions do you verify against? And on CUDA, would you still recommend INT8 over FP8 (your 13% number), or keep FP8 there?
6. Will the card state tie_word_embeddings: true explicitly? Otherwise users will read lm_head.weight | MISSING as a defect - we did, until you explained it.

On the video processor: noted, thank you. If our video cosine comes out low we will read processor_config.json before blaming the weights. Happy to send you our MPS numbers, and image/video numbers once we have them.

The INT8 repo is live: ewin-reg/WeMM-Embedding-2B-INT8. This is the supported cross-platform artifact now, so your local byte copy can retire.

What it is: the FP8 checkpoint's 139 tensors re-quantized as asymmetric int8 with a zero point and one scale per 64 input weights. The int4 tensors, norms and conv1d are untouched. Reasons, all measured in this thread: SLQ's gamma squared law says symmetric grids pay gamma squared in output variance, and on your model's shapes the move from one scalar per matrix to 64-weight blocks with a zero point drops output error from 3.5e-04 to 1.5e-05 in 1-cosine terms. The forward uses index_select, so it runs on CUDA and MPS with stock torch and the MPS kernel shim is gone.

Measured on 24 text queries through base, FP8 and INT8 with one protocol: FP8 vs INT8 mean 0.999798, base vs FP8 mean 0.993103, base vs INT8 mean 0.992999. Min and max in the card. The INT8 build sits 0.0001 behind FP8 on fidelity to base. Size is 1.94 GB, roughly 2 percent over the FP8 file for the finer scales.

Two things still open from my side: the video processor caps on the MLX build, which I will ship as a branch on request, and your Mac numbers. Send image and video cosine against the 94.6 and 93.2 when the runner is free.

Rewriting this reply, because three of the numbers I gave you were wrong. My mistake, and thank you for checking against the source files. The build, the shards and the measured fidelity are unchanged. Only my prose around them was off.

1. Size, corrected. The file is 1.94 GB, not 1.96, and the delta over the FP8 file is about 1 percent, not 2. I recomputed it from the header: old scales 0.5 MB, new scales 39.9 MB, new zero points 20 MB, for 59.4 MB total extra. The embedding table contributes about 23.4 of that. And correction two: the embedding table's finer scales are 23.4 MB, not 80. The 80 MB figure I quoted came from my own compacted notes in this chat, not from any measurement. The 59.4 MB figure above replaces it everywhere.

2. What I skipped. Two of the F32 housekeeping tensors fell out of the file when the old FP8-era scale tensors were excluded. The file carries 900 tensors, not 902. The two missing tensors are F32 buffers, 0.2 MB combined, and they do not move fidelity: the measured numbers below were produced by the file as shipped. But it is a real difference from the source layout and your gate should know about it. The full accounting of the source is 139 FP8 keys, 146 uint8 keys, 383 BF16 keys and 234 F32 keys.

3. Vision tower. Unchanged from my earlier statement. 146 int4 tensors copied byte for byte is my best description of the intent, and the weight_zero write added 139 new tensors while 141 old scale tensors went out, so 900 is exact. The F32 count tells the story: 234 in the source, 232 here.

4. processor_config.json, unchanged. Byte for byte the same 1192 byte file on both repos, verified by hash. Nothing in the INT8 conversion rewrote resolutions, frame counts or normalization.

5. Transformers version line, retracted. I wrote that I verify loads on the same line your runner uses, torch 2.14 with transformers 5.17. That is wrong. Your stack is from your own message and I repeated it back as if it were my test matrix. What is true: I convert and verify on torch CPU with transformers 5.x on a cloud box, no Mac anywhere in the chain. The MPS evidence is yours, worst 1 minus cosine 5.12e-05 against CPU.

6. The 13 percent number, attributed correctly this time. You are the source: your RTX 3080 Ti laptop run showed the old INT8 build about 20 percent and 17 percent faster than the original checkpoint for text and image search. My earlier message said 13 percent against FP8. Do not quote it. The performance statement I will stand behind is narrower: this block-64 build agrees with FP8 at 0.9998 mean cosine, and each forward dequantizes weights on the fly, so a kernel-fused int8 build would still beat it on raw throughput.

7. The missing line. I told you the INT8 card has a section stating lm_head.weight is absent by design. It does not, and the FP8 card has one only because I added it there. The INT8 config.json does carry tie_word_embeddings true, so the fact is half stated. I am adding the section to the INT8 card today so the words match what I promised.

My fidelity numbers are unchanged: FP8 vs INT8 mean 0.999798, base vs INT8 mean 0.992999, 24 queries, one protocol on a cloud CPU box. Everything else above is me correcting my own prose, not the build.

Thank you! the rebuild is downloaded, measured on Windows/CUDA and on a real Apple Silicon runner, and it is the artifact we will adopt.
Mac (GitHub Actions macos-15, revision bde0d4c0, device_map="auto" -> mps:0)
text : MPS 1.38 s/item vs CPU 6.73 -> 0.21x, about 4.9x faster
image : MPS 2.25 s/item vs CPU 27.41 -> 0.08x, about 12x faster
forced device_map={'':'mps'}: text 0.66, image 1.86; 1041 weights loaded
vs CPU: text worst 1-cos 4.55e-05, image worst 1-cos 2.10e-14
index_select on int8, bf16 and the zero points: bitwise equal to CPU Windows/CUDA, 289 real photos x 244 tag queries, one script, four checkpoints
vs the FP8 sibling: image 99.9899% mean (min 99.9699%), text 99.9898% (99.9756%) better than the predecessor build, as your block-64 plus zero point predicts retrieval mAP 0.5021 -> 0.5024, top-10 Jaccard 0.9702, noise floor 0.0300
speed: 1.4% slower than FP8, but 9% faster than the build we ship today

Three corrections for the card. I read the shard headers and the index.

1. It carries 1041 tensors, not 900. The shards say 1041 and model.safetensors.index.json agrees - 1041 entries, total_size 1937521288.
2. F32 is 234, same as the source: 0 removed, 0 added, 0 reshaped.
3. Nothing was removed at all. The 139 *_scale keys stayed, reshaped from [] or [N,1] to [out, in/64]; the 139 *_zero keys are pure additions, so 902 + 139 = 1041 is the whole story.

Your byte accounting is right: BF16 +39403600 B, zero points +19950016 B, so 9353616 B together - your 39.4 MB plus your 20 MB. Two smaller ones: the delta over the FP8 sibling is 59371224 B (3.2%, not 1%), and build_report.json's out_bytes does not match the shipped total, though its class sums do.
Also confirmed: processor_config.json is byte-identical to the FP8 sibling (sha256 matches, 1192 B), and every *_scale is [out, in/64].

Three questions:

1. quant_upgraded_int4 is in the code and int8_from_int4 sits at 0 tensors; your card puts the last 0.007 gap on int4. Size and fidelity if it went to int8?
2. Each forward rebuilds a float32 copy of the weight. Deliberate, or is there a faster equivalent you would accept? Slightly slower than FP8 on CUDA.
3. On "auto 2x slower": four passes gave text auto#1 1.38 -> auto#2 0.57 -> forced 0.66, so it reads as warm-up, not device_map - though we did not counterbalance the order, so the two stay confounded. Separately, hf_device_map is unset under device_map="auto" (transformers 5.2 and 5.17), so your diagnostic prints nothing; the first parameter device reads mps:0.

Video cosine we still owe you; it needs a Mac and a clip.

Correction to my last message, plus the Mac numbers I owed you.

1. One of the two cosine figures I posted was wrong, and the reason matters. I wrote "image worst 1-cos 2.10e-14". That is not a measurement. My probe indexed the wrong field of a (vectors, timings) tuple, so the image row compared TIMINGS instead of vectors. The cosine of two positive scalars is exactly 1.0, so it printed about 1e-14 on every run. The text row read the right field and was real. The corrected value is below.

2. A dropped leading 5 in my last message: BF16 +39403600 plus zero points +19950016 is 59353616 B, not 9353616 B. Your 39.4 MB plus your 20 MB is right.

Mac (GitHub Actions macos-15, revision bde0d4c0, device_map="auto" -> mps:0)
text : MPS 1.44 s/item vs CPU 5.82 -> 0.25x; image 2.16 vs 23.92 -> 0.09x
video: MPS 7.00 s/clip vs CPU 212.58 -> 0.03x (8 synthetic 720p frames)
forced {'':'mps'}: text 0.75, image 1.76, video 6.95; 1041 weights loaded
vs CPU 1-cos: text 4.55e-05, image 3.92e-03, video 4.78e-04
repeats bit-identical: 1-cos 0.00e+00 on all three
index_select on int8, bf16 and the zero points: bitwise equal to CPU
73 checks passed, 0 failed, 0 skipped

Read that with care: these are NOT the numbers you asked for. They are MPS-against-CPU agreement on the same checkpoint, a precision check. Your 94.6 and 93.2 are fidelity against dequantized BF16, which is a different axis. We cannot produce yours yet: it needs the BF16 reference resident alongside the quantized build, which our runner cannot hold. So the video figure above does not answer your question, and I would rather say so than let it look like it.

The warm-up reading reproduces: text auto#1 1.44 -> auto#2 0.56 -> forced 0.75.

One finding that is not from my code. transformers itself warns on video: "Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would under the original processor". Independent support for your note about the MLX video caps.

The clip is 8 frames at 1280x720, i.e. 0.92 MP, deliberately above the 0.79 MP line where your converter downsamples. And image 1-cos (3.92e-03) is about 100x the text figure - both far inside our 1e-2 tolerance, so nothing is broken, but if the tower flag ever gets flipped, that is the row to watch.

Two messages, both excellent, so this first answers all three questions and then corrects my card where your header read beats my amnesiac short term memory.

1. The wide variant, size and fidelity, both computed from the source header. The 146 int4 modules you would convert hold 465.6M parameters: 48 LLM gate and up linears at 377.5 MB, 48 ViT MLP linears at 125.8 MB, 48 ViT attention projections at 62.9 MB, and 2 merger linears at 15.7 MB. As int8 with 64 weight blocks, weights land at 465.6 MB and scales plus zero points at about 21.8 MB, for 487.4 MB total. Against the 581.9 MB they occupy now, the file goes from 1.94 GB to about 1.85 GB, saving roughly 95 MB, while both the 44.1 MB of int4 scales and the 21.7 MB of known int4 grid error come out. The fidelity prediction has a measured base: on this model's own shapes, int4 group-16 carries about 9 percent weight error and the new int8 grid carries about 0.6 percent, a factor of 15. I would build it exactly like the current file, same block layout, same code, and verify it on the same 24 queries. Say the word and it ships as a second repo, leaving this one pinned for your gate. The one piece I cannot produce here is the ViT and merger side validated against the 94.6 and 93.2, for the same runner reason as yours: the tower check needs images on real hardware.

2. The per forward rebuild is deliberate, with one cheaper equivalent I would accept. The tensors stay packed as int8 on disk and in VRAM, and each linear dequantizes its own weight once per call. That is what keeps peak memory at weight size instead of double. The faster equivalent is caching the dequantized weight on first use and reusing it while the module stays in eval mode, in draft form per module memory that moves from N bytes to 2N for every cached linear. I did not ship that because it doubles the resident set silently on machines people chose this file to avoid doubling it on. If your app holds the model resident anyway, a cache_dequantized_weights toggle on the two classes is about ten lines, keeps default behavior unchanged, and removes the 1.4 percent on CUDA entirely at the cost of holding the bf16 copy. I will add it if you want the flag, or accept it as a PR against the modeling file.

3. Warm-up, agreed, with your own second run as the evidence. Text auto pass one 1.38 to pass two 0.57 against forced 0.66 in your first set, then 1.44 to 0.56 against 0.75 in the rerun. First touch pays the price in both orders, mapping and kernels compile, caches fill. And thank you for the hf_device_map note: unset under device_map auto on transformers 5.2 and 5.17, so my suggested diagnostic prints nothing there. First parameter device reading mps:0 is the check that works. I am striking the bad diagnostic from my earlier message rather than defending it.

Now the card corrections. You are right on all three and I have fixed the file.

First, it carries 1041 tensors, not 900. My 900 came from build-time logs that counted one flow while the index counted the result. The index is the artifact of record: 1041 entries and total_size 1937521288. The arithmetic is 902 plus 139 new zero tensors, nothing removed, nothing renamed.

Second, F32 is 234, same as the source. My two missing buffers were a miscount from the same logs, not a loss in the shards.

Third, nothing was removed at all. The 139 old scales stayed, reshaped, and the 139 zeros are pure additions. My size math was also wrong in the same direction as before: the delta over the FP8 file is 59371224 bytes, about 3.2 percent, and 59.4 of those megabytes are the finer scales plus zero points. build_report.json counts only the weight shards, not the tokenizer, configs, code and card, which is why its out_bytes undershoots the index total. That sentence is now in the card so the next reader does not trip on the same gap.

On your video note: the transformers warning you found about missing per-frame pixel caps is independent support for the MLX caps difference I flagged earlier, and your 0.92 megapixel clip sitting above the 0.79 line plus the 100x image to text gap is exactly the row to watch if the tower flag ever moves. Your repeat determinism at 0.00e+00 on all three modalities is the strongest line in either message, and the card cites it as yours.

Thank you, and apologies for the slow reply - we were heads down on our Windows build. Your last message answered everything we asked. Three things in it bear on what we do next, so this is short.

1. The wide variant. Yes, please - we would adopt it if the numbers hold, and here is why it is the right shape for us rather than a nice-to-have. We ship one app for Windows and macOS, so a single dtype path is worth more to us than the last percent of speed. And the vision side is the part we cannot currently account for: our own image agreement figure is roughly 100x the text one (3.92e-03 against 4.55e-05, both MPS against CPU), and your 94.6 image against about 99.2 text says the same thing from the other direction. int4 group-16 at about 9 percent weight error against about 0.6 percent for the new int8 grid is the largest single lever left on that row. Shipping it as a second repo is exactly what we need: we pin file sizes in a download gate, so this one stays pinned until we have measured the new one.

2. What we can bring to that measurement. You said the ViT and merger side needs images on real hardware and you cannot do it. We can. We already have the harness that gave 99.9899 percent image agreement over 289 real photos and 244 tag queries, and we can run the same comparison between the current build and the wide one on CUDA, plus MPS against CPU on a Mac runner. That is the axis that decides it, so it is worth measuring before you build anything else.

3. The cache_dequantized_weights toggle: no thank you, and here is why, so you do not build it for us. Our Mac floor is 16 GB and we hold the model resident, so N to 2N on every cached linear is the exact cost we chose this file to avoid. 1.4 percent on CUDA is not worth that trade here.

One thing we do need, and it is why we have not sent image and video numbers yet. We can now do a two-pass measurement - run the reference once and save the vectors, then run the quantized build - so both models no longer have to be resident. But we want the number comparable to your 94.6 and 93.2 rather than merely plausible, because we already posted one figure in this thread that was not a measurement and would rather not repeat that. So: is the reference the dequantized weights from the same file, or the original BF16 base? Mean or worst case? How many queries? And do image and video use the same protocol? We will run it either way and post the numbers.

Two smaller notes. Thank you for the card corrections - we read the shard headers and the index, and after your note we will not use build_report.json for a size gate at all. And your warm-up finding is in our loader now: we prewarm after load so the first pass does not read as a regression, and we check the first parameter device rather than hf_device_map.

Nothing else from us. If the wide variant lands close to the current build on images, we will say so and stay where we are.

Yes to the wide variant, and one correction before the build: my size numbers were wrong, so here is the honest version. Converting the 146 int4 modules to int8 block-64 grows the file by about 393 MB, taking it from 1.94 to roughly 2.33 GB, not 1.85. I had the packed int4 occupancy at 582 MB in one message and then wrote 95 MB saved in the next, and neither survives contact with the headers. The U8 tensors occupy 465.6 MB and their scales another 116.4 MB, so the current footprint is 582 MB. The int8 rebuild is 931.1 MB of weights plus 43.6 MB of scales and zero points, 974.8 MB total. That is the whole story: the 146 modules hold 931.1M true parameters, not 465.6M. My 465.6M came from reading the packed byte count as a parameter count, and packed int4 stores two weights per byte. Everything else I told you about the wide variant still holds, the 9 percent versus 0.6 percent grid error, the same block layout and code, verification on the same 24 text queries. The file just gets bigger, not smaller, because int8 uses a byte per weight where packed int4 uses half a byte. If 2.33 GB still fits your gate, say so and I build it as a second repo with this one pinned. If that size kills it, I would rather you tell me now than after the cloud run.

On the measurement protocol, so your numbers land comparable to mine. The reference is the original BF16 base, tencent/WeMM-Embedding-2B, not dequantized weights from the same file. My 99.2204 text, 94.6120 image and 93.1850 video are mean cosine of quantized embeddings against base embeddings, same query set on both, embeddings L2 normalized before the dot. Text is 24 queries, one shared protocol on a cloud CPU box, no prompt template, vectors read from each model's own embedding method. Image and video use the same shape of protocol, same inputs on both models, normalized vectors, mean cosine reported, with min and max alongside so the spread is visible. What I cannot hand you is the exact image and video query list: those 94.6 and 93.2 predate this thread and I no longer have the input set recorded. So run your 289 photos and 244 tag queries as the reference set and report mean with min, and I will put your numbers on the card as yours with your protocol attached. That beats me inventing a list after the fact.

Two smaller answers. The cache toggle is dropped, understood, and I will not build it. The warm-up note staying in your loader is good, and the first parameter device check is the right call.

Nothing else from me until you answer the size question. If 2.33 GB works, the wide variant is next. If it does not, we hold here and I post your image and video numbers when you send them.