WeMM-Embedding-2B-INT8

INT8 rebuild of ewin-reg/WeMM-Embedding-2B-Quantized. Same model, 139 FP8 tensors re-encoded as asymmetric int8. 1.94 GB on disk, runs under plain PyTorch on CPU, CUDA and MPS with no kernel shim.

Why this exists

The FP8 checkpoint cannot run on Apple Silicon because torch.float8_e4m3fn has no MPS kernels, and its forward path needs a third party shim to work around that. This build replaces every FP8 tensor with an asymmetric int8 encoding that dequantizes through ordinary casts, so the same file runs on all three backends. If you ship one app for Windows and macOS, this is the single checkpoint that covers both.

Model details

Qwen3.5 hybrid backbone: 18 gated delta net linear-attention layers, 6 full-attention layers, a 24 block vision transformer, a 248078 by 2048 vocabulary table, tied input and output embeddings, native Matryoshka support from 64 to 2048 dimensions.

Variant of tencent/WeMM-Embedding-2B via the FP8 checkpoint above. The conversion is mechanical re-quantization, no retraining, no distillation.

Exact allocation, measured from the shipped shards

Module namespace	Count	Stored format	Deployed size
language_model.embed_tokens	1	asymmetric int8, scale and zero per 64 weights	531.9 MB
attention, linear attention and mlp.down_proj linears	138	asymmetric int8, scale and zero per 64 weights	804.8 MB
mlp.{gate_proj,up_proj}, 48 LLM linears	48	uint8 packed int4, group 16	377.9 MB
visual.blocks attention, 48 projections	48	uint8 packed int4, group 16	62.9 MB
visual.blocks MLP, 48 linears	48	uint8 packed int4, group 16	125.8 MB
merger, 2 linears	2	uint8 packed int4, group 16	15.7 MB
norms, conv1d, scales	misc	BF16 and F32, 234 F32 tensors same as source	18.5 MB
Checkpoint total	1041 tensors	mixed custom	1.94 GB

The file carries 1041 tensors with an index total_size of 1937521288. The arithmetic is 902 source tensors plus 139 new zero-point tensors: the old scales stayed and were reshaped from scalar or per-row to one scale per 64 weights, and the zeros are pure additions. Nothing was removed, nothing renamed. build_report.json counts the three weight shards only, so its out_bytes undershoots the index total, which also covers tokenizer, configs, code and card.

The int4 tensors are copied byte for byte from the FP8 checkpoint. Unpacking reads even columns from the low nibble and odd columns from the high nibble, which is the order that checkpoint ships.

The int8 recipe

The FP8 checkpoint stores each attention and down projection matrix with one scalar scale for the whole matrix, and the vocabulary table with one scale per row. This build re-quantizes those 139 tensors with one scale and one zero point per 64 input weights.

The zero point carries the minus 128 shift, so block minimum maps to minus 128 instead of wasting half the codebook. Blocks whose values are all identical fall back to a symmetric grid around zero.

Three sources back this exact choice. SLQ, arXiv 2605.02404, proves symmetric grids inflate output variance by gamma squared with gamma = 2M/R over the dynamic range, making asymmetry a prerequisite rather than a nicety. NVFP4's 16 element blocks in arXiv 2609.04098 localize outliers the same way the 64 weight blocks do here. The gated delta net study in that same paper keeps embeddings, conv1d and norms out of low precision, which is why this build leaves the F32 parts and the int4 granularity exactly where the FP8 checkpoint put them.

Platform notes

CUDA, MPS and CPU all run from the same files with stock torch. The FP8 forward dispatched F.embedding on a float8 weight, which is the call with no MPS kernel; the INT8 embedding reads rows with index_select, which takes integer dtypes on all three backends. Matrix weights dequantize to the input dtype inside each forward call, weight only, activations stay in full width throughout. Nothing in this repo imports the fp8 MPS shim package.

lm_head.weight is absent by design

config.json sets tie_word_embeddings: true, so lm_head.weight is not stored and ties to embed_tokens at load. Under device_map="auto" the meta placeholder never materializes, which is why a device dump shows lm_head on the meta device. Nothing is missing and the forward pass is unaffected.

Measured fidelity

24 text queries through the unquantized base, the FP8 checkpoint and this build, one shared protocol, CPU runtime, no prompt template, embeddings read from each model's own embedding method and compared as cosines:

Pair	Mean cosine	Min	Max
FP8 vs INT8	0.999798	0.999723	0.999848
Base BF16 vs FP8	0.993103	0.987118	0.995475
Base BF16 vs INT8	0.992999	0.987443	0.995423

The INT8 build sits 0.0001 behind the FP8 build in fidelity to the base, while the direct FP8 to INT8 agreement is 0.9998. For reference, the previous INT8 workaround measured 99.17 percent against base on the same kind of workload; this build measures 99.30 percent. Weight space reconstruction error of the new int8 tensors against the FP8 path is 0.00586 for the 138 linears and 0.00600 for the embedding table.

A third party ran the predecessor INT8 build on real Apple Silicon hardware and reported load onto mps:0, agreement with CPU to a worst case 1 minus cosine of 5.12e-05, and about 11x the CPU speed. Those are their numbers from discussion #2, kept here because this build inherits the same forward code paths.

Quickstart

Plain transformers, text input:

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("ewin-reg/WeMM-Embedding-2B-INT8", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("ewin-reg/WeMM-Embedding-2B-INT8", trust_remote_code=True)
model.eval()

inputs = tokenizer(["Testing INT8 embedding."], padding=True, return_tensors="pt")
emb = model.embedding(**inputs)
print(emb.shape)

SentenceTransformers pipeline, same as the FP8 card:

from sentence_transformers import SentenceTransformer
from PIL import Image

model = SentenceTransformer("ewin-reg/WeMM-Embedding-2B-INT8", trust_remote_code=True)

text_embeddings = model.encode([
    "High-throughput vector indexing with post-training quantization.",
    "Recent advances in multimodal foundation embeddings in 2026.",
])

image = Image.new("RGB", (224, 224), color=(73, 109, 137))
image_embedding = model.encode(image)

frames = [Image.new("RGB", (224, 224), color=(i * 20, 100, 150)) for i in range(4)]
frame_embeddings = model.encode(frames)
video_embedding = frame_embeddings.mean(axis=0)

Matryoshka truncation, 256 dimensions as an example:

import torch.nn.functional as F

raw = torch.tensor(text_embeddings)
mrl_256 = F.normalize(raw[:, :256], p=2, dim=-1)

What this build does not change

It does not touch the int4 tensors, so the dominant quantization error in the model is unchanged. Measured on this model's own shapes, the int4 group-16 grid carries about 9 percent weight error against 0.6 percent for the new int8 grid, which is where the remaining 0.007 gap to the base lives. That gap is documented, not hidden: a wider variant exists only as a measurement, and this repo stays behavior compatible with the FP8 card.