monju-lab/all-MiniLM-L6-v2-onnx

all-MiniLM-L6-v2 — quantized ONNX only

Quantized ONNX graphs and tokenizer files from sentence-transformers/all-MiniLM-L6-v2, with nothing else. The upstream repo (and a full mirror of it) carries safetensors, PyTorch, TensorFlow, Rust and OpenVINO weights plus the fp32 ONNX graphs — about 1 GB — to use one 23 MB file. This repo exists so a container build can fetch the graph without the other 900 MB.

Weights are byte-identical to upstream; nothing was re-exported or re-quantized.

Contents

File	Size	Use
onnx/model_quint8_avx2.onnx	23 MB	safe x86 default
onnx/model_qint8_avx512_vnni.onnx	23 MB	x86 with AVX-512 VNNI (fastest)
onnx/model_qint8_avx512.onnx	23 MB	x86 with AVX-512, no VNNI
onnx/model_qint8_arm64.onnx	23 MB	ARM64 / AWS Graviton

Plus tokenizer.json, vocab.txt, config.json and the sentence-transformers module configs. The quantized graphs are CPU-architecture specific — picking the wrong one is a silent performance loss or a load failure.

Usage

384-dimensional embeddings. The graph emits last_hidden_state, so apply attention-masked mean pooling then L2-normalize, per 1_Pooling/config.json and the Normalize module in modules.json.

Being a BERT model, the graph declares three inputs — input_ids, attention_mask and token_type_ids. Runtimes that feed only the first two will fail on a missing required input; pass zeros for token_type_ids.

import numpy as np, onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
tok.enable_padding(); tok.enable_truncation(max_length=256)
sess = ort.InferenceSession("onnx/model_quint8_avx2.onnx",
                            providers=["CPUExecutionProvider"])

enc = tok.encode_batch(["Is magnesium good for sleeping?"])
ids = np.array([e.ids for e in enc], dtype=np.int64)
mask = np.array([e.attention_mask for e in enc], dtype=np.int64)
out = sess.run(None, {"input_ids": ids, "attention_mask": mask,
                      "token_type_ids": np.zeros_like(ids)})[0]

w = mask[..., None].astype(np.float32)
vec = (out * w).sum(1) / np.clip(w.sum(1), 1e-9, None)
vec /= np.linalg.norm(vec, axis=1, keepdims=True)

License

Apache-2.0, inherited from the upstream model.