litoo13/leafmind-gte-multilingual-4lang

gte-multilingual (base + reranker), vocabulary pruned for English, German, Persian and Arabic

Smaller copies of two Alibaba models, made for leafmind (on-device document question answering, AGPL-3.0), usable by anyone under the original Apache-2.0 licence:

Folder	Original	model_int8.onnx (CPU)	model_q4f16.onnx (browser, WebGPU)
gte-embed/	Alibaba-NLP/gte-multilingual-base via onnx-community/gte-multilingual-base @ 2edbf5e	340 → 171 MB	465 → 177 MB
gte-reranker/	Alibaba-NLP/gte-multilingual-reranker-base via onnx-community/gte-multilingual-reranker-base @ ee64367	341 → 172 MB	466 → 178 MB

Both formats are cut from onnx-community's files of the same revision and use the same tokenizer.json in each folder. model_int8.onnx is for ONNX Runtime on a CPU (leafmind's native build); model_q4f16.onnx (4-bit weights, 16-bit activations) is for ONNX Runtime Web on WebGPU (leafmind's browser build).

Both models together use about half the memory once loaded (on an Apple M4 with ONNX Runtime 1.30, after 512-token inputs: 1,230 MB → 669 MB).

What was changed

Nothing in the network itself. Two tables were cut:

1. Vocabulary: 72,928 of the 250,002 word pieces are kept (the same list for both models, build/keep_ids.json): special tokens, every piece without letters (digits, punctuation, symbols), every piece with Arabic-script or Greek letters, every one-character piece, and the Latin-script pieces that English and German text uses (found by tokenizing the paragraphs of the training splits of SQuAD 2.0, GermanQuAD and the English part of TyDi QA; no text from them is in these files). The word-embedding table loses the other rows (the kept rows are unchanged: in int8 the table has a single scale and zero point; in q4f16 it is a plain 16-bit table of 250,048 padded rows), and each tokenizer.json keeps the same pieces with their original scores, with new ids.
2. Rotary position tables: the first 1,024 of 65,536 positions are kept, so inputs must be at most 1,024 tokens (leafmind uses 512).

Text in other writing systems (Cyrillic, CJK, …) now becomes unknown tokens: use these files only for English, German, Persian and Arabic. English and German words built from pieces that were not kept are split into smaller pieces; on the test pages and questions of the benchmark below that happened in 15–31 % of English/German texts (1.5–12 % for Persian/Arabic), usually for one rare word.

Checks

int8 (model_int8.onnx):

• Same pieces and identical outputs (difference 0.0) as the original pair on test sentences in all four languages, wherever the split is unchanged.
• Broad question-answering benchmark of leafmind (14 public test sets: SQuAD 2.0, GermanQuAD, ARCD, PQuAD, TyDi QA en/ar, MLQA en/de/ar, XQuAD en/de/ar, QASPER, MMLongBench-Doc; 4,190 questions): original → pruned, answers right 1,439 → 1,444, wrong 102 → 103, "not in the document" right 910 → 907.

q4f16 (model_q4f16.onnx):

• Identical outputs (difference 0.0) to onnx-community's unpruned q4f16 files on the test sentences.
• Same benchmark, run natively on a CPU: answers right 1,375, wrong 95, "not in the document" right 907 (int8: 1,444 / 103 / 907) — about 5 % fewer answers shown, a little more cautious.
• In Chrome with ONNX Runtime Web 1.30.0 on WebGPU (Apple M4 Max): 14 ms per paragraph embedded, about 135 ms per question in leafmind's browser build; 152 of 153 sample answers the same as the native run of these files. (ONNX Runtime Web 1.22.0 computed these models wrongly on WebGPU; use 1.30.0 or newer.)

Rebuild

build/ holds the scripts: train_texts.py (training paragraphs) → prune.py <train_texts.json> <out> (keep list and tokenizer) → build.py <out> (cuts the int8 tables from the onnx-community files and compares outputs); prune_variants.py cuts the q4f16 (and fp16) files with the same keep list. Paths in the scripts point to leafmind's working folders; adjust them to yours. build/keep_ids.json has SHA-256 a5394a2be2e6b9ce83bd3d036314729a936c6c38208f290eeec18e193d3a8694.

Licence

Apache-2.0, as the original models (Alibaba-NLP); LICENSE holds the licence text. Changes: the vocabulary and rotary tables were cut as described above.

SHA-256: gte-embed/model_q4f16.onnx aa9ae652c7f6effbfc7aa57465f6072256926c944922dee3845cec456168864f, gte-reranker/model_q4f16.onnx 8514babe9d028c6314a0ff8d6495bb778b55f54d7d4d3928589314a32c833188 (from onnx-community's onnx/model_q4f16.onnx: 9726ec27b72918ca0ed491f7f28bb52e6a5c7b8e47a49bd5f6c08a93675a6b55 and d24aacb414ccd10944c00622eb2d36c7f90019a8c8bdb882c320dc20f746abec).