stephvax/cosyvoice3-onnx-slim

CosyVoice3 — slimmed text-token embedding table

text_embedding_fp32.onnx in ayousanz/cosy-voice3-onnx is a graph containing one node: a Gather over embed_tokens.weight, [151936, 896] float32 — Qwen2's token embedding table, which CosyVoice3 uses as its LLM backbone. The file is 519.3 MiB and, measured through ONNX Runtime, an open session costs 1751 MiB of resident memory to serve a lookup that runs for 5 µs per synthesis. ONNX Runtime keeps roughly 3.4 copies of the table, at any precision.

The sibling graph llm_speech_embedding_fp16.onnx does the identical job for speech tokens as Gather + Cast over a float16 table. Same model, same backbone downstream, two precisions — an export inconsistency rather than a decision.

What is here

text_embedding_fp16.bin — the same table, float16, as a flat little-endian blob. No graph, no runtime.

shape   151936 × 896  (vocab × hidden)
dtype   IEEE-754 binary16, little-endian
layout  row-major; row i starts at byte offset i * 896 * 2 = i * 1792
size    272 269 312 bytes (259.7 MiB)
sha256  a25aa22be223a26845be0e4c6309d159264263c922f9238b856d81391338323b

Reading rows straight out of this file leaves them in the OS page cache — clean, file-backed pages, which macOS/iOS exclude from phys_footprint, the figure jetsam actually kills on. A synthesis embeds on the order of 80 tokens, i.e. 0.05 % of the table.

Fidelity — measured on the real table, not argued

Every value in the table lies within [-0.176, 0.138], so the dynamic range float32 buys is not used at all.

rows exceeding the fp16 ceiling (65504)	0
max absolute error	5.9e-5
mean absolute error	2.5e-6
per-row cosine, minimum over all 151936 rows	0.9999997
rows below cosine 0.9999	0

A row is what the backbone consumes, and that backbone already carries every one of its own weights in float16 — so rounding the embedding adds error strictly below what the very next layer already imposes.

End-to-end, one variant per process (measuring several in a row is worthless: the first session grows the heap and later ones then read as costing ~0 MiB):

form	disk	load	Δ footprint	per synthesis	min cosine
fp32 ONNX (upstream)	519.3 MiB	196 ms	1751 MiB	0.005 ms	—
fp16 ONNX + Cast	259.7 MiB	88 ms	851 MiB	0.023 ms	0.99999982
int8 ONNX + per-row scale	130.4 MiB	42 ms	405 MiB	0.020 ms	0.99991089
this blob, gathered by the host	259.7 MiB	—	~6 MiB	1.0 ms	0.99999997

The blob has the best fidelity of the three because there is no round trip through the graph optimiser.

Provenance

Derived from ayousanz/cosy-voice3-onnx (Apache-2.0) by casting the single initializer to float16 and writing it out raw. The conversion script lives at tools/cosyvoice3_text_embedding/build_variants.py in the project that uses this.