voyage-4-nano — LiteRT

voyageai/voyage-4-nano converted to LiteRT (.tflite) for on-device inference. Voyage AI's multilingual text-embedding model producing 2048-dimensional L2-normalized vectors — fully offline, on CPU.

The 1024→2048 projection, mean pooling and L2 normalization are inside the graph: one call in, one finished embedding out.

File	Recipe	Signatures	Size
voyage-4-nano_wi8fc.tflite	int8 dynamic-range	64, 128, 256, 512	364 MB	recommended
voyage-4-nano_fp16.tflite	fp16 weights, float compute	64, 128, 256, 512	696 MB	desktop only — see Memory

Both are task-lossless against the PyTorch reference on every gate below; fp16 additionally reproduces the reference to every printed digit.

Shared embedding space, on device

The Voyage 4 series shares one embedding space across sizes: vectors from voyage-4-large, voyage-4, voyage-4-lite and voyage-4-nano are directly comparable. The deployment this enables — index built server-side at full precision, queried on device by the int8 nano — is exactly what we gated: documents encoded with the PyTorch reference, queries with the int8 artifact, retrieval nDCG@10 0.9283 vs the 0.9292 all-PyTorch control (recall@5 and hit@1 unchanged). Quantization does not break the space.

Prompts

The upstream contract puts a prompt on both sides (from config_sentence_transformers.json):

• queries: "Represent the query for retrieving supporting documents: "
• documents: "Represent the document for retrieval: "

For symmetric tasks (similarity, clustering) upstream defines no prompt — encode raw. (In our Japanese retrieval gate the query prompt happened to be score-neutral, but the prompts above are the documented contract this artifact was verified under.)

Matryoshka dimensions

The model is MRL-trained: you may truncate the 2048-d output to 1024/512/256 and re-normalize. Measured on this artifact (int8, Japanese retrieval): 2048-d nDCG@10 0.9275 → 256-d 0.8962, and truncated int8 queries stay compatible with a truncated full-precision index (0.8989 vs 0.8990 control) — an 8× smaller vector for ~0.03 nDCG.

Signatures

Batch-1, right-padded static shapes: input_ids int32 [1, S], attention_mask int32 [1, S] (1 = real token, then 0s — padding must be contiguous on the right), for S in 64 / 128 / 256 / 512.

Signature	Output
embed_64 / embed_128 / embed_256 / embed_512	output_0 float32 [1, 2048] — masked mean over the projected token states, L2-normalized

Pad the token ids into the smallest signature that fits and set the mask accordingly. The mean is taken over sum(mask) inside the graph, so the result is independent of which signature you route through: the same text through embed_64/128/256/512 returns bitwise identical vectors, and pad-region token ids cannot influence the output at all.

Embeddings are L2-normalized, so cosine similarity is a dot product.

Usage (Python)

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from transformers import AutoTokenizer

PAD_ID = 151643
QUERY_PROMPT = "Represent the query for retrieving supporting documents: "
DOC_PROMPT = "Represent the document for retrieval: "

tok = AutoTokenizer.from_pretrained("voyageai/voyage-4-nano")
it = Interpreter(model_path="voyage-4-nano_wi8fc.tflite", num_threads=8)

LENS = sorted(int(n.split("_")[1]) for n in it.get_signature_list())
runners = {s: it.get_signature_runner(f"embed_{s}") for s in LENS}

def embed(text, prompt=""):
    ids = tok(prompt + text)["input_ids"][:LENS[-1]]
    S = next(s for s in LENS if len(ids) <= s)
    x = np.full((1, S), PAD_ID, np.int32)
    m = np.zeros((1, S), np.int32)
    x[0, :len(ids)] = ids
    m[0, :len(ids)] = 1
    return list(runners[S](input_ids=x, attention_mask=m).values())[0][0]

q = embed("日本で一番高い山は？", prompt=QUERY_PROMPT)
d = embed("Mount Fuji is the highest mountain in Japan, at 3,776 meters.",
          prompt=DOC_PROMPT)
print("cosine:", float(q @ d))
# Matryoshka: 256-d variant of the same embedding
q256 = q[:256] / np.linalg.norm(q[:256])

Texts longer than 512 tokens must be chunked (the upstream model accepts 32768, but a static on-device graph at that length is not practical).

Quality

Four independent checks, each run on every variant against the PyTorch fp32 reference.

1. The base card's own usage example (Red Planet query × 4 planet documents): every variant ranks the Mars document first; fp16 reproduces the reference score matrix bit-exactly, int8 within 0.003 cosine.

2. JSTS (Japanese semantic similarity, JGLUE v1.3 validation, 300 pairs, Spearman, no prompt): PyTorch 0.8406, int8 0.8407, fp16 0.8406.

3. JSQuAD retrieval (Japanese question → Wikipedia paragraph, 800-paragraph corpus, 150 questions, prompts both sides): PyTorch nDCG@10 0.9292 / hit@1 0.860; int8 0.9275 / 0.860; fp16 identical to PyTorch. The corpus is subsampled, so absolute numbers are not comparable to published benchmarks; at 150 queries the gate cannot resolve differences below a couple of percent.

4. Cross-variant retrieval — the shared-space deployment shape. Documents encoded with the PyTorch model, queries with the int8 artifact: nDCG@10 0.9283 vs the 0.9292 all-PyTorch control, recall@5/hit@1 unchanged, and the same at 256 Matryoshka dimensions (0.8989 vs 0.8990).

Multilingual spot-check (STS17 Spearman, 100 pairs each, int8): en-en 0.873 / ko-ko 0.844 / es-en 0.758 / en-ar 0.775, all within 0.009 of the PyTorch reference.

The int8 deltas here are unusually small for a 4-signature int8 conversion — consistent with the base model's quantization-aware training.

Speed

CPU/XNNPACK, median of 10 runs, 75%-full signatures:

Variant	Machine	embed_128	embed_512
wi8fc	M4 Max Mac, 16 threads	74 ms	245 ms
fp16	M4 Max Mac, 16 threads	81 ms	267 ms

A static signature computes all S positions regardless of how many are real, so route each text to the smallest signature that fits.

Memory

Measured at load+invoke on an M4 Max Mac (8 threads), all four signatures loaded: the int8 file peaks at ~1.15 GiB. The fp16 file peaks at ~7 GiB (XNNPACK expands fp16 weights to fp32 per signature subgraph) — it is a desktop artifact; use int8 on device.

Conversion

Encoder lane — a direct multi-signature litert_torch trace of the HF model (not an LLM export). Despite the config's Qwen3ForCausalLM label, the model is bidirectional (the repo's remote code flips every attention layer non-causal); the bidirectional+padding attention bias is built by hand inside the traced wrapper, and the 1024→2048 projection is applied per-token before the mean pool, exactly as the reference does. Gated on: bidirectionality (editing the last token must move earlier hidden states), pad-content invariance (bitwise), mask liveness (dropping the last real token must move the output), cross-signature bitwise agreement, and equivalence to the repo's own remote-code forward (cos ≥ 0.999999).

Script and full notes: hf-to-litertlm.

License

Apache 2.0, inherited from the base model by Voyage AI.

Modification notice: these files are converted, not original. The weights were exported to LiteRT and quantized (int8 dynamic-range / fp16); the output projection, mean pooling and L2 normalization were folded into the graph. No fine-tuning or weight modification beyond quantization was performed.