Giga-Embeddings-instruct-480M-0826 — LiteRT

ai-sage/Giga-Embeddings-instruct-480M-0826 converted to LiteRT (.tflite) and to LiteRT-LM EmbeddingEngine bundles (.litertlm) for on-device inference. A Russian + English text-embedding model (Qwen3 architecture made bidirectional, 28 layers, 484M parameters) for retrieval, RAG, semantic similarity, classification and clustering, producing 1024-dimensional L2-normalized vectors — fully offline, on CPU. The upstream card reports MTEB (rus, v1.1) 70.98 and MTEB (eng, v2) 69.52.

Mean pooling and L2 normalization are inside the graph: one call in, one finished embedding out.

File	Recipe	Signatures	Size
Giga-Embeddings-instruct-480M-0826_wi8fc.tflite	int8 dynamic-range (linears + embedding table)	64, 128, 256, 512	512 MB	recommended
Giga-Embeddings-instruct-480M-0826_fp16.tflite	fp16 weights, float compute	64, 128, 256, 512	976 MB	desktop
Giga-Embeddings-instruct-480M-0826_wi8fc.litertlm	same int8 weights, EmbeddingEngine bundle	64, 128, 256, 512	509 MB	LiteRT-LM ≥ 0.17.0
Giga-Embeddings-instruct-480M-0826_fp16.litertlm	int8 embedding table + fp16 encoder, EmbeddingEngine bundle	64, 128, 256, 512	848 MB	LiteRT-LM ≥ 0.17.0, desktop

Prompts: instruction on the query, nothing on the document

The model was trained in the instruction style. For retrieval and other asymmetric tasks prepend a one-sentence task instruction to the query; documents are embedded raw. The format (from the upstream config_sentence_transformers.json, note the newline and the trailing space):

Instruct: Given a query, retrieve relevant passages
Query: {your text}

For symmetric tasks (semantic similarity, deduplication) the upstream card says a generic instruction or none at all both work; the numbers below use none. The prompt is plain text — it is tokenized normally and, like the <s> / </s> tokens, included in the mean (include_prompt: true upstream).

Signatures (.tflite)

Batch-1, right-padded static shapes: input_ids int32 [1, S], attention_mask int32 [1, S] (1 = real token, 0 = pad).

Signature	Output
embed_<S> for S in 64 / 128 / 256 / 512	output_0 float32 [1, 1024] — mean-pooled over valid positions, L2-normalized

input_ids must carry what the upstream tokenizer produces with special tokens: <s> (id 1) first and </s> (id 2) last — the tokenizer's post-processor adds both, and both are inside the mean. Pad with id 2 (</s> doubles as the pad token upstream; the mask, not the id, separates pad from eos) into the smallest signature that fits and set the mask accordingly. Padding is fully masked inside the graph: the same text through embed_64 / 128 / 256 / 512 returns identical vectors (max abs difference 0.0), and pad-region token ids cannot influence the output at all (measured 0.0).

Embeddings are L2-normalized, so cosine similarity is a dot product.

Usage (Python, .tflite)

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from transformers import AutoTokenizer

PAD_ID = 2
INSTRUCT = "Instruct: Given a query, retrieve relevant passages\nQuery: "
tok = AutoTokenizer.from_pretrained("ai-sage/Giga-Embeddings-instruct-480M-0826")
it = Interpreter(model_path="Giga-Embeddings-instruct-480M-0826_wi8fc.tflite", num_threads=8)
LENS = sorted(int(n.split("_")[1]) for n in it.get_signature_list())
runners = {s: it.get_signature_runner(f"embed_{s}") for s in LENS}

def embed(text, prefix=""):
    ids = tok(prefix + text)["input_ids"][:LENS[-1]]      # adds <s> ... </s>
    S = next(s for s in LENS if len(ids) <= s)
    x = np.full((1, S), PAD_ID, np.int32)
    m = np.zeros((1, S), np.int32)
    x[0, :len(ids)] = ids
    m[0, :len(ids)] = 1
    return list(runners[S](input_ids=x, attention_mask=m).values())[0][0]

q = embed("Где столица России?", INSTRUCT)
d = embed("Москва — столица Российской Федерации.")
print("cosine:", float(q @ d))

Texts longer than 512 tokens must be chunked (the upstream model accepts 8192, but a static on-device graph at that length is not practical).

Quality

Checks run on every variant against the PyTorch fp32 reference (the upstream "Transformers" example: remote-code forward, manual mean pooling, L2 normalization). That reference agrees with the upstream sentence-transformers stack at cosine 1.000000 on the check set, so the <s>/</s>-inside-the-mean contract is the one the authors ship.

1. Parity on a 10-text check set (3 instruction-prefixed queries in Russian and English, 4 documents, a 3-token input, an 81-token English sentence, one Japanese sentence), cosine vs the PyTorch fp32 reference:

Variant	min cos	mean cos
fp32 .tflite	1.000000	1.000000
int8 .tflite	0.992158	0.994439
fp16 .tflite	1.000000	1.000000

The upstream README example (the Russian query vs "Moscow is the capital of the Russian Federation" / "Paris is the capital of France") through the upstream sentence-transformers stack gives 0.6627 / 0.4297 (Moscow / Paris; the PyTorch fp32 reference and the fp32 and fp16 .tflite files give the same to four decimals); the int8 .tflite gives 0.6726 / 0.4184.

2. Semantic similarity, Spearman, no prompt, both sides raw:

Variant	STS17 en-en (150 pairs)	STS22 ru (150 pairs)
PyTorch fp32	0.8754	0.6781
fp32 .tflite	0.8754	0.6781
int8 .tflite	0.8802	0.6795
fp16 .tflite	0.8754	0.6781

3. Retrieval (SciFact-derived, 50 queries over a 600-document corpus, instruction on the query, documents raw):

Variant	nDCG@10	recall@5
PyTorch fp32	0.8450	0.9000
int8 .tflite	0.8450	0.9000
fp16 .tflite	0.8450	0.9000

The corpus is subsampled, so the absolute numbers are not comparable to published BEIR scores; the gate is the int8-vs-fp32 delta.

Speed

CPU/XNNPACK, single text, median:

Variant	Machine	embed_64	embed_128	embed_256	embed_512
int8	Mac (M4 Max), 16 threads	82 ms	116 ms	182 ms	343 ms
fp16	Mac (M4 Max), 16 threads	89 ms	132 ms	208 ms	373 ms

A static signature computes all S positions regardless of how many are real, so route each text to the smallest signature that fits. On-device numbers are in the EmbeddingEngine section below.

Conversion

Encoder lane — a direct multi-signature litert_torch trace of the HF model (remote code Qwen3BidirectionalModel, a Qwen3Model subclass with every attention layer non-causal), not an LLM export. Two things worth knowing if you reproduce it:

• transformers' bidirectional mask builder returns no mask when nothing is padded; under torch.export that branch bakes a graph that ignores attention_mask. Trace with padded samples, hand the model a [1,1,1,S] additive bias built from the mask (bit-exact with the vendor's 2-D-mask path in eager), and gate on pad-content invariance.
• Bidirectionality is proven by experiment (changing a late token moves position 0), not by reading a flag.

Script and full notes: hf-to-litertlm.

LiteRT-LM EmbeddingEngine bundles (.litertlm, litert-lm ≥ 0.17.0)

Since litert-lm 0.17.0 the runtime hosts embedding models directly through EmbeddingEngine (Python, C and Kotlin), so the same weights are also published as bundles that the engine loads without any host-side tokenization or pooling code:

file	contents	size
Giga-Embeddings-instruct-480M-0826_wi8fc.litertlm	int8 embedding table + int8 (dynamic-range) encoder, signatures for 64/128/256/512 tokens	509 MB
Giga-Embeddings-instruct-480M-0826_fp16.litertlm	int8 embedding table + fp16 encoder	848 MB

The vectors are identical to the .tflite path (cosine 1.000000 on the check set below; mean pooling over <s> + prompt + text + </s> and L2 normalization are inside the graph). Keep insert_special_tokens at its default (True): the bundle declares <s> as the bos token and </s> as the eos token, and the engine inserts both — the runtime's tokenizer does not run the tokenizer's own post-processor, so turning the option off drops both from the mean and silently returns wrong vectors (cosine 0.02–0.6 against the correct ones).

Python (pip install litert-lm>=0.17.0):

import litert_lm
from litert_lm.embedding_engine import EmbeddingEngine, EmbeddingOptions

INSTRUCT = "Instruct: Given a query, retrieve relevant passages\nQuery: "
engine = EmbeddingEngine("Giga-Embeddings-instruct-480M-0826_wi8fc.litertlm", backend=litert_lm.Backend.CPU())
q = engine.compute_embedding(INSTRUCT + "Где столица России?").embedding        # 1024 floats, L2-normalized
docs = engine.compute_embedding_batch(["Москва — столица Российской Федерации.",   # documents: no prompt
                                       "Париж — столица Франции."])

Inputs longer than 512 tokens are an error by default; pass EmbeddingOptions(input_overflow_strategy=InputOverflowStrategy.TRUNCATE) (the </s> stays at the end of the truncated window) or CHUNK_AND_AVERAGE to choose. EmbeddingOptions(output_size=256) keeps the first 256 dimensions.

Kotlin (com.google.ai.edge.litertlm:litertlm-android:0.17.0, needs a Kotlin 2.4 project):

val engine = EmbeddingEngine(EmbeddingEngineConfig(modelPath = "/data/local/tmp/Giga-Embeddings-instruct-480M-0826_wi8fc.litertlm", backend = Backend.CPU()))
engine.initialize()
val vec = engine.computeEmbedding(listOf(InputData.Text("Instruct: Given a query, retrieve relevant passages\nQuery: Где столица России?"))).embedding
engine.close()

Measured (CPU, single text, 10-text check set of 3–81 tokens, median):

device	runtime	init	per text
Galaxy S26 (SM-S942Q, Android 16)	litertlm-android 0.17.0, Kotlin EmbeddingEngine	2.6 s	68–84 ms up to 64 tokens, 143 ms for the 81-token text (encoder_128)
Mac (M4 Max)	litert-lm 0.17.0, Python	0.2 s (wi8fc) / 0.6 s (fp16)	52 ms (wi8fc) / 65 ms (fp16)

License

The base model is released under the MIT license by its authors (declared in the upstream model card). These files are converted, not original: the weights were exported to LiteRT and quantized (int8 dynamic-range / fp16); mean pooling and L2 normalization were folded into the graph. No fine-tuning or weight modification beyond quantization was performed.