isaxtsystems/isaxt-quiz-embeddings-v5

ISAXT Quiz Embeddings v5 — built by VELXT Studio

Studio: VELXT Studio Family: ISAXT Repository: isaxtsystems/isaxt-quiz-embeddings-v5 · Manifest branding: brand: VELXT, studio: VELXT Studio, family: ISAXT, domain: quiz, version: 5.

What it is — and what it is not

This is a retrieval encoder. It is not a chat model and it does not generate text.

You give it a trivia question or a paraphrase of one, and it returns a single 128-dimension vector. You compare that vector against vectors you have already stored, and you serve the answer you stored with the nearest question. The model never writes an answer, never invents one, and never sees an option list — it cannot choose A, B, C or D.

If you are looking for something that answers a question directly, this is the wrong artifact. If you want to find the closest question in your own corpus and return its verified answer verbatim, use this model as the retrieval component.

Intended use

ISAXT Quiz Embeddings v5 is designed for semantic retrieval over a curated trivia and quiz corpus.

The v5 corpus contains 11,172 rows, including 11,167 rows belonging to the training tenant. The intended serving pattern is to embed a user question, compare it against stored question embeddings, and retrieve the nearest verified question; the application then returns the answer already stored with it.

This makes the model suitable as the retrieval layer for large quiz systems with thousands of known questions. Multiple-choice records such as A/B/C/D can be stored and served by the application, but the embedding model itself does not select or rank answer options.

Model details

Architecture	BertModel (2 layers, 2 attention heads, hidden 128, intermediate 512)
Base model	prajjwal1/bert-tiny
Output	128-dim, mean-pooled over the attention mask, L2-normalized
Objective	Contrastive InfoNCE, temperature 0.05
Curriculum	Enabled
Epochs	2
Batch size	8
Learning rate	1e-5 (10% linear warmup)
Seed	20260915
Early stopping	patience 2 (never fired)
Grid search	Disabled
Max sequence length	128 tokens
Training device	CPU only — this trainer has no GPU code path
Training wall time	30.23 CPU minutes

hidden_size: 128 comes from the bert-tiny base, not from a projection head. This is a very small model; read the limitations before assuming otherwise.

Usage

import numpy as np
import torch
from transformers import AutoTokenizer, AutoModel

model_id = "isaxtsystems/isaxt-quiz-embeddings-v5"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()

def embed(text: str) -> np.ndarray:
    """Encode one complete question into a normalized 128-d vector."""
    normalized = " ".join(text.lower().split())
    encoded = tokenizer(
        normalized,
        truncation=True,
        max_length=128,
        padding="max_length",
        return_tensors="pt",
    )
    with torch.no_grad():
        hidden = model(**encoded)[0]
    mask = encoded["attention_mask"].unsqueeze(-1).to(hidden.dtype)
    pooled = (hidden * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1.0)
    return torch.nn.functional.normalize(pooled, p=2, dim=1)[0].cpu().numpy()

query = embed("who wrote the play hamlet")
corpus = [embed(q) for q in stored_questions]
best = int(np.argmax(corpus @ query))          # cosine, because both sides are unit norm
answer = stored_answers[best]                  # your stored answer, returned verbatim

⚠️ Warning: Pass the whole normalized string to the tokenizer — never a list of words. A list is interpreted as a batch of separate sequences, and [0] would return the embedding of the first word only, not the full question.

ONNX

The onnx/ directory holds the ADR-006 serving artifact: model.onnx (fp32, with its external-data sidecar model.onnx.data) and model.int8.onnx (dynamic INT8 weights).

import onnxruntime as ort
session = ort.InferenceSession("onnx/model.int8.onnx", providers=["CPUExecutionProvider"])

Batch size 1 only. Both exported graphs hard-fail on any input with more than one row. torch.onnx.export traced from a batch-1 sample, and although dynamic_axes names the batch axis on the graph inputs and outputs, the internal LayerNormalization node kept a literal dim_value of 1. You will see Shape mismatch attempting to re-use buffer. {1,128,128} != {2,128,128}. The sequence axis is dynamic. Loop over queries, or use the transformers encoder above, which batches fine. This is measured and reported by ml/scripts/verify_onnx_roundtrip.py on every run — it is a known limitation of this release.

Training data

Corpus	11,172 rows, of which 11,167 belong to the training tenant
Training examples	106,873 contrastive triplets (question + hard negatives)
Generation	Zero API calls. Every question was authored from a verified local fact pool
Structure	100% clean: 265 leaky/placeholder rows retired before this run
Dedup	Exact + fuzzy (Dice > 0.95) + semantic (cosine > 0.98), zero duplicates
Provenance	Per-row in the corpus; Open Trivia DB rows carry CC-BY-SA 4.0

The previous encoder (v4a) was measured on a corpus containing 216 questions that stated their own answer in the question text and 82 circular ones — exactly the rows that make a retrieval encoder look better than it is. 51 rows fall in both classes, so 265 unique rows were retired before this model was trained. This is why the numbers below are lower than v4a's and more trustworthy. Do not quote v4a's acc@1 0.9975 as a comparison; it was measured on a different, leaky corpus.

30 categories are covered: art, astronomy, biology, board_games, chemistry, economics, food, general, geography, history, law, literature, mathematics, medicine, movies, music, mythology, people, philosophy, physics, political_science, politics, psychology, science, sociology, sports, technology, tv, video_games, anime. Categories outside this list are out of scope and will miss rather than guess.

Training note

Training was performed in Google Colab on CPU. The final training run completed successfully, and the released model was validated using the reported evaluation results and ONNX round-trip checks.

Known serving limitation: the exported ONNX artifacts support batch size 1 only.

Performance (measured only)

metric	value	how measured
best_val_loss	1.748240	train_embeddings.py, 2 epochs (epoch 1: 1.754241, epoch 2: 1.748240), seed 20260915
held-out acc@1	0.9998	evaluate_retrieval.py --v2, 13,316-row test split
held-out MRR	0.9999	same run, same rows
zero-shot base acc@1	0.9887	prajjwal1/bert-tiny on the identical 13,316 rows (paired comparison)
zero-shot base MRR	0.9934	same
torch vs fp32 ONNX	min 1.000000 over 30 probes	verify_onnx_roundtrip.py, one probe per category, dev tenant, floor 0.999
torch vs int8 ONNX	min 0.995093, mean 0.996662 over 30 probes	same run, floor 0.970
fp32 vs int8 ONNX (exporter's own figure)	0.990005	single sample, recorded in onnx/manifest.json at export time
fp32 → int8 size	4.44 MiB vs 16.95 MiB	onnx/manifest.json (int8) + onnx/ dir bytes (fp32 graph + external data)

The round-trip gate is reproducible: every number in the ONNX rows above comes from re-encoding real corpus questions through the torch encoder and both graphs and comparing them. Every probe cleared both floors, so the exported artifact reproduces the trained model rather than merely loading without error.

Note the gap between the two int8 figures: the exporter measured one sample at 0.990005, the 30-probe battery measures 0.995093 minimum. Different onnxruntime versions quantize slightly differently. Both clear the 0.970 floor.

Benchmark scope clarification: held-out acc@1 and MRR are measured against each example's positive candidate and its sampled hard negatives (5-8 per example), NOT against the entire serving corpus. Full-corpus retrieval quality is reported separately by the platform's hybrid search evaluation.

Limitations (honest)

• Retrieval only. No generation, no answer selection, no option ranking.
• Tiny model. bert-tiny with a 128-dim output. Excellent at "find the nearest question", weak at telling confusable entities apart.
• Near-ceiling metrics on an easy task. acc@1 0.9998 is a property of the benchmark: distractors are near-duplicates of the answer drawn from the same category pool. Read it as "this corpus is well separated", not as evidence of general-purpose language understanding.
• No accent folding. Júpiter and Jupiter are different strings to the tokenizer. Normalize upstream.
• Lowercase only. The encoder was trained on normalize_text() output: lowercased, whitespace-collapsed. Match that preprocessing or your vectors will drift.
• Not calibrated as a standalone accept/reject threshold. In the serving stack (SEMANTIC_SEPARABLE = false) cosine contributes ranking signal through Reciprocal Rank Fusion and never accepts a candidate on its own. Do not invent a cosine cut-off from these numbers.
• CPU-only training lineage. 30.23 CPU minutes, no GPU code path. Reproducing this run on a GPU host needs a trainer change, not just different hardware.
• Single-tenant embedding build. See the repository note below.

Model size (vs 30 MiB budget)

artifact	size
Folder total (ONNX + tokenizer + config)	22.07 MiB
model.onnx (fp32 graph)	0.29 MiB
model.onnx.data (fp32 external data)	16.67 MiB
model.int8.onnx (INT8, self-contained)	4.44 MiB
Budget	30 MiB

Size is fixed by the architecture and never grows with data.

License

MIT. Copyright © 2026 VELXT Studio.

Citation

@misc{velxt2026isaxt,
  author = {VELXT Studio},
  title = {ISAXT Quiz Embeddings v5},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/isaxtsystems/isaxt-quiz-embeddings-v5}}
}

Versioning policy

Versions are immutable: v5 is frozen and never renamed. A retrain ships as v6 with its own card, manifest and eval table. Publishing requires a green round-trip gate plus explicit owner go-ahead.

Repository note: multi-tenant platform security

The embed_corpus.py tool used to build the serving database alongside this model was originally released with a cross-tenant isolation defect: it read every tenant's rows rather than scoping to the --tenant-id flag.

UPDATE — Platform Patched: As of the platform repository's Task 81 fix, this pipeline has been strictly secured:

• --tenant-id now enforces canonical UUID validation (8-4-4-4-12 hex) and lowercases the value to match the stored schema.
• The embedding SELECT is scoped with WHERE tenant_id = ?.
• The destructive DROP TABLE question_embeddings was replaced with tenant-scoped row-level writes, preventing one tenant's run from erasing another tenant's vectors.
• 48 TDD tests prove Tenant B is neither read nor written during Tenant A's embedding run.
• The fix was mutation-tested: removing the predicate, restoring the DROP, and restoring the global count all fail correctly.
• Full 9-gate CI passed (436 ML tests, 330 TypeScript tests).

The v5 model weights are frozen and unchanged. This note documents the security posture of the serving platform, not a change to the artifact.