VelaVec

A lightweight hybrid retrieval encoder: frozen distilled table + 3 layers of bidirectional attention + dual pooling heads.

	VelaVec	Teacher (bge-small)	Ratio
Parameters	9.8M	33M	30%
Embedding dim	256	384	—
Query latency (CPU)	103 µs	5.5 ms	53.9×
Doc latency 128 tok (CPU)	~580 µs (2-thread)	13 ms	~22×
Cold start	5.4 ms (Rust)	~29 s (PyTorch)	~2600×

Built for latency-sensitive retrieval on edge hardware: a pure-Rust inference path (BNNS/CoreML + NEON kernels) with no Python runtime.

Model details

• Trunk: frozen 30,522×256 static table (distilled from bge-small token embeddings + PCA) → 3 transformer blocks (RMSNorm, RoPE, SwiGLU FFN) → attention pooling.
• Dual heads: head=0 symmetric semantics (STS / similarity), head=1 asymmetric retrieval (query ↔ doc). One trunk, two uses.
• Training: 5-domain distillation (NLI + STS + MS MARCO retrieval + SciFact/NFCorpus/ArguAna) with teacher-embedding regression (relational + InfoNCE). See training notes.

Evaluation (MTEB, no prompts, max_len=256, CPU)

Task	VelaVec	bge-small	% of teacher
STS12 (cosine spearman)	0.7117	0.7744	91.9%
STSBenchmark	0.7596	0.8586	88.4%
Banking77 (kNN acc)	0.8365	0.8175	102.3%
SciFact (ndcg@10)	0.6310	0.7200	87.6%
ArguAna (ndcg@10)	0.4560	0.5950	76.6%
NFCorpus (ndcg@10)	0.2449	0.3371	72.7%

Domain-internal retrieval (own eval pools): NLI r@1 = 0.912, MS MARCO dev r@1 = 0.694 (1k-candidate pool).

Usage

Python (no transformers dependency for inference)

from modeling_velavec import VelaVec

model = VelaVec.from_pretrained("velavec_hf")   # or a HF repo id

# Symmetric semantics (STS / clustering) — head 0
sim = model.encode(["a man is playing guitar on stage",
                    "a child reads a book under a tree"], head=0)

# Asymmetric retrieval (query vs passage) — head 1
q = model.encode(["what is the capital of france"], head=1)
docs = model.encode(["Paris is the capital of France and its largest city."], head=1)
score = q @ docs.T

Tokenization uses BAAI/bge-small-en-v1.5 (bundled under tokenizer.* / vocab.txt).

Rust (edge inference)

weights.bin (39 MB, fp32, dual-head layout) is consumed by the Rust inference engine — BNNS/CoreML pre-packed GEMM graphs + NEON kernels, no Python/PyTorch at runtime.

Source code (cargo build; runs NLI+MARCO eval and latency benchmarks out of the box): xagent2025/velavec-rust-inference

• Query (len=15): 103 µs / 53.9× faster than teacher on CPU
• Doc (len=128): ~580 µs dual-threaded / ~780 µs single-threaded
• Cold start 5.4 ms; NLI throughput ~11.5k items/s, MARCO passages ~2.9k/s (2 threads)
• Byte-identical to PyTorch forward pass (max |Δ| = 0)

Head routing: NLI/similarity → head 0, MARCO retrieval → head 1.

Limitations

• OOD retrieval domains (medical / scientific / debate) reach 73–88% of teacher — the frozen 9.8M trunk can't fully absorb the 33M teacher's domain knowledge.
• Short input focus: best with ≤64 tokens for queries; docs up to 256 tokens.
• No instruction prompting (models trained without prompts; use as-is).

Training notes

Distillation protocol: teacher bge-small embeds → static table (PCA 256) → hybrid trunk warm-started from an earlier retrieval run → 5-domain multi-task distillation (relational KD + InfoNCE + embedding regression, symmetric & retrieval heads). Training data includes MS MARCO (research license) — review your use case for commercial compliance.

Files

File	Description
model.safetensors	PyTorch weights (fp32, dual-head, self-contained)
config.json	Architecture config
modeling_velavec.py	Standalone inference module
tokenizer.*, vocab.txt	bge-small tokenizer (BAAI/bge-small-en-v1.5, MIT)
weights.bin	Rust inference weights (39 MB, dual-head layout)