emrevrg/Norovox-Embed-Zero

Norovox Embed Zero

🔬 Preview — This model is under active development; stronger versions will be released in this repository.

Text embeddings from the Norovox Alpha core, with three modes. The fast mode turns a sentence, a question or a passage into a 768-dimensional unit vector; the precise mode keeps one small vector per token and matches query and document word by word (late interaction, MaxSim) for stronger retrieval; the hybrid mode adds exact-word matching (BM25) to the precise mode; similar meanings land close together, so the vectors can be used for semantic search, retrieval-augmented generation, deduplication, clustering and similarity scoring.

architecture	NorovoxAlphaCore (RoPE, RMSNorm, grouped-query attention, SwiGLU), bidirectional encoder mode
parameters	192.2M
output	fast: 768-d vector (masked mean, L2; also 256/128/64-d and 1-bit) · precise: 128-d vector per token (MaxSim)
input	up to 256 tokens
runtime	PyTorch (CPU or GPU)
languages	trained and measured in English

Results

Measured with this repository's code on public evaluation sets that were not used for training. EmbeddingGemma-300M was run with sentence-transformers and its own official prompts (encode_query / encode_document), on the same machine and the same items. Higher is better.

model	params	STS-B test (Spearman)	SciFact (NDCG@10)	NFCorpus (NDCG@10)	CPU ms / text (1 thread)
Norovox Embed Zero — hybrid mode (precise + BM25)	192.2M	—	0.728	0.348	—
Norovox Embed Zero — precise mode	192.2M	—	0.661	0.315	—
Norovox Embed Zero — fast mode	192.2M	0.824	0.560	0.264	102
EmbeddingGemma-300M	307.6M	0.882	0.787	0.394	129

The precise mode adds +10.1 points on SciFact and +5.1 on NFCorpus over the fast mode of the same model and closes about 40% of the retrieval gap to EmbeddingGemma-300M; EmbeddingGemma is still higher on all three columns. The fast mode is about 1.27x faster than EmbeddingGemma-300M on one CPU thread (102 vs 129 ms per text, same machine; latency measured on the previous release, which has the identical architecture). The hybrid weight (0.7 precise + 0.3 BM25, min-max normalised) was chosen on the SciFact train and NFCorpus dev splits only; the test numbers above were measured once. Precise-mode storage is 256 bytes per token (128 × fp16). Retrieval is trained and measured in English.

Use

pip install "git+https://huggingface.co/emrevrg/Norovox-Embed-Zero#subdirectory=code"

from norovox_embed import NorovoxEmbed

m = NorovoxEmbed.from_pretrained("emrevrg/Norovox-Embed-Zero")
q = m.encode(["how do floods form in cities?"], kind="query")
d = m.encode(["Covered streams and old drains overflow during heavy rain.",
              "The museum opens at nine on weekdays."], kind="document")
print(m.similarity(q, d))          # cosine similarities (vectors are unit length)
s = m.encode(["A man is playing a guitar.", "Someone plays an instrument."])   # symmetric similarity: no kind

Precise mode (late interaction): index token vectors once, then rank with MaxSim.

docs = ["Covered streams and old drains overflow during heavy rain.", "The museum opens at nine on weekdays."]
index = m.encode_tokens(docs, kind="document")            # list of (n_tokens, 128) arrays — store these
print(m.search("how do floods form in cities?", doc_tokens=index, top_k=2))   # [(doc_index, score), ...]

Hybrid mode (precise + exact words): build the word index once and pass it with the token index.

from norovox_embed import BM25
words = BM25(docs)
print(m.search("how do floods form in cities?", documents=docs, doc_tokens=index, bm25=words, mode="hybrid", top_k=2))

Agent: memory, web research, self-learning

norovox_embed.agent turns the embeddings into a working memory and research loop:

• Memory that does not forget. Every conversation turn, every passage read from the web and everything you teach is stored on disk with its Norovox Embed Zero vector (~/.norovox/memory). Each new question recalls the most similar past turns and facts, so nothing is lost when the context window fills up.
• Web research. When memory has nothing close enough, the agent searches (Wikipedia full-text search in Turkish and English, DuckDuckGo Instant Answers; optionally your own SearXNG server or a Brave Search API key), reads the pages, splits them into passages, ranks the passages with Norovox Embed Zero and answers from the best ones with sources.
• Self-learning. Passages used in an answer are added to memory, so the next related question is answered without going to the web.
• Long tasks. The thinking model can ask for another search (ARA: <query>) up to max_steps times before answering.
• Pluggable thinking model. Norovox Embed Zero provides memory, search and ranking; the text itself is written by a generator: the local Norovox Alpha core by default, or any OpenAI-compatible endpoint (for example llama-server).

from norovox_embed import NorovoxEmbed
from norovox_embed.agent import Agent, EndpointThinker

agent = Agent(NorovoxEmbed.from_pretrained("emrevrg/Norovox-Embed-Zero"),
              thinker=EndpointThinker("http://127.0.0.1:8080/v1/chat/completions"))
agent.learn("Norovox Labs builds the Norovox and AUBIN model families.")
print(agent.chat("What does Norovox Labs build?"))
print(agent.trace)            # which steps ran: web search, learning

Command line: python -m norovox_embed.agent --endpoint http://127.0.0.1:8080/v1/chat/completions

How it was trained

1. Core. Norovox Alpha (NorovoxAlphaCore, 12 layers, d 768, Qwen2.5-compatible tokenizer) continued pre-training on 1133M tokens of FineWeb-Edu (2×T4, packed sequences of 1024, fp16).
2. Encoder mode. The same weights run with bidirectional attention and a padding mask (no causal mask).
3. Embedding training. 1.84M query–passage and sentence pairs from 25 open English sources (NLI, Natural Questions, S2ORC, SQuAD, HotpotQA, GooAQ, TriviaQA, PubMedQA, SPECTER, Quora, StackExchange, ELI5, Yahoo Answers, PAQ, Amazon-QA, CCNews, AGNews, NPR, XSum, SimpleWiki, AltLex, sentence-compression, WikiAnswers, CodeSearchNet, STS-B train). Every pair whose text matches an evaluation set (STS-B validation/test, SciFact and NFCorpus documents and queries) was removed. Noisy web pairs were dropped when the teacher did not rank the paired passage in its top 10; up to 7 hard negatives per pair were mined, never using passages the teacher scores within 5% of the correct one (likely false negatives). Distillation from Qwen3-Embedding-0.6B (Apache-2.0; its weights are not copied) on pre-computed teacher vectors, verified: the student learns the teacher's ranking only for queries where the teacher ranks the correct passage first (94% here); elsewhere it learns from the correct pair alone. Losses are applied at 768/256/128/64 dimensions and on 1-bit vectors, so shorter and binary vectors stay usable.
4. Precise mode. Continued from the fast-mode weights with a 768→128 token projection: MaxSim contrastive loss with hard negatives plus the same verified distillation (teacher scores used only where the teacher ranks the correct passage first), on a freshly built 25% slice of the data pipeline, 4 hours on 2×T4. The fast-mode vector is trained jointly so both modes stay usable.
5. Selection. Candidates (several fast-mode runs, weight averages, two precise-mode runs) compared on development splits only (STS-B validation, SciFact train queries, NFCorpus dev queries); test sets used once, for the tables.

Notes

• Every number comes from code/ (embed_train.py evaluation functions) on public data; nothing is hand-edited.
• CPU latency is one thread on a Kaggle CPU machine, the same machine and texts for both models.
• This product includes tokenizer vocabulary files compatible with and derived from Qwen2.5 (Alibaba Cloud Qwen team), licensed under the Apache License, Version 2.0. Training data: FineWeb-Edu (ODC-By), sentence-transformers datasets.

---

Support Norovox

Norovox is built by one 17-year-old high school student in Türkiye: no company, no sponsor, no budget. Every model on this page was trained on free GPU quotas that run out within days. When they run out, training stops halfway and the next, stronger version has to wait, sometimes for a week.

Your support goes directly into GPU time. Even a small contribution lets a training run finish instead of stopping, and helps the Nythos, N1, AUBIN and Norovox Embed models get better faster. Supporters are credited in the model cards and get early access to new releases.

Contact: zgremre@gmail.com · emrevrgdev@gmail.com

_{🇹🇷 Norovox'u Türkiye'de 17 yaşında bir lise öğrencisi tek başına geliştiriyor: şirket yok, sponsor yok, bütçe yok. Buradaki modellerin hepsi birkaç günde biten ücretsiz GPU kotalarıyla eğitildi; kota bitince eğitim yarıda kalıyor ve daha güçlü sürüm bekliyor. Desteğin doğrudan GPU süresine gider; küçük bir katkı bile bir eğitimin yarıda kalmadan tamamlanmasını sağlar. Destekçiler model kartlarında anılır ve yeni sürümlere erken erişir. İletişim: zgremre@gmail.com · emrevrgdev@gmail.com}

_{License: Apache-2.0. Norovox Alpha core and Norovox Embed Zero weights: built from scratch by Norovox Labs.}