tiny-embed-27m

Model Details | Usage | Evaluation | Contact | License

tiny-embed-27m maps Chinese text to a low-dimensional dense vector which can be used for tasks like retrieval, classification, clustering, or semantic search. It is a compact (27.6M parameters, 512-dim) model distilled from larger open Chinese embedding models. Its backbone shares 4 unique layers applied 16 times (recursive depth), which keeps the parameter count low at inference-friendly latency.

No query instruction is required — the model is trained and evaluated without any prefix. Passages and queries both take the original text.

************* 🌟Highlights🌟 *************

  • 27.6M parameters, 512-dim output, 512 tokens max length.
  • 0.5963 average on MTEB(cmn, v1) (31 tasks, no instruction), vs. bge-small-zh-v1.5 (24M) 0.5734 measured on the same harness.
  • Standard sentence-transformers pipeline: Transformer → Pooling(mean) → Normalize (no extra projection layer).

Model Details

Item Value
Model type Dense text embedding (bi-encoder)
Parameters 27,599,872
Embedding dimension 512
Backbone hidden size 512
Layers 4 unique, repeated 16× (weight-shared recursive depth)
Attention heads 8
Pooling mean, then L2 normalize
Max sequence length 512
Tokenizer Chinese BERT vocabulary (21,128 tokens)
Similarity cosine (embeddings are L2-normalized)
Precision fp16 / bf16 supported

The model was distilled from a three-teacher vector soup built from BAAI/bge-small-zh-v1.5, BAAI/bge-base-zh-v1.5 and thenlper/gte-base-zh (all MIT-licensed).

Training data & benchmark overlap

Training data are Chinese retrieval, STS and instruction corpora (webqa, cmedqa, chinese_agriculture_qa, pawsx, wiki_zh, fineweb_edu_zh, clean_passages, alpaca_zh) plus synthesized queries. Benchmark-related subsets are explicitly declared in the MTEB model metadata:

Status Benchmark tasks
Declared (their corpora appear in training data) PAWSX, CmedqaRetrieval, CMedQAv1-reranking, CMedQAv2-reranking
Excluded (corpora removed from training data → zero-shot) T2Retrieval, T2Reranking (T2Ranking family), DuRetrieval (DuReader), ATEC, BQ, LCQMC, STSB

Usage

Using Sentence-Transformers

pip install -U sentence-transformers

trust_remote_code=True is required: the backbone is shipped as tiny_st_module.py inside this repository. Unlike tiny-embed-61m, this checkpoint has no projection head: the pipeline is exactly Transformer → Pooling(mean) → Normalize, so the 512-d mean-pooled vector is the final embedding.

from sentence_transformers import SentenceTransformer

sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-4"]

model = SentenceTransformer("dynasty-ai/tiny-embed-27m", trust_remote_code=True)
embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)   # shape: (2, 512)

Short-query to long-passage retrieval — encode both sides identically, no instruction:

queries = ["停经半年后又现点红是什么原因", "中医治疗失眠的方法"]
passages = ["样例文档-1", "样例文档-2"]

q_embeddings = model.encode(queries, normalize_embeddings=True)
p_embeddings = model.encode(passages, normalize_embeddings=True)
scores = q_embeddings @ p_embeddings.T

Using Huggingface Transformers

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dynasty-ai/tiny-embed-27m", trust_remote_code=True)
model = AutoModel.from_pretrained("dynasty-ai/tiny-embed-27m", trust_remote_code=True)
model.eval()

sentences = ["样例数据-1", "样例数据-2"]
encoded_input = tokenizer(sentences, padding=True, truncation=True,
                          max_length=512, return_tensors="pt")

with torch.no_grad():
    token_embeddings = model(**encoded_input)["token_embeddings"]
    mask = encoded_input["attention_mask"].unsqueeze(-1).to(token_embeddings.dtype)
    pooled = (token_embeddings * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
    embeddings = F.normalize(pooled, p=2, dim=1)      # mean pooling + L2 normalize

print(embeddings.shape)   # (2, 512)

Evaluation

MTEB(cmn, v1) — Chinese MTEB, 31 tasks, no instruction prefix, cosine similarity, evaluated with mteb 1.39.7. Scores are the average of the standard task metric per task group.

Model Dimension Average Retrieval (8) Reranking (4) Clustering (4) Classification (9) PairClassification (2) STS (4)
tiny-embed-27m 512 59.63 62.36 65.04 48.61 61.22 68.64 51.71
bge-small-zh-v1.5 512 57.34 61.74 61.24 44.17 59.91 65.56 47.91
m3e-small — 52.75 — — — — — —
bge-base-zh-v1.5 768 62.70 69.40 64.87 47.30 64.21 73.17 53.90

Same-harness observations:

  • vs. bge-small-zh-v1.5 (24M): +2.29 average, better in every task group.
  • vs. bge-base-zh-v1.5 (102M, 3.7× parameters): −3.07 average; the gap is concentrated in retrieval (−7.04) and pair classification (−4.53).

Retrieval detail (T2Ranking dev, no instruction): recall@10 89.56; T2Retrieval zero-shot (T2Ranking corpora excluded from training).

Note on benchmark tasks: T2Retrieval / T2Reranking are zero-shot for this model. Subsets that do appear in training data are listed in the MTEB model metadata.

Contact

Questions, bug reports and suggestions are welcome:

  • Open a discussion on the Community tab of this model page (dynasty-ai/tiny-embed-27m) — this is the model "issue tracker"; please include library versions, hardware and a minimal reproduction.
  • Or reach the maintainers of the dynasty-ai organization via their Hugging Face profile: https://huggingface.co/dynasty-ai.

License

tiny-embed-27m is licensed under the Apache-2.0 License.

It is produced by distillation from the MIT-licensed models BAAI/bge-small-zh-v1.5, BAAI/bge-base-zh-v1.5 and thenlper/gte-base-zh; the copyright notices and permission notices of those upstream works apply to the corresponding upstream models. Training corpora retain their original licenses. The released weights can be used for commercial purposes free of charge.

Downloads last month
41
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Space using dynasty-ai/tiny-embed-27m 1