tiny-embed-61m

Model Details | Usage | Evaluation | Contact | License

tiny-embed-61m maps Chinese text to a low-dimensional dense vector which can be used for tasks like retrieval, classification, clustering, or semantic search. It is a compact (61.4M parameters, 768-dim) model distilled from larger open Chinese embedding models, and it is designed to run cheaply on CPU or a single consumer GPU while keeping retrieval quality close to bge-base-zh-v1.5 (102M).

No query instruction is required — the model is trained and evaluated without any prefix. Passages and queries both take the original text.

************* 🌟Highlights🌟 *************

  • 61.4M parameters, 768-dim output, 512 tokens max length.
  • 0.6058 average on MTEB(cmn, v1) (31 tasks, no instruction), vs. bge-small-zh-v1.5 0.5734 and bge-base-zh-v1.5 0.6270 measured on the same harness.
  • Pure sentence-transformers pipeline: Transformer → Pooling(mean) → Normalize → Dense(512→768) → Normalize.

Model Details

Item Value
Model type Dense text embedding (bi-encoder)
Parameters 61,425,664 (backbone) + 393,216 (projection head)
Embedding dimension 768
Backbone hidden size 512
Layers 12
Attention heads 8
Pooling mean, then L2 normalize
Projection head linear 512 → 768, bias-free, identity activation (part of the trained model, not a post-processing step)
Max sequence length 512
Tokenizer Chinese BERT vocabulary (21,128 tokens)
Similarity cosine (embeddings are L2-normalized)
Precision fp16 / bf16 supported

The model was distilled from thenlper/gte-base-zh at its native 768 dimensions, so the student and the teacher share the same embedding space without dimensionality reduction.

Training data & benchmark overlap

Training data are Chinese retrieval, STS and instruction corpora (webqa, cmedqa, chinese_agriculture_qa, pawsx, wiki_zh, fineweb_edu_zh, clean_passages, alpaca_zh) plus synthesized queries. Benchmark-related subsets are explicitly declared in the MTEB model metadata:

Status Benchmark tasks
Declared (their corpora appear in training data) PAWSX, CmedqaRetrieval, CMedQAv1-reranking, CMedQAv2-reranking
Excluded (corpora removed from training data → zero-shot) T2Retrieval, T2Reranking (T2Ranking family), DuRetrieval (DuReader), ATEC, BQ, LCQMC, STSB

Usage

Using Sentence-Transformers

pip install -U sentence-transformers

Two things are non-optional when loading this checkpoint:

  1. trust_remote_code=True — the backbone is shipped as tiny_st_module.py inside this repository.
  2. Keep the 3_Dense module (512 → 768). It is part of the trained model, not a post-processing step: training used normalize(mean_pool(hidden)) → Dense(512→768) → normalize, and the published scores are measured on those 768-d outputs. Loading only the backbone and skipping 3_Dense yields 512-d embeddings that do not reproduce the scores below.
from sentence_transformers import SentenceTransformer

sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-4"]

model = SentenceTransformer("dynasty-ai/tiny-embed-61m", trust_remote_code=True)
embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)   # shape: (2, 768)

For short-query to long-passage retrieval, encode queries and passages the same way — no instruction is needed on either side:

queries = ["停经半年后又现点红是什么原因", "中医治疗失眠的方法"]
passages = ["样例文档-1", "样例文档-2"]

q_embeddings = model.encode(queries, normalize_embeddings=True)
p_embeddings = model.encode(passages, normalize_embeddings=True)
scores = q_embeddings @ p_embeddings.T

Using Huggingface Transformers

The checkpoint also loads with transformers, but you must reproduce the pooling and the projection head yourself:

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dynasty-ai/tiny-embed-61m", trust_remote_code=True)
model = AutoModel.from_pretrained("dynasty-ai/tiny-embed-61m", trust_remote_code=True)
model.eval()

sentences = ["样例数据-1", "样例数据-2"]
encoded_input = tokenizer(sentences, padding=True, truncation=True,
                          max_length=512, return_tensors="pt")

with torch.no_grad():
    token_embeddings = model(**encoded_input)["token_embeddings"]
    mask = encoded_input["attention_mask"].unsqueeze(-1).to(token_embeddings.dtype)
    pooled = (token_embeddings * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
    pooled = F.normalize(pooled, p=2, dim=1)          # mean pooling
    head = torch.nn.Linear(512, 768, bias=False)      # load from 3_Dense/pytorch_model.bin
    head.load_state_dict({"weight": torch.load("3_Dense/pytorch_model.bin",
                                               map_location="cpu")["linear.weight"]})
    embeddings = F.normalize(head(pooled), p=2, dim=1)

print(embeddings.shape)   # (2, 768)

For most users the sentence-transformers route above is simpler and bit-identical.

Evaluation

MTEB(cmn, v1) — Chinese MTEB, 31 tasks, no instruction prefix, cosine similarity, evaluated with mteb 1.39.7. Scores are the average of the standard task metric per task group.

Model Dimension Average Retrieval (8) Reranking (4) Clustering (4) Classification (9) PairClassification (2) STS (4)
tiny-embed-61m 768 60.58 63.50 66.23 49.61 62.10 68.78 52.56
bge-small-zh-v1.5 512 57.34 61.74 61.24 44.17 59.91 65.56 47.91
bge-base-zh-v1.5 768 62.70 69.40 64.87 47.30 64.21 73.17 53.90
gte-base-zh 768 65.72 71.73 67.10 53.63 66.52 74.75 58.13

Same-harness observations:

  • vs. bge-small-zh-v1.5 (24M): +3.24 average, better in every task group.
  • vs. bge-base-zh-v1.5 (102M, 1.7× parameters): −2.12 average — competitive on reranking / clustering / STS, still behind on retrieval.

Retrieval detail (T2Ranking dev, nDCG@10 / recall@10, no instruction): T2Retrieval nDCG@10 78.21 / recall@10 90.10; MMarcoRetrieval 65.29.

Note on benchmark tasks: T2Retrieval / T2Reranking are zero-shot for this model (T2Ranking corpora were excluded from training). Subsets that do appear in training data are listed in the MTEB model metadata.

Contact

Questions, bug reports and suggestions are welcome:

  • Open a discussion on the Community tab of this model page (dynasty-ai/tiny-embed-61m) — this is the model "issue tracker"; please include library versions, hardware and a minimal reproduction.
  • Or reach the maintainers of the dynasty-ai organization via their Hugging Face profile: https://huggingface.co/dynasty-ai.

License

tiny-embed-61m is licensed under the Apache-2.0 License.

It is produced by distillation from the MIT-licensed model thenlper/gte-base-zh; the copyright notice and permission notice of the upstream work apply to the corresponding upstream model. Training corpora retain their original licenses. The released weights can be used for commercial purposes free of charge.

Downloads last month
47
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Space using dynasty-ai/tiny-embed-61m 1