Edit model card

ZPoint Large Embedding for Chinese

  • [2024-06-04] Release zpoint_large_embedding_zh, and upload model weight to huggingface
  • [2024-06-05] Add training details

Training Details

Base Model

  1. We chose Stella as our base model.

Training Data

  1. Hard negative samping
  • For retrieval task, We sampled 10 hard negative passages/answers from top50-top200 related passages/answers for each query.
  • For classification/clustering tasks, we sampled 5 hard negative samples from other classes/cluster for each sample.
  • For classification/clustering tasks, we also used the category names of each class and cluster as positive and negative samples.
  1. Data synthesis by LLM (ZPoint-72B)
  • For retrieval tasks, we used LLM to rewrite each query, generating five different rewritten results.
  • For retrieval tasks, we also generated five new queries for some documents by LLM.
  • For non-retrieval tasks, we used LLM to rewrite the queries, generating five rewritten results for each query.
  • Finally, total amount of synthesized data is about 30 million.
  1. Collect more data for retrieval-type tasks

We constructed a dataset of approximately 100 million training samples through collection, machine translation, and LLM synthesis. This dataset includes data from various fields such as healthcare, law, electricity, automotive, and 3C (Consumer Electronics).

Training loss

  1. Multi-Task loss like Piccolo
  2. Matryoshka Representation Learning

Example

from sentence_transformers import SentenceTransformer
sentences1 = ["这个产品真垃圾"]
sentences2 = ["我太喜欢这个产品了"]
model = SentenceTransformer('iampanda/zpoint_large_embedding_zh')
embeddings_1 = model.encode(sentences1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)
Downloads last month
15,340
Safetensors
Model size
326M params
Tensor type
F32
·
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Space using iampanda/zpoint_large_embedding_zh 1

Evaluation results