hanzceo/Jasper-Token-Compression-600M-GGUF

Jasper-Token-Compression-600M GGUF

To run this model, please use my fork at https://github.com/HanzCEO/llama.cpp/tree/impl/Jasper-Token-Compression-600M.

Benchmark MTEB English

Quantization	Accuracy	Precision	Recall	F1 Score	Spearman	Mean Sim
Q8_0	0.8448	0.8763	0.9556	0.9143	0.3062	0.6997
Q4_K_M	0.8477	0.8739	0.9631	0.9163	0.3080	0.7095
Q1_0	0.8484	0.8645	0.9782	0.9179	0.1435	0.8002

Running the Model

$ ./llama-embedding -m Jasper-Token-Compression-600M_Q4_K_M.gguf -p "An apple a day keeps the doctors away"
embedding 0: -0.018346 -0.009776 -0.053883  ... -0.006476  0.008699  0.009683 
embedding 1: -0.027236 -0.004766 -0.063195  ... -0.002329  0.007740  0.009831 
embedding 2: -0.039688  0.005092 -0.046118  ... -0.010706  0.001788  0.003643 
embedding 3: -0.031832  0.002946 -0.062942  ... -0.008288  0.002735  0.000762 
embedding 4: -0.048793 -0.006710 -0.065345  ... -0.010053  0.000352  0.007763 
embedding 5: -0.039835  0.009338 -0.063589  ... -0.010378 -0.001117 -0.000184 
embedding 6: -0.041486  0.012871 -0.071283  ... -0.006661  0.003343 -0.004718 
embedding 7: -0.012782 -0.002994 -0.052196  ... -0.011036  0.005822  0.009337 
embedding 8: -0.009351 -0.000308 -0.041070  ... -0.005398  0.001508  0.010260 

Original model card below.

---

Jasper-Token-Compression-600M

Introduction

Inspired by Deepseek-OCR, this is the first vector model in the Jasper and Stella series to use dynamic text token compression. Through the combination of vector distillation and contrastive learning, our model can compress text by 10x while still achieving excellent performance!

Original training codes: https://github.com/DunZhang/Jasper-Token-Compression-Training

Report: https://arxiv.org/abs/2511.14405

Wechat: zhdunt

X: https://x.com/dunn_zhang

Features

• ⭐⭐⭐ Supports bilingual (Chinese and English)
• ⭐⭐⭐⭐⭐⭐ Dynamic token compression - tested to achieve excellent results even when compressing text to 0.33x of original length
• ⭐⭐⭐ Combines vector distillation with contrastive learning to further improve performance on retrieval tasks
• ⭐⭐ 12 million unsupervised data distillation
• ⭐⭐ 0.6B parameter size

Technical Details

Dynamic Text Token Compression

My implementation is very simple: After text passes through the word_embedding layer, it immediately goes through a Qwen3MLP (approximately 3 fully connected layers), then I calculate the compressed length, and finally use adaptive_avg_pool1d to compress tokens to that length.

The compression length calculation logic is as follows:

real_length = 1000  # Actual token count of the text
length_threshold = 80  # Compress only if exceeding this threshold
compression_ratio = 0.333
if real_length <= length_threshold:
    # No compression
    pass
else:
    target_length = int(length_threshold + (real_length - length_threshold) * compression_ratio)

For implementation details, please refer to the modeling_qwen3_jasper.py file in this directory.

Vector Distillation + Contrastive Learning

First, we compute teacher vectors for texts in the contrastive learning training set, then use the following three losses during training:

1. Cosine loss: Standard vector distillation loss
2. InfoNCE (hard loss): Standard contrastive learning loss function
3. KL divergence (soft loss): KL divergence between student score matrix and teacher score matrix. The score matrix is the scores between query and all documents(i.e. positive doc, hard negative docs, other in-batch docs).

Evaluation

My prompt strategy and specific content are consistent with the QZhou model. Please refer to their evaluation script: https://github.com/Kingsoft-LLM/QZhou-Embedding

Usage

import torch
from sentence_transformers import SentenceTransformer

if __name__ == "__main__":
    model_name_or_path = "infgrad/Jasper-Token-Compression-600M"
    model = SentenceTransformer(
        model_name_or_path,
        model_kwargs={
            "torch_dtype": torch.bfloat16,
            "attn_implementation": "sdpa",  # We support flash_attention_2; sdpa; eager
            "trust_remote_code": True
        },
        trust_remote_code=True,
        tokenizer_kwargs={"padding_side": "left"},
        device="cpu",
    )

    queries = [
        "What is photosynthesis?",
        "Who invented the telephone?",
    ]
    documents = [
        "Photosynthesis is the process by which green plants use sunlight, carbon dioxide, and water to produce glucose and oxygen",
        "Alexander Graham Bell is credited with inventing the first practical telephone in 1876, receiving US patent number 174,465 for his device."
    ]
    # The smaller the compression_ratio parameter, the faster the speed, but the quality will correspondingly decrease.
    # Based on our parameter settings during training and test results, we recommend a range between 0.3-0.8.
    query_embeddings = model.encode(queries, prompt_name="query", normalize_embeddings=True, compression_ratio=0.3333)
    document_embeddings = model.encode(documents, normalize_embeddings=True, compression_ratio=0.3333)

    similarity = model.similarity(query_embeddings, document_embeddings)
    print(similarity)

Limitations and TODO

Retrieval performance

I found that distilled models struggle to approach the retrieval performance of teacher models, which is why I specifically used contrastive learning + distillation learning to enhance the student model. However, I found that while the enhanced model showed improvement on retrieval test sets, there is still a significant gap compared to mainstream models. Therefore, I believe that how to improve the retrieval performance of distilled models is a very necessary and valuable research direction.

More reasonable text token compression modules

There is limited research on text token compression currently, and I have only tried the simplest approach. I believe more reasonable text compression modules can definitely be found.

Text length

I only distilled texts up to 1024 tokens in length, so there should be performance degradation when text length exceeds 1024.

Citation

If you find our work worth citing, please use the following citation.

@misc{zhang2025jasperstelladistillationsota,
      title={Jasper and Stella: distillation of SOTA embedding models}, 
      author={Dun Zhang and Jiacheng Li and Ziyang Zeng and Fulong Wang},
      year={2025},
      eprint={2412.19048},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2412.19048}, 
}

@misc{zhang2025jaspertokencompression600mtechnicalreport,
      title={Jasper-Token-Compression-600M Technical Report}, 
      author={Dun Zhang and Ziyang Zeng and Yudong Zhou and Shuyang Lu},
      year={2025},
      eprint={2511.14405},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2511.14405}, 
}