Redis fine-tuned BiEncoder model for semantic caching on LangCache

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-modernbert-base on the LangCache Sentence Pairs (all) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for sentence pair similarity.

Model Details

Model Description

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 100, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("redis/langcache-embed-v3")
# Run inference
sentences = [
    '"If you click ""like"" on an old post that someone made on your wall yet you\'re no longer Facebook friends, will they still receive a notification?"',
    '"If you click ""like"" on an old post that someone made on your wall yet you\'re no longer Facebook friends, will they still receive a notification?"',
    '"If your teenage son posted ""La commedia e finita"" on his Facebook wall, would you be concerned?"',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 1.0000, 0.6758],
#         [1.0000, 1.0000, 0.6758],
#         [0.6758, 0.6758, 1.0078]], dtype=torch.bfloat16)

Evaluation

Metrics

Custom Information Retrieval

  • Dataset: test
  • Evaluated with ir_evaluator.CustomInformationRetrievalEvaluator
Metric Value
cosine_accuracy@1 0.5956
cosine_precision@1 0.5956
cosine_recall@1 0.5781
cosine_ndcg@10 0.7776
cosine_mrr@1 0.5956
cosine_map@100 0.7276
cosine_auc_precision_cache_hit_ratio 0.364
cosine_auc_similarity_distribution 0.154

Training Details

Training Dataset

LangCache Sentence Pairs (all)

  • Dataset: LangCache Sentence Pairs (all)
  • Size: 132,354 training samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 4 tokens
    • mean: 25.33 tokens
    • max: 100 tokens
    • min: 4 tokens
    • mean: 24.98 tokens
    • max: 100 tokens
    • min: 5 tokens
    • mean: 19.06 tokens
    • max: 68 tokens
  • Samples:
    anchor positive negative
    What high potential jobs are there other than computer science? What high potential jobs are there other than computer science? Why IT or Computer Science jobs are being over rated than other Engineering jobs?
    Would India ever be able to develop a missile system like S300 or S400 missile? Would India ever be able to develop a missile system like S300 or S400 missile? Should India buy the Russian S400 air defence missile system?
    water from the faucet is being drunk by a yellow dog A yellow dog is drinking water from the faucet Childlessness is low in Eastern European countries.
  • Loss: losses.ArcFaceInBatchLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "gather_across_devices": false
    }
    

Evaluation Dataset

LangCache Sentence Pairs (all)

  • Dataset: LangCache Sentence Pairs (all)
  • Size: 132,354 evaluation samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 4 tokens
    • mean: 25.33 tokens
    • max: 100 tokens
    • min: 4 tokens
    • mean: 24.98 tokens
    • max: 100 tokens
    • min: 5 tokens
    • mean: 19.06 tokens
    • max: 68 tokens
  • Samples:
    anchor positive negative
    What high potential jobs are there other than computer science? What high potential jobs are there other than computer science? Why IT or Computer Science jobs are being over rated than other Engineering jobs?
    Would India ever be able to develop a missile system like S300 or S400 missile? Would India ever be able to develop a missile system like S300 or S400 missile? Should India buy the Russian S400 air defence missile system?
    water from the faucet is being drunk by a yellow dog A yellow dog is drinking water from the faucet Childlessness is low in Eastern European countries.
  • Loss: losses.ArcFaceInBatchLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "gather_across_devices": false
    }
    

Training Logs

Epoch Step test_cosine_ndcg@10
-1 -1 0.7776

Framework Versions

  • Python: 3.12.3
  • Sentence Transformers: 5.1.0
  • Transformers: 4.56.0
  • PyTorch: 2.8.0+cu128
  • Accelerate: 1.10.1
  • Datasets: 4.0.0
  • Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
Downloads last month
932
Safetensors
Model size
0.1B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for redis/langcache-embed-v3

Finetuned
(17)
this model

Dataset used to train redis/langcache-embed-v3

Evaluation results