Redis fine-tuned BiEncoder model for semantic caching on LangCache
This is a sentence-transformers model finetuned from Alibaba-NLP/gte-modernbert-base on the LangCache Sentence Pairs (all) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for sentence pair similarity.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: Alibaba-NLP/gte-modernbert-base
- Maximum Sequence Length: 100 tokens
- Output Dimensionality: 768 dimensions
- Similarity Function: Cosine Similarity
- Training Dataset:
- Language: en
- License: apache-2.0
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 100, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("redis/langcache-embed-v3")
# Run inference
sentences = [
'"If you click ""like"" on an old post that someone made on your wall yet you\'re no longer Facebook friends, will they still receive a notification?"',
'"If you click ""like"" on an old post that someone made on your wall yet you\'re no longer Facebook friends, will they still receive a notification?"',
'"If your teenage son posted ""La commedia e finita"" on his Facebook wall, would you be concerned?"',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 1.0000, 0.6758],
# [1.0000, 1.0000, 0.6758],
# [0.6758, 0.6758, 1.0078]], dtype=torch.bfloat16)
Evaluation
Metrics
Custom Information Retrieval
- Dataset:
test
- Evaluated with
ir_evaluator.CustomInformationRetrievalEvaluator
Metric | Value |
---|---|
cosine_accuracy@1 | 0.5956 |
cosine_precision@1 | 0.5956 |
cosine_recall@1 | 0.5781 |
cosine_ndcg@10 | 0.7776 |
cosine_mrr@1 | 0.5956 |
cosine_map@100 | 0.7276 |
cosine_auc_precision_cache_hit_ratio | 0.364 |
cosine_auc_similarity_distribution | 0.154 |
Training Details
Training Dataset
LangCache Sentence Pairs (all)
- Dataset: LangCache Sentence Pairs (all)
- Size: 132,354 training samples
- Columns:
anchor
,positive
, andnegative
- Approximate statistics based on the first 1000 samples:
anchor positive negative type string string string details - min: 4 tokens
- mean: 25.33 tokens
- max: 100 tokens
- min: 4 tokens
- mean: 24.98 tokens
- max: 100 tokens
- min: 5 tokens
- mean: 19.06 tokens
- max: 68 tokens
- Samples:
anchor positive negative What high potential jobs are there other than computer science?
What high potential jobs are there other than computer science?
Why IT or Computer Science jobs are being over rated than other Engineering jobs?
Would India ever be able to develop a missile system like S300 or S400 missile?
Would India ever be able to develop a missile system like S300 or S400 missile?
Should India buy the Russian S400 air defence missile system?
water from the faucet is being drunk by a yellow dog
A yellow dog is drinking water from the faucet
Childlessness is low in Eastern European countries.
- Loss:
losses.ArcFaceInBatchLoss
with these parameters:{ "scale": 20.0, "similarity_fct": "cos_sim", "gather_across_devices": false }
Evaluation Dataset
LangCache Sentence Pairs (all)
- Dataset: LangCache Sentence Pairs (all)
- Size: 132,354 evaluation samples
- Columns:
anchor
,positive
, andnegative
- Approximate statistics based on the first 1000 samples:
anchor positive negative type string string string details - min: 4 tokens
- mean: 25.33 tokens
- max: 100 tokens
- min: 4 tokens
- mean: 24.98 tokens
- max: 100 tokens
- min: 5 tokens
- mean: 19.06 tokens
- max: 68 tokens
- Samples:
anchor positive negative What high potential jobs are there other than computer science?
What high potential jobs are there other than computer science?
Why IT or Computer Science jobs are being over rated than other Engineering jobs?
Would India ever be able to develop a missile system like S300 or S400 missile?
Would India ever be able to develop a missile system like S300 or S400 missile?
Should India buy the Russian S400 air defence missile system?
water from the faucet is being drunk by a yellow dog
A yellow dog is drinking water from the faucet
Childlessness is low in Eastern European countries.
- Loss:
losses.ArcFaceInBatchLoss
with these parameters:{ "scale": 20.0, "similarity_fct": "cos_sim", "gather_across_devices": false }
Training Logs
Epoch | Step | test_cosine_ndcg@10 |
---|---|---|
-1 | -1 | 0.7776 |
Framework Versions
- Python: 3.12.3
- Sentence Transformers: 5.1.0
- Transformers: 4.56.0
- PyTorch: 2.8.0+cu128
- Accelerate: 1.10.1
- Datasets: 4.0.0
- Tokenizers: 0.22.0
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
- Downloads last month
- 932
Model tree for redis/langcache-embed-v3
Base model
answerdotai/ModernBERT-base
Finetuned
Alibaba-NLP/gte-modernbert-base
Dataset used to train redis/langcache-embed-v3
Evaluation results
- Cosine Accuracy@1 on testself-reported0.596
- Cosine Precision@1 on testself-reported0.596
- Cosine Recall@1 on testself-reported0.578
- Cosine Ndcg@10 on testself-reported0.778
- Cosine Mrr@1 on testself-reported0.596
- Cosine Map@100 on testself-reported0.728
- Cosine Auc Precision Cache Hit Ratio on testself-reported0.364
- Cosine Auc Similarity Distribution on testself-reported0.154