redis/langcache-embed-v3

Redis fine-tuned BiEncoder model for semantic caching on LangCache

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-modernbert-base on the LangCache Sentence Pairs (all) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for sentence pair similarity.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Alibaba-NLP/gte-modernbert-base
• Maximum Sequence Length: 100 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • LangCache Sentence Pairs (all)
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 100, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("redis/langcache-embed-v3")
# Run inference
sentences = [
    'According to him , the earth is the carrier of his artistic work , which is only integrated into the creative process by minimal changes .',
    'According to him , earth is the carrier of his artistic work being integrated into the creative process only by minimal changes .',
    'According to him , earth is the carrier of his creative work being integrated into the artistic process only by minimal changes .',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9609, 0.6602],
#         [0.9609, 1.0078, 0.6875],
#         [0.6602, 0.6875, 0.9961]], dtype=torch.bfloat16)

Evaluation

Metrics

Information Retrieval

• Dataset: test
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.6051
cosine_precision@1	0.6051
cosine_recall@1	0.5871
cosine_ndcg@10	0.7757
cosine_mrr@1	0.6051
cosine_map@100	0.7289

Training Details

Training Dataset

LangCache Sentence Pairs (all)

• Dataset: LangCache Sentence Pairs (all)
• Size: 126,938 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 8 tokens mean: 27.27 tokens max: 49 tokens	min: 8 tokens mean: 27.27 tokens max: 48 tokens	min: 7 tokens mean: 26.54 tokens max: 61 tokens

• Samples:

  anchor	positive	negative
  The newer Punts are still very much in existence today and race in the same fleets as the older boats .	The newer punts are still very much in existence today and run in the same fleets as the older boats .	how can I get financial freedom as soon as possible?
  The newer punts are still very much in existence today and run in the same fleets as the older boats .	The newer Punts are still very much in existence today and race in the same fleets as the older boats .	The older Punts are still very much in existence today and race in the same fleets as the newer boats .
  Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .	Turner Valley , , was located at Turner Valley Bar N Ranch Airport , southwest of Turner Valley Bar N Ranch , Alberta , Canada .	Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .

• Loss: losses.ArcFaceInBatchLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

LangCache Sentence Pairs (all)

• Dataset: LangCache Sentence Pairs (all)
• Size: 126,938 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 8 tokens mean: 27.27 tokens max: 49 tokens	min: 8 tokens mean: 27.27 tokens max: 48 tokens	min: 7 tokens mean: 26.54 tokens max: 61 tokens

• Samples:

  anchor	positive	negative
  The newer Punts are still very much in existence today and race in the same fleets as the older boats .	The newer punts are still very much in existence today and run in the same fleets as the older boats .	how can I get financial freedom as soon as possible?
  The newer punts are still very much in existence today and run in the same fleets as the older boats .	The newer Punts are still very much in existence today and race in the same fleets as the older boats .	The older Punts are still very much in existence today and race in the same fleets as the newer boats .
  Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .	Turner Valley , , was located at Turner Valley Bar N Ranch Airport , southwest of Turner Valley Bar N Ranch , Alberta , Canada .	Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .

• Loss: losses.ArcFaceInBatchLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• weight_decay: 0.001
• adam_beta2: 0.98
• adam_epsilon: 1e-06
• max_steps: 75000
• warmup_ratio: 0.1
• load_best_model_at_end: True
• optim: stable_adamw
• ddp_find_unused_parameters: False
• push_to_hub: True
• hub_model_id: redis/langcache-embed-v3
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.001
• adam_beta1: 0.9
• adam_beta2: 0.98
• adam_epsilon: 1e-06
• max_grad_norm: 1.0
• num_train_epochs: 3.0
• max_steps: 75000
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: stable_adamw
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: redis/langcache-embed-v3
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	test_cosine_ndcg@10
-1	-1	-	-	0.7730
0.0069	500	0.4944	0.3538	0.7662
0.0139	1000	0.2093	0.3354	0.7636
0.0208	1500	0.1626	0.3065	0.7666
0.0277	2000	0.1468	0.2998	0.7660
0.0347	2500	0.1329	0.2788	0.7692
0.0416	3000	0.1219	0.2857	0.7685
0.0485	3500	0.1192	0.2827	0.7679
0.0555	4000	0.1102	0.2771	0.7682
0.0624	4500	0.1067	0.2799	0.7685
0.0693	5000	0.1046	0.2746	0.7690
0.0762	5500	0.1012	0.2594	0.7688
0.0832	6000	0.0982	0.2798	0.7672
0.0901	6500	0.0939	0.3265	0.7645
0.0970	7000	0.0931	0.2835	0.7678
0.1040	7500	0.0893	0.2878	0.7653
0.1109	8000	0.0864	0.2687	0.7684
0.1178	8500	0.0878	0.2700	0.7691
0.1248	9000	0.0892	0.2659	0.7666
0.1317	9500	0.0853	0.2796	0.7665
0.1386	10000	0.0811	0.2681	0.7649
0.1456	10500	0.0817	0.2578	0.7659
0.1525	11000	0.082	0.2594	0.7658
0.1594	11500	0.0806	0.2790	0.7641
0.1664	12000	0.0807	0.2698	0.7653
0.1733	12500	0.0771	0.2750	0.7680
0.1802	13000	0.0759	0.2598	0.7676
0.1871	13500	0.0741	0.2620	0.7678
0.1941	14000	0.0735	0.2648	0.7677
0.2010	14500	0.075	0.2596	0.7683
0.2079	15000	0.0751	0.2578	0.7678
0.2149	15500	0.0698	0.2457	0.7701
0.2218	16000	0.0703	0.2667	0.7674
0.2287	16500	0.0708	0.2402	0.7692
0.2357	17000	0.0699	0.2889	0.7662
0.2426	17500	0.0698	0.2607	0.7690
0.2495	18000	0.0697	0.2931	0.7640
0.2565	18500	0.0662	0.2651	0.7666
0.2634	19000	0.0667	0.2666	0.7684
0.2703	19500	0.0661	0.2850	0.7687
0.2773	20000	0.0664	0.2698	0.7677
0.2842	20500	0.0636	0.2565	0.7691
0.2911	21000	0.064	0.2563	0.7679
0.2980	21500	0.0651	0.2529	0.7670
0.3050	22000	0.0658	0.2522	0.7685
0.3119	22500	0.065	0.2472	0.7694
0.3188	23000	0.0644	0.2534	0.7700
0.3258	23500	0.063	0.2497	0.7697
0.3327	24000	0.0616	0.2864	0.7613
0.3396	24500	0.0644	0.2886	0.7634
0.3466	25000	0.059	0.2611	0.7674
0.3535	25500	0.0633	0.2508	0.7707
0.3604	26000	0.0602	0.2371	0.7740
0.3674	26500	0.0581	0.2410	0.7714
0.3743	27000	0.0628	0.2380	0.7712
0.3812	27500	0.0621	0.2554	0.7686
0.3882	28000	0.0553	0.2587	0.7700
0.3951	28500	0.0579	0.2517	0.7708
0.4020	29000	0.0587	0.2401	0.7737
0.4089	29500	0.0596	0.2488	0.7688
0.4159	30000	0.0602	0.2574	0.7692
0.4228	30500	0.057	0.2502	0.7697
0.4297	31000	0.0555	0.2366	0.7721
0.4367	31500	0.0542	0.2350	0.7728
0.4436	32000	0.0549	0.2433	0.7704
0.4505	32500	0.0566	0.2329	0.7718
0.4575	33000	0.0547	0.2351	0.7721
0.4644	33500	0.0565	0.2391	0.7728
0.4713	34000	0.0581	0.2361	0.7746
0.4783	34500	0.0552	0.2407	0.7711
0.4852	35000	0.0553	0.2423	0.7724
0.4921	35500	0.0554	0.2341	0.7727
0.4991	36000	0.0536	0.2361	0.7726
0.5060	36500	0.0538	0.2414	0.7714
0.5129	37000	0.0563	0.2400	0.7725
0.5198	37500	0.0546	0.2366	0.7727
0.5268	38000	0.0529	0.2455	0.7709
0.5337	38500	0.0526	0.2353	0.7734
0.5406	39000	0.0542	0.2352	0.7736
0.5476	39500	0.0544	0.2385	0.7724
0.5545	40000	0.052	0.2345	0.7721
0.5614	40500	0.0532	0.2558	0.7701
0.5684	41000	0.0505	0.2533	0.7709
0.5753	41500	0.0527	0.2617	0.7708
0.5822	42000	0.0537	0.2538	0.7736
0.5892	42500	0.0522	0.2582	0.7698
0.5961	43000	0.0504	0.2554	0.7705
0.6030	43500	0.0512	0.2503	0.7734
0.6100	44000	0.0496	0.2414	0.7719
0.6169	44500	0.051	0.2313	0.7731
0.6238	45000	0.0491	0.2307	0.7733
0.6307	45500	0.0491	0.2340	0.7726
0.6377	46000	0.0494	0.2369	0.7710
0.6446	46500	0.0496	0.2382	0.7720
0.6515	47000	0.0483	0.2345	0.7728
0.6585	47500	0.0478	0.2315	0.7735
0.6654	48000	0.0484	0.2315	0.7744
0.6723	48500	0.048	0.2238	0.7745
0.6793	49000	0.0485	0.2288	0.7742
0.6862	49500	0.0485	0.2261	0.7752
0.6931	50000	0.049	0.2278	0.7751
0.7001	50500	0.0473	0.2307	0.7759
0.7070	51000	0.0466	0.2284	0.7766
0.7139	51500	0.0485	0.2317	0.7764
0.7209	52000	0.0494	0.2197	0.7765
0.7278	52500	0.0476	0.2260	0.7761
0.7347	53000	0.0428	0.2243	0.7761
0.7416	53500	0.0446	0.2283	0.7756
0.7486	54000	0.0461	0.2350	0.7754
0.7555	54500	0.0439	0.2399	0.7744
0.7624	55000	0.0452	0.2427	0.7736
0.7694	55500	0.0436	0.2383	0.7749
0.7763	56000	0.0472	0.2379	0.7750
0.7832	56500	0.042	0.2423	0.7744
0.7902	57000	0.0452	0.2317	0.7755
0.7971	57500	0.0436	0.2320	0.7753
0.8040	58000	0.0451	0.2287	0.7749
0.8110	58500	0.0442	0.2265	0.7762
0.8179	59000	0.0422	0.2266	0.7751
0.8248	59500	0.044	0.2275	0.7766
0.8318	60000	0.0429	0.2274	0.7767
0.8387	60500	0.0434	0.2325	0.7752
0.8456	61000	0.0431	0.2347	0.7756
0.8525	61500	0.0402	0.2283	0.7766
0.8595	62000	0.0438	0.2343	0.7762
0.8664	62500	0.0435	0.2355	0.7760
0.8733	63000	0.042	0.2316	0.7763
0.8803	63500	0.0404	0.2319	0.7761
0.8872	64000	0.043	0.2278	0.7772
0.8941	64500	0.0428	0.2321	0.7761
0.9011	65000	0.0432	0.2310	0.7760
0.9080	65500	0.0405	0.2307	0.7765
0.9149	66000	0.0442	0.2329	0.7760
0.9219	66500	0.0409	0.2308	0.7762
0.9288	67000	0.0423	0.2308	0.7767
0.9357	67500	0.0425	0.2272	0.7771
0.9427	68000	0.0382	0.2291	0.7759
0.9496	68500	0.0408	0.2267	0.7761
0.9565	69000	0.0418	0.2269	0.7765
0.9634	69500	0.0406	0.2299	0.7765
0.9704	70000	0.0404	0.2292	0.7759
0.9773	70500	0.0398	0.2291	0.7755
0.9842	71000	0.043	0.2279	0.7764
0.9912	71500	0.0401	0.2265	0.7761
0.9981	72000	0.0424	0.2251	0.7766
1.0050	72500	0.0314	0.2256	0.7759
1.0120	73000	0.0281	0.2265	0.7752
1.0189	73500	0.0271	0.2259	0.7756
1.0258	74000	0.0258	0.2258	0.7756
1.0328	74500	0.0247	0.2259	0.7759
1.0397	75000	0.0266	0.2258	0.7757

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 5.1.0
• Transformers: 4.56.0
• PyTorch: 2.8.0+cu128
• Accelerate: 1.10.1
• Datasets: 4.0.0
• Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}