tomaarsen/multivector-ModernBERT-base-msmarco-cached-contrastive

ColBERT ModernBERT-base trained on MS MARCO triplets with GradCache

This is a Multi-Vector Encoder model finetuned from answerdotai/ModernBERT-base on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

• Model Type: Multi-Vector Encoder
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: maxsim
• Supported Modality: Text
• Training Dataset:
  • msmarco-bm25
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Multi-Vector Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_expansion': {'strategy': 'min', 'attend': False, 'token': None, 'length': 32}, 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-ModernBERT-base-msmarco-cached-contrastive")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'what is territorial sovereignty',
]
documents = [
    'territorial sovereignty. Exclusive right of a state to exercise its powers within the boundaries of its territory.',
    'Territorial preservation, as Agnew explains is merely one aspect of a states territorial integrity (2005). The lack of territorial sovereignty is often a key characteristic of so-called failed states where effective monopoly over the internal means of violence is lost.',
    '1 Active Transportï\x82§ Active Transport requires the cell to use energy, usually in the form of ATP.ï\x82§ Active Transport creates a charge gradient in the cell membrane. 2  For example in the mitochondrion, hydrogen ion pumps pump hydrogen ions into the intermembrane space of the organelle as part of making ATP. 3  15.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (32, 128) (24, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[28.2512, 25.4390,  6.2058]])

Evaluation

Metrics

Multi Vector Information Retrieval

• Datasets: NanoMSMARCO, NanoNQ, NanoFiQA2018, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with MultiVectorInformationRetrievalEvaluator

Metric	NanoMSMARCO	NanoNQ	NanoFiQA2018	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoHotpotQA	NanoNFCorpus	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
maxsim_accuracy@1	0.26	0.46	0.36	0.18	0.62	0.62	0.62	0.36	0.8	0.36	0.08	0.56	0.4694
maxsim_accuracy@3	0.5	0.6	0.46	0.32	0.78	0.88	0.82	0.5	0.94	0.62	0.48	0.66	0.7347
maxsim_accuracy@5	0.56	0.7	0.5	0.44	0.8	0.94	0.84	0.54	0.98	0.7	0.64	0.7	0.8163
maxsim_accuracy@10	0.76	0.8	0.6	0.6	0.88	1.0	0.88	0.58	1.0	0.78	0.8	0.76	1.0
maxsim_precision@1	0.26	0.46	0.36	0.18	0.62	0.62	0.62	0.36	0.8	0.36	0.08	0.56	0.4694
maxsim_precision@3	0.1667	0.2	0.2067	0.1133	0.5	0.3133	0.3533	0.3467	0.3867	0.2667	0.16	0.2267	0.4966
maxsim_precision@5	0.112	0.14	0.148	0.092	0.444	0.2	0.232	0.296	0.248	0.236	0.128	0.148	0.4939
maxsim_precision@10	0.076	0.084	0.09	0.072	0.408	0.106	0.128	0.226	0.132	0.166	0.08	0.086	0.4265
maxsim_recall@1	0.26	0.42	0.2184	0.085	0.0786	0.5767	0.31	0.0232	0.7007	0.0767	0.08	0.54	0.0365
maxsim_recall@3	0.5	0.56	0.315	0.1517	0.142	0.8433	0.53	0.0631	0.8987	0.1657	0.48	0.635	0.1163
maxsim_recall@5	0.56	0.63	0.3698	0.2167	0.1837	0.9033	0.58	0.0971	0.9487	0.2427	0.64	0.68	0.1859
maxsim_recall@10	0.76	0.75	0.4162	0.299	0.2782	0.9633	0.64	0.1163	0.986	0.3397	0.8	0.76	0.2997
maxsim_ndcg@10	0.4914	0.5801	0.3626	0.2205	0.5049	0.7963	0.5912	0.2812	0.8998	0.3195	0.4325	0.653	0.4669
maxsim_mrr@10	0.4082	0.554	0.4177	0.2862	0.7062	0.7632	0.726	0.4347	0.8812	0.5047	0.3155	0.6207	0.6344
maxsim_map@100	0.4174	0.5227	0.3272	0.1668	0.3796	0.7321	0.5168	0.1229	0.8643	0.2409	0.3245	0.6228	0.4014

Multi Vector Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with MultiVectorNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nq",
          "fiqa2018"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Metric	Value
maxsim_accuracy@1	0.36
maxsim_accuracy@3	0.4867
maxsim_accuracy@5	0.5667
maxsim_accuracy@10	0.6867
maxsim_precision@1	0.36
maxsim_precision@3	0.1778
maxsim_precision@5	0.1293
maxsim_precision@10	0.08
maxsim_recall@1	0.3078
maxsim_recall@3	0.4318
maxsim_recall@5	0.5044
maxsim_recall@10	0.6165
maxsim_ndcg@10	0.4659
maxsim_mrr@10	0.4492
maxsim_map@100	0.4172

Multi Vector Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with MultiVectorNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Metric	Value
maxsim_accuracy@1	0.4423
maxsim_accuracy@3	0.6381
maxsim_accuracy@5	0.7043
maxsim_accuracy@10	0.8031
maxsim_precision@1	0.4423
maxsim_precision@3	0.2874
maxsim_precision@5	0.2245
maxsim_precision@10	0.16
maxsim_recall@1	0.262
maxsim_recall@3	0.4154
maxsim_recall@5	0.4798
maxsim_recall@10	0.5699
maxsim_ndcg@10	0.5077
maxsim_mrr@10	0.5579
maxsim_map@100	0.4338

Training Details

Training Dataset

msmarco-bm25

• Dataset: msmarco-bm25 at ce8a493
• Size: 99,000 training samples
• Columns: query, positive, and negative
• Approximate statistics based on the first 100 samples:

  	query	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 5 tokens mean: 9.4 tokens max: 18 tokens	min: 28 tokens mean: 83.3 tokens max: 197 tokens	min: 24 tokens mean: 76.0 tokens max: 226 tokens

• Samples:

  query	positive	negative
  what is an agate made of	Agate is the name given to a group of silicate minerals that are made up primarily of chalcedony. Chalcedony is a member of the quartz family of minerals.	What is an agate, though, and what are the properties and feng shui meaning of agate? Let's find out. WHAT IS THE MEANING OF AGATE? As a form of chalcedony (type of quartz) agate exhibits a variety of colours , shapes, as well as an often present gentle iridescence.
  what is the socratic method?	The Socratic Learning Method (SLM) is a constructivist learning approach consisting of four key. steps: eliciting relevant preconceptions, clarifying preconceptions, testing ones own. hypotheses or encountered propositions, and deciding whether to accept the hypotheses or. propositions.	The Socratic Learning Method and the Inquiry-Based Learning Method. Since the Socratic dialogues are among the earliest documented instances of learning through. inquiry, it is reasonable to argue that what is now known as inquiry-based learning can trace its. origin to the Socratic Learning Method.
  what ditto means	• DITTO (noun). The noun DITTO has 1 sense: 1. a mark used to indicate the word above it should be repeated. Familiarity information: DITTO used as a noun is very rare. • DITTO (verb). The verb DITTO has 1 sense: 1. repeat an action or statement. Familiarity information: DITTO used as a verb is very rare.	Valerie Hill 0 I had texted one of my Auntie's and she sent the word Ditto back to me I was like what do that mean. Now I know!!! Acronym. DITTO in text means the same, or me too, or I agree.

• Loss: CachedMultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "score_metric": "colbert_scores",
      "mini_batch_size": 32,
      "mini_batch_num_tokens": null,
      "score_mini_batch_size": 32,
      "scale": 1.0,
      "size_average": true,
      "gather_across_devices": false
  }
  

Evaluation Dataset

msmarco-bm25

• Dataset: msmarco-bm25 at ce8a493
• Size: 1,000 evaluation samples
• Columns: query, positive, and negative
• Approximate statistics based on the first 100 samples:

  	query	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 4 tokens mean: 9.18 tokens max: 21 tokens	min: 22 tokens mean: 76.76 tokens max: 210 tokens	min: 30 tokens mean: 84.36 tokens max: 210 tokens

• Samples:

  query	positive	negative
  expected return is a function of	security market line Security market line (SML) is the representation of the capital asset pricing model. It displays the expected rate of return of an individual security as a function of systematic, non-diversifiable risk (its beta). beta Average sensitivity of a security's price to overall securities market prices.	A portfolio's expected return is the sum of the weighted average of each asset's expected return. Calculate a portfolio's expected return. To calculate the expected return of a portfolio, you need to know the expected return and weight of each asset in a portfolio.
  what smoker temperature for barbecue chicken	Chickens smoked hot and fast over indirect heat on the grill. But for pulled chicken, I wanted a slightly more intense smokiness to balance the barbecue sauce. I decided to do a side-by-side comparison, cooking one on the grill over indirect heat at around 375°F, and one in the smoker at 225°F.	Cook for an hour and 15 minutes. Check your smoked chicken breasts to make sure the temperature is still holding at about 250 degrees. Flip the chicken breasts and close the lid again. After 30 minutes, check the internal temperature of the chicken with a meat thermometer. You are looking for a temperature of 160 degrees before you can pull them off the smoker. Serve your smoked chicken breasts with a side of barbecue sauce for dipping.
  in classification of matter what is an element	Matter can be in the same phase or in two different phases for this separation to take place. Key Terms. mixture: Something that consists of diverse, non-bonded elements or molecules. element: A chemical substance that is made up of a particular kind of atom and cannot be broken down or transformed by a chemical reaction.	2. What four elements make up 96% of all living matter? The four elements that make up 96% of all living matter are oxygen, carbon, hydrogen and nitrogen. 3. What is the difference between an essential element and a trace element? An essential element is an element that an organism needs to live a healthy life and reproduce. A trace element is required by an organism in only minute quantities. Section 2 4. Sketch a model of an atom of helium, showing the electrons, protons, neutrons, and atomic nucleus. Neutrons Protons Electrons 5.

• Loss: CachedMultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "score_metric": "colbert_scores",
      "mini_batch_size": 32,
      "mini_batch_num_tokens": null,
      "score_mini_batch_size": 32,
      "scale": 1.0,
      "size_average": true,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 256
• num_train_epochs: 1
• learning_rate: 3e-05
• warmup_steps: 0.05
• bf16: True
• per_device_eval_batch_size: 32
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 256
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 3e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• max_length: None

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_maxsim_ndcg@10	NanoNQ_maxsim_ndcg@10	NanoFiQA2018_maxsim_ndcg@10	NanoBEIR_mean_maxsim_ndcg@10	NanoClimateFEVER_maxsim_ndcg@10	NanoDBPedia_maxsim_ndcg@10	NanoFEVER_maxsim_ndcg@10	NanoHotpotQA_maxsim_ndcg@10	NanoNFCorpus_maxsim_ndcg@10	NanoQuoraRetrieval_maxsim_ndcg@10	NanoSCIDOCS_maxsim_ndcg@10	NanoArguAna_maxsim_ndcg@10	NanoSciFact_maxsim_ndcg@10	NanoTouche2020_maxsim_ndcg@10
-1	-1	-	-	0.1000	0.1497	0.1543	0.1347	-	-	-	-	-	-	-	-	-	-
0.0103	4	5.8267	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0207	8	5.6903	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0310	12	5.4767	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0413	16	5.0988	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0517	20	4.2354	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0620	24	3.0957	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0724	28	2.4439	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0827	32	2.1717	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0930	36	1.8270	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1008	39	-	0.9998	0.4304	0.3701	0.3428	0.3811	-	-	-	-	-	-	-	-	-	-
0.1034	40	1.5308	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1137	44	1.3514	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1240	48	1.2883	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1344	52	1.1446	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1447	56	1.1350	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1550	60	1.1031	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1654	64	0.9816	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1757	68	0.9093	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1860	72	0.9494	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1964	76	0.8593	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2016	78	-	0.6296	0.4370	0.4710	0.3982	0.4354	-	-	-	-	-	-	-	-	-	-
0.2067	80	0.9200	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2171	84	0.8756	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2274	88	0.8491	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2377	92	0.7880	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2481	96	0.8130	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2584	100	0.8458	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2687	104	0.8617	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2791	108	0.7989	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2894	112	0.7277	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2997	116	0.7122	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3023	117	-	0.5629	0.4515	0.5383	0.3923	0.4607	-	-	-	-	-	-	-	-	-	-
0.3101	120	0.7367	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3204	124	0.7998	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3307	128	0.7553	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3411	132	0.7310	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3514	136	0.6776	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3618	140	0.7449	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3721	144	0.7055	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3824	148	0.6598	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3928	152	0.6659	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4031	156	0.6886	0.5203	0.4490	0.5421	0.3685	0.4532	-	-	-	-	-	-	-	-	-	-
0.4134	160	0.7026	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4238	164	0.5817	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4341	168	0.6364	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4444	172	0.6715	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4548	176	0.7659	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4651	180	0.6232	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4755	184	0.6615	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4858	188	0.6722	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4961	192	0.6091	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5039	195	-	0.5031	0.4836	0.5390	0.3795	0.4673	-	-	-	-	-	-	-	-	-	-
0.5065	196	0.6499	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5168	200	0.6139	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5271	204	0.6491	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5375	208	0.6666	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5478	212	0.6648	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5581	216	0.6836	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5685	220	0.6901	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5788	224	0.6772	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5891	228	0.6522	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5995	232	0.6261	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6047	234	-	0.4855	0.4740	0.5641	0.3426	0.4602	-	-	-	-	-	-	-	-	-	-
0.6098	236	0.5898	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6202	240	0.6333	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6305	244	0.6759	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6408	248	0.6389	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6512	252	0.6231	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6615	256	0.5636	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6718	260	0.5843	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6822	264	0.6028	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6925	268	0.6060	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7028	272	0.6343	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7054	273	-	0.4707	0.4728	0.5686	0.3417	0.4610	-	-	-	-	-	-	-	-	-	-
0.7132	276	0.6449	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7235	280	0.6012	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7339	284	0.6065	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7442	288	0.5939	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7545	292	0.6006	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7649	296	0.6582	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7752	300	0.5814	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7855	304	0.5842	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7959	308	0.6006	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8062	312	0.6268	0.4647	0.4914	0.5771	0.3454	0.4713	-	-	-	-	-	-	-	-	-	-
0.8165	316	0.5788	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8269	320	0.6145	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8372	324	0.5372	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8475	328	0.5663	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8579	332	0.5719	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8682	336	0.5525	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8786	340	0.5992	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8889	344	0.5530	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8992	348	0.5658	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.907	351	-	0.4543	0.4914	0.5801	0.3626	0.4781	-	-	-	-	-	-	-	-	-	-
0.9096	352	0.5474	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9199	356	0.5791	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9302	360	0.5582	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9406	364	0.5540	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9509	368	0.5548	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9612	372	0.5430	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9716	376	0.5718	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9819	380	0.5771	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9922	384	0.5648	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.0	387	-	0.4473	0.4747	0.5813	0.3416	0.4659	-	-	-	-	-	-	-	-	-	-
-1	-1	-	-	0.4914	0.5801	0.3626	0.5077	0.2205	0.5049	0.7963	0.5912	0.2812	0.8998	0.3195	0.4325	0.6530	0.4669

• The bold row denotes the saved checkpoint.

Training Time

• Training: 42.4 minutes
• Evaluation: 7.1 minutes
• Total: 49.5 minutes

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 5.7.0.dev0
• Transformers: 5.13.1
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultiVectorMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}