rafiazarin/arxiv-ms-marco-embedding-mixed

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("rafiazarin/arxiv-ms-marco-embedding-mixed")
# Run inference
sentences = [
    "What were the upper limits on branching ratios for b' -> bZ and b' -> cW obtained from the DELPHI experiment at LEP-II?",
    "A search for the pair production of fourth generation b'-quarks was performed using data taken by the DELPHI detector at LEP-II. The analysed data were collected at centre-of-mass energies ranging from 196 to 209 GeV, corresponding to an integrated luminosity of 420 pb^{-1}. No evidence for a signal was found. Upper limits on BR(b' -> bZ) and BR(b' -> cW) were obtained for b' masses ranging from 96 to 103 GeV/c^2. These limits, together with the theoretical branching ratios predicted by a sequential four generations model, were used to constrain the value of R_{CKM}=|V_{cb'}/V_{tb'}V_{tb}|, where V_{cb'}, V_{tb'} and V_{tb} are elements of the extended CKM matrix.",
    'We study the statefinder parameter in the five-dimensional big bounce model, and apply it to differentiate the attractor solutions of quintessence and phantom field. It is found that the evolving trajectories of these two attractor solutions in the statefinder parameters plane are quite different, and that are different from the statefinder trajectories of other dark energy models.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.5622, 0.0138],
#         [0.5622, 1.0000, 0.0719],
#         [0.0138, 0.0719, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: val
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.995
cosine_accuracy@3	0.995
cosine_accuracy@5	0.995
cosine_accuracy@10	1.0
cosine_precision@1	0.995
cosine_precision@3	0.3317
cosine_precision@5	0.199
cosine_precision@10	0.1
cosine_recall@1	0.995
cosine_recall@3	0.995
cosine_recall@5	0.995
cosine_recall@10	1.0
cosine_ndcg@10	0.9967
cosine_mrr@10	0.9957
cosine_map@100	0.9957

Training Details

Training Dataset

Unnamed Dataset

• Size: 2,700 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 5 tokens mean: 18.11 tokens max: 59 tokens	min: 27 tokens mean: 126.15 tokens max: 256 tokens

• Samples:

  anchor	positive
  where did gloria vanderbilt get her fortunes	Gloria Vanderbilt inherited most of her net worth from her father Reginald who was the only male heir to the Vanderbilt railroad fortune. Her father died when Gloria was just one year old. He left her a trust fund that was worth $5 million in 1925 which is equal to roughly $70 million in today's dollars.
  when flexing the hip what muscles shorten	Found deep in the abdominal cavity, they are some of the strongest muscles in the body, notes Stephanie E. Siegrist, MD, an orthopedic surgeon in Rochester, N.Y., and spokeswoman for the American Academy of Orthopaedic Surgeons. You put a lot of stress on your hip flexors when you sprint or kick. Hip Flexor Pain: It's All in the Hips. To avoid hip flexor pain, you should pay more attention to these muscles, Dr. Siegrist explains. When you are seated, your knees are bent and your hip muscles are flexed and often tighten up or become shortened. “Because we spend so much of our time in a seated position with the hip flexed, the hip flexor has the potential to shorten up
  What was the average accuracy achieved by the Bayesian rough set approach when estimating the risk of HIV given demographic data?	This paper proposes an approach to training rough set models using Bayesian framework trained using Markov Chain Monte Carlo (MCMC) method. The prior probabilities are constructed from the prior knowledge that good rough set models have fewer rules. Markov Chain Monte Carlo sampling is conducted through sampling in the rough set granule space and Metropolis algorithm is used as an acceptance criteria. The proposed method is tested to estimate the risk of HIV given demographic data. The results obtained shows that the proposed approach is able to achieve an average accuracy of 58% with the accuracy varying up to 66%. In addition the Bayesian rough set give the probabilities of the estimated HIV status as well as the linguistic rules describing how the demographic parameters drive the risk of HIV.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 200 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 200 samples:

  	anchor	positive
  type	string	string
  details	min: 9 tokens mean: 26.74 tokens max: 51 tokens	min: 37 tokens mean: 157.56 tokens max: 256 tokens

• Samples:

  anchor	positive
  What spin-flip and skyrmion-like textures are possible if the sign of the 2DEG g-factor is changed?	A theory of collective states in a magnetically quantized two-dimensional electron gas (2DEG) with half-filled Landau level (quantized Hall ferromagnet) in the presence of magnetic 3d impurities is developed. The spectrum of bound and delocalized spin-excitons as well as the renormalization of Zeeman splitting of the impurity 3d levels due to the indirect exchange interaction with the 2DEG are studied for the specific case of n-type GaAs doped with Mn where the Lande` g-factors of impurity and 2DEG have opposite signs. If the sign of the 2DEG g-factor is changed due to external influences, then impurity related transitions to new ground state phases, presenting various spin-flip and skyrmion-like textures, are possible. Conditions for existence of these phases are discussed. PACS: 73.43.Lp, 73.21.Fg, 72.15.Rn
  How do right modules over operads relate to functors on categories of algebras?	In the theory of operads we consider functors of generalized symmetric powers defined by sums of coinvariant modules under actions of symmetric groups. One observes classically that the construction of symmetric functors provides an isomorphism from the category of symmetric modules to a subcategory of the category of functors on the base category. The purpose of this book is to obtain a similar relationship for functors on a category of algebras over an operad. We observe that right modules over operads, symmetric modules equipped with a right operad action, give rise to functors on categories of algebras and we prove that this construction yields an embedding of categories. Then we check that right modules over operads form a model category. In addition we prove that weak-equivalences of right modules correspond to pointwise weak-equivalences at the functor level. As a conclusion, we obtain that right modules over operads supply good models for the homotopy of associated functors on ...
  What specific wave function modification, such as the Jastrow or Villars correlator, is used in the fixed center-of-mass approximation for the ^{4}He nucleus?	The approach exposed in the recent paper (A. Shebeko, P. Papakonstantinou, E. Mavrommatis, Eur. Phys. J. A 27, 143 (2006)) has been applied in studying center-of-mass motion effects on the nucleon density and momentum distributions in nuclei. We are focused upon effects due to the center-of-mass and short-range nucleon correlations embedded in translationally invariant ground-state wavefunctions. The latter are constructed in the so-called fixed center-of-mass approximation, starting with a Slater determinant wave function modified by some correlator (e.g., after Jastrow or Villars). It is shown how one can simplify evaluation of the corresponding expectation values that determine the distributions. The analytic expressions derived here involve the own "Tassie-Barker" factors for each distribution. As an illustration, numerical calculations have been carried out for the nucleus ^{4}He with the Slater determinant to describe the nucleon (1s)^4 configuration composed of single-particle o...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_steps: 0.1
• fp16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	val_cosine_ndcg@10
0.2941	25	0.0158	-	-
0.5882	50	0.0145	-	-
0.8824	75	0.0118	-	-
1.0	85	-	0.0083	0.995
1.1765	100	0.0081	-	-
1.4706	125	0.0027	-	-
1.7647	150	0.0028	-	-
2.0	170	-	0.006	0.9967
2.0588	175	0.0039	-	-
2.3529	200	0.0030	-	-
2.6471	225	0.0016	-	-
2.9412	250	0.0023	-	-
3.0	255	-	0.0054	0.9967
3.2353	275	0.0019	-	-
3.5294	300	0.0029	-	-
3.8235	325	0.0039	-	-
4.0	340	-	0.0049	0.9967
4.1176	350	0.0015	-	-
4.4118	375	0.0016	-	-
4.7059	400	0.0020	-	-
5.0	425	0.0024	0.0050	0.9967

• The bold row denotes the saved checkpoint.

Training Time

• Training: 2.7 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}