rafiazarin/arxiv-cs-embedding-finetuned

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("rafiazarin/arxiv-cs-embedding-finetuned")
# Run inference
sentences = [
    "What were the upper limits on branching ratios for b' -> bZ and b' -> cW obtained from the DELPHI experiment at LEP-II?",
    "A search for the pair production of fourth generation b'-quarks was performed using data taken by the DELPHI detector at LEP-II. The analysed data were collected at centre-of-mass energies ranging from 196 to 209 GeV, corresponding to an integrated luminosity of 420 pb^{-1}. No evidence for a signal was found. Upper limits on BR(b' -> bZ) and BR(b' -> cW) were obtained for b' masses ranging from 96 to 103 GeV/c^2. These limits, together with the theoretical branching ratios predicted by a sequential four generations model, were used to constrain the value of R_{CKM}=|V_{cb'}/V_{tb'}V_{tb}|, where V_{cb'}, V_{tb'} and V_{tb} are elements of the extended CKM matrix.",
    'We study the statefinder parameter in the five-dimensional big bounce model, and apply it to differentiate the attractor solutions of quintessence and phantom field. It is found that the evolving trajectories of these two attractor solutions in the statefinder parameters plane are quite different, and that are different from the statefinder trajectories of other dark energy models.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.5103, -0.0397],
#         [ 0.5103,  1.0000,  0.0270],
#         [-0.0397,  0.0270,  1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: val
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.995
cosine_accuracy@3	1.0
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.995
cosine_precision@3	0.3333
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.995
cosine_recall@3	1.0
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9975
cosine_mrr@10	0.9967
cosine_map@100	0.9967

Training Details

Training Dataset

Unnamed Dataset

• Size: 2,700 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 9 tokens mean: 27.46 tokens max: 61 tokens	min: 32 tokens mean: 157.29 tokens max: 256 tokens

• Samples:

  anchor	positive
  How does the transition from a scaling solution to a single-field dominated ekpyrotic collapse affect isocurvature perturbations?	A scale-invariant spectrum of isocurvature perturbations is generated during collapse in the ekpyrotic scaling solution in models where multiple fields have steep negative exponential potentials. The scale invariance of the spectrum is realized by a tachyonic instability in the isocurvature field. This instability drives the scaling solution to the late time attractor that is the old ekpyrotic collapse dominated by a single field. We show that the transition from the scaling solution to the single field dominated ekpyrotic collapse automatically converts the initial isocurvature perturbations about the scaling solution to comoving curvature perturbations about the late-time attractor. The final amplitude of the comoving curvature perturbation is determined by the Hubble scale at the transition.
  What are the explicit upper and lower bounds for the first moment of a general 2-CUSUM stopping rule, as derived from closed-form expressions?	This work examines the problem of sequential detection of a change in the drift of a Brownian motion in the case of two-sided alternatives. Applications to real life situations in which two-sided changes can occur are discussed. Traditionally, 2-CUSUM stopping rules have been used for this problem due to their asymptotically optimal character as the mean time between false alarms tends to . In particular, attention has focused on 2-CUSUM harmonic mean rules due to the simplicity in calculating their first moments. In this paper, we derive closed-form expressions for the first moment of a general 2-CUSUM stopping rule. We use these expressions to obtain explicit upper and lower bounds for it. Moreover, we derive an expression for the rate of change of this first moment as one of the threshold parameters changes. Based on these expressions we obtain explicit upper and lower bounds to this rate of change. Using these expressions we are able to find the best 2-CUSUM stopping rule with resp...
  What is the role of frustration and anisotropy in the one-dimensionalization of the -(BEDT-TTF)Cu(CN) material's excitation spectra?	Motivated by the observation of a gapless spin liquid state in -(BEDT-TTF)Cu(CN), we analyze the anisotropic triangular lattice Heisenberg model with the resonating valence bond mean-field approximation. Paying attention to the small quasi-one-dimensional anisotropy of the material, we take an approach from one-dimensional (1D) chains coupled with frustrating zig-zag bonds. By calculating one-particle excitation spectra changing anisotropy parameter from the decoupled 1D chains to the isotropic triangular lattice, we find almost gapless excitations in the wide range from the 1D limit. This one-dimensionalization by frustration is considered to be a candidate for the mechanism of the gapless spin liquid state.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 200 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 200 samples:

  	anchor	positive
  type	string	string
  details	min: 9 tokens mean: 26.74 tokens max: 51 tokens	min: 37 tokens mean: 157.56 tokens max: 256 tokens

• Samples:

  anchor	positive
  What spin-flip and skyrmion-like textures are possible if the sign of the 2DEG g-factor is changed?	A theory of collective states in a magnetically quantized two-dimensional electron gas (2DEG) with half-filled Landau level (quantized Hall ferromagnet) in the presence of magnetic 3d impurities is developed. The spectrum of bound and delocalized spin-excitons as well as the renormalization of Zeeman splitting of the impurity 3d levels due to the indirect exchange interaction with the 2DEG are studied for the specific case of n-type GaAs doped with Mn where the Lande` g-factors of impurity and 2DEG have opposite signs. If the sign of the 2DEG g-factor is changed due to external influences, then impurity related transitions to new ground state phases, presenting various spin-flip and skyrmion-like textures, are possible. Conditions for existence of these phases are discussed. PACS: 73.43.Lp, 73.21.Fg, 72.15.Rn
  How do right modules over operads relate to functors on categories of algebras?	In the theory of operads we consider functors of generalized symmetric powers defined by sums of coinvariant modules under actions of symmetric groups. One observes classically that the construction of symmetric functors provides an isomorphism from the category of symmetric modules to a subcategory of the category of functors on the base category. The purpose of this book is to obtain a similar relationship for functors on a category of algebras over an operad. We observe that right modules over operads, symmetric modules equipped with a right operad action, give rise to functors on categories of algebras and we prove that this construction yields an embedding of categories. Then we check that right modules over operads form a model category. In addition we prove that weak-equivalences of right modules correspond to pointwise weak-equivalences at the functor level. As a conclusion, we obtain that right modules over operads supply good models for the homotopy of associated functors on ...
  What specific wave function modification, such as the Jastrow or Villars correlator, is used in the fixed center-of-mass approximation for the ^{4}He nucleus?	The approach exposed in the recent paper (A. Shebeko, P. Papakonstantinou, E. Mavrommatis, Eur. Phys. J. A 27, 143 (2006)) has been applied in studying center-of-mass motion effects on the nucleon density and momentum distributions in nuclei. We are focused upon effects due to the center-of-mass and short-range nucleon correlations embedded in translationally invariant ground-state wavefunctions. The latter are constructed in the so-called fixed center-of-mass approximation, starting with a Slater determinant wave function modified by some correlator (e.g., after Jastrow or Villars). It is shown how one can simplify evaluation of the corresponding expectation values that determine the distributions. The analytic expressions derived here involve the own "Tassie-Barker" factors for each distribution. As an illustration, numerical calculations have been carried out for the nucleus ^{4}He with the Slater determinant to describe the nucleon (1s)^4 configuration composed of single-particle o...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_steps: 0.1
• fp16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	val_cosine_ndcg@10
0.2941	25	0.0486	-	-
0.5882	50	0.0297	-	-
0.8824	75	0.0159	-	-
1.0	85	-	0.0048	0.9968
1.1765	100	0.0125	-	-
1.4706	125	0.0062	-	-
1.7647	150	0.0061	-	-
2.0	170	-	0.0032	0.9969
2.0588	175	0.0044	-	-
2.3529	200	0.0046	-	-
2.6471	225	0.0052	-	-
2.9412	250	0.0042	-	-
3.0	255	-	0.0029	0.9972
3.2353	275	0.0035	-	-
3.5294	300	0.0032	-	-
3.8235	325	0.0055	-	-
4.0	340	-	0.0026	0.9975
4.1176	350	0.0046	-	-
4.4118	375	0.0029	-	-
4.7059	400	0.0035	-	-
5.0	425	0.0023	0.0026	0.9975

• The bold row denotes the saved checkpoint.

Training Time

• Training: 1.3 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}