Hugging Face's logo

sjmeis
/
skills-embedding-large-v1

Sentence Similarity
sentence-transformers
Safetensors
modernbert
feature-extraction
Generated from Trainer
dataset_size:3057742
loss:MultipleNegativesRankingLoss
text-embeddings-inference
Model card Files
xet
 Community

SentenceTransformer

This is a sentence-transformers model trained. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Maximum Sequence Length: 128 tokens
  • Output Dimensionality: 1024 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Subject: Your Culinary Arts Transcript is Ready.',
    'Thank you for contacting Community College of Rhode Island Student Services. We understand how important it is for you to provide verified proof of your culinary training for the Bucks County youth mentorship program, and we’re pleased to let you know that your transcript is now ready.',
    'Basic Requirement for Economist: A.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.6778,  0.0104],
#         [ 0.6778,  1.0000, -0.0491],
#         [ 0.0104, -0.0491,  1.0000]])

Training Details

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 64
  • learning_rate: 2e-05
  • warmup_steps: 0.1
  • weight_decay: 0.01
  • bf16: True
  • per_device_eval_batch_size: 128
  • load_best_model_at_end: True
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 64
  • num_train_epochs: 3
  • max_steps: -1
  • learning_rate: 2e-05
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.1
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.01
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 128
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 42
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • dataloader_multiprocessing_context: None
  • dataloader_in_order: True
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • warmup_ratio: None

Training Logs

Click to expand
Epoch Step Training Loss Validation Loss
0.0105 500 3.4128 -
0.0209 1000 2.2413 -
0.0314 1500 1.8003 -
0.0419 2000 1.5853 -
0.0523 2500 1.4051 -
0.0628 3000 1.2792 -
0.0733 3500 1.2056 -
0.0837 4000 1.1367 -
0.0942 4500 1.0590 -
0.1047 5000 1.0062 -
0.1151 5500 0.9471 -
0.1256 6000 0.8998 -
0.1360 6500 0.8607 -
0.1465 7000 0.8332 -
0.1570 7500 0.8328 -
0.1674 8000 0.8269 -
0.1779 8500 0.7928 -
0.1884 9000 0.7758 -
0.1988 9500 0.7590 -
0.2093 10000 0.7779 1.0340
0.2198 10500 0.7434 -
0.2302 11000 0.7424 -
0.2407 11500 0.7209 -
0.2512 12000 0.7027 -
0.2616 12500 0.7117 -
0.2721 13000 0.7009 -
0.2826 13500 0.7195 -
0.2930 14000 0.6895 -
0.3035 14500 0.7156 -
0.3140 15000 0.8232 -
0.3244 15500 0.7589 -
0.3349 16000 0.6728 -
0.3453 16500 0.6482 -
0.3558 17000 0.6323 -
0.3663 17500 0.6503 -
0.3767 18000 0.6439 -
0.3872 18500 0.6530 -
0.3977 19000 0.6169 -
0.4081 19500 0.6129 -
0.4186 20000 0.6114 0.8654
0.4291 20500 0.6166 -
0.4395 21000 0.6297 -
0.4500 21500 0.6137 -
0.4605 22000 0.5950 -
0.4709 22500 0.5960 -
0.4814 23000 0.5946 -
0.4919 23500 0.5814 -
0.5023 24000 0.6160 -
0.5128 24500 0.5888 -
0.5233 25000 0.5769 -
0.5337 25500 0.5676 -
0.5442 26000 0.5645 -
0.5546 26500 0.5671 -
0.5651 27000 0.5547 -
0.5756 27500 0.5521 -
0.5860 28000 0.5508 -
0.5965 28500 0.5555 -
0.6070 29000 0.5578 -
0.6174 29500 0.5445 -
0.6279 30000 0.5425 0.7659
0.6384 30500 0.5445 -
0.6488 31000 0.5461 -
0.6593 31500 0.5527 -
0.6698 32000 0.5495 -
0.6802 32500 0.5451 -
0.6907 33000 0.5404 -
0.7012 33500 0.5294 -
0.7116 34000 0.5264 -
0.7221 34500 0.5288 -
0.7326 35000 0.5462 -
0.7430 35500 0.5681 -
0.7535 36000 0.5366 -
0.7639 36500 0.5260 -
0.7744 37000 0.5214 -
0.7849 37500 0.5196 -
0.7953 38000 0.5104 -
0.8058 38500 0.5286 -
0.8163 39000 0.5159 -
0.8267 39500 0.5100 -
0.8372 40000 0.5140 0.7517
0.8477 40500 0.5227 -
0.8581 41000 0.5165 -
0.8686 41500 0.5399 -
0.8791 42000 0.5228 -
0.8895 42500 0.5160 -
0.9000 43000 0.5213 -
0.9105 43500 0.5115 -
0.9209 44000 0.5167 -
0.9314 44500 0.5147 -
0.9419 45000 0.5158 -
0.9523 45500 0.5137 -
0.9628 46000 0.5010 -
0.9733 46500 0.4988 -
0.9837 47000 0.5055 -
0.9942 47500 0.4823 -
1.0046 48000 0.4849 -
1.0151 48500 0.4769 -
1.0256 49000 0.4646 -
1.0360 49500 0.4655 -
1.0465 50000 0.4680 0.6869
1.0570 50500 0.4722 -
1.0674 51000 0.4612 -
1.0779 51500 0.4618 -
1.0884 52000 0.4661 -
1.0988 52500 0.4617 -
1.1093 53000 0.4638 -
1.1198 53500 0.4626 -
1.1302 54000 0.4598 -
1.1407 54500 0.4572 -
1.1512 55000 0.4596 -
1.1616 55500 0.4579 -
1.1721 56000 0.4718 -
1.1826 56500 0.4605 -
1.1930 57000 0.4615 -
1.2035 57500 0.4482 -
1.2139 58000 0.4540 -
1.2244 58500 0.4563 -
1.2349 59000 0.4629 -
1.2453 59500 0.4577 -
1.2558 60000 0.4579 0.6699
1.2663 60500 0.4537 -
1.2767 61000 0.4500 -
1.2872 61500 0.4500 -
1.2977 62000 0.4500 -
1.3081 62500 0.4568 -
1.3186 63000 0.4511 -
1.3291 63500 0.4427 -
1.3395 64000 0.4392 -
1.3500 64500 0.4445 -
1.3605 65000 0.4503 -
1.3709 65500 0.4426 -
1.3814 66000 0.4394 -
1.3919 66500 0.4365 -
1.4023 67000 0.4367 -
1.4128 67500 0.4344 -
1.4232 68000 0.4361 -
1.4337 68500 0.4394 -
1.4442 69000 0.4516 -
1.4546 69500 0.4452 -
1.4651 70000 0.4364 0.6353
1.4756 70500 0.4327 -
1.4860 71000 0.4280 -
1.4965 71500 0.4287 -
1.5070 72000 0.4280 -
1.5174 72500 0.4310 -
1.5279 73000 0.4195 -
1.5384 73500 0.4241 -
1.5488 74000 0.4279 -
1.5593 74500 0.4233 -
1.5698 75000 0.4271 -
1.5802 75500 0.4241 -
1.5907 76000 0.4261 -
1.6012 76500 0.4301 -
1.6116 77000 0.4297 -
1.6221 77500 0.4413 -
1.6326 78000 0.4325 -
1.6430 78500 0.4283 -
1.6535 79000 0.4261 -
1.6639 79500 0.4220 -
1.6744 80000 0.4523 0.6471
1.6849 80500 0.4378 -
1.6953 81000 0.4399 -
1.7058 81500 0.4291 -
1.7163 82000 0.4222 -
1.7267 82500 0.4237 -
1.7372 83000 0.4176 -
1.7477 83500 0.4223 -
1.7581 84000 0.4202 -
1.7686 84500 0.4179 -
1.7791 85000 0.4207 -
1.7895 85500 0.4034 -
1.8000 86000 0.4148 -
1.8105 86500 0.4206 -
1.8209 87000 0.4193 -
1.8314 87500 0.4065 -
1.8419 88000 0.4132 -
1.8523 88500 0.4074 -
1.8628 89000 0.4188 -
1.8732 89500 0.4146 -
1.8837 90000 0.4078 0.6025
1.8942 90500 0.4114 -
1.9046 91000 0.4139 -
1.9151 91500 0.4057 -
1.9256 92000 0.4212 -
1.9360 92500 0.4171 -
1.9465 93000 0.4166 -
1.9570 93500 0.4060 -
1.9674 94000 0.4095 -
1.9779 94500 0.4078 -
1.9884 95000 0.4031 -
1.9988 95500 0.4016 -
2.0093 96000 0.3951 -
2.0198 96500 0.3934 -
2.0302 97000 0.3903 -
2.0407 97500 0.3835 -
2.0512 98000 0.3874 -
2.0616 98500 0.3782 -
2.0721 99000 0.3790 -
2.0825 99500 0.3774 -
2.0930 100000 0.3803 0.5844
2.1035 100500 0.3711 -
2.1139 101000 0.3746 -
2.1244 101500 0.3774 -
2.1349 102000 0.3805 -
2.1453 102500 0.3773 -
2.1558 103000 0.3721 -
2.1663 103500 0.3792 -
2.1767 104000 0.3755 -
2.1872 104500 0.3829 -
2.1977 105000 0.3819 -
2.2081 105500 0.3733 -
2.2186 106000 0.3723 -
2.2291 106500 0.3748 -
2.2395 107000 0.3725 -
2.2500 107500 0.3778 -
2.2605 108000 0.3735 -
2.2709 108500 0.3637 -
2.2814 109000 0.3696 -
2.2918 109500 0.3632 -
2.3023 110000 0.3723 0.5753
2.3128 110500 0.3768 -
2.3232 111000 0.3707 -
2.3337 111500 0.3714 -
2.3442 112000 0.3709 -
2.3546 112500 0.3732 -
2.3651 113000 0.3592 -
2.3756 113500 0.3666 -
2.3860 114000 0.3656 -
2.3965 114500 0.3718 -
2.4070 115000 0.3705 -
2.4174 115500 0.3621 -
2.4279 116000 0.3616 -
2.4384 116500 0.3596 -
2.4488 117000 0.3681 -
2.4593 117500 0.3554 -
2.4698 118000 0.3490 -
2.4802 118500 0.3639 -
2.4907 119000 0.3582 -
2.5012 119500 0.3583 -
2.5116 120000 0.3476 0.5583
2.5221 120500 0.3627 -
2.5325 121000 0.3581 -
2.5430 121500 0.3555 -
2.5535 122000 0.3629 -
2.5639 122500 0.3507 -
2.5744 123000 0.3600 -
2.5849 123500 0.3595 -
2.5953 124000 0.3662 -
2.6058 124500 0.3474 -
2.6163 125000 0.3584 -
2.6267 125500 0.3550 -
2.6372 126000 0.3577 -
2.6477 126500 0.3556 -
2.6581 127000 0.3453 -
2.6686 127500 0.3487 -
2.6791 128000 0.3472 -
2.6895 128500 0.3488 -
2.7000 129000 0.3466 -
2.7105 129500 0.3537 -
2.7209 130000 0.3470 0.5477
2.7314 130500 0.3433 -
2.7418 131000 0.3481 -
2.7523 131500 0.3462 -
2.7628 132000 0.3513 -
2.7732 132500 0.3571 -
2.7837 133000 0.3523 -
2.7942 133500 0.3551 -
2.8046 134000 0.3464 -
2.8151 134500 0.3420 -
2.8256 135000 0.3491 -
2.8360 135500 0.3482 -
2.8465 136000 0.3498 -
2.8570 136500 0.3462 -
2.8674 137000 0.3476 -
2.8779 137500 0.3436 -
2.8884 138000 0.3414 -
2.8988 138500 0.3453 -
2.9093 139000 0.3542 -
2.9198 139500 0.3368 -
2.9302 140000 0.3379 0.5407
2.9407 140500 0.3457 -
2.9511 141000 0.3498 -
2.9616 141500 0.3450 -
2.9721 142000 0.3418 -
2.9825 142500 0.3425 -
2.9930 143000 0.3462 -
3.0 143334 - 0.5386
  • The bold row denotes the saved checkpoint.

Training Time

  • Training: 1.6 days

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 5.6.0
  • Transformers: 5.16.1
  • PyTorch: 2.13.0+cu130
  • Accelerate: 1.14.0
  • Datasets: 5.0.0
  • Tokenizers: 0.23.2

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss 

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}
Downloads last month
26
Safetensors
Model size
0.4B params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Papers for sjmeis/skills-embedding-large-v1