Hugging Face's logo

s-rahma
/
ft-bgem3-finetuned

Sentence Similarity
sentence-transformers
Safetensors
xlm-roberta
feature-extraction
Generated from Trainer
dataset_size:1970
loss:MultipleNegativesRankingLoss
text-embeddings-inference
Model card Files
xet
 Community

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BAAI/bge-m3
  • Maximum Sequence Length: 8192 tokens
  • Output Dimensionality: 1024 dimensions
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Ghee نباتي بطعم وريحة السَمنة الفلاحي من كريستال - 2.5 كيلو',
    'سمن نباتي بطعم وريحة السَمنة الفلاحي من كريستال - 2.5 كيلو',
    'سمن نباتي بطعم وريحة الزبدة الصفراء من كريستال - 2.5 كيلو',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

  • Size: 1,970 training samples
  • Columns: sentence_0 and sentence_1
  • Approximate statistics based on the first 1000 samples:
    sentence_0 sentence_1
    type string string
    details
    • min: 3 tokens
    • mean: 9.23 tokens
    • max: 33 tokens
    • min: 4 tokens
    • mean: 14.56 tokens
    • max: 33 tokens
  • Samples:
    sentence_0 sentence_1
    shower gel matey kids شاور ماتي مولي لطيف للبشرة الحساسة للأطفال الصغيرة
    خبز سن فاخر عيش السن*5
    شتورا سمنة جاموسى خليط -700جرام شتورا سمن جاموسى خليط -700جرام
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 4
  • per_device_eval_batch_size: 4
  • num_train_epochs: 150
  • multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: no
  • prediction_loss_only: True
  • per_device_train_batch_size: 4
  • per_device_eval_batch_size: 4
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 150
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: round_robin

Training Logs

Click to expand
Epoch Step Training Loss
1.0142 500 0.1635
2.0284 1000 0.0815
3.0426 1500 0.0465
4.0568 2000 0.0255
5.0710 2500 0.0213
6.0852 3000 0.0234
7.0994 3500 0.0254
8.1136 4000 0.0328
9.1278 4500 0.022
10.1420 5000 0.0145
11.1562 5500 0.0192
12.1704 6000 0.0263
13.1846 6500 0.0292
14.1988 7000 0.0255
15.2130 7500 0.0165
16.2272 8000 0.0273
17.2414 8500 0.0196
18.2556 9000 0.0277
19.2698 9500 0.024
20.2840 10000 0.0268
21.2982 10500 0.0394
22.3124 11000 0.0192
23.3266 11500 0.0223
24.3408 12000 0.0186
25.3550 12500 0.0257
26.3692 13000 0.0334
27.3834 13500 0.0212
28.3976 14000 0.0159
29.4118 14500 0.0304
30.4260 15000 0.0256
31.4402 15500 0.0298
32.4544 16000 0.0283
33.4686 16500 0.028
34.4828 17000 0.0219
35.4970 17500 0.0189
36.5112 18000 0.0247
37.5254 18500 0.0287
38.5396 19000 0.0238
39.5538 19500 0.0283
40.5680 20000 0.0206
41.5822 20500 0.0301
42.5963 21000 0.0241
43.6105 21500 0.0198
44.6247 22000 0.0246
45.6389 22500 0.0173
46.6531 23000 0.0318
47.6673 23500 0.0221
48.6815 24000 0.0278
49.6957 24500 0.0275
50.7099 25000 0.0269
51.7241 25500 0.0205
52.7383 26000 0.0176
53.7525 26500 0.0178
54.7667 27000 0.0204
55.7809 27500 0.0178
56.7951 28000 0.013
57.8093 28500 0.0194
58.8235 29000 0.0223
59.8377 29500 0.0236
60.8519 30000 0.0204
61.8661 30500 0.0225
62.8803 31000 0.0173
63.8945 31500 0.0241
64.9087 32000 0.0268
65.9229 32500 0.0205
66.9371 33000 0.0127
67.9513 33500 0.0214
68.9655 34000 0.0176
69.9797 34500 0.0225
70.9939 35000 0.0209
72.0081 35500 0.0188
73.0223 36000 0.0203
74.0365 36500 0.0176
75.0507 37000 0.0173
76.0649 37500 0.02
77.0791 38000 0.0216
78.0933 38500 0.0227
79.1075 39000 0.0188
80.1217 39500 0.0188
81.1359 40000 0.0217
82.1501 40500 0.0265
83.1643 41000 0.0189
84.1785 41500 0.0206
85.1927 42000 0.0194
86.2069 42500 0.0246
87.2211 43000 0.0225
88.2353 43500 0.016
89.2495 44000 0.0168
90.2637 44500 0.0264
91.2779 45000 0.017
92.2921 45500 0.0144
93.3063 46000 0.0233
94.3205 46500 0.022
95.3347 47000 0.0246
96.3489 47500 0.0182
97.3631 48000 0.0184
98.3773 48500 0.0155
99.3915 49000 0.0177
100.4057 49500 0.014
101.4199 50000 0.0263
102.4341 50500 0.0184
103.4483 51000 0.023
104.4625 51500 0.0212
105.4767 52000 0.0156
106.4909 52500 0.0192
107.5051 53000 0.0178
108.5193 53500 0.0196
109.5335 54000 0.0176
110.5477 54500 0.0128
111.5619 55000 0.0171
112.5761 55500 0.0157
113.5903 56000 0.0135
114.6045 56500 0.025
115.6187 57000 0.0131
116.6329 57500 0.0146
117.6471 58000 0.0146
118.6613 58500 0.0184
119.6755 59000 0.014
120.6897 59500 0.0242
121.7039 60000 0.0137
122.7181 60500 0.0148
123.7323 61000 0.0174
124.7465 61500 0.0126
125.7606 62000 0.022
126.7748 62500 0.0187
127.7890 63000 0.0235
128.8032 63500 0.0127
129.8174 64000 0.0154
130.8316 64500 0.0169
131.8458 65000 0.0148
132.8600 65500 0.01
133.8742 66000 0.0174
134.8884 66500 0.0155
135.9026 67000 0.0147
136.9168 67500 0.0191
137.9310 68000 0.0199
138.9452 68500 0.0207
139.9594 69000 0.0193
140.9736 69500 0.0137
141.9878 70000 0.0166
143.0020 70500 0.0146
144.0162 71000 0.0212
145.0304 71500 0.0145
146.0446 72000 0.0184
147.0588 72500 0.0211
148.0730 73000 0.0203
149.0872 73500 0.0229

Framework Versions

  • Python: 3.11.13
  • Sentence Transformers: 4.1.0
  • Transformers: 4.52.4
  • PyTorch: 2.6.0+cu124
  • Accelerate: 1.8.1
  • Datasets: 3.6.0
  • Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
7
Safetensors
Model size
0.6B params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for s-rahma/ft-bgem3-finetuned

Base model

BAAI/bge-m3
Finetuned
(591)
this model

Papers for s-rahma/ft-bgem3-finetuned