mjaliz/bslm-pair-combined-safe

SentenceTransformer based on mjaliz/bslm-pair-206k

This is a sentence-transformers model finetuned from mjaliz/bslm-pair-206k on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: mjaliz/bslm-pair-206k
• Maximum Sequence Length: 96 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'شومیز و شلوار عروسکی زنانه',
]
documents = [
    'ست شومیز شلوار زنانه ایوا مدل عروسکی قهوه\u200cای',
    'سرویس غذاخوری 27 پارچه چینی زرین قالب کواترو طرح ادلین طوسی',
    'روسری مجلسی ژان ژان ترک شاین با حاشیه نگین\u200cکاری قواره ۱۲۰ سانتی\u200cمتر',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.8277, -0.0022, -0.0189]])

Evaluation

Metrics

Information Retrieval

• Dataset: dev
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7932
cosine_accuracy@5	0.9386
cosine_accuracy@10	0.9634
cosine_accuracy@50	0.9883
cosine_precision@1	0.7932
cosine_precision@5	0.2715
cosine_precision@10	0.1584
cosine_precision@50	0.0411
cosine_recall@1	0.6622
cosine_recall@5	0.8925
cosine_recall@10	0.9373
cosine_recall@50	0.9821
cosine_ndcg@10	0.8712
cosine_mrr@10	0.8558
cosine_map@10	0.8377

Training Details

Training Dataset

json

• Dataset: json
• Size: 2,991,559 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 5 tokens mean: 7.99 tokens max: 15 tokens	min: 12 tokens mean: 22.05 tokens max: 41 tokens

• Samples:

  anchor	positive
  مانتو تابستانه سایزبزرگ	مانتو زنانه بلند کتان لمه مهتابان (Mahtaban) مدل قیطون‌دوزی شده سایزبزرگ
  شال توری بهاره	شال رینگی زنانه بافت توری مشکی ریزش ملایم بهاره تابستانه
  کاور صندلی خودرو	روکش صندلی خودرو ماتین کاور طرح لاماری جودون برای پراید 111 هاچبک رنگ عسلی

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 32,
      "gather_across_devices": true,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

json

• Dataset: json
• Size: 25,835 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 4 tokens mean: 7.61 tokens max: 12 tokens	min: 10 tokens mean: 22.74 tokens max: 32 tokens

• Samples:

  anchor	positive
  رژگونه توت فرنگی	بالم لب توت فرنگی Anakan (آناکان) آبرسان ویتامینه گیاهی
  چای ساز کوخ	چای ساز برقی Kouch (کوخ) مدل KT-2150 مشکی، 2200 وات، کتری استیل دوجداره
  میز کنسول	میز کنسول FARASHAHGROUP با درب طرح چوب و قفسه بندی باز

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 32,
      "gather_across_devices": true,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 256
• num_train_epochs: 5.0
• learning_rate: 2e-05
• warmup_steps: 0.1
• weight_decay: 0.01
• bf16: True
• gradient_checkpointing: True
• per_device_eval_batch_size: 64
• dataloader_num_workers: 4
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 256
• num_train_epochs: 5.0
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 64
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 4
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	dev_cosine_ndcg@10
0.0257	100	1.5386	-	-
0.0513	200	1.2748	-	-
0.0770	300	1.1664	-	-
0.1027	400	1.1179	-	-
0.1284	500	1.0774	0.4263	0.8280
0.1540	600	1.0482	-	-
0.1797	700	1.0287	-	-
0.2054	800	0.9916	-	-
0.2311	900	0.9794	-	-
0.2567	1000	0.9707	0.3903	0.8408
0.2824	1100	0.9690	-	-
0.3081	1200	0.9540	-	-
0.3338	1300	0.9484	-	-
0.3594	1400	0.9298	-	-
0.3851	1500	0.9201	0.3788	0.8507
0.4108	1600	0.9160	-	-
0.4365	1700	0.9118	-	-
0.4621	1800	0.9137	-	-
0.4878	1900	0.8985	-	-
0.5135	2000	0.8995	0.3692	0.8506
0.5392	2100	0.9045	-	-
0.5648	2200	0.8824	-	-
0.5905	2300	0.8808	-	-
0.6162	2400	0.8804	-	-
0.6418	2500	0.8681	0.3605	0.8543
0.6675	2600	0.8748	-	-
0.6932	2700	0.8669	-	-
0.7189	2800	0.8633	-	-
0.7445	2900	0.8595	-	-
0.7702	3000	0.8555	0.3593	0.8589
0.7959	3100	0.8541	-	-
0.8216	3200	0.8445	-	-
0.8472	3300	0.8448	-	-
0.8729	3400	0.8438	-	-
0.8986	3500	0.8375	0.3616	0.8606
0.9243	3600	0.8392	-	-
0.9499	3700	0.8397	-	-
0.9756	3800	0.8311	-	-
1.0072	3900	0.8389	-	-
1.0329	4000	0.8259	0.3666	0.8634
1.0585	4100	0.8214	-	-
1.0842	4200	0.8278	-	-
1.1099	4300	0.8237	-	-
1.1356	4400	0.8246	-	-
1.1612	4500	0.8110	0.3707	0.8647
1.1869	4600	0.8107	-	-
1.2126	4700	0.7897	-	-
1.2383	4800	0.8023	-	-
1.2639	4900	0.7967	-	-
1.2896	5000	0.8017	0.3566	0.8657
1.3153	5100	0.7930	-	-
1.3409	5200	0.7983	-	-
1.3666	5300	0.7898	-	-
1.3923	5400	0.7811	-	-
1.4180	5500	0.7804	0.3653	0.8656
1.4436	5600	0.7856	-	-
1.4693	5700	0.7796	-	-
1.4950	5800	0.7728	-	-
1.5207	5900	0.7783	-	-
1.5463	6000	0.7827	0.3749	0.8655
1.5720	6100	0.7598	-	-
1.5977	6200	0.7670	-	-
1.6234	6300	0.7707	-	-
1.6490	6400	0.7646	-	-
1.6747	6500	0.7644	0.3736	0.8686
1.7004	6600	0.7633	-	-
1.7261	6700	0.7571	-	-
1.7517	6800	0.7665	-	-
1.7774	6900	0.7646	-	-
1.8031	7000	0.7566	0.3612	0.8673
1.8288	7100	0.7545	-	-
1.8544	7200	0.7563	-	-
1.8801	7300	0.7535	-	-
1.9058	7400	0.7576	-	-
1.9315	7500	0.7536	0.3724	0.8690
1.9571	7600	0.7502	-	-
1.9828	7700	0.7497	-	-
2.0144	7800	0.7588	-	-
2.0401	7900	0.7414	-	-
2.0657	8000	0.7442	0.3745	0.8688
2.0914	8100	0.7562	-	-
2.1171	8200	0.7512	-	-
2.1427	8300	0.7454	-	-
2.1684	8400	0.7419	-	-
2.1941	8500	0.7301	0.3740	0.8694
2.2198	8600	0.7290	-	-
2.2454	8700	0.7324	-	-
2.2711	8800	0.7319	-	-
2.2968	8900	0.7365	-	-
2.3225	9000	0.7291	0.3737	0.8693
2.3481	9100	0.7339	-	-
2.3738	9200	0.7251	-	-
2.3995	9300	0.7214	-	-
2.4252	9400	0.7207	-	-
2.4508	9500	0.7298	0.3670	0.8702
2.4765	9600	0.7257	-	-
2.5022	9700	0.7124	-	-
2.5279	9800	0.7313	-	-
2.5535	9900	0.7211	-	-
2.5792	10000	0.7107	0.3712	0.8710
2.6049	10100	0.7149	-	-
2.6306	10200	0.7209	-	-
2.6562	10300	0.7112	-	-
2.6819	10400	0.7192	-	-
2.7076	10500	0.7061	0.3628	0.8712

Training Time

• Training: 1.7 days
• Evaluation: 17.8 minutes
• Total: 1.7 days

Framework Versions

• Python: 3.11.14
• Sentence Transformers: 5.6.0
• Transformers: 5.12.1
• PyTorch: 2.12.1+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}