bslm-pair-206k

This is a sentence-transformers bi-encoder finetuned from mjaliz/bslm-mlm-25M-ptdrw on 202,149 ecommerce query/product-title pairs with MultipleNegativesRankingLoss. It maps Persian ecommerce queries and product titles to a 1024-dimensional dense vector space for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: mjaliz/bslm-mlm-25M-ptdrw
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'قلک',
]
documents = [
    'ماشین حمل پول اسباب بازی قلک موزیکال رمزدار لگویی سفید',
    'سه چرخه کودک تاشو PARYANTOYS (پاریان تویز) مدل سوپر تیتان',
    'جارو برقی پرتابل Homatis هوماتیس مدل V20 کیسه جارو برقی عصایی تنظیم مکش',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.4066,  0.0768, -0.0410]])

Evaluation

Metrics

Information Retrieval

• Dataset: dev
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8362
cosine_accuracy@5	0.9821
cosine_accuracy@10	0.9956
cosine_accuracy@50	0.999
cosine_precision@1	0.8362
cosine_precision@5	0.1964
cosine_precision@10	0.0996
cosine_precision@50	0.02
cosine_recall@1	0.8362
cosine_recall@5	0.9821
cosine_recall@10	0.9956
cosine_recall@50	0.999
cosine_ndcg@10	0.9239
cosine_mrr@10	0.9
cosine_map@10	0.9

Held-out test set

• Dataset: test
• Queries: 2,063
• Corpus: 2,046 product titles
• Qrels: 2,063 query-product relevance labels

Metric	Value
cosine_accuracy@1	0.8333
cosine_accuracy@5	0.9777
cosine_accuracy@10	0.9927
cosine_accuracy@50	0.9995
cosine_precision@1	0.8333
cosine_precision@5	0.1955
cosine_precision@10	0.0993
cosine_precision@50	0.0200
cosine_recall@1	0.8333
cosine_recall@5	0.9777
cosine_recall@10	0.9927
cosine_recall@50	0.9995
cosine_ndcg@10	0.9206
cosine_ndcg@50	0.9222
cosine_mrr@10	0.8966
cosine_mrr@50	0.8970
cosine_map@10	0.8966
cosine_map@50	0.8970

Training Details

Training Dataset

json

• Dataset: json
• Size: 202,149 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 5 tokens mean: 7.99 tokens max: 15 tokens	min: 12 tokens mean: 22.05 tokens max: 41 tokens

• Samples:

  anchor	positive
  مانتو تابستانه سایزبزرگ	مانتو زنانه بلند کتان لمه مهتابان (Mahtaban) مدل قیطون‌دوزی شده سایزبزرگ
  شال توری بهاره	شال رینگی زنانه بافت توری مشکی ریزش ملایم بهاره تابستانه
  کاور صندلی خودرو	روکش صندلی خودرو ماتین کاور طرح لاماری جودون برای پراید 111 هاچبک رنگ عسلی

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": true,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

json

• Dataset: json
• Size: 2,063 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 4 tokens mean: 7.61 tokens max: 12 tokens	min: 10 tokens mean: 22.74 tokens max: 32 tokens

• Samples:

  anchor	positive
  رژگونه توت فرنگی	بالم لب توت فرنگی Anakan (آناکان) آبرسان ویتامینه گیاهی
  چای ساز کوخ	چای ساز برقی Kouch (کوخ) مدل KT-2150 مشکی، 2200 وات، کتری استیل دوجداره
  میز کنسول	میز کنسول FARASHAHGROUP با درب طرح چوب و قفسه بندی باز

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": true,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• num_train_epochs: 1.0
• learning_rate: 2e-05
• warmup_steps: 0.1
• weight_decay: 0.01
• bf16: True
• per_device_eval_batch_size: 128
• dataloader_num_workers: 4
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 64
• num_train_epochs: 1.0
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 128
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 4
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	dev_cosine_ndcg@10
0.0238	25	3.7237	-	-
0.0475	50	0.8812	-	-
0.0713	75	0.4181	-	-
0.0951	100	0.3205	0.3463	0.8755
0.1188	125	0.2469	-	-
0.1426	150	0.2423	-	-
0.1663	175	0.2189	-	-
0.1901	200	0.2111	0.2686	0.8980
0.2139	225	0.2034	-	-
0.2376	250	0.1820	-	-
0.2614	275	0.1809	-	-
0.2852	300	0.1854	0.2303	0.9102
0.3089	325	0.1847	-	-
0.3327	350	0.1769	-	-
0.3565	375	0.1665	-	-
0.3802	400	0.1651	0.2139	0.9133
0.4040	425	0.1583	-	-
0.4278	450	0.1560	-	-
0.4515	475	0.1651	-	-
0.4753	500	0.1501	0.1971	0.9172
0.4990	525	0.1483	-	-
0.5228	550	0.1436	-	-
0.5466	575	0.1595	-	-
0.5703	600	0.1418	0.1901	0.9176
0.5941	625	0.1471	-	-
0.6179	650	0.1410	-	-
0.6416	675	0.1422	-	-
0.6654	700	0.1442	0.1886	0.9195
0.6892	725	0.1317	-	-
0.7129	750	0.1415	-	-
0.7367	775	0.1339	-	-
0.7605	800	0.1365	0.1803	0.9223
0.7842	825	0.1357	-	-
0.8080	850	0.1281	-	-
0.8317	875	0.1244	-	-
0.8555	900	0.1285	0.1770	0.9235
0.8793	925	0.1258	-	-
0.9030	950	0.1287	-	-
0.9268	975	0.1264	-	-
0.9506	1000	0.1217	0.1745	0.9236
0.9743	1025	0.1286	-	-
0.9981	1050	0.1304	-	-
1.0	1052	-	0.1738	0.9239

Training Time

• Training: 33.0 minutes
• Evaluation: 1.1 minutes
• Total: 34.1 minutes

Framework Versions

• Python: 3.11.14
• Sentence Transformers: 5.6.0
• Transformers: 5.12.1
• PyTorch: 2.12.1+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}