slovak-nlp/e5-sk-large

SentenceTransformer based on mrshu/e5-large-trim-sk

This is a sentence-transformers model finetuned from mrshu/e5-large-trim-sk. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: mrshu/e5-large-trim-sk
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Vidiecke mydlo „Emmerdale“ vo štvrtok sledovalo 8,3 milióna ľudí, zatiaľ čo živá epizóda „The Bill“ mala 7,9 milióna divákov.',
    'Hodinové špeciálne epizódy seriálov ITV „Emmerdale“ a „The Bill“ boli najlepšie hodnotené britské televízne programy pri príležitosti 50. výročia komerčného kanála.',
    '4. mája 88 bolo unesených niekoľko honorárnych konzulov.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8783, 0.7403],
#         [0.8783, 1.0000, 0.7233],
#         [0.7403, 0.7233, 1.0000]])

Evaluation

Metrics

Semantic Similarity

• Dataset: validation_sts
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.8404
spearman_cosine	0.8424

Binary Classification

• Datasets: validation_nli and validation_rte
• Evaluated with BinaryClassificationEvaluator

Metric	validation_nli	validation_rte
cosine_accuracy	0.6663	0.5235
cosine_accuracy_threshold	0.9897	0.9791
cosine_f1	0.4997	0.6453
cosine_f1_threshold	0.7574	0.7797
cosine_precision	0.3333	0.4764
cosine_recall	0.9976	1.0
cosine_ap	0.2791	0.3795
cosine_mcc	0.0	0.0808

Multi Task Dev

• Evaluated with slovak_embeddings_v1.train.MultiTaskDevEvaluator

Metric	Value
validation_sts_pearson_cosine	0.8404
validation_sts_spearman_cosine	0.8424
validation_nli_cosine_accuracy	0.6663
validation_nli_cosine_accuracy_threshold	0.9897
validation_nli_cosine_f1	0.4997
validation_nli_cosine_f1_threshold	0.7574
validation_nli_cosine_precision	0.3333
validation_nli_cosine_recall	0.9976
validation_nli_cosine_ap	0.2791
validation_nli_cosine_mcc	0.0
validation_rte_cosine_accuracy	0.5235
validation_rte_cosine_accuracy_threshold	0.9791
validation_rte_cosine_f1	0.6453
validation_rte_cosine_f1_threshold	0.7797
validation_rte_cosine_precision	0.4764
validation_rte_cosine_recall	1.0
validation_rte_cosine_ap	0.3795
validation_rte_cosine_mcc	0.0808
validation_dev_overall	0.5003

Training Details

Training Datasets

Unnamed Dataset

• Size: 5,604 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 5 tokens mean: 19.41 tokens max: 71 tokens	min: 6 tokens mean: 19.54 tokens max: 87 tokens	min: 0.0 mean: 0.52 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  Akcie spoločnosti Corixa si na burze Nasdaq pripísali 71 centov alebo 10 % na 7,91 USD.	Počas neskorého ranného obchodovania na burze Nasdaq sa hodnota spoločnosti Corixa zvýšila o 74 centov, teda o 10 %, na 7,94 USD.	0.65
  Od Floridy po Aljašku tisíce účastníkov sľubovali, že budú presadzovať viac zákonných práv vrátane manželstiev osôb rovnakého pohlavia.	Tisíce ľudí, ktorí oslavovali toto rozhodnutie, prisľúbili, že budú presadzovať ďalšie zákonné práva vrátane manželstiev osôb rovnakého pohlavia.	0.8399999618530274
  Morrillova manželka Ellie počas obradu vzlykala a objímala Bondesonovu švagrinú.	Vdova po Morrillovi, Ellie, na obrade vzlykala a objímala Bondesonovu švagrinú, keď ju ľudia utešovali.	0.6800000190734863

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss"
  }
  

Unnamed Dataset

• Size: 130,900 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 4 tokens mean: 35.12 tokens max: 256 tokens	min: 6 tokens mean: 19.41 tokens max: 58 tokens

• Samples:

  sentence_0	sentence_1
  Od roku 1909 táto budova označuje sútok riek Gombak a Klang , kde baníci cínu nakladali zásoby , ktoré sa mali prepraviť proti prúdu rieky a vykladali cín , ktorý sa mal prepraviť na západ do prístavu Klang , a kde boli položené korene mesta .	Táto pošta je tam , kde sa stretávajú rieky Gombak a Klang .
  Žiarivé a trochu vyblednuté pozadie intenzívnej červenej farby sa dodnes nazýva Pompejská červená .	Odtieň červenej je úplne unikátny a vidieť len v Pompejach .
  Jemný obor sa hrá s malou myškou - Myši a ľudia . Zranený kat je požehnaný svojou blaženou obeťou - Billy Budd . Môžete pridať desiatky väzenských filmov , spolu s E.T. , Starmanom a dokonca aj nejaké filmy o samozvancoch .	V knihe O myšiach a ľuďoch sa jemný obor hrá s malou bielou myšou .

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Unnamed Dataset

• Size: 1,241 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 9 tokens mean: 72.48 tokens max: 256 tokens	min: 6 tokens mean: 17.56 tokens max: 60 tokens

• Samples:

  sentence_0	sentence_1
  Chirac potreboval pre svoju vládu nový mandát od voličov, alebo bola potrebná nová ľavicová vláda, ktorá by sa mohla spoľahnúť na podporu odborovej byrokracie a medzi robotníckou triedou, a tak by narazila na menší odpor.	Parlamentné voľby vytvárajú vo Francúzsku novú vládu.
  Hoci je pôrodnosť najvyššia za posledných päť rokov, v Škótsku bolo vlani stále viac úmrtí ako pôrodov.	Škótsko je európskou krajinou s najvyššou pôrodnosťou.
  Keď búrka prešla, v meste zavládla oslavná nálada až do skorého nedeľného rána, keď bol rapový magnát Suge Knight terčom streľby na párty Kanye Westa.	Najpozoruhodnejšia udalosť sa stala v nedeľu skoro ráno, keď majiteľku rapovej značky Marion "Suge" Knight postrelili do nohy na párty Kanye Westa.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	validation_sts_spearman_cosine	validation_nli_cosine_ap	validation_rte_cosine_ap	validation_dev_overall
0.3333	20	0.8424	0.2791	0.3795	0.5003

Framework Versions

• Python: 3.13.0
• Sentence Transformers: 5.2.0
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 4.4.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}