nwachuks/MedEmbed-base-v0.1-finetuned

SentenceTransformer

This is a sentence-transformers model trained on the csv dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • csv

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("nwachuks/MedEmbed-base-v0.1-finetuned")
# Run inference
sentences = [
    '1. Participants who have an HLA-matched sibling who is able and willing to donate bone marrow. Patients with a HLA-matched unrelated donor are not excluded.',
    'Individuals with an HLA-matched sibling, ready and capable of donating bone marrow, are eligible to participate.',
    'Eligible candidates have no history of smoking or using tobacco products.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Datasets: ai-job-validation, ai-job-test and ai-job-test
• Evaluated with TripletEvaluator

Metric	ai-job-validation	ai-job-test
cosine_accuracy	0.9483	0.9474

Training Details

Training Dataset

csv

• Dataset: csv
• Size: 8,049 training samples
• Columns: eligibility_criteria, positives, normal_negatives, and hard_negatives
• Approximate statistics based on the first 1000 samples:

  	eligibility_criteria	positives	normal_negatives	hard_negatives
  type	string	string	string	string
  details	min: 5 tokens mean: 29.1 tokens max: 194 tokens	min: 6 tokens mean: 25.87 tokens max: 114 tokens	min: 4 tokens mean: 14.31 tokens max: 66 tokens	min: 6 tokens mean: 21.45 tokens max: 179 tokens

• Samples:

  eligibility_criteria	positives	normal_negatives	hard_negatives
  3. Extracted or missing upper permanent tooth/teeth (except for third molars).	One or more missing or extracted upper permanent teeth (excluding third molars) is required.	A history of braces does not affect eligibility.	No gaps in the upper permanent teeth (excluding third molars) are allowed.
  1. Patient is intolerant of existing therapy(ies) known to provide clinical benefit for their condition	Existing therapies that benefit the patient's condition are not tolerated by the patient.	The patient has no contraindications for MRI scans.	The patient does not exhibit intolerance towards established therapies that benefit their condition.
  1. Apparently healthy adult of 18 to 45 years of age.	Individuals with no known health issues, aged between 18 and 45.	Participants who have undergone a major surgical procedure in the past year.	Participants with known health issues, despite being younger than 18 or older than 45.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

csv

• Dataset: csv
• Size: 1,006 evaluation samples
• Columns: eligibility_criteria, positives, normal_negatives, and hard_negatives
• Approximate statistics based on the first 1000 samples:

  	eligibility_criteria	positives	normal_negatives	hard_negatives
  type	string	string	string	string
  details	min: 4 tokens mean: 28.38 tokens max: 230 tokens	min: 5 tokens mean: 25.18 tokens max: 156 tokens	min: 4 tokens mean: 13.96 tokens max: 67 tokens	min: 6 tokens mean: 20.65 tokens max: 129 tokens

• Samples:

  eligibility_criteria	positives	normal_negatives	hard_negatives
  1. Age of the patients is 40 years or older.	Participants must be aged 40 years or more.	Participants must have a history of diabetes mellitus.	Participants must be under 40 years old.
  12. Patients taking narcotics prior to elective colorectal surgery	Individuals with narcotic use before undergoing elective colorectal surgery.	Participants with a flu shot before the influenza season.	Participants taking narcotics following, rather than before, elective colorectal surgery.
  10. Intends to initiate a weight reduction program during the study	Plans to start a diet regimen over the course of the study.	Has no plans to start a new exercise routine during the study.	Refuses to start any weight reduction program during the study.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	ai-job-validation_cosine_accuracy	ai-job-test_cosine_accuracy
-1	-1	-	-	0.7018	-
0.7937	100	0.8566	0.3594	0.8837	-
1.5873	200	0.3644	0.2623	0.9235	-
2.3810	300	0.2665	0.2278	0.9404	-
3.1746	400	0.2186	0.2131	0.9463	-
3.9683	500	0.1928	0.2038	0.9503	-
4.7619	600	0.177	0.2014	0.9483	-
-1	-1	-	-	0.9483	0.9474

Framework Versions

• Python: 3.9.22
• Sentence Transformers: 4.1.0
• Transformers: 4.52.3
• PyTorch: 2.7.0+cu126
• Accelerate: 1.5.2
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}