Hugging Face's logo

seanfarrell
/
icd11_ner_model_v2

Sentence Similarity
sentence-transformers
Safetensors
bert
feature-extraction
dense
Generated from Trainer
dataset_size:8089600
loss:MultipleNegativesRankingLoss
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Community

SentenceTransformer based on abhinand/MedEmbed-small-v0.1

This is a sentence-transformers model finetuned from abhinand/MedEmbed-small-v0.1. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: abhinand/MedEmbed-small-v0.1
  • Maximum Sequence Length: 30 tokens
  • Output Dimensionality: 384 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("seanfarrell/icd11_ner_model_v2")
# Run inference
queries = [
    'multiple delivery by combination of methods',
]
documents = [
    'multiple delivery by combination of methods without caesarean',
    'bipolar or related disorders, unspecified',
    'rectal spasm',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 384] [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.9466, -0.0751, -0.0982]])

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.7649
cosine_accuracy@5 0.8421
cosine_accuracy@10 0.869
cosine_precision@1 0.7649
cosine_precision@3 0.6776
cosine_precision@5 0.5796
cosine_precision@10 0.427
cosine_recall@1 0.1662
cosine_recall@3 0.4103
cosine_recall@5 0.5203
cosine_recall@10 0.6468
cosine_ndcg@10 0.7242
cosine_mrr@10 0.7982
cosine_map@10 0.6694

Training Details

Training Dataset

Unnamed Dataset

  • Size: 8,089,600 training samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 100 samples:
    anchor positive negative
    type string string string
    modality text text text
    details
    • min: 4 tokens
    • mean: 9.52 tokens
    • max: 25 tokens
    • min: 3 tokens
    • mean: 9.1 tokens
    • max: 22 tokens
    • min: 3 tokens
    • mean: 9.37 tokens
    • max: 22 tokens
  • Samples:
    anchor positive negative
    rheumatic myocardiopathy chronic rheumatic heart abnormal liver lobulation
    chronic rheumatic coronary arteritis inactive or quiescent rheumatic fever with left ventricular failure other specified impairment of visual functions
    chronic rheumatic heart disease with endocarditis rheumatic endocarditis listeriosis, unspecified
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false,
    "directions": [
        "query_to_doc"
    ],
    "partition_mode": "joint",
    "hardness_mode": null,
    "hardness_strength": 0.0
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 512
  • num_train_epochs: 1
  • learning_rate: 2e-05
  • lr_scheduler_type: cosine
  • warmup_steps: 1580
  • weight_decay: 0.01
  • bf16: True
  • per_device_eval_batch_size: 256
  • eval_on_start: True
  • load_best_model_at_end: True
  • dataloader_drop_last: True

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 512
  • num_train_epochs: 1
  • max_steps: -1
  • learning_rate: 2e-05
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: None
  • warmup_steps: 1580
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.01
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 256
  • prediction_loss_only: True
  • eval_on_start: True
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 42
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: True
  • dataloader_num_workers: 0
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • dataloader_multiprocessing_context: None
  • dataloader_in_order: True
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • warmup_ratio: None

Training Logs

Click to expand
Epoch Step Training Loss icd11-val_cosine_ndcg@10
0 0 - 0.5035
0.0032 50 3.3034 -
0.0063 100 3.0570 -
0.0095 150 2.6980 -
0.0127 200 2.3748 -
0.0158 250 2.1693 -
0.0190 300 2.0670 -
0.0222 350 2.0079 -
0.0253 400 1.9661 -
0.0285 450 1.9317 -
0.0316 500 1.9053 0.5508
0.0348 550 1.8528 -
0.0380 600 1.8550 -
0.0411 650 1.7966 -
0.0443 700 1.7828 -
0.0475 750 1.7346 -
0.0506 800 1.6919 -
0.0538 850 1.6980 -
0.0570 900 1.6606 -
0.0601 950 1.6058 -
0.0633 1000 1.5859 0.5926
0.0665 1050 1.5658 -
0.0696 1100 1.5588 -
0.0728 1150 1.5080 -
0.0759 1200 1.4797 -
0.0791 1250 1.4802 -
0.0823 1300 1.4226 -
0.0854 1350 1.4006 -
0.0886 1400 1.3809 -
0.0918 1450 1.3675 -
0.0949 1500 1.3573 0.6178
0.0981 1550 1.3354 -
0.1013 1600 1.3037 -
0.1044 1650 1.2889 -
0.1076 1700 1.2706 -
0.1108 1750 1.2340 -
0.1139 1800 1.2057 -
0.1171 1850 1.1862 -
0.1203 1900 1.1948 -
0.1234 1950 1.1557 -
0.1266 2000 1.1571 0.6386
0.1297 2050 1.1288 -
0.1329 2100 1.1162 -
0.1361 2150 1.0780 -
0.1392 2200 1.0784 -
0.1424 2250 1.0607 -
0.1456 2300 1.0435 -
0.1487 2350 1.0256 -
0.1519 2400 1.0248 -
0.1551 2450 0.9990 -
0.1582 2500 0.9757 0.6572
0.1614 2550 0.9813 -
0.1646 2600 0.9761 -
0.1677 2650 0.9549 -
0.1709 2700 0.9369 -
0.1741 2750 0.9127 -
0.1772 2800 0.9223 -
0.1804 2850 0.9096 -
0.1835 2900 0.8837 -
0.1867 2950 0.8843 -
0.1899 3000 0.8898 0.6649
0.1930 3050 0.8656 -
0.1962 3100 0.8796 -
0.1994 3150 0.8464 -
0.2025 3200 0.8261 -
0.2057 3250 0.8280 -
0.2089 3300 0.8319 -
0.2120 3350 0.8292 -
0.2152 3400 0.7955 -
0.2184 3450 0.8011 -
0.2215 3500 0.7908 0.6771
0.2247 3550 0.7760 -
0.2278 3600 0.7667 -
0.2310 3650 0.7683 -
0.2342 3700 0.7523 -
0.2373 3750 0.7344 -
0.2405 3800 0.7471 -
0.2437 3850 0.7250 -
0.2468 3900 0.7163 -
0.25 3950 0.7003 -
0.2532 4000 0.7103 0.6836
0.2563 4050 0.7073 -
0.2595 4100 0.7046 -
0.2627 4150 0.6804 -
0.2658 4200 0.6806 -
0.2690 4250 0.6792 -
0.2722 4300 0.6708 -
0.2753 4350 0.6678 -
0.2785 4400 0.6475 -
0.2816 4450 0.6444 -
0.2848 4500 0.6395 0.6841
0.2880 4550 0.6399 -
0.2911 4600 0.6415 -
0.2943 4650 0.6407 -
0.2975 4700 0.6220 -
0.3006 4750 0.6198 -
0.3038 4800 0.6145 -
0.3070 4850 0.6029 -
0.3101 4900 0.6042 -
0.3133 4950 0.5962 -
0.3165 5000 0.5979 0.6933
0.3196 5050 0.5811 -
0.3228 5100 0.5738 -
0.3259 5150 0.5731 -
0.3291 5200 0.5787 -
0.3323 5250 0.5770 -
0.3354 5300 0.5539 -
0.3386 5350 0.5573 -
0.3418 5400 0.5619 -
0.3449 5450 0.5387 -
0.3481 5500 0.5357 0.6979
0.3513 5550 0.5502 -
0.3544 5600 0.5316 -
0.3576 5650 0.5386 -
0.3608 5700 0.5257 -
0.3639 5750 0.5032 -
0.3671 5800 0.5197 -
0.3703 5850 0.5147 -
0.3734 5900 0.5194 -
0.3766 5950 0.5195 -
0.3797 6000 0.5241 0.6999
0.3829 6050 0.5032 -
0.3861 6100 0.4955 -
0.3892 6150 0.5055 -
0.3924 6200 0.4951 -
0.3956 6250 0.5022 -
0.3987 6300 0.4916 -
0.4019 6350 0.4840 -
0.4051 6400 0.4684 -
0.4082 6450 0.4569 -
0.4114 6500 0.4737 0.7041
0.4146 6550 0.4609 -
0.4177 6600 0.4694 -
0.4209 6650 0.4588 -
0.4241 6700 0.4710 -
0.4272 6750 0.4526 -
0.4304 6800 0.4605 -
0.4335 6850 0.4720 -
0.4367 6900 0.4594 -
0.4399 6950 0.4497 -
0.4430 7000 0.4475 0.7087
0.4462 7050 0.4415 -
0.4494 7100 0.4275 -
0.4525 7150 0.4349 -
0.4557 7200 0.4375 -
0.4589 7250 0.4340 -
0.4620 7300 0.4346 -
0.4652 7350 0.4281 -
0.4684 7400 0.4277 -
0.4715 7450 0.4188 -
0.4747 7500 0.4179 0.7109
0.4778 7550 0.4129 -
0.4810 7600 0.4137 -
0.4842 7650 0.4136 -
0.4873 7700 0.4091 -
0.4905 7750 0.4068 -
0.4937 7800 0.4089 -
0.4968 7850 0.4277 -
0.5 7900 0.3952 -
0.5032 7950 0.4009 -
0.5063 8000 0.3889 0.7134
0.5095 8050 0.4008 -
0.5127 8100 0.4031 -
0.5158 8150 0.3950 -
0.5190 8200 0.3807 -
0.5222 8250 0.3891 -
0.5253 8300 0.3940 -
0.5285 8350 0.3929 -
0.5316 8400 0.3935 -
0.5348 8450 0.3755 -
0.5380 8500 0.3788 0.7176
0.5411 8550 0.3770 -
0.5443 8600 0.3882 -
0.5475 8650 0.3747 -
0.5506 8700 0.3785 -
0.5538 8750 0.3689 -
0.5570 8800 0.3706 -
0.5601 8850 0.3756 -
0.5633 8900 0.3679 -
0.5665 8950 0.3696 -
0.5696 9000 0.3722 0.7176
0.5728 9050 0.3647 -
0.5759 9100 0.3616 -
0.5791 9150 0.3635 -
0.5823 9200 0.3447 -
0.5854 9250 0.3612 -
0.5886 9300 0.3609 -
0.5918 9350 0.3368 -
0.5949 9400 0.3509 -
0.5981 9450 0.3631 -
0.6013 9500 0.3422 0.7188
0.6044 9550 0.3537 -
0.6076 9600 0.3507 -
0.6108 9650 0.3460 -
0.6139 9700 0.3527 -
0.6171 9750 0.3441 -
0.6203 9800 0.3509 -
0.6234 9850 0.3396 -
0.6266 9900 0.3350 -
0.6297 9950 0.3493 -
0.6329 10000 0.3299 0.7220
0.6361 10050 0.3377 -
0.6392 10100 0.3401 -
0.6424 10150 0.3414 -
0.6456 10200 0.3385 -
0.6487 10250 0.3408 -
0.6519 10300 0.3450 -
0.6551 10350 0.3482 -
0.6582 10400 0.3261 -
0.6614 10450 0.3350 -
0.6646 10500 0.3335 0.7215
0.6677 10550 0.3458 -
0.6709 10600 0.3194 -
0.6741 10650 0.3224 -
0.6772 10700 0.3289 -
0.6804 10750 0.3239 -
0.6835 10800 0.3283 -
0.6867 10850 0.3277 -
0.6899 10900 0.3243 -
0.6930 10950 0.3196 -
0.6962 11000 0.3204 0.7220
0.6994 11050 0.3334 -
0.7025 11100 0.3216 -
0.7057 11150 0.3286 -
0.7089 11200 0.3119 -
0.7120 11250 0.3185 -
0.7152 11300 0.3058 -
0.7184 11350 0.3157 -
0.7215 11400 0.3136 -
0.7247 11450 0.3179 -
0.7278 11500 0.3187 0.7235
0.7310 11550 0.3144 -
0.7342 11600 0.3156 -
0.7373 11650 0.3139 -
0.7405 11700 0.3104 -
0.7437 11750 0.3187 -
0.7468 11800 0.3154 -
0.75 11850 0.3132 -
0.7532 11900 0.3110 -
0.7563 11950 0.3143 -
0.7595 12000 0.3119 0.7236
0.7627 12050 0.3068 -
0.7658 12100 0.3227 -
0.7690 12150 0.3170 -
0.7722 12200 0.3046 -
0.7753 12250 0.3085 -
0.7785 12300 0.3135 -
0.7816 12350 0.3042 -
0.7848 12400 0.3152 -
0.7880 12450 0.3122 -
0.7911 12500 0.3201 0.7235
0.7943 12550 0.3148 -
0.7975 12600 0.2993 -
0.8006 12650 0.3040 -
0.8038 12700 0.2973 -
0.8070 12750 0.3074 -
0.8101 12800 0.3074 -
0.8133 12850 0.2971 -
0.8165 12900 0.3060 -
0.8196 12950 0.3136 -
0.8228 13000 0.3095 0.7244
0.8259 13050 0.3032 -
0.8291 13100 0.3070 -
0.8323 13150 0.3014 -
0.8354 13200 0.2965 -
0.8386 13250 0.2977 -
0.8418 13300 0.3077 -
0.8449 13350 0.3145 -
0.8481 13400 0.3019 -
0.8513 13450 0.3201 -
0.8544 13500 0.2986 0.7248
0.8576 13550 0.3001 -
0.8608 13600 0.3047 -
0.8639 13650 0.3040 -
0.8671 13700 0.2923 -
0.8703 13750 0.3029 -
0.8734 13800 0.2942 -
0.8766 13850 0.2970 -
0.8797 13900 0.2971 -
0.8829 13950 0.2927 -
0.8861 14000 0.3006 0.7239
0.8892 14050 0.2930 -
0.8924 14100 0.3001 -
0.8956 14150 0.3124 -
0.8987 14200 0.3083 -
0.9019 14250 0.3059 -
0.9051 14300 0.2926 -
0.9082 14350 0.2905 -
0.9114 14400 0.2959 -
0.9146 14450 0.3038 -
0.9177 14500 0.3045 0.7243
0.9209 14550 0.2908 -
0.9241 14600 0.3043 -
0.9272 14650 0.3007 -
0.9304 14700 0.2933 -
0.9335 14750 0.2951 -
0.9367 14800 0.2874 -
0.9399 14850 0.3041 -
0.9430 14900 0.3019 -
0.9462 14950 0.3073 -
0.9494 15000 0.3064 0.7245
0.9525 15050 0.3038 -
0.9557 15100 0.3043 -
0.9589 15150 0.2990 -
0.9620 15200 0.3016 -
0.9652 15250 0.3073 -
0.9684 15300 0.2959 -
0.9715 15350 0.2936 -
0.9747 15400 0.2928 -
0.9778 15450 0.2945 -
0.9810 15500 0.2930 0.7247
0.9842 15550 0.3092 -
0.9873 15600 0.3079 -
0.9905 15650 0.2975 -
0.9937 15700 0.2969 -
0.9968 15750 0.2890 -
1.0 15800 0.2956 0.7242
  • The bold row denotes the saved checkpoint.

Training Time

  • Training: 26.3 minutes
  • Evaluation: 1.5 minutes
  • Total: 27.8 minutes

Framework Versions

  • Python: 3.13.15
  • Sentence Transformers: 5.7.0
  • Transformers: 5.18.0
  • PyTorch: 2.11.0+cu130
  • Accelerate: 1.15.0
  • Datasets: 4.8.5
  • Tokenizers: 0.23.2

Additional Resources

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss 

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}
Downloads last month
14
Safetensors
Model size
33.4M params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for seanfarrell/icd11_ner_model_v2

Base model

BAAI/bge-small-en-v1.5
Finetuned
abhinand/MedEmbed-small-v0.1
Finetuned
(6)
this model

Papers for seanfarrell/icd11_ner_model_v2

Evaluation results