seanfarrell/icd11_ner_model_v2_large

SentenceTransformer based on google/embeddinggemma-300m

This is a sentence-transformers model finetuned from google/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/embeddinggemma-300m
• Maximum Sequence Length: 30 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (4): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("seanfarrell/icd11_ner_model_v2_large")
# Run inference
queries = [
    'multiple delivery by combination of methods',
]
documents = [
    'multiple delivery by combination of methods without caesarean',
    'bipolar or related disorders, unspecified',
    'rectal spasm',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.7717, -0.0575,  0.0184]])

Evaluation

Metrics

Information Retrieval

• Dataset: icd11-val
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7731
cosine_accuracy@5	0.848
cosine_accuracy@10	0.8713
cosine_precision@1	0.7731
cosine_precision@3	0.6729
cosine_precision@5	0.5754
cosine_precision@10	0.429
cosine_recall@1	0.1676
cosine_recall@3	0.402
cosine_recall@5	0.5068
cosine_recall@10	0.6314
cosine_ndcg@10	0.7209
cosine_mrr@10	0.8068
cosine_map@10	0.6633

Training Details

Training Dataset

Unnamed Dataset

• Size: 8,089,600 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 4 tokens mean: 8.06 tokens max: 20 tokens	min: 3 tokens mean: 7.71 tokens max: 20 tokens	min: 4 tokens mean: 8.25 tokens max: 19 tokens

• Samples:

  anchor	positive	negative
  rheumatic myocardiopathy	chronic rheumatic heart	abnormal liver lobulation
  chronic rheumatic coronary arteritis	inactive or quiescent rheumatic fever with left ventricular failure	other specified impairment of visual functions
  chronic rheumatic heart disease with endocarditis	rheumatic endocarditis	listeriosis, unspecified

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 512
• num_train_epochs: 1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 1580
• weight_decay: 0.01
• bf16: True
• per_device_eval_batch_size: 256
• eval_on_start: True
• load_best_model_at_end: True
• dataloader_drop_last: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 512
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 1580
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 256
• prediction_loss_only: True
• eval_on_start: True
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Epoch	Step	Training Loss	icd11-val_cosine_ndcg@10
0	0	-	0.5241
0.0032	50	2.0893	-
0.0063	100	1.6578	-
0.0095	150	1.4329	-
0.0127	200	1.2170	-
0.0158	250	1.0541	-
0.0190	300	0.9107	-
0.0222	350	0.7910	-
0.0253	400	0.6937	-
0.0285	450	0.5987	-
0.0316	500	0.5487	0.6342
0.0348	550	0.4945	-
0.0380	600	0.4407	-
0.0411	650	0.3987	-
0.0443	700	0.3634	-
0.0475	750	0.3231	-
0.0506	800	0.3094	-
0.0538	850	0.3041	-
0.0570	900	0.2820	-
0.0601	950	0.2629	-
0.0633	1000	0.2460	0.6571
0.0665	1050	0.2443	-
0.0696	1100	0.2154	-
0.0728	1150	0.2221	-
0.0759	1200	0.2117	-
0.0791	1250	0.2019	-
0.0823	1300	0.1920	-
0.0854	1350	0.1913	-
0.0886	1400	0.1777	-
0.0918	1450	0.1851	-
0.0949	1500	0.1792	0.6835
0.0981	1550	0.1725	-
0.1013	1600	0.1605	-
0.1044	1650	0.1743	-
0.1076	1700	0.1514	-
0.1108	1750	0.1455	-
0.1139	1800	0.1410	-
0.1171	1850	0.1209	-
0.1203	1900	0.1257	-
0.1234	1950	0.1135	-
0.1266	2000	0.1105	0.7033
0.1297	2050	0.1046	-
0.1329	2100	0.1068	-
0.1361	2150	0.0919	-
0.1392	2200	0.0985	-
0.1424	2250	0.0920	-
0.1456	2300	0.0868	-
0.1487	2350	0.0825	-
0.1519	2400	0.0826	-
0.1551	2450	0.0757	-
0.1582	2500	0.0778	0.7126
0.1614	2550	0.0697	-
0.1646	2600	0.0697	-
0.1677	2650	0.0649	-
0.1709	2700	0.0645	-
0.1741	2750	0.0661	-
0.1772	2800	0.0628	-
0.1804	2850	0.0553	-
0.1835	2900	0.0530	-
0.1867	2950	0.0565	-
0.1899	3000	0.0554	0.7209

Training Time

• Training: 29.3 minutes
• Evaluation: 48.8 seconds
• Total: 30.1 minutes

Framework Versions

• Python: 3.13.15
• Sentence Transformers: 5.7.0
• Transformers: 5.18.0
• PyTorch: 2.11.0+cu130
• Accelerate: 1.15.0
• Datasets: 4.8.5
• Tokenizers: 0.23.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}