vab46/nomic-embed-text-v1.5_Clinical-Trials_Matryoshka_final

nomic-embed text-v1.5 Clinical-Trials Matryoshka

This is a sentence-transformers model finetuned from nomic-ai/nomic-embed-text-v1.5 on the dataset from Clinical_trials_anchor-positive-pairs_EmbeddingModel-data_final. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/nomic-embed-text-v1.5
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset: Clinical_trials_anchor-positive-pairs_EmbeddingModel-data_final
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'NomicBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("vab46/nomic-embed-text-v1.5_Clinical-Trials_Matryoshka_final")
# Run inference
documents = [
    'TITLE: Safety and Efficacy of Six-Channel Radiofrequency Ablation System for Renal Denervation in Patients With Untreated Grade I Hypertension: a Pilot Study\nSUMMARY: Prospective, Multi-Center, Randomized, shame-Controlled, Uptake clinical trial to evaluate the efficacy and safety of the six-channel radio-frequency（RF) renal denervation system-comprising the six-channel RF generator (specification model: 25D1G, software release version: SRG-V1) and the disposable ultra-guiding RF denervation catheter (specification model: 25C6W127F115T)-for renal denervation in patients with grade I hypertension and without taking antihypertensive medicines.\nINCLUSION_CRITERIA: 1. Male or female, aged 18 to 65 years inclusive\n2. Hypertension duration longer than 3 months\n3. Hypertensive subjects who have been stopped taking antihypertensive drugs continuously and stably for at least 4 weeks or who do not take antihypertensive drugs , with office systolic/diastolic blood pressure still ≥140/90 mmHg and \\<160/100 mmHg, and 24-hour ambulatory mean systolic /diastolic pressure ≥130/80 mmHg and \\<140/90 mmHg;\n4. The subject or his/her legal representative fully understands the content of the informed consent form for this trial and voluntarily signs the written informed consent form.',
]
queries = [
    "I have high blood pressure, but I'm not on any medication. Can I participate in this study?",
    'What is the primary goal of this clinical trial regarding pain management in premature newborns?',
    "I'm on a ventilator and feel really short of breath. Can anything be done to make me feel more comfortable?",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5783, 0.0218, 0.0940]])

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768, dim_512, dim_256, dim_128, dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

Metric	dim_768	dim_512	dim_256	dim_128	dim_64
cosine_accuracy@1	0.5476	0.5349	0.5235	0.4905	0.4409
cosine_accuracy@3	0.662	0.6417	0.6531	0.6048	0.5667
cosine_accuracy@5	0.6938	0.6874	0.6836	0.6607	0.6213
cosine_accuracy@10	0.7395	0.7395	0.7306	0.7116	0.6773
cosine_precision@1	0.5476	0.5349	0.5235	0.4905	0.4409
cosine_precision@3	0.2207	0.2139	0.2177	0.2016	0.1889
cosine_precision@5	0.1388	0.1375	0.1367	0.1321	0.1243
cosine_precision@10	0.074	0.074	0.0731	0.0712	0.0677
cosine_recall@1	0.5476	0.5349	0.5235	0.4905	0.4409
cosine_recall@3	0.662	0.6417	0.6531	0.6048	0.5667
cosine_recall@5	0.6938	0.6874	0.6836	0.6607	0.6213
cosine_recall@10	0.7395	0.7395	0.7306	0.7116	0.6773
cosine_ndcg@10	0.6433	0.6346	0.628	0.5974	0.5563
cosine_mrr@10	0.6126	0.6015	0.5951	0.5612	0.5179
cosine_map@100	0.6162	0.6049	0.5986	0.5654	0.5228

Comparison with the baseline model performance on same data

	metric	dimensions	base_value	ft_value	diff	%change
0	accuracy@1	768	0.43202	0.547649	0.115629	26.76471
1	accuracy@1	512	0.43075	0.536213	0.105464	24.48378
2	accuracy@1	256	0.400254	0.518424	0.11817	29.52381
3	accuracy@1	128	0.3723	0.499365	0.127065	34.12969
4	accuracy@1	64	0.29479	0.456163	0.161372	54.74138
5	accuracy@3	768	0.52859	0.653113	0.124524	23.55769
6	accuracy@3	512	0.519695	0.641677	0.121982	23.47188
7	accuracy@3	256	0.501906	0.636595	0.134689	26.83544
8	accuracy@3	128	0.473952	0.60737	0.133418	28.15013
9	accuracy@3	64	0.416773	0.570521	0.153748	36.89024
10	accuracy@5	768	0.574333	0.700127	0.125794	21.90265
11	accuracy@5	512	0.570521	0.687421	0.1169	20.48998
12	accuracy@5	256	0.550191	0.674714	0.124524	22.63279
13	accuracy@5	128	0.520966	0.653113	0.132147	25.36585
14	accuracy@5	64	0.465057	0.617535	0.152478	32.78689
15	accuracy@10	768	0.622618	0.740788	0.11817	18.97959
16	accuracy@10	512	0.609911	0.734435	0.124524	20.41667
17	accuracy@10	256	0.60737	0.724269	0.1169	19.24686
18	accuracy@10	128	0.574333	0.702668	0.128335	22.34513
19	accuracy@10	64	0.523507	0.670902	0.147395	28.15534
20	recall@1	768	0.43202	0.547649	0.115629	26.76471
21	recall@1	512	0.43075	0.536213	0.105464	24.48378
22	recall@1	256	0.400254	0.518424	0.11817	29.52381
23	recall@1	128	0.3723	0.499365	0.127065	34.12969
24	recall@1	64	0.29479	0.456163	0.161372	54.74138
25	recall@3	768	0.52859	0.653113	0.124524	23.55769
26	recall@3	512	0.519695	0.641677	0.121982	23.47188
27	recall@3	256	0.501906	0.636595	0.134689	26.83544
28	recall@3	128	0.473952	0.60737	0.133418	28.15013
29	recall@3	64	0.416773	0.570521	0.153748	36.89024
30	recall@5	768	0.574333	0.700127	0.125794	21.90265
31	recall@5	512	0.570521	0.687421	0.1169	20.48998
32	recall@5	256	0.550191	0.674714	0.124524	22.63279
33	recall@5	128	0.520966	0.653113	0.132147	25.36585
34	recall@5	64	0.465057	0.617535	0.152478	32.78689
35	recall@10	768	0.622618	0.740788	0.11817	18.97959
36	recall@10	512	0.609911	0.734435	0.124524	20.41667
37	recall@10	256	0.60737	0.724269	0.1169	19.24686
38	recall@10	128	0.574333	0.702668	0.128335	22.34513
39	recall@10	64	0.523507	0.670902	0.147395	28.15534
40	precision@1	768	0.43202	0.547649	0.115629	26.76471
41	precision@1	512	0.43075	0.536213	0.105464	24.48378
42	precision@1	256	0.400254	0.518424	0.11817	29.52381
43	precision@1	128	0.3723	0.499365	0.127065	34.12969
44	precision@1	64	0.29479	0.456163	0.161372	54.74138
45	precision@3	768	0.176197	0.217704	0.041508	23.55769
46	precision@3	512	0.173232	0.213892	0.040661	23.47188
47	precision@3	256	0.167302	0.212198	0.044896	26.83544
48	precision@3	128	0.157984	0.202457	0.044473	28.15013
49	precision@3	64	0.138924	0.190174	0.051249	36.89024
50	precision@5	768	0.114867	0.140025	0.025159	21.90265
51	precision@5	512	0.114104	0.137484	0.02338	20.48998
52	precision@5	256	0.110038	0.134943	0.024905	22.63279
53	precision@5	128	0.104193	0.130623	0.026429	25.36585
54	precision@5	64	0.093011	0.123507	0.030496	32.78689
55	precision@10	768	0.062262	0.074079	0.011817	18.97959
56	precision@10	512	0.060991	0.073443	0.012452	20.41667
57	precision@10	256	0.060737	0.072427	0.01169	19.24686
58	precision@10	128	0.057433	0.070267	0.012834	22.34513
59	precision@10	64	0.052351	0.06709	0.01474	28.15534
60	map@100	768	0.496152	0.613222	0.11707	23.59553
61	map@100	512	0.492057	0.603853	0.111796	22.72006
62	map@100	256	0.469749	0.592448	0.122699	26.12
63	map@100	128	0.441045	0.570548	0.129503	29.36267
64	map@100	64	0.374852	0.529623	0.15477	41.28835
65	mrr@10	768	0.491678	0.610103	0.118425	24.08598
66	mrr@10	512	0.486797	0.600289	0.113492	23.31392
67	mrr@10	256	0.464769	0.58866	0.123891	26.6565
68	mrr@10	128	0.435313	0.566223	0.130911	30.07274
69	mrr@10	64	0.368128	0.524602	0.156474	42.50527
70	ndcg@10	768	0.523048	0.641648	0.1186	22.67486
71	ndcg@10	512	0.516347	0.632601	0.116254	22.5146
72	ndcg@10	256	0.498844	0.621455	0.122611	24.57903
73	ndcg@10	128	0.468557	0.599204	0.130646	27.88273
74	ndcg@10	64	0.405503	0.559806	0.154303	38.05217
	AVERAGE					27.30715
	VARIANCE					60.62231
	STD_DEV					7.786033

For the list of Sequential Evaluator score on nested mashtroyska embeddings [768, 512, 256, 128, 64]:-

Initial(pre-trained) score:- 0.405----------->Final(fine-tuned) score:- 0.56

Training Details

Training Dataset

Unnamed Dataset

• Size: 7,079 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 100 samples:

  	positive	anchor
  type	string	string
  modality	text	text
  details	min: 72 tokens mean: 290.18 tokens max: 675 tokens	min: 11 tokens mean: 21.08 tokens max: 42 tokens

• Samples:

  positive	anchor
  TITLE: Risk Classification and Prediction of Histopathological Subtypes in Basal Cell Carcinoma Using a CNN-Based Artificial Intelligence Model on Dermoscopic Images SUMMARY: This retrospective observational study aims to develop and evaluate a convolutional neural network (CNN)-based artificial intelligence model for risk classification and histopathological subtype prediction of basal cell carcinoma (BCC) using clinical and dermoscopic images. Histopathologically confirmed BCC cases from a dermatology archive will be included. The primary objective is to assess the diagnostic performance of the CNN model in classifying BCC as low-risk or high-risk. Secondary objectives include predicting histopathological subtypes and comparing the model's performance with that of dermatology physicians. Histopathological diagnosis will serve as the reference standard. All archived data will be anonymized before analysis. INCLUSION_CRITERIA: Patients with histopathologically confirmed basal cell carc...	Are dermoscopic images with sufficient image quality and resolution required for participation in this trial?
  TITLE: Postural Control Mechanism During a Stationary Wheelchair Wheelie SUMMARY: The purpose of this study was to investigate the postural control during a stationary wheelchair wheelie. A group of participants was recruited for this observational study. Participants were asked to perform and maintain the stationary phase of a wheelchair wheelie, during which kinematic and kinetic data were recorded. INCLUSION_CRITERIA: Spinal cord injury at or below the T1 (first thoracic) level. Between 20 and 65 years of age. Currently using a manual wheelchair as the primary means of mobility. Able to perform and maintain a wheelchair wheelie for at least 20 seconds	Is there an age restriction for participants in this clinical trial, and if so, what is it?
  TITLE: The Effect of Mandala Art Therapy on Fear and Anxiety of Childbirth in Expectant Fathers Waiting Outside the Delivery Room: A Randomized Controlled Study SUMMARY: This randomized controlled trial is designed to evaluate the effectiveness of mandala art therapy in reducing childbirth fear and state anxiety among fathers whose partners are nulliparous pregnant women and who are waiting in the obstetric ward waiting area during labor. Eligible participants will be randomly allocated to either the intervention or the control group. Fathers assigned to the intervention group will participate in a single 20-25-minute mandala art therapy session, whereas those in the control group will receive standard care without any additional intervention. Childbirth fear and state anxiety will be assessed using the Fathers' Birth Fear Scale and the State subscale of the Spielberger State-Trait Anxiety Inventory, respectively, at baseline and immediately following the intervention. The findings of ...	Could a father with a partner expecting their first child qualify for this study?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 4
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 0.1
• gradient_accumulation_steps: 8
• fp16: True
• per_device_eval_batch_size: 32
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 8
• num_train_epochs: 4
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 8
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.0904	10	4.1897	-	-	-	-	-
0.1808	20	3.0473	-	-	-	-	-
0.2712	30	2.3585	-	-	-	-	-
0.3616	40	1.5994	-	-	-	-	-
0.4520	50	1.6686	-	-	-	-	-
0.5424	60	1.5741	-	-	-	-	-
0.6328	70	1.5577	-	-	-	-	-
0.7232	80	1.5720	-	-	-	-	-
0.8136	90	1.3066	-	-	-	-	-
0.9040	100	1.3134	-	-	-	-	-
0.9944	110	1.3546	-	-	-	-	-
1.0	111	-	0.6286	0.6207	0.6117	0.5864	0.5414
1.0814	120	1.0738	-	-	-	-	-
1.1718	130	1.1824	-	-	-	-	-
1.2621	140	0.9235	-	-	-	-	-
1.3525	150	1.1639	-	-	-	-	-
1.4429	160	0.9303	-	-	-	-	-
1.5333	170	0.8958	-	-	-	-	-
1.6237	180	1.0578	-	-	-	-	-
1.7141	190	1.0462	-	-	-	-	-
1.8045	200	0.9778	-	-	-	-	-
1.8949	210	0.8817	-	-	-	-	-
1.9853	220	0.8784	-	-	-	-	-
2.0	222	-	0.6416	0.6326	0.6215	0.5992	0.5598
2.0723	230	0.7649	-	-	-	-	-
2.1627	240	0.6925	-	-	-	-	-
2.2531	250	0.7085	-	-	-	-	-
2.3435	260	0.6575	-	-	-	-	-
2.4339	270	0.7791	-	-	-	-	-
2.5243	280	0.6990	-	-	-	-	-
2.6147	290	0.6887	-	-	-	-	-
2.7051	300	0.8579	-	-	-	-	-
2.7955	310	0.7612	-	-	-	-	-
2.8859	320	0.5202	-	-	-	-	-
2.9763	330	0.7271	-	-	-	-	-
3.0	333	-	0.6422	0.6360	0.6284	0.5985	0.5562
3.0633	340	0.6966	-	-	-	-	-
3.1537	350	0.5423	-	-	-	-	-
3.2441	360	0.6177	-	-	-	-	-
3.3345	370	0.5960	-	-	-	-	-
3.4249	380	0.5821	-	-	-	-	-
3.5153	390	0.4473	-	-	-	-	-
3.6056	400	0.6135	-	-	-	-	-
3.6960	410	0.5474	-	-	-	-	-
3.7864	420	0.4391	-	-	-	-	-
3.8768	430	0.5270	-	-	-	-	-
3.9672	440	0.5693	-	-	-	-	-
4.0	444	-	0.6433	0.6346	0.6280	0.5974	0.5563

• The bold row denotes the saved checkpoint.

Training Time

• Training: 31.6 minutes

Framework Versions

• Python: 3.13.15
• Sentence Transformers: 5.7.0
• Transformers: 5.15.0
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}