Static SELFIES embedding (320d) trained on COCONUT + ChEMBL34 pairs

This is a sentence-transformers model trained on the coconut-chembl34-mol-sim dataset. It maps inputs to a 320-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Maximum Sequence Length: 200 tokens
  • Output Dimensionality: 320 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text
  • Training Dataset:
    • coconut-chembl34-mol-sim
  • Language: en
  • License: cc-by-nc-sa-4.0

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): StaticEmbedding({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/static-chemselfies-molsim")
# Run inference
sentences = [
    '[O] [=S] [Branch1] [Ring1] [O] [C] [S] [S] [S] [S] [S] [N]',
    '[S] [C] [=C] [C] [=C] [Ring1] [Branch1] [C] [=C]',
    '[O] [=C] [Branch1] [O] [C] [C] [=C] [C] [#C] [C] [#C] [C] [=C] [C] [C] [C]',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 320]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6880, 0.1453],
#         [0.6880, 1.0000, 0.1629],
#         [0.1453, 0.1629, 1.0000]])

Evaluation

Metrics

Semantic Similarity

Metric molsim molsim-fulltest
pearson_cosine 0.7647 0.7735
spearman_cosine 0.7428 0.7519

Baseline comparison

All numbers are Spearman/Pearson of embedding cosine similarity against MACCS fingerprint similarity labels on held-out SELFIES pairs of gbyuvd/coconut-chembl34-mol-sim.

Model Type Params Spearman cosine Pearson cosine
gbyuvd/chemembed-chemselfies (8-layer BERT, hidden 320) contextual 11.1M 0.952* 0.961*
This model, untrained (token-overlap floor) static ~1M ~0.54 -
This model (Matryoshka 320/160/80) static ~1M 0.7519 0.7735

* as reported by the authors of gbyuvd/chemembed-chemselfies on their held-out "combined-test" split of the same data family, with EmbeddingSimilarityEvaluator-style Spearman/Pearson of cosine vs MACCS. This model was evaluated on the dataset's own full test split (118,318 pairs), so the numbers are directly comparable in kind but not bit-identical in split.

The ~0.20 Spearman gap is the price of going static: a mean-pooled embedding table (one vector per SELFIES symbol, no context, ~1M parameters) cannot model token interactions the way the 11.1M-parameter BERT can. In exchange, this model is orders of magnitude faster at inference (a lookup + average; ~20x faster than a comparable-size encoder on GPU and ~80x on CPU) and supports Matryoshka truncation to 160/80 dims.

Limitations & Notes

  • Static model: mean-pooled SELFIES token embeddings, no context or attention. It trails contextual encoders on fine-grained similarity; it trades that for ~1M parameters and very fast inference.
  • Input format: SELFIES strings must be space-separated tokens, e.g. [C] [=C] [N] (use " ".join(re.findall(r"\[[^\]]*\]", s)) on raw SELFIES output). Raw glued SELFIES ([C][=C][N]) encodes to [UNK].
  • The tokenizer's [CLS]/[SEP] post-processing was stripped for training and inference (noise for a token-bag model). Vocab: 3,095 SELFIES symbols, taken from gbyuvd/chemembed-chemselfies.
  • Labels are MACCS fingerprint cosine similarity, so the embeddings primarily encode MACCS-like similarity. 1 epoch reached 0.7473 Spearman, 3 epochs 0.7519 — diminishing returns. Recipe: CosineSimilarityLoss + MatryoshkaLoss [320/160/80], lr 0.05, batch 2048, cosine scheduler, best-checkpoint restore; ~15 min training on a T4.
  • License cc-by-nc-sa-4.0, inherited from the baseline model and dataset by gbyuvd.

Training Details

Training Dataset

coconut-chembl34-mol-sim

  • Dataset: coconut-chembl34-mol-sim
  • Size: 1,049,286 training samples
  • Columns: sentence1, sentence2, and label
  • Approximate statistics based on the first 100 samples:
    sentence1 sentence2 label
    type string string float
    modality text text
    details
    • min: 60 characters
    • mean: 436.88 characters
    • max: 1766 characters
    • min: 48 characters
    • mean: 446.99 characters
    • max: 1875 characters
    • min: 0.2
    • mean: 0.54
    • max: 0.92
  • Samples:
    sentence1 sentence2 label
    [O] [=C] [N] [C] [C] [C] [C] [Ring1] [=Branch1] [=C] [O] [C] [Branch2] [Ring1] [Branch1] [O] [C] [O] [C] [Branch1] [Ring1] [C] [O] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Ring1] [#Branch2] [O] [C] [Ring2] [Ring1] [C] [C] [C] [O] [=C] [C] [=C] [Branch2] [Ring2] [=Branch1] [O] [C] [=Branch1] [Branch1] [=C] [Ring1] [=Branch1] [O] [C] [NH1] [C] [=Branch1] [=N] [=N] [C] [=Ring1] [Branch1] [C] [=C] [C] [=C] [C] [=C] [Ring1] [=Branch1] [C] [=C] [C] [=C] [Branch1] [C] [O] [C] [=C] [Ring1] [#Branch1] [O] [C] 0.57
    [O] [=C] [O] [C] [Branch1] [C] [C] [C] [Branch2] [Ring2] [O] [N] [=C] [Branch1] [C] [O] [C] [Branch2] [Ring1] [Branch1] [N] [=C] [Branch1] [C] [O] [C] [Branch1] [Ring1] [O] [C] [C] [O] [S] [=Branch1] [C] [=O] [=Branch1] [C] [=O] [O] [C] [C] [C] [=C] [C] [=C] [Branch1] [C] [O] [C] [=C] [Ring1] [#Branch1] [C] [=Branch2] [=Branch1] [#Branch2] [=N] [C] [Branch2] [=Branch1] [Ring2] [C] [=Branch2] [Branch1] [#C] [=N] [C] [C] [=Branch1] [C] [=O] [N] [Branch1] [=Branch2] [C] [Branch1] [C] [O] [C] [C] [Ring1] [Branch2] [C] [Branch2] [Ring2] [Ring2] [C] [=Branch1] [C] [=O] [N] [Branch1] [C] [C] [C] [Branch1] [S] [C] [=Branch1] [N] [=N] [C] [Ring2] [Branch1] [Ring2] [C] [Branch1] [C] [C] [C] [C] [O] [C] [C] [=C] [C] [=C] [C] [=C] [Ring1] [=Branch1] [C] [Branch1] [C] [C] [C] [C] [O] [C] [O] [O] [=C] [C] [C] [C] [Ring1] [Ring2] [C] [C] [C] [C] [C] [=C] [C] [C] [=C] [C] [C] [C] [C] [C] 0.38
    [O] [=C] [N] [C] [Branch2] [Ring1] [C] [O] [C] [=Branch1] [C] [=O] [C] [C] [Branch1] [Ring2] [N] [C] [=O] [C] [Branch1] [C] [C] [C] [C] [Branch1] [C] [C] [C] [O] [=C] [Branch3] [Ring1] [C] [Branch2] [O] [C] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Branch1] [=Branch1] [O] [C] [Ring1] [Branch2] [C] [O] [C] [Branch1] [Ring1] [C] [=C] [Branch1] [C] [C] [C] [C] [C] [=C] [Branch2] [S] [Branch1] [C] [=Branch1] [C] [=O] [O] [C] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Branch1] [=Branch1] [O] [C] [Ring1] [Branch2] [C] [O] [C] [Branch1] [Ring1] [C] [=C] [Branch1] [C] [C] [C] [C] [C] [=C] [Branch2] [=N] [=C] [C] [=Branch1] [C] [=O] [O] [C] [C] [C] [Branch2] [Branch1] [O] [C] [=Branch1] [C] [=O] [O] [C] [O] [C] [Branch1] [Ring1] [C] [O] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Ring1] [#Branch2] [O] [C] [O] [C] [Branch1] [C] [C] [C] [Branch1] [P] [O] [C] [O] [C] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Ring1] [Branch2] [O] [C] [Branch1] [C] [O] [C] [Ring2] [Ring1] [C] [O] [C] [Branch1] [C] [O] [C] [C] [Branch2] [Branch2] [#C] [C] [=Branch2] [#Branch1] [=C] [=C] [C] [C] [C] [Branch1] [C] [C] [C] [C] [C] [Branch2] [Branch1]... 0.49
  • Loss: MatryoshkaLoss with these parameters:
    {
        "loss": "CosineSimilarityLoss",
        "matryoshka_dims": [
            320,
            160,
            80
        ],
        "matryoshka_weights": [
            1,
            1,
            1
        ],
        "n_dims_per_step": -1
    }
    

Evaluation Dataset

coconut-chembl34-mol-sim

  • Dataset: coconut-chembl34-mol-sim
  • Size: 10,000 evaluation samples
  • Columns: sentence1, sentence2, and label
  • Approximate statistics based on the first 100 samples:
    sentence1 sentence2 label
    type string string float
    modality text text
    details
    • min: 47 characters
    • mean: 424.23 characters
    • max: 1441 characters
    • min: 37 characters
    • mean: 459.26 characters
    • max: 1583 characters
    • min: 0.0
    • mean: 0.53
    • max: 0.92
  • Samples:
    sentence1 sentence2 label
    [O] [=C] [C] [Branch2] [Ring2] [=Branch2] [O] [C] [O] [C] [Branch2] [Ring1] [Branch1] [C] [O] [C] [O] [C] [Branch1] [C] [C] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Ring1] [=Branch2] [O] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Ring2] [Ring1] [Ring2] [O] [=C] [Branch2] [Branch1] [O] [O] [C] [=C] [C] [Branch2] [Ring2] [Branch2] [O] [C] [O] [C] [Branch1] [Ring1] [C] [O] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Ring1] [#Branch2] [O] [C] [O] [C] [Branch1] [Ring1] [C] [O] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Ring1] [#Branch2] [O] [=C] [C] [Branch1] [C] [O] [=C] [Ring2] [Branch1] [Branch2] [Ring2] [Ring1] [=C] [C] [=C] [C] [=C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [=C] [Ring1] [Branch2] [S] [C] [C] [N] [C] [C] [C] [N] [C] [C] [Ring1] [#Branch2] 0.33
    [F] [C] [=Branch2] [Ring1] [C] [=C] [Branch1] [C] [F] [C] [C] [S] [C] [=N] [N] [=C] [Branch1] [Ring2] [S] [Ring1] [Branch1] [N] [F] [O] [=C] [Branch1] [#Branch2] [N] [C] [C] [S] [C] [=C] [C] [=Ring1] [Branch1] [C] [C] [C] [N] [Branch1] [C] [C] [C] [C] [C] [N] [Branch2] [Ring1] [Branch1] [C] [C] [=C] [C] [=C] [C] [Branch1] [#Branch2] [O] [C] [Branch1] [C] [F] [Branch1] [C] [F] [F] [=C] [Ring1] [O] [C] [Ring1] [P] [C] [=Branch1] [C] [=O] [N] [C] [Ring2] [Ring1] [Branch2] 0.49
    [O] [=C] [Branch2] [=Branch1] [N] [O] [C] [C] [O] [C] [Branch2] [Branch1] [#Branch1] [O] [C] [=C] [C] [=C] [Branch2] [Ring1] [Branch1] [O] [C] [O] [C] [Branch1] [Ring1] [C] [O] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Ring1] [#Branch2] [O] [C] [=C] [Branch1] [C] [O] [C] [=C] [Ring2] [Ring1] [Ring1] [OH1+1] [C] [Ring2] [Ring1] [#Branch1] [C] [=C] [C] [=C] [Branch1] [C] [O] [C] [=C] [Ring1] [#Branch1] [C] [Branch1] [C] [O] [C] [Branch1] [C] [O] [C] [Ring2] [Ring2] [#Branch1] [O] [C] [=C] [C] [=C] [C] [=C] [Branch1] [C] [O] [C] [Branch1] [Ring1] [O] [C] [=C] [Ring1] [=Branch2] [O] [C] [Branch1] [C] [C] [C] [C] [C] [C] [C] [Ring1] [Ring1] [C] [C] [Ring1] [=Branch2] [C] [C] [C] [Ring1] [Branch1] [Branch1] [C] [O] [C] 0.42
  • Loss: MatryoshkaLoss with these parameters:
    {
        "loss": "CosineSimilarityLoss",
        "matryoshka_dims": [
            320,
            160,
            80
        ],
        "matryoshka_weights": [
            1,
            1,
            1
        ],
        "n_dims_per_step": -1
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 2048
  • learning_rate: 0.05
  • lr_scheduler_type: cosine
  • warmup_steps: 0.1
  • bf16: True
  • per_device_eval_batch_size: 2048
  • load_best_model_at_end: True
  • seed: 12

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 2048
  • num_train_epochs: 3
  • max_steps: -1
  • learning_rate: 0.05
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.1
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 2048
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 12
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • dataloader_multiprocessing_context: None
  • dataloader_in_order: True
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • warmup_ratio: None

Training Logs

Click to expand
Epoch Step Training Loss Validation Loss molsim_spearman_cosine molsim-fulltest_spearman_cosine
-1 -1 - - 0.5371 -
0.0019 1 0.4891 - - -
0.0312 16 0.4880 - - -
0.0624 32 0.4098 - - -
0.0936 48 0.2228 - - -
0.1248 64 0.1083 - - -
0.1559 80 0.0858 - - -
0.1871 96 0.0715 - - -
0.2183 112 0.0621 - - -
0.2495 128 0.0568 - - -
0.2807 144 0.0528 - - -
0.3002 154 - 0.0509 0.7186 -
0.3119 160 0.0497 - - -
0.3431 176 0.0484 - - -
0.3743 192 0.0465 - - -
0.4055 208 0.0452 - - -
0.4366 224 0.0444 - - -
0.4678 240 0.0442 - - -
0.4990 256 0.0434 - - -
0.5302 272 0.0432 - - -
0.5614 288 0.0430 - - -
0.5926 304 0.0418 - - -
0.6004 308 - 0.0438 0.7342 -
0.6238 320 0.0424 - - -
0.6550 336 0.0412 - - -
0.6862 352 0.0414 - - -
0.7173 368 0.0422 - - -
0.7485 384 0.0418 - - -
0.7797 400 0.0417 - - -
0.8109 416 0.0419 - - -
0.8421 432 0.0417 - - -
0.8733 448 0.0414 - - -
0.9006 462 - 0.0426 0.7393 -
0.9045 464 0.0410 - - -
0.9357 480 0.0414 - - -
0.9669 496 0.0409 - - -
0.9981 512 0.0416 - - -
1.0292 528 0.0414 - - -
1.0604 544 0.0409 - - -
1.0916 560 0.0410 - - -
1.1228 576 0.0413 - - -
1.1540 592 0.0411 - - -
1.1852 608 0.0402 - - -
1.2008 616 - 0.0420 0.7388 -
1.2164 624 0.0418 - - -
1.2476 640 0.0408 - - -
1.2788 656 0.0406 - - -
1.3099 672 0.0410 - - -
1.3411 688 0.0404 - - -
1.3723 704 0.0404 - - -
1.4035 720 0.0405 - - -
1.4347 736 0.0406 - - -
1.4659 752 0.0410 - - -
1.4971 768 0.0409 - - -
1.5010 770 - 0.0417 0.7415 -
1.5283 784 0.0405 - - -
1.5595 800 0.0407 - - -
1.5906 816 0.0404 - - -
1.6218 832 0.0404 - - -
1.6530 848 0.0403 - - -
1.6842 864 0.0400 - - -
1.7154 880 0.0408 - - -
1.7466 896 0.0401 - - -
1.7778 912 0.0405 - - -
1.8012 924 - 0.0415 0.7413 -
1.8090 928 0.0407 - - -
1.8402 944 0.0404 - - -
1.8713 960 0.0404 - - -
1.9025 976 0.0405 - - -
1.9337 992 0.0407 - - -
1.9649 1008 0.0406 - - -
1.9961 1024 0.0402 - - -
2.0273 1040 0.0404 - - -
2.0585 1056 0.0399 - - -
2.0897 1072 0.0397 - - -
2.1014 1078 - 0.0415 0.7425 -
2.1209 1088 0.0405 - - -
2.1520 1104 0.0404 - - -
2.1832 1120 0.0399 - - -
2.2144 1136 0.0405 - - -
2.2456 1152 0.0403 - - -
2.2768 1168 0.0398 - - -
2.3080 1184 0.0402 - - -
2.3392 1200 0.0412 - - -
2.3704 1216 0.0402 - - -
2.4016 1232 0.0409 0.0414 0.7428 -
2.4327 1248 0.0403 - - -
2.4639 1264 0.0407 - - -
2.4951 1280 0.0406 - - -
2.5263 1296 0.0396 - - -
2.5575 1312 0.0399 - - -
2.5887 1328 0.0395 - - -
2.6199 1344 0.0405 - - -
2.6511 1360 0.0405 - - -
2.6823 1376 0.0407 - - -
2.7018 1386 - 0.0414 0.7427 -
2.7135 1392 0.0397 - - -
2.7446 1408 0.0403 - - -
2.7758 1424 0.0402 - - -
2.8070 1440 0.0400 - - -
2.8382 1456 0.0398 - - -
2.8694 1472 0.0400 - - -
2.9006 1488 0.0407 - - -
2.9318 1504 0.0402 - - -
2.9630 1520 0.0403 - - -
2.9942 1536 0.0401 - - -
3.0 1539 - 0.0414 0.7426 -
-1 -1 - - 0.7428 0.7519
  • The bold row denotes the saved checkpoint.

Training Time

  • Training: 14.4 minutes
  • Evaluation: 1.2 minutes
  • Total: 15.6 minutes

Framework Versions

  • Python: 3.12.12
  • Sentence Transformers: 6.0.1
  • Transformers: 5.16.1
  • PyTorch: 2.14.0+cu130
  • Accelerate: 1.14.0
  • Datasets: 5.0.1
  • Tokenizers: 0.23.2

Additional Resources

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
990k params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Papers for tomaarsen/static-chemselfies-molsim

Evaluation results