the-token-architects/distilbert-triplet

SentenceTransformer based on distilbert/distilbert-base-uncased

This is a sentence-transformers model finetuned from distilbert/distilbert-base-uncased. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'DistilBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("the-token-architects/distilbert-triplet")
# Run inference
sentences = [
    'where is the name jason in the bible',
    'Jason of Thessalonica was a Jewish convert and early Christian believer mentioned in the New Testament in Acts 17:5-9 and Romans 16:21. According to tradition, Jason is numbered among the Seventy Disciples.[citation needed]',
    'The literary source (hagiographic legend) of the life of Jason and Sosipater was newly edited and translated by B. Kindt as appendix to "La version longue du récit légendaire de l\'évangelisation de Corfou par les saints Jason and Sosipatre," Analecta Bollandiana 116 (1998) 259–295.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.7319, -0.7970],
#         [ 0.7319,  1.0000, -0.4059],
#         [-0.7970, -0.4059,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 203,501 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 9 tokens mean: 11.58 tokens max: 18 tokens	min: 18 tokens mean: 140.37 tokens max: 312 tokens	min: 2 tokens mean: 105.06 tokens max: 450 tokens

• Samples:

  anchor	positive	negative
  when did the smithsonian air and space museum open	The National Air and Space Museum of the Smithsonian Institution, also called the NASM, is a museum in Washington, D.C.. It was established in 1946 as the National Air Museum and opened its main building on the National Mall near L'Enfant Plaza in 1976. In 2016, the museum saw approximately 7.5 million visitors, making it the second most visited museum in the world, and the most visited museum in the United States. The museum contains the Apollo 11 command module, the Friendship 7 capsule which was flown by John Glenn, Charles Lindbergh's Spirit of St. Louis, the Bell X-1 which broke the sound barrier, and the Wright brothers' plane near the entrance.	The National Air and Space Museum is a center for research into the history and science of aviation and spaceflight, as well as planetary science and terrestrial geology and geophysics. Almost all space and aircraft on display are originals or the original backup craft. It operates an annex, the Steven F. Udvar-Hazy Center, at Dulles International Airport, which opened in 2003 and itself encompasses 760,000 square feet (71,000 m2). The museum currently conducts restoration of its collection at the Paul E. Garber Preservation, Restoration, and Storage Facility in Suitland, Maryland, while steadily moving such restoration and archival activities into the Mary Baker Engen Restoration Hangar, a part of the Udvar-Hazy annex facilities as of 2014[update].
  when did the smithsonian air and space museum open	The National Air and Space Museum of the Smithsonian Institution, also called the NASM, is a museum in Washington, D.C.. It was established in 1946 as the National Air Museum and opened its main building on the National Mall near L'Enfant Plaza in 1976. In 2016, the museum saw approximately 7.5 million visitors, making it the second most visited museum in the world, and the most visited museum in the United States. The museum contains the Apollo 11 command module, the Friendship 7 capsule which was flown by John Glenn, Charles Lindbergh's Spirit of St. Louis, the Bell X-1 which broke the sound barrier, and the Wright brothers' plane near the entrance.	Because of the museum's close proximity to the United States Capitol, the Smithsonian wanted a building that would be architecturally impressive but would not stand out too boldly against the Capitol building. St. Louis-based architect Gyo Obata of HOK designed the museum as four simple marble-encased cubes containing the smaller and more theatrical exhibits, connected by three spacious steel-and-glass atria which house the larger exhibits such as missiles, airplanes and spacecraft. The mass of the museum is similar to the National Gallery of Art across the National Mall, and uses the same pink Tennessee marble as the National Gallery. Built by Gilbane Building Company, the museum was completed in 1976. The west glass wall of the building is used for the installation of airplanes, functioning as a giant door.
  when did the smithsonian air and space museum open	The National Air and Space Museum of the Smithsonian Institution, also called the NASM, is a museum in Washington, D.C.. It was established in 1946 as the National Air Museum and opened its main building on the National Mall near L'Enfant Plaza in 1976. In 2016, the museum saw approximately 7.5 million visitors, making it the second most visited museum in the world, and the most visited museum in the United States. The museum contains the Apollo 11 command module, the Friendship 7 capsule which was flown by John Glenn, Charles Lindbergh's Spirit of St. Louis, the Bell X-1 which broke the sound barrier, and the Wright brothers' plane near the entrance.	The museum's prominent site on the National Mall once housed the city's armory, and during the Civil War, Armory Square Hospital nursed the worst wounded cases who were transported to Washington after battles.

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 1.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 1
• learning_rate: 2e-05
• fp16: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 8
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0039	100	0.7963
0.0079	200	0.5151
0.0118	300	0.4567
0.0157	400	0.3929
0.0197	500	0.3726
0.0236	600	0.3938
0.0275	700	0.3697
0.0314	800	0.3358
0.0354	900	0.3083
0.0393	1000	0.2984
0.0432	1100	0.3126
0.0472	1200	0.2916
0.0511	1300	0.3006
0.0550	1400	0.2746
0.0590	1500	0.2847
0.0629	1600	0.2831
0.0668	1700	0.2435
0.0708	1800	0.2566
0.0747	1900	0.2568
0.0786	2000	0.2429
0.0826	2100	0.2390
0.0865	2200	0.2295
0.0904	2300	0.2053
0.0943	2400	0.2129
0.0983	2500	0.2127
0.1022	2600	0.1969
0.1061	2700	0.2294
0.1101	2800	0.2120
0.1140	2900	0.2013
0.1179	3000	0.2139
0.1219	3100	0.1722
0.1258	3200	0.1939
0.1297	3300	0.1838
0.1337	3400	0.1529
0.1376	3500	0.1794
0.1415	3600	0.1831
0.1455	3700	0.1703
0.1494	3800	0.1777
0.1533	3900	0.1582
0.1572	4000	0.1360
0.1612	4100	0.1373
0.1651	4200	0.1717
0.1690	4300	0.1327
0.1730	4400	0.1519
0.1769	4500	0.1420
0.1808	4600	0.1175
0.1848	4700	0.1487
0.1887	4800	0.1240
0.1926	4900	0.1369
0.1966	5000	0.1393
0.2005	5100	0.1142
0.2044	5200	0.1332
0.2083	5300	0.1250
0.2123	5400	0.1285
0.2162	5500	0.0995
0.2201	5600	0.1290
0.2241	5700	0.1239
0.2280	5800	0.1071
0.2319	5900	0.1242
0.2359	6000	0.1063
0.2398	6100	0.0986
0.2437	6200	0.1068
0.2477	6300	0.1310
0.2516	6400	0.1102
0.2555	6500	0.0966
0.2595	6600	0.1202
0.2634	6700	0.1204
0.2673	6800	0.1274
0.2712	6900	0.1076
0.2752	7000	0.0954
0.2791	7100	0.0979
0.2830	7200	0.1062
0.2870	7300	0.0929
0.2909	7400	0.1219
0.2948	7500	0.1094
0.2988	7600	0.0779
0.3027	7700	0.0698
0.3066	7800	0.1020
0.3106	7900	0.1016
0.3145	8000	0.1008
0.3184	8100	0.0921
0.3224	8200	0.0877
0.3263	8300	0.0923
0.3302	8400	0.0749
0.3341	8500	0.1089
0.3381	8600	0.1027
0.3420	8700	0.0987
0.3459	8800	0.0853
0.3499	8900	0.0877
0.3538	9000	0.0927
0.3577	9100	0.0872
0.3617	9200	0.0647
0.3656	9300	0.0673
0.3695	9400	0.0693
0.3735	9500	0.0863
0.3774	9600	0.0689
0.3813	9700	0.0710
0.3853	9800	0.0615
0.3892	9900	0.0910
0.3931	10000	0.0754
0.3970	10100	0.0593
0.4010	10200	0.0639
0.4049	10300	0.0657
0.4088	10400	0.0600
0.4128	10500	0.0675
0.4167	10600	0.0674
0.4206	10700	0.0764
0.4246	10800	0.0895
0.4285	10900	0.0647
0.4324	11000	0.0566
0.4364	11100	0.0621
0.4403	11200	0.0666
0.4442	11300	0.0728
0.4481	11400	0.0566
0.4521	11500	0.0698
0.4560	11600	0.0688
0.4599	11700	0.0626
0.4639	11800	0.0464
0.4678	11900	0.0627
0.4717	12000	0.0534
0.4757	12100	0.0535
0.4796	12200	0.0542
0.4835	12300	0.0590
0.4875	12400	0.0644
0.4914	12500	0.0512
0.4953	12600	0.0572
0.4993	12700	0.0537
0.5032	12800	0.0568
0.5071	12900	0.0512
0.5110	13000	0.0636
0.5150	13100	0.0361
0.5189	13200	0.0528
0.5228	13300	0.0473
0.5268	13400	0.0569
0.5307	13500	0.0625
0.5346	13600	0.0590
0.5386	13700	0.0396
0.5425	13800	0.0537
0.5464	13900	0.0418
0.5504	14000	0.0338
0.5543	14100	0.0527
0.5582	14200	0.0483
0.5622	14300	0.0531
0.5661	14400	0.0594
0.5700	14500	0.0482
0.5739	14600	0.0528
0.5779	14700	0.0464
0.5818	14800	0.0728
0.5857	14900	0.0457
0.5897	15000	0.0379
0.5936	15100	0.0552
0.5975	15200	0.0574
0.6015	15300	0.0427
0.6054	15400	0.0457
0.6093	15500	0.0423
0.6133	15600	0.0572
0.6172	15700	0.0467
0.6211	15800	0.0429
0.6250	15900	0.0502
0.6290	16000	0.0412
0.6329	16100	0.0512
0.6368	16200	0.0349
0.6408	16300	0.0438
0.6447	16400	0.0489
0.6486	16500	0.0482
0.6526	16600	0.0491
0.6565	16700	0.0339
0.6604	16800	0.0369
0.6644	16900	0.0405
0.6683	17000	0.0393
0.6722	17100	0.0433
0.6762	17200	0.0557
0.6801	17300	0.0353
0.6840	17400	0.0371
0.6879	17500	0.0325
0.6919	17600	0.0451
0.6958	17700	0.0375
0.6997	17800	0.0439
0.7037	17900	0.0416
0.7076	18000	0.0363
0.7115	18100	0.0267
0.7155	18200	0.0362
0.7194	18300	0.0410
0.7233	18400	0.0300
0.7273	18500	0.0411
0.7312	18600	0.0424
0.7351	18700	0.0310
0.7391	18800	0.0441
0.7430	18900	0.0374
0.7469	19000	0.0426
0.7508	19100	0.0438
0.7548	19200	0.0296
0.7587	19300	0.0293
0.7626	19400	0.0279
0.7666	19500	0.0337
0.7705	19600	0.0381
0.7744	19700	0.0350
0.7784	19800	0.0300
0.7823	19900	0.0425
0.7862	20000	0.0384
0.7902	20100	0.0209
0.7941	20200	0.0361
0.7980	20300	0.0312
0.8019	20400	0.0329
0.8059	20500	0.0347
0.8098	20600	0.0366
0.8137	20700	0.0319
0.8177	20800	0.0353
0.8216	20900	0.0383
0.8255	21000	0.0360
0.8295	21100	0.0260
0.8334	21200	0.0327
0.8373	21300	0.0357
0.8413	21400	0.0363
0.8452	21500	0.0366
0.8491	21600	0.0312
0.8531	21700	0.0330
0.8570	21800	0.0293
0.8609	21900	0.0135
0.8648	22000	0.0369
0.8688	22100	0.0254
0.8727	22200	0.0311
0.8766	22300	0.0301
0.8806	22400	0.0354
0.8845	22500	0.0311
0.8884	22600	0.0373
0.8924	22700	0.0288
0.8963	22800	0.0327
0.9002	22900	0.0435
0.9042	23000	0.0223
0.9081	23100	0.0351
0.9120	23200	0.0252
0.9160	23300	0.0329
0.9199	23400	0.0436
0.9238	23500	0.0253
0.9277	23600	0.0268
0.9317	23700	0.0195
0.9356	23800	0.0368
0.9395	23900	0.0266
0.9435	24000	0.0389
0.9474	24100	0.0298
0.9513	24200	0.0242
0.9553	24300	0.0308
0.9592	24400	0.0322
0.9631	24500	0.0205
0.9671	24600	0.0333
0.9710	24700	0.0198
0.9749	24800	0.0219
0.9789	24900	0.0293
0.9828	25000	0.0301
0.9867	25100	0.0211
0.9906	25200	0.0285
0.9946	25300	0.0300
0.9985	25400	0.0241

Training Time

• Training: 59.2 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.7.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}