swardiantara/bert-tiny-snli-k10-adaptive-cosine

SentenceTransformer based on google/bert_uncased_L-2_H-128_A-2

This is a sentence-transformers model finetuned from google/bert_uncased_L-2_H-128_A-2. It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/bert_uncased_L-2_H-128_A-2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 128, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("swardiantara/bert-tiny-snli-k10-adaptive-cosine")
# Run inference
sentences = [
    'The school is having a special event in order to show the american culture on how other cultures are dealt with in parties. [SEP] A school is hosting an event.',
    'A large, black dog is running in the sand on the beach. [SEP] The dog is chasing a Frisbee.',
    'A group of people walking down the street with one on the phone in a green hat. [SEP] People are outside',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3961, 0.5340],
#         [0.3961, 1.0000, 0.0176],
#         [0.5340, 0.0176, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 11,537,277 training samples
• Columns: text_a, text_b, and label
• Approximate statistics based on the first 100 samples:

  	text_a	text_b	label
  type	string	string	list
  modality	text	text
  details	min: 13 tokens mean: 21.08 tokens max: 28 tokens	min: 20 tokens mean: 26.1 tokens max: 34 tokens	size: 2 elements

• Samples:

  text_a	text_b	label
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	The man is on a black and white bike. [SEP] The man is going to be in a bicycle race.	[1.0, 0.0]
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A man, on a basketball team, dunking the ball through the hoop. [SEP] The man is playing basketball.	[0.0, 0.5]
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A young man wearing blue jeans and a t-shirt sits in the grass, with a ball in the air. [SEP] The young man is throwing a ball.	[0.0, 0.5]

• Loss: main.OrdinalProxyContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1024
• num_train_epochs: 10
• learning_rate: 2e-05
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 1024
• num_train_epochs: 10
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0444	500	0.0429
0.0888	1000	0.0262
0.1331	1500	0.0248
0.1775	2000	0.0241
0.2219	2500	0.0235
0.2663	3000	0.0224
0.3106	3500	0.0203
0.3550	4000	0.0188
0.3994	4500	0.0180
0.4438	5000	0.0174
0.4882	5500	0.0167
0.5325	6000	0.0162
0.5769	6500	0.0158
0.6213	7000	0.0155
0.6657	7500	0.0152
0.7100	8000	0.0148
0.7544	8500	0.0146
0.7988	9000	0.0144
0.8432	9500	0.0140
0.8875	10000	0.0138
0.9319	10500	0.0135
0.9763	11000	0.0136
1.0	11267	-
1.0207	11500	0.0134
1.0651	12000	0.0130
1.1094	12500	0.0130
1.1538	13000	0.0128
1.1982	13500	0.0126
1.2426	14000	0.0124
1.2869	14500	0.0124
1.3313	15000	0.0121
1.3757	15500	0.0121
1.4201	16000	0.0120
1.4645	16500	0.0119
1.5088	17000	0.0117
1.5532	17500	0.0119
1.5976	18000	0.0116
1.6420	18500	0.0115
1.6863	19000	0.0116
1.7307	19500	0.0113
1.7751	20000	0.0113
1.8195	20500	0.0112
1.8639	21000	0.0113
1.9082	21500	0.0111
1.9526	22000	0.0110
1.9970	22500	0.0111
2.0	22534	-
2.0414	23000	0.0107
2.0857	23500	0.0108
2.1301	24000	0.0108
2.1745	24500	0.0105
2.2189	25000	0.0105
2.2632	25500	0.0104
2.3076	26000	0.0104
2.3520	26500	0.0104
2.3964	27000	0.0104
2.4408	27500	0.0103
2.4851	28000	0.0102
2.5295	28500	0.0102
2.5739	29000	0.0102
2.6183	29500	0.0100
2.6626	30000	0.0100
2.7070	30500	0.0099
2.7514	31000	0.0100
2.7958	31500	0.0100
2.8402	32000	0.0098
2.8845	32500	0.0098
2.9289	33000	0.0098
2.9733	33500	0.0098
3.0	33801	-
3.0177	34000	0.0098
3.0620	34500	0.0096
3.1064	35000	0.0096
3.1508	35500	0.0095
3.1952	36000	0.0095
3.2395	36500	0.0095
3.2839	37000	0.0095
3.3283	37500	0.0094
3.3727	38000	0.0093
3.4171	38500	0.0093
3.4614	39000	0.0094
3.5058	39500	0.0093
3.5502	40000	0.0094
3.5946	40500	0.0093
3.6389	41000	0.0092
3.6833	41500	0.0093
3.7277	42000	0.0092
3.7721	42500	0.0092
3.8165	43000	0.0091
3.8608	43500	0.0090
3.9052	44000	0.0091
3.9496	44500	0.0091
3.9940	45000	0.0090
4.0	45068	-
4.0383	45500	0.0090
4.0827	46000	0.0089
4.1271	46500	0.0089
4.1715	47000	0.0088
4.2159	47500	0.0089
4.2602	48000	0.0088
4.3046	48500	0.0089
4.3490	49000	0.0088
4.3934	49500	0.0088
4.4377	50000	0.0088
4.4821	50500	0.0088
4.5265	51000	0.0088
4.5709	51500	0.0088
4.6152	52000	0.0087
4.6596	52500	0.0087
4.7040	53000	0.0087
4.7484	53500	0.0086
4.7928	54000	0.0087
4.8371	54500	0.0086
4.8815	55000	0.0086
4.9259	55500	0.0086
4.9703	56000	0.0086
5.0	56335	-
5.0146	56500	0.0085
5.0590	57000	0.0086
5.1034	57500	0.0086
5.1478	58000	0.0083
5.1922	58500	0.0084
5.2365	59000	0.0085
5.2809	59500	0.0084
5.3253	60000	0.0084
5.3697	60500	0.0084
5.4140	61000	0.0084
5.4584	61500	0.0084
5.5028	62000	0.0084
5.5472	62500	0.0084
5.5916	63000	0.0083
5.6359	63500	0.0083
5.6803	64000	0.0082
5.7247	64500	0.0084
5.7691	65000	0.0083
5.8134	65500	0.0084
5.8578	66000	0.0083
5.9022	66500	0.0083
5.9466	67000	0.0084
5.9909	67500	0.0083
6.0	67602	-
6.0353	68000	0.0082
6.0797	68500	0.0082
6.1241	69000	0.0081
6.1685	69500	0.0081
6.2128	70000	0.0081
6.2572	70500	0.0081
6.3016	71000	0.0082
6.3460	71500	0.0083
6.3903	72000	0.0081
6.4347	72500	0.0082
6.4791	73000	0.0081
6.5235	73500	0.0082
6.5679	74000	0.0081
6.6122	74500	0.0081
6.6566	75000	0.0081
6.7010	75500	0.0081
6.7454	76000	0.0082
6.7897	76500	0.0082
6.8341	77000	0.0082
6.8785	77500	0.0080
6.9229	78000	0.0081
6.9672	78500	0.0080
7.0	78869	-
7.0116	79000	0.0080
7.0560	79500	0.0080
7.1004	80000	0.0080
7.1448	80500	0.0079
7.1891	81000	0.0079
7.2335	81500	0.0080
7.2779	82000	0.0079
7.3223	82500	0.0079
7.3666	83000	0.0078
7.4110	83500	0.0080
7.4554	84000	0.0079
7.4998	84500	0.0079
7.5442	85000	0.0079
7.5885	85500	0.0079
7.6329	86000	0.0079
7.6773	86500	0.0079
7.7217	87000	0.0079
7.7660	87500	0.0080
7.8104	88000	0.0078
7.8548	88500	0.0079
7.8992	89000	0.0079
7.9436	89500	0.0079
7.9879	90000	0.0079
8.0	90136	-
8.0323	90500	0.0078
8.0767	91000	0.0078
8.1211	91500	0.0077
8.1654	92000	0.0078
8.2098	92500	0.0079
8.2542	93000	0.0078
8.2986	93500	0.0078
8.3429	94000	0.0078
8.3873	94500	0.0077
8.4317	95000	0.0078
8.4761	95500	0.0078
8.5205	96000	0.0079
8.5648	96500	0.0078
8.6092	97000	0.0078
8.6536	97500	0.0077
8.6980	98000	0.0077
8.7423	98500	0.0078
8.7867	99000	0.0078
8.8311	99500	0.0078
8.8755	100000	0.0078
8.9199	100500	0.0079
8.9642	101000	0.0077
9.0	101403	-
9.0086	101500	0.0077
9.0530	102000	0.0078
9.0974	102500	0.0077
9.1417	103000	0.0078
9.1861	103500	0.0077
9.2305	104000	0.0078
9.2749	104500	0.0077
9.3193	105000	0.0076
9.3636	105500	0.0076
9.4080	106000	0.0077
9.4524	106500	0.0077
9.4968	107000	0.0078
9.5411	107500	0.0077
9.5855	108000	0.0077
9.6299	108500	0.0077
9.6743	109000	0.0077
9.7186	109500	0.0078
9.7630	110000	0.0078
9.8074	110500	0.0078
9.8518	111000	0.0077
9.8962	111500	0.0078
9.9405	112000	0.0077
9.9849	112500	0.0077
10.0	112670	-

• The bold row denotes the saved checkpoint.

Training Time

• Training: 4.9 hours
• Evaluation: 1.0 minutes
• Total: 4.9 hours

Framework Versions

• Python: 3.12.4
• Sentence Transformers: 5.5.1
• Transformers: 5.11.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.13.0
• Datasets: 2.21.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}