milhamapratama/agree-model-v1

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("milhamapratama/finetuned-agree-all-miniLM-L6-v2")
# Run inference
sentences = [
    'apa tindakan agree terhadap pengguna yang memberikan informasi palsu saat pendaftaran?',
    'jika pengguna dan/atau pelanggan memberikan informasi atau data pribadi yang tidak benar, tidak jelas, tidak akurat, atau tidak lengkap, agree berhak menolak permohonan pembuatan akun dan dapat menangguhkan atau menghentikan sebagian atau seluruh layanan yang diberikan.',
    'dengan layanan ini, pengguna dapat berkontribusi dalam menyebarkan pengetahuan dan wawasan terkait berbagai topik agrikultur kepada komunitas lainnya.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Dataset: val
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.8274
spearman_cosine	0.7265
pearson_manhattan	0.8054
spearman_manhattan	0.7261
pearson_euclidean	0.8059
spearman_euclidean	0.7265
pearson_dot	0.8274
spearman_dot	0.7265
pearson_max	0.8274
spearman_max	0.7265

Training Details

Training Dataset

Unnamed Dataset

• Size: 9,912 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 11 tokens mean: 26.33 tokens max: 49 tokens	min: 21 tokens mean: 88.24 tokens max: 256 tokens	min: 0.0 mean: 0.7 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  bagaimana cara menambahkan nama produk di bagian overview produk?	pada bagian "overview produk" di halaman pertama formulir tambah traceability story, pengguna dapat mengisi nama produk yang akan digunakan sebagai judul utama pada halaman traceability story. nama produk ini berperan penting dalam memberikan identitas yang jelas terhadap produk yang didaftarkan dalam sistem.	1.0
  apakah ada batasan usia atau latar belakang tertentu untuk menjadi agree expert?	apabila pengguna ingin menjadi expert, pengguna harus memiliki keahlian dan pengalaman di bidang agrikultur serta mengisi formulir pendaftaran dengan informasi yang relevan untuk mendukung proses validasi.	1.0
  apakah perubahan dalam pusat resolusi akan berdampak pada cara mengajukan keluhan?	ketentuan pusat resolusi pada platform agree poin kedua: dalam hal adanya penambahan, pengurangan dan/atau perubahan channel pengaduan akan diinformasikan kemudian melalui dashboard atau situs agree.	1.0

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 4
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss	val_spearman_max
0.0806	50	-	0.5255
0.1613	100	-	0.5470
0.2419	150	-	0.5847
0.3226	200	-	0.6196
0.4032	250	-	0.6456
0.4839	300	-	0.6653
0.5645	350	-	0.6740
0.6452	400	-	0.6750
0.7258	450	-	0.6746
0.8065	500	0.1211	0.6870
0.8871	550	-	0.6942
0.9677	600	-	0.6963
1.0	620	-	0.7018
1.0484	650	-	0.7001
1.1290	700	-	0.7047
1.2097	750	-	0.7070
1.2903	800	-	0.7044
1.3710	850	-	0.7096
1.4516	900	-	0.7065
1.5323	950	-	0.7014
1.6129	1000	0.0848	0.7082
1.6935	1050	-	0.7148
1.7742	1100	-	0.7175
1.8548	1150	-	0.7142
1.9355	1200	-	0.7126
2.0	1240	-	0.7158
2.0161	1250	-	0.7154
2.0968	1300	-	0.7137
2.1774	1350	-	0.7142
2.2581	1400	-	0.7218
2.3387	1450	-	0.7203
2.4194	1500	0.0709	0.7205
2.5	1550	-	0.7194
2.5806	1600	-	0.7175
2.6613	1650	-	0.7194
2.7419	1700	-	0.7185
2.8226	1750	-	0.7195
2.9032	1800	-	0.7241
2.9839	1850	-	0.7230
3.0	1860	-	0.7234
3.0645	1900	-	0.7251
3.1452	1950	-	0.7260
3.2258	2000	0.0614	0.7243
3.3065	2050	-	0.7257
3.3871	2100	-	0.7264
3.4677	2150	-	0.7248
3.5484	2200	-	0.7255
3.6290	2250	-	0.7245
3.7097	2300	-	0.7263
3.7903	2350	-	0.7262
3.8710	2400	-	0.7265
3.9516	2450	-	0.7265
4.0	2480	-	0.7265

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.1.1
• Transformers: 4.45.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}