Upload folder using huggingface_hub

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -0,0 +1,544 @@
+---
+language: []
+library_name: sentence-transformers
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- generated_from_trainer
+- dataset_size:362208
+- loss:ContrastiveLoss
+base_model: vinai/phobert-base-v2
+datasets: []
+widget:
+- source_sentence: vùng khí_hậu nóng ẩm ghi_nhận sự tồn_tại của Virus SARS-CoV -2
+    .
+  sentences:
+  - vùng khí_hậu nóng ẩm ghi_nhận sự tồn_tại của Virus SARS-CoV -2 .
+  - Chế_độ ăn_uống nghỉ_ngơi như_thế_nào là hợp_lý ?
+  - Và bé đang có tình_trạng bị đọng cặn nước_tiểu ở đầu bộ sinh_dục ( bé trai ) .
+- source_sentence: Khoan xương và chèn vào hai bên hai bong_bóng được gọi là đệm xương
+    .
+  sentences:
+  - Khoan xương và chèn vào hai bên hai bong_bóng được gọi là đệm xương .
+  - 3 hôm_nay bé đi phân chủ_yếu là nhầy có bọt rồi nước lẫn hoa_cà_hoa_cải , thi_thoảng
+    phân có màu xanh đậm .
+  - Sau khi chẩn_đoán , bác_sĩ khuyên anh Khánh nên sớm điều_trị kịp_thời để tránh
+    nhiều biến_chứng .
+- source_sentence: Những phụ_nữ có tiền_sử bệnh như trên chính là những đối_tượng
+    nguy_cơ của tình_trạng tắc ống dẫn trứng .
+  sentences:
+  - 'Dùng các thiết_bị hỗ_trợ quá_trình di_chuyển đồng_thời giúp cải_thiện chức_năng
+    của các khớp như :'
+  - Những phụ_nữ có tiền_sử bệnh như trên chính là những đối_tượng nguy_cơ của tình_trạng
+    tắc ống dẫn trứng .
+  - Trong các nguyên_nhân sau đây , đâu là các nguyên_nhân khách_quan , không đến
+    từ mẹ và thai_nhi ?
+- source_sentence: Bé nhà con nay được 1 tháng 23 ngày .
+  sentences:
+  - 'Vú phải : - bất đối_xứng ở vùng dưới ( kích_thước :'
+  - Thưa bác_sĩ tôi 18 tuổi , bị sưng chân răng , nhức tai , nhức đầu với có kêu tiếng
+    trong quai_hàm .
+  - Bé nhà con nay được 1 tháng 23 ngày .
+- source_sentence: Tuy_nhiên , nếu bệnh không tự lành và vẫn tiếp_tục chảy_máu , cần
+    phải sử_dụng các liệu_pháp cầm máu để bù lại lượng máu đã mất .
+  sentences:
+  - Nguyễn_Thị_Thanh_Tuyền ( 1995 ) .
+  - Bệnh_viện Bệnh Nhiệt_đới Trung_ương cơ_sở Kim_Chung là bệnh_viện khám_chữa bệnh
+    đa_khoa phục_vụ cho người_dân trong cả nước .
+  - 'Một_số yếu_tố làm tăng nguy_cơ mắc bệnh như : Yếu_tố nội_tiết : bệnh thường gặp
+    ở phụ_nữ chậm có kinh và sớm mãn_kinh .'
+pipeline_tag: sentence-similarity
+---
+
+# SentenceTransformer based on vinai/phobert-base-v2
+
+This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [vinai/phobert-base-v2](https://huggingface.co/vinai/phobert-base-v2). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+
+## Model Details
+
+### Model Description
+- **Model Type:** Sentence Transformer
+- **Base model:** [vinai/phobert-base-v2](https://huggingface.co/vinai/phobert-base-v2) <!-- at revision 2b51e367d92093c9688112098510e6a58bab67cd -->
+- **Maximum Sequence Length:** 256 tokens
+- **Output Dimensionality:** 768 tokens
+- **Similarity Function:** Cosine Similarity
+<!-- - **Training Dataset:** Unknown -->
+<!-- - **Language:** Unknown -->
+<!-- - **License:** Unknown -->
+
+### Model Sources
+
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
+
+### Full Model Architecture
+
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: RobertaModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+)
+```
+
+## Usage
+
+### Direct Usage (Sentence Transformers)
+
+First install the Sentence Transformers library:
+
+```bash
+pip install -U sentence-transformers
+```
+
+Then you can load this model and run inference.
+```python
+from sentence_transformers import SentenceTransformer
+
+# Download from the 🤗 Hub
+model = SentenceTransformer("sentence_transformers_model_id")
+# Run inference
+sentences = [
+    'Tuy_nhiên , nếu bệnh không tự lành và vẫn tiếp_tục chảy_máu , cần phải sử_dụng các liệu_pháp cầm máu để bù lại lượng máu đã mất .',
+    'Một_số yếu_tố làm tăng nguy_cơ mắc bệnh như : Yếu_tố nội_tiết : bệnh thường gặp ở phụ_nữ chậm có kinh và sớm mãn_kinh .',
+    'Nguyễn_Thị_Thanh_Tuyền ( 1995 ) .',
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 768]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities.shape)
+# [3, 3]
+```
+
+<!--
+### Direct Usage (Transformers)
+
+<details><summary>Click to see the direct usage in Transformers</summary>
+
+</details>
+-->
+
+<!--
+### Downstream Usage (Sentence Transformers)
+
+You can finetune this model on your own dataset.
+
+<details><summary>Click to expand</summary>
+
+</details>
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Dataset
+
+#### Unnamed Dataset
+
+
+* Size: 362,208 training samples
+* Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>label</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | sentence_0                                                                         | sentence_1                                                                         | label                                                          |
+  |:--------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------|
+  | type    | string                                                                             | string                                                                             | float                                                          |
+  | details | <ul><li>min: 3 tokens</li><li>mean: 22.64 tokens</li><li>max: 104 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 23.25 tokens</li><li>max: 222 tokens</li></ul> | <ul><li>min: 0.1</li><li>mean: 0.82</li><li>max: 1.0</li></ul> |
+* Samples:
+  | sentence_0                                                                                                                                                                      | sentence_1                                                                                                                                           | label                           |
+  |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------|
+  | <code>Hiệu_lực của vaccine AstraZeneca ra sao ?</code>                                                                                                                          | <code>Hiệu_lực của vaccine AstraZeneca ra sao ?</code>                                                                                               | <code>1.0</code>                |
+  | <code>Gần đây , tôi có quen một bạn gái , mỗi lần ngồi gần nhau có cử_chỉ thân_mật thì tôi gần như không kìm chế được có_thể nói là giống như hiện_tượng xuất_tinh sớm .</code> | <code>Chụp CT scanner sọ não : là hình_ảnh tốt nhất để đánh_giá tổn_thương não vì có_thể hiển_thị mô não hoặc xuất_huyết não hoặc nhũn_não .</code> | <code>0.6540138125419617</code> |
+  | <code>Sốt siêu_vi sau quan_hệ tình_dục không an_toàn có phải đã nhiễm HIV không ?</code>                                                                                        | <code>Sốt siêu_vi sau quan_hệ tình_dục không an_toàn có phải đã nhiễm HIV không ?</code>                                                             | <code>1.0</code>                |
+* Loss: [<code>ContrastiveLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#contrastiveloss) with these parameters:
+  ```json
+  {
+      "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
+      "margin": 0.5,
+      "size_average": true
+  }
+  ```
+
+### Training Hyperparameters
+#### Non-Default Hyperparameters
+
+- `per_device_train_batch_size`: 16
+- `per_device_eval_batch_size`: 16
+- `num_train_epochs`: 4
+- `multi_dataset_batch_sampler`: round_robin
+
+#### All Hyperparameters
+<details><summary>Click to expand</summary>
+
+- `overwrite_output_dir`: False
+- `do_predict`: False
+- `prediction_loss_only`: True
+- `per_device_train_batch_size`: 16
+- `per_device_eval_batch_size`: 16
+- `per_gpu_train_batch_size`: None
+- `per_gpu_eval_batch_size`: None
+- `gradient_accumulation_steps`: 1
+- `eval_accumulation_steps`: None
+- `learning_rate`: 5e-05
+- `weight_decay`: 0.0
+- `adam_beta1`: 0.9
+- `adam_beta2`: 0.999
+- `adam_epsilon`: 1e-08
+- `max_grad_norm`: 1
+- `num_train_epochs`: 4
+- `max_steps`: -1
+- `lr_scheduler_type`: linear
+- `lr_scheduler_kwargs`: {}
+- `warmup_ratio`: 0.0
+- `warmup_steps`: 0
+- `log_level`: passive
+- `log_level_replica`: warning
+- `log_on_each_node`: True
+- `logging_nan_inf_filter`: True
+- `save_safetensors`: True
+- `save_on_each_node`: False
+- `save_only_model`: False
+- `no_cuda`: False
+- `use_cpu`: False
+- `use_mps_device`: False
+- `seed`: 42
+- `data_seed`: None
+- `jit_mode_eval`: False
+- `use_ipex`: False
+- `bf16`: False
+- `fp16`: False
+- `fp16_opt_level`: O1
+- `half_precision_backend`: auto
+- `bf16_full_eval`: False
+- `fp16_full_eval`: False
+- `tf32`: None
+- `local_rank`: 0
+- `ddp_backend`: None
+- `tpu_num_cores`: None
+- `tpu_metrics_debug`: False
+- `debug`: []
+- `dataloader_drop_last`: False
+- `dataloader_num_workers`: 0
+- `dataloader_prefetch_factor`: None
+- `past_index`: -1
+- `disable_tqdm`: False
+- `remove_unused_columns`: True
+- `label_names`: None
+- `load_best_model_at_end`: False
+- `ignore_data_skip`: False
+- `fsdp`: []
+- `fsdp_min_num_params`: 0
+- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
+- `fsdp_transformer_layer_cls_to_wrap`: None
+- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}
+- `deepspeed`: None
+- `label_smoothing_factor`: 0.0
+- `optim`: adamw_torch
+- `optim_args`: None
+- `adafactor`: False
+- `group_by_length`: False
+- `length_column_name`: length
+- `ddp_find_unused_parameters`: None
+- `ddp_bucket_cap_mb`: None
+- `ddp_broadcast_buffers`: False
+- `dataloader_pin_memory`: True
+- `dataloader_persistent_workers`: False
+- `skip_memory_metrics`: True
+- `use_legacy_prediction_loop`: False
+- `push_to_hub`: False
+- `resume_from_checkpoint`: None
+- `hub_model_id`: None
+- `hub_strategy`: every_save
+- `hub_private_repo`: False
+- `hub_always_push`: False
+- `gradient_checkpointing`: False
+- `gradient_checkpointing_kwargs`: None
+- `include_inputs_for_metrics`: False
+- `fp16_backend`: auto
+- `push_to_hub_model_id`: None
+- `push_to_hub_organization`: None
+- `mp_parameters`: 
+- `auto_find_batch_size`: False
+- `full_determinism`: False
+- `torchdynamo`: None
+- `ray_scope`: last
+- `ddp_timeout`: 1800
+- `torch_compile`: False
+- `torch_compile_backend`: None
+- `torch_compile_mode`: None
+- `dispatch_batches`: None
+- `split_batches`: None
+- `include_tokens_per_second`: False
+- `include_num_input_tokens_seen`: False
+- `neftune_noise_alpha`: None
+- `optim_target_modules`: None
+- `batch_sampler`: batch_sampler
+- `multi_dataset_batch_sampler`: round_robin
+
+</details>
+
+### Training Logs
+<details><summary>Click to expand</summary>
+
+| Epoch  | Step  | Training Loss |
+|:------:|:-----:|:-------------:|
+| 0.0221 | 500   | 0.0168        |
+| 0.0442 | 1000  | 0.0139        |
+| 0.0663 | 1500  | 0.0142        |
+| 0.0883 | 2000  | 0.0139        |
+| 0.1104 | 2500  | 0.0137        |
+| 0.1325 | 3000  | 0.0139        |
+| 0.1546 | 3500  | 0.0137        |
+| 0.1767 | 4000  | 0.0139        |
+| 0.1988 | 4500  | 0.0136        |
+| 0.2209 | 5000  | 0.0135        |
+| 0.2430 | 5500  | 0.0137        |
+| 0.2650 | 6000  | 0.0138        |
+| 0.2871 | 6500  | 0.0136        |
+| 0.3092 | 7000  | 0.0137        |
+| 0.3313 | 7500  | 0.0138        |
+| 0.3534 | 8000  | 0.0135        |
+| 0.3755 | 8500  | 0.0138        |
+| 0.3976 | 9000  | 0.0138        |
+| 0.4196 | 9500  | 0.0141        |
+| 0.4417 | 10000 | 0.0139        |
+| 0.4638 | 10500 | 0.0139        |
+| 0.4859 | 11000 | 0.0138        |
+| 0.5080 | 11500 | 0.0141        |
+| 0.5301 | 12000 | 0.0138        |
+| 0.5522 | 12500 | 0.0138        |
+| 0.5743 | 13000 | 0.0138        |
+| 0.5963 | 13500 | 0.0138        |
+| 0.6184 | 14000 | 0.0136        |
+| 0.6405 | 14500 | 0.0139        |
+| 0.6626 | 15000 | 0.0151        |
+| 0.6847 | 15500 | 0.019         |
+| 0.7068 | 16000 | 0.0184        |
+| 0.7289 | 16500 | 0.018         |
+| 0.7509 | 17000 | 0.0163        |
+| 0.7730 | 17500 | 0.0164        |
+| 0.7951 | 18000 | 0.0158        |
+| 0.8172 | 18500 | 0.0155        |
+| 0.8393 | 19000 | 0.0151        |
+| 0.8614 | 19500 | 0.0151        |
+| 0.8835 | 20000 | 0.0152        |
+| 0.9056 | 20500 | 0.0152        |
+| 0.9276 | 21000 | 0.0151        |
+| 0.9497 | 21500 | 0.0148        |
+| 0.9718 | 22000 | 0.015         |
+| 0.9939 | 22500 | 0.0147        |
+| 1.0160 | 23000 | 0.0149        |
+| 1.0381 | 23500 | 0.0151        |
+| 1.0602 | 24000 | 0.015         |
+| 1.0823 | 24500 | 0.0148        |
+| 1.1043 | 25000 | 0.0147        |
+| 1.1264 | 25500 | 0.0149        |
+| 1.1485 | 26000 | 0.0147        |
+| 1.1706 | 26500 | 0.015         |
+| 1.1927 | 27000 | 0.0146        |
+| 1.2148 | 27500 | 0.0145        |
+| 1.2369 | 28000 | 0.0147        |
+| 1.2589 | 28500 | 0.0149        |
+| 1.2810 | 29000 | 0.0147        |
+| 1.3031 | 29500 | 0.0144        |
+| 1.3252 | 30000 | 0.0147        |
+| 1.3473 | 30500 | 0.0147        |
+| 1.3694 | 31000 | 0.0145        |
+| 1.3915 | 31500 | 0.0149        |
+| 1.4136 | 32000 | 0.0147        |
+| 1.4356 | 32500 | 0.0148        |
+| 1.4577 | 33000 | 0.0148        |
+| 1.4798 | 33500 | 0.0145        |
+| 1.5019 | 34000 | 0.0149        |
+| 1.5240 | 34500 | 0.0147        |
+| 1.5461 | 35000 | 0.0146        |
+| 1.5682 | 35500 | 0.0144        |
+| 1.5902 | 36000 | 0.0146        |
+| 1.6123 | 36500 | 0.0143        |
+| 1.6344 | 37000 | 0.0145        |
+| 1.6565 | 37500 | 0.0145        |
+| 1.6786 | 38000 | 0.0146        |
+| 1.7007 | 38500 | 0.0143        |
+| 1.7228 | 39000 | 0.0149        |
+| 1.7449 | 39500 | 0.0143        |
+| 1.7669 | 40000 | 0.0146        |
+| 1.7890 | 40500 | 0.0146        |
+| 1.8111 | 41000 | 0.0146        |
+| 1.8332 | 41500 | 0.0142        |
+| 1.8553 | 42000 | 0.0144        |
+| 1.8774 | 42500 | 0.0146        |
+| 1.8995 | 43000 | 0.0147        |
+| 1.9215 | 43500 | 0.0144        |
+| 1.9436 | 44000 | 0.0145        |
+| 1.9657 | 44500 | 0.0143        |
+| 1.9878 | 45000 | 0.0146        |
+| 2.0099 | 45500 | 0.0143        |
+| 2.0320 | 46000 | 0.0147        |
+| 2.0541 | 46500 | 0.0146        |
+| 2.0762 | 47000 | 0.0144        |
+| 2.0982 | 47500 | 0.0144        |
+| 2.1203 | 48000 | 0.0144        |
+| 2.1424 | 48500 | 0.0145        |
+| 2.1645 | 49000 | 0.0144        |
+| 2.1866 | 49500 | 0.0144        |
+| 2.2087 | 50000 | 0.0141        |
+| 2.2308 | 50500 | 0.0142        |
+| 2.2528 | 51000 | 0.0145        |
+| 2.2749 | 51500 | 0.0143        |
+| 2.2970 | 52000 | 0.0141        |
+| 2.3191 | 52500 | 0.0144        |
+| 2.3412 | 53000 | 0.0143        |
+| 2.3633 | 53500 | 0.0144        |
+| 2.3854 | 54000 | 0.0144        |
+| 2.4075 | 54500 | 0.0144        |
+| 2.4295 | 55000 | 0.0145        |
+| 2.4516 | 55500 | 0.0145        |
+| 2.4737 | 56000 | 0.0144        |
+| 2.4958 | 56500 | 0.0147        |
+| 2.5179 | 57000 | 0.0145        |
+| 2.5400 | 57500 | 0.0144        |
+| 2.5621 | 58000 | 0.0143        |
+| 2.5842 | 58500 | 0.0144        |
+| 2.6062 | 59000 | 0.0143        |
+| 2.6283 | 59500 | 0.0142        |
+| 2.6504 | 60000 | 0.0143        |
+| 2.6725 | 60500 | 0.0143        |
+| 2.6946 | 61000 | 0.0143        |
+| 2.7167 | 61500 | 0.0144        |
+| 2.7388 | 62000 | 0.0143        |
+| 2.7608 | 62500 | 0.0143        |
+| 2.7829 | 63000 | 0.0146        |
+| 2.8050 | 63500 | 0.0144        |
+| 2.8271 | 64000 | 0.0141        |
+| 2.8492 | 64500 | 0.0142        |
+| 2.8713 | 65000 | 0.0143        |
+| 2.8934 | 65500 | 0.0146        |
+| 2.9155 | 66000 | 0.0143        |
+| 2.9375 | 66500 | 0.0143        |
+| 2.9596 | 67000 | 0.0141        |
+| 2.9817 | 67500 | 0.0144        |
+| 3.0038 | 68000 | 0.0143        |
+| 3.0259 | 68500 | 0.0145        |
+| 3.0480 | 69000 | 0.0142        |
+| 3.0701 | 69500 | 0.0145        |
+| 3.0921 | 70000 | 0.0142        |
+| 3.1142 | 70500 | 0.0143        |
+| 3.1363 | 71000 | 0.0142        |
+| 3.1584 | 71500 | 0.0143        |
+| 3.1805 | 72000 | 0.0143        |
+| 3.2026 | 72500 | 0.014         |
+| 3.2247 | 73000 | 0.0141        |
+| 3.2468 | 73500 | 0.0142        |
+| 3.2688 | 74000 | 0.0143        |
+| 3.2909 | 74500 | 0.0141        |
+| 3.3130 | 75000 | 0.0141        |
+| 3.3351 | 75500 | 0.0143        |
+| 3.3572 | 76000 | 0.0141        |
+| 3.3793 | 76500 | 0.0143        |
+| 3.4014 | 77000 | 0.0143        |
+| 3.4234 | 77500 | 0.0146        |
+| 3.4455 | 78000 | 0.0144        |
+| 3.4676 | 78500 | 0.0143        |
+| 3.4897 | 79000 | 0.0144        |
+| 3.5118 | 79500 | 0.0145        |
+| 3.5339 | 80000 | 0.0142        |
+| 3.5560 | 80500 | 0.0144        |
+| 3.5781 | 81000 | 0.0143        |
+| 3.6001 | 81500 | 0.0142        |
+| 3.6222 | 82000 | 0.0142        |
+| 3.6443 | 82500 | 0.0142        |
+| 3.6664 | 83000 | 0.014         |
+| 3.6885 | 83500 | 0.0144        |
+| 3.7106 | 84000 | 0.0141        |
+| 3.7327 | 84500 | 0.0143        |
+| 3.7547 | 85000 | 0.014         |
+| 3.7768 | 85500 | 0.0146        |
+| 3.7989 | 86000 | 0.0143        |
+| 3.8210 | 86500 | 0.0142        |
+| 3.8431 | 87000 | 0.0139        |
+| 3.8652 | 87500 | 0.0143        |
+| 3.8873 | 88000 | 0.0144        |
+| 3.9094 | 88500 | 0.0143        |
+| 3.9314 | 89000 | 0.0142        |
+| 3.9535 | 89500 | 0.0142        |
+| 3.9756 | 90000 | 0.0142        |
+
+</details>
+
+### Framework Versions
+- Python: 3.10.13
+- Sentence Transformers: 3.1.0.dev0
+- Transformers: 4.39.3
+- PyTorch: 2.1.2
+- Accelerate: 0.29.3
+- Datasets: 2.18.0
+- Tokenizers: 0.15.2
+
+## Citation
+
+### BibTeX
+
+#### Sentence Transformers
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084",
+}
+```
+
+#### ContrastiveLoss
+```bibtex
+@inproceedings{hadsell2006dimensionality,
+    author={Hadsell, R. and Chopra, S. and LeCun, Y.},
+    booktitle={2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)}, 
+    title={Dimensionality Reduction by Learning an Invariant Mapping}, 
+    year={2006},
+    volume={2},
+    number={},
+    pages={1735-1742},
+    doi={10.1109/CVPR.2006.100}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<mask>": 64000
+}

config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "vinai/phobert-base-v2",
+  "architectures": [
+    "RobertaModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 258,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "tokenizer_class": "PhobertTokenizer",
+  "torch_dtype": "float32",
+  "transformers_version": "4.39.3",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 64001
+}

config_sentence_transformers.json

@@ -0,0 +1,10 @@
+{
+  "__version__": {
+    "sentence_transformers": "3.1.0.dev0",
+    "transformers": "4.39.3",
+    "pytorch": "2.1.2"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": null
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9d4a6db7578a9c2f29c471ecba2540b8522a455faff430999a8dadf897826b8
+size 540015464

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3d341f133c19d9a676f147da1a70460996cc9db7d9616a4b897900a4a508c2d
+size 1080152634

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0406bf623d5d04576a8b9be491c9c293d43e339baa4b502fdf4e9218b981bc8
+size 14244

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf1f98f00221cad9904202f2abd85131a25f85b16287bb2dab3e092ee2ace761
+size 1064

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 256,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,9 @@
+{
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,54 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64000": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "model_max_length": 256,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "PhobertTokenizer",
+  "unk_token": "<unk>"
+}

trainer_state.json

@@ -0,0 +1,1281 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.97561622051418,
+  "eval_steps": 0,
+  "global_step": 90000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.02,
+      "grad_norm": 0.010943206027150154,
+      "learning_rate": 1.6563604240282684e-06,
+      "loss": 0.0168,
+      "step": 500
+    },
+    {
+      "epoch": 0.04,
+      "grad_norm": 0.010196206159889698,
+      "learning_rate": 3.312720848056537e-06,
+      "loss": 0.0139,
+      "step": 1000
+    },
+    {
+      "epoch": 0.07,
+      "grad_norm": 0.009697528555989265,
+      "learning_rate": 4.969081272084806e-06,
+      "loss": 0.0142,
+      "step": 1500
+    },
+    {
+      "epoch": 0.09,
+      "grad_norm": 0.011516646482050419,
+      "learning_rate": 6.625441696113074e-06,
+      "loss": 0.0139,
+      "step": 2000
+    },
+    {
+      "epoch": 0.11,
+      "grad_norm": 0.01910693757236004,
+      "learning_rate": 8.281802120141344e-06,
+      "loss": 0.0137,
+      "step": 2500
+    },
+    {
+      "epoch": 0.13,
+      "grad_norm": 0.004923399072140455,
+      "learning_rate": 9.938162544169612e-06,
+      "loss": 0.0139,
+      "step": 3000
+    },
+    {
+      "epoch": 0.15,
+      "grad_norm": 0.007313380483537912,
+      "learning_rate": 1.159452296819788e-05,
+      "loss": 0.0137,
+      "step": 3500
+    },
+    {
+      "epoch": 0.18,
+      "grad_norm": 0.003835548646748066,
+      "learning_rate": 1.3250883392226147e-05,
+      "loss": 0.0139,
+      "step": 4000
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.007223702035844326,
+      "learning_rate": 1.4907243816254417e-05,
+      "loss": 0.0136,
+      "step": 4500
+    },
+    {
+      "epoch": 0.22,
+      "grad_norm": 0.010532427579164505,
+      "learning_rate": 1.6563604240282687e-05,
+      "loss": 0.0135,
+      "step": 5000
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.0022224171552807093,
+      "learning_rate": 1.8219964664310956e-05,
+      "loss": 0.0137,
+      "step": 5500
+    },
+    {
+      "epoch": 0.27,
+      "grad_norm": 0.0046217963099479675,
+      "learning_rate": 1.9876325088339224e-05,
+      "loss": 0.0138,
+      "step": 6000
+    },
+    {
+      "epoch": 0.29,
+      "grad_norm": 0.007814540527760983,
+      "learning_rate": 2.1532685512367493e-05,
+      "loss": 0.0136,
+      "step": 6500
+    },
+    {
+      "epoch": 0.31,
+      "grad_norm": 0.011144978925585747,
+      "learning_rate": 2.318904593639576e-05,
+      "loss": 0.0137,
+      "step": 7000
+    },
+    {
+      "epoch": 0.33,
+      "grad_norm": 0.0014108135364949703,
+      "learning_rate": 2.484540636042403e-05,
+      "loss": 0.0138,
+      "step": 7500
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.0039382693357765675,
+      "learning_rate": 2.6501766784452294e-05,
+      "loss": 0.0135,
+      "step": 8000
+    },
+    {
+      "epoch": 0.38,
+      "grad_norm": 0.014407187700271606,
+      "learning_rate": 2.8158127208480566e-05,
+      "loss": 0.0138,
+      "step": 8500
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.003938812296837568,
+      "learning_rate": 2.9814487632508834e-05,
+      "loss": 0.0138,
+      "step": 9000
+    },
+    {
+      "epoch": 0.42,
+      "grad_norm": 0.0075125317089259624,
+      "learning_rate": 2.983655639540591e-05,
+      "loss": 0.0141,
+      "step": 9500
+    },
+    {
+      "epoch": 0.44,
+      "grad_norm": 0.004186388570815325,
+      "learning_rate": 2.9652498282124278e-05,
+      "loss": 0.0139,
+      "step": 10000
+    },
+    {
+      "epoch": 0.46,
+      "grad_norm": 0.004882327280938625,
+      "learning_rate": 2.9468440168842645e-05,
+      "loss": 0.0139,
+      "step": 10500
+    },
+    {
+      "epoch": 0.49,
+      "grad_norm": 0.002360760699957609,
+      "learning_rate": 2.9284382055561012e-05,
+      "loss": 0.0138,
+      "step": 11000
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 0.003932056948542595,
+      "learning_rate": 2.910032394227938e-05,
+      "loss": 0.0141,
+      "step": 11500
+    },
+    {
+      "epoch": 0.53,
+      "grad_norm": 0.015781084075570107,
+      "learning_rate": 2.8916265828997743e-05,
+      "loss": 0.0138,
+      "step": 12000
+    },
+    {
+      "epoch": 0.55,
+      "grad_norm": 0.002987222746014595,
+      "learning_rate": 2.873220771571611e-05,
+      "loss": 0.0138,
+      "step": 12500
+    },
+    {
+      "epoch": 0.57,
+      "grad_norm": 0.002478894544765353,
+      "learning_rate": 2.8548149602434473e-05,
+      "loss": 0.0138,
+      "step": 13000
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 0.00281110149808228,
+      "learning_rate": 2.836409148915284e-05,
+      "loss": 0.0138,
+      "step": 13500
+    },
+    {
+      "epoch": 0.62,
+      "grad_norm": 0.0016416048165410757,
+      "learning_rate": 2.8180033375871207e-05,
+      "loss": 0.0136,
+      "step": 14000
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.003159256186336279,
+      "learning_rate": 2.7995975262589574e-05,
+      "loss": 0.0139,
+      "step": 14500
+    },
+    {
+      "epoch": 0.66,
+      "grad_norm": 0.03868310526013374,
+      "learning_rate": 2.781191714930794e-05,
+      "loss": 0.0151,
+      "step": 15000
+    },
+    {
+      "epoch": 0.68,
+      "grad_norm": 0.026466334238648415,
+      "learning_rate": 2.7627859036026308e-05,
+      "loss": 0.019,
+      "step": 15500
+    },
+    {
+      "epoch": 0.71,
+      "grad_norm": 0.014158655889332294,
+      "learning_rate": 2.7443800922744675e-05,
+      "loss": 0.0184,
+      "step": 16000
+    },
+    {
+      "epoch": 0.73,
+      "grad_norm": 0.0531819723546505,
+      "learning_rate": 2.7259742809463042e-05,
+      "loss": 0.018,
+      "step": 16500
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.031152892857789993,
+      "learning_rate": 2.707568469618141e-05,
+      "loss": 0.0163,
+      "step": 17000
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 0.045258529484272,
+      "learning_rate": 2.6891626582899776e-05,
+      "loss": 0.0164,
+      "step": 17500
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.08072955161333084,
+      "learning_rate": 2.6707568469618143e-05,
+      "loss": 0.0158,
+      "step": 18000
+    },
+    {
+      "epoch": 0.82,
+      "grad_norm": 0.06747995316982269,
+      "learning_rate": 2.652351035633651e-05,
+      "loss": 0.0155,
+      "step": 18500
+    },
+    {
+      "epoch": 0.84,
+      "grad_norm": 0.029322072863578796,
+      "learning_rate": 2.6339452243054877e-05,
+      "loss": 0.0151,
+      "step": 19000
+    },
+    {
+      "epoch": 0.86,
+      "grad_norm": 0.03149860352277756,
+      "learning_rate": 2.615539412977324e-05,
+      "loss": 0.0151,
+      "step": 19500
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.03845517709851265,
+      "learning_rate": 2.5971336016491608e-05,
+      "loss": 0.0152,
+      "step": 20000
+    },
+    {
+      "epoch": 0.91,
+      "grad_norm": 0.02114521712064743,
+      "learning_rate": 2.578727790320997e-05,
+      "loss": 0.0152,
+      "step": 20500
+    },
+    {
+      "epoch": 0.93,
+      "grad_norm": 0.04469970241189003,
+      "learning_rate": 2.560321978992834e-05,
+      "loss": 0.0151,
+      "step": 21000
+    },
+    {
+      "epoch": 0.95,
+      "grad_norm": 0.03439483791589737,
+      "learning_rate": 2.5419161676646705e-05,
+      "loss": 0.0148,
+      "step": 21500
+    },
+    {
+      "epoch": 0.97,
+      "grad_norm": 0.016261784359812737,
+      "learning_rate": 2.5235103563365072e-05,
+      "loss": 0.015,
+      "step": 22000
+    },
+    {
+      "epoch": 0.99,
+      "grad_norm": 0.04864068329334259,
+      "learning_rate": 2.505104545008344e-05,
+      "loss": 0.0147,
+      "step": 22500
+    },
+    {
+      "epoch": 1.02,
+      "grad_norm": 0.024570118635892868,
+      "learning_rate": 2.4866987336801806e-05,
+      "loss": 0.0149,
+      "step": 23000
+    },
+    {
+      "epoch": 1.04,
+      "grad_norm": 0.015043354593217373,
+      "learning_rate": 2.4682929223520173e-05,
+      "loss": 0.0151,
+      "step": 23500
+    },
+    {
+      "epoch": 1.06,
+      "grad_norm": 0.038648445159196854,
+      "learning_rate": 2.449887111023854e-05,
+      "loss": 0.015,
+      "step": 24000
+    },
+    {
+      "epoch": 1.08,
+      "grad_norm": 0.2623123824596405,
+      "learning_rate": 2.4314812996956907e-05,
+      "loss": 0.0148,
+      "step": 24500
+    },
+    {
+      "epoch": 1.1,
+      "grad_norm": 0.02235906571149826,
+      "learning_rate": 2.4130754883675274e-05,
+      "loss": 0.0147,
+      "step": 25000
+    },
+    {
+      "epoch": 1.13,
+      "grad_norm": 0.005854467861354351,
+      "learning_rate": 2.394669677039364e-05,
+      "loss": 0.0149,
+      "step": 25500
+    },
+    {
+      "epoch": 1.15,
+      "grad_norm": 0.011547247879207134,
+      "learning_rate": 2.376263865711201e-05,
+      "loss": 0.0147,
+      "step": 26000
+    },
+    {
+      "epoch": 1.17,
+      "grad_norm": 0.03119933046400547,
+      "learning_rate": 2.3578580543830375e-05,
+      "loss": 0.015,
+      "step": 26500
+    },
+    {
+      "epoch": 1.19,
+      "grad_norm": 0.047728102654218674,
+      "learning_rate": 2.339452243054874e-05,
+      "loss": 0.0146,
+      "step": 27000
+    },
+    {
+      "epoch": 1.21,
+      "grad_norm": 0.04931659996509552,
+      "learning_rate": 2.3210464317267106e-05,
+      "loss": 0.0145,
+      "step": 27500
+    },
+    {
+      "epoch": 1.24,
+      "grad_norm": 0.22793345153331757,
+      "learning_rate": 2.3026406203985473e-05,
+      "loss": 0.0147,
+      "step": 28000
+    },
+    {
+      "epoch": 1.26,
+      "grad_norm": 0.046943288296461105,
+      "learning_rate": 2.2842348090703837e-05,
+      "loss": 0.0149,
+      "step": 28500
+    },
+    {
+      "epoch": 1.28,
+      "grad_norm": 0.07938718795776367,
+      "learning_rate": 2.2658289977422203e-05,
+      "loss": 0.0147,
+      "step": 29000
+    },
+    {
+      "epoch": 1.3,
+      "grad_norm": 0.02574564516544342,
+      "learning_rate": 2.247423186414057e-05,
+      "loss": 0.0144,
+      "step": 29500
+    },
+    {
+      "epoch": 1.33,
+      "grad_norm": 0.011776907369494438,
+      "learning_rate": 2.2290173750858937e-05,
+      "loss": 0.0147,
+      "step": 30000
+    },
+    {
+      "epoch": 1.35,
+      "grad_norm": 0.0066869258880615234,
+      "learning_rate": 2.2106115637577304e-05,
+      "loss": 0.0147,
+      "step": 30500
+    },
+    {
+      "epoch": 1.37,
+      "grad_norm": 0.010923570953309536,
+      "learning_rate": 2.192205752429567e-05,
+      "loss": 0.0145,
+      "step": 31000
+    },
+    {
+      "epoch": 1.39,
+      "grad_norm": 0.03816843405365944,
+      "learning_rate": 2.173799941101404e-05,
+      "loss": 0.0149,
+      "step": 31500
+    },
+    {
+      "epoch": 1.41,
+      "grad_norm": 0.04863005876541138,
+      "learning_rate": 2.1553941297732405e-05,
+      "loss": 0.0147,
+      "step": 32000
+    },
+    {
+      "epoch": 1.44,
+      "grad_norm": 0.09232094883918762,
+      "learning_rate": 2.1369883184450772e-05,
+      "loss": 0.0148,
+      "step": 32500
+    },
+    {
+      "epoch": 1.46,
+      "grad_norm": 0.006221645046025515,
+      "learning_rate": 2.118582507116914e-05,
+      "loss": 0.0148,
+      "step": 33000
+    },
+    {
+      "epoch": 1.48,
+      "grad_norm": 0.008674496784806252,
+      "learning_rate": 2.1001766957887506e-05,
+      "loss": 0.0145,
+      "step": 33500
+    },
+    {
+      "epoch": 1.5,
+      "grad_norm": 0.013610797002911568,
+      "learning_rate": 2.081770884460587e-05,
+      "loss": 0.0149,
+      "step": 34000
+    },
+    {
+      "epoch": 1.52,
+      "grad_norm": 0.020487351343035698,
+      "learning_rate": 2.0633650731324237e-05,
+      "loss": 0.0147,
+      "step": 34500
+    },
+    {
+      "epoch": 1.55,
+      "grad_norm": 0.004388992674648762,
+      "learning_rate": 2.0449592618042604e-05,
+      "loss": 0.0146,
+      "step": 35000
+    },
+    {
+      "epoch": 1.57,
+      "grad_norm": 0.029407095164060593,
+      "learning_rate": 2.026553450476097e-05,
+      "loss": 0.0144,
+      "step": 35500
+    },
+    {
+      "epoch": 1.59,
+      "grad_norm": 0.04079248011112213,
+      "learning_rate": 2.0081476391479335e-05,
+      "loss": 0.0146,
+      "step": 36000
+    },
+    {
+      "epoch": 1.61,
+      "grad_norm": 0.033315982669591904,
+      "learning_rate": 1.98974182781977e-05,
+      "loss": 0.0143,
+      "step": 36500
+    },
+    {
+      "epoch": 1.63,
+      "grad_norm": 0.00864444486796856,
+      "learning_rate": 1.971336016491607e-05,
+      "loss": 0.0145,
+      "step": 37000
+    },
+    {
+      "epoch": 1.66,
+      "grad_norm": 0.03435393422842026,
+      "learning_rate": 1.9529302051634436e-05,
+      "loss": 0.0145,
+      "step": 37500
+    },
+    {
+      "epoch": 1.68,
+      "grad_norm": 0.008053929544985294,
+      "learning_rate": 1.9345243938352803e-05,
+      "loss": 0.0146,
+      "step": 38000
+    },
+    {
+      "epoch": 1.7,
+      "grad_norm": 0.004771470092236996,
+      "learning_rate": 1.916118582507117e-05,
+      "loss": 0.0143,
+      "step": 38500
+    },
+    {
+      "epoch": 1.72,
+      "grad_norm": 0.016594666987657547,
+      "learning_rate": 1.8977127711789537e-05,
+      "loss": 0.0149,
+      "step": 39000
+    },
+    {
+      "epoch": 1.74,
+      "grad_norm": 0.01181800477206707,
+      "learning_rate": 1.8793069598507904e-05,
+      "loss": 0.0143,
+      "step": 39500
+    },
+    {
+      "epoch": 1.77,
+      "grad_norm": 0.03508065640926361,
+      "learning_rate": 1.860901148522627e-05,
+      "loss": 0.0146,
+      "step": 40000
+    },
+    {
+      "epoch": 1.79,
+      "grad_norm": 0.028093870729207993,
+      "learning_rate": 1.8424953371944638e-05,
+      "loss": 0.0146,
+      "step": 40500
+    },
+    {
+      "epoch": 1.81,
+      "grad_norm": 0.05029403790831566,
+      "learning_rate": 1.8240895258663005e-05,
+      "loss": 0.0146,
+      "step": 41000
+    },
+    {
+      "epoch": 1.83,
+      "grad_norm": 0.006528925616294146,
+      "learning_rate": 1.8056837145381368e-05,
+      "loss": 0.0142,
+      "step": 41500
+    },
+    {
+      "epoch": 1.86,
+      "grad_norm": 0.02557162009179592,
+      "learning_rate": 1.7872779032099735e-05,
+      "loss": 0.0144,
+      "step": 42000
+    },
+    {
+      "epoch": 1.88,
+      "grad_norm": 0.025360217317938805,
+      "learning_rate": 1.7688720918818102e-05,
+      "loss": 0.0146,
+      "step": 42500
+    },
+    {
+      "epoch": 1.9,
+      "grad_norm": 0.04788580909371376,
+      "learning_rate": 1.750466280553647e-05,
+      "loss": 0.0147,
+      "step": 43000
+    },
+    {
+      "epoch": 1.92,
+      "grad_norm": 0.02906920574605465,
+      "learning_rate": 1.7320604692254836e-05,
+      "loss": 0.0144,
+      "step": 43500
+    },
+    {
+      "epoch": 1.94,
+      "grad_norm": 0.012823808006942272,
+      "learning_rate": 1.71365465789732e-05,
+      "loss": 0.0145,
+      "step": 44000
+    },
+    {
+      "epoch": 1.97,
+      "grad_norm": 0.008996455930173397,
+      "learning_rate": 1.6952488465691567e-05,
+      "loss": 0.0143,
+      "step": 44500
+    },
+    {
+      "epoch": 1.99,
+      "grad_norm": 0.010119748301804066,
+      "learning_rate": 1.6768430352409934e-05,
+      "loss": 0.0146,
+      "step": 45000
+    },
+    {
+      "epoch": 2.01,
+      "grad_norm": 0.02591855265200138,
+      "learning_rate": 1.65843722391283e-05,
+      "loss": 0.0143,
+      "step": 45500
+    },
+    {
+      "epoch": 2.03,
+      "grad_norm": 0.013729671947658062,
+      "learning_rate": 1.6400314125846668e-05,
+      "loss": 0.0147,
+      "step": 46000
+    },
+    {
+      "epoch": 2.05,
+      "grad_norm": 0.0771203562617302,
+      "learning_rate": 1.6216256012565035e-05,
+      "loss": 0.0146,
+      "step": 46500
+    },
+    {
+      "epoch": 2.08,
+      "grad_norm": 0.04501279070973396,
+      "learning_rate": 1.60321978992834e-05,
+      "loss": 0.0144,
+      "step": 47000
+    },
+    {
+      "epoch": 2.1,
+      "grad_norm": 0.03493111953139305,
+      "learning_rate": 1.584813978600177e-05,
+      "loss": 0.0144,
+      "step": 47500
+    },
+    {
+      "epoch": 2.12,
+      "grad_norm": 0.01472916454076767,
+      "learning_rate": 1.5664081672720136e-05,
+      "loss": 0.0144,
+      "step": 48000
+    },
+    {
+      "epoch": 2.14,
+      "grad_norm": 0.04763146862387657,
+      "learning_rate": 1.54800235594385e-05,
+      "loss": 0.0145,
+      "step": 48500
+    },
+    {
+      "epoch": 2.16,
+      "grad_norm": 0.024467509239912033,
+      "learning_rate": 1.5295965446156866e-05,
+      "loss": 0.0144,
+      "step": 49000
+    },
+    {
+      "epoch": 2.19,
+      "grad_norm": 0.01768341101706028,
+      "learning_rate": 1.5111907332875235e-05,
+      "loss": 0.0144,
+      "step": 49500
+    },
+    {
+      "epoch": 2.21,
+      "grad_norm": 0.06102894991636276,
+      "learning_rate": 1.49278492195936e-05,
+      "loss": 0.0141,
+      "step": 50000
+    },
+    {
+      "epoch": 2.23,
+      "grad_norm": 0.01851697266101837,
+      "learning_rate": 1.4743791106311966e-05,
+      "loss": 0.0142,
+      "step": 50500
+    },
+    {
+      "epoch": 2.25,
+      "grad_norm": 0.015444310382008553,
+      "learning_rate": 1.4559732993030333e-05,
+      "loss": 0.0145,
+      "step": 51000
+    },
+    {
+      "epoch": 2.27,
+      "grad_norm": 0.013120009563863277,
+      "learning_rate": 1.43756748797487e-05,
+      "loss": 0.0143,
+      "step": 51500
+    },
+    {
+      "epoch": 2.3,
+      "grad_norm": 0.01589464209973812,
+      "learning_rate": 1.4191616766467067e-05,
+      "loss": 0.0141,
+      "step": 52000
+    },
+    {
+      "epoch": 2.32,
+      "grad_norm": 0.040490709245204926,
+      "learning_rate": 1.4007558653185433e-05,
+      "loss": 0.0144,
+      "step": 52500
+    },
+    {
+      "epoch": 2.34,
+      "grad_norm": 0.025874989107251167,
+      "learning_rate": 1.38235005399038e-05,
+      "loss": 0.0143,
+      "step": 53000
+    },
+    {
+      "epoch": 2.36,
+      "grad_norm": 0.022394156083464622,
+      "learning_rate": 1.3639442426622166e-05,
+      "loss": 0.0144,
+      "step": 53500
+    },
+    {
+      "epoch": 2.39,
+      "grad_norm": 0.010273805819451809,
+      "learning_rate": 1.3455384313340531e-05,
+      "loss": 0.0144,
+      "step": 54000
+    },
+    {
+      "epoch": 2.41,
+      "grad_norm": 0.028374383226037025,
+      "learning_rate": 1.3271326200058898e-05,
+      "loss": 0.0144,
+      "step": 54500
+    },
+    {
+      "epoch": 2.43,
+      "grad_norm": 0.018441613763570786,
+      "learning_rate": 1.3087268086777265e-05,
+      "loss": 0.0145,
+      "step": 55000
+    },
+    {
+      "epoch": 2.45,
+      "grad_norm": 0.006460436619818211,
+      "learning_rate": 1.2903209973495632e-05,
+      "loss": 0.0145,
+      "step": 55500
+    },
+    {
+      "epoch": 2.47,
+      "grad_norm": 0.00770485308021307,
+      "learning_rate": 1.2719151860213999e-05,
+      "loss": 0.0144,
+      "step": 56000
+    },
+    {
+      "epoch": 2.5,
+      "grad_norm": 0.00849447026848793,
+      "learning_rate": 1.2535093746932366e-05,
+      "loss": 0.0147,
+      "step": 56500
+    },
+    {
+      "epoch": 2.52,
+      "grad_norm": 0.035621609538793564,
+      "learning_rate": 1.2351035633650733e-05,
+      "loss": 0.0145,
+      "step": 57000
+    },
+    {
+      "epoch": 2.54,
+      "grad_norm": 0.008923010900616646,
+      "learning_rate": 1.2166977520369098e-05,
+      "loss": 0.0144,
+      "step": 57500
+    },
+    {
+      "epoch": 2.56,
+      "grad_norm": 0.0056849876418709755,
+      "learning_rate": 1.1982919407087464e-05,
+      "loss": 0.0143,
+      "step": 58000
+    },
+    {
+      "epoch": 2.58,
+      "grad_norm": 0.0071659935638308525,
+      "learning_rate": 1.179886129380583e-05,
+      "loss": 0.0144,
+      "step": 58500
+    },
+    {
+      "epoch": 2.61,
+      "grad_norm": 0.021617043763399124,
+      "learning_rate": 1.1614803180524198e-05,
+      "loss": 0.0143,
+      "step": 59000
+    },
+    {
+      "epoch": 2.63,
+      "grad_norm": 0.011144719086587429,
+      "learning_rate": 1.1430745067242565e-05,
+      "loss": 0.0142,
+      "step": 59500
+    },
+    {
+      "epoch": 2.65,
+      "grad_norm": 0.010943782515823841,
+      "learning_rate": 1.1246686953960932e-05,
+      "loss": 0.0143,
+      "step": 60000
+    },
+    {
+      "epoch": 2.67,
+      "grad_norm": 0.010286700911819935,
+      "learning_rate": 1.1062628840679299e-05,
+      "loss": 0.0143,
+      "step": 60500
+    },
+    {
+      "epoch": 2.69,
+      "grad_norm": 0.010169615969061852,
+      "learning_rate": 1.0878570727397666e-05,
+      "loss": 0.0143,
+      "step": 61000
+    },
+    {
+      "epoch": 2.72,
+      "grad_norm": 0.032067082822322845,
+      "learning_rate": 1.069451261411603e-05,
+      "loss": 0.0144,
+      "step": 61500
+    },
+    {
+      "epoch": 2.74,
+      "grad_norm": 0.008680183440446854,
+      "learning_rate": 1.0510454500834396e-05,
+      "loss": 0.0143,
+      "step": 62000
+    },
+    {
+      "epoch": 2.76,
+      "grad_norm": 0.01648719422519207,
+      "learning_rate": 1.0326396387552763e-05,
+      "loss": 0.0143,
+      "step": 62500
+    },
+    {
+      "epoch": 2.78,
+      "grad_norm": 0.0210120789706707,
+      "learning_rate": 1.014233827427113e-05,
+      "loss": 0.0146,
+      "step": 63000
+    },
+    {
+      "epoch": 2.81,
+      "grad_norm": 0.034336596727371216,
+      "learning_rate": 9.958280160989497e-06,
+      "loss": 0.0144,
+      "step": 63500
+    },
+    {
+      "epoch": 2.83,
+      "grad_norm": 0.03138417750597,
+      "learning_rate": 9.774222047707864e-06,
+      "loss": 0.0141,
+      "step": 64000
+    },
+    {
+      "epoch": 2.85,
+      "grad_norm": 0.01799875684082508,
+      "learning_rate": 9.590163934426231e-06,
+      "loss": 0.0142,
+      "step": 64500
+    },
+    {
+      "epoch": 2.87,
+      "grad_norm": 0.02960127592086792,
+      "learning_rate": 9.406105821144595e-06,
+      "loss": 0.0143,
+      "step": 65000
+    },
+    {
+      "epoch": 2.89,
+      "grad_norm": 0.012712860479950905,
+      "learning_rate": 9.222047707862962e-06,
+      "loss": 0.0146,
+      "step": 65500
+    },
+    {
+      "epoch": 2.92,
+      "grad_norm": 0.009180006571114063,
+      "learning_rate": 9.037989594581329e-06,
+      "loss": 0.0143,
+      "step": 66000
+    },
+    {
+      "epoch": 2.94,
+      "grad_norm": 0.0106426402926445,
+      "learning_rate": 8.853931481299696e-06,
+      "loss": 0.0143,
+      "step": 66500
+    },
+    {
+      "epoch": 2.96,
+      "grad_norm": 0.03638075664639473,
+      "learning_rate": 8.669873368018063e-06,
+      "loss": 0.0141,
+      "step": 67000
+    },
+    {
+      "epoch": 2.98,
+      "grad_norm": 0.02028089202940464,
+      "learning_rate": 8.48581525473643e-06,
+      "loss": 0.0144,
+      "step": 67500
+    },
+    {
+      "epoch": 3.0,
+      "grad_norm": 0.004987742286175489,
+      "learning_rate": 8.301757141454797e-06,
+      "loss": 0.0143,
+      "step": 68000
+    },
+    {
+      "epoch": 3.03,
+      "grad_norm": 0.012421207502484322,
+      "learning_rate": 8.117699028173162e-06,
+      "loss": 0.0145,
+      "step": 68500
+    },
+    {
+      "epoch": 3.05,
+      "grad_norm": 0.05489884316921234,
+      "learning_rate": 7.933640914891527e-06,
+      "loss": 0.0142,
+      "step": 69000
+    },
+    {
+      "epoch": 3.07,
+      "grad_norm": 0.007833471521735191,
+      "learning_rate": 7.749582801609894e-06,
+      "loss": 0.0145,
+      "step": 69500
+    },
+    {
+      "epoch": 3.09,
+      "grad_norm": 0.014776123687624931,
+      "learning_rate": 7.565524688328261e-06,
+      "loss": 0.0142,
+      "step": 70000
+    },
+    {
+      "epoch": 3.11,
+      "grad_norm": 0.015590249560773373,
+      "learning_rate": 7.381466575046628e-06,
+      "loss": 0.0143,
+      "step": 70500
+    },
+    {
+      "epoch": 3.14,
+      "grad_norm": 0.018214261159300804,
+      "learning_rate": 7.197408461764995e-06,
+      "loss": 0.0142,
+      "step": 71000
+    },
+    {
+      "epoch": 3.16,
+      "grad_norm": 0.029773008078336716,
+      "learning_rate": 7.013350348483361e-06,
+      "loss": 0.0143,
+      "step": 71500
+    },
+    {
+      "epoch": 3.18,
+      "grad_norm": 0.028139958158135414,
+      "learning_rate": 6.8292922352017276e-06,
+      "loss": 0.0143,
+      "step": 72000
+    },
+    {
+      "epoch": 3.2,
+      "grad_norm": 0.024558302015066147,
+      "learning_rate": 6.6452341219200945e-06,
+      "loss": 0.014,
+      "step": 72500
+    },
+    {
+      "epoch": 3.22,
+      "grad_norm": 0.05188705772161484,
+      "learning_rate": 6.461176008638461e-06,
+      "loss": 0.0141,
+      "step": 73000
+    },
+    {
+      "epoch": 3.25,
+      "grad_norm": 0.02240253984928131,
+      "learning_rate": 6.277117895356828e-06,
+      "loss": 0.0142,
+      "step": 73500
+    },
+    {
+      "epoch": 3.27,
+      "grad_norm": 0.015994379296898842,
+      "learning_rate": 6.093059782075194e-06,
+      "loss": 0.0143,
+      "step": 74000
+    },
+    {
+      "epoch": 3.29,
+      "grad_norm": 0.014095323160290718,
+      "learning_rate": 5.909001668793561e-06,
+      "loss": 0.0141,
+      "step": 74500
+    },
+    {
+      "epoch": 3.31,
+      "grad_norm": 0.0076615894213318825,
+      "learning_rate": 5.724943555511927e-06,
+      "loss": 0.0141,
+      "step": 75000
+    },
+    {
+      "epoch": 3.34,
+      "grad_norm": 0.023330098018050194,
+      "learning_rate": 5.540885442230294e-06,
+      "loss": 0.0143,
+      "step": 75500
+    },
+    {
+      "epoch": 3.36,
+      "grad_norm": 0.022397508844733238,
+      "learning_rate": 5.35682732894866e-06,
+      "loss": 0.0141,
+      "step": 76000
+    },
+    {
+      "epoch": 3.38,
+      "grad_norm": 0.01998765394091606,
+      "learning_rate": 5.172769215667027e-06,
+      "loss": 0.0143,
+      "step": 76500
+    },
+    {
+      "epoch": 3.4,
+      "grad_norm": 0.07193479686975479,
+      "learning_rate": 4.988711102385393e-06,
+      "loss": 0.0143,
+      "step": 77000
+    },
+    {
+      "epoch": 3.42,
+      "grad_norm": 0.030124777927994728,
+      "learning_rate": 4.80465298910376e-06,
+      "loss": 0.0146,
+      "step": 77500
+    },
+    {
+      "epoch": 3.45,
+      "grad_norm": 0.0762249082326889,
+      "learning_rate": 4.620594875822126e-06,
+      "loss": 0.0144,
+      "step": 78000
+    },
+    {
+      "epoch": 3.47,
+      "grad_norm": 0.030013220384716988,
+      "learning_rate": 4.4365367625404925e-06,
+      "loss": 0.0143,
+      "step": 78500
+    },
+    {
+      "epoch": 3.49,
+      "grad_norm": 0.013210024684667587,
+      "learning_rate": 4.2524786492588595e-06,
+      "loss": 0.0144,
+      "step": 79000
+    },
+    {
+      "epoch": 3.51,
+      "grad_norm": 0.021476522088050842,
+      "learning_rate": 4.068420535977226e-06,
+      "loss": 0.0145,
+      "step": 79500
+    },
+    {
+      "epoch": 3.53,
+      "grad_norm": 0.005120801739394665,
+      "learning_rate": 3.884362422695593e-06,
+      "loss": 0.0142,
+      "step": 80000
+    },
+    {
+      "epoch": 3.56,
+      "grad_norm": 0.03378542512655258,
+      "learning_rate": 3.7003043094139592e-06,
+      "loss": 0.0144,
+      "step": 80500
+    },
+    {
+      "epoch": 3.58,
+      "grad_norm": 0.004559422377496958,
+      "learning_rate": 3.5162461961323254e-06,
+      "loss": 0.0143,
+      "step": 81000
+    },
+    {
+      "epoch": 3.6,
+      "grad_norm": 0.022087154909968376,
+      "learning_rate": 3.3321880828506924e-06,
+      "loss": 0.0142,
+      "step": 81500
+    },
+    {
+      "epoch": 3.62,
+      "grad_norm": 0.027302134782075882,
+      "learning_rate": 3.1481299695690585e-06,
+      "loss": 0.0142,
+      "step": 82000
+    },
+    {
+      "epoch": 3.64,
+      "grad_norm": 0.0070667564868927,
+      "learning_rate": 2.964071856287425e-06,
+      "loss": 0.0142,
+      "step": 82500
+    },
+    {
+      "epoch": 3.67,
+      "grad_norm": 0.004392644390463829,
+      "learning_rate": 2.7800137430057916e-06,
+      "loss": 0.014,
+      "step": 83000
+    },
+    {
+      "epoch": 3.69,
+      "grad_norm": 0.004756265785545111,
+      "learning_rate": 2.595955629724158e-06,
+      "loss": 0.0144,
+      "step": 83500
+    },
+    {
+      "epoch": 3.71,
+      "grad_norm": 0.028167065232992172,
+      "learning_rate": 2.4118975164425248e-06,
+      "loss": 0.0141,
+      "step": 84000
+    },
+    {
+      "epoch": 3.73,
+      "grad_norm": 0.04241223633289337,
+      "learning_rate": 2.2278394031608913e-06,
+      "loss": 0.0143,
+      "step": 84500
+    },
+    {
+      "epoch": 3.75,
+      "grad_norm": 0.0073333000764250755,
+      "learning_rate": 2.043781289879258e-06,
+      "loss": 0.014,
+      "step": 85000
+    },
+    {
+      "epoch": 3.78,
+      "grad_norm": 0.022960776463150978,
+      "learning_rate": 1.8597231765976245e-06,
+      "loss": 0.0146,
+      "step": 85500
+    },
+    {
+      "epoch": 3.8,
+      "grad_norm": 0.009491208009421825,
+      "learning_rate": 1.675665063315991e-06,
+      "loss": 0.0143,
+      "step": 86000
+    },
+    {
+      "epoch": 3.82,
+      "grad_norm": 0.04244249686598778,
+      "learning_rate": 1.4916069500343576e-06,
+      "loss": 0.0142,
+      "step": 86500
+    },
+    {
+      "epoch": 3.84,
+      "grad_norm": 0.0845978856086731,
+      "learning_rate": 1.307548836752724e-06,
+      "loss": 0.0139,
+      "step": 87000
+    },
+    {
+      "epoch": 3.87,
+      "grad_norm": 0.012684383429586887,
+      "learning_rate": 1.1234907234710905e-06,
+      "loss": 0.0143,
+      "step": 87500
+    },
+    {
+      "epoch": 3.89,
+      "grad_norm": 0.010290221311151981,
+      "learning_rate": 9.394326101894571e-07,
+      "loss": 0.0144,
+      "step": 88000
+    },
+    {
+      "epoch": 3.91,
+      "grad_norm": 0.009122644551098347,
+      "learning_rate": 7.553744969078238e-07,
+      "loss": 0.0143,
+      "step": 88500
+    },
+    {
+      "epoch": 3.93,
+      "grad_norm": 0.02026693895459175,
+      "learning_rate": 5.713163836261902e-07,
+      "loss": 0.0142,
+      "step": 89000
+    },
+    {
+      "epoch": 3.95,
+      "grad_norm": 0.03241865336894989,
+      "learning_rate": 3.8725827034455676e-07,
+      "loss": 0.0142,
+      "step": 89500
+    },
+    {
+      "epoch": 3.98,
+      "grad_norm": 0.04753628000617027,
+      "learning_rate": 2.0320015706292333e-07,
+      "loss": 0.0142,
+      "step": 90000
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 90552,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 10000,
+  "total_flos": 0.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c9d2d8c930913ff2f4ae4382de12a1443b6cd2be1903c4675e15ed7f1a359e1
+size 5176