|
--- |
|
tags: |
|
- generated_from_trainer |
|
metrics: |
|
- accuracy |
|
model-index: |
|
- name: roberta-tiny-2l-10M |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# roberta-tiny-2l-10M |
|
|
|
This model was trained from scratch on an unknown dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 3.1695 |
|
- Accuracy: 0.4534 |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 0.0004 |
|
- train_batch_size: 16 |
|
- eval_batch_size: 32 |
|
- seed: 42 |
|
- gradient_accumulation_steps: 32 |
|
- total_train_batch_size: 512 |
|
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08 |
|
- lr_scheduler_type: cosine |
|
- lr_scheduler_warmup_steps: 50 |
|
- num_epochs: 100.0 |
|
- mixed_precision_training: Native AMP |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Accuracy | |
|
|:-------------:|:-----:|:----:|:---------------:|:--------:| |
|
| 7.7619 | 1.04 | 50 | 7.2338 | 0.0748 | |
|
| 7.0524 | 2.08 | 100 | 6.6252 | 0.1331 | |
|
| 6.8423 | 3.12 | 150 | 6.4622 | 0.1463 | |
|
| 6.7298 | 4.16 | 200 | 6.3971 | 0.1488 | |
|
| 6.669 | 5.21 | 250 | 6.3628 | 0.1519 | |
|
| 6.2038 | 6.25 | 300 | 6.3371 | 0.1518 | |
|
| 6.1783 | 7.29 | 350 | 6.3115 | 0.1532 | |
|
| 6.1459 | 8.33 | 400 | 6.2922 | 0.1530 | |
|
| 6.1096 | 9.37 | 450 | 6.2696 | 0.1536 | |
|
| 6.0745 | 10.41 | 500 | 6.2545 | 0.1541 | |
|
| 6.0689 | 11.45 | 550 | 6.2496 | 0.1533 | |
|
| 6.0562 | 12.49 | 600 | 6.2313 | 0.1542 | |
|
| 6.0324 | 13.53 | 650 | 6.2248 | 0.1536 | |
|
| 5.9907 | 14.58 | 700 | 6.2179 | 0.1544 | |
|
| 5.9683 | 15.62 | 750 | 6.1832 | 0.1545 | |
|
| 5.9236 | 16.66 | 800 | 6.1413 | 0.1550 | |
|
| 5.8808 | 17.7 | 850 | 6.0900 | 0.1558 | |
|
| 5.8392 | 18.74 | 900 | 6.0543 | 0.1566 | |
|
| 5.7962 | 19.78 | 950 | 6.0222 | 0.1575 | |
|
| 5.7473 | 20.82 | 1000 | 5.9471 | 0.1617 | |
|
| 5.5787 | 21.86 | 1050 | 5.7038 | 0.1891 | |
|
| 5.2316 | 22.9 | 1100 | 5.2708 | 0.2382 | |
|
| 4.6613 | 23.95 | 1150 | 4.7075 | 0.2975 | |
|
| 4.3006 | 24.99 | 1200 | 4.4180 | 0.3222 | |
|
| 4.3754 | 26.04 | 1250 | 4.2383 | 0.3385 | |
|
| 4.2531 | 27.08 | 1300 | 4.1157 | 0.3491 | |
|
| 4.0987 | 28.12 | 1350 | 4.0197 | 0.3578 | |
|
| 4.0045 | 29.16 | 1400 | 3.9504 | 0.3656 | |
|
| 3.9145 | 30.21 | 1450 | 3.8819 | 0.3718 | |
|
| 3.5808 | 31.25 | 1500 | 3.8279 | 0.3781 | |
|
| 3.5354 | 32.29 | 1550 | 3.7830 | 0.3826 | |
|
| 3.4788 | 33.33 | 1600 | 3.7400 | 0.3872 | |
|
| 3.4315 | 34.37 | 1650 | 3.7028 | 0.3911 | |
|
| 3.3906 | 35.41 | 1700 | 3.6629 | 0.3956 | |
|
| 3.3508 | 36.45 | 1750 | 3.6344 | 0.3984 | |
|
| 3.288 | 37.49 | 1800 | 3.6046 | 0.4019 | |
|
| 3.2678 | 38.53 | 1850 | 3.5799 | 0.4053 | |
|
| 3.2382 | 39.58 | 1900 | 3.5549 | 0.4074 | |
|
| 3.2151 | 40.62 | 1950 | 3.5285 | 0.4103 | |
|
| 3.1777 | 41.66 | 2000 | 3.5069 | 0.4132 | |
|
| 3.1499 | 42.7 | 2050 | 3.4917 | 0.4150 | |
|
| 3.131 | 43.74 | 2100 | 3.4701 | 0.4168 | |
|
| 3.0942 | 44.78 | 2150 | 3.4530 | 0.4189 | |
|
| 3.0683 | 45.82 | 2200 | 3.4320 | 0.4212 | |
|
| 3.0363 | 46.86 | 2250 | 3.4195 | 0.4227 | |
|
| 3.0264 | 47.9 | 2300 | 3.4046 | 0.4249 | |
|
| 3.0079 | 48.95 | 2350 | 3.3874 | 0.4267 | |
|
| 2.9869 | 49.99 | 2400 | 3.3792 | 0.4277 | |
|
| 3.1592 | 51.04 | 2450 | 3.3655 | 0.4289 | |
|
| 3.1353 | 52.08 | 2500 | 3.3548 | 0.4310 | |
|
| 3.1257 | 53.12 | 2550 | 3.3489 | 0.4308 | |
|
| 3.0822 | 54.16 | 2600 | 3.3353 | 0.4327 | |
|
| 3.0771 | 55.21 | 2650 | 3.3220 | 0.4341 | |
|
| 2.8639 | 56.25 | 2700 | 3.3119 | 0.4354 | |
|
| 2.8477 | 57.29 | 2750 | 3.3104 | 0.4360 | |
|
| 2.8373 | 58.33 | 2800 | 3.2954 | 0.4378 | |
|
| 2.818 | 59.37 | 2850 | 3.2935 | 0.4381 | |
|
| 2.8137 | 60.41 | 2900 | 3.2786 | 0.4394 | |
|
| 2.7985 | 61.45 | 2950 | 3.2747 | 0.4401 | |
|
| 2.7936 | 62.49 | 3000 | 3.2668 | 0.4411 | |
|
| 2.7764 | 63.53 | 3050 | 3.2569 | 0.4419 | |
|
| 2.7819 | 64.58 | 3100 | 3.2492 | 0.4434 | |
|
| 2.7672 | 65.62 | 3150 | 3.2494 | 0.4433 | |
|
| 2.7629 | 66.66 | 3200 | 3.2410 | 0.4443 | |
|
| 2.747 | 67.7 | 3250 | 3.2368 | 0.4446 | |
|
| 2.7303 | 68.74 | 3300 | 3.2246 | 0.4460 | |
|
| 2.7461 | 69.78 | 3350 | 3.2212 | 0.4462 | |
|
| 2.7179 | 70.82 | 3400 | 3.2217 | 0.4470 | |
|
| 2.7184 | 71.86 | 3450 | 3.2132 | 0.4479 | |
|
| 2.7077 | 72.9 | 3500 | 3.2086 | 0.4487 | |
|
| 2.6916 | 73.95 | 3550 | 3.2057 | 0.4482 | |
|
| 2.6934 | 74.99 | 3600 | 3.2010 | 0.4495 | |
|
| 2.8585 | 76.04 | 3650 | 3.1980 | 0.4497 | |
|
| 2.8559 | 77.08 | 3700 | 3.1940 | 0.4503 | |
|
| 2.8519 | 78.12 | 3750 | 3.1940 | 0.4506 | |
|
| 2.8391 | 79.16 | 3800 | 3.1897 | 0.4509 | |
|
| 2.845 | 80.21 | 3850 | 3.1858 | 0.4510 | |
|
| 2.6636 | 81.25 | 3900 | 3.1819 | 0.4518 | |
|
| 2.6569 | 82.29 | 3950 | 3.1834 | 0.4517 | |
|
| 2.647 | 83.33 | 4000 | 3.1798 | 0.4517 | |
|
| 2.6665 | 84.37 | 4050 | 3.1786 | 0.4525 | |
|
| 2.6382 | 85.41 | 4100 | 3.1733 | 0.4525 | |
|
| 2.6346 | 86.45 | 4150 | 3.1700 | 0.4532 | |
|
| 2.6457 | 87.49 | 4200 | 3.1714 | 0.4529 | |
|
| 2.6328 | 88.53 | 4250 | 3.1686 | 0.4537 | |
|
| 2.6429 | 89.58 | 4300 | 3.1715 | 0.4534 | |
|
| 2.6369 | 90.62 | 4350 | 3.1687 | 0.4538 | |
|
| 2.628 | 91.66 | 4400 | 3.1651 | 0.4539 | |
|
| 2.6373 | 92.7 | 4450 | 3.1660 | 0.4539 | |
|
| 2.6357 | 93.74 | 4500 | 3.1662 | 0.4537 | |
|
| 2.6302 | 94.78 | 4550 | 3.1695 | 0.4533 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.24.0 |
|
- Pytorch 1.11.0+cu113 |
|
- Datasets 2.6.1 |
|
- Tokenizers 0.12.1 |
|
|
