--- tags: - generated_from_trainer metrics: - accuracy model-index: - name: roberta-tiny-2l-10M results: [] --- # roberta-tiny-2l-10M This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 3.1695 - Accuracy: 0.4534 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0004 - train_batch_size: 16 - eval_batch_size: 32 - seed: 42 - gradient_accumulation_steps: 32 - total_train_batch_size: 512 - optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 50 - num_epochs: 100.0 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | Accuracy | |:-------------:|:-----:|:----:|:---------------:|:--------:| | 7.7619 | 1.04 | 50 | 7.2338 | 0.0748 | | 7.0524 | 2.08 | 100 | 6.6252 | 0.1331 | | 6.8423 | 3.12 | 150 | 6.4622 | 0.1463 | | 6.7298 | 4.16 | 200 | 6.3971 | 0.1488 | | 6.669 | 5.21 | 250 | 6.3628 | 0.1519 | | 6.2038 | 6.25 | 300 | 6.3371 | 0.1518 | | 6.1783 | 7.29 | 350 | 6.3115 | 0.1532 | | 6.1459 | 8.33 | 400 | 6.2922 | 0.1530 | | 6.1096 | 9.37 | 450 | 6.2696 | 0.1536 | | 6.0745 | 10.41 | 500 | 6.2545 | 0.1541 | | 6.0689 | 11.45 | 550 | 6.2496 | 0.1533 | | 6.0562 | 12.49 | 600 | 6.2313 | 0.1542 | | 6.0324 | 13.53 | 650 | 6.2248 | 0.1536 | | 5.9907 | 14.58 | 700 | 6.2179 | 0.1544 | | 5.9683 | 15.62 | 750 | 6.1832 | 0.1545 | | 5.9236 | 16.66 | 800 | 6.1413 | 0.1550 | | 5.8808 | 17.7 | 850 | 6.0900 | 0.1558 | | 5.8392 | 18.74 | 900 | 6.0543 | 0.1566 | | 5.7962 | 19.78 | 950 | 6.0222 | 0.1575 | | 5.7473 | 20.82 | 1000 | 5.9471 | 0.1617 | | 5.5787 | 21.86 | 1050 | 5.7038 | 0.1891 | | 5.2316 | 22.9 | 1100 | 5.2708 | 0.2382 | | 4.6613 | 23.95 | 1150 | 4.7075 | 0.2975 | | 4.3006 | 24.99 | 1200 | 4.4180 | 0.3222 | | 4.3754 | 26.04 | 1250 | 4.2383 | 0.3385 | | 4.2531 | 27.08 | 1300 | 4.1157 | 0.3491 | | 4.0987 | 28.12 | 1350 | 4.0197 | 0.3578 | | 4.0045 | 29.16 | 1400 | 3.9504 | 0.3656 | | 3.9145 | 30.21 | 1450 | 3.8819 | 0.3718 | | 3.5808 | 31.25 | 1500 | 3.8279 | 0.3781 | | 3.5354 | 32.29 | 1550 | 3.7830 | 0.3826 | | 3.4788 | 33.33 | 1600 | 3.7400 | 0.3872 | | 3.4315 | 34.37 | 1650 | 3.7028 | 0.3911 | | 3.3906 | 35.41 | 1700 | 3.6629 | 0.3956 | | 3.3508 | 36.45 | 1750 | 3.6344 | 0.3984 | | 3.288 | 37.49 | 1800 | 3.6046 | 0.4019 | | 3.2678 | 38.53 | 1850 | 3.5799 | 0.4053 | | 3.2382 | 39.58 | 1900 | 3.5549 | 0.4074 | | 3.2151 | 40.62 | 1950 | 3.5285 | 0.4103 | | 3.1777 | 41.66 | 2000 | 3.5069 | 0.4132 | | 3.1499 | 42.7 | 2050 | 3.4917 | 0.4150 | | 3.131 | 43.74 | 2100 | 3.4701 | 0.4168 | | 3.0942 | 44.78 | 2150 | 3.4530 | 0.4189 | | 3.0683 | 45.82 | 2200 | 3.4320 | 0.4212 | | 3.0363 | 46.86 | 2250 | 3.4195 | 0.4227 | | 3.0264 | 47.9 | 2300 | 3.4046 | 0.4249 | | 3.0079 | 48.95 | 2350 | 3.3874 | 0.4267 | | 2.9869 | 49.99 | 2400 | 3.3792 | 0.4277 | | 3.1592 | 51.04 | 2450 | 3.3655 | 0.4289 | | 3.1353 | 52.08 | 2500 | 3.3548 | 0.4310 | | 3.1257 | 53.12 | 2550 | 3.3489 | 0.4308 | | 3.0822 | 54.16 | 2600 | 3.3353 | 0.4327 | | 3.0771 | 55.21 | 2650 | 3.3220 | 0.4341 | | 2.8639 | 56.25 | 2700 | 3.3119 | 0.4354 | | 2.8477 | 57.29 | 2750 | 3.3104 | 0.4360 | | 2.8373 | 58.33 | 2800 | 3.2954 | 0.4378 | | 2.818 | 59.37 | 2850 | 3.2935 | 0.4381 | | 2.8137 | 60.41 | 2900 | 3.2786 | 0.4394 | | 2.7985 | 61.45 | 2950 | 3.2747 | 0.4401 | | 2.7936 | 62.49 | 3000 | 3.2668 | 0.4411 | | 2.7764 | 63.53 | 3050 | 3.2569 | 0.4419 | | 2.7819 | 64.58 | 3100 | 3.2492 | 0.4434 | | 2.7672 | 65.62 | 3150 | 3.2494 | 0.4433 | | 2.7629 | 66.66 | 3200 | 3.2410 | 0.4443 | | 2.747 | 67.7 | 3250 | 3.2368 | 0.4446 | | 2.7303 | 68.74 | 3300 | 3.2246 | 0.4460 | | 2.7461 | 69.78 | 3350 | 3.2212 | 0.4462 | | 2.7179 | 70.82 | 3400 | 3.2217 | 0.4470 | | 2.7184 | 71.86 | 3450 | 3.2132 | 0.4479 | | 2.7077 | 72.9 | 3500 | 3.2086 | 0.4487 | | 2.6916 | 73.95 | 3550 | 3.2057 | 0.4482 | | 2.6934 | 74.99 | 3600 | 3.2010 | 0.4495 | | 2.8585 | 76.04 | 3650 | 3.1980 | 0.4497 | | 2.8559 | 77.08 | 3700 | 3.1940 | 0.4503 | | 2.8519 | 78.12 | 3750 | 3.1940 | 0.4506 | | 2.8391 | 79.16 | 3800 | 3.1897 | 0.4509 | | 2.845 | 80.21 | 3850 | 3.1858 | 0.4510 | | 2.6636 | 81.25 | 3900 | 3.1819 | 0.4518 | | 2.6569 | 82.29 | 3950 | 3.1834 | 0.4517 | | 2.647 | 83.33 | 4000 | 3.1798 | 0.4517 | | 2.6665 | 84.37 | 4050 | 3.1786 | 0.4525 | | 2.6382 | 85.41 | 4100 | 3.1733 | 0.4525 | | 2.6346 | 86.45 | 4150 | 3.1700 | 0.4532 | | 2.6457 | 87.49 | 4200 | 3.1714 | 0.4529 | | 2.6328 | 88.53 | 4250 | 3.1686 | 0.4537 | | 2.6429 | 89.58 | 4300 | 3.1715 | 0.4534 | | 2.6369 | 90.62 | 4350 | 3.1687 | 0.4538 | | 2.628 | 91.66 | 4400 | 3.1651 | 0.4539 | | 2.6373 | 92.7 | 4450 | 3.1660 | 0.4539 | | 2.6357 | 93.74 | 4500 | 3.1662 | 0.4537 | | 2.6302 | 94.78 | 4550 | 3.1695 | 0.4533 | ### Framework versions - Transformers 4.24.0 - Pytorch 1.11.0+cu113 - Datasets 2.6.1 - Tokenizers 0.12.1