--- tags: - generated_from_trainer metrics: - accuracy model-index: - name: roberta-tiny-10M results: [] --- # roberta-tiny-10M This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 2.7391 - Accuracy: 0.5148 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0004 - train_batch_size: 16 - eval_batch_size: 32 - seed: 42 - gradient_accumulation_steps: 32 - total_train_batch_size: 512 - optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 50 - num_epochs: 100.0 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | Accuracy | |:-------------:|:-----:|:----:|:---------------:|:--------:| | 7.8031 | 1.04 | 50 | 7.3560 | 0.0606 | | 7.1948 | 2.08 | 100 | 6.7374 | 0.1182 | | 6.8927 | 3.12 | 150 | 6.5022 | 0.1415 | | 6.7339 | 4.16 | 200 | 6.4005 | 0.1483 | | 6.6609 | 5.21 | 250 | 6.3535 | 0.1510 | | 6.1972 | 6.25 | 300 | 6.3324 | 0.1519 | | 6.1685 | 7.29 | 350 | 6.3029 | 0.1528 | | 6.1302 | 8.33 | 400 | 6.2828 | 0.1521 | | 6.093 | 9.37 | 450 | 6.2568 | 0.1536 | | 6.0543 | 10.41 | 500 | 6.2430 | 0.1544 | | 6.0479 | 11.45 | 550 | 6.2346 | 0.1541 | | 6.0372 | 12.49 | 600 | 6.2232 | 0.1546 | | 6.0127 | 13.53 | 650 | 6.2139 | 0.1541 | | 5.968 | 14.58 | 700 | 6.2053 | 0.1547 | | 5.9635 | 15.62 | 750 | 6.1996 | 0.1549 | | 5.9479 | 16.66 | 800 | 6.1953 | 0.1548 | | 5.9371 | 17.7 | 850 | 6.1887 | 0.1545 | | 5.9046 | 18.74 | 900 | 6.1613 | 0.1545 | | 5.8368 | 19.78 | 950 | 6.0952 | 0.1557 | | 5.7914 | 20.82 | 1000 | 6.0330 | 0.1569 | | 5.7026 | 21.86 | 1050 | 5.9430 | 0.1612 | | 5.491 | 22.9 | 1100 | 5.6100 | 0.1974 | | 4.9289 | 23.95 | 1150 | 4.9607 | 0.2702 | | 4.5214 | 24.99 | 1200 | 4.5795 | 0.3051 | | 4.5663 | 26.04 | 1250 | 4.3454 | 0.3265 | | 4.3717 | 27.08 | 1300 | 4.1738 | 0.3412 | | 4.1483 | 28.12 | 1350 | 4.0336 | 0.3555 | | 3.9988 | 29.16 | 1400 | 3.9180 | 0.3677 | | 3.8695 | 30.21 | 1450 | 3.8108 | 0.3782 | | 3.5017 | 31.25 | 1500 | 3.7240 | 0.3879 | | 3.4311 | 32.29 | 1550 | 3.6426 | 0.3974 | | 3.3517 | 33.33 | 1600 | 3.5615 | 0.4068 | | 3.2856 | 34.37 | 1650 | 3.4915 | 0.4156 | | 3.227 | 35.41 | 1700 | 3.4179 | 0.4255 | | 3.1675 | 36.45 | 1750 | 3.3636 | 0.4325 | | 3.0908 | 37.49 | 1800 | 3.3083 | 0.4394 | | 3.0561 | 38.53 | 1850 | 3.2572 | 0.4473 | | 3.0139 | 39.58 | 1900 | 3.2159 | 0.4525 | | 2.9837 | 40.62 | 1950 | 3.1789 | 0.4575 | | 2.9387 | 41.66 | 2000 | 3.1431 | 0.4618 | | 2.9034 | 42.7 | 2050 | 3.1163 | 0.4654 | | 2.8822 | 43.74 | 2100 | 3.0842 | 0.4694 | | 2.836 | 44.78 | 2150 | 3.0583 | 0.4727 | | 2.8129 | 45.82 | 2200 | 3.0359 | 0.4760 | | 2.7733 | 46.86 | 2250 | 3.0173 | 0.4776 | | 2.7589 | 47.9 | 2300 | 2.9978 | 0.4812 | | 2.7378 | 48.95 | 2350 | 2.9788 | 0.4831 | | 2.7138 | 49.99 | 2400 | 2.9674 | 0.4844 | | 2.8692 | 51.04 | 2450 | 2.9476 | 0.4874 | | 2.8462 | 52.08 | 2500 | 2.9342 | 0.4893 | | 2.8312 | 53.12 | 2550 | 2.9269 | 0.4900 | | 2.7834 | 54.16 | 2600 | 2.9111 | 0.4917 | | 2.7822 | 55.21 | 2650 | 2.8987 | 0.4934 | | 2.584 | 56.25 | 2700 | 2.8844 | 0.4949 | | 2.5668 | 57.29 | 2750 | 2.8808 | 0.4965 | | 2.5536 | 58.33 | 2800 | 2.8640 | 0.4982 | | 2.5403 | 59.37 | 2850 | 2.8606 | 0.4982 | | 2.5294 | 60.41 | 2900 | 2.8441 | 0.5008 | | 2.513 | 61.45 | 2950 | 2.8402 | 0.5013 | | 2.5105 | 62.49 | 3000 | 2.8316 | 0.5022 | | 2.4897 | 63.53 | 3050 | 2.8237 | 0.5027 | | 2.4974 | 64.58 | 3100 | 2.8187 | 0.5040 | | 2.4799 | 65.62 | 3150 | 2.8129 | 0.5044 | | 2.4741 | 66.66 | 3200 | 2.8056 | 0.5057 | | 2.4582 | 67.7 | 3250 | 2.8025 | 0.5061 | | 2.4389 | 68.74 | 3300 | 2.7913 | 0.5076 | | 2.4539 | 69.78 | 3350 | 2.7881 | 0.5072 | | 2.4252 | 70.82 | 3400 | 2.7884 | 0.5082 | | 2.4287 | 71.86 | 3450 | 2.7784 | 0.5093 | | 2.4131 | 72.9 | 3500 | 2.7782 | 0.5099 | | 2.4016 | 73.95 | 3550 | 2.7724 | 0.5098 | | 2.3998 | 74.99 | 3600 | 2.7659 | 0.5111 | | 2.5475 | 76.04 | 3650 | 2.7650 | 0.5108 | | 2.5443 | 77.08 | 3700 | 2.7620 | 0.5117 | | 2.5381 | 78.12 | 3750 | 2.7631 | 0.5115 | | 2.5269 | 79.16 | 3800 | 2.7578 | 0.5122 | | 2.5288 | 80.21 | 3850 | 2.7540 | 0.5124 | | 2.3669 | 81.25 | 3900 | 2.7529 | 0.5125 | | 2.3631 | 82.29 | 3950 | 2.7498 | 0.5132 | | 2.3499 | 83.33 | 4000 | 2.7454 | 0.5136 | | 2.3726 | 84.37 | 4050 | 2.7446 | 0.5141 | | 2.3411 | 85.41 | 4100 | 2.7403 | 0.5144 | | 2.3321 | 86.45 | 4150 | 2.7372 | 0.5146 | | 2.3456 | 87.49 | 4200 | 2.7389 | 0.5146 | | 2.3372 | 88.53 | 4250 | 2.7384 | 0.5151 | | 2.343 | 89.58 | 4300 | 2.7398 | 0.5144 | ### Framework versions - Transformers 4.24.0 - Pytorch 1.11.0+cu113 - Datasets 2.6.1 - Tokenizers 0.12.1