--- license: mit tags: - generated_from_trainer model-index: - name: BERiT_2000_custom_architecture_relu_40_epochs results: [] --- # BERiT_2000_custom_architecture_relu_40_epochs This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 6.3968 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0005 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - num_epochs: 40 - label_smoothing_factor: 0.2 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:------:|:---------------:| | 14.8756 | 0.19 | 500 | 8.3404 | | 7.8098 | 0.39 | 1000 | 7.3110 | | 7.2696 | 0.58 | 1500 | 7.1646 | | 7.1277 | 0.77 | 2000 | 7.0953 | | 7.0939 | 0.97 | 2500 | 7.0701 | | 7.0621 | 1.16 | 3000 | 7.0003 | | 7.0236 | 1.36 | 3500 | 6.9189 | | 6.9898 | 1.55 | 4000 | 6.8536 | | 6.9625 | 1.74 | 4500 | 6.8450 | | 6.9125 | 1.94 | 5000 | 6.7799 | | 6.9115 | 2.13 | 5500 | 6.8028 | | 6.8954 | 2.32 | 6000 | 6.7288 | | 6.8289 | 2.52 | 6500 | 6.7664 | | 6.855 | 2.71 | 7000 | 6.7064 | | 6.8134 | 2.9 | 7500 | 6.7155 | | 6.7907 | 3.1 | 8000 | 6.7050 | | 6.791 | 3.29 | 8500 | 6.6695 | | 6.7659 | 3.49 | 9000 | 6.6815 | | 6.775 | 3.68 | 9500 | 6.6449 | | 6.7508 | 3.87 | 10000 | 6.6684 | | 6.7627 | 4.07 | 10500 | 6.6397 | | 6.7229 | 4.26 | 11000 | 6.6417 | | 6.7336 | 4.45 | 11500 | 6.6824 | | 6.7138 | 4.65 | 12000 | 6.6252 | | 6.7123 | 4.84 | 12500 | 6.6374 | | 6.703 | 5.03 | 13000 | 6.6400 | | 6.7054 | 5.23 | 13500 | 6.6264 | | 6.6978 | 5.42 | 14000 | 6.6094 | | 6.6944 | 5.62 | 14500 | 6.6627 | | 6.6857 | 5.81 | 15000 | 6.6363 | | 6.694 | 6.0 | 15500 | 6.6026 | | 6.6882 | 6.2 | 16000 | 6.6123 | | 6.6694 | 6.39 | 16500 | 6.5918 | | 6.6781 | 6.58 | 17000 | 6.6201 | | 6.6605 | 6.78 | 17500 | 6.6316 | | 6.6435 | 6.97 | 18000 | 6.5789 | | 6.6658 | 7.16 | 18500 | 6.5999 | | 6.6551 | 7.36 | 19000 | 6.5425 | | 6.6603 | 7.55 | 19500 | 6.5790 | | 6.6589 | 7.75 | 20000 | 6.5767 | | 6.675 | 7.94 | 20500 | 6.6005 | | 6.6362 | 8.13 | 21000 | 6.5962 | | 6.6391 | 8.33 | 21500 | 6.5716 | | 6.6379 | 8.52 | 22000 | 6.5830 | | 6.6164 | 8.71 | 22500 | 6.6137 | | 6.638 | 8.91 | 23000 | 6.5877 | | 6.6255 | 9.1 | 23500 | 6.6197 | | 6.6284 | 9.3 | 24000 | 6.5573 | | 6.6198 | 9.49 | 24500 | 6.5717 | | 6.6025 | 9.68 | 25000 | 6.5627 | | 6.6334 | 9.88 | 25500 | 6.5902 | | 6.6305 | 10.07 | 26000 | 6.5628 | | 6.5797 | 10.26 | 26500 | 6.5625 | | 6.5906 | 10.46 | 27000 | 6.5808 | | 6.5904 | 10.65 | 27500 | 6.5690 | | 6.5935 | 10.84 | 28000 | 6.5845 | | 6.6231 | 11.04 | 28500 | 6.5282 | | 6.5923 | 11.23 | 29000 | 6.6107 | | 6.6136 | 11.43 | 29500 | 6.5475 | | 6.5954 | 11.62 | 30000 | 6.5823 | | 6.5821 | 11.81 | 30500 | 6.5721 | | 6.5993 | 12.01 | 31000 | 6.5492 | | 6.5584 | 12.2 | 31500 | 6.4938 | | 6.5886 | 12.39 | 32000 | 6.6026 | | 6.5625 | 12.59 | 32500 | 6.5902 | | 6.572 | 12.78 | 33000 | 6.5436 | | 6.5807 | 12.97 | 33500 | 6.5588 | | 6.5853 | 13.17 | 34000 | 6.5555 | | 6.5727 | 13.36 | 34500 | 6.5606 | | 6.5456 | 13.56 | 35000 | 6.5386 | | 6.5538 | 13.75 | 35500 | 6.5712 | | 6.5456 | 13.94 | 36000 | 6.5582 | | 6.5734 | 14.14 | 36500 | 6.4951 | | 6.5639 | 14.33 | 37000 | 6.5323 | | 6.5712 | 14.52 | 37500 | 6.5049 | | 6.5739 | 14.72 | 38000 | 6.5523 | | 6.5534 | 14.91 | 38500 | 6.5188 | | 6.5401 | 15.1 | 39000 | 6.5968 | | 6.5456 | 15.3 | 39500 | 6.5413 | | 6.5555 | 15.49 | 40000 | 6.5347 | | 6.538 | 15.69 | 40500 | 6.5180 | | 6.537 | 15.88 | 41000 | 6.5372 | | 6.537 | 16.07 | 41500 | 6.5514 | | 6.5445 | 16.27 | 42000 | 6.5242 | | 6.5285 | 16.46 | 42500 | 6.5071 | | 6.5046 | 16.65 | 43000 | 6.5342 | | 6.5609 | 16.85 | 43500 | 6.5329 | | 6.527 | 17.04 | 44000 | 6.5569 | | 6.5199 | 17.23 | 44500 | 6.5438 | | 6.5328 | 17.43 | 45000 | 6.5380 | | 6.5183 | 17.62 | 45500 | 6.5273 | | 6.5349 | 17.82 | 46000 | 6.5209 | | 6.5283 | 18.01 | 46500 | 6.4884 | | 6.5111 | 18.2 | 47000 | 6.5036 | | 6.4895 | 18.4 | 47500 | 6.5675 | | 6.5308 | 18.59 | 48000 | 6.5378 | | 6.5159 | 18.78 | 48500 | 6.4792 | | 6.4875 | 18.98 | 49000 | 6.4846 | | 6.5076 | 19.17 | 49500 | 6.5203 | | 6.4991 | 19.36 | 50000 | 6.5007 | | 6.5269 | 19.56 | 50500 | 6.4796 | | 6.4887 | 19.75 | 51000 | 6.5197 | | 6.4995 | 19.95 | 51500 | 6.5009 | | 6.4762 | 20.14 | 52000 | 6.5049 | | 6.4872 | 20.33 | 52500 | 6.4880 | | 6.5117 | 20.53 | 53000 | 6.4917 | | 6.5035 | 20.72 | 53500 | 6.4791 | | 6.4784 | 20.91 | 54000 | 6.4771 | | 6.4749 | 21.11 | 54500 | 6.5230 | | 6.4867 | 21.3 | 55000 | 6.4954 | | 6.4921 | 21.49 | 55500 | 6.5079 | | 6.4587 | 21.69 | 56000 | 6.5309 | | 6.4839 | 21.88 | 56500 | 6.4476 | | 6.5011 | 22.08 | 57000 | 6.5025 | | 6.471 | 22.27 | 57500 | 6.5122 | | 6.4689 | 22.46 | 58000 | 6.4689 | | 6.4764 | 22.66 | 58500 | 6.5073 | | 6.4764 | 22.85 | 59000 | 6.4741 | | 6.4751 | 23.04 | 59500 | 6.4978 | | 6.4823 | 23.24 | 60000 | 6.4857 | | 6.4594 | 23.43 | 60500 | 6.4817 | | 6.4795 | 23.63 | 61000 | 6.5292 | | 6.4565 | 23.82 | 61500 | 6.4684 | | 6.4627 | 24.01 | 62000 | 6.4900 | | 6.4542 | 24.21 | 62500 | 6.4373 | | 6.4692 | 24.4 | 63000 | 6.4787 | | 6.4772 | 24.59 | 63500 | 6.4553 | | 6.4613 | 24.79 | 64000 | 6.4695 | | 6.4673 | 24.98 | 64500 | 6.5077 | | 6.466 | 25.17 | 65000 | 6.4919 | | 6.4595 | 25.37 | 65500 | 6.4451 | | 6.444 | 25.56 | 66000 | 6.4750 | | 6.438 | 25.76 | 66500 | 6.4672 | | 6.4499 | 25.95 | 67000 | 6.4358 | | 6.4578 | 26.14 | 67500 | 6.4762 | | 6.4701 | 26.34 | 68000 | 6.4462 | | 6.4296 | 26.53 | 68500 | 6.4879 | | 6.4305 | 26.72 | 69000 | 6.4519 | | 6.443 | 26.92 | 69500 | 6.4530 | | 6.4571 | 27.11 | 70000 | 6.4564 | | 6.4477 | 27.3 | 70500 | 6.4557 | | 6.443 | 27.5 | 71000 | 6.4862 | | 6.4429 | 27.69 | 71500 | 6.4498 | | 6.4374 | 27.89 | 72000 | 6.4225 | | 6.4363 | 28.08 | 72500 | 6.4723 | | 6.4127 | 28.27 | 73000 | 6.4733 | | 6.4116 | 28.47 | 73500 | 6.4499 | | 6.4312 | 28.66 | 74000 | 6.4600 | | 6.4251 | 28.85 | 74500 | 6.4451 | | 6.4318 | 29.05 | 75000 | 6.4337 | | 6.4432 | 29.24 | 75500 | 6.4713 | | 6.4183 | 29.43 | 76000 | 6.4699 | | 6.4109 | 29.63 | 76500 | 6.4591 | | 6.3939 | 29.82 | 77000 | 6.4768 | | 6.4194 | 30.02 | 77500 | 6.4786 | | 6.4262 | 30.21 | 78000 | 6.4407 | | 6.4392 | 30.4 | 78500 | 6.4202 | | 6.4311 | 30.6 | 79000 | 6.4361 | | 6.3963 | 30.79 | 79500 | 6.4346 | | 6.3872 | 30.98 | 80000 | 6.3810 | | 6.4277 | 31.18 | 80500 | 6.4451 | | 6.4112 | 31.37 | 81000 | 6.4243 | | 6.4202 | 31.56 | 81500 | 6.4502 | | 6.444 | 31.76 | 82000 | 6.4572 | | 6.4066 | 31.95 | 82500 | 6.4033 | | 6.4101 | 32.15 | 83000 | 6.4154 | | 6.3985 | 32.34 | 83500 | 6.4377 | | 6.4294 | 32.53 | 84000 | 6.4392 | | 6.397 | 32.73 | 84500 | 6.4387 | | 6.4217 | 32.92 | 85000 | 6.4305 | | 6.4061 | 33.11 | 85500 | 6.4541 | | 6.4014 | 33.31 | 86000 | 6.4173 | | 6.4223 | 33.5 | 86500 | 6.4403 | | 6.3953 | 33.69 | 87000 | 6.4333 | | 6.4135 | 33.89 | 87500 | 6.4183 | | 6.3955 | 34.08 | 88000 | 6.3958 | | 6.4064 | 34.28 | 88500 | 6.3913 | | 6.3997 | 34.47 | 89000 | 6.4330 | | 6.4212 | 34.66 | 89500 | 6.3955 | | 6.3957 | 34.86 | 90000 | 6.4438 | | 6.3936 | 35.05 | 90500 | 6.4382 | | 6.3927 | 35.24 | 91000 | 6.4055 | | 6.3972 | 35.44 | 91500 | 6.4006 | | 6.4137 | 35.63 | 92000 | 6.4245 | | 6.3947 | 35.82 | 92500 | 6.4057 | | 6.3798 | 36.02 | 93000 | 6.4006 | | 6.4011 | 36.21 | 93500 | 6.3943 | | 6.4012 | 36.41 | 94000 | 6.3766 | | 6.3961 | 36.6 | 94500 | 6.4260 | | 6.3819 | 36.79 | 95000 | 6.3801 | | 6.3795 | 36.99 | 95500 | 6.4019 | | 6.3954 | 37.18 | 96000 | 6.4387 | | 6.3874 | 37.37 | 96500 | 6.4477 | | 6.3844 | 37.57 | 97000 | 6.4177 | | 6.3898 | 37.76 | 97500 | 6.4213 | | 6.3855 | 37.96 | 98000 | 6.3838 | | 6.3825 | 38.15 | 98500 | 6.4048 | | 6.3615 | 38.34 | 99000 | 6.4636 | | 6.392 | 38.54 | 99500 | 6.4197 | | 6.3773 | 38.73 | 100000 | 6.4505 | | 6.3834 | 38.92 | 100500 | 6.3889 | | 6.3846 | 39.12 | 101000 | 6.4394 | | 6.376 | 39.31 | 101500 | 6.3923 | | 6.3699 | 39.5 | 102000 | 6.4025 | | 6.3826 | 39.7 | 102500 | 6.3951 | | 6.373 | 39.89 | 103000 | 6.3968 | ### Framework versions - Transformers 4.24.0 - Pytorch 1.12.1+cu113 - Datasets 2.7.1 - Tokenizers 0.13.2