roberta-base-nsp-500000-1e-06-128

This model is a fine-tuned version of roberta-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1111

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 512
• eval_batch_size: 1024
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
0.2798	1.0	977	0.2218
0.2027	2.0	1954	0.1736
0.1759	3.0	2931	0.1549
0.1574	4.0	3908	0.1422
0.1464	5.0	4885	0.1356
0.1383	6.0	5862	0.1288
0.1307	7.0	6839	0.1243
0.1242	8.0	7816	0.1196
0.122	9.0	8793	0.1171
0.115	10.0	9770	0.1147
0.1113	11.0	10747	0.1136
0.1098	12.0	11724	0.1147
0.1061	13.0	12701	0.1125
0.1031	14.0	13678	0.1133
0.102	15.0	14655	0.1119
0.1005	16.0	15632	0.1107
0.0959	17.0	16609	0.1116
0.0972	18.0	17586	0.1109
0.0928	19.0	18563	0.1111

Framework versions

• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1