roberta-large-nsp-1000-1e-06-8

This model is a fine-tuned version of roberta-large on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5884

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 32
• eval_batch_size: 1024
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	32	0.6935
No log	2.0	64	0.6888
No log	3.0	96	0.6834
No log	4.0	128	0.6600
No log	5.0	160	0.6272
No log	6.0	192	0.6098
0.6812	7.0	224	0.5968
0.6812	8.0	256	0.5925
0.6812	9.0	288	0.5899
0.6812	10.0	320	0.5873
0.6812	11.0	352	0.5866
0.6812	12.0	384	0.5870
0.6056	13.0	416	0.5884
0.6056	14.0	448	0.5889
0.6056	15.0	480	0.5887
0.6056	16.0	512	0.5884

Framework versions

• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1