roberta-small-ucic

This model is a fine-tuned version of smallbenchnlp/roberta-small on the trec dataset. It achieves the following results on the evaluation set:

• Loss: 2.2140
• Accuracy: 0.542
• F1: 0.1365
• Precision: 0.1192
• Recall: 0.1703

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• num_epochs: 10
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1	Precision	Recall
3.73	0.6462	50	3.5638	0.176	0.0067	0.0039	0.0222
3.436	1.2843	100	3.3628	0.176	0.0067	0.0039	0.0222
3.2804	1.9305	150	3.1425	0.26	0.0321	0.0488	0.0520
3.0182	2.5687	200	2.9318	0.384	0.0672	0.0688	0.0930
2.8872	3.2068	250	2.7501	0.468	0.1066	0.1073	0.1278
2.7232	3.8530	300	2.6119	0.492	0.1109	0.1051	0.1370
2.5765	4.4911	350	2.4985	0.508	0.1228	0.1211	0.1477
2.4639	5.1292	400	2.4120	0.514	0.1250	0.1241	0.1502
2.4198	5.7754	450	2.3442	0.524	0.1307	0.1230	0.1575
2.332	6.4136	500	2.2935	0.53	0.1321	0.1202	0.1606
2.2779	7.0517	550	2.2594	0.536	0.1359	0.1210	0.1672
2.2767	7.6979	600	2.2363	0.538	0.1379	0.1243	0.1676
2.2479	8.3360	650	2.2224	0.542	0.1367	0.1195	0.1703
2.2666	8.9822	700	2.2161	0.542	0.1365	0.1192	0.1703
2.2366	9.6204	750	2.2140	0.542	0.1365	0.1192	0.1703

Framework versions

• Transformers 4.53.3
• Pytorch 2.6.0+cu124
• Datasets 4.8.4
• Tokenizers 0.21.4