t5-small-mlm-pubmed

This model is a fine-tuned version of t5-small on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.8008
• Rouge2 Precision: 0.6071
• Rouge2 Recall: 0.4566
• Rouge2 Fmeasure: 0.5079

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 40
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge2 Precision	Rouge2 Recall	Rouge2 Fmeasure
0.914	0.75	500	0.8691	0.5901	0.4357	0.4879
0.9093	1.51	1000	0.8646	0.5867	0.4372	0.488
0.895	2.26	1500	0.8618	0.5891	0.4387	0.49
0.8842	3.02	2000	0.8571	0.5899	0.4374	0.4891
0.8796	3.77	2500	0.8544	0.5903	0.4406	0.4916
0.8759	4.52	3000	0.8513	0.5921	0.4395	0.4912
0.8621	5.28	3500	0.8485	0.5934	0.4413	0.493
0.8613	6.03	4000	0.8442	0.5944	0.4428	0.4944
0.8537	6.79	4500	0.8406	0.594	0.4414	0.4932
0.8518	7.54	5000	0.8399	0.5956	0.4424	0.4945
0.8438	8.3	5500	0.8365	0.5953	0.4452	0.4964
0.8339	9.05	6000	0.8353	0.5983	0.4468	0.4983
0.8307	9.8	6500	0.8331	0.5979	0.4461	0.4976
0.8328	10.56	7000	0.8304	0.5975	0.4465	0.4979
0.8263	11.31	7500	0.8283	0.5977	0.4467	0.4981
0.8168	12.07	8000	0.8267	0.5971	0.4463	0.4976
0.8165	12.82	8500	0.8248	0.5969	0.4462	0.4976
0.8084	13.57	9000	0.8245	0.6018	0.4527	0.5035
0.8136	14.33	9500	0.8219	0.6023	0.4509	0.5023
0.8073	15.08	10000	0.8206	0.6002	0.4486	0.5001
0.808	15.84	10500	0.8185	0.6009	0.4506	0.5019
0.8027	16.59	11000	0.8173	0.5978	0.4478	0.4989
0.8061	17.35	11500	0.8169	0.6022	0.4513	0.5026
0.7922	18.1	12000	0.8152	0.6016	0.4501	0.5016
0.7928	18.85	12500	0.8141	0.6009	0.45	0.5012
0.7909	19.61	13000	0.8143	0.6019	0.4521	0.5028
0.7909	20.36	13500	0.8115	0.5997	0.4505	0.5011
0.7949	21.12	14000	0.8115	0.6043	0.4536	0.5048
0.7853	21.87	14500	0.8095	0.6033	0.4527	0.5038
0.7819	22.62	15000	0.8095	0.6054	0.4541	0.5056
0.7828	23.38	15500	0.8075	0.6036	0.453	0.5042
0.787	24.13	16000	0.8068	0.6031	0.4528	0.504
0.7739	24.89	16500	0.8072	0.6043	0.4529	0.5045
0.7782	25.64	17000	0.8073	0.606	0.4551	0.5063
0.7772	26.4	17500	0.8063	0.6055	0.4549	0.5062
0.7718	27.15	18000	0.8057	0.606	0.4546	0.5059
0.7747	27.9	18500	0.8045	0.6046	0.4543	0.5054
0.7738	28.66	19000	0.8035	0.6059	0.4549	0.506
0.7642	29.41	19500	0.8041	0.6053	0.4545	0.5058
0.7666	30.17	20000	0.8039	0.6066	0.457	0.508
0.7686	30.92	20500	0.8027	0.6075	0.4571	0.5081
0.7664	31.67	21000	0.8026	0.6062	0.4566	0.5076
0.77	32.43	21500	0.8022	0.6068	0.4571	0.5081
0.7618	33.18	22000	0.8015	0.6065	0.4563	0.5072
0.7615	33.94	22500	0.8013	0.6064	0.4565	0.5074
0.7611	34.69	23000	0.8017	0.607	0.4567	0.5078
0.7611	35.44	23500	0.8013	0.608	0.4565	0.5082
0.7604	36.2	24000	0.8012	0.6069	0.4561	0.5072
0.7599	36.95	24500	0.8013	0.6078	0.4571	0.5085
0.7542	37.71	25000	0.8016	0.6083	0.4579	0.5091
0.7637	38.46	25500	0.8009	0.6072	0.4569	0.5081
0.7596	39.22	26000	0.8008	0.6069	0.4566	0.5078
0.7604	39.97	26500	0.8008	0.6071	0.4566	0.5079

Framework versions

• Transformers 4.12.3
• Pytorch 1.9.0+cu111
• Datasets 1.15.1
• Tokenizers 0.10.3