saeedehj/led-base-finetune-xsum

led-base-16384-finetune-xsum

This model is a fine-tuned version of allenai/led-base-16384 on the xsum dataset. It achieves the following results on the evaluation set:

• Loss: 3.3325
• Rouge1: 31.3157
• Rouge2: 9.2183
• Rougel: 23.7641
• Rougelsum: 23.8202
• Gen Len: 19.89

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
No log	1.0	125	2.6311	32.5653	10.8601	25.3811	25.5187	19.84
No log	2.0	250	2.7544	31.6321	9.9595	25.0264	25.0779	19.85
No log	3.0	375	2.8261	32.0246	10.1415	25.2121	25.2632	19.89
0.1515	4.0	500	2.9240	31.6961	11.1892	25.0684	25.1019	19.92
0.1515	5.0	625	3.0229	31.1022	9.294	24.3075	24.309	19.9
0.1515	6.0	750	3.0900	31.7063	10.2344	25.1885	25.3359	19.89
0.1515	7.0	875	3.0958	31.6973	10.2856	25.5433	25.6242	19.91
0.0437	8.0	1000	3.1248	30.9445	10.3904	24.0861	24.109	19.91
0.0437	9.0	1125	3.1542	31.4694	9.4087	24.3248	24.4039	19.97
0.0437	10.0	1250	3.1986	30.428	9.6657	24.2568	24.4035	19.86
0.0437	11.0	1375	3.2040	32.3325	9.8754	25.117	25.1563	19.95
0.0229	12.0	1500	3.2044	30.8435	8.6959	23.4129	23.5211	19.99
0.0229	13.0	1625	3.2419	31.8807	9.6734	24.5748	24.6672	19.96
0.0229	14.0	1750	3.2926	31.8181	9.5238	24.3606	24.4569	19.88
0.0229	15.0	1875	3.2935	30.7551	8.9042	23.9581	24.1074	19.98
0.0107	16.0	2000	3.3219	31.3919	9.3308	24.1432	24.2162	19.93
0.0107	17.0	2125	3.3167	31.7918	9.4813	23.9672	24.0244	19.9
0.0107	18.0	2250	3.3281	31.0624	9.3608	23.6247	23.6658	19.89
0.0107	19.0	2375	3.3248	31.7832	9.5257	23.9738	24.0255	19.96
0.0063	20.0	2500	3.3325	31.3157	9.2183	23.7641	23.8202	19.89

Framework versions

• Transformers 4.30.2
• Pytorch 2.0.1+cu118
• Datasets 2.13.1
• Tokenizers 0.13.3