arxiv-summarization-t5-small-2022-09-05

This model is a fine-tuned version of t5-small on the ccdv/arxiv-summarization dataset. It achieves the following results on the evaluation set:

• Loss: 2.2994
• Rouge1: 37.7982
• Rouge2: 12.5571
• Rougel: 24.9325
• Rougelsum: 32.9189
• Gen Len: 122.0061

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
2.5925	0.39	10000	2.4566	17.8432	6.6779	14.2303	16.1952	19.0
2.518	0.79	20000	2.3868	18.0354	6.8565	14.3552	16.3664	19.0
2.4587	1.18	30000	2.3600	18.2076	6.9618	14.5349	16.5626	19.0
2.4365	1.58	40000	2.3295	18.3579	7.0312	14.6145	16.6845	19.0
2.4306	1.97	50000	2.3190	18.4551	7.0861	14.6879	16.7627	19.0
2.4005	2.37	60000	2.3056	18.3521	7.0496	14.6413	16.6832	19.0
2.396	2.76	70000	2.3012	18.348	7.0439	14.6509	16.6994	19.0

Framework versions

• Transformers 4.22.0.dev0
• Pytorch 1.12.1+cu102
• Datasets 2.4.0
• Tokenizers 0.12.1