long-t5-tglobal-large-pubmed-3k-booksum-16384-WIP
NOTE: this is still a work-in-progress (WIP) and not completed/converged by any means, but sharing to maybe save some time for others :)
Updates
As I update this WIP checkpoint, I will post a note here.
- July 26, 2022: add two more epochs of training, metrics starting to be almost as good as the more-tuned
base
variant - July 8, 2022: add checkpoint with ~4 epochs of training on A100, equating to approx 350 steps of functional batch size 128
- July 4, 2022: add checkpoint with six additional epochs of training with the dataset summary outputs filtered to 1024 tokens, resolving the prior issue of short summaries.
About
- a checkpoint of Stancld/longt5-tglobal-large-16384-pubmed-3k_steps trained on
kmfoda/booksum
for about 26 epochs - max input lengths during training vary between 8192 and 16384 tokens depending on GPU availability. This checkpoint was trained with 16384 tokens as the max input length for the final 10+ epochs
Comparisons
- compare to pszemraj/led-large-book-summary.
- inference API has been disabled because it's too compute-intensive :/
- Downloads last month
- 20
Dataset used to train pszemraj/long-t5-tglobal-large-pubmed-3k-booksum-16384-WIP
Evaluation results
- ROUGE-1 on kmfoda/booksumtest set verified35.997
- ROUGE-2 on kmfoda/booksumtest set verified5.927
- ROUGE-L on kmfoda/booksumtest set verified16.014
- ROUGE-LSUM on kmfoda/booksumtest set verified32.941
- loss on kmfoda/booksumtest set verified2.934
- gen_len on kmfoda/booksumtest set verified283.720
- ROUGE-1 on samsumtest set verified26.241
- ROUGE-2 on samsumtest set verified5.979
- ROUGE-L on samsumtest set verified18.747
- ROUGE-LSUM on samsumtest set verified22.557
- loss on samsumtest set verified2.878
- gen_len on samsumtest set verified47.653
- ROUGE-1 on xsumtest set verified19.321
- ROUGE-2 on xsumtest set verified2.798
- ROUGE-L on xsumtest set verified12.582
- ROUGE-LSUM on xsumtest set verified15.024
- loss on xsumtest set verified4.484
- gen_len on xsumtest set verified82.729
- ROUGE-1 on billsumtest set verified36.569
- ROUGE-2 on billsumtest set verified12.585
- ROUGE-L on billsumtest set verified22.246
- ROUGE-LSUM on billsumtest set verified30.651
- loss on billsumtest set verified2.646
- gen_len on billsumtest set verified139.040
- ROUGE-1 on launch/gov_reporttest set verified37.025
- ROUGE-2 on launch/gov_reporttest set verified9.045
- ROUGE-L on launch/gov_reporttest set verified18.052
- ROUGE-LSUM on launch/gov_reporttest set verified33.472
- loss on launch/gov_reporttest set verified3.381
- gen_len on launch/gov_reporttest set verified211.207