license:
- apache-2.0
- bsd-3-clause
tags:
- summarization
- summary
- booksum
- long-document
- long-form
- tglobal-xl
- XL
datasets:
- kmfoda/booksum
metrics:
- rouge
inference: false
model-index:
- name: pszemraj/long-t5-tglobal-xl-16384-book-summary
results:
- task:
type: summarization
name: Summarization
dataset:
name: multi_news
type: multi_news
config: default
split: test
metrics:
- type: rouge
value: 36.2043
name: ROUGE-1
verified: true
verifyToken: >-
eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzRmMmUyOTVjMmJmZTRiZDcyYzY3MTQ1MmUyNDA5NjVhYzEzYzBiNzcxYTRhMDQ3OTlhMGZjYmJlNDM1M2NjYyIsInZlcnNpb24iOjF9._uArOQ1_0znXDPXMq7unA1OHB-XbgqzzKRbFRcVUzTUJdWk26LiSa2pEEVNNmJPg6Uo7CAvONmhpEswLvl9TAg
- type: rouge
value: 8.424
name: ROUGE-2
verified: true
verifyToken: >-
eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzg0MzljYjVjYWQ3MmRkZDBlOGI5M2RiMGU0M2UwZGUzMDg2NTU0NjcwMTNiN2ZmODEzNTQ0MmEwNDA3NDA5MSIsInZlcnNpb24iOjF9.Dzj85ld6TjosQ8KyUdoadzicMLedEFrICC6Q-08O3qx28d9B9Uke1zw-VWabiuesPEDTRGbWuBgPA5vxYWUZAw
- type: rouge
value: 17.3721
name: ROUGE-L
verified: true
verifyToken: >-
eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDA3ZjZmODAwMTNlM2RlZmJlMDI5MGVkMGRkMTBjMTYzNDk5ZjFiNTY5MWE1MDUwNWI2MDE4ZDA2YWMwMmI2NCIsInZlcnNpb24iOjF9.MOV_nId0XAK1eMQssG5GN9DsitZaTrxl4jdCJnOg9EZ0-vAw227ln599YV5YfZ1OPJnWwek6rneqqyONiHn9AQ
- type: rouge
value: 32.3994
name: ROUGE-LSUM
verified: true
verifyToken: >-
eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmY3MDMwOTZjNWI0YTk1MDgwMzJkYTFiN2U5YWU0Mzc0MWRiMzc1NzZlMDhjMWUwMmY2ODI2MjI5ODBkYWUxOSIsInZlcnNpb24iOjF9._BwGIZbcA4pUBkEAL0cW-JPPta0KSoGug4Z7vogHacUz-AEhIOI5ICUldZh0pt9OK67MpUSzpShJOu3rSt5YDQ
- type: loss
value: 2.0843334197998047
name: loss
verified: true
verifyToken: >-
eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOWFhMmE5ZjA3ODM4YmVjMDMyMjk5YjNlMjA1MGMzOWY0NTRlYzk1YjZiMzQxMDMxOTMwMjFkNTdmNjM1NDcyMyIsInZlcnNpb24iOjF9.3wbXV4CIIgnfXAnnRztdOR12PwsWsEfiglQQ09K-C1EgW4gai4x9l-wTE2OZ7CTWkuk_tr4tL_uqOCXLZRMtCQ
- type: gen_len
value: 248.3572
name: gen_len
verified: true
verifyToken: >-
eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMWZhOGMwMDJjNGU2MzA2YzI1OWU1ZDY5N2NjZmM1YTA5NDg1MzUwNmU1YTBhNjQyNWYwYzA3OGNmODFjMmE2NSIsInZlcnNpb24iOjF9.Rc9u89zCdbFnjsnmq65l_JvCtUwOX_ZWapKJpTZ-rC8HxcUVfi2Ash2QfvvvxHH_YWhwklxxdnNa0HCm46qLAA
- task:
type: summarization
name: Summarization
dataset:
name: billsum
type: billsum
config: default
split: test
metrics:
- name: ROUGE-1
type: rouge
value: 41.3645
verified: true
- name: ROUGE-2
type: rouge
value: 16.144
verified: true
- name: ROUGE-L
type: rouge
value: 24.2981
verified: true
- name: ROUGE-LSUM
type: rouge
value: 35.3234
verified: true
- name: loss
type: loss
value: 1.282260775566101
verified: true
- name: gen_len
type: gen_len
value: 291.8158
verified: true
- task:
type: summarization
name: Summarization
dataset:
name: ccdv/arxiv-summarization
type: ccdv/arxiv-summarization
config: document
split: test
metrics:
- name: ROUGE-1
type: rouge
value: 36.3225
verified: true
- name: ROUGE-2
type: rouge
value: 9.3743
verified: true
- name: ROUGE-L
type: rouge
value: 19.8396
verified: true
- name: ROUGE-LSUM
type: rouge
value: 32.2532
verified: true
- name: loss
type: loss
value: 2.146871566772461
verified: true
- name: gen_len
type: gen_len
value: 186.2966
verified: true
long-t5-tglobal-xl + BookSum
Summarize long text and get a SparkNotes-esque summary of arbitrary topics!
- Generalizes reasonably well to academic & narrative text.
- This is the XL checkpoint, which from a human-evaluation perspective, produces even better summaries.
A simple example/use case with the base model on ASR is here.
Cheeky Proof-of-Concept
A summary of the infamous navy seals copypasta:
In this chapter, the monster explains how he intends to exact revenge on "the little b****" who insulted him. He tells the kiddo that he is a highly trained and experienced killer who will use his arsenal of weapons--including his access to the internet--to exact justice on the little brat.
While a somewhat crude example, try running this copypasta through other summarization models to see the difference in comprehension (despite it not even being a "long" text!)
Description
A fine-tuned version of google/long-t5-tglobal-xl on the kmfoda/booksum dataset.
Read the paper by Guo et al. here: LongT5: Efficient Text-To-Text Transformer for Long Sequences
How-To in Python
🚧
LLM.int8()appears to be compatible with summarization and does not degrade the quality of the outputs; this is a crucial enabler for using this model on standard GPUs. A PR for this is in-progress here, and this model card will be updated with instructions once done :) 🚧
Install/update transformers pip install -U transformers
Summarize text with pipeline:
import torch
from transformers import pipeline
summarizer = pipeline(
"summarization",
"pszemraj/long-t5-tglobal-xl-16384-book-summary",
device=0 if torch.cuda.is_available() else -1,
)
long_text = "Here is a lot of text I don't want to read. Replace me"
result = summarizer(long_text)
print(result[0]["summary_text"])
Pass other parameters related to beam search textgen when calling summarizer to get even higher quality results.
About
Intended uses & limitations
While this model seems to improve upon factual consistency, do not take summaries to be foolproof and check things that seem odd.
Specifically: negation statements (i.e., model says: This thing does not have [ATTRIBUTE] where instead it should have said This thing has a lot of [ATTRIBUTE]).
- I'm sure someone will write a paper on this eventually (if there isn't one already), but you can usually fact-check this by comparing a specific claim to what the surrounding sentences imply.
Training and evaluation data
kmfoda/booksum dataset on HuggingFace - read the original paper here.
- Initial fine-tuning only used input text with 12288 tokens input or less and 1024 tokens output or less (i.e. rows with longer were dropped before training) for memory reasons. Per brief analysis, summaries in the 12288-16384 range in this dataset are in the small minority
- In addition, this initial training combined the training and validation sets and trained on these in aggregate to increase the functional dataset size. Therefore, take the validation set results with a grain of salt; primary metrics should be (always) the test set.
- final phases of fine-tuning used the standard conventions of 16384 input/1024 output keeping everything (truncating longer sequences). This did not appear to change the loss/performance much.
Eval results
Official results with the model evaluator will be computed and posted here.
Please read the note above as due to training methods, validation set performance looks better than the test set results will be. The model achieves the following results on the evaluation set:
- eval_loss: 1.2756
- eval_rouge1: 41.8013
- eval_rouge2: 12.0895
- eval_rougeL: 21.6007
- eval_rougeLsum: 39.5382
- eval_gen_len: 387.2945
- eval_runtime: 13908.4995
- eval_samples_per_second: 0.107
- eval_steps_per_second: 0.027
***** predict/test metrics (initial) *****
predict_gen_len = 506.4368
predict_loss = 2.028
predict_rouge1 = 36.8815
predict_rouge2 = 8.0625
predict_rougeL = 17.6161
predict_rougeLsum = 34.9068
predict_runtime = 2:04:14.37
predict_samples = 1431
predict_samples_per_second = 0.192
predict_steps_per_second = 0.048
* evaluating big model not as easy as it seems. Doing a bit more investigating
FAQ
How can I run inference with this on CPU?
lol
How to run inference over a very long (30k+ tokens) document in batches?
See summarize.py in the code for my hf space Document Summarization :)
You can also use the same code to split a document into batches of 4096, etc., and run over those with the model. This is useful in situations where CUDA memory is limited.
How to fine-tune further?
See train with a script and the summarization scripts
Training procedure
Updates
Updates to this model/model card will be posted here as relevant. The model seems fairly converged; if updates/improvements are possible using the BookSum dataset, this repo will be updated.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0006
- train_batch_size: 1
- eval_batch_size: 1
- seed: 10350
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 32
- total_train_batch_size: 128
- total_eval_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- num_epochs: 1.0
*Prior training sessions used roughly similar parameters (learning rates were higher); multiple sessions were required as this takes eons to train.
Framework versions
- Transformers 4.25.0.dev0
- Pytorch 1.13.0+cu117
- Datasets 2.6.1
- Tokenizers 0.13.1