# long-t5-tglobal-xl + BookSum

Summarize long text and get a SparkNotes-like summary of any topic!

A simple example/use case with the base model on ASR is here.

## Cheeky Proof-of-Concept

A summary of the infamous navy seals copypasta:

In this chapter, the monster explains how he intends to exact revenge on "the little b****" who insulted him. He tells the kiddo that he is a highly trained and experienced killer who will use his arsenal of weapons--including his access to the internet--to exact justice on the little brat.

While this is a crude example, try running this copypasta through other summarization models to see the difference in comprehension (even though it's not even a "long" text!).

Contents

## Description

A fine-tuned version of google/long-t5-tglobal-xl on the kmfoda/booksum dataset.

Read the paper by Guo et al. here: LongT5: Efficient Text-To-Text Transformer for Long Sequences

## How-To in Python

install/update transformers pip install -U transformers

summarize text with pipeline:

import torch
from transformers import pipeline

summarizer = pipeline(
"summarization",
"pszemraj/long-t5-tglobal-xl-16384-book-summary",
device=0 if torch.cuda.is_available() else -1,
)
long_text = "Here is a lot of text I don't want to read. Replace me"

result = summarizer(long_text)
print(result[0]["summary_text"])


### Beyond the basics

There are two additional points to consider beyond simple inference: adjusting decoding parameters for improved performance, and quantization for reduced memory consumption.

Pass other parameters related to beam search textgen when calling summarizer to get even higher quality results.

#### LLM.int8 Quantization

alternative section title: how to get this monster to run inference on free colab runtimes

Per this PR LLM.int8 is now supported for long-t5 models. Per initial tests the summarization quality seems to hold while using significantly less memory! *

How-to: basically make sure you have pip-installed the latest GitHub repo main version of transformers, and also the bitsandbytes package.

install the latest main branch:

pip install bitsandbytes
pip install git+https://github.com/huggingface/transformers.git


load in 8-bit (voodoo magic-the good kind-completed by bitsandbytes behind the scenes)

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained(
"pszemraj/long-t5-tglobal-xl-16384-book-summary"
)

model = AutoModelForSeq2SeqLM.from_pretrained(
"pszemraj/long-t5-tglobal-xl-16384-book-summary",
device_map="auto",
)


The above is already present in the Colab demo linked at the top of the model map.

Do you like to ask questions? Great. But first, check out the how LLM.int8 works blog post by huggingface.

* More rigorous metrics-based research comparing beam-search summarization with and without LLM.int8 will take place over time.

### Intended uses & limitations

While this model seems to improve factual consistency, don't take summaries as foolproof and check things that seem odd.

Specifically: negation statements (i.e., the model says: this thing does not have [ATTRIBUTE], when instead it should have said this thing has lots of [ATTRIBUTE]).

• I'm sure someone will write a paper on this eventually (if there isn't one already), but you can usually check this by comparing a particular statement with what the surrounding sentences imply.

### Training and evaluation data

kmfoda/booksum dataset on HuggingFace - read the original paper here.

• For initial fine-tuning, only input text with 12288 input tokens or less and 1024 output tokens or less was used (i.e. lines longer than that were dropped before training) for memory reasons. After a quick analysis, summaries in the 12288-16384 range are in the small minority in this dataset.
• In addition, this initial training combined the training and validation sets and trained on them in aggregate to increase the functional dataset size. Therefore, take the validation set results with a grain of salt; primary metrics should (always) be the test set..
• The final stages of fine-tuning used the standard 16384 input/1024 output conventions, preserving the standard in/out lengths (and truncating longer sequences). This did not seem to change the loss/performance much.

### Eval results

Official results with the model evaluator will be computed and posted here.

Please read the note above, as due to the training methods, the performance on the validation set looks better than the results on the test set will be. The model achieves the following results on the evaluation set:

• eval_loss: 1.2756

• eval_rouge1: 41.8013

• eval_rouge2: 12.0895

• eval_rougeL: 21.6007

• eval_rougeLsum: 39.5382

• eval_gen_len: 387.2945

• eval_runtime: 13908.4995

• eval_samples_per_second: 0.107

• eval_steps_per_second: 0.027

***** predict/test metrics (initial) ***** predict_gen_len = 506.4368 predict_loss = 2.028 predict_rouge1 = 36.8815 predict_rouge2 = 8.0625 predict_rougeL = 17.6161 predict_rougeLsum = 34.9068 predict_runtime = 2:04:14.37 predict_samples = 1431 predict_samples_per_second = 0.192 predict_steps_per_second = 0.048

* evaluating big model not as easy as it seems. Doing a bit more investigating

## FAQ

lol

### How to run inference over a very long (30k+ tokens) document in batches?

See summarize.py in the code for my hf space Document Summarization :)

You can also use the same code to split a document into batches of 4096, etc., and iterate over them with the model. This is useful in situations where CUDA memory is limited.

Update: see the section on the textsum package below.

### Are there simpler ways to run this?

For this reason, I created a Python package utility. It's called textsum, and you can use it to load models and summarize things in a few lines of code.

pip install textsum


Use textsum in python with this model:

from textsum.summarize import Summarizer

summarizer = Summarizer(
model_name_or_path="pszemraj/long-t5-tglobal-xl-16384-book-summary"
)

long_string = "This is a long string of text that will be summarized."
out_str = summarizer.summarize_string(long_string)
print(f"summary: {out_str}")


This package provides easy-to-use interfaces for applying summarization models to text documents of arbitrary length. Currently implemented interfaces include a Python API, a CLI, and a shareable demo application.

For details, explanations, and documentation, see the README (linked above) or the wiki.

## Training procedure

Updates to this model/model card will be posted here when relevant. The model seems to be fairly converged; if updates/improvements are possible using the BookSum dataset, this repo will be updated.

### Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0006
• train_batch_size: 1
• eval_batch_size: 1
• seed: 10350
• distributed_type: multi-GPU
• num_devices: 4
• total_train_batch_size: 128
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• num_epochs: 1.0

*Prior training sessions used roughly similar parameters (learning rates were higher); multiple sessions were required as this takes eons to train.

### Framework versions

• Transformers 4.25.0.dev0
• Pytorch 1.13.0+cu117
• Datasets 2.6.1
• Tokenizers 0.13.1