README.md

---
license:
  - apache-2.0
  - bsd-3-clause
tags:
  - summarization
  - summary
  - booksum
  - long-document
  - long-form
  - tglobal-xl
  - XL
datasets:
  - kmfoda/booksum
metrics:
  - rouge
inference: false
---


# long-t5-tglobal-xl-16384-book-summary: the 8-bit quantized version

This is an 8-bit quantized version of the `pszemraj/long-t5-tglobal-xl-16384-book-summary` model, The model has been compressed using `bitsandbytes` and can be loaded with low memory usage.

Refer to the [original model](https://huggingface.co/pszemraj/long-t5-tglobal-xl-16384-book-summary) for all details about the model architecture and training process. For more information on loading 8-bit models, refer to the `4.28.0` [release information](https://github.com/huggingface/transformers/releases/tag/v4.28.0) and the [example repository](https://huggingface.co/ybelkada/bloom-1b7-8bit).

- The total size of the model is only ~3.5 GB, much smaller than the original size.
- This allows for low-RAM loading, making it easier to use in memory-limited environments.
- `bitsandbytes` - AFAIK at time of writing - only works on GPU


## Basic Usage

To use the model, install or upgrade `transformers`, `accelerate`, and `bitsandbytes`. Make sure to have `transformers>=4.28.0` and `bitsandbytes>0.37.2`.

```bash
pip install -U -q transformers bitsandbytes accelerate
```

Load the model with `AutoTokenizer` and `AutoModelForSeq2SeqLM`:

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_name = "pszemraj/long-t5-tglobal-xl-16384-book-summary-8bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
```

## More information about long-t5-tglobal-xl-16384-book-summary

- This is an 8-bit quantized version of `pszemraj/long-t5-tglobal-xl-16384-book-summary`.
- It generalizes reasonably well to academic and narrative text, producing high-quality summaries.
- The XL checkpoint is used, resulting in even better summaries from a human evaluation perspective.
- A simple example/use case with the base model on ASR can be found [here](https://huggingface.co/pszemraj/long-t5-tglobal-xl-16384-book-summary/tree/main/examples/asr).
- A proof-of-concept example using the infamous Navy Seals copypasta demonstrates the model's ability to generate summaries from even short text inputs.