---
license:
- apache-2.0
- bsd-3-clause
tags:
- summarization
- summary
- booksum
- long-document
- long-form
- tglobal-xl
- XL
- 8bit
- quantized
datasets:
- kmfoda/booksum
metrics:
- rouge
inference: false
pipeline_tag: summarization
---


# long-t5-tglobal-xl-16384-book-summary: 8-bit quantized version

<a href="https://colab.research.google.com/gist/pszemraj/c19e32baf876deb866c31cd46c86e893/long-t5-xl-accelerate-test.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

This is an 8-bit quantized version of the `pszemraj/long-t5-tglobal-xl-16384-book-summary` model, The model has been compressed using `bitsandbytes` and can be loaded with low memory usage.

Refer to the [original model](https://huggingface.co/pszemraj/long-t5-tglobal-xl-16384-book-summary) for all details about the model architecture and training process. For more information on loading 8-bit models, refer to the `4.28.0` [release information](https://github.com/huggingface/transformers/releases/tag/v4.28.0) and the [example repository](https://huggingface.co/ybelkada/bloom-1b7-8bit).

- The total size of the model is only ~3.5 GB (vs original 12 GB)
- Enables low-RAM loading, making it easier to use in memory-limited environments like Colab
- Requires `bitsandbytes` - AFAIK at time of writing, only works on GPU


## Basic Usage

To use the model, install or upgrade `transformers`, `accelerate`, and `bitsandbytes`. Make sure to have `transformers>=4.28.0` and `bitsandbytes>0.37.2`.

```bash
pip install -U -q transformers bitsandbytes accelerate
```

Load the model with `AutoTokenizer` and `AutoModelForSeq2SeqLM`:

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_name = "pszemraj/long-t5-tglobal-xl-16384-book-summary-8bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
```

## More information about long-t5-tglobal-xl-16384-book-summary

- This is an 8-bit quantized version of `pszemraj/long-t5-tglobal-xl-16384-book-summary`.
  - It generalizes reasonably well to academic and narrative text.
  - The XL checkpoint typically generates summaries that are considerably better from a human evaluation perspective.