license: apache-2.0
Paper: Adapting Language Models to Compress Contexts
Code: https://github.com/princeton-nlp/AutoCompressors
Models:
- Llama-2-7b fine-tuned models: AutoCompressor-Llama-2-7b-6k, FullAttention-Llama-2-7b-6k
- OPT-2.7b fine-tuned models: AutoCompressor-2.7b-6k, AutoCompressor-2.7b-30k, RMT-2.7b-8k, FullAttention-2.7b-4k
- OPT-1.3b fine-tuned models: AutoCompressor-1.3b-30k, RMT-1.3b-30k
FullAttention-2.7b-4k is a model fine-tuned from facebook/opt-2.7b following the context window extension method described in Adapting Language Models to Compress Contexts. The 2,048 positional embeddings of the pre-trained OPT-2.7b are duplicated and the model is fine-tuned on sequences of 4,096 tokens from 2B tokens from The Pile.
To get started, download the AutoCompressor
repository and load the model as follows:
from auto_compressor import AutoCompressorModel
model = AutoCompressorModel.from_pretrained("princeton-nlp/FullAttention-2.7b-4k")
Evaluation
We record the perplexity achieved by our OPT-2.7b models on segments of 2,048 tokens, conditioned on different amounts of context. FullAttention-2.7-4k uses full uncompressed contexts whereas AutoCompressor-2.7b-6k and RMT-2.7b-8k compress segments of 2,048 tokens into 50 summary vectors.
In-domain Evaluation
Context Tokens | 0 | 512 | 2048 | 4096 | 6144 |
---|---|---|---|---|---|
FullAttention-2.7b-4k | 6.57 | 6.15 | 5.94 | - | - |
RMT-2.7b-8k | 6.34 | 6.19 | 6.02 | 6.02 | 6.01 |
AutoCompressor-2.7b-6k | 6.31 | 6.04 | 5.98 | 5.94 | 5.93 |
Out-of-domain Evaluation
Context Tokens | 0 | 512 | 2048 | 4096 | 6144 |
---|---|---|---|---|---|
FullAttention-2.7b-4k | 8.94 | 8.28 | 7.93 | - | - |
RMT-2.7b-8k | 8.62 | 8.44 | 8.21 | 8.20 | 8.20 |
AutoCompressor-2.7b-6k | 8.60 | 8.26 | 8.17 | 8.12 | 8.10 |
Bibtex
@misc{chevalier2023adapting,
title={Adapting Language Models to Compress Contexts},
author={Alexis Chevalier and Alexander Wettig and Anirudh Ajith and Danqi Chen},
year={2023},
eprint={2305.14788},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
- Downloads last month
- 5