arxiv:2401.15024

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

Published on Jan 26

· Featured in Daily Papers on Jan 29

Upvote

Authors:

Saleh Ashkboos ,

Maximilian L. Croci ,

Marcelo Gennari do Nascimento ,

James Hensman

Abstract

Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models. Code is available at: https://github.com/microsoft/TransformerCompression

View arXiv page View PDF Add to collection

Community

PixelRage

Jan 29

Not APril FOols, Rad!

Chat-Error

Jan 29

Code is 404

Jameshensman

Paper author Jan 29

•

edited Jan 29

Code is 404

it's there now!

leegao19

Jan 29

On the calibration set - it looks like this is running a bunch of samples of Wiki2 and Alpaca truncated to the maximum length of the LLM.

I'm guessing the intuition here is to have a representative set of samples that's either seen during pretraining or are likely to be encountered in day-to-day use.

Have you guys explored the idea of using this for feature activation steering? E.g. prime the model (maybe even using contrastive pairs to get positive and negatives) using calibration prompts that steer behavior to, for e.g., avoid sycophancy (vs emulates typical interactions).

Admittedly, it's really heavy for what can be done, but it'd be interesting to see if the PCA picks up anything interesting (e.g. do they decompose into basic feature directions? can they be interpreted? can they be manipulated to amplify/suppress certain behaviors? do these components become sparser as steering becomes more directed?)

Jameshensman

Paper author Jan 29

Hi @leegao19 ,

Your intuition is correct. I think about it as trying to perturb all the directions in the model that are going to be important. I've been pleasantly surprised by how few samples are needed to get a representative set. Just 2k samples from Alpaca seems to be enough to preserve llm-eval metrics.

We have not experimented with steering! Feel free to check out the code and pull out the Principal components to see what they each do. I'd be interested to hear what happens.