Transformers documentation


You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v4.41.3).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started


Quantization techniques focus on representing data with less information while also trying to not lose too much accuracy. This often means converting a data type to represent the same information with fewer bits. For example, if your model weights are stored as 32-bit floating points and they’re quantized to 16-bit floating points, this halves the model size which makes it easier to store and reduces memory-usage. Lower precision can also speedup inference because it takes less time to perform calculations with fewer bits.

Interested in adding a new quantization method to Transformers? Read the HfQuantizer guide to learn how!

If you are new to the quantization field, we recommend you to check out these beginner-friendly courses about quantization in collaboration with DeepLearning.AI:

When to use what?

The community has developed many quantization methods for various use cases. With Transformers, you can run any of these integrated methods depending on your use case because each method has their own pros and cons.

For example, some quantization methods require calibrating the model with a dataset for more accurate and β€œextreme” compression (up to 1-2 bits quantization), while other methods work out of the box with on-the-fly quantization.

Another parameter to consider is compatibility with your target device. Do you want to quantize on a CPU, GPU, or Apple silicon?

In short, supporting a wide range of quantization methods allows you to pick the best quantization method for your specific use case.

Use the table below to help you decide which quantization method to use.

Quantization method On the fly quantization CPU CUDA GPU RoCm GPU (AMD) Metal (Apple Silicon) torch.compile() support Number of bits Supports fine-tuning (through PEFT) Serializable with πŸ€— transformers πŸ€— transformers support Link to library
AQLM πŸ”΄ 🟒 🟒 πŸ”΄ πŸ”΄ ? 1 / 2 🟒 🟒 🟒
AWQ πŸ”΄ πŸ”΄ 🟒 🟒 πŸ”΄ ? 4 🟒 🟒 🟒
bitsandbytes 🟒 πŸ”΄ 🟒 πŸ”΄ πŸ”΄ πŸ”΄ 4 / 8 🟒 🟒 🟒
EETQ 🟒 πŸ”΄ 🟒 πŸ”΄ πŸ”΄ ? 8 🟒 🟒 🟒
GGUF / GGML (llama.cpp) 🟒 🟒 🟒 πŸ”΄ 🟒 πŸ”΄ 1 - 8 πŸ”΄ See GGUF section See GGUF section
GPTQ πŸ”΄ πŸ”΄ 🟒 🟒 πŸ”΄ πŸ”΄ 2 - 3 - 4 - 8 🟒 🟒 🟒
HQQ 🟒 🟒 🟒 πŸ”΄ πŸ”΄ 🟒 1 - 8 🟒 πŸ”΄ 🟒
Quanto 🟒 🟒 🟒 πŸ”΄ 🟒 🟒 2 / 4 / 8 πŸ”΄ πŸ”΄ 🟒
< > Update on GitHub