|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
|
|
# Molmo-7B-GPTQ-4bit 🚀 |
|
|
|
## Overview |
|
|
|
The **Molmo-7B-GPTQ-4bit** model is a transformer-based model fine-tuned for NLP tasks. It has been quantized to 4-bit precision for efficient deployment. This model has been prepared using **bitsandbytes** for 4-bit quantization rather than using **AutoGPTQ**, which does not natively support this model format as of now. The quantization leverages the `BitsAndBytesConfig` from the `transformers` library, enabling highly optimized GPU inference with reduced memory usage. |
|
|
|
<div align="center"> |
|
<img src="https://molmo.allenai.org/opengraph-image.png" alt="Model Architecture" width="80%" /> |
|
</div> |
|
|
|
|
|
### Model Information |
|
|
|
- **Model Name**: Molmo-7B-GPTQ-4bit |
|
- **Base Model**: [allenai/Molmo-7B-D-0924](https://huggingface.co/allenai/Molmo-7B-D-0924) |
|
- **Quantization**: 4-bit quantization using `bitsandbytes` instead of `AutoGPTQ` |
|
- **Repository URL**: [zamal/Molmo-7B-GPTQ-4bit](https://huggingface.co/zamal/Molmo-7B-GPTQ-4bit) |
|
|
|
### Technical Details |
|
|
|
This model is quantized using **bitsandbytes** (not **AutoGPTQ**), as GPTQ currently lacks direct support for NF4 4-bit quantization via the native `AutoGPTQ` methods. This approach allows for highly efficient 4-bit precision inference with minimal loss in performance and reduced memory overhead. |
|
|
|
#### Key Quantization Configurations: |
|
|
|
- **bnb_4bit_use_double_quant**: Enabled, for more efficient handling of smaller models. |
|
- **bnb_4bit_quant_type**: NF4 (Normal Float 4-bit), which is more efficient and accurate for smaller models. |
|
- **bnb_4bit_compute_dtype**: FP16 (float16) to accelerate GPU-based inference. |
|
|
|
#### Device Compatibility: |
|
|
|
- **bitsandbytes** automatically handles device mapping for GPUs via the `device_map="auto"` parameter. |
|
- **4-bit models** are ideal for GPUs with limited VRAM, allowing inference on larger models without exceeding hardware memory limits. |
|
|
|
### Limitations |
|
|
|
- **Precision Loss**: While the model has been quantized for efficiency, there is a minor trade-off in precision due to the 4-bit quantization, which may slightly affect performance compared to the original full-precision model. |
|
- **AutoGPTQ Limitation**: As mentioned, **AutoGPTQ** does not natively support this kind of quantization, and this has been achieved through `bitsandbytes` and the `transformers` library. |
|
|
|
## Usage |
|
|
|
### Installation |
|
|
|
Make sure you have the necessary dependencies installed: |
|
|
|
```bash |
|
pip install transformers torch Pillow torchvision einops accelerate tensorflow bitsandbytes |
|
|
