license: llama3
I wanted to be able to go from the Meta model weights to an AWQ quantised model myself, rather than grab the weights from casperhansen or elsewhere.
First Attempt
Initially I tried running autoawq on an aws g5.12xlarge instance (4xA10), ubuntu 22, cuda 12.2, nvidia 535.113.01 drivers.
I tried different combinations of torch (2.1.2, 2.2.2), autoawq (0.2.4, 0.2.5) and transformers (4.38.2, 4.41.2), but I couldnt get it to work, even with the 8B model, (which all below errors are for). I kept getting errors like:
- 0.2.4 4.38.2 2.1.2, No device map, failed at 3%
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1) - 0.2.4 4.38.2 2.1.2, device map
"auto"failed at 16%& index < sizes[i] && "index out of bounds"failed.` - 0.2.5 4.40.0 2.1.2, No device map failed at 3%
File "{redacted}/.venv/lib/python3.11/site-packages/awq/quantize/quantizer.py"", line 69, in pseudo_quantize_tensor assert torch.isnan(w).sum() == 0" - 0.2.4 4.38.2 2.2.2, No device map failed at 3%
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1)
The only thing that worked was setting CUDA_VISIBLE_DEVICES=0 or to a single device, but this would not work for the 70B model (vram) Though the comment from casper here makes me think quantising llama 3 70B with multiple GPUs should be possible.
Working Approach
The following worked for me:
Machine: vast.ai 2xA100 PCIE instance with AMD EPYC 9554, CUDA 12.2 (~ half the price of the g5.12x large!)
Container: pytorch:2.2.0-cuda12.1-cudnn8-devel image
AutoAWQ @ 5f3785dc
Followed commands in the readme:
git clone https://github.com/casper-hansen/AutoAWQ
cd AutoAWQ
pip install -e .
Installed vim to edit the example script: apt install vim, vi examples/quantize.py
Changed model path to:
meta-llama/Meta-Llama-3-70B-Instruct
Changed output path to:
Meta-Llama-3-70B-Instruct-awq
Used a script to set the token so we can pull llama 3
#!/usr/bin/env bash
export HF_TOKEN=${your token here - used to grab llama weights}
python quantize.py
This worked, took ~ 100 mins for the 70B model to quantise. Not sure if the second A100 was used, once I set the thing running I couldnt figure out how to open a second ssh session to run nvidia-smi or similar without joining the same tmux session running the quantisation, so just left it to it.