README.md

---
license: llama3
---

I wanted to be able to go from the Meta model weights to an AWQ quantised model myself,
rather than grab the weights from [casperhansen](https://huggingface.co/casperhansen/llama-3-70b-instruct-awq) or elsewhere.

# First Attempt

Initially I tried running autoawq on an aws g5.12xlarge instance (4xA10), ubuntu 22, cuda 12.2, nvidia 535.113.01 drivers.

I tried different combinations of torch (2.1.2, 2.2.2), autoawq (0.2.4, 0.2.5) and transformers (4.38.2, 4.41.2), but I couldnt get it to work, even with the 8B model, (which all below errors are for). I kept getting errors like:

* 0.2.4	4.38.2 2.1.2, No device map, failed at 3% `RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1)`
* 0.2.4	4.38.2 2.1.2, device map `"auto"`	failed at 16%	`& index < sizes[i] && "index out of bounds"` failed.`
* 0.2.5	4.40.0 2.1.2, No device map		failed at 3%	`File "{redacted}/.venv/lib/python3.11/site-packages/awq/quantize/quantizer.py"", line 69, in pseudo_quantize_tensor  assert torch.isnan(w).sum() == 0"`
* 0.2.4	4.38.2	2.2.2, No device map		failed at 3%    `RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1)`

  
The only thing that worked was setting CUDA_VISIBLE_DEVICES=0 or to a single device, but this would not work for the 70B model (vram)
Though the comment from casper [here](https://github.com/casper-hansen/AutoAWQ/issues/450#issuecomment-2065870629) makes me think quantising llama 3 70B with multiple GPUs should be possible.

# Working Approach

The following worked for me:

Machine: vast.ai 2xA100 PCIE instance with AMD EPYC 9554, CUDA 12.2 (~ half the price of the g5.12x large!)
Container: `pytorch:2.2.0-cuda12.1-cudnn8-devel` image

AutoAWQ @ [5f3785dc](https://github.com/casper-hansen/AutoAWQ/commit/5f3785dcaa107ca76f5fa5355f459370c86f82d6) 


Followed commands in the readme:
```
git clone https://github.com/casper-hansen/AutoAWQ
cd AutoAWQ
pip install -e .
```

Installed vim to edit the example script: `apt install vim`, `vi examples/quantize.py`

Changed model path to:
`meta-llama/Meta-Llama-3-70B-Instruct`

Changed output path to:
`Meta-Llama-3-70B-Instruct-awq`
 
Used a script to set the token so we can pull llama 3

```
#!/usr/bin/env bash

export HF_TOKEN=${your token here - used to grab llama weights}

python quantize.py
```

This worked, took ~ 100 mins for the 70B model to quantise. Not sure if the second A100 was used, 
once I set the thing running I couldnt figure out how to open a second ssh session to run nvidia-smi 
or similar without joining the same tmux session running the quantisation, so just left it to it.