Update README.md

README.md

@@ -1,3 +1,63 @@
----
-license: llama3
----
+---
+license: llama3
+---
+
+I wanted to be able to go from the Meta model weights to an AWQ quantised model myself,
+rather than grab the weights from [casperhansen](https://huggingface.co/casperhansen/llama-3-70b-instruct-awq) or elsewhere.
+
+# First Attempt
+
+Initially I tried running autoawq on an aws g5.12xlarge instance (4xA10), ubuntu 22, cuda 12.2, nvidia 535.113.01 drivers.
+
+I tried different combinations of torch (2.1.2, 2.2.2), autoawq (0.2.4, 0.2.5) and transformers (4.38.2, 4.41.2), but I couldnt get it to work, even with the 8B model, (which all below errors are for). I kept getting errors like:
+
+* 0.2.4	4.38.2 2.1.2, No device map, failed at 3% `RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1)`
+* 0.2.4	4.38.2 2.1.2, device map `"auto"`	failed at 16%	`& index < sizes[i] && "index out of bounds"` failed.`
+* 0.2.5	4.40.0 2.1.2, No device map		failed at 3%	`File "{redacted}/.venv/lib/python3.11/site-packages/awq/quantize/quantizer.py"", line 69, in pseudo_quantize_tensor  assert torch.isnan(w).sum() == 0"`
+* 0.2.4	4.38.2	2.2.2, No device map		failed at 3%    `RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1)`
+
+  
+The only thing that worked was setting CUDA_VISIBLE_DEVICES=0 or to a single device, but this would not work for the 70B model (vram)
+Though the comment from casper [here](https://github.com/casper-hansen/AutoAWQ/issues/450#issuecomment-2065870629) makes me think quantising llama 3 70B with multiple GPUs should be possible.
+
+# Working Approach
+
+The following worked for me:
+
+Machine: vast.ai 2xA100 PCIE instance with AMD EPYC 9554, CUDA 12.2 (~ half the price of the g5.12x large!)
+Container: `pytorch:2.2.0-cuda12.1-cudnn8-devel` image
+
+AutoAWQ @ [5f3785dc](https://github.com/casper-hansen/AutoAWQ/commit/5f3785dcaa107ca76f5fa5355f459370c86f82d6) 
+
+
+Followed commands in the readme:
+```
+git clone https://github.com/casper-hansen/AutoAWQ
+cd AutoAWQ
+pip install -e .
+```
+
+Installed vim to edit the example script: `apt install vim`, `vi examples/quantize.py`
+
+Changed model path to:
+`meta-llama/Meta-Llama-3-70B-Instruct`
+
+Changed output path to:
+`Meta-Llama-3-70B-Instruct-awq`
+ 
+Used a script to set the token so we can pull llama 3
+
+```
+#!/usr/bin/env bash
+
+export HF_TOKEN=${your token here - used to grab llama weights}
+
+python quantize.py
+```
+
+This worked, took ~ 100 mins for the 70B model to quantise. Not sure if the second A100 was used, 
+once I set the thing running I couldnt figure out how to open a second ssh session to run nvidia-smi 
+or similar without joining the same tmux session running the quantisation, so just left it to it.
+
+
+