hugging-quants
/

Meta-Llama-3.1-405B-Instruct-GPTQ-INT4

@@ -35,25 +35,11 @@ In order to use the current quantized model, support is offered for different so
 ### 🤗 transformers
-In order to run the inference with Llama 3.1 405B Instruct GPTQ in INT4, both `torch` and `autogptq` need to be installed as:
 ```bash
-pip install "torch>=2.2.0,<2.3.0" --upgrade
-pip install auto-gptq --no-build-isolation
-```
-Otherwise, running the model may fail, since the AutoGPTQ kernels are built with PyTorch 2.2.1, meaning that those will break with PyTorch 2.3.0.
-Then, the latest version of `transformers` need to be installed including the `accelerate` extra, being 4.43.0 or higher, as:
-```bash
-pip install "transformers[accelerate]>=4.43.0" --upgrade
-```
-Finally, in order to use `autogptq`, `optimum` also needs to be installed:
-```bash
-pip install optimum --upgrade
 ```
 To run the inference on top of Llama 3.1 405B Instruct GPTQ in INT4 precision, the GPTQ model can be instantiated as any other causal language modeling model via `AutoModelForCausalLM` and run the inference normally.
@@ -91,30 +77,14 @@ print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
 ### AutoGPTQ
-Alternatively, one may want to run that via `AutoGPTQ` even though it's built on top of 🤗 `transformers`, which is the recommended approach instead as described above.
-In order to run the inference with Llama 3.1 405B Instruct GPTQ in INT4, both `torch` and `autogptq` need to be installed as:
-```bash
-pip install "torch>=2.2.0,<2.3.0" --upgrade
-pip install auto-gptq --no-build-isolation
-```
-Otherwise, running the model may fail, since the AutoGPTQ kernels are built with PyTorch 2.2.1, meaning that those will break with PyTorch 2.3.0.
-Then, the latest version of `transformers` need to be installed including the `accelerate` extra, being 4.43.0 or higher, as:
-```bash
-pip install "transformers[accelerate]>=4.43.0" --upgrade
-```
-Finally, in order to use `autogptq`, `optimum` also needs to be installed:
 ```bash
-pip install optimum --upgrade
 ```
-And then run it as follows:
 ```python
 import torch
@@ -148,7 +118,7 @@ outputs = model.generate(**inputs, do_sample=True, max_new_tokens=256)
 print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
 ```
-The AutoGPTQ script has been adapted from [AutoGPTQ/examples/quantization/basic_usage.py](https://github.com/AutoGPTQ/AutoGPTQ/blob/main/examples/quantization/basic_usage.py).
 ### 🤗 Text Generation Inference (TGI)
@@ -159,28 +129,14 @@ Coming soon!
 > [!NOTE]
 > In order to quantize Llama 3.1 405B Instruct using AutoGPTQ, you will need to use an instance with at least enough CPU RAM to fit the whole model i.e. ~800GiB, and an NVIDIA GPU with 80GiB of VRAM to quantize it.
-In order to quantize Llama 3.1 405B Instruct, first install `torch` and `autoqptq` as follows:
-```bash
-pip install "torch>=2.2.0,<2.3.0" --upgrade
-pip install auto-gptq --no-build-isolation
-```
-Otherwise the quantization may fail, since the AutoGPTQ kernels are built with PyTorch 2.2.1, meaning that those will break with PyTorch 2.3.0.
-Then install the latest version of `transformers` as follows:
-```bash
-pip install "transformers>=4.43.0" --upgrade
-```
-Finally, in order to use `autogptq`, `optimum` also needs to be installed:
 ```bash
-pip install optimum --upgrade
 ```
-And then, run the following script, adapted from [AutoGPTQ/examples/quantization/basic_usage.py](https://github.com/AutoGPTQ/AutoGPTQ/blob/main/examples/quantization/basic_usage.py).
 ```python
 import random

 ### 🤗 transformers
+In order to run the inference with Llama 3.1 405B Instruct GPTQ in INT4, you need to install the following packages:
 ```bash
+pip install -q --upgrade transformers accelerate optimum
+pip install -q --no-build-isolation auto-gptq
 ```
 To run the inference on top of Llama 3.1 405B Instruct GPTQ in INT4 precision, the GPTQ model can be instantiated as any other causal language modeling model via `AutoModelForCausalLM` and run the inference normally.
 ### AutoGPTQ
+In order to run the inference with Llama 3.1 405B Instruct GPTQ in INT4, you need to install the following packages:
 ```bash
+pip install -q --upgrade transformers accelerate optimum
+pip install -q --no-build-isolation auto-gptq
 ```
+Alternatively, one may want to run that via `AutoGPTQ` even though it's built on top of 🤗 `transformers`, which is the recommended approach instead as described above.
 ```python
 import torch
 print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
 ```
+The AutoGPTQ script has been adapted from [`AutoGPTQ/examples/quantization/basic_usage.py`](https://github.com/AutoGPTQ/AutoGPTQ/blob/main/examples/quantization/basic_usage.py).
 ### 🤗 Text Generation Inference (TGI)
 > [!NOTE]
 > In order to quantize Llama 3.1 405B Instruct using AutoGPTQ, you will need to use an instance with at least enough CPU RAM to fit the whole model i.e. ~800GiB, and an NVIDIA GPU with 80GiB of VRAM to quantize it.
+In order to quantize Llama 3.1 405B Instruct with GPTQ in INT4, you need to install the following packages:
 ```bash
+pip install -q --upgrade transformers accelerate optimum
+pip install -q --no-build-isolation auto-gptq
 ```
+Then run the following script, adapted from [`AutoGPTQ/examples/quantization/basic_usage.py`](https://github.com/AutoGPTQ/AutoGPTQ/blob/main/examples/quantization/basic_usage.py).
 ```python
 import random