AWQ 4bit Inference
We integrated AWQ into FastChat to provide efficient and accurate 4bit LLM inference.
Install AWQ
Setup environment (please refer to this link for more details):
conda create -n fastchat-awq python=3.10 -y
conda activate fastchat-awq
# cd /path/to/FastChat
pip install --upgrade pip # enable PEP 660 support
pip install -e . # install fastchat
git clone https://github.com/mit-han-lab/llm-awq repositories/llm-awq
cd repositories/llm-awq
pip install -e . # install awq package
cd awq/kernels
python setup.py install # install awq CUDA kernels
Chat with the CLI
# Download quantized model from huggingface
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/mit-han-lab/vicuna-7b-v1.3-4bit-g128-awq
# You can specify which quantized model to use by setting --awq-ckpt
python3 -m fastchat.serve.cli \
--model-path models/vicuna-7b-v1.3-4bit-g128-awq \
--awq-wbits 4 \
--awq-groupsize 128
Benchmark
Through 4-bit weight quantization, AWQ helps to run larger language models within the device memory restriction and prominently accelerates token generation. All benchmarks are done with group_size 128.
Benchmark on NVIDIA RTX A6000:
Model Bits Max Memory (MiB) Speed (ms/token) AWQ Speedup vicuna-7b 16 13543 26.06 / vicuna-7b 4 5547 12.43 2.1x llama2-7b-chat 16 13543 27.14 / llama2-7b-chat 4 5547 12.44 2.2x vicuna-13b 16 25647 44.91 / vicuna-13b 4 9355 17.30 2.6x llama2-13b-chat 16 25647 47.28 / llama2-13b-chat 4 9355 20.28 2.3x NVIDIA RTX 4090:
Model AWQ 4bit Speed (ms/token) FP16 Speed (ms/token) AWQ Speedup vicuna-7b 8.61 19.09 2.2x llama2-7b-chat 8.66 19.97 2.3x vicuna-13b 12.17 OOM / llama2-13b-chat 13.54 OOM / NVIDIA Jetson Orin:
Model AWQ 4bit Speed (ms/token) FP16 Speed (ms/token) AWQ Speedup vicuna-7b 65.34 93.12 1.4x llama2-7b-chat 75.11 104.71 1.4x vicuna-13b 115.40 OOM / llama2-13b-chat 136.81 OOM /