FP8 LLMs for vLLM
Collection
Accurate FP8 quantized models by Neural Magic, ready for use with vLLM!
•
10 items
•
Updated
•
15
Qwen2-1.5B-Instruct quantized to FP8 weights and activations using per-tensor quantization through the AutoFP8 repository, ready for inference with vLLM >= 0.5.0. Calibrated with 512 UltraChat samples to achieve 99% performance recovery on the Open LLM Benchmark evaluations. Reduces space on disk by ~40%. Part of the FP8 LLMs for vLLM collection.
Produced using AutoFP8 with calibration samples from ultrachat.
from datasets import load_dataset
from transformers import AutoTokenizer
from auto_fp8 import AutoFP8ForCausalLM, BaseQuantizeConfig
pretrained_model_dir = "Qwen/Qwen2-1.5B-Instruct"
quantized_model_dir = "Qwen2-1.5B-Instruct-FP8"
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True, model_max_length=4096)
tokenizer.pad_token = tokenizer.eos_token
ds = load_dataset("mgoin/ultrachat_2k", split="train_sft").select(range(512))
examples = [tokenizer.apply_chat_template(batch["messages"], tokenize=False) for batch in ds]
examples = tokenizer(examples, padding=True, truncation=True, return_tensors="pt").to("cuda")
quantize_config = BaseQuantizeConfig(quant_method="fp8", activation_scheme="static")
model = AutoFP8ForCausalLM.from_pretrained(
pretrained_model_dir, quantize_config=quantize_config
)
model.quantize(examples)
model.save_quantized(quantized_model_dir)
Evaluated through vLLM with the following script:
#!/bin/bash
# Example usage:
# CUDA_VISIBLE_DEVICES=0 ./eval_openllm.sh "neuralmagic/Qwen2-1.5B-Instruct-FP8" "tensor_parallel_size=1,max_model_len=4096,add_bos_token=True,gpu_memory_utilization=0.7"
export MODEL_DIR=${1}
export MODEL_ARGS=${2}
declare -A tasks_fewshot=(
["arc_challenge"]=25
["winogrande"]=5
["truthfulqa_mc2"]=0
["hellaswag"]=10
["mmlu"]=5
["gsm8k"]=5
)
declare -A batch_sizes=(
["arc_challenge"]="auto"
["winogrande"]="auto"
["truthfulqa_mc2"]="auto"
["hellaswag"]="auto"
["mmlu"]=1
["gsm8k"]="auto"
)
for TASK in "${!tasks_fewshot[@]}"; do
NUM_FEWSHOT=${tasks_fewshot[$TASK]}
BATCH_SIZE=${batch_sizes[$TASK]}
lm_eval --model vllm \
--model_args pretrained=$MODEL_DIR,$MODEL_ARGS \
--tasks ${TASK} \
--num_fewshot ${NUM_FEWSHOT} \
--write_out \
--show_config \
--device cuda \
--batch_size ${BATCH_SIZE} \
--output_path="results/${TASK}"
done
Evaluated on the Open LLM Leaderboard evaluations through vLLM.
Qwen2-1.5B-Instruct | Qwen2-1.5B-Instruct-FP8 (this model) |
|
---|---|---|
arc-c 25-shot |
43.09 | 41.81 |
hellaswag 10-shot |
67.48 | 67.18 |
mmlu 5-shot |
55.87 | 55.60 |
truthfulqa 0-shot |
43.34 | 43.09 |
winogrande 5-shot |
63.61 | 63.38 |
gsm8k 5-shot |
57.70 | 56.48 |
Average Accuracy |
55.18 | 54.59 |
Recovery | 100% | 98.93% |