just use for

vllm serve lee5j/Athene-V2-Chat-gptq4 --gpu-memory-utilization 1

Downloads last month
35
Safetensors
Model size
11.9B params
Tensor type
I32
BF16
FP16
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no library tag.

Model tree for lee5j/Athene-V2-Chat-gptq4

Base model

Qwen/Qwen2.5-72B
Quantized
(24)
this model