Why was the config.json changed to "tq" instead of "tq_quant" ?
#2
by Akash4444 - opened
Why was the config.json changed to "tq" instead of "tq_quant" ? When trying to run this model through docker , I am getting error .
Only way to run is to modify the config.json before the container runs .
We changed the name of the quant in our vllm extension to "tq" instead of "tq_quant" because we felt quant word is redundant. You can run with this docker image (4bit-main instead of 4bit) like this and it should work now:
docker run --rm --gpus all
-p 8000:8000
docker.io/textclf/tq-quant:4bit-main
vllm serve textclf/Qwen3.8-27B-TQ-4bit
--quantization tq
We updated the readme accordingly. Thanks.