vLLM Compatibility Issue with Unsloth's 4-bit Quantized Models - Shape Mismatch During Weight Loading

by joserey1 - opened Nov 26, 2025

Nov 26, 2025

just like in this discussion, using this model in vllm got this issue

tested by using nightly docker image
"vllm/vllm-openai@sha256:0d0104a260b69ce0bff9badde7620b8d796abfd067c327451f7ae0b09c761c9f"

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment