Repository: unslothai/unsloth

Issue #1597: Error on unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit
Created: 2025-01-31T10:35:33Z
I run the model with your official instrcutions, but it seems that there's an error about block quantization. How to solve it？

Duplicate issue #1596: Error when loading Mistral-Small-24B-Base-2501-unsloth-bnb-4bit
Created: 2025-01-31T07:50:38Z
Hi, I get the following error when trying to load unsloth/Mistral-Small-24B-Base-2501-unsloth-bnb-4bit.
My environment is as follows:
==((====))==  Unsloth 2025.1.6: Fast Mistral patching. Transformers: 4.48.1.
   \\   /|    GPU: NVIDIA H100 80GB HBM3. Max memory: 79.109 GB. Platform: Linux.
O^O/ \_/ \    Torch: 2.5.1+cu124. CUDA: 9.0. CUDA Toolkit: 12.4. Triton: 3.1.0
\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.28.post3. FA2 = False]
 "-____-"     Free Apache license: http://github.com/unslothai/unsloth

The error I get (when fine-tuning with load-in-4bit):
 File "/.../.venv/lib/python3.12/site-packages/unsloth/models/loader.py", line 258, in from_pretrained
    model, tokenizer = dispatch_model.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/unsloth/models/mistral.py", line 348, in from_pretrained
    return FastLlamaModel.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/unsloth/models/llama.py", line 1691, in from_pretrained
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4224, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4794, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 875, in _load_state_dict_into_meta_model
    hf_quantizer.create_quantized_param(model, param, param_name, param_device, state_dict, unexpected_keys)
  File "/.../.venv/lib/python3.12/site-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 242, in create_quantized_param
    new_value = bnb.nn.Params4bit(new_value, requires_grad=False, **kwargs).to(target_device)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/bitsandbytes/nn/modules.py", line 331, in to
    return self._quantize(device)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/bitsandbytes/nn/modules.py", line 296, in _quantize
    w_4bit, quant_state = bnb.functional.quantize_4bit(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/bitsandbytes/functional.py", line 1243, in quantize_4bit
    raise ValueError(f"Blockwise quantization only supports 16/32-bit floats, but got {A.dtype}")
ValueError: Blockwise quantization only supports 16/32-bit floats, but got torch.uint8

If I don't use load-in-4bit, I get the following error:
ValueError: Trying to set a tensor of shape torch.Size([83886080, 1]) in "weight" (which has shape torch.Size([5120, 32768])), this looks incorrect.

Closing comment (2025-01-31T12:39:32Z): Could it be related to #1597?