Issues with lora finetuning
Hi, I'm having a problem trying to fine-tune the model with lora.
Lora set as below :
peft_config = LoraConfig(
r=64,
lora_alpha=128,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
],
bias="none",
lora_dropout=0.05,
task_type="CAUSAL_LM",
auto_mapping=True,
)
Error Information as below :
93 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
94 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/torch/utils/checkpoint.py", line 261, in forward
95 outputs = run_function(*args)
96 ^^^^^^^^^^^^^^^^^^^
97 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
98 return self._call_impl(*args, **kwargs)
99 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
100 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
101 return forward_call(*args, **kwargs)
102 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
103 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/accelerate/hooks.py", line 169, in new_forward
104 output = module._old_forward(*args, **kwargs)
105 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
106 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/transformers/models/mistral/modeling_mistral.py", line 757, in forward
107 hidden_states, self_attn_weights, present_key_value = self.self_attn(
108 ^^^^^^^^^^^^^^^
109 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
110 return self._call_impl(*args, **kwargs)
111 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
112 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
113 return forward_call(*args, **kwargs)
114 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
115 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/accelerate/hooks.py", line 169, in new_forward
116 output = module._old_forward(*args, **kwargs)
117 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
118 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/transformers/models/mistral/modeling_mistral.py", line 653, in forward
119 query_states = self.q_proj(hidden_states)
120 ^^^^^^^^^^^^^^^^^^^^^^^^^^
121 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
122 return self._call_impl(*args, **kwargs)
123 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
124 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
125 return forward_call(*args, **kwargs)
126 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
127 File "/home/xinye/miniconda3/envs/translation/lib/python3.12/site-packages/peft/tuners/lora/bnb.py", line 480, in forward
128 result = result + output
129 ~~~~~~~^~~~~~~~
130 RuntimeError: The size of tensor a (4096) must match the size of tensor b (5120) at non-singleton dimension 2
Same problem here. Use the latest peft and transformers package. I tried using both AutoModelForSequenceClassification and MistralForSequenceClassification to load the model. All get the same error.
did you use git to load the package??
I solved this issue by upgrading transformers package to latest dev version by: pip install git+https://github.com/huggingface/transformers.git
I managed to start the fine-tuning after installing transformers from source. However, the model is not learning at all. It seems to be related to the tokenizer configuration, but despite trying various settings, the output model only generates nonsensical results. The issue persists regardless of the LoRA or tokenizer configuration used. Here are the current LoRA and Tokenizer parameters:
LoRA:
- bits: 4
- lora_r: 256
- lora_alpha: 128
- lora_dropout: 0.05
- bias: "none"
- target_modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
- task_type: "CAUSAL_LM"
Tokenizer:
- padding: True
- padding_side: 'right'
- add_bos_token: False
- add_eos_token: True
- trust_remote_code: True
- use_auth_token: True
- eos_token: < /s>
- pad_token: < /s>
did you use git to load the package??
I solved this issue by upgrading transformers package to latest dev version by: pip install git+https://github.com/huggingface/transformers.git
Thanks, that's very useful.
did you use git to load the package??
I downloaded all the files .