Finetuning with PEFT - Some weights of MistralForSequenceClassification were not initialized from the model

#116
by RobbieTheRobot - opened

Q1: Finetuning for a classification task (dataset imdb) using PEFT generates the following error:

Some weights of MistralForSequenceClassification were not initialized from the model checkpoint at mistralai/Mistral-7B-v0.1 and are newly initialized: ['score.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Q2: What is the max_length in mistral for Padding?

Thanks in advance

hi @RobbieTheRobot you need to properly set modules_to_save=["score"] in the peft config, can you provide more details on how you do fine-tuning with PEFT + sequence classification using mistral?

@ybelkada set module_to_save=["score"] does not eliminate the issue.

quantization_config = BitsAndBytesConfig(
    load_in_4bit = True, 
    bnb_4bit_quant_type = 'nf4', 
    bnb_4bit_use_double_quant = True, 
    bnb_4bit_compute_dtype = torch.bfloat16 
)
lora_config = LoraConfig(
    r = 16, 
    lora_alpha = 8, 
    target_modules = ['q_proj', 'k_proj', 'v_proj', 'o_proj'],
    lora_dropout = 0.05,
    bias = 'none', 
    task_type = 'SEQ_CLS',
    modules_to_save=["score"]
)
model = AutoModelForSequenceClassification.from_pretrained(
    'mistralai/Mistral-7B-v0.1',
    quantization_config=quantization_config,
    num_labels=2
)
model = prepare_model_for_kbit_training(model)
model.config.use_cache = False
model.config.pad_token_id = tokenizer.pad_token_id
model = get_peft_model(model, lora_config)

@wuflyh can you try with the latest version of PEFT? pip install -U peft ? If this do not work can you openan issue on PEFT repository with a reproducer?

Thanks @ybelkada
Hower,
I've already used the latest version. Version: 0.8.2.
the version is the same before and after upgrade attempt using "pip install -U peft"

Hi man, I met the same problem. Have u found the solution?

Any updates?

Sign up or log in to comment