mistralai

/

Mistral-7B-Instruct-v0.2

like 2.29k

This is my current work flow :

-Merge fine tuned apadter weights with base mistal model
-convert merge model to GGUF format using llama.cpp

For additional context:

• Repo containing fine-tuned model only contains the adapter_config.json and adapter_model.safetensors, The repo is named Rmote6603/MedPrescription-FineTuning
  *The python code I use to merge is attached to thread, this is most likely where I am doing something wrong I think.
  *When the merged model is created in it's own directory it contains the following:

generation_config.json,
model.safetensors.index.json,
6 model-0000X-of-00006.safetensors (X represents numbers from 1 to 6)

*I had to manually add in the tokenizer.model and tokenizer.json from the base instruct model so could convert to GGUF with llama.cpp.
*I run python3 llama.cpp/convert-hf-to-gguf.py remote-hf --outfile remote2.gguf --outtype f16.

Here is the merge code

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
peft_model_id = "Rmote6603/MedPrescription-FineTuning"
model = PeftModel.from_pretrained(base_model, peft_model_id)
merged_model = model.merge_and_unload()
local_save_path = "remote-hf"

# Save the merged model locally
merged_model.save_pretrained(local_save_path)

Does any part of my process look off? I'm getting wacky behaviors when I try to create the model in ollama from the GGUF I get from this process.