szymonrucinski

/

krakowiak-7b

like 8

Hi Grzegorz! 🤝😃
It is not a model file. In this repo there is only a PEFT adapter that you need to merge into the model using the following code:

from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
from pprint import pprint

config = PeftConfig.from_pretrained("szymonrucinski/krakowiak-7b")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-"7b-hf,

tokenizer = AutoTokenizer.from_pretrained(f"meta-llama/Llama-2-7b-hf", )
model = PeftModel.from_pretrained(model, f"szymonrucinski/krakowiak-7b")

pipeline = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    device_map="auto",
)

Good luck!

Thanks for educating me about the PEFT adapters!

I had to tweak your snippet a little bit and finally got the following to run:

import os
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline


HF_TOKEN = os.getenv("HF_TOKEN")

peft_model_id = "szymonrucinski/krakowiak-7b"
peft_config = PeftConfig.from_pretrained(peft_model_id)

base_model_id = peft_config.base_model_name_or_path
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, token=HF_TOKEN)
base_tokenizer = AutoTokenizer.from_pretrained(base_model_id, token=HF_TOKEN)

peft_model = PeftModel.from_pretrained(base_model, peft_model_id)

pipeline = pipeline(task="text-generation", model=peft_model, tokenizer=base_tokenizer, device_map="auto")