navdeepdh/prompt-gen-adapters

---

license: apache-2.0

---

Testing Model

Prompt gen model can be used to generate prompts from provided chunks, which can further be used to create domain specific dataset for any base LLMs.

In this test model Mistral-7B-v0.1 is used as base model to finetune as prompt-gen model, fintuned version mimic gpt-3.5 in generating prompts from text chunks.

These type of model can be helpful in generating dataset at low cost compared to gpt and large amount of datapoint for fintenuning any llm for domain specific tasks.

load base model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

base_model_id = "mistralai/Mistral-7B-v0.1"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,  # Mistral, same as before
    quantization_config=bnb_config,  # Same quantization config as before
    device_map="auto",
    trust_remote_code=True,
)

eval_tokenizer = AutoTokenizer.from_pretrained(base_model_id, add_bos_token=True, trust_remote_code=True)

load finetuned PEFT-LORA WEIGHTS

from peft import PeftModel

ft_model = PeftModel.from_pretrained(base_model, "navdeepdh/prompt-gen-adapters")

prompt generation

eval_prompt = """[INST]Generate 5 question from this text in array format to get most info about the text.
add your text chunk here[/INST]"""
model_input = eval_tokenizer(eval_prompt, return_tensors="pt").to("cuda")
ft_model.eval()
with torch.no_grad():
    print(eval_tokenizer.decode(ft_model.generate(**model_input, max_new_tokens=200, repetition_penalty=1.15)[0], skip_special_tokens=True))