FuelProp-LM

FuelProp-LM is a LoRA fine-tuned adapter based on Qwen/Qwen2.5-7B-Instruct.

This repository contains the adapter weights and tokenizer files needed to load the model with transformers and peft. The base model weights are not included; they are loaded from Qwen/Qwen2.5-7B-Instruct.

Model Details

• Base model: Qwen/Qwen2.5-7B-Instruct
• Fine-tuning method: LoRA / PEFT
• Task type: causal language modeling
• LoRA rank: 32
• LoRA alpha: 64
• LoRA dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, up_proj, down_proj

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "Qwen/Qwen2.5-7B-Instruct"
adapter_id = "tcy0512/FuelProp-LM"

tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_id)

messages = [
    {"role": "user", "content": "Please introduce yourself."}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Results

• Train loss: 0.3761
• Eval loss: 0.4702
• Epochs: 3.0

Limitations

This adapter inherits the capabilities and limitations of Qwen/Qwen2.5-7B-Instruct. Users should evaluate the model carefully for their own domain and use case.

License

Please follow the license terms of the base model Qwen/Qwen2.5-7B-Instruct and any applicable license terms for the fine-tuning data.