SagaLM-slm1

This is a LoRA fine-tuned adapter for SagaLM-slm1, built on top of the base model Qwen/Qwen2.5-3B-Instruct.

Training Data

• OpenOrca (25K samples)
• UltraChat 200k (25K samples)

LoRA Configuration

• r = 16
• lora_alpha = 32
• lora_dropout = 0.05
• target_modules = ['o_proj', 'up_proj', 'k_proj', 'v_proj', 'q_proj', 'down_proj', 'gate_proj']

How to use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", device_map="auto")
model = PeftModel.from_pretrained(base, "venkateshchsagalm/SagaLM-slm1")
tokenizer = AutoTokenizer.from_pretrained("venkateshchsagalm/SagaLM-slm1")