Qwen-0.8B-unsloth 🦥

A fine-tuned version of Qwen/Qwen3.5-0.8B optimized for instruction-following and technical Q&A regarding Unsloth AI documentation.

This model was trained using Supervised Fine-Tuning (SFT) with LoRA (Low-Rank Adaptation) and 4-bit quantization via bitsandbytes and Hugging Face's trl library.

---

📌 Model Details

• Developed by: maghrane
• Base Model: Qwen/Qwen3.5-0.8B
• Model Type: Causal Language Model (Decoder-Only)
• Fine-Tuning Technique: LoRA (PEFT) + QLoRA (4-bit NF4)
• Dataset: maghrane/unsloth-qwen
• Prompt Format: <s>[INST] {prompt} [/INST] {response} </s>
• License: Apache-2.0

---

⚙️ Training Hyperparameters & Configuration

🔹 LoRA Configuration (PEFT)

Parameter	Value
LoRA Rank ($r$)	64
LoRA Alpha ($\alpha$)	16
LoRA Dropout	0.1
Bias	none
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, fc1, fc2

🔹 Quantization & Precision (BitsAndBytes)

Parameter	Value
Quantization	4-bit (nf4)
Compute Dtype	float16
Double Quantization	False

🔹 SFT Training Arguments

Parameter	Value
Optimizer	paged_adamw_32bit
Learning Rate	2e-4
Learning Rate Scheduler	constant
Weight Decay	0.001
Warmup Ratio	0.03
Epochs	1
Per Device Batch Size	4
Gradient Accumulation	1
Max Gradient Norm	0.3

---

💻 How to Use & Run Inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# 1. Disable cuDNN algorithm search (resolves Qwen 3.5 1D convolution kernel lookup issue)
torch.backends.cudnn.enabled = False

base_model_name = "Qwen/Qwen3.5-0.8B"
adapter_model_name = "maghrane/Qwen-0.8B-unsloth"

# 2. Load Base Model & Adapter
tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, adapter_model_name)

# 3. Re-enable KV Cache & set to evaluation mode
model.config.use_cache = True
model.eval()

# 4. Generate Response
prompt = "What is Unsloth and what can it do?"
formatted_prompt = f"<s>[INST] {prompt} [/INST]"
inputs = tokenizer(formatted_prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.15,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id
    )

generated_text = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
print(generated_text)

---

📊 Results & Sample Inference

Input Prompt:

<s>[INST] What is Unsloth and what can it do? [/INST]

Model Output:

Unsloth lets you run and train AI models on your own local hardware via an open-source UI. It supports inference and training for 500+ models across text, audio, embedding, vision, and more. Key features include:

• Inference: Search, download, and run models (GGUFs, LoRA adapters, safetensors), self-healing tool calling, auto parameter tuning, model export, and side-by-side output comparison.
• Training: ~2x faster training with ~70% less VRAM, full/pre-training, 4-bit/16-bit/FP8 support, auto-dataset creation from PDF/CSV/DOCX, live observability, and efficient reinforcement learning (RL).

---

📜 Disclaimer

This model is fine-tuned for educational and documentation guidance purposes regarding local LLM fine-tuning and inference.