ChatWithMe — Qwen3-4B Fine-tuned on Alpaca
A Qwen3-4B model fine-tuned with LoRA on the Alpaca dataset for instruction-following conversations.
Model Details
- Base model: Qwen/Qwen3-4B
- Fine-tuning method: LoRA (r=8, alpha=16)
- Dataset: yahma/alpaca-cleaned (~52k examples)
- Chat format: ChatML
- Training: 1 epoch, SFTTrainer, 4-bit quantization, bf16, cosine scheduler
- Hardware: NVIDIA A100 80GB
- Final training loss: 1.0875
- Final validation loss: 1.0976
- Perplexity: ~3.00
LoRA Configuration
- Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- Rank: 8
- Alpha: 16
- Dropout: 0.05
- Trainable parameters: ~13M / 4B total (0.3%)
Usage
- Downloads last month
- 31