giangkh19/qwen-0.5b-coder-r8

Qwen2.5-Coder-0.5B-Instruct — LoRA r8 (coder_r8)

Fine-tuned Qwen/Qwen2.5-Coder-0.5B-Instruct (494M) on 6,425 code samples with LoRA rank 8.
+6.6 pts on a frozen code-only benchmark (84 Python unit-test tasks) vs the base model.

Available as a lightweight adapter (~17 MB). Load it on top of the base — no 1 GB merge needed.

Metrics

Frozen benchmark data/eval/coding.jsonl (84 diverse Python functions, each with 4–5 unit tests, greedy decoding temperature=0). All 84 verified 100% on reference solutions; no train-eval leak (max 6-gram Jaccard 0.009).

Model	Coding (84)	Avg
Qwen2.5-Coder-0.5B-Instruct (base)	73.9	73.9
This adapter — coder_r8 (r=8)	80.5	80.5
Δ	+6.6	+6.6

Eval is code-only (24 original + 60 algorithmic extensions, each function-masked and run in a subprocess).

Training Data (6,425 rows, ≤4000 tokens, no truncation)

Source	Rows	What
Claude production code (claude_opus_743_short → chunked)	743	Real production tasks (queue consumer, file watcher, migration helpers) in Python/JS/TS/Rust/Java — chunked at block boundaries so every sample <4000 tokens
Curated code knowledge (authored)	685	OOP / Big-O / HTTP-REST / SQL / data structures / debugging / code-trace output
Magicoder-OSS-Instruct-75K (MIT) — Python slice	5,000	Real open-source Python instructions, deduplicated against eval entry function names
Total	6,425 → chunked to 2,018	Long answers split at newline boundaries; median ~1305 tokens

• License: base Apache-2.0, Magicoder slice MIT.
• De-duplicated across sources, then chunked rather than truncated — no tail loss.

Training Configuration

Param	Value
Base	Qwen/Qwen2.5-Coder-0.5B-Instruct
Method	LoRA — r=8, alpha=16, target q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj, dropout 0.05
Rank	8 → 4.4M trainable (0.88 %) of 498M total
Max seq length	4096 (with gradient_checkpointing=True + enable_input_require_grads() for LoRA CKPT)
Epochs	1
LR	2e-4 (linear decay)
Batch	1 × grad-accum 8 = effective 8
Optim	adamw_torch (bf16, max_grad_norm=1.0)
Hardware	RTX 3090 24 GB
Samples/sec	~4.0 (post-fix)
Final train loss	~0.885
Code	train.py (standard Trainer, completion-only masking via response template \nassistant)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen2.5-Coder-0.5B-Instruct"
adapter = "giangkh1908/qwen-0.5b-coder-r8"  # <-- update to your actual HF repo id

tok = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base, trust_remote_code=True, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)
model.eval()

# chat template (single-turn) — same as training
messages = [{"role": "user", "content": "Write a Python function to check if a string is palindrome."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False, temperature=0)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

CLI reproduction (from this repo):

git clone https://github.com/giangkh1908/Finetune_Qwen2.5-0.5B.git && cd Finetune_Qwen2.5-0.5B
python harness/run_eval.py --model Qwen/Qwen2.5-Coder-0.5B-Instruct --adapter giangkh1908/qwen-0.5b-coder-r8 --tag r8
# → coding 80.5 on 84 tasks vs base 73.9

Limitations

• 0.5B is tiny; it learns style + short-function patterns but cannot reliably produce 25K-token reasoning.
• The eval is 84 Python function-implementation tasks. It does not measure long-form production coding, IDE file-watcher tasks, or multi-language depth.
• The training set is Python-heavy (Magicoder Python slice); other languages (Rust/Java/JS) are underrepresented relative to the original Claude dataset.

Citation

@misc{qwen25coder-r8,
  title  = {Qwen2.5-Coder-0.5B LoRA r=8 on 6.4k code samples},
  author = {giangkh1908},
  howpublished = {Hugging Face},
  year   = {2026}
}