talzoomanzoo/qwen3-4b-base-math-32k

Qwen3-4B Base math LoRA (32k)

LoRA adapter from math SFT at 32k context, trained on Qwen/Qwen3-4B-Base.

Checkpoint: step 228 (2 epochs).

Load

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-4B-Base"
adapter = "talzoomanzoo/qwen3-4b-base-math-32k"

tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype="auto", device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(model, adapter)

Training

• Base model: Qwen/Qwen3-4B-Base
• Method: LoRA SFT (r=64, alpha=128) via TRL
• Max sequence length: 32768