iromu/Qwen3-0.6B-tools-RL-step25

Qwen3-0.6B Tools (RL, GRPO step 25)

Qwen3-0.6B fine-tuned for tool calling and agent-style interactions: first LoRA SFT, then further trained with GRPO reinforcement learning (NeMo-RL) on multi-turn tool-calling rollouts. This checkpoint is the GRPO LoRA adapter from training step 25 merged back into the SFT model.

Base model

This model was built on top of the SFT fine-tune iromu/Qwen3-0.6B-tools (itself fine-tuned from Qwen/Qwen3-0.6B). The GRPO policy starts from the merged SFT weights and trains a LoRA adapter on top of them.

Training

Stage 1 — SFT (LoRA)

LoRA SFT with NVIDIA NeMo AutoModel (see iromu/Qwen3-0.6B-tools for the full SFT configuration).

Stage 2 — RL (GRPO + LoRA)

GRPO with LoRA using NVIDIA NeMo-RL:

• Algorithm: GRPO, 4 rollouts per prompt, up to 8 tool-call turns per rollout
• Training data: rl_tool_prompts split of the distillation dataset (3 tool families)
• Environment: in-memory tool-calling environment
• LoRA dimension: 32, alpha: 32, dropout: 0.05, target modules: *.proj (mirrors the SFT LoRA)
• Max sequence length: 4096
• Learning rate: 2.0e-6 (AdamW), weight decay: 0.01
• Global batch size: 8 (2 prompts x 4 generations, micro batch 2)
• Training steps: 50
• Mixed precision: bf16
• Rollout backend: vLLM (temperature 1.0, top-p 0.999)

Validation (val:accuracy on rl_tool_prompts, 64 prompts) across the run:

Step	val:accuracy
0 (SFT baseline)	0.0625
10	0.1016
20	0.0859
30	0.0625
40	0.0469
50	0.0156

This checkpoint is the GRPO LoRA adapter at training step 25, merged into the SFT weights and exported as a standalone BF16 model.

Dataset

SFT stage: sft_tools split. RL stage: rl_tool_prompts split of the r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation dataset.

Intended use

• Structured tool/function calling
• Agent-style multi-step interactions
• Small-footprint on-device or edge deployment
• Research: SFT → RL (GRPO) comparison for small tool-calling models

It is not intended to be a general replacement for larger Qwen models.

Usage

The model is a standard HF checkpoint (BF16, LoRA merged in):

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "iromu/Qwen3-0.6B-tools-RL-step25", torch_dtype="bfloat16"
)
tokenizer = AutoTokenizer.from_pretrained("iromu/Qwen3-0.6B-tools-RL-step25")