davidnichols-ops/qwen3-4b-devin-sft

Qwen3-4B Devin SFT (Instruct)

Fine-tuned Qwen3-4B on Devin session data with tool-calling format. Trained on a GB10 (NVIDIA DGX Spark) with 128GB VRAM.

Training Details

• Base model: Qwen/Qwen3-4B
• Dataset: davidnichols-ops/adaptive-operator-v4-dataset (4,992 samples)
• Format: Chat messages (system/user/assistant) with tool calls in <tool>{json}</tool> format
• Epochs: 3
• Batch size: 16 (effective 32 with grad accumulation)
• Learning rate: 2e-5, cosine schedule
• Precision: BF16
• Hardware: NVIDIA GB10 (128GB VRAM, CC 12.1)
• Training time: ~48 min (cleaned instruct data, no control tokens)

Tool Calling Format

The model expects tool calls in this format:

<tool>
{"name": "shell", "arguments": {"command": "uv run pytest"}}
</tool>

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "davidnichols-ops/qwen3-4b-devin-sft",
    dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("davidnichols-ops/qwen3-4b-devin-sft")

messages = [
    {"role": "system", "content": "You are a helpful engineering assistant."},
    {"role": "user", "content": "Run the test suite and fix any failures"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Speculative Decoding

Pair with the draft model davidnichols-ops/qwen3-0.6b-devin-draft for 2-3x inference speedup. The 0.6B draft model proposes tokens; this 4B target model verifies them.