davidnichols-ops/adaptive-operator-v4

Adaptive Operator v4

A Qwen3.5-9B model fine-tuned with a custom 5-token control system for adaptive compute allocation in agentic workflows. The model routes every task to one of five compute levels at inference time, emitting a control token at the start of each response before executing structured tool calls.

This is a research artifact, not a production model. The fine-tune successfully taught control token routing and tool call formatting, but it degraded the base model's coding ability by 47 percentage points. The lessons learned from this project are more valuable than the model itself. Read on for the full honest breakdown.

---

Nerd TL;DR

For engineers who want the raw facts, warts and all.

What this is: A LoRA SFT + DPO fine-tune of Qwen3.5-9B that teaches the model to emit one of five control tokens ([FAST], [THINK], [VERIFY], [RECOVER], [ESCALATE]) and make structured XML+JSON tool calls. Trained on 4,992 SFT examples + 5,000 DPO pairs synthesized from Qwen v3.1 teacher responses, corrected by a pure-Python 5-reviewer orchestrator.

What this is NOT: A coding model. The fine-tune degraded raw coding ability. HumanEval+ pass@1 dropped from 64.0% (base) to 17.1% (fine-tuned, raw completion) to 13.4% (fine-tuned, chat template). The 95.1% / 81.1% HumanEval scores in the benchmark section are the base model with execution-based multi-sample selection (50 samples, T=0.2+0.8), not this fine-tuned model. Those benchmarks establish the ceiling of the base model's coding ability and validate the evaluation methodology.

Should you use this? Probably not for coding. Use the base Qwen3.5-9B for code generation. This model is useful if you need a standalone model that emits structured tool calls without a Python routing layer. If you already have a routing layer (like MAOS, LangChain, etc.), the base model with a good system prompt is better.

Training at a glance:

Stage	Steps	Time	Final Loss	Key Metric
SFT (LoRA r=64, bf16)	1,887	105 min	0.060	99.2% token accuracy
DPO (beta=0.1, 1 epoch)	630	77 min	3.9e-05	100% reward accuracy, margin ~43

The DPO loss went to near-zero. The chosen/rejected pairs were trivially separable -- the teacher narrated tools (950 chars avg) while the synthesized responses emitted structured calls (185 chars avg). The DPO signal is dominated by format differences, not reasoning quality. The model learned "don't narrate, emit structured calls" but did not learn deeper preference signals.

The control token system works but has routing edge cases. The model sometimes routes debugging tasks to [RECOVER] instead of [THINK] when the prompt contains failure-related keywords ("failing", "broken", "error"). Keyword sensitivity from the training data distribution.

Coding SFT LoRA was attempted and abandoned. A separate coding LoRA trained on MBPP + synthetic data (1,256 examples) hurt performance (57.3% vs 64% base). Overfitting on simple patterns. The base model was already strong; fine-tuning on narrow data degraded it.

Infrastructure: Trained on a rented RTX PRO 5000 Blackwell (48GB VRAM) via vast.ai. Total cost ~$3.50 ($2-3 Together AI inference + $0.30 GPU rental). bf16, no quantization, SDPA attention (flash-attn wouldn't build), no packing (cross-contamination risk without flash attention).

Deployment formats: bf16 GGUF (17GB), Q8_0 (8.9GB), Q6_K (6.9GB), Q5_K_M (6.0GB), Q5_K_S (5.9GB), Q4_K_M (5.2GB), Q4_K_S (5.0GB), Q4_0 (4.9GB), Q3_K_M (4.3GB), MLX 4-bit (4.7GB). fp8 is NOT supported by llama.cpp.

Thinking mode gotcha: Qwen3.5 has built-in thinking mode. You MUST pass enable_thinking=False in the chat template, or the model generates a thinking block before the control token. This is why the chat-template benchmark (13.4%) was worse than raw completion (17.1%) -- thinking tokens contaminated the code output.

---

The Honest Assessment

This project succeeded at what it set out to do (teach control tokens + tool call format) and failed at what we hoped it would do (maintain coding ability while adding agentic behavior). Both outcomes are instructive.

What we got right

• The pipeline works. Data generation (5K prompts -> teacher inference -> 5-reviewer orchestrator -> synthesis -> quality filter) produced clean training data in 3 seconds for 5,000 responses, at zero API cost.
• The training converged. SFT hit 99.2% token accuracy in 105 minutes. DPO hit 100% reward accuracy in 77 minutes. The format/routing behavior is easy to learn.
• The cost was $3.50. Together AI inference ($2-3) + vast.ai GPU rental ($0.30). No expensive API bills, no long training runs.
• The benchmarking methodology is sound. Execution-based multi-sample selection (50 samples, T=0.2+0.8) is the standard technique used by DeepSeek-Coder and CodeLlama. We validated it on the base model: 64% greedy -> 81.1% with execution selection.
• The deployment pipeline works. fp16 safetensors, MLX 4-bit, and 9 GGUF quantizations all produced and uploaded.

What we got wrong

• Fine-tuning degraded coding ability by 47 percentage points. HumanEval+ dropped from 64.0% (base) to 17.1% (fine-tuned). The LoRA adapters optimized for format compliance at the expense of semantic code generation. This is the fundamental trade-off of fine-tuning on format data: you get format, you lose substance.
• The DPO signal was format-only. Cross-model DPO pairs (teacher narration vs synthesized structured calls) are trivially separable. The loss went to 3.9e-05. The model learned "don't narrate" but not "reason better." For deeper preference learning, self-preference pairs with multiple temperature samples would have been better.
• The coding SFT LoRA overfit. Training on 1,256 MBPP + synthetic examples hurt the base model (57.3% vs 64%). The base model's coding ability was already strong. Fine-tuning on narrow, simple data degraded generalization. Lesson: don't fine-tune a strong model on weak data.
• The control token routing has keyword sensitivity. The model routes debugging tasks to [RECOVER] instead of [THINK] when the prompt contains "failing", "broken", "error". The training data distribution biased the routing.
• Chat template evaluation was misleading. The thinking mode contamination (13.4% vs 17.1%) took time to diagnose. Should have tested raw completion first.
• Flash attention never built. Python 3.12 + CUDA 13.0 had no prebuilt wheel. Used SDPA with packing=False, accepting a 20-30% throughput hit.

Lessons that matter more than the model

These lessons cost $3.50 and 2 days to learn. They're worth more than the model:

1. Fine-tuning is a trade-off, not an upgrade. If your base model is already good at something, fine-tuning on different data will degrade that ability. The LoRA adapters don't add capability -- they redirect capacity. Know what you're trading away.

2. DPO is only as good as your pairs. If your chosen/rejected pairs differ in format (not reasoning), the model learns format (not reasoning). Near-zero DPO loss is a red flag, not a success. It means the task was trivial.

3. Execution-based benchmarking is the correct methodology. Greedy pass@1 underestimates model ability. Multi-sample with execution selection (50 samples, T=0.2+0.8) gives the true ceiling. 28/164 problems had no passing sample at T=0.2; 15 of those were solved at T=0.8. High-temperature diversity is critical.

4. Test the base model first. We spent time trying to "recover" coding ability with a coding SFT LoRA, not realizing the base model was already strong (64% HumanEval+). If we'd benchmarked the base model first, we'd have known the ceiling and not wasted time on the coding LoRA.

5. Template-based synthesis is deterministic but shallow. The 5-reviewer orchestrator + template synthesis produced correctly formatted responses at 1,623 responses/sec for $0. But the improved responses were structurally correct and semantically simple. The base model provides the reasoning; the templates provide the format.

6. $3.50 is enough for a 9B LoRA fine-tune. Together AI inference ($2-3 for 5K responses) + vast.ai RTX PRO 5000 Blackwell ($0.30 for 3 hours). You don't need expensive infrastructure for experimentation.

---

Model Details

Field	Value
Base model	Qwen/Qwen3.5-9B
Parameters	9B
Training method	LoRA SFT + DPO
LoRA rank	64 (alpha 128, dropout 0.05)
Training precision	bf16 (no quantization during training)
Training data	4,992 SFT examples + 5,000 DPO pairs
Training hardware	RTX PRO 5000 Blackwell (48GB VRAM)
Context length	2048 tokens
Attention implementation	SDPA (PyTorch native)
Packing	Disabled (no flash attention available)
Total training time	~3 hours (SFT 105 min + DPO 77 min)
Total cost	~$3.50

Control Token System

The model emits one of five control tokens at the start of every response:

Token	Usage	Distribution in training data
[FAST]	Direct action, simple tasks	47.4% (2,366)
[THINK]	Multi-step reasoning before acting	41.6% (2,078)
[VERIFY]	Act then confirm result (destructive/irreversible)	2.9% (145)
[RECOVER]	Reassess after failure, try different strategy	7.1% (354)
[ESCALATE]	Surface for human decision (high-risk/security)	1.0% (49)

Tool Call Format

<tool_call>
{"name": "shell", "arguments": {"command": "uv run pytest"}}
</tool_call>```

Supported tools: `shell`, `file_read`, `file_write`, `file_edit`, `file_list`, `grep`, `find_file`, `git`, `web_search`, `web_fetch`, `todo_write`, `ask_user`.

## Usage

### MLX (Apple Silicon)

```python
import mlx_lm

model, tokenizer = mlx_lm.load("davidnichols-ops/adaptive-operator-v4-mlx-4bit")

SYSTEM = """You are an adaptive engineering operator. You inspect systems, reason through problems, take actions, verify results, recover from failures, and decide when deeper thinking is necessary. You work from evidence, not assumptions.

## Compute Modes (control tokens)
Begin every response with exactly one control token to signal how much reasoning the task needs:
  [FAST]     -- direct, low-latency action. Use for simple tool calls, file reads, status checks, and routine edits.
  [THINK]    -- extended reasoning before acting. Use for design, debugging, refactoring, multi-step planning, and ambiguous problems.
  [VERIFY]   -- act, then verify the result before declaring success. Use when a change must be confirmed (tests pass, file written, deploy succeeded).
  [RECOVER]  -- a previous attempt failed; reassess and try a different strategy. Use after errors, broken builds, or unexpected output.
  [ESCALATE] -- the task exceeds safe autonomous scope; stop and surface the situation for a human. Use for irreversible or high-risk actions.

## Tool Use
When a task requires a tool, emit a structured tool call in this exact format:
  <tool_call>
  {"name": "tool_name", "arguments": {"param": "value"}}
  </tool_call>

After each tool call you will receive the result. Use it to decide your next action. When the task is complete, give a concise final answer with no tool calls."""

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "Show me the last 10 git commits."},
]
chat = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
response = mlx_lm.generate(model, tokenizer, prompt=chat, max_tokens=256)
print(response)

HuggingFace Transformers (fp16)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("davidnichols-ops/adaptive-operator-v4", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("davidnichols-ops/adaptive-operator-v4")

# CRITICAL: disable thinking mode or control tokens won't appear at response start
chat = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)

Ollama (GGUF)

ollama create adaptive-operator-v4 -f Modelfile
ollama run adaptive-operator-v4 "Show me the last 10 git commits."

GGUF Quantizations

All quantizations are in the GGUF repo, generated from the bf16 source with llama.cpp:

Format	Size	Bits/weight	Best for
bf16	17 GB	16.0	Maximum fidelity, CPU inference
Q8_0	8.9 GB	8.5	Near-lossless, good speed/quality balance
Q6_K	6.9 GB	6.6	High quality, moderate size
Q5_K_M	6.0 GB	5.8	Good quality/size trade-off
Q5_K_S	5.9 GB	5.6	Slightly smaller than Q5_K_M
Q4_K_M	5.2 GB	5.0	Recommended default for 4-bit
Q4_K_S	5.0 GB	4.8	Smaller 4-bit variant
Q4_0	4.9 GB	4.7	Fastest 4-bit, legacy format
Q3_K_M	4.3 GB	4.1	Smallest viable, quality degradation

fp8 is not supported by llama.cpp. The highest-fidelity 16-bit option is bf16.

Training Pipeline

Prompt Generator (5K unique prompts across 10 categories)
    -> Qwen v3.1 Teacher Inference (Together AI, 32 parallel workers, ~$2-3)
    -> 5-Reviewer Orchestrator (pure Python heuristics, 1,623 responses/sec)
       - Code Quality & Correctness
       - Tool Selection & Usage
       - Control Token Routing
       - Error Handling & Edge Cases
       - Response Format & Clarity
    -> Data Quality Filter (dedup + n-gram decontamination)
    -> SFT Export + DPO Pair Generation
    -> LoRA SFT Training (bf16, 3 epochs, lr=2e-4, r=64)
    -> LoRA DPO Training (bf16, 1 epoch, lr=5e-5, beta=0.1)
    -> Merge LoRA -> fp16 model
    -> Quantization (MLX 4-bit + 9 GGUF variants)

Training Metrics

SFT (3 epochs, 1,887 steps, 105 min)

Metric	Value
Final train loss	0.060
Final token accuracy	99.2%
Convergence	2.09 -> 0.08 in 110 steps (6% of training)
Throughput	2.39 samples/sec, 0.30 steps/sec
Total tokens	5.9M

DPO (1 epoch, 630 steps, 77 min)

Metric	Value
Final train loss	3.9e-05 (near-zero -- see lessons above)
Reward accuracy	100%
Reward margin	~43 (chosen +8.5, rejected -35.5)
Throughput	1.09 samples/sec, 0.14 steps/sec

Benchmark Results

HumanEval / HumanEval+ (EvalPlus) -- Base Qwen3.5-9B

These benchmarks were run on the base model to establish the coding ceiling and validate the evaluation methodology. The fine-tuned model's coding ability is lower (see below).

Method	HumanEval pass@1	HumanEval+ pass@1
Greedy (temp=0)	70.7%	64.0%
20 samples (T=0.2), random pick	69.9%	62.1%
20 samples, execution-selected (base tests)	82.9%	70.7%
20 samples, execution-selected (base+plus tests)	82.9%	72.6%
50 samples (T=0.2+0.8), execution-selected	95.1%	81.1%
pass@10 (20 samples, T=0.2)	81.5%	74.6%

Fine-tuned Model -- Coding Ability Impact

Model	HumanEval+ pass@1 (raw completion)
Base Qwen3.5-9B	64.0%
Fine-tuned (chat template)	13.4%
Fine-tuned (raw completion)	17.1%
Coding SFT LoRA merged	57.3% (abandoned -- overfit)

Fine-tuned Model -- Agentic Metrics

Metric	Value
SFT final loss	0.060
DPO reward accuracy	100%
Control token routing accuracy	86% FAST mode on simple tasks
Tool call format compliance	>99%

Dataset Statistics

SFT Dataset (4,992 examples)

Category	Count	Percentage
tool_use_file_ops	889	17.8%
coding_write	828	16.6%
coding_debug	615	12.3%
tool_use_shell	543	10.9%
tool_use_git	435	8.7%
coding_test	404	8.1%
coding_refactor	376	7.5%
planning	374	7.5%
tool_use_search	298	6.0%
recovery	238	4.8%

Difficulty Distribution

Level	Count	Percentage
1 (simple)	2,374	47.5%
2 (moderate)	2,035	40.8%
3 (complex)	591	11.8%

DPO Dataset (5,000 pairs)

Cross-model preference pairs: original Qwen v3.1 teacher response (rejected) vs synthesized improved response (chosen). The rejected responses are 5x longer (950 chars vs 185 chars) -- the teacher was verbose with narration. The improved responses are concise and structured.

Repository Contents

Path	Description
model.safetensors	Merged fp16 model (17GB)
sft_adapter/	SFT LoRA adapter (465MB)
dpo_adapter/	DPO LoRA adapter (908MB)
config.json	Model configuration
tokenizer.json	Tokenizer
chat_template.jinja	Chat template
TRAINING_REPORT.md	Full training report with 14 issues encountered
PIPELINE_ARCHITECTURE.md	Pipeline architecture reference
benchmark/	Benchmark scripts (multi-sample, execution selection, etc.)
training/	Training scripts (SFT, DPO)

Related Repositories

Repository	Description
adaptive-operator-v4-mlx-4bit	MLX 4-bit quantized for Apple Silicon (4.7GB)
adaptive-operator-v4-gguf	9 GGUF quantizations (bf16, Q8_0, Q6_K, Q5_K_M, Q5_K_S, Q4_K_M, Q4_K_S, Q4_0, Q3_K_M)
adaptive-operator-v4-dataset	SFT + DPO training data

Training Cost

Resource	Usage	Cost
Together AI inference (5K responses)	5.4M tokens	~$2-3
Vast.ai RTX PRO 5000 Blackwell	~3 hours	~$0.30
Total		~$3.50

Important: Thinking Mode

Qwen3.5 has a built-in thinking mode that prepends internal reasoning before the response. To get control tokens at the start of the output, you must pass enable_thinking=False when applying the chat template:

chat = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)

Without this, the model generates a thinking block first, and the control token appears later in the response.

Limitations

• Coding ability degradation: The fine-tune degraded raw coding ability (64% -> 17% HumanEval+). Use the base Qwen3.5-9B for coding tasks.
• Context length: 2048 tokens (may truncate long conversations)
• Tool call accuracy: The model emits correctly formatted tool calls but may select suboptimal tools for ambiguous tasks
• Reasoning depth: SFT responses are structurally correct but semantically simple -- the base model provides semantic content, but the synthesis templates produce minimal stubs
• Repetition: 4-bit quantization with greedy decoding can cause repetition loops; use temperature sampling (0.7-0.8)
• System prompt required: The control token system only activates when the system prompt is provided
• Thinking mode: Must disable enable_thinking in the chat template
• THINK routing: The model sometimes routes debugging tasks to [RECOVER] instead of [THINK] when the prompt contains failure-related keywords
• DPO overfitting: Near-zero DPO loss indicates format-dominated signal, not reasoning quality
• Training data bias: 47.5% of training data is difficulty level 1 (simple). The model may underperform on complex, multi-step tasks

Training Configuration

BASE_MODEL = "Qwen/Qwen3.5-9B"
LORA_R = 64
LORA_ALPHA = 128
LORA_DROPOUT = 0.05

# SFT
SFT_EPOCHS = 3
SFT_LEARNING_RATE = 2e-4
SFT_BATCH_SIZE = 4
SFT_GRADIENT_ACCUMULATION = 2  # effective batch = 8
MAX_SEQ_LENGTH = 2048

# DPO
DPO_EPOCHS = 1
DPO_LEARNING_RATE = 5e-5
DPO_BETA = 0.1

# Infrastructure
PRECISION = "bf16"
ATTENTION = "sdpa"
PACKING = False
GRADIENT_CHECKPOINTING = True
GPU = "RTX PRO 5000 Blackwell (48GB)"

License

Apache 2.0 (derived from Qwen3.5-9B)

Citation

@misc{adaptive-operator-v4,
  title={Adaptive Operator v4: Qwen3.5-9B with Control Token Routing},
  author={David Nichols},
  year={2025},
  url={https://huggingface.co/davidnichols-ops/adaptive-operator-v4}
}