dunks/granite-350m-witness

Granite 350M Witness

A fine-tuned IBM Granite 4.0-H-350M model for autonomous agent supervision in Gas Town.

What it does

This model acts as a Witness — a lightweight patrol agent that monitors worker agents (polecats) and makes operational decisions:

• Detect stuck or idle workers and nudge them
• Identify zombie processes (running state, dead session)
• Escalate crash loops and infrastructure failures
• Route completion notifications to the merge queue
• Close cleanup beads after successful merges
• Flag orphaned work with no assigned worker

Every response is a single JSON tool call:

{"tool": "gt_nudge", "args": {"target": "gastown/furiosa", "message": "No progress detected."}}

Why 350M?

The witness role is narrow and structured — it reads a patrol snapshot and picks one tool from a fixed set. A 350M hybrid Mamba2/Transformer model handles this perfectly at:

• ~700MB VRAM (bf16) or ~300MB (Q4)
• 128K context window (Mamba2 architecture, constant-memory inference)
• 100% accuracy on 16 evaluation scenarios
• No API costs — runs entirely on-device

This replaces a full LLM API call (Qwen, Claude, etc.) that costs money and adds latency.

Architecture

Spec	Value
Base model	ibm-granite/granite-4.0-h-350m
Parameters	350M
Architecture	Hybrid Mamba2/Transformer (4 attention + 28 Mamba2 layers)
Context window	128K tokens
Training	Full fine-tune, 3 epochs, bf16
Training loss	0.346
Eval token accuracy	91.0%
Tool selection accuracy	16/16 (100%)
Avg inference latency	~4.5s (RTX 3060, PyTorch)

Training Data

Fine-tuned on curated witness patrol decisions extracted from real Gas Town sessions. The dataset includes:

• Patrol snapshots with polecat status, inbox messages, infrastructure health
• Correct tool-call responses for each scenario
• Format B (pure JSON, no prose)

Available Tools

gt_polecat_list, gt_polecat_nuke, gt_peek, gt_session_status,
gt_nudge, gt_mail_inbox, gt_mail_read, gt_mail_send,
gt_patrol_report, gt_handoff, gt_escalate,
bd_show, bd_list, bd_close, bd_children,
check_git_state, check_tmux_session, bash, none

Usage

Python (transformers)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "dunks/granite-350m-witness",
    dtype=torch.bfloat16,
    trust_remote_code=True,
).to("cuda").eval()

tokenizer = AutoTokenizer.from_pretrained("dunks/granite-350m-witness", trust_remote_code=True)

messages = [
    {"role": "system", "content": 'You are a Witness agent. Respond ONLY with JSON: {"tool": "<name>", "args": {}}'},
    {"role": "user", "content": "Polecats\n\n  ● gastown/furiosa  working\n\nLast activity: 45 minutes ago. No progress."},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=100, do_sample=False)

print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# {"tool": "gt_nudge", "args": {"target": "gastown/furiosa", "message": "Are you still working? No progress detected."}}

API Server

# Serve locally
python api_server.py --config serve.yaml

# Query
curl http://127.0.0.1:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "witness", "messages": [...]}'

Evaluation Results (16 scenarios)

Scenario	Tool Selected	Correct
Idle patrol	gt_patrol_report	Yes
Healthy polecat	gt_patrol_report	Yes
Stuck polecat (45min)	gt_nudge	Yes
Completed polecat	gt_mail_send	Yes
Crash loop (4 restarts)	gt_mail_send	Yes
Unpushed work	gt_mail_send	Yes
Infrastructure down	gt_mail_send	Yes
Idle with mail	gt_mail_send	Yes
Polecat done notification	gt_mail_send	Yes
Lifecycle shutdown	gt_polecat_nuke	Yes
Merged notification	bd_close	Yes
Help request	gt_mail_send	Yes
Zombie polecat	gt_mail_send	Yes
Stale spawn (20min)	gt_escalate	Yes
Orphaned bead	gt_mail_send	Yes
Full infra down	gt_mail_send	Yes

Requirements

• mamba-ssm and causal-conv1d for efficient Mamba2 CUDA kernels during training
• PyTorch with CUDA for inference (CPU works but ~10x slower)
• trust_remote_code=True required for Granite hybrid architecture

License

Apache 2.0 (inherits from base model)

Links

• Training pipeline: sfncore/tiny-model
• Gas Town: steveyegge/gastown
• Base model: ibm-granite/granite-4.0-h-350m