oku006/bussing-whiteboard-act-gripper

INSAIT whiteboard policy — gripper-weighted variant

A variant of oku006/bussing-whiteboard-act, not a separate model. Same architecture, same data, one training change: the gripper channel is normalized alongside the seven velocity channels instead of being left raw.

Why it exists: left raw (std 0.27) against normalized velocities (std 1.0), the gripper received only ~3.7% of the L1 signal. Normalizing it gives it a fair 1/8. The effect is a clean trade — gripper error improves 20% (0.0266 → 0.0212) and clipping drops from 2.9% to 0.5%, while the velocity metrics get slightly worse — because the total loss signal is conserved.

Use the main checkpoint by default. Reach for this one if the arm reaches the right place but the grasp closes at the wrong moment, which is the specific failure this addresses.

from bussing_act import BussingActPolicy      # shipped in this repo
policy = BussingActPolicy.from_pretrained(".", device="cuda")
out = policy.infer({
    "observation/image":       main_uint8_hwc,    # overhead camera
    "observation/wrist_image": wrist_uint8_hwc,   # wrist camera
    "observation/state":       joints_and_gripper # 8 floats
})
out["actions"]      # (16, 8) = 7 joint velocities rad/s + gripper

Loads with no training repository and no network access — the frozen encoder is rebuilt from a config stored here, not downloaded.

Architecture

vision encoder	frozen DINOv2-base ViT (86.6M), swappable
decoder	7-layer transformer decoder, learned action queries
encoder	4-layer, d_model 512, 8 heads, FFN 3200
CVAE	off
language conditioning	supported (lang_mode token / FiLM); unused here, single task
parameters	142.3M total, 55.7M trainable
chunk	16 steps, first 8 executed

The frozen swappable encoder is the point of the project: the vision backbone is pretrained and never updated, so the policy learns the task rather than the representation, and SigLIP / DINOv2 / JEPA can be exchanged without touching anything else. A SigLIP variant of this checkpoint also exists.

Action space

7 joint velocities (rad/s) + gripper position, matching JointVelocitiesAction(velocities=action[:7]). No conversion step, no end-effector pose needed, no workspace clamp — a velocity is not a position.

Set flip_gripper: False: trained on the raw recorded convention (1.0 = open, ~0.3 = closed on an object).

Measured — held-out episodes [2, 4, 10, 20, 21], 8 executed steps

Identical protocol for every row; the shared zero-motion baseline of 0.02671 rad/s is the check that they are comparable.

	π0.5	π0.5 own-stats	main whiteboard	this (gripper-weighted)	LeRobot ACT
beats zero-motion	29.9%	28.9%	27.4%	25.5%	24.1%
velocity error (rad/s)	0.0187	0.0190	0.0194	0.0199	0.0203
gripper error	0.0073	0.0097	0.0266	0.0212	0.0349
motion ratio (1.0 = human)	0.921	0.928	0.857	0.845	0.940
joint error, 8 steps	0.0286	0.0289	0.0296	0.0304	0.0307
clipped by controller	1.8%	2.8%	2.9%	0.5%	2.1%

At 142M parameters with no robot pretraining, this sits within 2.5 points of a 3B model pretrained on thousands of hours of DROID data, and ahead of LeRobot's stock ACT. Inference 15 ms.

Limits

33 of 38 episodes for training, 5 held out. One task. No generalization to new objects or layouts expected. Every recording starts from ≈ the same home pose (1 cm spread), so starting far from it is out of distribution. Best validation came at step 13,000 of 20,000 — it overfits after that, which is why the best-validation checkpoint is the one shipped.