oku006/bussing-whiteboard-act

INSAIT whiteboard policy — bussing task (joint velocity)

The custom Action Chunking Transformer built in the INSAIT whiteboard project, fine-tuned on Sombit/bussing_task_new. This is not LeRobot's stock ACT — it is a different architecture (see below).

from bussing_act import BussingActPolicy      # shipped in this repo
policy = BussingActPolicy.from_pretrained(".", device="cuda")
out = policy.infer({
    "observation/image":       main_uint8_hwc,    # overhead camera
    "observation/wrist_image": wrist_uint8_hwc,   # wrist camera
    "observation/state":       joints_and_gripper # 8 floats
})
out["actions"]      # (16, 8) = 7 joint velocities rad/s + gripper

Loads with no training repository and no network access — the frozen encoder is rebuilt from a config stored here, not downloaded.

Architecture

vision encoder	frozen DINOv2-base ViT (86.6M), swappable
decoder	7-layer transformer decoder, learned action queries
encoder	4-layer, d_model 512, 8 heads, FFN 3200
CVAE	off
language conditioning	supported (lang_mode token / FiLM); unused here, single task
parameters	142.3M total, 55.7M trainable
chunk	16 steps, first 8 executed

The frozen swappable encoder is the point of the project: the vision backbone is pretrained and never updated, so the policy learns the task rather than the representation, and SigLIP / DINOv2 / JEPA can be exchanged without touching anything else. A SigLIP variant of this checkpoint also exists.

Action space

7 joint velocities (rad/s) + gripper position, matching JointVelocitiesAction(velocities=action[:7]). No conversion step, no end-effector pose needed, no workspace clamp — a velocity is not a position.

Set flip_gripper: False: trained on the raw recorded convention (1.0 = open, ~0.3 = closed on an object).

Measured — held-out episodes [2, 4, 10, 20, 21], 8 executed steps

Identical protocol for every row; the shared zero-motion baseline of 0.02671 rad/s is the check that they are comparable.

	π0.5	π0.5 own-stats	this (DINOv2)	SigLIP variant	LeRobot ACT
beats zero-motion	29.9%	28.9%	27.4%	24.9%	24.1%
velocity error (rad/s)	0.0187	0.0190	0.0194	0.0201	0.0203
gripper error	0.0073	0.0097	0.0266	0.0246	0.0349
motion ratio (1.0 = human)	0.921	0.928	0.857	0.729	0.940
joint error, 8 steps	0.0286	0.0289	0.0296	0.0306	0.0307
clipped by controller	1.8%	2.8%	2.9%	0.3%	2.1%

At 142M parameters with no robot pretraining, this sits within 2.5 points of a 3B model pretrained on thousands of hours of DROID data, and ahead of LeRobot's stock ACT. Inference 15 ms.

Limits

33 of 38 episodes for training, 5 held out. One task. No generalization to new objects or layouts expected. Every recording starts from ≈ the same home pose (1 cm spread), so starting far from it is out of distribution. Best validation came at step 13,000 of 20,000 — it overfits after that, which is why the best-validation checkpoint is the one shipped.