pepijn223/odsc-so101-act

SO-101 five-object kitting ACT · ODSC 2026

Pick the marked active object into the lined blue output tray, leaving four spare objects in place. Five native-scale objects require body, narrow-cylinder, offset-handle, rim and precision pinches. This checkpoint is specific to the exact workshop scene.

Training and inputs

250 accepted contact-based scripted demonstrations, 98,065 frames at 20 Hz, from 255 logged attempts. Balance: 5 objects × 5 pickup layouts × 10 varied accepted repetitions. Dataset revision 9c632abbcfefbd3f6c1430f8af883fc67f6c50f7. Official LeRobot v3 data and saved normalization processors are required.

ACT: resnet18; transformer width 512, 8 heads, feedforward 3200, 4 encoder layers, 4 VAE encoder layers, 1 decoder layers. The exact optimizer, batch size, schedule and model initialization are retained in train_config.json and train.sh. Candidate selection and all validation outcomes are retained in checkpoint-selection.json. Selected checkpoint: 40,000 base + 10,000 lower-LR refinement. Predict 30 actions and execute 30 (1.5 seconds) before a new observation. Offline loss does not depend on the action execution window. Hardware: one RTX 5090.

The complete recipe has two stages: 40,000 base ACT steps at learning rate 1e-4 (backbone 1e-5), then up to 20,000 refinement steps at 1e-5 (backbone 1e-6), with fresh AdamW initialized from the base policy. Refinement checkpoints are saved every 5,000 steps. The supplied checkpoint uses 10,000 refinement steps. Architecture, dataset, cameras and action contract stay the same. Balanced validation selects the checkpoint; a separate fresh test follows selection. All outcomes are retained. The full recipe includes the measured continuation from the saved 10,000-step refinement optimizer and RNG state.

Training provenance:

{
  "base_checkpoint_step": 40000,
  "base_model_sha256": "208822fbd4efefb2d1b9b50310434f4dd3b0677d840eccad002a76e6e9819c40",
  "selected_refinement_step": 10000,
  "combined_selected_updates": 50000,
  "optimizer_initialization": "Fresh AdamW; lower learning rate; same pinned dataset and architecture"
}

The saved train_config.json describes the refinement stage; train.sh and train-base.sh reproduce both stages.

Inputs: two 384 × 384 RGB cameras (fixed angled top and wrist) and six measured joint positions. Outputs: six absolute joint targets in radians at 20 Hz. Load saved preprocessor/postprocessor with the weights. Object poses are used only by the scripted teacher and success evaluator. ACT has no privileged pose input, attachment, teleport or scripted fallback.

Balanced evaluation · all attempts retained

Validation: 35/50 ACT across all 25 object/layout strata, two jittered resets each. Candidate checkpoints/windows use this set only; tie rules and all outcomes are retained. Fresh test at seed base 30000 is run after selection: ACT 35/50, expert 48/50.

The workshop acceptance criterion is strictly more than 50% overall (at least 26/50), as requested by the presenter after validation and before the fresh test. Earlier stricter selection criteria remain in the history. Each object's result is reported below, including failures on the small precision-grasp objects.

Object	Fresh ACT	Fresh expert
bottle	8/10	10/10
socket	5/10	10/10
screwdriver	8/10	10/10
tape	10/10	10/10
bracket	4/10	8/10

Each rollout has a 650-control-step maximum. Success requires previous lift above 5 cm, full oriented-bounds XY containment in the tray near floor height, released/open gripper, speed below 3 cm/s, and spare displacement staying below 1 cm throughout, for five consecutive 20 Hz steps. Bottle/socket must be vertical and tape flat within 15°; screwdriver/bracket may settle in any orientation. All successful validation, fresh-test and selected CPU runs are additionally audited for force-bearing contact with the tray floor at termination. The exact predicate is in the pinned scene source. All failures count; this is not a demonstration-success count.

This small, bounded test covers the five known objects and pickup layouts with pose jitter. It does not establish unseen-object robustness, changed-camera/lighting robustness, or real-robot transfer. Successful demonstration filtering is selection-biased.

CPU examples for the workshop

These are selected successful cases, separate from the full benchmark. cpu-examples.json retains all CPU trials used to find the examples. Each Space run generates a new simulation video. Allow a few minutes on CPU.

• socket, layoutC, seed30121: actual CPU ACT recording
• screwdriver, layoutC, seed30220: actual CPU ACT recording
• tape, layoutC, seed30320: actual CPU ACT recording
• bracket, layoutB, seed30410: actual CPU ACT recording
• bottle, layoutC, seed30020: actual CPU ACT recording

Participants inspect the prepared dataset, read the training recipe, and run this checkpoint during 45 minutes; they do not train during the session. Each eligible participant can duplicate the Space for an independent CPU simulator; PREREQUISITES.md explains account eligibility and a local fallback.

Provenance

MuJoCo 3.3.1, LeRobot 7521ab48cb55dd04cbd2961820ae766b0f1f7e3b. Full SO101 from MuJoCo Menagerie commit f3b6013629c7b94f8ee93fd977b161fa8350f9bc, Apache 2.0. Native-scale NVIDIA SimReady USD source 524747d9b5e1f21b5b18b56041788c01d745f1a1, CC BY 4.0: five objects, output tray, toolbox, bench and carton. Converted visuals retain transforms, material subsets, UVs and albedo. MuJoCo shading, compound/convex contacts, protective liner, masses and friction are authored approximations. NVIDIA physics and MDL/normal/ORM shaders are not ported. Individual licenses and attribution travel with the Space and dataset.

Weights SHA-256: dc5e3e8b1b23e6115ba542903c9331834685bea664d1a509fc6c069f393895cb. Dataset, scene, cameras, joint order, normalization and radians contract form this checkpoint's compatibility contract.

Hosted CPU delivery check

The balanced 50-reset benchmark above uses CUDA inference and NVIDIA EGL on Madeleine. The selected successful CPU videos also use NVIDIA EGL.

Hosted CPU Basic / Mesa EGL preset smoke check: 4/5 successes. These selected presets are separate from the balanced CUDA/NVIDIA EGL benchmark. Per-object outcomes: bottle: success, socket: failure, screwdriver: success, tape: success, bracket: success.