rocochallenge2025/rocochallengeIROS2026-ACT-roco-umi

ROCO IROS 2026 ACT Policy

Model Description

This repository contains the official competition ACT baseline checkpoint for the ROCO IROS 2026 UMI dataset and the ISuit-V3 robot. ACT (Action Chunking with Transformers) predicts a one-second chunk of bimanual absolute Cartesian target actions from three RGB observations and the current 16-dimensional robot state.

The release layout follows the information split used by the designated Embodied-AI-6/pi0.5 reference: inference weights, policy configuration, training metadata, preprocessing state, postprocessing state, and a model card. The policy and all parameters in this repository are ACT-specific and come from the actual training run.

The inference-relevant ACT source is included in act_source/. It is derived from tonyzhaozh/act commit 742c753c0d4a5d87076c8f69e5628c79a8cc5488 and contains the trained run's three-independent-backbone modification. The included ACT source retains its MIT license; the bundled DETR-derived source retains its Apache-2.0 license in act_source/detr/LICENSE.

Model weights license: MIT

The root LICENSE applies to the trained ACT model weights and release-specific assets. It does not replace the preserved upstream ACT MIT notice in act_source/LICENSE or the DETR-derived Apache-2.0 notice in act_source/detr/LICENSE.

Training Dataset

Field	Value
Dataset	rocochallenge2025/roco_iros2026_umi_dataset
Robot	ISuit-V3
Dataset episodes	4,928
Training / validation episodes	3,942 / 986
Dataset frames	7,459,440
Tasks	5
Control frequency	30 Hz
Split	deterministic 80/20 by source_worldcode, seed 0
Model cameras	fpv_left, left_hand_left, right_hand_left
Image source	H264 MP4, YUV420P, decoded to RGB

The ACT conversion stores lightweight per-episode HDF5 indexes with external references to the three source H264 streams. Normalization statistics were computed from the training split only. The source metadata does not declare the physical unit for Cartesian positions or gripper commands.

Training Configuration

Field	Value
Batch size	64
Gradient accumulation	1
Optimizer	AdamW
Transformer learning rate	5e-5
Backbone learning rate	1e-5
Weight decay	1e-4
Betas / epsilon	(0.9, 0.999) / 1e-8
Scheduler	none
Gradient clipping	none
Epoch cap	700
Completed epoch	645
Completed optimizer steps	40,000
Random seed	0
Loss	masked normalized L1 + 10 × KL
Chunk size	30
Backbone	three independent ResNet-18 backbones
Hidden / feed-forward dimension	512 / 3,200
Transformer encoder / decoder layers	4 / 7
CVAE action encoder layers	4
Attention heads / dropout	8 / 0.1

The full machine-readable record is in train_config.json.

Model Input

The model consumes one observation:

• observation.state: [B, 16] float32. The order is left XYZ, left quaternion XYZW, right XYZ, right quaternion XYZW, left gripper, right gripper.
• Three RGB images in this exact order: fpv_left, left_hand_left, right_hand_left.
• Each raw image is [384, 480, 3] uint8 RGB. Images are not resized.
• The public tensor layout is [B, 3 cameras, 384, 480, 3]; preprocessing converts it to [B, 3, 3, 384, 480], divides by 255, and applies ImageNet mean/std inside the policy forward pass.
• Robot state is normalized with training-split mean and standard deviation.

qvel, task ID, timestamp, and the other source camera streams are not model inputs. The exact operation order and statistics filename are recorded in policy_preprocessor.json.

Model Output

The model emits normalized float32 actions with shape [B, 30, 16]. Each step contains left target XYZ, left target quaternion XYZW, right target XYZ, right target quaternion XYZW, and the two gripper commands. The postprocessor applies action * train_std + train_mean to recover dataset action units.

For real data, training paired observation t with an action chunk beginning at max(0, t-1). The default deployment setting queries every 30 frames and does not use temporal aggregation.

Checkpoint

model.safetensors was exported from policy_best.ckpt and contains only the inference model state. It contains no optimizer, scheduler, random state, or training history.

Field	Value
Epoch	629
Optimizer step	39,060
Samples seen	2,483,460
Posterior validation loss	0.0823896201
Selection method	lowest recorded posterior validation total loss
Trainable parameters	106,225,169
FP32 state values in the file	106,299,153

The training run continued to step 40,000, but the final checkpoint had a higher offline prior action MAE than the selected checkpoint.

Usage

Install the tested dependencies and run the end-to-end smoke observation:

python -m pip install -r requirements.txt
python inference.py --device cuda

The command constructs ACT, loads model.safetensors, preprocesses a public input-format observation, predicts a 30-step action chunk, applies action unnormalization, and checks that the result is finite.

For recorded observations, save arrays with these shapes:

• state.npy: [B, 16] float32 raw robot state.
• images.npy: [B, 3, 384, 480, 3] uint8 RGB in the camera order above.

Then run:

python inference.py \
  --device cuda \
  --state-npy state.npy \
  --images-npy images.npy \
  --output-npy action_chunk.npy

action_chunk.npy has shape [B, 30, 16] in dataset action units. Callers that decode video with OpenCV must convert its BGR frames to RGB before forming images.npy.

The same executable pipeline can be imported directly:

from pathlib import Path
import numpy as np
import torch
from inference import load_policy, load_processor_state
from inference import preprocess_observation, postprocess_action

release = Path(".")
policy, config = load_policy(release, device="cuda")
state_stats, action_stats = load_processor_state(release)

# raw_state: [B,16]; rgb_images: [B,3,384,480,3], uint8 RGB.
raw_state = np.load("state.npy")
rgb_images = np.load("images.npy")
state, images = preprocess_observation(
    raw_state, rgb_images, state_stats, config, device="cuda"
)
with torch.inference_mode():
    normalized_chunk = policy(state, images)
    action_chunk = postprocess_action(normalized_chunk, action_stats)

Evaluation

On the fixed offline validation protocol, the selected checkpoint was tested on 986 episodes with five horizon-safe contexts per episode. Its zero-latent prior normalized action MAE was 0.0738040. This is an open-loop comparison against recorded expert actions; it is not a robot-controlled rollout.

Closed-loop task success results are not included in this release. The offline MAE must not be interpreted as a success rate.

Release Validation

validation_report.json records checkpoint loading, exact tensor conversion, preprocessing, postprocessing, complete inference, output consistency, and credential-scan results. SHA256SUMS provides hashes for all published files.