rubatotree/classify-blocks-act

classify-blocks ACT

An ACT policy fine-tuned for one long-horizon pick-and-place task: take a loose plastic block out of a transparent organiser, find the compartment whose resident block shows the same class, and place the loose block there.

Trained on rubatotree/classify-blocks-512-local (512 episodes / 201,004 frames at 15 fps: 20 measured teleoperation episodes plus 492 synthesised ones) for 30,000 steps at batch 16 in bfloat16 on a single RTX 4090.

Interface

observation.images.front	(3, 480, 640) float32 RGB in [0, 1], wrist camera, 15 fps
observation.state	(6,) float32: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll in degrees, gripper in percent
action	(6,) float32, absolute joint position command in the same units, 15 Hz

There is no language input; the task is fixed. The action chunk is 30, and temporal ensembling at coefficient 0.02 consumes the whole chunk on every observation, so the policy re-plans at the full 15 Hz. n_action_steps is 1 and is inert on this path: it never reaches the action queue.

Recipe

resnet18 ImageNet backbone, dim_model 512, 4 encoder and 1 decoder layer, VAE latent 32 with kl_weight 10, dropout 0.1, AdamW at 1e-5 for both the head and the backbone with weight decay 1e-4. VISUAL, STATE and ACTION are all MEAN_STD normalised, so the normaliser statistics stored beside the weights are part of the model and must be loaded with it.

Notes and limits

Nothing here has been executed on a physical arm. Success rates are unmeasured, and this checkpoint is a candidate for a first closed-loop trial rather than a validated policy. The measured fifth of the data comes from one session, one operator and one organiser, so no claim is made about other operators, other objects or other mounting. The synthetic episodes are a geometric and photometric approximation of the recorded scene: joints and contacts are simulated and only appearance is checked against the recording, so they augment the measured episodes rather than standing in as independent ground truth.

The dataset carries one extra column, training.action_valid. It is a supervision mask that is folded into the action padding mask during training and is not an input; do not pass it at inference.

Usage

import torch
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.factory import make_pre_post_processors

model_id = "rubatotree/classify-blocks-act"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

policy = ACTPolicy.from_pretrained(model_id).to(device).eval()
preprocess, postprocess = make_pre_post_processors(
    policy.config, model_id,
    preprocessor_overrides={"device_processor": {"device": str(device)}},
)

# `frame` is one LeRobotDataset row: observation.images.front and observation.state.
with torch.inference_mode():
    action = postprocess(policy.select_action(preprocess(frame)))

The saved config selects device: cuda. On a CPU-only machine, override the device while loading:

from lerobot.configs.policies import PreTrainedConfig

config = PreTrainedConfig.from_pretrained(model_id, cli_overrides=["--device=cpu"])
policy = ACTPolicy.from_pretrained(model_id, config=config)

To run the policy on an SO-100 follower with lerobot-record:

lerobot-record \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}}" \
  --robot.id=my_follower \
  --dataset.repo_id=${HF_USER}/classify-blocks-act-eval \
  --dataset.single_task="Match each loose plastic block to the examples in the organizer and put it in the matching compartment." \
  --policy.path=rubatotree/classify-blocks-act \
  --policy.device=cuda \
  --policy.use_degrees=true \
  --policy.temporal_ensemble_coeff=0.02