iMaxxgx/act-so101-conveyor-sort

ACT policy: SO-101 conveyor sort (simulation, OpenVINO)

An ACT (Action Chunking Transformer) policy for a simulated SO-101 arm in the conveyor_sort scene of the Physical AI Runtime MuJoCo plugin. Items ride a belt past the arm; it drops red, blue and green items into the bin of their colour, and purple or cracked items into the reject bin.

It was trained with Physical AI Studio on about 100 episodes recorded from the simulation's built-in demonstrator, and exported to OpenVINO as a policy package that physicalai.inference.InferenceModel loads directly.

Files

File	Content
manifest.json	Policy package manifest: inputs, outputs and preprocessing
act.xml, act.bin	OpenVINO IR

Inputs: state [6] (SO-101 joint positions), images.gripper and images.overview (640×480 RGB, letterboxed to 384×384 by the manifest's preprocessor). Output: action [100, 6], a chunk of joint targets at 30 Hz.

Run it

The tutorial notebook examples/tutorials/run_policy_in_simulation.ipynb runs this policy end to end with a live viewer. In short:

from physicalai.capture import IPCamera
from physicalai.inference import InferenceModel
from physicalai.runtime import ChunkedActionQueue, LerpSmoother, PolicySource, RobotRuntime, SyncExecution
from physicalai_mujoco_plugin import MuJoCoRobot

robot = MuJoCoRobot(profile="so101", scene="conveyor_sort", rate_hz=30, http_port=8770)
robot.connect()
robot.get_observation()  # renders the first camera frames

policy = InferenceModel.from_pretrained("iMaxxgx/act-so101-conveyor-sort", backend="openvino", device="CPU")
runtime = RobotRuntime(
    robot=robot,
    cameras={
        "overview": IPCamera(url="http://127.0.0.1:8770/cameras/overview/mjpeg", width=640, height=480, fps=30),
        "gripper": IPCamera(url="http://127.0.0.1:8770/cameras/wrist/mjpeg", width=640, height=480, fps=30),
    },
    action_source=PolicySource(
        model=policy,
        execution=SyncExecution(request_threshold=0.95),
        action_queue=ChunkedActionQueue(smoother=LerpSmoother()),
    ),
    fps=30.0,
)
runtime.connect()
runtime.run(duration_s=120)
runtime.disconnect()

request_threshold=0.95 asks for a fresh chunk after about every fifth action. The policy needs it: it does not recover from a missed grasp by itself, and with 0.5 it sorted nothing in a 120 s test.

Results

In 120 s at the trained belt speed (3 cm/s), with the setup above: 10 items fed, 5 sorted correctly, 0 into a wrong bin, 1 missed; the rest were still on the belt. Expect more misses at higher belt speeds.

Limitations

• Simulation only: trained and evaluated in the MuJoCo conveyor_sort scene, not on a physical SO-101.
• It expects the scene's default overview camera pose and the SO-101 wrist camera, at 640×480.