ravediamond/act-so101-gguf

ACT, SO-101 bowl task, GGUF for vla.cpp

A LeRobot ACT policy for an SO-101 arm, trained for 40k steps on ravediamond/so101_bowl_blue_in_pink and converted to GGUF for vla.cpp. Weights are unchanged F32. The ResNet's frozen batch norms are folded into the convs, the VAE encoder is left out (the latent is zero at inference), and the normalizer stats are stored in the file, so vla.cpp takes raw joint state and pixels and returns joint targets in degrees.

Two cameras at 480x640, sent in this order: front_top_camera, then wrist_camera. 6 dim state and action, 100 action chunk.

With PyTorch on a Jetson Orin NX the policy succeeded 9 times out of 10 on the real arm. It has not been run on the arm through vla.cpp yet.

Run

./build/vla-server act-so101-bowl-f32.gguf
./build/vla-bench --ckpt act-so101-bowl-f32.gguf --images 2 --size 640 --height 480

On a Jetson Orin NX 16 GB one chunk takes 57.7 ms (45.0 ms with --weight-dtype f16 --flash-attn), against 57.0 ms for PyTorch on the same board. For the arm, the --arch act client is in khanhnd61-vr/lerobot#2.

Parity

Against LeRobot's PyTorch ACTPolicy, going through its own pre and post processors, on the same synthetic input, the max abs action error is 3.4e-5 on CPU and about 0.02 on CUDA and Metal, on actions that span about 100 degrees.

ACT is from Zhao et al. 2023.