MolmoAct2-RoboDojo

MolmoAct2 is an open vision-language-action model for robot control. It builds on Molmo2-ER and attaches a flow-matching continuous action expert that conditions on the VLM key-value cache through a per-layer connection.

This 90K-step checkpoint is fine-tuned on the RoboDojo training set with absolute joint-pose control and language instructions. It is intended for both further fine-tuning and dual-arm RoboDojo policy inference.

Quick Links

• 📂 Models: Base Model, MolmoAct2 Models, Finetuned Models
• 📂 Dataset and Benchmark: RoboDojo
• 📄 Paper: arXiv:2605.02881
• 💻 Code: allenai/molmoact2, RoboDojo-Benchmark/RoboDojo
• 🎥 Blog Post: MolmoAct2

Intended Use

Use this checkpoint for dual ARX X5 inference in RoboDojo or for further fine-tuning. Dataset normalization metadata is stored in norm_stats.json. Pass norm_tag="robodojo" at inference time. The model predicts 25-step, 14D absolute joint-pose action chunks.

Continuous action prediction is the intended and recommended inference mode. Discrete action prediction is exposed for parity and debugging, but we use continuous actions by default.

Install

pip install torch transformers pillow numpy huggingface_hub

Sample Input

This sample comes from the RoboDojo training dataset RoboDojo_lerobot_v30_video, episode 0, frame 0. The camera order for this checkpoint is high, left wrist, right wrist.

Top RGB	Left RGB	Right RGB

from huggingface_hub import hf_hub_download
from PIL import Image
import numpy as np

repo_id = "hqfang/MolmoAct2-RoboDojo"

top_rgb = Image.open(
    hf_hub_download(repo_id, "assets/sample_top_rgb.png")
).convert("RGB")
left_rgb = Image.open(
    hf_hub_download(repo_id, "assets/sample_left_rgb.png")
).convert("RGB")
right_rgb = Image.open(
    hf_hub_download(repo_id, "assets/sample_right_rgb.png")
).convert("RGB")

task = "Arrange the numbers from left to right to form the largest possible number, and place them on the pad."
robot_state = np.array(
    [
        4.0714453293164143e-13,
        -5.980358068871964e-16,
        1.4396729160394704e-16,
        3.3858371238235303e-16,
        -6.58480106378867e-13,
        1.9262460550228955e-12,
        1.0,
        4.0714296083849133e-13,
        -5.980402008706103e-16,
        1.4397026945415223e-16,
        3.385876563795137e-16,
        -6.584741432669183e-13,
        1.9262503918315854e-12,
        1.0,
    ],
    dtype=np.float32,
)

Continuous Actions

import numpy as np
import torch
from huggingface_hub import hf_hub_download
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

repo_id = "hqfang/MolmoAct2-RoboDojo"

top_rgb = Image.open(
    hf_hub_download(repo_id, "assets/sample_top_rgb.png")
).convert("RGB")
left_rgb = Image.open(
    hf_hub_download(repo_id, "assets/sample_left_rgb.png")
).convert("RGB")
right_rgb = Image.open(
    hf_hub_download(repo_id, "assets/sample_right_rgb.png")
).convert("RGB")
task = "Arrange the numbers from left to right to form the largest possible number, and place them on the pad."
robot_state = np.array(
    [
        4.0714453293164143e-13,
        -5.980358068871964e-16,
        1.4396729160394704e-16,
        3.3858371238235303e-16,
        -6.58480106378867e-13,
        1.9262460550228955e-12,
        1.0,
        4.0714296083849133e-13,
        -5.980402008706103e-16,
        1.4397026945415223e-16,
        3.385876563795137e-16,
        -6.584741432669183e-13,
        1.9262503918315854e-12,
        1.0,
    ],
    dtype=np.float32,
)

processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype=torch.float32,
).to("cuda").eval()

out = model.predict_action(
    processor=processor,
    images=[top_rgb, left_rgb, right_rgb],
    task=task,
    state=robot_state,
    norm_tag="robodojo",
    inference_action_mode="continuous",
    enable_depth_reasoning=False,
    num_steps=10,
    normalize_language=True,
    enable_cuda_graph=True,
)

actions = out.actions

MolmoAct2 was trained with mixed precision. For the RoboDojo evaluation, we ran inference in bfloat16. The float32 path uses the most GPU memory: roughly 26GB with CUDA graph enabled, or around 24GB without CUDA graph.

If you have a GPU with less memory, you can run inference with bfloat16 instead:

model = AutoModelForImageTextToText.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    out = model.predict_action(...)

Using bfloat16 is much more memory efficient and can run under 16GB of GPU memory in our tests. It usually does not hurt performance much.

images should preserve camera order: [top_rgb, left_rgb, right_rgb], corresponding to high, left-wrist, and right-wrist cameras. Images may be PIL images or RGB arrays. state is the raw 14D robot state, and absolute 14D actions are returned in robot scale.

normalize_language=True is the default. It lowercases the task string and removes trailing sentence punctuation to match training preprocessing. Set it to False if you need to preserve the task text exactly.

enable_cuda_graph=True is the default. The first few calls can be slow because the model warms up and captures CUDA graphs. run several random warm-up calls before measuring deployment latency. num_steps controls the continuous flow solver and defaults to the checkpoint config value, 10.

Depth reasoning is disabled for this checkpoint. Calling enable_depth_reasoning=True will raise an error.

Discrete Actions

Discrete action inference requires a caller-provided action tokenizer. It is not saved in this repository. Discrete mode decodes action tokens directly. the continuous action expert is not used.

action_tokenizer = AutoProcessor.from_pretrained(
    "allenai/MolmoAct2-FAST-Tokenizer",
    trust_remote_code=True,
)

out = model.predict_action(
    processor=processor,
    images=[top_rgb, left_rgb, right_rgb],
    task=task,
    state=robot_state,
    norm_tag="robodojo",
    inference_action_mode="discrete",
    action_tokenizer=action_tokenizer,
    enable_depth_reasoning=False,
)

Model and Hardware Safety

MolmoAct2 generate robot actions from visual observations and language instructions, but their behavior may vary across embodiments, environments, and hardware configurations. Users should carefully validate model outputs before deployment, especially when operating physical robots or other actuated systems. Where possible, actions should be monitored through interpretable intermediate outputs (adaptive depth map), simulation rollouts, action limits, or other safety checks before execution on hardware. The model’s action space should be bounded by the training data, robot controller limits, and task-specific safety constraints, including limits on speed, workspace, torque, and contact force. Users should follow the hardware manufacturer’s safety guidelines, use appropriate emergency-stop mechanisms, and operate the system only in a safely configured environment with human supervision.

Citation

@misc{fang2026molmoact2actionreasoningmodels,
      title={MolmoAct2: Action Reasoning Models for Real-world Deployment}, 
      author={Haoquan Fang and Jiafei Duan and Donovan Clay and Sam Wang and Shuo Liu and Weikai Huang and Xiang Fan and Wei-Chuan Tsai and Shirui Chen and Yi Ru Wang and Shanli Xing and Jaemin Cho and Jae Sung Park and Ainaz Eftekhar and Peter Sushko and Karen Farley and Angad Wadhwa and Cole Harrison and Winson Han and Ying-Chun Lee and Eli VanderBilt and Rose Hendrix and Suveen Ellawela and Lucas Ngoo and Joyce Chai and Zhongzheng Ren and Ali Farhadi and Dieter Fox and Ranjay Krishna},
      year={2026},
      eprint={2605.02881},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2605.02881}, 
}