vla_ma

A FineART-VLA LoRA adapter. The included loader restores the adapter, its processors and tokenizers on the pinned base model.

Install

Use Python 3.12 and a CUDA-compatible PyTorch installation. Install LeRobot with FineART-VLA and PEFT support; the loader was checked with LeRobot commit 72e1d9e5e76ee719bacfc78bc34d817bee5fb521, Transformers 5.5.4 and PEFT 0.21.2.

pip install "lerobot[fineart_vla,peft] @ git+https://github.com/huggingface/lerobot.git@72e1d9e5e76ee719bacfc78bc34d817bee5fb521" "transformers==5.5.4" "peft==0.21.2"

Authenticate with your usual Hugging Face login, or set HF_TOKEN_PATH to your credential file if authentication is required.

Load

import sys
from huggingface_hub import snapshot_download

root = snapshot_download("pmolodyk/vla_ma")
sys.path.insert(0, root)
from load_policy import load_policy

policy, preprocessor, postprocessor = load_policy(root, device="cuda")

The loader downloads lerobot/FineART-VLA-subtask-midtrained at revision b6cf4faf2630f971dd390bc32365e9d9c330da8e. To use an existing copy of that exact base checkpoint, pass base_path="/path/to/base" to load_policy.

Predict actions

Provide a 14-element state vector and three RGB images under these keys:

• observation.state: float32 tensor, shape [14].
• observation.images.cam_high: float32 RGB tensor, shape [3, H, W], range [0, 1].
• observation.images.cam_left_wrist: same image format.
• observation.images.cam_right_wrist: same image format.

Use the joint order in config.json. Keep the provided processors to apply the required normalization and image processing. The following code assumes obs contains those four observation tensors:

import torch
from lerobot.utils.constants import QUERY_KIND, QUERY_TEXT

task = "Place the object in the container."

with torch.inference_mode():
    planner = {**obs, "task": task, QUERY_KIND: "next_subtask", QUERY_TEXT: task}
    subtask = policy.select_message(
        preprocessor(planner), max_new_tokens=48, suppress_loc_tokens=True
    ).strip()
    batch = {
        **obs,
        "task": task,
        "messages_rendered": [{"role": "user", "content": subtask}],
        "message_streams": ["low_level"],
        "target_message_indices": [],
    }
    actions = postprocessor(policy.predict_action_chunk(preprocessor(batch)))
    # actions: [1, 50, 14] absolute action targets

For a supplied subtask, skip the planner call and set subtask to your instruction. The Gemma terms inherited from the base model apply; see LICENSE and NOTICE.