mangocloud/robotwin-cover-vla

RoboTwin CoVer-VLA verifier

A three-member CoVer-VLA trajectory verifier trained on the ten training tasks of the RoboTwin select split. It scores candidate robot actions conditioned on the current head-camera image and the original language instruction. Higher scores indicate stronger image/instruction/action alignment; scores are not calibrated success probabilities.

This repository contains verifier weights, not an action-generating robot policy. The verifier can rerank outputs from another policy when its actions follow the contract below. It can be loaded without installing GR00T or running a rephrase API.

Files

• cover_verifier_robotwin.pt: merged trainable components for three ensemble members.
• verifier.json: portable manifest with a relative checkpoint path and action contract.
• requirements.txt: pinned standalone inference dependencies, including upstream CoVer.
• score.py: standalone candidate-scoring CLI.
• training_config.json, task_split.json: training settings and task membership.
• SHA256SUMS, LICENSE: file integrity checks and license.

The frozen SigLIP2 ViT-L/16 384 backbone is loaded separately from timm/ViT-L-16-SigLIP2-384. Its pretrained weights and tokenizer are downloaded on first use; the 302 MiB checkpoint here contains the trained verifier components. The pinned loader also resolves Hub metadata, so the quick-start below assumes access to Hugging Face.

Download and install

Use Python 3.12 and a compatible PyTorch/CUDA environment. The CLI also supports --device cpu, which is useful for testing but slower than CUDA.

hf download mangocloud/robotwin-cover-vla --local-dir ./robotwin-cover-vla
python -m pip install -r ./robotwin-cover-vla/requirements.txt

The hf command is provided by huggingface_hub. Alternatively, download with huggingface_hub.snapshot_download("mangocloud/robotwin-cover-vla").

Input contract

• Image: the current cam_high head-camera frame, RGB uint8 [height,width,3]. Training images were 240 x 320. The backbone applies its own image preprocessing.
• Text: the original task instruction in English.
• Trajectories: float32 [K,20,14], with one complete trajectory per candidate. The first 12 steps are previously executed actions; the final 8 steps are the proposed actions. Every candidate must share the same execution history.
• Units: absolute RoboTwin joint/gripper targets in environment units, after policy decoding. Do not convert them to end-effector deltas or apply policy normalization to the verifier input.
• History padding: left-pad unavailable history with -5.0 in all 14 dimensions. Proposed actions must contain actual decoded targets, not padding.

The action order is:

Slice	Group	Meaning
0:6	left_arm	Six absolute left-arm joint targets
6:7	left_gripper	Absolute left gripper target
7:13	right_arm	Six absolute right-arm joint targets
13:14	right_gripper	Absolute right gripper target

The original policy predicts 16 steps. Score its first eight proposed steps and execute those eight before replanning. The verifier's 20-step input is 12 past + 8 future, not 20 future actions.

For example, to construct candidates.npy:

import numpy as np

# past: [N,14] executed absolute targets, in chronological order (N can be zero).
# future: [K,16,14] decoded candidate action chunks from your policy.
past = np.asarray(past, dtype=np.float32)[-12:]
history = np.full((12, 14), -5.0, dtype=np.float32)
if len(past):
    history[-len(past):] = past
shared_history = np.broadcast_to(history, (len(future), 12, 14))
trajectories = np.concatenate([shared_history, future[:, :8]], axis=1).astype(np.float32)
np.save("candidates.npy", trajectories)

Score candidates

python ./robotwin-cover-vla/score.py \
  --image head_rgb.png \
  --instruction "Click the top center of the bell." \
  --candidates candidates.npy \
  --device cuda:0

The final JSON prints one score per candidate and best_index. The upstream loader also prints initialization messages. For Python integration:

import numpy as np
from PIL import Image
from bridge_verifier.ensemble_eval import EfficientEnsembleMerged

verifier = EfficientEnsembleMerged(
    "robotwin-cover-vla/cover_verifier_robotwin.pt", device="cuda:0"
)
image = np.asarray(Image.open("head_rgb.png").convert("RGB"))
trajectories = np.load("candidates.npy", allow_pickle=False).astype(np.float32)
best_trajectory, scores = verifier.predict(
    image, "Click the top center of the bell.", trajectories
)
best_index = max(range(len(trajectories)), key=lambda i: scores[str(i)])

When reproducing hierarchical CoVer selection, group candidates by rephrased instruction, select the instruction group with the highest mean score, then its highest-scoring candidate. Always score against the original instruction. The standalone CLI returns a flat best index; grouping is the caller's responsibility.

Training provenance and scope

• Ten select.train tasks, 50 demonstration episodes each: 450 train / 50 validation.
• Episode split by task dataset, before language augmentation; validation text is original.
• GPT-6.1 Sol rephrases: 128-candidate pools, Qwen3-Embedding-0.6B/k-means selection of 16.
• Native 14D action targets; stride 1; 12 history + 8 future steps.
• 1,293,472 augmented training windows and 8,947 validation windows.
• Three ensemble members, seeds 42, 43, 44, 2,000 steps each.
• Four GPUs; per-GPU batch 4,096, global contrastive batch 16,384; LR 1e-6.
• Frozen SigLIP2 backbone and released CoVer Transformer verifier architecture.

This is a RoboTwin adaptation of the released CoVer method. The 12+8 window and global batch 16,384 differ from the paper's 6+4 window and 32,768 global batch. No simulator success-rate claim is made by this model card. Finite synthetic scores only verify loading and the inference interface. Other embodiments, action units, camera views, and execution horizons require separate validation or retraining.

Implementation: CoVer-VLA, pinned commit 21a9960639a8340c1e8511fb04776fb11271f246, plus the RoboTwin data adapter used for this training run. The existing augmented_train.* files are training provenance; they are not required for inference or for downloading the backbone.