justintiensmith/Spa-Bench-VLA0-Epoch12

VLA-0 — Spa-Bench Epoch 12

This is the VLA-0 checkpoint evaluated as a partial in-distribution baseline in Spa-Bench, a real-robot benchmark of spatially grounded reasoning in vision-language-action policies. It controls a single-arm SO-101 from two RGB views, a six-dimensional joint state, and a natural-language instruction.

The checkpoint is published under Justin Tien-Smith's account so the thesis artifact has a stable, maintained model endpoint. The model files and Git history were copied server-side from mattpidden/vla0-justin-epoch12 at immutable source revision 9c14bd78de26b66f16a66f98e64f17f00a6740fc. This copy preserves that provenance; it does not claim that the checkpoint was independently retrained after copying.

Model details

Field	Value
Base model	Qwen/Qwen2.5-VL-3B-Instruct
Evaluated checkpoint	End of epoch 12; 77,246 optimizer updates
Robot	SO-101 single-arm manipulator
Inputs	Fixed middle RGB, wrist RGB, six absolute joint positions, text instruction
Image preprocessing	224×224 policy inputs; random crop and colour jitter during training
Action representation	Six joint targets encoded as text with 1,000 bins
Action horizon	8
Adaptation	Full Qwen backbone; no LoRA or QLoRA
Optimizer	AdamW, learning rate 5e-6, weight decay 0.01, cosine annealing
Hardware and batch	Four NVIDIA GH200 GPUs; 24 samples per device, global batch 96
Training data	Motion-trimmed 1,200-episode Spa-Bench derivative

The archived config.yaml records a 16-epoch training plan. Spa-Bench selected and evaluated the model_12 checkpoint, following the study's fixed epoch-12 selection rule.

The thesis author confirms that the training dataset was justintiensmith/VLA_Reasoning_Training_Dataset_1200_Trimmed_Start_5_Frame. The original VLA-0 launcher and the dataset revision used by that run were not retained, so this card does not infer a revision from the current dataset head.

Evaluation

The checkpoint completed 0/72 familiar/in-distribution physical rollouts. It received at least ten trials from each of the six Spa-Bench task families and a 90-second per-rollout limit because autoregressive action generation was slower than the other evaluated policies. Evaluation was stopped before the out-of-distribution and diagnostic protocol, so this checkpoint must not be compared with the fully evaluated policies on the headline OOD benchmark.

• Rollouts: justintiensmith/Spa_Bench_Partial_VLA-0
• Thesis artifact: justintiensmith/Imperial-Thesis
• Rollout integration: justintiensmith/lerobot_rollout_vla0

The reported result applies only to this checkpoint, embodiment, camera setup, prompts, and physical protocol. It is not a general assessment of the VLA-0 method.

Intended use

This release supports reproducibility, inspection, and research on the Spa-Bench experiment. It is not a drop-in LeRobot policy: deployment uses the project's VLA-0 inference service plus the HTTP-backed rollout adapter in the linked LeRobot fork.

Robot policies can move hardware unexpectedly. Use conservative limits, a clear workspace, an accessible emergency stop, and direct human supervision. Do not use this checkpoint for unattended, safety-critical, medical, or industrial operation.

Limitations

• Physical evaluation is partial and contains no OOD trials.
• Training used one robot embodiment, one workspace, two selected camera views, and English task instructions.
• The motion-trimmed dataset is a policy-specific adaptation and is not controlled independently from model family.
• No held-out validation loss was used to select among checkpoints; epoch 12 was fixed before physical evaluation.
• The copied source repository did not include a model card, so this card draws configuration details from the archived files and the Spa-Bench report.

Files and loading

model_12.pth is the training checkpoint used by the VLA-0 service. The model_12/ directory contains the Qwen-format model, tokenizer, processor, and generation configuration. dataset_stats.pkl contains training-data scaling statistics. Retain these files together when reproducing deployment.

This repository does not relicense third-party model or software components. Use is subject to the Qwen base-model license and the licenses of the VLA-0 and rollout code used for deployment.

Citation

Please cite the completed Spa-Bench MSc report, the thesis artifact, and the VLA-0 method.

@article{goyal2025vla0,
  title   = {VLA-0: Building State-of-the-Art VLAs with Zero Modification},
  author  = {Goyal, Ankit and Hadfield, Hugo and Yang, Xianqiang and Blukis, Valts and Ramos, Fabio},
  journal = {arXiv preprint arXiv:2510.13054},
  year    = {2025}
}