TrAct

TrAct is a π0.5-based vision-language-action policy trained to predict 2D point tracks together with actions. At test time it can optionally rank candidate actions with a track-conditioned dual-view video world model and the VLAC critic. It is built on openpi. The Python package is tract.

This repository contains code only: no datasets, model checkpoints, or experiment results. The sections below explain where to get each one.

Repository layout

src/tract/                 policy, model, training code (fork of openpi's `openpi` package)
packages/openpi-client/    websocket client used by the eval scripts (unchanged from openpi)
scripts/                   training, data preprocessing, LIBERO suite builders, evaluation
  vlac_pipeline/           world-model + VLAC action selection
examples/                  openpi examples (LIBERO, ALOHA, DROID, ...)
third_party/
  libero/                  LIBERO with added suites (bddl + init files); assets downloaded separately
  LIBERO-Pro/              LIBERO-PRO perturbation bddl / init files
  track_dualview/          track-conditioned dual-view SVD world model
  Ctrl-World/              Ctrl-World world model (baseline)
  VLAC/                    VLAC critic
  co-tracker/              CoTracker3 (used to label tracks)
  aloha/                   git submodule (openpi ALOHA examples)

Installation

git clone --recurse-submodules <this repo> tract && cd tract
GIT_LFS_SKIP_SMUDGE=1 uv sync
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e .
uv pip install -e third_party/libero -e third_party/co-tracker

third_party/Ctrl-World, third_party/VLAC and third_party/track_dualview have their own requirements files. You only need them for world-model training or WM-based action selection.

The first time LIBERO is imported, it asks where its config lives. Point it at third_party/libero/libero/libero. The eval scripts set LIBERO_CONFIG_PATH=.libero_local automatically.

Data

No data is included. Download or generate the following, with all paths relative to the repo root:

What	Source	Where it goes
LIBERO demos (LeRobot)	physical-intelligence/libero, downloaded automatically by LeRobot	$HF_LEROBOT_HOME
LIBERO / LIBERO-plus scene assets	assets.zip from Sylvest/LIBERO-plus	unzip to third_party/libero/libero/libero/assets/
LIBERO + CoTracker track labels (libero_cotracker_densegrid_merged)	generated, see below	$HF_LEROBOT_HOME/physical-intelligence/

Generate the track-labelled training set (CoTracker3 checkpoint from facebook/cotracker3):

mkdir -p checkpoints/cotracker
wget -P checkpoints/cotracker https://huggingface.co/facebook/cotracker3/resolve/main/scaled_offline.pth
uv run scripts/lerobot_preprocess_cotracker.py \
  --source-repo-id physical-intelligence/libero \
  --target-repo-id physical-intelligence/libero_cotracker_densegrid_merged

Checkpoints

No weights are included. The π0.5 base and LIBERO weights are downloaded from gs://openpi-assets automatically. The world-model / critic components default to these paths:

Component	Source	Default path
Stable Video Diffusion	stabilityai/stable-video-diffusion-img2vid	checkpoints/svd
CLIP	openai/clip-vit-base-patch32	checkpoints/clip-vit-base-patch32
Ctrl-World	yjguo/Ctrl-World, or train with third_party/Ctrl-World/scripts/train_wm_libero.py	checkpoints/ctrl_world/
VLAC critic	InternRobotics/VLAC	checkpoints/VLAC-8b
Track dual-view world model	train with third_party/track_dualview/train_code/	checkpoints/track_dualview/
TrAct policy	train with the configs below	checkpoints/pi05_libero_stage2p5_joint/<exp>/<step>

The standalone scripts inside third_party/* keep their projects' own relative layout (e.g. third_party/Ctrl-World/checkpoints/svd).

Training

TrAct is trained in three stages, one train config each in src/tract/training/config.py:

uv run scripts/compute_norm_stats.py --config-name pi05_libero_stage1_align
XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py pi05_libero_stage1_align   --exp-name=<exp>
XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py pi05_libero_stage2_track   --exp-name=<exp>
XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py pi05_libero_stage2p5_joint --exp-name=<exp>

Evaluation (LIBERO and variants)

# policy only
uv run scripts/evaluate.py            --checkpoint-dir checkpoints/pi05_libero_stage2p5_joint/<exp>/<step>
# policy + track world model + VLAC action selection
uv run scripts/evaluate_full_track.py --checkpoint-dir checkpoints/pi05_libero_stage2p5_joint/<exp>/<step>

The extra suites (*_swap, *_object, *_lan, *_task, *_cross, *_sawyer, libero_viewpoint, libero_background, libero_integral, ...) are defined in third_party/libero and built by scripts/build_libero_*.py. Outputs go to results/, which is git-ignored.

License

Code from openpi is Apache-2.0 (LICENSE, LICENSE_GEMMA.txt for Gemma weights). Each project under third_party/ keeps its own license file.