wkrfugs/lingbot-va-piper-4task-step30000

LingBot-VA Piper teacher — four tasks, step 30,000

A LingBot-VA base model fine-tuned on four Piper tasks for 30,000 optimizer steps. This is the multi-step teacher, trained with video and action flow matching. It is not a Flash-WAM or DriftWAM distilled student and is not an OpenPI policy checkpoint. The transformer is published in its original FP32 precision, without conversion.

Contents

• transformer/: model configuration and original FP32 weights (20,355,577,064 bytes for the weight file; approximately 18.96 GiB).
• norm_stats.json, piper_contract.json: the exact training normalization and input/action contract.
• split.json, data_identity.json: the episode split and data fingerprints.
• training_summary.json, train_action_l1_definition.json: training settings and metric definition.
• validation_metrics.json: offline validation history.
• base_components.json, provenance.json: pinned base-component and runtime versions.
• runtime/piper_inference.patch: local inference/model changes against the pinned official LingBot-VA commit.
• piper_adapter/: the data helpers, preprocessing and offline evaluator used for this model, plus a small preencoded-input inference wrapper.

This inference release does not contain optimizer, scheduler or RNG states; it is not an exact-resume training checkpoint. Those are retained separately.

Download

hf download wkrfugs/lingbot-va-piper-4task-step30000 --local-dir ./piper_teacher

For reproducible experiments, pass --revision with the desired repository commit.

The frozen VAE, text encoder and tokenizer are unchanged from the base model. Download only those components (do not overwrite this repository's transformer):

hf download robbyant/lingbot-va-base \
  --revision 68b7bc1b35da6ddc67ea94c4ceb58d768fbb3f9c \
  --include 'vae/*' 'text_encoder/*' 'tokenizer/*' \
  --local-dir ./piper_teacher

The base components are only needed to encode new RGB observations and text; preencoded inference uses the fine-tuned transformer alone.

Runtime

Use the official runtime at this commit and apply the included patch once:

git clone https://github.com/Robbyant/lingbot-va.git
cd lingbot-va
git checkout 7c6ffa9bfc4b83582cafc860fab4c82cc7deeeeb
git apply ../piper_teacher/runtime/piper_inference.patch
pip install -r requirements.txt
cd ..

The tested environment uses PyTorch 2.9.0, Diffusers 0.36.0 and Transformers 4.55.2; see the runtime requirements and the upstream installation guide for CUDA/attention dependencies. Use the repository's WanTransformer3DModel, not AutoModel: the transformer configuration alone is not a generic Transformers robotics pipeline.

The patch preserves the locally used model implementation and adds the preencoded video/action rollout path and configurable action guidance. provenance.json contains the resulting model/server file hashes. The accompanying data scripts are offline preprocessing/evaluation utilities, not a robot client or controller.

Input contract

• Ordered cameras: head, hand (observation.images.head, observation.images.hand).
• RGB images: aspect-preserving PIL bilinear resize and centered black padding to 224 × 224 per camera. Do not stretch or center-crop.
• Source actions: 30 Hz; video samples: every third source frame (10 FPS).
• Encode each camera separately with the base causal VAE, using the posterior mean and the VAE's latent mean/std normalization. Concatenate head then hand along latent width, yielding [48, F, 14, 28]; do not concatenate the RGB images before VAE encoding.
• Streaming VAE processing: first image separately, then groups of four images. Reset the VAE cache for each camera/episode. See piper_adapter/prepare_latents.py for the exact encoding and text embedding procedure.
• Use the original full task instruction. Text and empty-prompt embeddings have shape [512, 4096], with padded token embeddings zeroed.

For an observed single frame, the normalized two-camera latent is [48, 1, 14, 28]. At nonzero positions in offline validation, cached latents include causal history; this is distinct from encoding every incoming frame with a freshly reset VAE.

Action contract — essential for integration

The physical output is seven values per timestep:

[joint_1, joint_2, joint_3, joint_4, joint_5, joint_6, gripper_opening]
[rad,     rad,     rad,     rad,     rad,     rad,     m]

These are absolute joint targets and gripper opening, not Cartesian positions, velocities or relative action deltas. The seven values occupy zero-based model channels [14, 15, 16, 17, 18, 19, 28] of a 30-channel tensor. The slot names come from the pretrained model layout; this dataset contains one Piper arm. Other channels are masked from supervision and must be ignored, not interpreted as trained robot commands or assumed to be meaningful zero predictions.

For each valid channel, training normalization is:

normalized = 2 * (physical - q01) / (q99 - q01 + 1e-6) - 1
physical = (normalized + 1) / 2 * (q99 - q01 + 1e-6) + q01

The quantiles in norm_stats.json were fitted on training episodes only. Labels were not clipped to [-1, 1]. These empirical quantiles are not robot limits. Use the provided seven-dimensional inverse normalization, not a RoboTwin or OpenPI action transform.

Each future video latent corresponds to 12 actions. The offline forecast uses four future latents, giving 48 valid actions (1.6 seconds at 30 Hz). The observed initial latent's dummy action block is discarded. The validated server path generates enough chunks and truncates surplus predictions to 48 actions.

Offline inference entry point

Create observation.pt containing tensors with these keys:

init_latent:    [48, 1, 14, 28]  # normalized, head then hand
text_emb:      [512, 4096]
empty_text_emb:[512, 4096]

Then run one GPU:

torchrun --standalone --nproc_per_node=1 \
  piper_teacher/piper_adapter/infer_preencoded.py \
  --model-root ./piper_teacher \
  --lingbot-root ./lingbot-va \
  --input ./observation.pt --output ./predicted_actions.npz --seed 42

The output actions array has shape [48, 7] in radians/meters; normalized_actions contains the corresponding normalized values. The wrapper uses the same settings as offline validation: 25 video / 50 action denoising steps, video/action guidance 5/1, SNR shifts 5/1, attention window 30, BF16 inference. This is a fresh-context forecast with generated future video. It does not implement camera acquisition, execution cadence, feedback or hardware control. Those are the collaborator's integration responsibilities.

The underlying sampling path has run throughout validation. The newly packaged wrapper has had syntax/interface checks; a fresh end-to-end GPU run of this release package has not been performed.

Training

• Initialization: robbyant/lingbot-va-base, revision in provenance.json.
• Dataset: Piper four-task dataset.
• 125 episodes: 113 train / 12 validation, three complete validation episodes per task, split seed 42. Raw demonstrations are not republished here.
• Effective batch 32 (four A800 GPUs, per-GPU batch 8, accumulation 1).
• 30,000 steps, BF16 compute with FP32 saved parameters.
• Video FM and action FM weights 1 / 1; SNR shifts 5 / 1.
• AdamW: peak LR 2.5e-5, warmup 1,000 steps, cosine decay to 2.5e-6, betas (0.9, 0.95), weight decay 1e-10, epsilon 1e-8, gradient clipping 1.
• Five-latent training windows, stride four, training chunk size four.
• Action L1 is a detached diagnostic, never an optimization loss. It is training-noise denoising reconstruction error, not full sampled-rollout error.

Final 100-update means:

Metric	Value
Video FM loss	0.00614567
Action FM loss	0.00033954
Training action L1	0.00373420

Offline validation

12 held-out episodes × early/middle/late = 36 equally weighted conditions. Each condition forecasts 48 actions from an observed latent and task text; future video is generated, not supplied from the reference episode. Joint MAE is averaged over six joints and timesteps; gripper MAE uses its own physical unit. Normalized L1 averages the seven normalized channels.

Scope (step 30,000)	Normalized L1	Joint MAE (degrees)	Gripper MAE (mm)
Full 48-action forecast	0.089670	4.27345	3.83985
First action	0.110426	5.57320	3.45282

These are offline action errors, not real-robot success rates. Training L1 and sampled validation L1 use different prediction procedures and are not directly interchangeable. Closed-loop robot performance has not been established for this checkpoint. The model is published for collaborator integration/evaluation.

Attribution

Based on LingBot-VA and its Apache-2.0 base weights. The upstream runtime license is included in runtime/LICENSE.txt. Piper data and robot setup are from the collaborator's OpenPI lab dataset.