wjstx/VLA-Touch-FRS-Tact

VLA-Touch on FRS_Tact: four task models

Four independently trained FRS robot policies: insert, press, bread, two_tubes. Training completed on 2026-11-09. Each model uses the original RDT-1B backbone with frozen SigLIP/T5 encoders, followed by a frozen-base tactile stochastic-interpolant controller. The deployment planner uses the DeepSeek API. This is an adaptation to the FRS robot and sensors; it does not reproduce the original Franka experimental success rates.

Training and offline results

Each task completed 20,000 RDT optimizer steps (batch 2), full feature-cache preparation, 100 controller epochs (batch 64), and held-out evaluation. Checkpoints were selected using validation loss. Splits are by episode and stratified within each dataset source, with train-only normalization statistics.

Task	State/action profile	Selected RDT step	Selected controller epoch	Translation MAE, base → refined (mm)
insert	single-right-arm-7x10	20000	95	0.566 → 0.431
press	single-right-arm-7x10	20000	75	0.463 → 0.415
bread	dual-arm-20x20	16000	100	0.397 → 0.362
two_tubes	dual-arm-20x20	18000	96	0.474 → 0.348

The controller improves translation and gripper MAE on all four tasks. Rotation6D MAE worsens by approximately 4.9% for press and 11.6% for bread; see each evaluation.json for the complete base/refined and tactile-ablation results. These are offline action errors; robot success rates have not yet been evaluated.

Source groups: insert_01–02, press_01–02, bread_01–03, two_tubes_01–04 under KaiyueChen/LeRobot v2.1. Insert/press use 7D right-arm state and 10D actions; bread/two_tubes use 20D bimanual state and actions. Actions are delta XYZ, rotation6D matrix columns, and absolute gripper width in meters, per arm. Data frequency is 30 Hz, controller horizon 32, and observation/execution stride 8. Tactile inputs are reference optical-flow displacement proxies, not calibrated force measurements.

Contents

• tasks/<task>/rdt/best/: complete trained RDT weights, architecture and FRS adapter metadata.
• tasks/<task>/controller/controller.pt and .json: selected EMA tactile controller and checksum contract.
• Per-task training/deployment YAML, audit, training metrics, and evaluation results.
• shared/octopi/: original pretrained ViFiCLIP, sensor adapters and property regressor used for deployment.
• source/: exact training/deployment source snapshot, including pinned upstream RDT/VLA-Touch code and licenses.
• dependencies.json: fixed official Hugging Face revisions for SigLIP, T5, DINOv2 and CLIP.
• SHA256SUMS and manifest.json: integrity records for this archive.

The original model files are byte-for-byte preserved. Their contracts include the training server's absolute paths. Run relocate.py after downloading to create deployment copies with your local paths; it preserves parameter tensors and updates contract hashes together. Optimizer/resume checkpoints and raw training data are kept on the training server and are not part of this deployment archive.

Download and prepare deployment

Use a Python 3.12 environment with the pinned dependencies in requirements.lock.txt. For a private repository, first authenticate locally with your Hugging Face credentials.

hf download wjstx/VLA-Touch-FRS-Tact --local-dir ./vla_touch_bundle
cd vla_touch_bundle
sha256sum -c SHA256SUMS

Download the unchanged foundation encoders from their official repositories, or reuse your existing pretrained directory containing the four subdirectories listed in dependencies.json:

import json
from pathlib import Path
from huggingface_hub import snapshot_download
bundle = Path("vla_touch_bundle").resolve()  # adjust to the downloaded directory
pretrained = Path("vla_touch_pretrained").resolve()
for name, spec in json.loads((bundle / "dependencies.json").read_text())["models"].items():
    snapshot_download(**spec, local_dir=pretrained / name,
                      allow_patterns=["*.json", "*.model", "*.bin", "*.safetensors"])

The full official T5-XXL file alone is approximately 44.5 GB. Existing verified encoders on the training machine are located at /DATA/ljl/substage/vla_touch/pretrained.

python ./vla_touch_bundle/relocate.py \
  --bundle ./vla_touch_bundle \
  --pretrained-root ./vla_touch_pretrained \
  --output ./vla_touch_runtime
cd ./vla_touch_bundle/source
python -m deploy_vla_touch.remote_client \
  --config /absolute/path/to/vla_touch_runtime/insert/deploy.yaml --check

Configure the robot bridge address/token and your DeepSeek API key locally before actual robot use. The --check command validates the artifact/configuration and does not connect to the robot. Use the supplied deployment README for observation-only checks and robot integration.

Provenance

• FRS_Tact eric base: https://github.com/chaoyiL/FRS_Tact/tree/eric (a4926aa).
• VLA-Touch: https://github.com/jxbi1010/VLA-Touch (c9f970366387fc23f1d64a48a7827d3263410d89).
• RDT: https://github.com/thu-ml/RoboticsDiffusionTransformer (cd79363a1387e8f81c7724d070ef7e45fd23150f).
• Full source hashes and installed dependency versions: training_provenance.json.

Upstream source licenses remain in their respective source directories. Foundation encoders retain their original repository licenses and usage terms.