MIT-Media-Lab/egotouch-tactile-encoder-p1-v1

EgoTouch tactile encoder P1 v1

Three completed seeds of the full-touch/full-pose Bash BiGRU P1 encoder trained on published clean EgoTouch atomic episodes. The model aligns an eight-frame tactile history with the synchronized eight-frame 21-joint hand-pose history using symmetric masked InfoNCE.

Source code and frozen configs: kaichen-z/VLA-HAND@31d75d2.

Method-D-aligned release (2026-09-08)

The mano-aligned-v1/ directory contains new H2, H4, and H8 checkpoints. These runs use EgoTouch revision 5131b279898b6db4eb7c131ee954e53c5f61b229. That revision repairs the left-hand rotation frame in the re-cut atomic dataset.

Unlike the original release below, the new pose target uses Method-D-compatible MANO-local keypoints:

R(global_orient_worldspace)^T @ (joints_worldspace - transl_worldspace)

Checkpoint	P2T mAP	T2P mAP	Mean mAP
H2	0.06310	0.05683	0.05996
H4	0.06904	0.06277	0.06590
H8	0.07307	0.07085	0.07196

Use H8 when retrieval is the selection criterion. These values are not directly comparable with the camera-space results below because the data release and pose target changed.

The source branch and full editor report are in hannah/tactile_editor. See mano-aligned-v1/README.md for exact files, checksums, and training settings.

Original camera-space v1 results

Values are three-seed mean +/- sample standard deviation on fixed 4,096-window galleries.

Split	Bidirectional retrieval mean mAP
validation	0.121405 +/- 0.001201
test seen	0.121559 +/- 0.000380
test unseen	0.048720 +/- 0.000976

Files

Each seed-* directory contains:

• training_checkpoint.pt: model, optimizer, scheduler, epoch, step, config, and selected validation metrics; use this to resume training.
• model.safetensors: complete FormalEncoderModel state dict.
• tactile_encoder.safetensors: standalone TactileEncoder state dict.
• pose_encoder.safetensors: standalone PoseEncoder state dict.
• exact resolved config, initialization report, runtime summary, and held-out metrics.

pressure_masks.npy contains the left/right 151-cell non-bend validity masks. export_manifest.json records every file size and SHA-256 checksum.

Input contract

• Tactile: [B, 8, 21, 21], target hand, pressure values as published.
• Pose: [B, 8, 21, 3], target-hand joints_camspace.
• Output: L2-normalized 64D tactile and pose embeddings.
• Canonical hand: right. Left tactile is flipped horizontally and left pose x is reflected before model input.
• Pose is wrist-centered and temporally hand-scale-normalized inside the pose encoder.
• Instruction and RGB are not model inputs.

Standalone tactile encoder

Clone the source repository and install Tactile/requirements.txt, then:

import numpy as np
import torch
import torch.nn.functional as F
from safetensors.torch import load_file
from egotouch_formal.model import TactileEncoder

encoder = TactileEncoder(embed_dim=64)
encoder.load_state_dict(load_file("seed-20260831/tactile_encoder.safetensors"))
encoder.eval()

pressure_masks = torch.from_numpy(np.load("pressure_masks.npy")).bool()
touch = torch.zeros(1, 8, 21, 21)  # already hand-canonicalized
side = torch.tensor([1])            # 0=left, 1=right; selects validity mask
with torch.no_grad():
    embedding = F.normalize(encoder(touch, pressure_masks[side]), dim=-1)
assert embedding.shape == (1, 64)

For the complete tactile-pose model, instantiate FormalEncoderModel with task="p1", pose_temporal="gru", and embed_dim=64, then load model.safetensors strictly.

Training protocol

Batch 1,024; AdamW at 1e-4 with weight decay .01; 500-step linear warmup and cosine decay; BF16; gradient norm clip 1.0; 40 epochs. The three runs completed 25,080 optimizer steps because the epoch limit was reached before the 30,000 step cap.

The initialization checkpoint is basharkMIT/opentouch-gru-pose-encoder/epoch_300.pt, pinned in the source downloader by SHA-256.

Data and limitations

The exact preprocessed cache is published separately as MIT-Media-Lab/egotouch-clean-atomic-p1-cache-v1. The test-unseen gap shows that cross-recording-population generalization remains substantially harder than validation/test-seen retrieval. This experiment does not demonstrate improved future-pose prediction; no future-pose head was trained.