sjmathy/final_ek100

EK100 Action Anticipation Package

This package contains the pieces needed by a collaborator who already has the EK100 mean-pooled context features and context Pi3 trajectories.

Included:

• V-JEPA 2.1 full-token context features for train, val, and test.
• Pi3 future-action trajectories for train, val, and test.
• Train/validation/test action-anticipation manifests.
• Full-token extraction shard manifests for reconstructing shard membership.

Not included:

• Mean-pooled context features.
• Context Pi3 trajectories from [t-2, t-1].
• Raw videos.

Layout After Extraction

The paths inside the tar files are relative to the original EK100 extracted root:

features/train/full_tokens/*.pt
features/val/full_tokens/*.pt
features/test/full_tokens/*.pt

pi3_future_action/atomic_train/pi3_trajectory_controls_atomic/m16/*.pt
pi3_future_action/atomic_dev/pi3_trajectory_controls_atomic/m16/*.pt
pi3_future_action/test/pi3_trajectory_controls_atomic/m16/*.pt

planning/atomic_train/manifests/segment_manifest.csv
planning/atomic_dev/manifests/segment_manifest.csv
planning/test/manifests/segment_manifest.csv

manifests/fulltoken_shards/{train,val,test}/vjepa_shard_*.csv

Timing Convention

For each EK100 segment i, the action anticipation context is the clip before the action:

• context interval: [t_i - 2s, t_i - 1s]
• anticipation gap: 1s
• target action interval: [action_start_sec, action_end_sec]

The full-token V-JEPA features are extracted from the same context interval used for the mean-pooled cache. The future Pi3 trajectories are sampled from the target action segment, using the same 16-knot center/owned-interval style used by the current predictor experiments.

Tensor Formats

Full-token feature .pt files are dictionaries. The main tensor is:

tokens: (4608, 1664)

Future Pi3 trajectory .pt files are dictionaries. The main trajectory tensor is:

u_middle_oracle: (16, 6)

The trajectory files also store query timestamps, requested frame indices, gravity anchor metadata, video metadata, and Pi3 provenance.

Counts

train manifest rows: 66,178
val manifest rows:    9,595
test manifest rows:  13,061

full-token train files: 66,178
full-token val files:    9,587
full-token test files:  13,061

future Pi3 train files: 66,178
future Pi3 val files:    9,593
future Pi3 test files:  13,061

Known missing validation files:

• full tokens: 8 validation segments
• future Pi3: P30_08_154, P30_08_29

The missing future Pi3 entries are from corrupt/decode-failing frames in P30_08.MP4.

Current Local Predictor Results

These are local EK100 validation mean-class Recall@5 numbers, matching the benchmark metric style. The trajectory-conditioned row uses context Pi3 plus oracle future-action Pi3, so it is an oracle-future-trajectory validation setting.

No-traj predictor:          Verb 17.74 / Noun 17.51 / Action 5.37 / VN avg 17.63
Oracle future Pi3 predictor Verb 19.57 / Noun 19.26 / Action 6.27 / VN avg 19.42

Both predictor rows initialize from the atomic causal text bridge checkpoint and use its original atomic CamFormer trajectory encoder. The separately trained EK100 CamFormer alignment diagnostic checkpoint is not used in those two numbers.