terryGyx/irp-latent-action

IRP artefacts: latent action representations for human activity video

Fine-tuned latent action models and supporting features from the MSc Independent Research Project Latent Action Representations for Human Activity Video: Diagnosing and Improving Cross-Scene Semantic Consistency (Imperial College London, 2026).

Code, results and the written report: https://github.com/ese-ada-lovelace-2025/irp-yg25

Every number in the report traces to a file in that repository's code/results/. This repository holds the weights and features behind those numbers, which are too large for version control and cannot be re-downloaded from their original sources because we produced them ourselves.

Contents

exp5/ — SACA fine-tuned DiLA (7 variants, 421 MB each)

Structure-aware contrastive alignment. The loss adds three terms to DiLA's phase-2 objective:

L = L_reconstruct + lam_sym * L_sym + lam_inv * L_inv + lam_margin * L_margin

Trainable parts are DiLA's inverse dynamics model (DeepConvIDM) and the projection head. Everything else stays frozen.

Directory	Variant
full_inv_high	The selected model. lam_sym 0.7, lam_inv 0.7, lam_margin 7.0
full	The smoke-calibrated weights, lam_inv 0.35
full_sym_low	Weight-sensitivity grid point
full_margin_high	Weight-sensitivity grid point
sym_only	Ablation, symmetry term only
inv_only	Ablation, invariance term only
sym_inv	Ablation, no margin term

full_inv_high is the model reported throughout the paper. It gives the highest class separability and the highest probing accuracy of anything in the project, while keeping symmetric-pair structure far better than inv_only.

exp4/ — Seq-Delta-REPA fine-tuned DiLA (3 lambdas, 420 MB each)

Label-free alignment to V-JEPA 2 features, L = L_reconstruct + lambda * L_align. Directories seq_repa_lam0.01, seq_repa_lam0.1, seq_repa_lam1.0. lambda = 1.0 is the one discussed in the report. This branch raises probing accuracy but damages the geometry of the latent space, which is what motivated the SACA work above.

latents_bridge_steps/ — per-step Bridge latents (1.3 GB per tag)

baseline and full_inv_high, one 256-d latent per BridgeV2 transition. These are the pseudo-action labels for stage 1 of the SimplerEnv experiment. Regenerating them costs about 7.6 GPU-hours and requires the exp5 weights above, so they are kept rather than recomputed.

Loading

The exp4 and exp5 checkpoints are plain torch.save dictionaries, not HuggingFace models:

import torch
ckpt = torch.load("exp5/full_inv_high/ckpt_final.pt", map_location="cpu")
print(ckpt.keys())

They rebuild against DiLA's architecture. See code/src/models/dila.py and code/scripts/exp5_finetune.py in the GitHub repository.

Licence

This repository is released under the MIT licence. Everything here is a derivative work, and the upstream terms are as follows.

Upstream	Used for	Licence
DiLA	Base model for exp4 and exp5	MIT
DINOv2	Visual encoder inside DiLA	Apache-2.0
V-JEPA 2 ViT-L	Alignment targets for exp4	MIT
BridgeV2 (LeRobot port)	Source video for latents_bridge_steps	Apache-2.0

All four permit redistribution of derivative works with attribution, which this section provides.

On Something-Something V2

The exp4 and exp5 weights were fine-tuned on Something-Something V2, which is distributed under a restrictive research licence that does not permit redistribution of the data. Model weights are a transformation rather than a copy of the training data, so they are published here.

The precomputed per-clip V-JEPA 2 feature tables over SSv2 are deliberately not included. A per-video feature table is closer in kind to a derived dataset than to a model, so it is held privately instead. It can be regenerated from the public V-JEPA 2 checkpoint once you have your own licensed copy of SSv2, using code/scripts/extract_tau.py in the GitHub repository.

Not here

The Experiment 6 vision-language-action policies (5 checkpoints, 41.5 GB) are not retained. They are largely Qwen2.5-VL-3B base weights, they require the full StarVLA and SimplerEnv stack to use, and their scientific content is recorded in code/results/exp6_simpler.json, which carries the protocol, per-task and per-repeat success rates, stage-1 losses and latent diagnostics.