luuuulinnnn/egohoi-control-hot3d-h2o

EgoHOI Control — HOT3D + H2O (Wan 2.1 I2V-14B)

Conditional egocentric world-model training on HOT3D + H2O (interleaved 19:1), built on a frozen Wan 2.1 I2V-14B-720P DiT. Code: edisiondyli/egohoi_extension @ branch obj_rope (commit 563354c).

Both training runs completed the full 13000 steps. Both were then evaluated against a frozen-Wan baseline on the same validation grid.

Results

Validation loss (training objective)

run	camera	hand	obj	view_render_root	val loss	train cum-avg
train_camera_obj	✅	❌	✅	hand_view_render (new)	0.04912	0.12306
train_camera_egohoihand_obj	✅	✅	✅	view_render_old	0.05026	0.12363

Validation uses fixed timesteps [357, 682, 833, 921, 978] with deterministic noise, so the two numbers are directly comparable.

Generation quality — PSNR / SSIM vs ground truth

All videos generated at 40 sampling steps, unipc, guide_scale 5.0, seed 23, then compared to the GT clip over the same frame interval at 480×480.

run	PSNR (dB)	SSIM	n	HOT3D PSNR / SSIM	H2O PSNR / SSIM
train_camera_obj (exp1)	20.460	0.6571	139	20.779 / 0.6600	19.794 / 0.6509
train_camera_egohoihand_obj (exp2)	18.708	0.6370	142	19.376 / 0.6501	17.400 / 0.6114
Wan 2.1 baseline (frozen, no control)	12.230	0.3686	142	12.287 / 0.3557	12.116 / 0.3946

Both conditioned models beat the frozen baseline by a wide margin (+6.5 to +8.2 dB PSNR, +0.27 to +0.29 SSIM), confirming the camera/hand/object conditioning is doing real work rather than the base model carrying the result.

Caveat on exp1 vs exp2. These two runs differ in two variables at once — the camera render set (hand_view_render vs view_render_old) and whether the egohoi hand encoder is enabled. exp1's advantage therefore cannot be attributed to the render set alone; a single-variable ablation would be needed to separate the two effects.

Baseline caveat. The frozen Wan baseline is image-to-video from the first GT frame with a text prompt only — it receives no camera, hand, or object conditioning, and nothing anchors it to the GT trajectory. Its low PSNR/SSIM reflects that missing conditioning, not a broken model. It was run at Wan's 480p setting (832*480) to match the fine-tuned runs' resolution.

Repository layout

<run_name>/                         # train_camera_obj, train_camera_egohoihand_obj
  checkpoints/                      # final.pt (step 13000), best.pt, best_average.pt
  train.log, slurm-*.out, resolved_config.yaml, submit.sbatch
  tensorboard/                      # training curves
  wandb/                            # offline W&B run

infer_<run>/                        # infer_camera_obj, infer_camera_egohoihand_obj,
  hot3d/<clip>/<clip>_<span>.mp4    # infer_wan21_baseline
  h2o/<clip>/<clip>_<span>.mp4      # all generated videos
  inference_summaries.jsonl         # GT <-> generated pairing, caption, seed, checkpoint
  slurm-*.out, submit.sbatch

metrics/
  psnr_ssim_summary.json            # aggregate, overall + per dataset
  <run>_per_video.csv               # per-video PSNR/SSIM
  eval_psnr_ssim.py                 # the driver used to produce them

View curves for both training runs at once:

tensorboard --logdir .

Training setup

Identical across both runs except the three enable flags and view_render_root.

Base model	Wan 2.1 I2V-14B-720P (frozen)
Steps	13000
Optimizer	AdamW, lr 1e-5 (decayed to 0), weight_decay 0
Precision	bf16 + gradient checkpointing
Datasets	HOT3D + H2O interleaved — 19 HOT3D steps : 1 H2O step
Batch size	1 per GPU
Video	81 frames @ 480×480
HOT3D start frames	[0, 10, 20, 30, 40, 50, 60, 69]
Validation	every 500 steps — HOT3D 100 explicit val clips + H2O subject3_ego
Checkpointing	every 1000 steps
obj rope	tss-rope (temporal −1, shifted spatial grid)
Object stages	A 1500 / B 1500 / C remainder
Seed	23
Hardware	8× NVIDIA H200, 2 CPUs per GPU

Conditioning modules

module	configuration
camera_encoder	ControlNet-style residuals into the first 20 Wan blocks; Plücker rays (256 hidden, 6 ch) + masks; control_dim 1024, 20 layers, 16 heads, ffn 4096, rms_norm
hand_encoder	Wan LoRA r128 / α128 on q,k,v,o,ffn.0,ffn.2 (kaiming); Conv3d hand stack (concat_dim 4) + reference hand-frame injection
obj_encoder	Stand-In image-branch LoRA r128 / α128 on self_attn.{q,k,v} (zero init), all attention blocks

Data

Preprocessed HOT3D (1516 clips) + H2O (4 subjects) from EndeavourDD/Jichen (outputs/).

• HOT3D train: clips ≥ 1849 excluding the validation grid
• HOT3D val: 100 explicit clip ids sampled with random.seed(23) over [1849, 3364]
• H2O train: subject1_ego, subject2_ego · H2O val: subject3_ego
• Captions: hot3d_qwen_captions.json, h2o_action_label_captions.json

Reproduction notes

Fixes required to run this outside the author's machine:

1. h2o_data.raw_root — the H2O camera intrinsics/poses (cam_intrinsics.txt, cam_pose/) ship inside outputs/h2o_preprocessed/<subject>/<seq>/cam4/, not at the configured data/h2o. Set --set h2o_data.raw_root=outputs/h2o_preprocessed.
2. H2O object_render_metadata.json stores absolute image_path values (/home/jc/jc/outputs/...), unlike HOT3D which uses repo-relative paths. They must be rewritten to outputs/h2o_preprocessed/... or every H2O object window is silently dropped.
3. scripts/eval.py is empty (0 bytes). The working metric tool is scripts/compute_video_psnr_ssim.py, which compares one pair at a time; metrics/eval_psnr_ssim.py here drives it over a whole run.
4. scripts/wan21_inference.py hardcodes CUDA_VISIBLE_DEVICES=7, /home/jc/jc/... paths, and 720p. It also writes no summary JSON, so the metric driver falls back to deriving GT pairs from the output filenames.
5. CUDA_HOME must point at a real CUDA toolkit (module load CUDA/12.6.0), otherwise transformers → deepspeed raises MissingCUDAException on the first generated item.

Notes

• Checkpoints are full training checkpoints (model + optimizer state), not merged weights. final.pt (step 13000) is the one to use; best.pt / best_average.pt track noisy single-step and running-average training loss.
• Both training runs exceeded the 48 h scheduler limit and were resumed from step checkpoints; the slurm-*.out files cover the separate legs.
• exp1 generated 139 unique videos rather than 143: the val_clips list contains 4 duplicate HOT3D ids (2024, 2604, 2780, 3022). The baseline is 142/145 — one clip (clip-001877) failed to generate on every attempt.