haoyuhsu/imu4d-baselines

IMU4D baseline checkpoints

These are the checkpoints behind the IMU4D baseline numbers. There are two groups:

• IMU → motion: IMUPoser, MobilePoser.
• Motion → text: MotionGPT, MotionGPT3, GeoMotionGPT.

There is also a larger MobilePoser model (XL), which is not part of the reported evaluation.

The folder layout mirrors the IMU4D repo under baseline/. Download into IMU4D_dev/baseline/ and the paths in data/eval_outputs/tools/eval_config.py and in each baseline's scripts work unchanged.

Every model was pretrained jointly on hiphi/v1 + humoto/v1 + motionmillion/v1 + omomo/v1, at 30 fps on the SMPL-X 22-joint skeleton. Below, <tag> stands for hiphi_v1+humoto_v1+motionmillion_v1+omomo_v1.

.
├── IMUPoser/checkpoints/
│   ├── imuposer_pretrain_global-09082026-113948/{epoch=48-val_loss=0.01113.ckpt, best_model.txt}
│   ├── imuposer_ft_humoto_v1_global-09172026-002305/{epoch=32-…ckpt, best_model.txt}
│   ├── imuposer_ft_omomo_v1_global-09172026-002400/{epoch=8-…ckpt, best_model.txt}
│   ├── imuposer_ft_hiphi_v1_global-09172026-002524/{epoch=16-…ckpt, best_model.txt}
│   ├── imuposer_ft_dipimu_v2_global-09092026-171210/{epoch=49-…ckpt, best_model.txt}
│   ├── imuposer_ft_imuposer_v2_global-09092026-171302/{epoch=49-…ckpt, best_model.txt}
│   └── imuposer_ft_ncsa_v3_mr3pt_mr2pt_global-09212026-125047/{epoch=189-…ckpt, best_model.txt}
├── MobilePoser/checkpoints/
│   ├── 1_<tag>/                                  # base preset (6.67M), used in the eval
│   │   ├── base_model.pth
│   │   └── model_finetuned_{humoto_v1,omomo_v1,hiphi_v1,dipimu_v2,imuposer_v2,ncsa_v3_mr3pt_mr2pt}.pth
│   └── 1_<tag>_xl_cont25/base_model.pth          # XL preset (162.9M), NOT in the reported eval
├── MotionGPT/
│   ├── configs/imu4d_m2t.yaml
│   ├── stats/<tag>/{Mean,Std,RawStd}.npy, meta.json
│   └── experiments/mgpt/
│       ├── IMU4D_M2T_<tag>/checkpoints/last.ckpt
│       └── IMU4D_M2T_FT_{humoto,omomo,hiphi}_v1_10ep/checkpoints/last.ckpt
├── MotionGPT3/
│   ├── configs/imu4d_m2t.yaml
│   ├── stats/<tag>/…
│   └── experiments/motgpt/
│       ├── IMU4D_M2T_<tag>/checkpoints/last.ckpt
│       └── IMU4D_M2T_FT_{humoto,omomo,hiphi}_v1_10ep/checkpoints/last.ckpt
└── GeoMotionGPT/
    ├── configs/train/m2t_imu4d.yaml
    ├── stats/<tag>/…
    ├── checkpoints/imu4d-dvq-gsst_<tag>.pt       # DVQ motion codebook (required)
    └── experiments/geomotiongpt/
        ├── IMU4D_GEO_M2T_<tag>/checkpoints/last.ckpt
        └── IMU4D_GEO_M2T_FT_{humoto,omomo,hiphi}_v1_10ep/checkpoints/last.ckpt

Which checkpoint scores which dataset

eval dataset	IMUPoser run	MobilePoser (base)	MotionGPT / MotionGPT3 / GeoMotionGPT
LINGO, HumanML	imuposer_pretrain_global-09082026-113948	base_model.pth	joint IMU4D_[GEO_]M2T_<tag>/last.ckpt
HUMOTO / OMOMO / HiPHI	imuposer_ft_<ds>_v1_global-0917…	model_finetuned_<ds>_v1.pth	IMU4D_[GEO_]M2T_FT_<ds>_v1_10ep/last.ckpt
DIP-IMU / IMUPoser	imuposer_ft_<ds>_v2_global-0909…	model_finetuned_<ds>_v2.pth	n/a
NCSA meeting room, ncsa_v3	imuposer_ft_ncsa_v3_mr3pt_mr2pt_global-09212026-125047	model_finetuned_ncsa_v3_mr3pt_mr2pt.pth	n/a

• Fine-tunes: every scene-level and real-IMU dataset is scored with a model fine-tuned on that dataset alone.
• NCSA checkpoint: one joint fine-tune on ncsa/v3 plus the meeting-room 3-point and 2-point captures. It uses watch (left wrist), phone (right thigh) and headphone, which is the lw_rp_h combo.
• Caveat: the IMUPoser and MobilePoser fine-tunes picked their checkpoint by validation loss on the evaluation data. The motion-to-text fine-tunes use last.ckpt, so no checkpoint was selected on the evaluation data.

IMU → motion

IMUPoser

• Files: Lightning checkpoints, 10.67M params, 42.7 MB each.
• best_model.txt: line 1 holds the repo-relative path of the checkpoint the eval loaded. It is the only .ckpt in each folder.
• Env: conda imuposer (py3.8, torch 2.4). Run with PYTHONPATH=src, because the hparams pickle references imuposer.*.

cd baseline/IMUPoser
PYTHONPATH=src python scripts/evaluate.py \
    --checkpoint checkpoints/imuposer_ft_humoto_v1_global-09172026-002305/epoch=32-val_loss=0.01573.ckpt \
    --datasets humoto/v1 --combo lw_rp_h --max_frames 120

MobilePoser (base preset, used in the eval)

• Files: combined weights for all four modules (poser, joints, foot_contact, velocity). Each is a plain state_dict, 27 MB.
• Default combo: lw_rp_h.
• Best validation losses of the pretrain:
  • poser 0.00394
  • joints 0.00508
  • foot_contact 0.129
• Env: conda mobileposer (py3.9, torch 2.1, lightning 2.1).

cd baseline/MobilePoser
MOBILEPOSER_SIZE=base PYTHONPATH=. python -m mobileposer.evaluate \
    --model checkpoints/1_<tag>/model_finetuned_humoto_v1.pth --datasets humoto/v1 --combo lw_rp_h --max-frames 120

MobilePoser XL (1_<tag>_xl_cont25/base_model.pth, not in the reported eval)

• Size: LSTM 1024×3, 162.9M params, 651 MB.
• Recipe: AdamW 5e-4, weight decay 0.01, cosine schedule with 1k warmup, grad clip 1.0, dropout 0.1, 25 epochs.
• Continuation: poser and joints then trained +25 epochs from their best checkpoints (fresh warmup, cosine, peak lr 2.5e-4).
  • foot_contact and velocity come from the 25-epoch run.
• Best validation losses:
  • poser 0.00189
  • joints 0.00360
  • foot_contact 0.0929
  • velocity 1.923
• There are no per-dataset XL fine-tunes yet.
• It must be loaded with MOBILEPOSER_SIZE=xl. Otherwise load_state_dict fails with a size mismatch.

MOBILEPOSER_SIZE=xl PYTHONPATH=. python -m mobileposer.evaluate \
    --model checkpoints/1_<tag>_xl_cont25/base_model.pth --datasets humoto/v1 --combo lw_rp_h --max-frames 120

Motion → text

All three models run in conda mgpt from their repo root.

• Input: HumanML3D-style 263-dim features on the SMPL-X neutral skeleton, at 30 fps.
• Checkpoints: original Lightning .ckpt files, optimizer state included.
• Joint model: VAE / VQ / DVQ trained from scratch, then the language model.
• Fine-tunes: 10 epochs on one dataset from the joint last.ckpt. Each fine-tune's provenance.json records the source checkpoint's sha256 and the stats hashes.
• Common flags:
  • Add --motion_source <dir of IMU→motion .npz exports> to caption predicted motion instead of GT motion.
  • --stats_dir stats/<tag> is always needed.

model	language model	motion tokenizer	deps/ needed (public, not shipped)
MotionGPT	flan-T5-base	VQ-VAE, inside the .ckpt (vae.*)	deps/flan-t5-base: bash prepare/prepare_t5.sh
MotionGPT3	MoT-GPT2	MLD VAE, inside the .ckpt (vae.*)	deps/gpt2 and deps/mot-gpt2: bash prepare/prepare_gpt2.sh && python -m scripts.gen_mot_gpt
GeoMotionGPT	GPT-2	DVQ codebook, separate file checkpoints/imu4d-dvq-gsst_<tag>.pt	deps/gpt2: bash prepare/prepare_gpt2.sh

# MotionGPT (baseline/MotionGPT)
python evaluate_m2t.py --cfg configs/imu4d_m2t.yaml --datasets humoto/v1 --stats_dir stats/<tag> \
    --checkpoint experiments/mgpt/IMU4D_M2T_FT_humoto_v1_10ep/checkpoints/last.ckpt
# MotionGPT3 (baseline/MotionGPT3)
python evaluate_m2t.py --cfg configs/imu4d_m2t.yaml --datasets humoto/v1 --stats_dir stats/<tag> \
    --checkpoint experiments/motgpt/IMU4D_M2T_FT_humoto_v1_10ep/checkpoints/last.ckpt
# GeoMotionGPT (baseline/GeoMotionGPT)
python evaluate_m2t.py --cfg configs/train/m2t_imu4d.yaml --datasets humoto/v1 --stats_dir stats/<tag> \
    --codebook_path checkpoints/imu4d-dvq-gsst_<tag>.pt \
    --checkpoint experiments/geomotiongpt/IMU4D_GEO_M2T_FT_humoto_v1_10ep/checkpoints/last.ckpt

Each repo also has run_eval.sh, which fills in these defaults (DATASETS=humoto/v1 CHECKPOINT=... bash run_eval.sh).

Integrity

SHA256SUMS covers every file except this README. To check it: sha256sum -c SHA256SUMS.