Video-to-Animation Research

Research repo archiving the attempts, data and findings from converting monocular videos of dancing people into bone animations driving a Ready Player Me / Mixamo avatar. Parked in April 2026; archived for future resume.

Goal

Given any video of a person dancing, produce a playable .glb animation on an RPM-style avatar (e.g. Sofia, Avaturn, Rayan) that visually matches the movement in the video. Quality is measured by a MediaPipe-based pose similarity score (MP score, 0–100).

Target: MP ≥ 60. Best achieved: MP ≈ 39.3 (wham-v18). Ceiling appears to be driven by the extractor's own accuracy on dance motion, not by the retargeting stage.

Pipeline overview

video.mp4
  │
  ├─► [extractor] ──► pose sequence (SMPL, 3D joints, or rotations)
  │                     ├─ WHAM / SMPL-est-X / 4D-Humans / PromptHMR-Vid
  │                     ├─ GVHMR / TRAM / HybrIK
  │                     └─ MediaPipe 3D
  │
  ├─► [retarget] ────► bone rotations on Mixamo skeleton
  │                     ├─ ARP (Auto-Rig Pro) — most used
  │                     ├─ analytical IK from 3D joints
  │                     └─ MoCapAnything-style Procrustes+swing
  │
  ├─► [post] ────────► smoothing / correction
  │                     ├─ SmoothNet
  │                     ├─ DNO / MotionFix refinement (not tried)
  │                     └─ learned-correction (static Δ per bone)
  │
  └─► [avatar bake] ─► final .glb with avatar mesh + animation

Alternative (Option C, MVP implemented): skip extraction entirely and retrieve the closest match from a pre-made Mixamo library via MediaPipe keypoints + DTW — scripts/dtw_match.py.

Approaches tried (chronological)

#	Approach	MP score	Outcome
1	WHAM raw → bone rotations (no post)	6.4	broken bind-pose / twist
2	WHAM + ARP baseline (`wham-v6`)	38.6	OK but shoulders off
3	WHAM + ARP `copy_bone_rest` on shoulders (v18)	39.3	current champion
4	GT rendered → ARP mirror-rest (closure sanity)	17.4	pipeline loses ~60pt even with perfect input
5	Learned per-bone static correction Δ	low	30.5° residual is frame-dependent, can't be static
6	Analytical IK from 3D joints (shortest-arc)	0	axis convention / frame mismatch
7	MoCapAnything-style IK (Procrustes + swing, temporal warm-start)	not measured	math verified symbolically; visually off because WHAM-derived joints inherit WHAM error
8	SMPL-est-X → ARP (with hands)	similar	hand mesh clipping unresolved
9	4D-Humans / HMR2.0 single-frame HD render	0	ViTDet detector failed on workbench renders
10	Retrieval — MediaPipe + DTW → library lookup	n/a	cluster correct (idle→idles, dance→dances); exact GT not always top-1

Key findings (the ones to remember)

Closure test failed (GT animation → our pipeline → compare to GT = MP 17.4). Any extraction approach is capped at ~60 MP because the pipeline itself drops ~60 points via avatar bind-pose, axis conventions, bone-roll mismatches.
Bind-pose mismatch dominates error. SMPL T-pose vs Sofia A-pose = 54.7° delta at the shoulder. Per-bone error breakdown from bone_error_metric.py:
- Hips: 170° (camera-frame/world-frame conversion)
- UpLegs: 170° (same)
- Shoulders: 125° (T-pose vs A-pose)
- Arms: 84°
- Spine: 10°
- Hands: 26°
"Fix the bind-pose" ideas that did NOT work:
- Re-rig Sofia as SMPL (manual Blender) — mesh distorted even with shape-key clean-up
- Swap to known T-pose avatar (avaturn/rayan) — didn't materially change score
- Learned static correction per bone (qlog mean) — error is frame-dependent, not static
MoCapAnything ≠ free lunch without their model. The paper's real contribution is a trained 3D joint predictor (DINOv2 + 4D mesh). Their §3.5 IK recipe applied to WHAM-derived joints cannot exceed WHAM's ceiling, because those joints come from WHAM's own SMPL forward-kinematics. Code not released as of April 2026.
WHAM output convention (critical, repeatedly rediscovered):
- poses_body = 23 SMPL-relative rotmats (body joints 1..23)
- poses_root_world = root rotation in world frame
- trans_world = root translation
- verts_cam = 6890 SMPL vertices in camera space
- Default Y-down unless return_y_up=True. yup2ydown flips Y-axis on root rotation and translation only — body-relative rots are frame-independent.
- For Mixamo retargeting the fast path is poses_body + poses_root_world directly, not via joint positions.
Pipeline loses information even for the "identity" case (GT → extract → retarget → render → score vs GT = 17.4). Sources: (a) GLB animation sampling at 30fps, (b) armature empty-node vs pose-bone copy conversions, (c) rotation order / quaternion sign ambiguities.
HumanMM repo is empty (README only, no code). No forks/mirrors exist. PromptHMR-Vid already integrates TRAM and tests similarly to WHAM.

Approaches worth trying next

Priority order for future-me:

MotionFix / DNO post-refinement over wham-v18 output. Diffusion-based motion denoising. Expected +5–10 MP. Low risk, well-documented.
Physics-based IK (joint limits + contact constraints). ECCV'24 "Biomechanically Accurate Neural IK". Useful if extraction is decent but poses look unnatural.
Trained retarget model. Take our 5–6 annotated GT videos + WHAM extractions → train a neural net that maps SMPL pose → Mixamo-correct pose per bone. Feedback loop with Groq/LLaMA already memoised (project_groq_autooptimize.md).
Better extractor. Wait for HumanMM release (CVPR 2025 specialist on multi-shot dance); or test GVHMR (script exists: scripts/gvhmr_client.py); or PromptHMR-Vid tuning.
Retrieval (Option C2). MVP in this repo is C1 (one-shot nearest-neighbour). C2 adds a sliding DTW window with crossfades between clips — likely produces visually much better output than any single extractor on dances.

Directory layout

Dir	Purpose
`scripts/`	All Python/Blender scripts (77 files). Each targets a specific extractor or stage. See "Key scripts" below.
`extractions/`	Per-extractor output data — `{wham,4d-humans,gvhmr,mediapipe,nlf,prompthmr,prompthmr-vid,smplest-x,wham-joints3d}/` subdirs.
`artifacts/`	Generated strategy GLBs playable in `ui/strategies.html`. Named `${anim}-${extractor}-${strategy}.glb`.
`videos/`	3 test videos: `01-idle.mp4`, `02-dance1.mp4`, `03-dance2.mp4` (rendered from F_Standing_Idle_001 / F_Dances_001 / F_Dances_005).
`ui/`	Side-by-side comparison HTML (`strategies.html`). Loads original vs strategy GLB with Babylon.js.
`motion-library/`	Library signatures (`library.npz`, 119 anims × 12 joints × T frames, body-local normalised) + per-video MediaPipe signatures + match results.
`mocapik/`	GLBs from MoCapAnything-style IK experiment (procrustes + swing, baked on Avaturn).
`docs/`	This README.
`autorig/`	Auto-Rig Pro paid addon binaries (pinned versions). Private backup — do not redistribute. See `autorig/README.md`.

Key scripts

Extraction:

wham_to_joints3d.py — convert WHAM pkl → per-frame 3D joint JSON.
fourdh_to_json.py — same for 4D-Humans.
hybrik_client.py, gvhmr_client.py — HTTP clients for remote extraction services.
extract_video_signature.py — MediaPipe → 12-joint body-local npz.
extract_library_signatures.py — Blender batch extractor for Mixamo library GLBs.

Retargeting:

arp_retarget.py — Auto-Rig Pro retargeting (champion path). Supports --mirror-target-rest for closure tests.
mocapanything_ik.py — Procrustes + swing IK from 3D joints. §3.5 of arxiv 2512.10881.
joints3d_to_animation.py — shortest-arc IK (failed, score 0).
smpl_to_rpm_retarget.py, sofia_source_retarget.py, retarget_fk.py, rokoko_retarget.py, blender_retarget.py — alternative retargeters.
apply_hybrik_correction.py, hybrik_calibration.py — HybrIK-specific post.

Diagnostics:

bone_error_metric.py — per-bone quaternion angular error GT vs extracted.
closure_test.py — GT → our JSON format → compare; validates pipeline correctness.
learn_correction.py — attempt to learn static per-bone Δ (failed, 30.5° residual).
compare_extraction.py, compare_mocap.py, compare_rotations.py — various side-by-side comparators.

Retrieval (Option C, MVP):

dtw_match.py — DTW matching with motion-energy penalty (fix for idle-bias). Results:
- 01-idle → M_Standing_Idle_002 (cost 0.25, correct cluster)
- 02-dance1 → F_Dances_006 (cost 0.80, GT was F_Dances_001)
- 03-dance2 → M_Dances_003 (cost 0.75, GT was F_Dances_005)

Rendering:

render_animated_avatar.py — HD render avatar + animation via workbench engine.
render_wham_mesh.py, render_smpl_verts.py — render raw extractor output (no retarget).

Running experiments (reproduction)

Pre-req: Blender ≥ 4.5, Python ≥ 3.10, pip install mediapipe opencv-python numpy joblib.

# 1. Extract library signatures (one-time, ~3 min for 121 GLBs)
blender -b -P scripts/extract_library_signatures.py -- \
  --lib-dir <path-to-rpm-library> --out motion-library/library.npz

# 2. Extract video signature
python3 scripts/extract_video_signature.py \
  --video videos/02-dance1.mp4 \
  --out motion-library/videos/02-dance1.npz \
  --model <path-to>/pose_landmarker_heavy.task

# 3. Match
python3 scripts/dtw_match.py \
  --library motion-library/library.npz \
  --video motion-library/videos/02-dance1.npz \
  --subsample 3 --top-k 10 --mode dtw

# 4. MoCapAnything IK from WHAM joints (alternative path)
blender -b -P scripts/mocapanything_ik.py -- \
  --avatar <path-to>/avaturn.glb \
  --joints extractions/wham-joints3d/02-dance1-joints3d.json \
  --out mocapik/02-dance1-wham-mocapik-arp.glb

Related work / external deps (not stored here, reconstruct if needed)

Originally under tests/ml-mocap/{extern,humor,synsp,video2anim,data}/, excluded to keep repo size down. Fetch from:

WHAM: https://github.com/yohanshin/WHAM
4D-Humans / HMR2.0: https://github.com/shubham-goel/4D-Humans
PromptHMR / PromptHMR-Vid: https://github.com/rawalkhirodkar/prompthmr
GVHMR: https://github.com/zju3dv/GVHMR
SMPL-est-X: https://github.com/SMPLCap/SMPLest-X
HybrIK: https://github.com/jeffffffli/HybrIK
MoCapAnything: https://animotionlab.github.io/MoCapAnything/ (code not released, Apr 2026)
HumanMM: CVPR 2025 — repo README-only as of Apr 2026
Auto-Rig Pro (Blender addon): paid, https://blendermarket.com/products/auto-rig-pro

Models required locally:

pose_landmarker_heavy.task (MediaPipe) — https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_heavy/float16/latest/pose_landmarker_heavy.task
SMPL body model (basicModel_neutral_lbs_10_207_0_v1.0.0.pkl) — https://smpl.is.tue.mpg.de/
SMPL-X models — https://smpl-x.is.tue.mpg.de/

License

Internal research. MIT on original code in scripts/. External test videos and Mixamo-derived GLBs retain their original licenses.

Status

Archived April 2026. Ceiling MP ≈ 39 was not surpassed by any extraction-based pipeline tried here. See "Approaches worth trying next" when resuming.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support