py-feat/bs_to_mesh

blendshape + pose -> MP 478 mesh PLS (v5)

Full-rank linear regression mapping 52 blendshape features + 3 pose covariates (Pitch, Yaw, Roll) to 478×3 = 1434 MediaPipe FaceMesh vertex coordinates in a pose-canonical (Procrustes-aligned) frame. Companion to au_to_mesh_pls — same recipe and SHARED mesh frame, so AU / emotion / blendshape mesh predictions are mutually consistent for visualization.

Training data

• 633,207 frames from 34,854 CelebV-HQ celebrity videos
• All predictions from the SAME forward on the CelebV-HQ chips: mesh478 + 52 blendshapes (MPDetector), AU + emotion + pose (Detectorv2 v2.x)
• Pose-filtered to |yaw| <= 40°, |pitch| <= 30°
• Per-frame Umeyama similarity Procrustes to a GPA reference of 12 stable upper-face anchors (forehead 10/9/8/151, nose bridge 6/168/197/195, outer canthi 33/263, inner canthi 133/362). Removes (R, s, t).
• Top 1% of frames by max anchor residual dropped.
• Absolute aligned coords — NO per-subject neutral subtraction (Cheong / py-feat tutorial-06 recipe).

Method

• Full-rank linear (== full-rank PLSRegression(scale=True); verified equal in prediction on a held-out subsample). Per user request, no low-rank truncation.
• Training inputs: [52 blendshape | 3 pose | 156 blendshape×pose] features.
• Deployed inputs: [52 blendshape | 3 pose] (interactions vanish at pose=0).
• Outputs: 1434-d absolute pose-canonical mesh coords, axis-major [x|y|z].

Out-of-sample performance (3-fold GroupKFold by video)

• Variance-weighted R² = 0.6389 ± 0.0028
• Per-fold R² = [0.6402, 0.6350, 0.6415]
• MAE = 1.2945 (au_to_mesh v5 canonical-frame units)

Inference

import numpy as np
m = np.load("bs_to_mesh_pls_v5.npz")
f = np.zeros(len(m["feature_columns"]))
# e.g. f[m["feature_columns"].tolist().index("happiness")] = 1.0
pose = np.zeros(3)                                  # [Pitch, Yaw, Roll]
x = np.concatenate([f, pose])                       # (55,)
flat = x @ m["coef"] + m["intercept"]               # (1434,)
mesh = np.stack([flat[:478], flat[478:956], flat[956:]], axis=1)  # (478,3) axis-major

File format

NPZ: coef (55, 1434) f32; intercept (1434,) f32; input_columns; feature_columns; feature_name; pose_columns; mean_aligned_mesh (478,3); mean_neutral_mesh (478,3); reference_anchors (12,3); anchor_indices (12,); n_components (); model_card; training_metadata (JSON).