MDFLD v2: cartoon face landmark detection (68 points)

Model weights for Towards Multi-Domain Face Landmark Detection with Synthetic Data from Diffusion Model (Yuanming Li, Gwantae Kim, Jeong-gi Kwak, Bon-hwa Ku, Hanseok Ko; ICASSP 2024).

Paper (arXiv:2401.13191) · Code · Project page

This is the cartoon model v2: the paper model fine-tuned with self-training on cartoon and caricature faces. On the CariFace caricature test set (1,560 caricatures) it lowers the 68-point NME from 6.70% to 5.03% (eyes 3.49% → 2.78%, jaw 10.42% → 7.75%) with the package's three-pass crop; on held-out real-face video frames it reaches 1.74% (v1: 3.03%). See the GitHub README for the full table, the training data and the limitations.

Use

The easiest way is the mdfld package, which downloads this model, finds the face and adds eyelid curves:

pip install "mdfld[sam] @ git+https://github.com/zenonLEE/MDFLD"
mdfld image face.png --out face.json --vis face_mdfld.png

The file mdfld_cartoon_v2.torchscript.pt can also be used directly with PyTorch, without any code from the repository:

import torch
net = torch.jit.load("mdfld_cartoon_v2.torchscript.pt", map_location="cuda")
# x: (N, 3, 256, 256) float tensor of BGR face crops, values pixel / 255 * 2 - 1
out = net(x)                        # (N, 68, 2), x and y in [-1, 1]
pts = ((out + 1) * 256 - 1) / 2     # pixels of the 256 x 256 crop

The crop is a square centred on the face box with side equal to the larger side of the box, resized to 256 × 256. For the best accuracy, run two more passes, each on a square of 1.1 × the larger side of the box of the previous points, as the package does (mdfld/landmarks.py). The 68 points follow the iBUG 300-W order (0-based).

Files

File	Use
mdfld_cartoon_v2.torchscript.pt	TorchScript model, used by the mdfld package

The same weights in the original checkpoint format (mdfld_cartoon_v2.pkl, for the STAR-based code in the repository) are in the v2.0 release.

Terms of use

Research use only, without warranty. The model was trained on third-party data with their own terms (CariFace is for research only; several cartoon sets contain third-party artwork). See the terms of use.

Citation

@inproceedings{li2024towards,
  title={Towards Multi-Domain Face Landmark Detection with Synthetic Data from Diffusion Model},
  author={Li, Yuanming and Kim, Gwantae and Kwak, Jeong-gi and Ku, Bon-hwa and Ko, Hanseok},
  booktitle={ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={6730--6734},
  year={2024},
  organization={IEEE}
}

Teaser images: public-domain works from Wikimedia Commons and frames of Sintel (© copyright Blender Foundation | durian.blender.org, CC BY 3.0).