hugging-mac/mediapipe-hand-coreml

MediaPipe Hand Detection & Landmarks — Core ML

A ready-to-use two-model Core ML pipeline for real-time hand tracking on Apple Silicon, converted from Qualcomm AI Hub's MediaPipe Hand ONNX release.

The bundle contains both models required by the pipeline:

1. Palm detector — finds hands and seven detector keypoints.
2. Hand landmark detector — predicts presence, handedness, and 21 three-dimensional landmarks from each detected hand crop.

Together they enable gesture-controlled apps, games, plugins, and agents. Gesture recognition itself is implemented by interpreting the 21 landmarks; these models do not contain a separate gesture classifier. See Hugging Mac for the complete local macOS pipeline.

Model details

Component	Input	Outputs	Parameters
hand_detector.mlpackage	image: 1 × 3 × 256 × 256 FP32 RGB	box_coords: 1 × 2944 × 18; box_scores: 1 × 2944 × 1	1.76M
hand_landmark_detector.mlpackage	image: 1 × 3 × 256 × 256 FP32 RGB crop	scores: 1; lr: 1; landmarks: 1 × 21 × 3	2.01M

• Precision: FP32
• Core ML deployment target: macOS 13 or later
• Combined package size: 15.3 MB
• The landmark model can be loaded lazily only when landmarks are requested.

Core ML example

from pathlib import Path

import coremltools as ct
import numpy as np
from huggingface_hub import snapshot_download
from PIL import Image, ImageOps

root = Path(snapshot_download(
    repo_id="hugging-mac/mediapipe-hand-coreml",
    allow_patterns=["coreml/**"],
)) / "coreml"

detector = ct.models.MLModel(
    root / "hand_detector.mlpackage",
    compute_units=ct.ComputeUnit.ALL,
)
image = ImageOps.pad(Image.open("hand.jpg").convert("RGB"), (256, 256))
value = np.asarray(image, dtype=np.float32) / 255.0
tensor = np.ascontiguousarray(value.transpose(2, 0, 1))[None]
outputs = detector.predict({"image": tensor})

print(outputs["box_coords"].shape)  # (1, 2944, 18)
print(outputs["box_scores"].shape)  # (1, 2944, 1)

Raw detector outputs require MediaPipe anchor decoding and weighted NMS. The resulting rotated hand crop is passed to the landmark model. Use the Hugging Mac MediaPipe Hand SDK for the complete pipeline, coordinate projection, and handedness handling.

Conversion verification

The Core ML models were compared with ONNX Runtime on identical random inputs. Maximum absolute differences:

• Detector boxes: 6.866e-5
• Detector scores: 1.526e-4
• Hand presence: 3.92e-8
• Handedness: 4.59e-6
• Landmarks: 4.47e-7

Provenance and integrity

• Upstream: Qualcomm MediaPipe Hand Detection
• Upstream revision: 013e27b599e37c3b4c69439de15de53cc5b5708e
• Qualcomm AI Hub release: v0.60.0, ONNX float
• hand_detector.mlpackage SHA-256: a81143ef0bfa896d4206caf226697141ab8c052d57ba5e95fc7c7ac2660dc83d
• hand_landmark_detector.mlpackage SHA-256: d4049538d8fe4b4ffa77f71e1efa43705fc010e721c0456a95d9c97453c13716
• Complete bundle SHA-256: 5432c988be7ed2fd6847097f823e0d813f97be7dc7a7d6587129386dbcd75929

License

The upstream repository declares a custom license and links to the MediaPipePyTorch license, which states that the implementation follows the Apache License 2.0 terms used by MediaPipe. Review the upstream Qualcomm distribution terms and linked source license before redistribution or commercial use. Hugging Mac is not affiliated with or endorsed by Qualcomm, Google, or MediaPipe.