Vision Cardio — rPPG Heart-Rate Model (PhysNet, UBFC fine-tuned)
Camera-based remote photoplethysmography (rPPG) model that estimates heart rate from short front-camera face clips. PhysNet-style 3D-CNN that outputs an rPPG waveform; HR is read off as the FFT peak in the 0.7–3.0 Hz band.
Powers the on-device VISION CARDIO iOS app (offline, Korean UI, wellness-only): front camera → HR, with running HR-phase zones (1–5) and weights set/rest recovery coaching measured against a calibrated baseline HR. Built and running on iPhone (iOS 16+); Core ML runs on GPU/CPU.
Files
| file | what |
|---|---|
rppg_physnet_ubfc.pt |
PyTorch checkpoint (PhysNet, width=32), UBFC fine-tuned |
VisionCardioHR.mlpackage |
Core ML model for iOS 16+ (same weights, traced) |
Performance
Trained on synthetic SCAMPS, then fine-tuned on real UBFC-rPPG faces with a strict by-participant split (no subject leakage).
| stage | HR MAE (vs contact-PPG) |
|---|---|
| zero-shot (SCAMPS only) on UBFC | 5.63 bpm |
| after UBFC fine-tune (val) | 2.80 bpm |
Fine-tune split: 40 train / 10 val subjects; best-by-val checkpoint (early-stopped — the tiny set overfits after a couple of epochs).
I/O contract
input "clip" : (1, 3, T=128, H=112, W=112) float, RGB, [0,1] normalized
output "waveform" : (1, 128) predicted rPPG pulse
HR = FFT peak in [0.7, 3.0] Hz, fs = clip_frames / clip_seconds = 128 / 20 = 6.4 Hz
Core ML metadata carries clip_frames, clip_size, clip_seconds, fs, hr_band_hz
so the app stays in sync automatically.
Usage (PyTorch)
import torch
from ml.physnet import PhysNet, hr_from_wave # repo: PFSV/ByeongYeok_RnD_NLP (vision_cardio)
ck = torch.load("rppg_physnet_ubfc.pt", map_location="cpu")
m = PhysNet(width=ck["width"]); m.load_state_dict(ck["state_dict"]); m.eval()
wave = m(clip)[0].numpy() # clip: (1,3,128,112,112) in [0,1]
bpm = hr_from_wave(wave, fs=6.4)
Caveats
- Wellness/research only — not a medical device, no diagnostic claims.
- Fine-tuned on a small real set (UBFC); generalization to arbitrary lighting / skin tone / motion is not guaranteed. Use confidence gating before trusting an estimate.
Code, pipeline & iOS app: PFSV/ByeongYeok_RnD_NLP, branch vision-cardio-ubfc-app
→ TUTORIAL_DRILL/neuro-nlp/vision_cardio.
- Downloads last month
- 5