heartwise/deepcoro_clip_mace

DeepCORO-CLIP MACE (transfer-learning heads)

One-year major adverse cardiovascular event (MACE) prediction from a complete multi-view coronary angiography study. The DeepCORO-CLIP video–language encoder is fine-tuned end-to-end with a multi-instance (study-level) pooling head and one sigmoid output per endpoint.

Research use only — not for clinical use. The model has not been prospectively validated and carries no regulatory clearance. Angiographic videos alone cannot capture the full clinical picture; outputs are risk scores for research and external-validation purposes.

Code, configs and the Docker external-validation pipeline: https://github.com/HeartWise-AI/DeepCORO_CLIP

Files

Path	Content
11zt0zl5_20250723-162407/models/best_model_epoch_18.pt	Released checkpoint. Full fine-tuned model (linear_probing state dict: MViT encoder + MIL pooling + 9 heads), epoch 18 of run 11zt0zl5.
11zt0zl5_20250723-162407/config.yaml	Training configuration of the released checkpoint.
config.json	Architecture / preprocessing summary (mean, std, num_videos, pooling).
class_mapping.json	Head names, task type and default 0.5 threshold.
inference_external_validation.yaml	Ready-to-run inference config for scripts/runner.sh in the GitHub repo.
m3wanj4z_20251201-225344/config.yaml	W&B training config of the manuscript Table 4 model (weights not included, see Provenance).
7vhghgbs_20260112-132350/config.yaml	W&B inference config that produced the manuscript Table 4 predictions.

Heads

Head	Endpoint
mace_urgent_revascularization_binary	Urgent revascularization within 1 year
mace_non_fatal_mi_binary	Non-fatal myocardial infarction within 1 year
mace_cv_death_binary	Cardiovascular death within 1 year (very few events; unreliable)
complete_occlusion_coronary_disease_binary	Complete coronary occlusion on the index angiogram
mace_binary	Composite of any component below
mace_non_fatal_stroke_binary, mace_heart_failure_hosp_binary, mace_life_threatening_arrhythmia_binary, mace_cardiogenic_shock_binary	Secondary components (few events; exploratory)

The manuscript's core-events composite (urgent revascularization, non-fatal MI, CV death) has no dedicated head in this checkpoint. Use mace_binary, or the maximum of the three component probabilities; both give equivalent discrimination (see below).

Architecture and preprocessing

• Encoder: MViT-v2-S video backbone (DeepCORO-CLIP pretrained), 16 frames, stride 1, 224×224.
• Study-level aggregation: cls_token multi-instance pooling over up to 3 videos per study (num_videos: 3); studies with more videos are subsampled, fewer are padded with a real-video mask.
• Heads: 9 × linear (dim 1) with BCE-with-logits loss.
• Normalization (training-set statistics of run 11zt0zl5, apply as-is on new data, do not recompute): mean = [107.362, 107.362, 107.362], std = [34.678, 34.678, 34.678] on 0–255 pixel values.
• Input CSV: α-separated, one row per video, FileName (AVI/MP4 path), StudyInstanceUID, Split == "inference". Ground-truth columns are optional.

Performance

Released checkpoint (11zt0zl5, epoch 18) on the manuscript MACE test cohort

350 studies (338 patients), Montreal Heart Institute, followed up to one year. Study-level AUROC with 1,000-resample bootstrap 95% CI. Predictions were regenerated with the public repository code (config/inference/mace_external_validation_inference.yaml).

Endpoint	Events	AUROC (95% CI)	AUPRC
Urgent revascularization	89/350	0.82 (0.78–0.87)	0.62
Non-fatal MI	37/350	0.78 (0.69–0.85)	0.30
Complete coronary occlusion	25/350	0.81 (0.72–0.89)	0.28
Composite (any component, mace_binary)	134/350	0.78 (0.73–0.83)	0.68
Core-events composite, scored with mace_binary	111/350	0.82 (0.76–0.86)	—
Core-events composite, max of component heads	111/350	0.81 (0.76–0.85)	—
CV death	4/350	0.42 (0.11–0.83)	0.01

Caveat. This cohort was the held-out test set of the manuscript model, not of this checkpoint: 49 of the 350 studies were in the validation split of run 11zt0zl5, and its training split file is no longer available, so training-set overlap cannot be excluded. Treat these numbers as internal performance, not as a clean held-out estimate. External validation on an independent cohort is the purpose of this release.

Validation-split AUROC logged during training (W&B, epoch 18): urgent revascularization 0.78, non-fatal MI 0.77, complete occlusion 0.87, composite 0.69.

Manuscript Table 4 (run m3wanj4z, same 350-study cohort, for reference)

Endpoint	AUROC (95% CI)
Urgent revascularization	0.82 (0.77–0.87)
Non-fatal MI	0.71 (0.63–0.80)
Complete coronary occlusion	0.77 (0.64–0.87)
Composite core events	0.79 (0.74–0.84)

Provenance

• 11zt0zl5 (July 2025): 9 binary heads, cls_token pooling, num_videos 3, 20 epochs, best epoch 18 by validation loss. This is the checkpoint released here.
• m3wanj4z (December 2025): 4 heads (urgent revascularization, non-fatal MI, complete occlusion, core-events composite), attention+cls_token pooling, num_videos 10, best epoch 1. Its epoch-1 checkpoint produced the manuscript Table 4 numbers via inference run 7vhghgbs (January 2026). The checkpoint lived in a working copy that has since been deleted and was not archived; only its W&B configs survive and are included so the recipe can be re-run.

Usage

git clone https://github.com/HeartWise-AI/DeepCORO_CLIP && cd DeepCORO_CLIP
# weights/deepcoro_clip_mace/ is populated by utils/download_pretrained_weights.py (needs api_key.json)
cp inference_external_validation.yaml config/inference/   # or use the copy shipped in the repo
bash scripts/runner.sh --base_config config/inference/mace_external_validation_inference.yaml \
     --run_mode inference --use_wandb false --selected_gpus 0

Docker external validation from DICOMs (DICOM → AVI → VasoVision view filtering → DeepCORO MACE):

docker run --rm --gpus all --ipc=host \
  -e EXTERNAL_VALIDATION_DATA_PATH=/app/data/input.csv \
  -e DEEPCORO_BASE_CONFIG=config/linear_probing/mace/docker_base_config_mace.yaml \
  -v /path/to/dicoms:/path/to/dicoms -v /path/to/input.csv:/app/data/input.csv \
  -v /path/to/results:/workspace/results deepcoro_clip-docker python scripts/external_validation.py

Outputs: inference_predictions_best_epoch_-1.csv with one row per study and a <head>_pred probability column per head.

Citation

DeepCORO-CLIP: a video–language foundation model for coronary angiography (HeartWise-AI, Montreal Heart Institute). See the GitHub repository for the current citation.