Medical Imaging Models

This repository contains two complementary medical imaging AI model collections:

BiomedCLIP Router - Routes medical images to the correct imaging modality
Disease-Specific Models - Classification and segmentation models for various medical imaging domains

Model 1: BiomedCLIP Router

Description

A modality classification model that routes medical images to one of 7 imaging categories. Built on Microsoft's BiomedCLIP backbone, this router enables automatic sorting of medical images to the appropriate downstream model.

Supported Modalities

Class	Modality
0	Endoscopy
1	Dermatology
2	X-Ray
3	Ultrasound
4	Mammography
5	Fundus / Retinography
6	Microscopy

Performance

Metric	Test Score
Accuracy	100%
Macro F1	100%
Macro Precision	100%
Macro Recall	100%

Training Details

Base Model: microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224
Training Images: 14,000 (2,000 per class)
Validation Images: 1,750 (250 per class)
Test Images: 1,750 (250 per class)
Best Epoch: 1
Validation Macro F1: 99.94%

Files

biomedclip_router_20260423_194004/
├── best_biomedclip_router.pt    # Trained model weights (748MB)
├── metrics.json                  # Training metrics and history
├── train.log                     # Training logs
├── session_name.txt              # Session identifier
└── unseen_holdout_300/          # Holdout test data info

Model 2: Disease-Specific Models

A collection of specialized models for disease classification and segmentation across 8 medical imaging domains.

2.1 Breast Ultrasound

Classification Model

Task: Classify breast ultrasound images as benign, malignant, or normal

Metric	Test Score
Accuracy	92.31%
Macro F1	90.75%
Macro Precision	89.79%
Macro Recall	91.85%
Best Epoch	23

Classes: benign, malignant, normal

Segmentation Model

Task: Segment lesions in breast ultrasound images

Metric	Test Score
Dice	83.80%
IoU	76.17%
Best Epoch	20
Architecture	SMP UNet++

Files:

breast_ultrasound/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmentor.pt
    ├── metrics.json
    └── plots/

2.2 Chest X-Ray

Classification Model

Task: 4-class chest X-ray classification (COVID-19 Radiography Database)

Metric	Test Score
Accuracy	96.17%
Macro F1	96.08%
Macro Precision	96.40%
Macro Recall	95.79%
Best Epoch	11

Classes: covid, lung_opacity, normal, viral_pneumonia

Architecture: DenseNet121 (torchvision IMAGENET1K_V1, the CheXNet chest-X-ray backbone) + fresh 4-class head (replaces the previous 15-class BiomedCLIP multi-label model that scored ~21% exact-match accuracy)
Dataset: COVID-19 Radiography Database (Kaggle tawsifurrahman/covid19-radiography-database); also mirrored at dataset repo umairinayat/fyp-dataset
Deduplication: near-duplicates removed via MD5 + perceptual pHash (mirror-aware) -> 19,133 unique images
Split: stratified 70/15/15 (train 13,393 / val 2,870 / test 2,870), deduplicated globally before splitting (no train/test leakage)
Input: 224x224, ImageNet normalization
Training: 15 epochs, AdamW + OneCycleLR (max_lr 2e-4, weight_decay 0.03), tempered class-weighted CrossEntropy (label smoothing 0.1), mixed precision; augmentation = RandomResizedCrop + TrivialAugmentWide + flip + RandomErasing (train only); no test-time augmentation

Images per class (after dedup; total unique = 19,133):

Class	Train	Val	Test	Total
covid	2,225	477	477	3,179
lung_opacity	4,088	876	876	5,840
normal	6,152	1,318	1,319	8,789
viral_pneumonia	928	199	198	1,325
Total	13,393	2,870	2,870	19,133

Segmentation Model

Task: Binary lung-region segmentation (using the dataset's masks/)

Metric	Test Score
Dice	98.71%
IoU	97.50%
Best Epoch	15
Architecture	SMP UNet (EfficientNet-B0 ImageNet encoder)

Loss: BCEWithLogits + Dice; same leakage-free split as classification; image & mask resized to 256x256; train augmentation = flip + rot90

Files:

chest_xray/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   ├── training_history.json
│   └── plots/
└── segmentation/
    ├── best_segmenter.pt
    ├── metrics.json
    ├── training_history.json
    └── plots/

2.3 Dermatology

Classification Model

Task: Classify skin lesions as malignant or non-malignant (2-class)

Metric	Test Score
Accuracy	92.25%
Macro F1	90.47%
Macro Precision	89.50%
Macro Recall	91.63%
Best Epoch	8

Classes: malignant, non_malignant

Architecture: ConvNeXt-Small (torchvision IMAGENET1K_V1) + 2-class head (replaces the previous BiomedCLIP model at 88.74%)
Dataset: combined 2 sources for variety - HAM10000 (Kaggle surajghuwalewala/ham1000-segmentation-and-classification, 7->2 class) + fanconic/skin-cancer-malignant-vs-benign
Leakage-free split: HAM10000 has multiple images per lesion, so it is split by lesion_id (all images of a lesion stay in one split); fanconic uses its own official train/test holdout -> train 9,667 / val 1,489 / test 2,156
Training: 12 epochs @ 224px, AdamW + OneCycleLR, tempered class-weighted CE (label smoothing 0.1), mixed precision; augmentation = RandomResizedCrop + TrivialAugmentWide + flip + RandomErasing

Images per class:

Class	Train	Val	Test	Total
malignant	2,559	307	585	3,451
non_malignant	7,108	1,182	1,571	9,861
Total	9,667	1,489	2,156	13,312

Segmentation Model

Task: Binary lesion segmentation (HAM10000 lesion masks)

Metric	Test Score
Dice	95.25%
IoU	91.52%
Best Epoch	18
Architecture	SegFormer-B3 (transformer)

Architecture: SegFormer-B3 (nvidia/segformer-b3-finetuned-ade-512-512, ADE20K-pretrained) with a 1-label binary head
Dataset: HAM10000 lesion masks (Kaggle surajghuwalewala/ham1000-segmentation-and-classification), 10,015 image+mask pairs
Split: lesion-level (by lesion_id) so the same lesion never appears in both train and test -> train 6,987 / val 1,478 / test 1,550 (no leakage)
Training: 18 epochs @ 256px, AdamW + OneCycleLR (max_lr 6e-5), BCEWithLogits + Dice, mixed precision; augmentation = hflip + rot90

Files:

dermatology/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmenter.pt
    ├── metrics.json
    └── plots/

2.4 Endoscopy

Classification Model

Task: Classify GI endoscopy images into 5 classes (healthy added vs. previous 4-class model)

Metric	Test Score
Accuracy	94.73%
Macro F1	80.35%
Macro Precision	81.96%
Macro Recall	79.66%
Best Epoch	20

Classes: barretts, esophagitis, polyp, ulcerative_colitis, healthy

Architecture: ConvNeXt-Small (torchvision IMAGENET1K_V1) + fresh 5-class head (replaces the previous 4-class BiomedCLIP model)
Dataset: HyperKvasir labeled images (23 source classes mapped to 5 targets); barretts<-{barretts,barretts-short-segment}, esophagitis<-{esophagitis-a,esophagitis-b-d}, polyp<-{polyps}, ulcerative_colitis<-{ulcerative-colitis-grade-*}, healthy<-{cecum,pylorus,z-line,retroflex-rectum,retroflex-stomach,ileum}
Deduplication: near-duplicates removed via MD5 + perceptual pHash (mirror-aware) -> 6,695 unique images
Split: stratified 70/15/15 (train 4,686 / val 1,004 / test 1,005), deduplicated globally before splitting (no train/test leakage)
Input: 384x384, ImageNet normalization
Training: 22 epochs, AdamW + OneCycleLR (max_lr 2e-4, weight_decay 0.05), tempered class-weighted CrossEntropy (label smoothing 0.1), mixed precision; augmentation = RandomResizedCrop + TrivialAugmentWide + flip + RandomErasing (train only)
Eval: no test-time augmentation (single center-crop)
Note: barretts has only 94 images in all of HyperKvasir (66 train / 14 test), so it remains the hardest class and caps macro-F1; the other 4 classes score F1 0.84-0.99

Images per class (after dedup; total unique = 6,695):

Class	Train	Val	Test	Total
barretts	66	14	14	94
esophagitis	463	100	99	662
polyp	713	152	153	1,018
ulcerative_colitis	587	126	126	839
healthy	2,857	612	613	4,082
Total	4,686	1,004	1,005	6,695

Segmentation Model

Task: Segment regions of interest in endoscopy images

Metric	Test Score
Dice	88.52%
IoU	82.18%
Best Epoch	38
Architecture	SMP UNet++

Files:

endoscopy/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmentor.pt
    ├── metrics.json
    └── plots/

2.5 Mammography

Classification Model

Task: Classify mammogram patches as benign, malignant, or normal (3-class)

Metric	Test Score
Accuracy	99.63%
Macro F1	99.72%
Macro Precision	99.71%
Macro Recall	99.72%
Best Epoch	12

Classes: benign, malignant, normal

Architecture: ResNet50 (torchvision IMAGENET1K_V2 pretrained weights) + fresh 3-class linear head
Dataset: INbreast + MIAS + DDSM (CLAHE preprocessed) - 26,602 raw images
Deduplication: 10,472 duplicate / near-duplicate images removed via MD5 + perceptual pHash (LSH banding, mirror-aware, Hamming <= 5) -> 16,130 unique images
Split: stratified 70/15/15 (train 11,291 / val 2,419 / test 2,420), deduplicated globally before splitting, so the held-out test set contains no image (or near-duplicate) seen in train/val - no leakage
Input: 224x224, grayscale replicated to 3 channels, ImageNet normalization
Training: 15 epochs, AdamW + OneCycleLR (max_lr 3e-4), class-weighted CrossEntropy, mixed precision
Augmentation (train only): RandomResizedCrop, RandomHorizontalFlip, RandomRotation(15), RandomAffine, ColorJitter, RandomErasing

Images per class (after dedup; total unique = 16,130):

Class	Train	Val	Test	Total
benign	4,505	965	966	6,436
malignant	5,428	1,163	1,163	7,754
normal	1,358	291	291	1,940
Total	11,291	2,419	2,420	16,130

Replaces the previous 2-class (BENIGN/MALIGNANT) BiomedCLIP mammography classifier (was 66% accuracy). The new model is a ResNet50 - load it with torchvision.models.resnet50 (see Usage below), not the BiomedCLIP loader.

Test confusion matrix (rows = true, cols = predicted benign, malignant, normal):

benign:    [963,    3,    0]
malignant: [  6, 1157,    0]
normal:    [  0,    0,  291]

Segmentation Model

Task: Binary mass segmentation in mammography (CBIS-DDSM)

Metric	Test Score
Dice	90.48%
IoU	83.01%
Best Epoch	13
Architecture	SegFormer-B3 (transformer)

Architecture: SegFormer-B3 (nvidia/segformer-b3-finetuned-ade-512-512, ADE20K-pretrained), fine-tuned with a 1-label binary head — replaces the previous UNet++ (86.31% Dice)
Dataset: CBIS-DDSM mass cases (Kaggle awsaf49/cbis-ddsm-breast-cancer-image-dataset); full mammogram + full-image ROI mask paired, square patch cropped around each lesion (2x margin) -> 512x512
Split: official CBIS-DDSM mass train/test (patient-disjoint, no leakage); 15% of train held as validation -> train 979 / val 172 / test 341 patches
Training: 15 epochs @ 384px, AdamW + OneCycleLR (max_lr 6e-5), BCEWithLogits + Dice, mixed precision; augmentation = hflip/vflip/rot90
Eval: no test-time augmentation

Files:

mammography/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmentor.pt
    ├── metrics.json
    └── plots/

2.6 Thyroid Ultrasound

Classification Model

Task: Classify thyroid nodules by risk level

Metric	Test Score
Accuracy	88.57%
Macro F1	79.89%
Macro Precision	76.79%
Macro Recall	85.00%
Best Epoch	8

Classes: low_risk, suspicious

Segmentation Model

Task: Segment thyroid nodules in ultrasound images

Metric	Test Score
Dice	83.43%
IoU	73.64%
Best Epoch	21
Architecture	SMP UNet++

Files:

thyroid_ultrasound/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmentor.pt
    ├── metrics.json
    └── plots/

Performance Summary

Classification Models Performance

Domain	Accuracy	Macro F1	Macro Precision	Macro Recall
BiomedCLIP Router	100%	100%	100%	100%
Mammography	99.63%	99.72%	99.71%	99.72%
Chest X-Ray	96.17%	96.08%	96.40%	95.79%
Endoscopy	94.73%	80.35%	81.96%	79.66%
Breast Ultrasound	92.31%	90.75%	89.79%	91.85%
Dermatology	92.25%	90.47%	89.50%	91.63%
Thyroid Ultrasound	88.57%	79.89%	76.79%	85.00%

Mammography uses ResNet50, Endoscopy uses ConvNeXt-Small, and Chest X-Ray uses DenseNet121 (other disease classifiers use BiomedCLIP ViT-B/16).

Segmentation Models Performance

Domain	Dice	IoU	Architecture
Chest X-Ray	98.71%	97.50%	SMP UNet (EfficientNet-B0)
Dermatology	95.25%	91.52%	SegFormer-B3
Mammography	90.48%	83.01%	SegFormer-B3
Endoscopy	88.52%	82.18%	SMP UNet++
Breast Ultrasound	83.80%	76.17%	SMP UNet++
Thyroid Ultrasound	83.43%	73.64%	SMP UNet++

Dataset Statistics (per-class splits)

Counts = train / val / test images after de-duplication. "Unique total" = images actually used. Legacy BiomedCLIP / SMP models (Breast US, Thyroid US, Endoscopy seg, Fundus, Microscopy) were trained externally before this work; their per-class splits were not recorded and are marked N/R.

BiomedCLIP Router (7 modalities)

Class	Train	Val	Test	Total
Endoscopy / Dermatology / X-Ray / Ultrasound / Mammography / Fundus / Microscopy (each)	2,000	250	250	2,500
Total (7 classes)	14,000	1,750	1,750	17,500

Mammography — Classification (3-class) · INbreast+MIAS+DDSM (CLAHE), deduped

Class	Train	Val	Test	Total
benign	4,505	965	966	6,436
malignant	5,428	1,163	1,163	7,754
normal	1,358	291	291	1,940
Total	11,291	2,419	2,420	16,130

Mammography — Segmentation · CBIS-DDSM mass (binary patches, official patient split)

Split	Train	Val	Test	Total
image-mask patches	979	172	341	1,492

Chest X-Ray — Classification (4-class) · COVID-19 Radiography, deduped

Class	Train	Val	Test	Total
covid	2,225	477	477	3,179
lung_opacity	4,088	876	876	5,840
normal	6,152	1,318	1,319	8,789
viral_pneumonia	928	199	198	1,325
Total	13,393	2,870	2,870	19,133

Chest X-Ray — Segmentation · COVID-19 (binary lung, same images)

Split	Train	Val	Test	Total
image-mask pairs	13,393	2,870	2,870	19,133

Endoscopy — Classification (5-class, healthy added) · HyperKvasir, deduped

Class	Train	Val	Test	Total
barretts	66	14	14	94
esophagitis	463	100	99	662
polyp	713	152	153	1,018
ulcerative_colitis	587	126	126	839
healthy	2,857	612	613	4,082
Total	4,686	1,004	1,005	6,695

Dermatology — Classification (2-class) · HAM10000 (lesion-level) + fanconic

Class	Train	Val	Test	Total
malignant	2,559	307	585	3,451
non_malignant	7,108	1,182	1,571	9,861
Total	9,667	1,489	2,156	13,312

Dermatology — Segmentation · HAM10000 lesion masks (binary, lesion-level split)

Split	Train	Val	Test	Total
image-mask pairs	6,987	1,478	1,550	10,015

Legacy models (per-class splits not recorded)

Model	Classes (from config)	Status
Breast Ultrasound (cls + seg)	benign / malignant / normal	BiomedCLIP / SMP UNet++ — N/R
Thyroid Ultrasound (cls + seg)	low_risk / suspicious	BiomedCLIP / SMP UNet++ — N/R
Endoscopy (segmentation)	polyp (binary)	SMP UNet++ — N/R
Fundus Retinography (cls)	no_retinal_disease / retinal_disease	BiomedCLIP — N/R; no segmentation
Microscopy (cls)	—	BiomedCLIP — N/R; no segmentation

Usage

Installation

pip install torch torchvision huggingface_hub open_clip_torch segmentation-models-pytorch

Download Models

# Download entire repository
hf download umairinayat/medical-models --local-dir ./medical-models

# Download specific components
hf download umairinayat/medical-models biomedclip_router_20260423_194004 --local-dir ./router
hf download umairinayat/medical-models disease_models --local-dir ./disease_models

Router Inference Example

import torch
import open_clip
from PIL import Image

# Load BiomedCLIP Router
model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms(
    'hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224'
)
tokenizer = open_clip.get_tokenizer(
    'hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224'
)

# Load trained router weights
checkpoint = torch.load('biomedclip_router_20260423_194004/best_biomedclip_router.pt', map_location='cpu')
model.load_state_dict(checkpoint['model_state_dict'])

# Inference
image = preprocess_val(Image.open('medical_image.png')).unsqueeze(0)
with torch.no_grad():
    output = model(image)
    predicted_class = output.argmax(dim=1).item()

class_names = ['Endoscopy', 'Dermatology', 'X-Ray', 'Ultrasound', 
               'Mammography', 'Fundus/Retinography', 'Microscopy']
print(f"Predicted modality: {class_names[predicted_class]}")

Mammography Classification Inference (ResNet50)

import torch
from PIL import Image
from torchvision import models, transforms
from huggingface_hub import hf_hub_download

# Load checkpoint
ckpt_path = hf_hub_download(
    'umairinayat/medical-models',
    'disease_models/mammography/classification/best_classifier.pt')
ckpt = torch.load(ckpt_path, map_location='cpu', weights_only=False)
class_names = ckpt['class_names']  # ['benign', 'malignant', 'normal']

# Build model
model = models.resnet50()
model.fc = torch.nn.Linear(model.fc.in_features, len(class_names))
model.load_state_dict(ckpt['model_state'])
model.eval()

# Preprocess (224x224, grayscale -> RGB, ImageNet stats)
tfm = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(ckpt['normalization_mean'], ckpt['normalization_std']),
])
img = Image.open('mammogram.png').convert('RGB')
x = tfm(img).unsqueeze(0)
with torch.no_grad():
    probs = torch.softmax(model(x), dim=1)[0]
print({c: round(float(p), 4) for c, p in zip(class_names, probs)})

Model Architecture

BiomedCLIP Router

Backbone: ViT-B/16 (Vision Transformer)
Pre-training: BiomedCLIP (PubMed 15M+ biomedical articles)
Classification Head: Linear layer with 7 output classes
Input Size: 224 x 224 pixels

Classification Models

Backbone: BiomedCLIP ViT-B/16 (except Mammography, which uses ResNet50, and Endoscopy, which uses ConvNeXt-Small)
Fine-tuned: Full model (backbone unfrozen)
Augmentation: Test-time augmentation (TTA) enabled

Segmentation Models

Architecture: SMP UNet++ (UNet with nested skip pathways)
Encoder: Pretrained on ImageNet
Loss: Weighted Dice + BCE Loss
Augmentation: Test-time augmentation (TTA) enabled

Intended Use

These models are designed for:

Medical image triage and routing
Automated modality classification
Disease detection and diagnosis support
Medical image segmentation
Medical imaging research

Limitations

Models trained on specific datasets; performance may vary on different populations
Chest X-Ray model shows lower performance due to multi-label complexity and class imbalance
Not a replacement for professional medical diagnosis
Should be validated before clinical deployment
May not generalize well to out-of-distribution images

Training Configuration

All classification models were trained with:

Base Model: hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224
Backbone: Unfrozen (fine-tuned)
Test-Time Augmentation: Enabled

Exception - Mammography: trained with torchvision.resnet50 (IMAGENET1K_V2), a fresh 3-class head, AdamW + OneCycleLR, class-weighted CrossEntropy, on a deduplicated INbreast+MIAS+DDSM (CLAHE) split with leakage-free train/val/test separation. See disease_models/mammography/classification/metrics.json.

All segmentation models were trained with:

Architecture: SMP UNet++
Loss: Combined Dice + BCE with positive class weighting
Test-Time Augmentation: Enabled

Citation

If you use these models, please cite:

@misc{medical-models-2026,
  author = {Umair Inayat},
  title = {Medical Imaging Models: BiomedCLIP Router and Disease-Specific Classifiers},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/umairinayat/medical-models}
}

License

MIT License

Contact

For questions or issues, please open an issue on the Hugging Face repository.

Models Trained in This Work

Models trained and pushed during the current update cycle (each under disease_models/<disease>/{classification,segmentation}/, with weights + metrics.json + training_history.json + plots).

#	Disease / task	Backbone	Result	Status
1	Mammography — Classification (3-class)	ResNet50	99.63% accuracy	pushed
2	Mammography — Segmentation (mass)	SegFormer-B3	90.48% Dice	pushed
3	Chest X-Ray — Classification (4-class)	DenseNet121	96.17% accuracy	pushed
4	Chest X-Ray — Segmentation (lung)	UNet (EfficientNet-B0)	98.71% Dice	pushed
5	Dermatology — Classification (2-class)	ConvNeXt-Small	92.25% accuracy	pushed
6	Dermatology — Segmentation (lesion)	SegFormer-B3	95.25% Dice	pushed
7	Endoscopy — Classification (5-class, healthy added)	ConvNeXt-Small	94.73% accuracy	pushed

Dataset also pushed: umairinayat/fyp-dataset — COVID-19 Radiography (21,165 images + masks).

Trained but not pushed (came out below the existing repo models, so the originals were kept):

Breast Ultrasound classification (74.5%) and segmentation (70% Dice) — BUSI is small (~626 unique images), honest held-out results were below the existing 92.3% / 83.8%.
Endoscopy segmentation — SegFormer-B3 (88.26% Dice) and UNet (86%) were at/below the existing 88.5% on a noisy 150-image test.

Untouched (original/legacy models retained): BiomedCLIP Router, Breast Ultrasound, Thyroid Ultrasound, Fundus Retinography, Microscopy, and Endoscopy segmentation.

Downloads last month: -; Downloads are not tracked for this model. How to track