README.md

metadata

title: NeuroSAM 3
emoji: 🏥
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 6.1.0
app_file: app.py
pinned: true
license: apache-2.0

NeuroSAM 3: Medical Image Segmentation

A comprehensive medical image segmentation application using SAM 3 (Segment Anything Model 3) for DICOM file analysis with advanced features.

✨ Features

Core Features

• 🧠 SAM 3 Integration: Uses the latest Segment Anything Model 3 for medical image segmentation
• 📁 DICOM Support: Process CT and MRI DICOM files
• 🖼️ Image Formats: Supports DICOM (.dcm), PNG, and JPG files
• 🎯 Text Prompts: Describe what you want to segment (e.g., "brain", "tumor", "skull")
• ⚙️ Windowing Strategies: Optimized windowing presets for CT images

Interactive Features

• 📊 Interactive Slice Viewer: Scroll through multiple slices with auto-play
• 🔍 Subject Detection: Automatically groups images by patient/subject ID
• ▶️ Auto-play: Automatically animate through slices
• 📈 Ground Truth Comparison: Compare SAM 3 results with expert annotations (Dice score, IoU)

Enhanced Features (NEW!)

• ✨ Enhanced Single Image: Brightness/contrast controls, colormap options, transparency adjustment
• ✨ Enhanced Batch Processing: Process multiple images with ZIP download
• 🎯 Point/Box Prompts: Interactive point-click and bounding box-based segmentation
• 📊 ROI Statistics: Detailed statistics (area, intensity, centroid, bounding box)
• 📥 NIFTI Export: Export segmentation masks to NIFTI format for medical imaging software
• 💾 Annotation Save/Load: Save and reload annotations for later use
• 🎭 Multi-Mask Output: Generate multiple mask candidates with confidence scores

Usage

Single Image Processing

1. Upload a DICOM (.dcm), PNG, or JPG file
2. Enter a text prompt describing what to segment
3. Select the imaging modality (CT or MRI)
4. Choose the windowing strategy (for CT images)
5. Click "Segment Structure" to process

Interactive Slice Viewer

1. Upload multiple slices/images from the same subject
2. Click "🔍 Detect Subjects" to auto-group by patient ID
3. Select a subject from the dropdown
4. Click "Process All Slices"
5. Use the slider, navigation buttons, or ▶️ Auto-play to scroll through slices

Enhanced Single Image (with Image Adjustments)

1. Upload an image
2. Adjust brightness and contrast sliders
3. Choose colormap and transparency for mask overlay
4. Click "Segment with Progress"
5. Download result using the download button

Point/Box Prompts

1. Upload an image
2. Point Prompt: Enter X, Y coordinates and click "Segment at Point"
3. Box Prompt: Enter bounding box coordinates (X1, Y1, X2, Y2) and click "Segment in Box"

ROI Statistics & Export

1. Process an image using "Process & Calculate Statistics"
2. View detailed ROI statistics (area, intensity, centroid)
3. Export to NIFTI format for use in medical imaging software
4. Save annotation for later use
5. Load previously saved annotations

Multi-Mask Output

1. Upload an image and enter prompt
2. Choose number of masks to generate (1-5)
3. Click "Generate Multiple Masks"
4. Compare mask candidates with confidence scores

Compare with Ground Truth

1. Upload a medical image and its ground truth mask
2. Enter segmentation prompt
3. Click "Compare Segmentation" to see metrics (Dice score, IoU)

Requirements

• Python 3.8+
• PyTorch
• Gradio
• PyDICOM
• Transformers (SAM 3 requires latest version from GitHub)
• nibabel (for NIFTI export)
• scipy (for ROI statistics)

Installation

To install the required dependencies:

pip install -r requirements.txt

Note: SAM 3 is only available in the development version of transformers. The requirements.txt file automatically installs it from the GitHub repository.

Model

This app uses the SAM 3 model from Facebook/Meta. You need:

• A Hugging Face account
• Access token with read permissions for the SAM 3 model (get it from https://huggingface.co/settings/tokens)
• Set HF_TOKEN environment variable in Space settings

Public Datasets

• UniqueData/dicom-brain-dataset (Hugging Face) - MRI Brain scans
• The Cancer Imaging Archive (TCIA) - Various medical imaging
• Imaging Data Commons (IDC) - Large collection of DICOM files
• Kaggle Brain MRI datasets - PNG/JPG format images

Tabs Overview

Tab	Description
Single Image	Basic single image segmentation
Interactive Slice Viewer	Browse through multiple slices with auto-play
Gallery View	View all segmentations in a grid
Compare with Ground Truth	Evaluate with Dice/IoU metrics
✨ Enhanced Single Image	Advanced controls with download
✨ Enhanced Batch Processing	Batch processing with ZIP export
🎯 Point/Box Prompts	Interactive region selection
📊 ROI Statistics & Export	Statistics + NIFTI/annotation export
🎭 Multi-Mask Output	Multiple mask candidates
🔬 Automatic Mask Generator	AMG - segment entire image without prompts
🔧 Advanced Transforms	ResizeLongestSide + CLAHE enhancement
🌊 Edge-Based Segmentation	Sobel edge detection (no AI required)

SAM-Medical-Imaging Utilities

Inspired by amine0110/SAM-Medical-Imaging, we've integrated several utilities:

• Automatic Mask Generator (AMG): Automatically segment all regions in an image using multiple prompts
• ResizeLongestSide Transform: Optimal image resizing that maintains aspect ratio
• CLAHE-like Enhancement: Contrast enhancement for low-visibility images
• Edge-Based Segmentation: Classical computer vision approach using Sobel gradients

License

Apache 2.0