Image Segmentation

Image Segmentation divides an image into segments where each pixel in the image is mapped to an object. This task has multiple variants such as instance segmentation, panoptic segmentation and semantic segmentation.

Image Segmentation Model

About Image Segmentation

Use Cases

Autonomous Driving

Segmentation models are used to identify road patterns such as lanes and obstacles for safer driving.

Background Removal

Image Segmentation models are used in cameras to erase the background of certain objects and apply filters to them.

Medical Imaging

Image Segmentation models are used to distinguish organs or tissues, improving medical imaging workflows. Models are used to segment dental instances, analyze X-Ray scans or even segment cells for pathological diagnosis. This dataset contains images of lungs of healthy patients and patients with COVID-19 segmented with masks. Another segmentation dataset contains segmented MRI data of the lower spine to analyze the effect of spaceflight simulation.

Task Variants

Semantic Segmentation

Semantic Segmentation is the task of segmenting parts of an image that belong to the same class. Semantic Segmentation models make predictions for each pixel and return the probabilities of the classes for each pixel. These models are evaluated on Mean Intersection Over Union (Mean IoU).

Instance Segmentation

Instance Segmentation is the variant of Image Segmentation where every distinct object is segmented, instead of one segment per class.

Panoptic Segmentation

Panoptic Segmentation is the Image Segmentation task that segments the image both by instance and by class, assigning each pixel a different instance of the class.


You can infer with Image Segmentation models using the image-segmentation pipeline. You need to install timm first.

!pip install timm
model = pipeline("image-segmentation")
#[{'label': 'cat',
#  'mask': mask_code,
#  'score': 0.999}
# ...]

Useful Resources

Would you like to learn more about image segmentation? Great! Here you can find some curated resources that you may find helpful!

Compatible libraries

Image Segmentation demo
Image Segmentation
Drag image file here or click to browse from your device
This model can be loaded on the Inference API on-demand.
Models for Image Segmentation Browse Models (44)

Note Solid panoptic segmentation model trained on the COCO 2017 benchmark dataset.

Note Semantic segmentation model trained on ADE20k benchmark dataset.

Metrics for Image Segmentation
Average Precision
Average Precision (AP) is the Area Under the PR Curve (AUC-PR). It is calculated for each semantic class separately
Mean Average Precision
Mean Average Precision (mAP) is the overall average of the AP values
Mean Intersection over Union
Intersection over Union (IoU) is the overlap of segmentation masks. Mean IoU is the average of the IoU of all semantic classes
APα is the Average Precision at the IoU threshold of a α value, for example, AP50 and AP75