MaskFormer
This is a recently introduced model so the API hasn’t been tested extensively. There may be some bugs or slight breaking changes to fix it in the future. If you see something strange, file a Github Issue.
Overview
The MaskFormer model was proposed in Per-Pixel Classification is Not All You Need for Semantic Segmentation by Bowen Cheng, Alexander G. Schwing, Alexander Kirillov. MaskFormer addresses semantic segmentation with a mask classification paradigm instead of performing classic pixel-level classification.
The abstract from the paper is the following:
Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.
The figure below illustrates the architecture of MaskFormer. Taken from the original paper.
This model was contributed by francesco. The original code can be found here.
Usage tips
- MaskFormer’s Transformer decoder is identical to the decoder of DETR. During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help the model output the correct number of objects of each class. If you set the parameter
use_auxiliary_lossof MaskFormerConfig toTrue, then prediction feedforward neural networks and Hungarian losses are added after each decoder layer (with the FFNs sharing parameters). - If you want to train the model in a distributed environment across multiple nodes, then one should update the
get_num_masksfunction inside in theMaskFormerLossclass ofmodeling_maskformer.py. When training on multiple nodes, this should be set to the average number of target masks across all nodes, as can be seen in the original implementation here. - One can use MaskFormerImageProcessor to prepare images for the model and optional targets for the model.
- To get the final segmentation, depending on the task, you can call post_process_semantic_segmentation() or post_process_panoptic_segmentation(). Both tasks can be solved using MaskFormerForInstanceSegmentation output, panoptic segmentation accepts an optional
label_ids_to_fuseargument to fuse instances of the target object/s (e.g. sky) together.
Resources
- All notebooks that illustrate inference as well as fine-tuning on custom data with MaskFormer can be found here.
- Scripts for finetuning
MaskFormerwith Trainer or Accelerate can be found here.
MaskFormer specific outputs
class transformers.models.maskformer.modeling_maskformer.MaskFormerModelOutput
< source >( encoder_last_hidden_state: Optional = None pixel_decoder_last_hidden_state: Optional = None transformer_decoder_last_hidden_state: Optional = None encoder_hidden_states: Optional = None pixel_decoder_hidden_states: Optional = None transformer_decoder_hidden_states: Optional = None hidden_states: Optional = None attentions: Optional = None )
Parameters
- encoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the encoder model (backbone). - pixel_decoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the pixel decoder model (FPN). - transformer_decoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size)) — Last hidden states (final feature map) of the last stage of the transformer decoder model. - encoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder model at the output of each stage. - pixel_decoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel decoder model at the output of each stage. - transformer_decoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the transformer decoder at the output of each stage. - hidden_states
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensorcontainingencoder_hidden_states,pixel_decoder_hidden_statesanddecoder_hidden_states - attentions (
tuple(torch.FloatTensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length). Attentions weights from Detr’s decoder after the attention softmax, used to compute the weighted average in the self-attention heads.
Class for outputs of MaskFormerModel. This class returns all the needed hidden states to compute the logits.
class transformers.models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput
< source >( loss: Optional = None class_queries_logits: FloatTensor = None masks_queries_logits: FloatTensor = None auxiliary_logits: FloatTensor = None encoder_last_hidden_state: Optional = None pixel_decoder_last_hidden_state: Optional = None transformer_decoder_last_hidden_state: Optional = None encoder_hidden_states: Optional = None pixel_decoder_hidden_states: Optional = None transformer_decoder_hidden_states: Optional = None hidden_states: Optional = None attentions: Optional = None )
Parameters
- loss (
torch.Tensor, optional) — The computed loss, returned when labels are present. - class_queries_logits (
torch.FloatTensor) — A tensor of shape(batch_size, num_queries, num_labels + 1)representing the proposed classes for each query. Note the+ 1is needed because we incorporate the null class. - masks_queries_logits (
torch.FloatTensor) — A tensor of shape(batch_size, num_queries, height, width)representing the proposed masks for each query. - encoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the encoder model (backbone). - pixel_decoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the pixel decoder model (FPN). - transformer_decoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size)) — Last hidden states (final feature map) of the last stage of the transformer decoder model. - encoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder model at the output of each stage. - pixel_decoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel decoder model at the output of each stage. - transformer_decoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, sequence_length, hidden_size). Hidden-states of the transformer decoder at the output of each stage. - hidden_states
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensorcontainingencoder_hidden_states,pixel_decoder_hidden_statesanddecoder_hidden_states. - attentions (
tuple(torch.FloatTensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length). Attentions weights from Detr’s decoder after the attention softmax, used to compute the weighted average in the self-attention heads.
Class for outputs of MaskFormerForInstanceSegmentation.
This output can be directly passed to post_process_semantic_segmentation() or or post_process_instance_segmentation() or post_process_panoptic_segmentation() depending on the task. Please, see [`~MaskFormerImageProcessor] for details regarding usage.
MaskFormerConfig
class transformers.MaskFormerConfig
< source >( fpn_feature_size: int = 256 mask_feature_size: int = 256 no_object_weight: float = 0.1 use_auxiliary_loss: bool = False backbone_config: Optional = None decoder_config: Optional = None init_std: float = 0.02 init_xavier_std: float = 1.0 dice_weight: float = 1.0 cross_entropy_weight: float = 1.0 mask_weight: float = 20.0 output_auxiliary_logits: Optional = None backbone: Optional = None use_pretrained_backbone: bool = False use_timm_backbone: bool = False backbone_kwargs: Optional = None **kwargs )
Parameters
- mask_feature_size (
int, optional, defaults to 256) — The masks’ features size, this value will also be used to specify the Feature Pyramid Network features’ size. - no_object_weight (
float, optional, defaults to 0.1) — Weight to apply to the null (no object) class. - use_auxiliary_loss(
bool, optional, defaults toFalse) — IfTrueMaskFormerForInstanceSegmentationOutputwill contain the auxiliary losses computed using the logits from each decoder’s stage. - backbone_config (
Dict, optional) — The configuration passed to the backbone, if unset, the configuration corresponding toswin-base-patch4-window12-384will be used. - backbone (
str, optional) — Name of backbone to use whenbackbone_configisNone. Ifuse_pretrained_backboneisTrue, this will load the corresponding pretrained weights from the timm or transformers library. Ifuse_pretrained_backboneisFalse, this loads the backbone’s config and uses that to initialize the backbone with random weights. - use_pretrained_backbone (
bool, optional,False) — Whether to use pretrained weights for the backbone. - use_timm_backbone (
bool, optional,False) — Whether to loadbackbonefrom the timm library. IfFalse, the backbone is loaded from the transformers library. - backbone_kwargs (
dict, optional) — Keyword arguments to be passed to AutoBackbone when loading from a checkpoint e.g.{'out_indices': (0, 1, 2, 3)}. Cannot be specified ifbackbone_configis set. - decoder_config (
Dict, optional) — The configuration passed to the transformer decoder model, if unset the base config fordetr-resnet-50will be used. - init_std (
float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - init_xavier_std (
float, optional, defaults to 1) — The scaling factor used for the Xavier initialization gain in the HM Attention map module. - dice_weight (
float, optional, defaults to 1.0) — The weight for the dice loss. - cross_entropy_weight (
float, optional, defaults to 1.0) — The weight for the cross entropy loss. - mask_weight (
float, optional, defaults to 20.0) — The weight for the mask loss. - output_auxiliary_logits (
bool, optional) — Should the model output itsauxiliary_logitsor not.
Raises
ValueError
ValueError— Raised if the backbone model type selected is not in["swin"]or the decoder model type selected is not in["detr"]
This is the configuration class to store the configuration of a MaskFormerModel. It is used to instantiate a MaskFormer model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the MaskFormer facebook/maskformer-swin-base-ade architecture trained on ADE20k-150.
Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.
Currently, MaskFormer only supports the Swin Transformer as backbone.
Examples:
>>> from transformers import MaskFormerConfig, MaskFormerModel
>>> # Initializing a MaskFormer facebook/maskformer-swin-base-ade configuration
>>> configuration = MaskFormerConfig()
>>> # Initializing a model (with random weights) from the facebook/maskformer-swin-base-ade style configuration
>>> model = MaskFormerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.configfrom_backbone_and_decoder_configs
< source >( backbone_config: PretrainedConfig decoder_config: PretrainedConfig **kwargs ) → MaskFormerConfig
Parameters
- backbone_config (PretrainedConfig) — The backbone configuration.
- decoder_config (PretrainedConfig) — The transformer decoder configuration to use.
Returns
An instance of a configuration object
Instantiate a MaskFormerConfig (or a derived class) from a pre-trained backbone model configuration and DETR model configuration.
MaskFormerImageProcessor
class transformers.MaskFormerImageProcessor
< source >( do_resize: bool = True size: Dict = None size_divisor: int = 32 resample: Resampling = <Resampling.BILINEAR: 2> do_rescale: bool = True rescale_factor: float = 0.00392156862745098 do_normalize: bool = True image_mean: Union = None image_std: Union = None ignore_index: Optional = None do_reduce_labels: bool = False num_labels: Optional = None **kwargs )
Parameters
- do_resize (
bool, optional, defaults toTrue) — Whether to resize the input to a certainsize. - size (
int, optional, defaults to 800) — Resize the input to the given size. Only has an effect ifdo_resizeis set toTrue. If size is a sequence like(width, height), output size will be matched to this. If size is an int, smaller edge of the image will be matched to this number. i.e, ifheight > width, then image will be rescaled to(size * height / width, size). - size_divisor (
int, optional, defaults to 32) — Some backbones need images divisible by a certain number. If not passed, it defaults to the value used in Swin Transformer. - resample (
int, optional, defaults toResampling.BILINEAR) — An optional resampling filter. This can be one ofPIL.Image.Resampling.NEAREST,PIL.Image.Resampling.BOX,PIL.Image.Resampling.BILINEAR,PIL.Image.Resampling.HAMMING,PIL.Image.Resampling.BICUBICorPIL.Image.Resampling.LANCZOS. Only has an effect ifdo_resizeis set toTrue. - do_rescale (
bool, optional, defaults toTrue) — Whether to rescale the input to a certainscale. - rescale_factor (
float, optional, defaults to1/ 255) — Rescale the input by the given factor. Only has an effect ifdo_rescaleis set toTrue. - do_normalize (
bool, optional, defaults toTrue) — Whether or not to normalize the input with mean and standard deviation. - image_mean (
int, optional, defaults to[0.485, 0.456, 0.406]) — The sequence of means for each channel, to be used when normalizing images. Defaults to the ImageNet mean. - image_std (
int, optional, defaults to[0.229, 0.224, 0.225]) — The sequence of standard deviations for each channel, to be used when normalizing images. Defaults to the ImageNet std. - ignore_index (
int, optional) — Label to be assigned to background pixels in segmentation maps. If provided, segmentation map pixels denoted with 0 (background) will be replaced withignore_index. - do_reduce_labels (
bool, optional, defaults toFalse) — Whether or not to decrement all label values of segmentation maps by 1. Usually used for datasets where 0 is used for background, and background itself is not included in all classes of a dataset (e.g. ADE20k). The background label will be replaced byignore_index. - num_labels (
int, optional) — The number of labels in the segmentation map.
Constructs a MaskFormer image processor. The image processor can be used to prepare image(s) and optional targets for the model.
This image processor inherits from BaseImageProcessor which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.
preprocess
< source >( images: Union segmentation_maps: Union = None instance_id_to_semantic_id: Optional = None do_resize: Optional = None size: Optional = None size_divisor: Optional = None resample: Resampling = None do_rescale: Optional = None rescale_factor: Optional = None do_normalize: Optional = None image_mean: Union = None image_std: Union = None ignore_index: Optional = None do_reduce_labels: Optional = None return_tensors: Union = None data_format: Union = <ChannelDimension.FIRST: 'channels_first'> input_data_format: Union = None )
encode_inputs
< source >( pixel_values_list: List segmentation_maps: Union = None instance_id_to_semantic_id: Union = None ignore_index: Optional = None do_reduce_labels: bool = False return_tensors: Union = None input_data_format: Union = None ) → BatchFeature
Parameters
- pixel_values_list (
List[ImageInput]) — List of images (pixel values) to be padded. Each image should be a tensor of shape(channels, height, width). - segmentation_maps (
ImageInput, optional) — The corresponding semantic segmentation maps with the pixel-wise annotations.(
bool, optional, defaults toTrue): Whether or not to pad images up to the largest image in a batch and create a pixel mask.If left to the default, will return a pixel mask that is:
- 1 for pixels that are real (i.e. not masked),
- 0 for pixels that are padding (i.e. masked).
- instance_id_to_semantic_id (
List[Dict[int, int]]orDict[int, int], optional) — A mapping between object instance ids and class ids. If passed,segmentation_mapsis treated as an instance segmentation map where each pixel represents an instance id. Can be provided as a single dictionary with a global/dataset-level mapping or as a list of dictionaries (one per image), to map instance ids in each image separately. - return_tensors (
stror TensorType, optional) — If set, will return tensors instead of NumPy arrays. If set to'pt', return PyTorchtorch.Tensorobjects.
Returns
A BatchFeature with the following fields:
- pixel_values — Pixel values to be fed to a model.
- pixel_mask — Pixel mask to be fed to a model (when
=Trueor ifpixel_maskis inself.model_input_names). - mask_labels — Optional list of mask labels of shape
(labels, height, width)to be fed to a model (whenannotationsare provided). - class_labels — Optional list of class labels of shape
(labels)to be fed to a model (whenannotationsare provided). They identify the labels ofmask_labels, e.g. the label ofmask_labels[i][j]ifclass_labels[i][j].
Pad images up to the largest image in a batch and create a corresponding pixel_mask.
MaskFormer addresses semantic segmentation with a mask classification paradigm, thus input segmentation maps
will be converted to lists of binary masks and their respective labels. Let’s see an example, assuming
segmentation_maps = [[2,6,7,9]], the output will contain mask_labels = [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]] (four binary masks) and class_labels = [2,6,7,9], the labels for
each mask.
post_process_semantic_segmentation
< source >( outputs target_sizes: Optional = None ) → List[torch.Tensor]
Parameters
- outputs (MaskFormerForInstanceSegmentation) — Raw outputs of the model.
- target_sizes (
List[Tuple[int, int]], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction. If left to None, predictions will not be resized.
Returns
List[torch.Tensor]
A list of length batch_size, where each item is a semantic segmentation map of shape (height, width)
corresponding to the target_sizes entry (if target_sizes is specified). Each entry of each
torch.Tensor correspond to a semantic class id.
Converts the output of MaskFormerForInstanceSegmentation into semantic segmentation maps. Only supports PyTorch.
post_process_instance_segmentation
< source >( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 target_sizes: Optional = None return_coco_annotation: Optional = False return_binary_maps: Optional = False ) → List[Dict]
Parameters
- outputs (MaskFormerForInstanceSegmentation) — Raw outputs of the model.
- threshold (
float, optional, defaults to 0.5) — The probability score threshold to keep predicted instance masks. - mask_threshold (
float, optional, defaults to 0.5) — Threshold to use when turning the predicted masks into binary values. - overlap_mask_area_threshold (
float, optional, defaults to 0.8) — The overlap mask area threshold to merge or discard small disconnected parts within each binary instance mask. - target_sizes (
List[Tuple], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction. If left to None, predictions will not be resized. - return_coco_annotation (
bool, optional, defaults toFalse) — If set toTrue, segmentation maps are returned in COCO run-length encoding (RLE) format. - return_binary_maps (
bool, optional, defaults toFalse) — If set toTrue, segmentation maps are returned as a concatenated tensor of binary segmentation maps (one per detected instance).
Returns
List[Dict]
A list of dictionaries, one per image, each dictionary containing two keys:
- segmentation — A tensor of shape
(height, width)where each pixel represents asegment_idorList[List]run-length encoding (RLE) of the segmentation map if return_coco_annotation is set toTrue. Set toNoneif no mask if found abovethreshold. - segments_info — A dictionary that contains additional information on each segment.
- id — An integer representing the
segment_id. - label_id — An integer representing the label / semantic class id corresponding to
segment_id. - score — Prediction score of segment with
segment_id.
- id — An integer representing the
Converts the output of MaskFormerForInstanceSegmentationOutput into instance segmentation predictions. Only
supports PyTorch.
post_process_panoptic_segmentation
< source >( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 label_ids_to_fuse: Optional = None target_sizes: Optional = None ) → List[Dict]
Parameters
- outputs (
MaskFormerForInstanceSegmentationOutput) — The outputs from MaskFormerForInstanceSegmentation. - threshold (
float, optional, defaults to 0.5) — The probability score threshold to keep predicted instance masks. - mask_threshold (
float, optional, defaults to 0.5) — Threshold to use when turning the predicted masks into binary values. - overlap_mask_area_threshold (
float, optional, defaults to 0.8) — The overlap mask area threshold to merge or discard small disconnected parts within each binary instance mask. - label_ids_to_fuse (
Set[int], optional) — The labels in this state will have all their instances be fused together. For instance we could say there can only be one sky in an image, but several persons, so the label ID for sky would be in that set, but not the one for person. - target_sizes (
List[Tuple], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction in batch. If left to None, predictions will not be resized.
Returns
List[Dict]
A list of dictionaries, one per image, each dictionary containing two keys:
- segmentation — a tensor of shape
(height, width)where each pixel represents asegment_id, set toNoneif no mask if found abovethreshold. Iftarget_sizesis specified, segmentation is resized to the correspondingtarget_sizesentry. - segments_info — A dictionary that contains additional information on each segment.
- id — an integer representing the
segment_id. - label_id — An integer representing the label / semantic class id corresponding to
segment_id. - was_fused — a boolean,
Trueiflabel_idwas inlabel_ids_to_fuse,Falseotherwise. Multiple instances of the same class / label were fused and assigned a singlesegment_id. - score — Prediction score of segment with
segment_id.
- id — an integer representing the
Converts the output of MaskFormerForInstanceSegmentationOutput into image panoptic segmentation
predictions. Only supports PyTorch.
MaskFormerFeatureExtractor
encode_inputs
< source >( pixel_values_list: List segmentation_maps: Union = None instance_id_to_semantic_id: Union = None ignore_index: Optional = None do_reduce_labels: bool = False return_tensors: Union = None input_data_format: Union = None ) → BatchFeature
Parameters
- pixel_values_list (
List[ImageInput]) — List of images (pixel values) to be padded. Each image should be a tensor of shape(channels, height, width). - segmentation_maps (
ImageInput, optional) — The corresponding semantic segmentation maps with the pixel-wise annotations.(
bool, optional, defaults toTrue): Whether or not to pad images up to the largest image in a batch and create a pixel mask.If left to the default, will return a pixel mask that is:
- 1 for pixels that are real (i.e. not masked),
- 0 for pixels that are padding (i.e. masked).
- instance_id_to_semantic_id (
List[Dict[int, int]]orDict[int, int], optional) — A mapping between object instance ids and class ids. If passed,segmentation_mapsis treated as an instance segmentation map where each pixel represents an instance id. Can be provided as a single dictionary with a global/dataset-level mapping or as a list of dictionaries (one per image), to map instance ids in each image separately. - return_tensors (
stror TensorType, optional) — If set, will return tensors instead of NumPy arrays. If set to'pt', return PyTorchtorch.Tensorobjects.
Returns
A BatchFeature with the following fields:
- pixel_values — Pixel values to be fed to a model.
- pixel_mask — Pixel mask to be fed to a model (when
=Trueor ifpixel_maskis inself.model_input_names). - mask_labels — Optional list of mask labels of shape
(labels, height, width)to be fed to a model (whenannotationsare provided). - class_labels — Optional list of class labels of shape
(labels)to be fed to a model (whenannotationsare provided). They identify the labels ofmask_labels, e.g. the label ofmask_labels[i][j]ifclass_labels[i][j].
Pad images up to the largest image in a batch and create a corresponding pixel_mask.
MaskFormer addresses semantic segmentation with a mask classification paradigm, thus input segmentation maps
will be converted to lists of binary masks and their respective labels. Let’s see an example, assuming
segmentation_maps = [[2,6,7,9]], the output will contain mask_labels = [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]] (four binary masks) and class_labels = [2,6,7,9], the labels for
each mask.
post_process_semantic_segmentation
< source >( outputs target_sizes: Optional = None ) → List[torch.Tensor]
Parameters
- outputs (MaskFormerForInstanceSegmentation) — Raw outputs of the model.
- target_sizes (
List[Tuple[int, int]], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction. If left to None, predictions will not be resized.
Returns
List[torch.Tensor]
A list of length batch_size, where each item is a semantic segmentation map of shape (height, width)
corresponding to the target_sizes entry (if target_sizes is specified). Each entry of each
torch.Tensor correspond to a semantic class id.
Converts the output of MaskFormerForInstanceSegmentation into semantic segmentation maps. Only supports PyTorch.
post_process_instance_segmentation
< source >( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 target_sizes: Optional = None return_coco_annotation: Optional = False return_binary_maps: Optional = False ) → List[Dict]
Parameters
- outputs (MaskFormerForInstanceSegmentation) — Raw outputs of the model.
- threshold (
float, optional, defaults to 0.5) — The probability score threshold to keep predicted instance masks. - mask_threshold (
float, optional, defaults to 0.5) — Threshold to use when turning the predicted masks into binary values. - overlap_mask_area_threshold (
float, optional, defaults to 0.8) — The overlap mask area threshold to merge or discard small disconnected parts within each binary instance mask. - target_sizes (
List[Tuple], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction. If left to None, predictions will not be resized. - return_coco_annotation (
bool, optional, defaults toFalse) — If set toTrue, segmentation maps are returned in COCO run-length encoding (RLE) format. - return_binary_maps (
bool, optional, defaults toFalse) — If set toTrue, segmentation maps are returned as a concatenated tensor of binary segmentation maps (one per detected instance).
Returns
List[Dict]
A list of dictionaries, one per image, each dictionary containing two keys:
- segmentation — A tensor of shape
(height, width)where each pixel represents asegment_idorList[List]run-length encoding (RLE) of the segmentation map if return_coco_annotation is set toTrue. Set toNoneif no mask if found abovethreshold. - segments_info — A dictionary that contains additional information on each segment.
- id — An integer representing the
segment_id. - label_id — An integer representing the label / semantic class id corresponding to
segment_id. - score — Prediction score of segment with
segment_id.
- id — An integer representing the
Converts the output of MaskFormerForInstanceSegmentationOutput into instance segmentation predictions. Only
supports PyTorch.
post_process_panoptic_segmentation
< source >( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 label_ids_to_fuse: Optional = None target_sizes: Optional = None ) → List[Dict]
Parameters
- outputs (
MaskFormerForInstanceSegmentationOutput) — The outputs from MaskFormerForInstanceSegmentation. - threshold (
float, optional, defaults to 0.5) — The probability score threshold to keep predicted instance masks. - mask_threshold (
float, optional, defaults to 0.5) — Threshold to use when turning the predicted masks into binary values. - overlap_mask_area_threshold (
float, optional, defaults to 0.8) — The overlap mask area threshold to merge or discard small disconnected parts within each binary instance mask. - label_ids_to_fuse (
Set[int], optional) — The labels in this state will have all their instances be fused together. For instance we could say there can only be one sky in an image, but several persons, so the label ID for sky would be in that set, but not the one for person. - target_sizes (
List[Tuple], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction in batch. If left to None, predictions will not be resized.
Returns
List[Dict]
A list of dictionaries, one per image, each dictionary containing two keys:
- segmentation — a tensor of shape
(height, width)where each pixel represents asegment_id, set toNoneif no mask if found abovethreshold. Iftarget_sizesis specified, segmentation is resized to the correspondingtarget_sizesentry. - segments_info — A dictionary that contains additional information on each segment.
- id — an integer representing the
segment_id. - label_id — An integer representing the label / semantic class id corresponding to
segment_id. - was_fused — a boolean,
Trueiflabel_idwas inlabel_ids_to_fuse,Falseotherwise. Multiple instances of the same class / label were fused and assigned a singlesegment_id. - score — Prediction score of segment with
segment_id.
- id — an integer representing the
Converts the output of MaskFormerForInstanceSegmentationOutput into image panoptic segmentation
predictions. Only supports PyTorch.
MaskFormerModel
class transformers.MaskFormerModel
< source >( config: MaskFormerConfig )
Parameters
- config (MaskFormerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
The bare MaskFormer Model outputting raw hidden-states without any specific head on top. This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( pixel_values: Tensor pixel_mask: Optional = None output_hidden_states: Optional = None output_attentions: Optional = None return_dict: Optional = None ) → transformers.models.maskformer.modeling_maskformer.MaskFormerModelOutput or tuple(torch.FloatTensor)
Parameters
- pixel_values (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See MaskFormerImageProcessor.call() for details. - pixel_mask (
torch.LongTensorof shape(batch_size, height, width), optional) — Mask to avoid performing attention on padding pixel values. Mask values selected in[0, 1]:- 1 for pixels that are real (i.e. not masked),
- 0 for pixels that are padding (i.e. masked).
- output_hidden_states (
bool, optional) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more detail. - output_attentions (
bool, optional) — Whether or not to return the attentions tensors of Detr’s decoder attention layers. - return_dict (
bool, optional) — Whether or not to return a~MaskFormerModelOutputinstead of a plain tuple.
Returns
transformers.models.maskformer.modeling_maskformer.MaskFormerModelOutput or tuple(torch.FloatTensor)
A transformers.models.maskformer.modeling_maskformer.MaskFormerModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MaskFormerConfig) and inputs.
- encoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the encoder model (backbone). - pixel_decoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the pixel decoder model (FPN). - transformer_decoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size)) — Last hidden states (final feature map) of the last stage of the transformer decoder model. - encoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder model at the output of each stage. - pixel_decoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel decoder model at the output of each stage. - transformer_decoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the transformer decoder at the output of each stage. - hidden_states
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensorcontainingencoder_hidden_states,pixel_decoder_hidden_statesanddecoder_hidden_states - attentions (
tuple(torch.FloatTensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length). Attentions weights from Detr’s decoder after the attention softmax, used to compute the weighted average in the self-attention heads.
The MaskFormerModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
>>> from transformers import AutoImageProcessor, MaskFormerModel
>>> from PIL import Image
>>> import requests
>>> # load MaskFormer fine-tuned on ADE20k semantic segmentation
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/maskformer-swin-base-ade")
>>> model = MaskFormerModel.from_pretrained("facebook/maskformer-swin-base-ade")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = image_processor(image, return_tensors="pt")
>>> # forward pass
>>> outputs = model(**inputs)
>>> # the decoder of MaskFormer outputs hidden states of shape (batch_size, num_queries, hidden_size)
>>> transformer_decoder_last_hidden_state = outputs.transformer_decoder_last_hidden_state
>>> list(transformer_decoder_last_hidden_state.shape)
[1, 100, 256]MaskFormerForInstanceSegmentation
forward
< source >( pixel_values: Tensor mask_labels: Optional = None class_labels: Optional = None pixel_mask: Optional = None output_auxiliary_logits: Optional = None output_hidden_states: Optional = None output_attentions: Optional = None return_dict: Optional = None ) → transformers.models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput or tuple(torch.FloatTensor)
Parameters
- pixel_values (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See MaskFormerImageProcessor.call() for details. - pixel_mask (
torch.LongTensorof shape(batch_size, height, width), optional) — Mask to avoid performing attention on padding pixel values. Mask values selected in[0, 1]:- 1 for pixels that are real (i.e. not masked),
- 0 for pixels that are padding (i.e. masked).
- output_hidden_states (
bool, optional) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more detail. - output_attentions (
bool, optional) — Whether or not to return the attentions tensors of Detr’s decoder attention layers. - return_dict (
bool, optional) — Whether or not to return a~MaskFormerModelOutputinstead of a plain tuple. - mask_labels (
List[torch.Tensor], optional) — List of mask labels of shape(num_labels, height, width)to be fed to a model - class_labels (
List[torch.LongTensor], optional) — list of target class labels of shape(num_labels, height, width)to be fed to a model. They identify the labels ofmask_labels, e.g. the label ofmask_labels[i][j]ifclass_labels[i][j].
Returns
transformers.models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput or tuple(torch.FloatTensor)
A transformers.models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MaskFormerConfig) and inputs.
- loss (
torch.Tensor, optional) — The computed loss, returned when labels are present. - class_queries_logits (
torch.FloatTensor) — A tensor of shape(batch_size, num_queries, num_labels + 1)representing the proposed classes for each query. Note the+ 1is needed because we incorporate the null class. - masks_queries_logits (
torch.FloatTensor) — A tensor of shape(batch_size, num_queries, height, width)representing the proposed masks for each query. - encoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the encoder model (backbone). - pixel_decoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the pixel decoder model (FPN). - transformer_decoder_last_hidden_state (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size)) — Last hidden states (final feature map) of the last stage of the transformer decoder model. - encoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder model at the output of each stage. - pixel_decoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel decoder model at the output of each stage. - transformer_decoder_hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings + one for the output of each stage) of shape(batch_size, sequence_length, hidden_size). Hidden-states of the transformer decoder at the output of each stage. - hidden_states
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensorcontainingencoder_hidden_states,pixel_decoder_hidden_statesanddecoder_hidden_states. - attentions (
tuple(torch.FloatTensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length). Attentions weights from Detr’s decoder after the attention softmax, used to compute the weighted average in the self-attention heads.
The MaskFormerForInstanceSegmentation forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
Semantic segmentation example:
>>> from transformers import AutoImageProcessor, MaskFormerForInstanceSegmentation
>>> from PIL import Image
>>> import requests
>>> # load MaskFormer fine-tuned on ADE20k semantic segmentation
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/maskformer-swin-base-ade")
>>> model = MaskFormerForInstanceSegmentation.from_pretrained("facebook/maskformer-swin-base-ade")
>>> url = (
... "https://huggingface.co/datasets/hf-internal-testing/fixtures_ade20k/resolve/main/ADE_val_00000001.jpg"
... )
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> # model predicts class_queries_logits of shape `(batch_size, num_queries)`
>>> # and masks_queries_logits of shape `(batch_size, num_queries, height, width)`
>>> class_queries_logits = outputs.class_queries_logits
>>> masks_queries_logits = outputs.masks_queries_logits
>>> # you can pass them to image_processor for postprocessing
>>> predicted_semantic_map = image_processor.post_process_semantic_segmentation(
... outputs, target_sizes=[image.size[::-1]]
... )[0]
>>> # we refer to the demo notebooks for visualization (see "Resources" section in the MaskFormer docs)
>>> list(predicted_semantic_map.shape)
[512, 683]Panoptic segmentation example:
>>> from transformers import AutoImageProcessor, MaskFormerForInstanceSegmentation
>>> from PIL import Image
>>> import requests
>>> # load MaskFormer fine-tuned on COCO panoptic segmentation
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/maskformer-swin-base-coco")
>>> model = MaskFormerForInstanceSegmentation.from_pretrained("facebook/maskformer-swin-base-coco")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> # model predicts class_queries_logits of shape `(batch_size, num_queries)`
>>> # and masks_queries_logits of shape `(batch_size, num_queries, height, width)`
>>> class_queries_logits = outputs.class_queries_logits
>>> masks_queries_logits = outputs.masks_queries_logits
>>> # you can pass them to image_processor for postprocessing
>>> result = image_processor.post_process_panoptic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
>>> # we refer to the demo notebooks for visualization (see "Resources" section in the MaskFormer docs)
>>> predicted_panoptic_map = result["segmentation"]
>>> list(predicted_panoptic_map.shape)
[480, 640]