init for demo

Change-Id: Iedfddfc377edab70464dd68ba4618336f41d2a2a

GETTING_STARTED.md

@@ -0,0 +1,65 @@
+## Getting Started with Mask2Former
+
+This document provides a brief intro of the usage of Mask2Former.
+
+Please see [Getting Started with Detectron2](https://github.com/facebookresearch/detectron2/blob/master/GETTING_STARTED.md) for full usage.
+
+
+### Inference Demo with Pre-trained Models
+
+1. Pick a model and its config file from
+  [model zoo](MODEL_ZOO.md),
+  for example, `configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml`.
+2. We provide `demo.py` that is able to demo builtin configs. Run it with:
+```
+cd demo/
+python demo.py --config-file ../configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml \
+  --input input1.jpg input2.jpg \
+  [--other-options]
+  --opts MODEL.WEIGHTS /path/to/checkpoint_file
+```
+The configs are made for training, therefore we need to specify `MODEL.WEIGHTS` to a model from model zoo for evaluation.
+This command will run the inference and show visualizations in an OpenCV window.
+
+For details of the command line arguments, see `demo.py -h` or look at its source code
+to understand its behavior. Some common arguments are:
+* To run __on your webcam__, replace `--input files` with `--webcam`.
+* To run __on a video__, replace `--input files` with `--video-input video.mp4`.
+* To run __on cpu__, add `MODEL.DEVICE cpu` after `--opts`.
+* To save outputs to a directory (for images) or a file (for webcam or video), use `--output`.
+
+
+### Training & Evaluation in Command Line
+
+We provide a script `train_net.py`, that is made to train all the configs provided in Mask2Former.
+
+To train a model with "train_net.py", first
+setup the corresponding datasets following
+[datasets/README.md](./datasets/README.md),
+then run:
+```
+python train_net.py --num-gpus 8 \
+  --config-file configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml
+```
+
+The configs are made for 8-GPU training.
+Since we use ADAMW optimizer, it is not clear how to scale learning rate with batch size.
+To train on 1 GPU, you need to figure out learning rate and batch size by yourself:
+```
+python train_net.py \
+  --config-file configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml \
+  --num-gpus 1 SOLVER.IMS_PER_BATCH SET_TO_SOME_REASONABLE_VALUE SOLVER.BASE_LR SET_TO_SOME_REASONABLE_VALUE
+```
+
+To evaluate a model's performance, use
+```
+python train_net.py \
+  --config-file configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml \
+  --eval-only MODEL.WEIGHTS /path/to/checkpoint_file
+```
+For more options, see `python train_net.py -h`.
+
+
+### Video instance segmentation
+Please use `demo_video/demo.py` for video instance segmentation demo and `train_net_video.py` to train
+and evaluate video instance segmentation models.

INSTALL.md

@@ -0,0 +1,48 @@
+## Installation
+
+### Requirements
+- Linux or macOS with Python ≥ 3.6
+- PyTorch ≥ 1.9 and [torchvision](https://github.com/pytorch/vision/) that matches the PyTorch installation.
+  Install them together at [pytorch.org](https://pytorch.org) to make sure of this. Note, please check
+  PyTorch version matches that is required by Detectron2.
+- Detectron2: follow [Detectron2 installation instructions](https://detectron2.readthedocs.io/tutorials/install.html).
+- OpenCV is optional but needed by demo and visualization
+- `pip install -r requirements.txt`
+
+### CUDA kernel for MSDeformAttn
+After preparing the required environment, run the following command to compile CUDA kernel for MSDeformAttn:
+
+`CUDA_HOME` must be defined and points to the directory of the installed CUDA toolkit.
+
+```bash
+cd mask2former/modeling/pixel_decoder/ops
+sh make.sh
+```
+
+#### Building on another system
+To build on a system that does not have a GPU device but provide the drivers:
+```bash
+TORCH_CUDA_ARCH_LIST='8.0' FORCE_CUDA=1 python setup.py build install
+```
+
+### Example conda environment setup
+```bash
+conda create --name mask2former python=3.8 -y
+conda activate mask2former
+conda install pytorch==1.9.0 torchvision==0.10.0 cudatoolkit=11.1 -c pytorch -c nvidia
+pip install -U opencv-python
+
+# under your working directory
+git clone git@github.com:facebookresearch/detectron2.git
+cd detectron2
+pip install -e .
+pip install git+https://github.com/cocodataset/panopticapi.git
+pip install git+https://github.com/mcordts/cityscapesScripts.git
+
+cd ..
+git clone git@github.com:facebookresearch/Mask2Former.git
+cd Mask2Former
+pip install -r requirements.txt
+cd mask2former/modeling/pixel_decoder/ops
+sh make.sh
+```

LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2022 Meta, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

app.py

@@ -0,0 +1,232 @@
+import os 
+import sys
+
+os.system("pip install gdown")
+
+os.system("pip install imutils")
+
+os.system('pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html')
+
+os.system("pip install git+https://github.com/cocodataset/panopticapi.git")
+
+import gradio as gr
+# check pytorch installation: 
+import detectron2
+from detectron2.utils.logger import setup_logger
+from contextlib import ExitStack
+# import some common libraries
+import numpy as np
+import cv2
+import torch
+import itertools
+# import some common detectron2 utilities
+from detectron2 import model_zoo
+from detectron2.config import get_cfg
+from detectron2.utils.visualizer import Visualizer, ColorMode, random_color
+from detectron2.data import MetadataCatalog
+from detectron2.projects.deeplab import add_deeplab_config
+
+
+coco_metadata = MetadataCatalog.get("coco_2017_val_panoptic")
+
+# import FCCLIP project
+from fcclip import add_maskformer2_config, add_fcclip_config
+from demo.predictor import DefaultPredictor, OpenVocabVisualizer
+from PIL import Image
+import imutils
+import json
+
+setup_logger()
+logger = setup_logger(name="fcclip")
+
+cfg = get_cfg()
+cfg.MODEL.DEVICE='cpu'
+add_deeplab_config(cfg)
+add_maskformer2_config(cfg)
+add_fcclip_config(cfg)
+cfg.merge_from_file("configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_ade20k.yaml")
+os.system("gdown 1-91PIns86vyNaL3CzMmDD39zKGnPMtvj")
+cfg.MODEL.WEIGHTS = './fcclip_cocopan.pth'
+cfg.MODEL.KMAX_DEEPLAB.TEST.SEMANTIC_ON = False
+cfg.MODEL.KMAX_DEEPLAB.TEST.INSTANCE_ON = False
+cfg.MODEL.KMAX_DEEPLAB.TEST.PANOPTIC_ON = True
+predictor = DefaultPredictor(cfg)
+
+# def inference(img):
+#     im = cv2.imread(img)
+#     #im = imutils.resize(im, width=512)
+#     outputs = predictor(im)
+#     v = OpenVocabVisualizer(im[:, :, ::-1], coco_metadata, scale=1.2, instance_mode=ColorMode.IMAGE_BW)
+#     panoptic_result = v.draw_panoptic_seg(outputs["panoptic_seg"][0].to("cpu"), outputs["panoptic_seg"][1]).get_image()
+#     return Image.fromarray(np.uint8(panoptic_result)).convert('RGB')
+    
+
+title = "FC-CLIP"
+description = """Gradio demo for FC-CLIP. To use it, simply upload your image, or click one of the examples to load them. FC-CLIP could perform open vocabulary segmentation, you may input more classes (separate by comma).
+The expected format is 'a1,a2;b1,b2', where a1,a2 are synonyms vocabularies for the first class. 
+The first word will be displayed as the class name.Read more at the links below."""
+
+article = "<p style='text-align: center'><a href='https://arxiv.org/abs/2207.04044' target='_blank'>kMaX-DeepLab</a> | <a href='https://github.com/google-research/deeplab2' target='_blank'>Github Repo</a></p>"
+
+examples = [
+    [
+        "demo/examples/coco.jpg",
+        "black pickup truck, pickup truck; blue sky, sky",
+        ["COCO (133 categories)", "ADE (150 categories)", "LVIS (1203 categories)"],
+    ],
+    [
+        "demo/examples/ade.jpg",
+        "luggage, suitcase, baggage;handbag",
+        ["ADE (150 categories)"],
+    ],
+    [
+        "demo/examples/ego4d.jpg",
+        "faucet, tap; kitchen paper, paper towels",
+        ["COCO (133 categories)"],
+    ],
+]
+
+
+coco_metadata = MetadataCatalog.get("openvocab_coco_2017_val_panoptic_with_sem_seg")
+ade20k_metadata = MetadataCatalog.get("openvocab_ade20k_panoptic_val")
+lvis_classes = open("./fcclip/data/datasets/lvis_1203_with_prompt_eng.txt", 'r').read().splitlines()
+lvis_classes = [x[x.find(':')+1:] for x in lvis_classes]
+lvis_colors = list(
+    itertools.islice(itertools.cycle(coco_metadata.stuff_colors), len(lvis_classes))
+)
+# rerrange to thing_classes, stuff_classes
+coco_thing_classes = coco_metadata.thing_classes
+coco_stuff_classes = [x for x in coco_metadata.stuff_classes if x not in coco_thing_classes]
+coco_thing_colors = coco_metadata.thing_colors
+coco_stuff_colors = [x for x in coco_metadata.stuff_colors if x not in coco_thing_colors]
+ade20k_thing_classes = ade20k_metadata.thing_classes
+ade20k_stuff_classes = [x for x in ade20k_metadata.stuff_classes if x not in ade20k_thing_classes]
+ade20k_thing_colors = ade20k_metadata.thing_colors
+ade20k_stuff_colors = [x for x in ade20k_metadata.stuff_colors if x not in ade20k_thing_colors]
+
+def build_demo_classes_and_metadata(vocab, label_list):
+    extra_classes = []
+
+    if vocab:
+        for words in vocab.split(";"):
+            extra_classes.append([word.strip() for word in words.split(",")])
+    extra_colors = [random_color(rgb=True, maximum=1) for _ in range(len(extra_classes))]
+
+    demo_thing_classes = extra_classes
+    demo_stuff_classes = []
+    demo_thing_colors = extra_colors
+    demo_stuff_colors = []
+
+    if any("COCO" in label for label in label_list):
+        demo_thing_classes += coco_thing_classes
+        demo_stuff_classes += coco_stuff_classes
+        demo_thing_colors += coco_thing_colors
+        demo_stuff_colors += coco_stuff_colors
+    if any("ADE" in label for label in label_list):
+        demo_thing_classes += ade20k_thing_classes
+        demo_stuff_classes += ade20k_stuff_classes
+        demo_thing_colors += ade20k_thing_colors
+        demo_stuff_colors += ade20k_stuff_colors
+    if any("LVIS" in label for label in label_list):
+        demo_thing_classes += lvis_classes
+        demo_thing_colors += lvis_colors
+
+    MetadataCatalog.pop("fcclip_demo_metadata", None)
+    demo_metadata = MetadataCatalog.get("fcclip_demo_metadata")
+    demo_metadata.thing_classes = [c[0] for c in demo_thing_classes]
+    demo_metadata.stuff_classes = [
+        *demo_metadata.thing_classes,
+        *[c[0] for c in demo_stuff_classes],
+    ]
+    demo_metadata.thing_colors = demo_thing_colors
+    demo_metadata.stuff_colors = demo_thing_colors + demo_stuff_colors
+    demo_metadata.stuff_dataset_id_to_contiguous_id = {
+        idx: idx for idx in range(len(demo_metadata.stuff_classes))
+    }
+    demo_metadata.thing_dataset_id_to_contiguous_id = {
+        idx: idx for idx in range(len(demo_metadata.thing_classes))
+    }
+
+    demo_classes = demo_thing_classes + demo_stuff_classes
+
+    return demo_classes, demo_metadata
+
+
+def inference(image_path, vocab, label_list):
+
+    logger.info("building class names")
+    demo_classes, demo_metadata = build_demo_classes_and_metadata(vocab, label_list)
+    predictor.set_metadata(demo_metadata)
+
+    im = cv2.imread(image_path)
+    outputs = predictor(im)
+    v = OpenVocabVisualizer(im[:, :, ::-1], demo_metadata, scale=1.2, instance_mode=ColorMode.IMAGE_BW)
+    panoptic_result = v.draw_panoptic_seg(outputs["panoptic_seg"][0].to("cpu"), outputs["panoptic_seg"][1]).get_image()
+    return Image.fromarray(np.uint8(panoptic_result)).convert('RGB')
+
+    
+with gr.Blocks(title=title) as demo:
+    gr.Markdown("<h1 style='text-align: center; margin-bottom: 1rem'>" + title + "</h1>")
+    gr.Markdown(description)
+    input_components = []
+    output_components = []
+
+    with gr.Row():
+        output_image_gr = gr.outputs.Image(label="Panoptic Segmentation", type="pil")
+        output_components.append(output_image_gr)
+
+    with gr.Row().style(equal_height=True, mobile_collapse=True):
+        with gr.Column(scale=3, variant="panel") as input_component_column:
+            input_image_gr = gr.inputs.Image(type="filepath")
+            extra_vocab_gr = gr.inputs.Textbox(default="", label="Extra Vocabulary")
+            category_list_gr = gr.inputs.CheckboxGroup(
+                choices=["COCO (133 categories)", "ADE (150 categories)", "LVIS (1203 categories)"],
+                default=["COCO (133 categories)", "ADE (150 categories)", "LVIS (1203 categories)"],
+                label="Category to use",
+            )
+            input_components.extend([input_image_gr, extra_vocab_gr, category_list_gr])
+
+        with gr.Column(scale=2):
+            examples_handler = gr.Examples(
+                examples=examples,
+                inputs=[c for c in input_components if not isinstance(c, gr.State)],
+                outputs=[c for c in output_components if not isinstance(c, gr.State)],
+                fn=inference,
+                cache_examples=torch.cuda.is_available(),
+                examples_per_page=5,
+            )
+            with gr.Row():
+                clear_btn = gr.Button("Clear")
+                submit_btn = gr.Button("Submit", variant="primary")
+
+    gr.Markdown(article)
+
+    submit_btn.click(
+        inference,
+        input_components,
+        output_components,
+        api_name="predict",
+        scroll_to_output=True,
+    )
+
+    clear_btn.click(
+        None,
+        [],
+        (input_components + output_components + [input_component_column]),
+        _js=f"""() => {json.dumps(
+                    [component.cleared_value if hasattr(component, "cleared_value") else None
+                     for component in input_components + output_components] + (
+                        [gr.Column.update(visible=True)]
+                    )
+                    + ([gr.Column.update(visible=False)])
+                )}
+                """,
+    )
+
+demo.launch()
+
+
+# gr.Interface(inference, inputs=gr.inputs.Image(type="filepath"), outputs=gr.outputs.Image(label="Panoptic segmentation",type="pil"), title=title,
+#     description=description,
+#     article=article,
+#     examples=examples).launch(enable_queue=True)

cog.yaml

@@ -0,0 +1,28 @@
+build:
+  gpu: true
+  cuda: "10.1"
+  python_version: "3.8"
+  system_packages:
+    - "libgl1-mesa-glx"
+    - "libglib2.0-0"
+  python_packages:
+    - "ipython==7.30.1"
+    - "numpy==1.21.4"
+    - "torch==1.8.1"
+    - "torchvision==0.9.1"
+    - "opencv-python==4.5.5.62"
+    - "Shapely==1.8.0"
+    - "h5py==3.6.0"
+    - "scipy==1.7.3"
+    - "submitit==1.4.1"
+    - "scikit-image==0.19.1"
+    - "Cython==0.29.27"
+    - "timm==0.4.12"
+  run:
+    - pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.8/index.html
+    - pip install git+https://github.com/cocodataset/panopticapi.git
+    - pip install git+https://github.com/mcordts/cityscapesScripts.git
+    - git clone https://github.com/facebookresearch/Mask2Former
+    - TORCH_CUDA_ARCH_LIST='7.5' FORCE_CUDA=1 python Mask2Former/mask2former/modeling/pixel_decoder/ops/setup.py build install
+
+predict: "predict.py:Predictor"

configs/coco/panoptic-segmentation/Base-COCO-PanopticSegmentation.yaml

@@ -0,0 +1,47 @@
+MODEL:
+  BACKBONE:
+    FREEZE_AT: 0
+    NAME: "build_resnet_backbone"
+  WEIGHTS: "detectron2://ImageNetPretrained/torchvision/R-50.pkl"
+  PIXEL_MEAN: [123.675, 116.280, 103.530]
+  PIXEL_STD: [58.395, 57.120, 57.375]
+  RESNETS:
+    DEPTH: 50
+    STEM_TYPE: "basic"  # not used
+    STEM_OUT_CHANNELS: 64
+    STRIDE_IN_1X1: False
+    OUT_FEATURES: ["res2", "res3", "res4", "res5"]
+    # NORM: "SyncBN"
+    RES5_MULTI_GRID: [1, 1, 1]  # not used
+DATASETS:
+  TRAIN: ("coco_2017_train_panoptic",)
+  TEST: ("coco_2017_val_panoptic_with_sem_seg",)  # to evaluate instance and semantic performance as well
+SOLVER:
+  IMS_PER_BATCH: 16
+  BASE_LR: 0.0001
+  STEPS: (327778, 355092)
+  MAX_ITER: 368750
+  WARMUP_FACTOR: 1.0
+  WARMUP_ITERS: 10
+  WEIGHT_DECAY: 0.05
+  OPTIMIZER: "ADAMW"
+  BACKBONE_MULTIPLIER: 0.1
+  CLIP_GRADIENTS:
+    ENABLED: True
+    CLIP_TYPE: "full_model"
+    CLIP_VALUE: 0.01
+    NORM_TYPE: 2.0
+  AMP:
+    ENABLED: True
+INPUT:
+  IMAGE_SIZE: 1024
+  MIN_SCALE: 0.1
+  MAX_SCALE: 2.0
+  FORMAT: "RGB"
+  DATASET_MAPPER_NAME: "coco_panoptic_lsj"
+TEST:
+  EVAL_PERIOD: 5000
+DATALOADER:
+  FILTER_EMPTY_ANNOTATIONS: True
+  NUM_WORKERS: 4
+VERSION: 2

configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_a847.yaml

@@ -0,0 +1,10 @@
+_BASE_: ./fcclip_convnext_large_eval_ade20k.yaml
+
+MODEL:
+  MASK_FORMER:
+    TEST:
+      PANOPTIC_ON: False
+      INSTANCE_ON: False
+
+DATASETS:
+  TEST: ("openvocab_ade20k_full_sem_seg_val",)

configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_ade20k.yaml

@@ -0,0 +1,28 @@
+_BASE_: ../maskformer2_R50_bs16_50ep.yaml
+MODEL:
+  META_ARCHITECTURE: "FCCLIP"
+  SEM_SEG_HEAD:
+    NAME: "FCCLIPHead"
+  # backbone part.
+  BACKBONE:
+    NAME: "CLIP"
+  WEIGHTS: ""
+  PIXEL_MEAN: [122.7709383, 116.7460125, 104.09373615]
+  PIXEL_STD: [68.5005327, 66.6321579, 70.32316305]
+  FC_CLIP:
+    CLIP_MODEL_NAME: "convnext_large_d_320"
+    CLIP_PRETRAINED_WEIGHTS: "laion2b_s29b_b131k_ft_soup"
+    EMBED_DIM: 768
+    GEOMETRIC_ENSEMBLE_ALPHA: 0.4
+    GEOMETRIC_ENSEMBLE_BETA: 0.8
+  MASK_FORMER:
+    NUM_OBJECT_QUERIES: 250
+    TEST:
+      SEMANTIC_ON: True
+      INSTANCE_ON: True
+      PANOPTIC_ON: True
+      OBJECT_MASK_THRESHOLD: 0.0
+
+DATASETS:
+  TRAIN: ("openvocab_coco_2017_train_panoptic_with_sem_seg",)
+  TEST: ("openvocab_ade20k_panoptic_val",)

configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_cityscapes.yaml

@@ -0,0 +1,8 @@
+_BASE_: ./fcclip_convnext_large_eval_ade20k.yaml
+
+INPUT:
+  MIN_SIZE_TEST: 1024
+  MAX_SIZE_TEST: 2560
+
+DATASETS:
+  TEST: ("openvocab_cityscapes_fine_panoptic_val",)

configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_coco.yaml

@@ -0,0 +1,3 @@
+_BASE_: ./fcclip_convnext_large_eval_ade20k.yaml
+DATASETS:
+  TEST: ("openvocab_coco_2017_val_panoptic_with_sem_seg",)

configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_mapillary_vistas.yaml

@@ -0,0 +1,12 @@
+_BASE_: ./fcclip_convnext_large_eval_ade20k.yaml
+
+MODEL:
+  MASK_FORMER:
+    TEST:
+      INSTANCE_ON: False
+INPUT:
+  MIN_SIZE_TEST: 1024
+  MAX_SIZE_TEST: 2560
+
+DATASETS:
+  TEST: ("openvocab_mapillary_vistas_panoptic_val",)

configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_pas20.yaml

@@ -0,0 +1,10 @@
+_BASE_: ./fcclip_convnext_large_eval_ade20k.yaml
+
+MODEL:
+  MASK_FORMER:
+    TEST:
+      PANOPTIC_ON: False
+      INSTANCE_ON: False
+
+DATASETS:
+  TEST: ("openvocab_pascal20_sem_seg_val",)

configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_pas21.yaml

@@ -0,0 +1,10 @@
+_BASE_: ./fcclip_convnext_large_eval_ade20k.yaml
+
+MODEL:
+  MASK_FORMER:
+    TEST:
+      PANOPTIC_ON: False
+      INSTANCE_ON: False
+
+DATASETS:
+  TEST: ("openvocab_pascal21_sem_seg_val",)

configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_pc459.yaml

@@ -0,0 +1,10 @@
+_BASE_: ./fcclip_convnext_large_eval_ade20k.yaml
+
+MODEL:
+  MASK_FORMER:
+    TEST:
+      PANOPTIC_ON: False
+      INSTANCE_ON: False
+
+DATASETS:
+  TEST: ("openvocab_pascal_ctx459_sem_seg_val",)

configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_pc59.yaml

@@ -0,0 +1,10 @@
+_BASE_: ./fcclip_convnext_large_eval_ade20k.yaml
+
+MODEL:
+  MASK_FORMER:
+    TEST:
+      PANOPTIC_ON: False
+      INSTANCE_ON: False
+
+DATASETS:
+  TEST: ("openvocab_pascal_ctx59_sem_seg_val",)

configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml

@@ -0,0 +1,45 @@
+_BASE_: Base-COCO-PanopticSegmentation.yaml
+MODEL:
+  META_ARCHITECTURE: "MaskFormer"
+  SEM_SEG_HEAD:
+    NAME: "MaskFormerHead"
+    IN_FEATURES: ["res2", "res3", "res4", "res5"]
+    IGNORE_VALUE: 255
+    NUM_CLASSES: 133
+    LOSS_WEIGHT: 1.0
+    CONVS_DIM: 256
+    MASK_DIM: 256
+    NORM: "GN"
+    # pixel decoder
+    PIXEL_DECODER_NAME: "MSDeformAttnPixelDecoder"
+    IN_FEATURES: ["res2", "res3", "res4", "res5"]
+    DEFORMABLE_TRANSFORMER_ENCODER_IN_FEATURES: ["res3", "res4", "res5"]
+    COMMON_STRIDE: 4
+    TRANSFORMER_ENC_LAYERS: 6
+  MASK_FORMER:
+    TRANSFORMER_DECODER_NAME: "MultiScaleMaskedTransformerDecoder"
+    TRANSFORMER_IN_FEATURE: "multi_scale_pixel_decoder"
+    DEEP_SUPERVISION: True
+    NO_OBJECT_WEIGHT: 0.1
+    CLASS_WEIGHT: 2.0
+    MASK_WEIGHT: 5.0
+    DICE_WEIGHT: 5.0
+    HIDDEN_DIM: 256
+    NUM_OBJECT_QUERIES: 100
+    NHEADS: 8
+    DROPOUT: 0.0
+    DIM_FEEDFORWARD: 2048
+    ENC_LAYERS: 0
+    PRE_NORM: False
+    ENFORCE_INPUT_PROJ: False
+    SIZE_DIVISIBILITY: 32
+    DEC_LAYERS: 10  # 9 decoder layers, add one for the loss on learnable query
+    TRAIN_NUM_POINTS: 12544
+    OVERSAMPLE_RATIO: 3.0
+    IMPORTANCE_SAMPLE_RATIO: 0.75
+    TEST:
+      SEMANTIC_ON: True
+      INSTANCE_ON: True
+      PANOPTIC_ON: True
+      OVERLAP_THRESHOLD: 0.8
+      OBJECT_MASK_THRESHOLD: 0.8

datasets/README.md

@@ -0,0 +1,135 @@
+# Prepare Datasets for FCCLIP
+
+A dataset can be used by accessing [DatasetCatalog](https://detectron2.readthedocs.io/modules/data.html#detectron2.data.DatasetCatalog)
+for its data, or [MetadataCatalog](https://detectron2.readthedocs.io/modules/data.html#detectron2.data.MetadataCatalog) for its metadata (class names, etc).
+This document explains how to setup the builtin datasets so they can be used by the above APIs.
+[Use Custom Datasets](https://detectron2.readthedocs.io/tutorials/datasets.html) gives a deeper dive on how to use `DatasetCatalog` and `MetadataCatalog`,
+and how to add new datasets to them.
+
+FCCLIP has builtin support for a few datasets.
+The datasets are assumed to exist in a directory specified by the environment variable
+`DETECTRON2_DATASETS`.
+Under this directory, detectron2 will look for datasets in the structure described below, if needed.
+```
+$DETECTRON2_DATASETS/
+  ADEChallengeData2016/
+  coco/
+  cityscapes/
+  mapillary_vistas/
+```
+
+You can set the location for builtin datasets by `export DETECTRON2_DATASETS=/path/to/datasets`.
+If left unset, the default is `./datasets` relative to your current working directory.
+
+
+## Expected dataset structure for [COCO](https://cocodataset.org/#download):
+
+```
+coco/
+  annotations/
+    instances_{train,val}2017.json
+    panoptic_{train,val}2017.json
+  {train,val}2017/
+    # image files that are mentioned in the corresponding json
+  panoptic_{train,val}2017/  # png annotations
+  panoptic_semseg_{train,val}2017/  # generated by the script mentioned below
+```
+
+Install panopticapi by:
+```
+pip install git+https://github.com/cocodataset/panopticapi.git
+```
+Then, run `python datasets/prepare_coco_semantic_annos_from_panoptic_annos.py`, to extract semantic annotations from panoptic annotations (only used for evaluation).
+
+
+## Expected dataset structure for [cityscapes](https://www.cityscapes-dataset.com/downloads/):
+```
+cityscapes/
+  gtFine/
+    train/
+      aachen/
+        color.png, instanceIds.png, labelIds.png, polygons.json,
+        labelTrainIds.png
+      ...
+    val/
+    test/
+    # below are generated Cityscapes panoptic annotation
+    cityscapes_panoptic_train.json
+    cityscapes_panoptic_train/
+    cityscapes_panoptic_val.json
+    cityscapes_panoptic_val/
+    cityscapes_panoptic_test.json
+    cityscapes_panoptic_test/
+  leftImg8bit/
+    train/
+    val/
+    test/
+```
+Install cityscapes scripts by:
+```
+pip install git+https://github.com/mcordts/cityscapesScripts.git
+```
+
+Note: to create labelTrainIds.png, first prepare the above structure, then run cityscapesescript with:
+```
+CITYSCAPES_DATASET=/path/to/abovementioned/cityscapes python cityscapesscripts/preparation/createTrainIdLabelImgs.py
+```
+These files are not needed for instance segmentation.
+
+Note: to generate Cityscapes panoptic dataset, run cityscapesescript with:
+```
+CITYSCAPES_DATASET=/path/to/abovementioned/cityscapes python cityscapesscripts/preparation/createPanopticImgs.py
+```
+These files are not needed for semantic and instance segmentation.
+
+
+## Expected dataset structure for [ADE20k](http://sceneparsing.csail.mit.edu/):
+```
+ADEChallengeData2016/
+  images/
+  annotations/
+  objectInfo150.txt
+  # download instance annotation
+  annotations_instance/
+  # generated by prepare_ade20k_sem_seg.py
+  annotations_detectron2/
+  # below are generated by prepare_ade20k_pan_seg.py
+  ade20k_panoptic_{train,val}.json
+  ade20k_panoptic_{train,val}/
+  # below are generated by prepare_ade20k_ins_seg.py
+  ade20k_instance_{train,val}.json
+```
+
+The directory `annotations_detectron2` is generated by running `python datasets/prepare_ade20k_sem_seg.py`.
+
+Install panopticapi by:
+```bash
+pip install git+https://github.com/cocodataset/panopticapi.git
+```
+
+Download the instance annotation from http://sceneparsing.csail.mit.edu/:
+```bash
+wget http://sceneparsing.csail.mit.edu/data/ChallengeData2017/annotations_instance.tar
+```
+
+Then, run `python datasets/prepare_ade20k_pan_seg.py`, to combine semantic and instance annotations for panoptic annotations.
+
+And run `python datasets/prepare_ade20k_ins_seg.py`, to extract instance annotations in COCO format.
+
+
+## Expected dataset structure for [Mapillary Vistas](https://www.mapillary.com/dataset/vistas):
+```
+mapillary_vistas/
+  training/
+    images/
+    instances/
+    labels/
+    panoptic/
+  validation/
+    images/
+    instances/
+    labels/
+    panoptic/
+```
+
+No preprocessing is needed for Mapillary Vistas on semantic and panoptic segmentation.

datasets/ade20k_instance_catid_mapping.txt

@@ -0,0 +1,104 @@
+Instacne100	SceneParse150	FullADE20K
+1		8		165
+2		9		3055
+3		11		350
+4		13		1831
+5		15		774
+5		15		783
+6		16		2684
+7		19		687
+8		20		471
+9		21		401
+10		23		1735
+11		24		2473
+12		25		2329
+13		28		1564
+14		31		57
+15		32		2272
+16		33		907
+17		34		724
+18		36		2985
+18		36		533
+19		37		1395
+20		38		155
+21		39		2053
+22		40		689
+23		42		266
+24		43		581
+25		44		2380
+26		45		491
+27		46		627
+28		48		2388
+29		50		943
+30		51		2096
+31		54		2530
+32		56		420
+33		57		1948
+34		58		1869
+35		59		2251
+36		63		239
+37		65		571
+38		66		2793
+39		67		978
+40		68		236
+41		70		181
+42		71		629
+43		72		2598
+44		73		1744
+45		74		1374
+46		75		591
+47		76		2679
+48		77		223
+49		79		47
+50		81		327
+51		82		2821
+52		83		1451
+53		84		2880
+54		86		480
+55		87		77
+56		88		2616
+57		89		246
+57		89		247
+58		90		2733
+59		91		14
+60		93		38
+61		94		1936
+62		96		120
+63		98		1702
+64		99		249
+65		103		2928
+66		104		2337
+67		105		1023
+68		108		2989
+69		109		1930
+70		111		2586
+71		112		131
+72		113		146
+73		116		95
+74		117		1563
+75		119		1708
+76		120		103
+77		121		1002
+78		122		2569
+79		124		2833
+80		125		1551
+81		126		1981
+82		127		29
+83		128		187
+84		130		747
+85		131		2254
+86		133		2262
+87		134		1260
+88		135		2243
+89		136		2932
+90		137		2836
+91		138		2850
+92		139		64
+93		140		894
+94		143		1919
+95		144		1583
+96		145		318
+97		147		2046
+98		148		1098
+99		149		530
+100		150		954

datasets/prepare_ade20k_ins_seg.py

@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Copyright (c) Facebook, Inc. and its affiliates.
+import glob
+import json
+import os
+from collections import Counter
+
+import numpy as np
+import tqdm
+from panopticapi.utils import IdGenerator, save_json
+from PIL import Image
+import pycocotools.mask as mask_util
+
+
+if __name__ == "__main__":
+    dataset_dir = os.getenv("DETECTRON2_DATASETS", "datasets")
+
+    for name, dirname in [("train", "training"), ("val", "validation")]:
+        image_dir = os.path.join(dataset_dir, f"ADEChallengeData2016/images/{dirname}/")
+        instance_dir = os.path.join(
+            dataset_dir, f"ADEChallengeData2016/annotations_instance/{dirname}/"
+        )
+
+        # img_id = 0
+        ann_id = 1
+
+        # json
+        out_file = os.path.join(dataset_dir, f"ADEChallengeData2016/ade20k_instance_{name}.json")
+
+        # json config
+        instance_config_file = "datasets/ade20k_instance_imgCatIds.json"
+        with open(instance_config_file) as f:
+            category_dict = json.load(f)["categories"]
+
+        # load catid mapping
+        # it is important to share category id for both instance and panoptic annotations
+        mapping_file = "datasets/ade20k_instance_catid_mapping.txt"
+        with open(mapping_file) as f:
+            map_id = {}
+            for i, line in enumerate(f.readlines()):
+                if i == 0:
+                    continue
+                ins_id, sem_id, _ = line.strip().split()
+                # shift id by 1 because we want it to start from 0!
+                # ignore_label becomes 255
+                map_id[int(ins_id)] = int(sem_id) - 1
+
+        for cat in category_dict:
+            cat["id"] = map_id[cat["id"]]
+
+        filenames = sorted(glob.glob(os.path.join(image_dir, "*.jpg")))
+
+        ann_dict = {}
+        images = []
+        annotations = []
+
+        for idx, filename in enumerate(tqdm.tqdm(filenames)):
+            image = {}
+            image_id = os.path.basename(filename).split(".")[0]
+
+            image["id"] = image_id
+            image["file_name"] = os.path.basename(filename)
+
+            original_format = np.array(Image.open(filename))
+            image["width"] = original_format.shape[1]
+            image["height"] = original_format.shape[0]
+
+            images.append(image)
+
+            filename_instance = os.path.join(instance_dir, image_id + ".png")
+            ins_seg = np.asarray(Image.open(filename_instance))
+            assert ins_seg.dtype == np.uint8
+
+            instance_cat_ids = ins_seg[..., 0]
+            # instance id starts from 1!
+            # because 0 is reserved as VOID label
+            instance_ins_ids = ins_seg[..., 1]
+
+            # process things
+            for thing_id in np.unique(instance_ins_ids):
+                if thing_id == 0:
+                    continue
+                mask = instance_ins_ids == thing_id
+                instance_cat_id = np.unique(instance_cat_ids[mask])
+                assert len(instance_cat_id) == 1
+
+                anno = {}
+                anno['id'] = ann_id
+                ann_id += 1
+                anno['image_id'] = image['id']
+                anno["iscrowd"] = int(0)
+                anno["category_id"] = int(map_id[instance_cat_id[0]])
+
+                inds = np.nonzero(mask)
+                ymin, ymax = inds[0].min(), inds[0].max()
+                xmin, xmax = inds[1].min(), inds[1].max()
+                anno["bbox"] = [int(xmin), int(ymin), int(xmax - xmin + 1), int(ymax - ymin + 1)]
+                # if xmax <= xmin or ymax <= ymin:
+                #     continue
+                rle = mask_util.encode(np.array(mask[:, :, None], order="F", dtype="uint8"))[0]
+                rle["counts"] = rle["counts"].decode("utf-8")
+                anno["segmentation"] = rle
+                anno["area"] = int(mask_util.area(rle))
+                annotations.append(anno)
+
+        # save this
+        ann_dict['images'] = images
+        ann_dict['categories'] = category_dict
+        ann_dict['annotations'] = annotations
+
+        save_json(ann_dict, out_file)

datasets/prepare_ade20k_pan_seg.py

@@ -0,0 +1,500 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Copyright (c) Facebook, Inc. and its affiliates.
+import glob
+import json
+import os
+from collections import Counter
+
+import numpy as np
+import tqdm
+from panopticapi.utils import IdGenerator, save_json
+from PIL import Image
+
+ADE20K_SEM_SEG_CATEGORIES = [
+    "wall",
+    "building",
+    "sky",
+    "floor",
+    "tree",
+    "ceiling",
+    "road, route",
+    "bed",
+    "window ",
+    "grass",
+    "cabinet",
+    "sidewalk, pavement",
+    "person",
+    "earth, ground",
+    "door",
+    "table",
+    "mountain, mount",
+    "plant",
+    "curtain",
+    "chair",
+    "car",
+    "water",
+    "painting, picture",
+    "sofa",
+    "shelf",
+    "house",
+    "sea",
+    "mirror",
+    "rug",
+    "field",
+    "armchair",
+    "seat",
+    "fence",
+    "desk",
+    "rock, stone",
+    "wardrobe, closet, press",
+    "lamp",
+    "tub",
+    "rail",
+    "cushion",
+    "base, pedestal, stand",
+    "box",
+    "column, pillar",
+    "signboard, sign",
+    "chest of drawers, chest, bureau, dresser",
+    "counter",
+    "sand",
+    "sink",
+    "skyscraper",
+    "fireplace",
+    "refrigerator, icebox",
+    "grandstand, covered stand",
+    "path",
+    "stairs",
+    "runway",
+    "case, display case, showcase, vitrine",
+    "pool table, billiard table, snooker table",
+    "pillow",
+    "screen door, screen",
+    "stairway, staircase",
+    "river",
+    "bridge, span",
+    "bookcase",
+    "blind, screen",
+    "coffee table",
+    "toilet, can, commode, crapper, pot, potty, stool, throne",
+    "flower",
+    "book",
+    "hill",
+    "bench",
+    "countertop",
+    "stove",
+    "palm, palm tree",
+    "kitchen island",
+    "computer",
+    "swivel chair",
+    "boat",
+    "bar",
+    "arcade machine",
+    "hovel, hut, hutch, shack, shanty",
+    "bus",
+    "towel",
+    "light",
+    "truck",
+    "tower",
+    "chandelier",
+    "awning, sunshade, sunblind",
+    "street lamp",
+    "booth",
+    "tv",
+    "plane",
+    "dirt track",
+    "clothes",
+    "pole",
+    "land, ground, soil",
+    "bannister, banister, balustrade, balusters, handrail",
+    "escalator, moving staircase, moving stairway",
+    "ottoman, pouf, pouffe, puff, hassock",
+    "bottle",
+    "buffet, counter, sideboard",
+    "poster, posting, placard, notice, bill, card",
+    "stage",
+    "van",
+    "ship",
+    "fountain",
+    "conveyer belt, conveyor belt, conveyer, conveyor, transporter",
+    "canopy",
+    "washer, automatic washer, washing machine",
+    "plaything, toy",
+    "pool",
+    "stool",
+    "barrel, cask",
+    "basket, handbasket",
+    "falls",
+    "tent",
+    "bag",
+    "minibike, motorbike",
+    "cradle",
+    "oven",
+    "ball",
+    "food, solid food",
+    "step, stair",
+    "tank, storage tank",
+    "trade name",
+    "microwave",
+    "pot",
+    "animal",
+    "bicycle",
+    "lake",
+    "dishwasher",
+    "screen",
+    "blanket, cover",
+    "sculpture",
+    "hood, exhaust hood",
+    "sconce",
+    "vase",
+    "traffic light",
+    "tray",
+    "trash can",
+    "fan",
+    "pier",
+    "crt screen",
+    "plate",
+    "monitor",
+    "bulletin board",
+    "shower",
+    "radiator",
+    "glass, drinking glass",
+    "clock",
+    "flag",  # noqa
+]
+
+PALETTE = [
+    [120, 120, 120],
+    [180, 120, 120],
+    [6, 230, 230],
+    [80, 50, 50],
+    [4, 200, 3],
+    [120, 120, 80],
+    [140, 140, 140],
+    [204, 5, 255],
+    [230, 230, 230],
+    [4, 250, 7],
+    [224, 5, 255],
+    [235, 255, 7],
+    [150, 5, 61],
+    [120, 120, 70],
+    [8, 255, 51],
+    [255, 6, 82],
+    [143, 255, 140],
+    [204, 255, 4],
+    [255, 51, 7],
+    [204, 70, 3],
+    [0, 102, 200],
+    [61, 230, 250],
+    [255, 6, 51],
+    [11, 102, 255],
+    [255, 7, 71],
+    [255, 9, 224],
+    [9, 7, 230],
+    [220, 220, 220],
+    [255, 9, 92],
+    [112, 9, 255],
+    [8, 255, 214],
+    [7, 255, 224],
+    [255, 184, 6],
+    [10, 255, 71],
+    [255, 41, 10],
+    [7, 255, 255],
+    [224, 255, 8],
+    [102, 8, 255],
+    [255, 61, 6],
+    [255, 194, 7],
+    [255, 122, 8],
+    [0, 255, 20],
+    [255, 8, 41],
+    [255, 5, 153],
+    [6, 51, 255],
+    [235, 12, 255],
+    [160, 150, 20],
+    [0, 163, 255],
+    [140, 140, 200],
+    [250, 10, 15],
+    [20, 255, 0],
+    [31, 255, 0],
+    [255, 31, 0],
+    [255, 224, 0],
+    [153, 255, 0],
+    [0, 0, 255],
+    [255, 71, 0],
+    [0, 235, 255],
+    [0, 173, 255],
+    [31, 0, 255],
+    [11, 200, 200],
+    [255, 82, 0],
+    [0, 255, 245],
+    [0, 61, 255],
+    [0, 255, 112],
+    [0, 255, 133],
+    [255, 0, 0],
+    [255, 163, 0],
+    [255, 102, 0],
+    [194, 255, 0],
+    [0, 143, 255],
+    [51, 255, 0],
+    [0, 82, 255],
+    [0, 255, 41],
+    [0, 255, 173],
+    [10, 0, 255],
+    [173, 255, 0],
+    [0, 255, 153],
+    [255, 92, 0],
+    [255, 0, 255],
+    [255, 0, 245],
+    [255, 0, 102],
+    [255, 173, 0],
+    [255, 0, 20],
+    [255, 184, 184],
+    [0, 31, 255],
+    [0, 255, 61],
+    [0, 71, 255],
+    [255, 0, 204],
+    [0, 255, 194],
+    [0, 255, 82],
+    [0, 10, 255],
+    [0, 112, 255],
+    [51, 0, 255],
+    [0, 194, 255],
+    [0, 122, 255],
+    [0, 255, 163],
+    [255, 153, 0],
+    [0, 255, 10],
+    [255, 112, 0],
+    [143, 255, 0],
+    [82, 0, 255],
+    [163, 255, 0],
+    [255, 235, 0],
+    [8, 184, 170],
+    [133, 0, 255],
+    [0, 255, 92],
+    [184, 0, 255],
+    [255, 0, 31],
+    [0, 184, 255],
+    [0, 214, 255],
+    [255, 0, 112],
+    [92, 255, 0],
+    [0, 224, 255],
+    [112, 224, 255],
+    [70, 184, 160],
+    [163, 0, 255],
+    [153, 0, 255],
+    [71, 255, 0],
+    [255, 0, 163],
+    [255, 204, 0],
+    [255, 0, 143],
+    [0, 255, 235],
+    [133, 255, 0],
+    [255, 0, 235],
+    [245, 0, 255],
+    [255, 0, 122],
+    [255, 245, 0],
+    [10, 190, 212],
+    [214, 255, 0],
+    [0, 204, 255],
+    [20, 0, 255],
+    [255, 255, 0],
+    [0, 153, 255],
+    [0, 41, 255],
+    [0, 255, 204],
+    [41, 0, 255],
+    [41, 255, 0],
+    [173, 0, 255],
+    [0, 245, 255],
+    [71, 0, 255],
+    [122, 0, 255],
+    [0, 255, 184],
+    [0, 92, 255],
+    [184, 255, 0],
+    [0, 133, 255],
+    [255, 214, 0],
+    [25, 194, 194],
+    [102, 255, 0],
+    [92, 0, 255],
+]
+
+
+if __name__ == "__main__":
+    dataset_dir = os.getenv("DETECTRON2_DATASETS", "datasets")
+
+    for name, dirname in [("train", "training"), ("val", "validation")]:
+        image_dir = os.path.join(dataset_dir, f"ADEChallengeData2016/images/{dirname}/")
+        semantic_dir = os.path.join(dataset_dir, f"ADEChallengeData2016/annotations/{dirname}/")
+        instance_dir = os.path.join(
+            dataset_dir, f"ADEChallengeData2016/annotations_instance/{dirname}/"
+        )
+
+        # folder to store panoptic PNGs
+        out_folder = os.path.join(dataset_dir, f"ADEChallengeData2016/ade20k_panoptic_{name}/")
+        # json with segmentations information
+        out_file = os.path.join(dataset_dir, f"ADEChallengeData2016/ade20k_panoptic_{name}.json")
+
+        if not os.path.isdir(out_folder):
+            print("Creating folder {} for panoptic segmentation PNGs".format(out_folder))
+            os.mkdir(out_folder)
+
+        # json config
+        config_file = "datasets/ade20k_instance_imgCatIds.json"
+        with open(config_file) as f:
+            config = json.load(f)
+
+        # load catid mapping
+        mapping_file = "datasets/ade20k_instance_catid_mapping.txt"
+        with open(mapping_file) as f:
+            map_id = {}
+            for i, line in enumerate(f.readlines()):
+                if i == 0:
+                    continue
+                ins_id, sem_id, _ = line.strip().split()
+                # shift id by 1 because we want it to start from 0!
+                # ignore_label becomes 255
+                map_id[int(ins_id) - 1] = int(sem_id) - 1
+
+        ADE20K_150_CATEGORIES = []
+        for cat_id, cat_name in enumerate(ADE20K_SEM_SEG_CATEGORIES):
+            ADE20K_150_CATEGORIES.append(
+                {
+                    "name": cat_name,
+                    "id": cat_id,
+                    "isthing": int(cat_id in map_id.values()),
+                    "color": PALETTE[cat_id],
+                }
+            )
+        categories_dict = {cat["id"]: cat for cat in ADE20K_150_CATEGORIES}
+
+        panoptic_json_categories = ADE20K_150_CATEGORIES[:]
+        panoptic_json_images = []
+        panoptic_json_annotations = []
+
+        filenames = sorted(glob.glob(os.path.join(image_dir, "*.jpg")))
+        for idx, filename in enumerate(tqdm.tqdm(filenames)):
+            panoptic_json_image = {}
+            panoptic_json_annotation = {}
+
+            image_id = os.path.basename(filename).split(".")[0]
+
+            panoptic_json_image["id"] = image_id
+            panoptic_json_image["file_name"] = os.path.basename(filename)
+
+            original_format = np.array(Image.open(filename))
+            panoptic_json_image["width"] = original_format.shape[1]
+            panoptic_json_image["height"] = original_format.shape[0]
+
+            pan_seg = np.zeros(
+                (original_format.shape[0], original_format.shape[1], 3), dtype=np.uint8
+            )
+            id_generator = IdGenerator(categories_dict)
+
+            filename_semantic = os.path.join(semantic_dir, image_id + ".png")
+            filename_instance = os.path.join(instance_dir, image_id + ".png")
+
+            sem_seg = np.asarray(Image.open(filename_semantic))
+            ins_seg = np.asarray(Image.open(filename_instance))
+
+            assert sem_seg.dtype == np.uint8
+            assert ins_seg.dtype == np.uint8
+
+            semantic_cat_ids = sem_seg - 1
+            instance_cat_ids = ins_seg[..., 0] - 1
+            # instance id starts from 1!
+            # because 0 is reserved as VOID label
+            instance_ins_ids = ins_seg[..., 1]
+
+            segm_info = []
+
+            # NOTE: there is some overlap between semantic and instance annotation
+            # thus we paste stuffs first
+
+            # process stuffs
+            for semantic_cat_id in np.unique(semantic_cat_ids):
+                if semantic_cat_id == 255:
+                    continue
+                if categories_dict[semantic_cat_id]["isthing"]:
+                    continue
+                mask = semantic_cat_ids == semantic_cat_id
+                # should not have any overlap
+                assert pan_seg[mask].sum() == 0
+
+                segment_id, color = id_generator.get_id_and_color(semantic_cat_id)
+                pan_seg[mask] = color
+
+                area = np.sum(mask)  # segment area computation
+                # bbox computation for a segment
+                hor = np.sum(mask, axis=0)
+                hor_idx = np.nonzero(hor)[0]
+                x = hor_idx[0]
+                width = hor_idx[-1] - x + 1
+                vert = np.sum(mask, axis=1)
+                vert_idx = np.nonzero(vert)[0]
+                y = vert_idx[0]
+                height = vert_idx[-1] - y + 1
+                bbox = [int(x), int(y), int(width), int(height)]
+
+                segm_info.append(
+                    {
+                        "id": int(segment_id),
+                        "category_id": int(semantic_cat_id),
+                        "area": int(area),
+                        "bbox": bbox,
+                        "iscrowd": 0,
+                    }
+                )
+
+            # process things
+            for thing_id in np.unique(instance_ins_ids):
+                if thing_id == 0:
+                    continue
+                mask = instance_ins_ids == thing_id
+                instance_cat_id = np.unique(instance_cat_ids[mask])
+                assert len(instance_cat_id) == 1
+
+                semantic_cat_id = map_id[instance_cat_id[0]]
+
+                segment_id, color = id_generator.get_id_and_color(semantic_cat_id)
+                pan_seg[mask] = color
+
+                area = np.sum(mask)  # segment area computation
+                # bbox computation for a segment
+                hor = np.sum(mask, axis=0)
+                hor_idx = np.nonzero(hor)[0]
+                x = hor_idx[0]
+                width = hor_idx[-1] - x + 1
+                vert = np.sum(mask, axis=1)
+                vert_idx = np.nonzero(vert)[0]
+                y = vert_idx[0]
+                height = vert_idx[-1] - y + 1
+                bbox = [int(x), int(y), int(width), int(height)]
+
+                segm_info.append(
+                    {
+                        "id": int(segment_id),
+                        "category_id": int(semantic_cat_id),
+                        "area": int(area),
+                        "bbox": bbox,
+                        "iscrowd": 0,
+                    }
+                )
+
+            panoptic_json_annotation = {
+                "image_id": image_id,
+                "file_name": image_id + ".png",
+                "segments_info": segm_info,
+            }
+
+            Image.fromarray(pan_seg).save(os.path.join(out_folder, image_id + ".png"))
+
+            panoptic_json_images.append(panoptic_json_image)
+            panoptic_json_annotations.append(panoptic_json_annotation)
+
+        # save this
+        d = {
+            "images": panoptic_json_images,
+            "annotations": panoptic_json_annotations,
+            "categories": panoptic_json_categories,
+        }
+
+        save_json(d, out_file)

datasets/prepare_ade20k_sem_seg.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Copyright (c) Facebook, Inc. and its affiliates.
+import os
+from pathlib import Path
+
+import numpy as np
+import tqdm
+from PIL import Image
+
+
+def convert(input, output):
+    img = np.asarray(Image.open(input))
+    assert img.dtype == np.uint8
+    img = img - 1  # 0 (ignore) becomes 255. others are shifted by 1
+    Image.fromarray(img).save(output)
+
+
+if __name__ == "__main__":
+    dataset_dir = Path(os.getenv("DETECTRON2_DATASETS", "datasets")) / "ADEChallengeData2016"
+    for name in ["training", "validation"]:
+        annotation_dir = dataset_dir / "annotations" / name
+        output_dir = dataset_dir / "annotations_detectron2" / name
+        output_dir.mkdir(parents=True, exist_ok=True)
+        for file in tqdm.tqdm(list(annotation_dir.iterdir())):
+            output_file = output_dir / file.name
+            convert(file, output_file)

datasets/prepare_coco_semantic_annos_from_panoptic_annos.py

@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Copyright (c) Facebook, Inc. and its affiliates.
+
+import functools
+import json
+import multiprocessing as mp
+import numpy as np
+import os
+import time
+from fvcore.common.download import download
+from panopticapi.utils import rgb2id
+from PIL import Image
+
+from detectron2.data.datasets.builtin_meta import COCO_CATEGORIES
+
+
+def _process_panoptic_to_semantic(input_panoptic, output_semantic, segments, id_map):
+    panoptic = np.asarray(Image.open(input_panoptic), dtype=np.uint32)
+    panoptic = rgb2id(panoptic)
+    output = np.zeros_like(panoptic, dtype=np.uint8) + 255
+    for seg in segments:
+        cat_id = seg["category_id"]
+        new_cat_id = id_map[cat_id]
+        output[panoptic == seg["id"]] = new_cat_id
+    Image.fromarray(output).save(output_semantic)
+
+
+def separate_coco_semantic_from_panoptic(panoptic_json, panoptic_root, sem_seg_root, categories):
+    """
+    Create semantic segmentation annotations from panoptic segmentation
+    annotations, to be used by PanopticFPN.
+    It maps all thing categories to class 0, and maps all unlabeled pixels to class 255.
+    It maps all stuff categories to contiguous ids starting from 1.
+    Args:
+        panoptic_json (str): path to the panoptic json file, in COCO's format.
+        panoptic_root (str): a directory with panoptic annotation files, in COCO's format.
+        sem_seg_root (str): a directory to output semantic annotation files
+        categories (list[dict]): category metadata. Each dict needs to have:
+            "id": corresponds to the "category_id" in the json annotations
+            "isthing": 0 or 1
+    """
+    os.makedirs(sem_seg_root, exist_ok=True)
+
+    id_map = {}  # map from category id to id in the output semantic annotation
+    assert len(categories) <= 254
+    for i, k in enumerate(categories):
+        id_map[k["id"]] = i
+    # what is id = 0?
+    # id_map[0] = 255
+    print(id_map)
+
+    with open(panoptic_json) as f:
+        obj = json.load(f)
+
+    pool = mp.Pool(processes=max(mp.cpu_count() // 2, 4))
+
+    def iter_annotations():
+        for anno in obj["annotations"]:
+            file_name = anno["file_name"]
+            segments = anno["segments_info"]
+            input = os.path.join(panoptic_root, file_name)
+            output = os.path.join(sem_seg_root, file_name)
+            yield input, output, segments
+
+    print("Start writing to {} ...".format(sem_seg_root))
+    start = time.time()
+    pool.starmap(
+        functools.partial(_process_panoptic_to_semantic, id_map=id_map),
+        iter_annotations(),
+        chunksize=100,
+    )
+    print("Finished. time: {:.2f}s".format(time.time() - start))
+
+
+if __name__ == "__main__":
+    dataset_dir = os.path.join(os.getenv("DETECTRON2_DATASETS", "datasets"), "coco")
+    for s in ["val2017", "train2017"]:
+        separate_coco_semantic_from_panoptic(
+            os.path.join(dataset_dir, "annotations/panoptic_{}.json".format(s)),
+            os.path.join(dataset_dir, "panoptic_{}".format(s)),
+            os.path.join(dataset_dir, "panoptic_semseg_{}".format(s)),
+            COCO_CATEGORIES,
+        )

datasets/prepare_pascal_ctx_full_sem_seg.py

@@ -0,0 +1,48 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# This work is made available under the Nvidia Source Code License.
+# To view a copy of this license, visit
+# https://github.com/NVlabs/ODISE/blob/main/LICENSE
+#
+# Written by Jiarui Xu
+# ------------------------------------------------------------------------------
+
+import os
+import numpy as np
+from pathlib import Path
+from PIL import Image
+import scipy.io as sio
+
+import tqdm
+
+
+def generate_labels(mat_file, out_dir):
+
+    mat = sio.loadmat(mat_file)
+    label_map = mat["LabelMap"]
+    assert label_map.dtype == np.uint16
+    label_map[label_map == 0] = 65535
+    label_map = label_map - 1
+    label_map[label_map == 65534] = 65535
+
+    out_file = out_dir / Path(mat_file.name).with_suffix(".tif")
+    Image.fromarray(label_map).save(out_file)
+
+
+if __name__ == "__main__":
+    dataset_dir = Path(os.getenv("DETECTRON2_DATASETS", "datasets")) / "pascal_ctx_d2"
+    voc_dir = Path(os.getenv("DETECTRON2_DATASETS", "datasets")) / "VOCdevkit/VOC2010"
+    mat_dir = voc_dir / "trainval"
+    for split in ["training", "validation"]:
+        file_names = list((dataset_dir / "images" / split).glob("*.jpg"))
+        output_img_dir = dataset_dir / "images" / split
+        output_ann_dir = dataset_dir / "annotations_ctx459" / split
+
+        output_img_dir.mkdir(parents=True, exist_ok=True)
+        output_ann_dir.mkdir(parents=True, exist_ok=True)
+
+        for file_name in tqdm.tqdm(file_names):
+            mat_file_path = mat_dir / f"{file_name.stem}.mat"
+
+            generate_labels(mat_file_path, output_ann_dir)

datasets/prepare_pascal_ctx_sem_seg.py

@@ -0,0 +1,84 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# This work is made available under the Nvidia Source Code License.
+# To view a copy of this license, visit
+# https://github.com/NVlabs/ODISE/blob/main/LICENSE
+#
+# Written by Jiarui Xu
+# ------------------------------------------------------------------------------
+
+import os
+from pathlib import Path
+import shutil
+
+import numpy as np
+import tqdm
+from PIL import Image
+import multiprocessing as mp
+import functools
+from detail import Detail
+
+# fmt: off
+_mapping = np.sort(
+    np.array([
+        0, 2, 259, 260, 415, 324, 9, 258, 144, 18, 19, 22, 23, 397, 25, 284,
+        158, 159, 416, 33, 162, 420, 454, 295, 296, 427, 44, 45, 46, 308, 59,
+        440, 445, 31, 232, 65, 354, 424, 68, 326, 72, 458, 34, 207, 80, 355,
+        85, 347, 220, 349, 360, 98, 187, 104, 105, 366, 189, 368, 113, 115
+    ]))
+# fmt: on
+_key = np.array(range(len(_mapping))).astype("uint8")
+
+
+def generate_labels(img_info, detail_api, out_dir):
+    def _class_to_index(mask, _mapping, _key):
+        # assert the values
+        values = np.unique(mask)
+        for i in range(len(values)):
+            assert values[i] in _mapping
+        index = np.digitize(mask.ravel(), _mapping, right=True)
+        return _key[index].reshape(mask.shape)
+
+    sem_seg = _class_to_index(detail_api.getMask(img_info), _mapping=_mapping, _key=_key)
+    sem_seg = sem_seg - 1  # 0 (ignore) becomes 255. others are shifted by 1
+    filename = img_info["file_name"]
+
+    Image.fromarray(sem_seg).save(out_dir / filename.replace("jpg", "png"))
+
+
+def copy_images(img_info, img_dir, out_dir):
+    filename = img_info["file_name"]
+    shutil.copy2(img_dir / filename, out_dir / filename)
+
+
+if __name__ == "__main__":
+    dataset_dir = Path(os.getenv("DETECTRON2_DATASETS", "datasets")) / "pascal_ctx_d2"
+    voc_dir = Path(os.getenv("DETECTRON2_DATASETS", "datasets")) / "VOCdevkit/VOC2010"
+    for split in ["training", "validation"]:
+        img_dir = voc_dir / "JPEGImages"
+        if split == "training":
+            detail_api = Detail(voc_dir / "trainval_merged.json", img_dir, "train")
+        else:
+            detail_api = Detail(voc_dir / "trainval_merged.json", img_dir, "val")
+        img_infos = detail_api.getImgs()
+
+        output_img_dir = dataset_dir / "images" / split
+        output_ann_dir = dataset_dir / "annotations_ctx59" / split
+
+        output_img_dir.mkdir(parents=True, exist_ok=True)
+        output_ann_dir.mkdir(parents=True, exist_ok=True)
+
+        pool = mp.Pool(processes=max(mp.cpu_count() // 2, 4))
+
+        pool.map(
+            functools.partial(copy_images, img_dir=img_dir, out_dir=output_img_dir),
+            tqdm.tqdm(img_infos, desc=f"Writing {split} images to {output_img_dir} ..."),
+            chunksize=100,
+        )
+
+        pool.map(
+            functools.partial(generate_labels, detail_api=detail_api, out_dir=output_ann_dir),
+            tqdm.tqdm(img_infos, desc=f"Writing {split} images to {output_ann_dir} ..."),
+            chunksize=100,
+        )

datasets/prepare_pascal_voc_sem_seg.py

@@ -0,0 +1,65 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# This work is made available under the Nvidia Source Code License.
+# To view a copy of this license, visit
+# https://github.com/NVlabs/ODISE/blob/main/LICENSE
+#
+# Written by Jiarui Xu
+# ------------------------------------------------------------------------------
+
+import os
+from pathlib import Path
+import shutil
+
+import numpy as np
+import tqdm
+from PIL import Image
+
+
+def convert_pas21(input, output):
+    img = np.asarray(Image.open(input))
+    assert img.dtype == np.uint8
+    # do nothing
+    Image.fromarray(img).save(output)
+
+def convert_pas20(input, output):
+    img = np.array(Image.open(input))
+    img[img == 0] = 255
+    img = img - 1
+    img[img == 254] = 255
+    assert img.dtype == np.uint8
+    # do nothing
+    Image.fromarray(img).save(output)
+
+
+if __name__ == "__main__":
+    dataset_dir = Path(os.getenv("DETECTRON2_DATASETS", "datasets")) / "pascal_voc_d2"
+    voc_dir = Path(os.getenv("DETECTRON2_DATASETS", "datasets")) / "VOCdevkit/VOC2012"
+    for split in ["training", "validation"]:
+        if split == "training":
+            img_name_path = voc_dir / "ImageSets/Segmentation/train.txt"
+        else:
+            img_name_path = voc_dir / "ImageSets/Segmentation/val.txt"
+        img_dir = voc_dir / "JPEGImages"
+        ann_dir = voc_dir / "SegmentationClass"
+
+        output_img_dir = dataset_dir / "images" / split
+        output_ann_dir_21 = dataset_dir / "annotations_pascal21" / split
+        output_ann_dir_20 = dataset_dir / "annotations_pascal20" / split
+
+        output_img_dir.mkdir(parents=True, exist_ok=True)
+        output_ann_dir_21.mkdir(parents=True, exist_ok=True)
+        output_ann_dir_20.mkdir(parents=True, exist_ok=True)
+
+        with open(img_name_path) as f:
+            for line in tqdm.tqdm(f.readlines()):
+                img_name = line.strip()
+                img_path = img_dir / f"{img_name}.jpg"
+                ann_path = ann_dir / f"{img_name}.png"
+
+                # print(f'copy2 {output_img_dir}')
+                shutil.copy2(img_path, output_img_dir)
+                # print(f"convert {ann_dir} to {output_ann_dir / f'{img_name}.png'}")
+                convert_pas21(ann_path, output_ann_dir_21 / f"{img_name}.png")
+                convert_pas20(ann_path, output_ann_dir_20 / f"{img_name}.png")

demo/demo.py

@@ -0,0 +1,195 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+# Modified by Bowen Cheng from: https://github.com/facebookresearch/detectron2/blob/master/demo/demo.py
+import argparse
+import glob
+import multiprocessing as mp
+import os
+
+# fmt: off
+import sys
+sys.path.insert(1, os.path.join(sys.path[0], '..'))
+# fmt: on
+
+import tempfile
+import time
+import warnings
+
+import cv2
+import numpy as np
+import tqdm
+
+from detectron2.config import get_cfg
+from detectron2.data.detection_utils import read_image
+from detectron2.projects.deeplab import add_deeplab_config
+from detectron2.utils.logger import setup_logger
+
+from fcclip import add_maskformer2_config, add_fcclip_config
+from predictor import VisualizationDemo
+
+
+# constants
+WINDOW_NAME = "fc-clip demo"
+
+
+def setup_cfg(args):
+    # load config from file and command-line arguments
+    cfg = get_cfg()
+    add_deeplab_config(cfg)
+    add_maskformer2_config(cfg)
+    add_fcclip_config(cfg)
+    cfg.merge_from_file(args.config_file)
+    cfg.merge_from_list(args.opts)
+    cfg.freeze()
+    return cfg
+
+
+def get_parser():
+    parser = argparse.ArgumentParser(description="fcclip demo for builtin configs")
+    parser.add_argument(
+        "--config-file",
+        default="configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_ade20k.yaml",
+        metavar="FILE",
+        help="path to config file",
+    )
+    parser.add_argument("--webcam", action="store_true", help="Take inputs from webcam.")
+    parser.add_argument("--video-input", help="Path to video file.")
+    parser.add_argument(
+        "--input",
+        nargs="+",
+        help="A list of space separated input images; "
+        "or a single glob pattern such as 'directory/*.jpg'",
+    )
+    parser.add_argument(
+        "--output",
+        help="A file or directory to save output visualizations. "
+        "If not given, will show output in an OpenCV window.",
+    )
+
+    parser.add_argument(
+        "--confidence-threshold",
+        type=float,
+        default=0.5,
+        help="Minimum score for instance predictions to be shown",
+    )
+    parser.add_argument(
+        "--opts",
+        help="Modify config options using the command-line 'KEY VALUE' pairs",
+        default=[],
+        nargs=argparse.REMAINDER,
+    )
+    return parser
+
+
+def test_opencv_video_format(codec, file_ext):
+    with tempfile.TemporaryDirectory(prefix="video_format_test") as dir:
+        filename = os.path.join(dir, "test_file" + file_ext)
+        writer = cv2.VideoWriter(
+            filename=filename,
+            fourcc=cv2.VideoWriter_fourcc(*codec),
+            fps=float(30),
+            frameSize=(10, 10),
+            isColor=True,
+        )
+        [writer.write(np.zeros((10, 10, 3), np.uint8)) for _ in range(30)]
+        writer.release()
+        if os.path.isfile(filename):
+            return True
+        return False
+
+
+if __name__ == "__main__":
+    mp.set_start_method("spawn", force=True)
+    args = get_parser().parse_args()
+    setup_logger(name="fvcore")
+    logger = setup_logger()
+    logger.info("Arguments: " + str(args))
+
+    cfg = setup_cfg(args)
+
+    demo = VisualizationDemo(cfg)
+
+    if args.input:
+        if len(args.input) == 1:
+            args.input = glob.glob(os.path.expanduser(args.input[0]))
+            assert args.input, "The input path(s) was not found"
+        for path in tqdm.tqdm(args.input, disable=not args.output):
+            # use PIL, to be consistent with evaluation
+            img = read_image(path, format="BGR")
+            start_time = time.time()
+            predictions, visualized_output = demo.run_on_image(img)
+            logger.info(
+                "{}: {} in {:.2f}s".format(
+                    path,
+                    "detected {} instances".format(len(predictions["instances"]))
+                    if "instances" in predictions
+                    else "finished",
+                    time.time() - start_time,
+                )
+            )
+
+            if args.output:
+                if os.path.isdir(args.output):
+                    assert os.path.isdir(args.output), args.output
+                    out_filename = os.path.join(args.output, os.path.basename(path))
+                else:
+                    assert len(args.input) == 1, "Please specify a directory with args.output"
+                    out_filename = args.output
+                visualized_output.save(out_filename)
+            else:
+                cv2.namedWindow(WINDOW_NAME, cv2.WINDOW_NORMAL)
+                cv2.imshow(WINDOW_NAME, visualized_output.get_image()[:, :, ::-1])
+                if cv2.waitKey(0) == 27:
+                    break  # esc to quit
+    elif args.webcam:
+        assert args.input is None, "Cannot have both --input and --webcam!"
+        assert args.output is None, "output not yet supported with --webcam!"
+        cam = cv2.VideoCapture(0)
+        for vis in tqdm.tqdm(demo.run_on_video(cam)):
+            cv2.namedWindow(WINDOW_NAME, cv2.WINDOW_NORMAL)
+            cv2.imshow(WINDOW_NAME, vis)
+            if cv2.waitKey(1) == 27:
+                break  # esc to quit
+        cam.release()
+        cv2.destroyAllWindows()
+    elif args.video_input:
+        video = cv2.VideoCapture(args.video_input)
+        width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH))
+        height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))
+        frames_per_second = video.get(cv2.CAP_PROP_FPS)
+        num_frames = int(video.get(cv2.CAP_PROP_FRAME_COUNT))
+        basename = os.path.basename(args.video_input)
+        codec, file_ext = (
+            ("x264", ".mkv") if test_opencv_video_format("x264", ".mkv") else ("mp4v", ".mp4")
+        )
+        if codec == ".mp4v":
+            warnings.warn("x264 codec not available, switching to mp4v")
+        if args.output:
+            if os.path.isdir(args.output):
+                output_fname = os.path.join(args.output, basename)
+                output_fname = os.path.splitext(output_fname)[0] + file_ext
+            else:
+                output_fname = args.output
+            assert not os.path.isfile(output_fname), output_fname
+            output_file = cv2.VideoWriter(
+                filename=output_fname,
+                # some installation of opencv may not support x264 (due to its license),
+                # you can try other format (e.g. MPEG)
+                fourcc=cv2.VideoWriter_fourcc(*codec),
+                fps=float(frames_per_second),
+                frameSize=(width, height),
+                isColor=True,
+            )
+        assert os.path.isfile(args.video_input)
+        for vis_frame in tqdm.tqdm(demo.run_on_video(video), total=num_frames):
+            if args.output:
+                output_file.write(vis_frame)
+            else:
+                cv2.namedWindow(basename, cv2.WINDOW_NORMAL)
+                cv2.imshow(basename, vis_frame)
+                if cv2.waitKey(1) == 27:
+                    break  # esc to quit
+        video.release()
+        if args.output:
+            output_file.release()
+        else:
+            cv2.destroyAllWindows()

demo/predictor.py

@@ -0,0 +1,275 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+# Copied from: https://github.com/facebookresearch/detectron2/blob/master/demo/predictor.py
+import atexit
+import bisect
+import multiprocessing as mp
+from collections import deque
+
+import cv2
+import torch
+import itertools
+
+
+from detectron2.data import DatasetCatalog, MetadataCatalog
+from detectron2.engine.defaults import DefaultPredictor as d2_defaultPredictor
+from detectron2.utils.video_visualizer import VideoVisualizer
+from detectron2.utils.visualizer import ColorMode, Visualizer, random_color
+import detectron2.utils.visualizer as d2_visualizer
+
+
+class DefaultPredictor(d2_defaultPredictor):
+
+    def set_metadata(self, metadata):
+        self.model.set_metadata(metadata)
+
+
+class OpenVocabVisualizer(Visualizer):
+    def draw_panoptic_seg(self, panoptic_seg, segments_info, area_threshold=None, alpha=0.7):
+        """
+        Draw panoptic prediction annotations or results.
+
+        Args:
+            panoptic_seg (Tensor): of shape (height, width) where the values are ids for each
+                segment.
+            segments_info (list[dict] or None): Describe each segment in `panoptic_seg`.
+                If it is a ``list[dict]``, each dict contains keys "id", "category_id".
+                If None, category id of each pixel is computed by
+                ``pixel // metadata.label_divisor``.
+            area_threshold (int): stuff segments with less than `area_threshold` are not drawn.
+
+        Returns:
+            output (VisImage): image object with visualizations.
+        """
+        pred = d2_visualizer._PanopticPrediction(panoptic_seg, segments_info, self.metadata)
+
+        if self._instance_mode == ColorMode.IMAGE_BW:
+            self.output.reset_image(self._create_grayscale_image(pred.non_empty_mask()))
+        # draw mask for all semantic segments first i.e. "stuff"
+        for mask, sinfo in pred.semantic_masks():
+            category_idx = sinfo["category_id"]
+            try:
+                mask_color = [x / 255 for x in self.metadata.stuff_colors[category_idx]]
+            except AttributeError:
+                mask_color = None
+
+            text = self.metadata.stuff_classes[category_idx].split(',')[0]
+            self.draw_binary_mask(
+                mask,
+                color=mask_color,
+                edge_color=d2_visualizer._OFF_WHITE,
+                text=text,
+                alpha=alpha,
+                area_threshold=area_threshold,
+            )
+        # draw mask for all instances second
+        all_instances = list(pred.instance_masks())
+        if len(all_instances) == 0:
+            return self.output
+        masks, sinfo = list(zip(*all_instances))
+        category_ids = [x["category_id"] for x in sinfo]
+
+        try:
+            scores = [x["score"] for x in sinfo]
+        except KeyError:
+            scores = None
+        stuff_classes = self.metadata.stuff_classes
+        stuff_classes = [x.split(',')[0] for x in stuff_classes]
+        labels = d2_visualizer._create_text_labels(
+            category_ids, scores, stuff_classes, [x.get("iscrowd", 0) for x in sinfo]
+        )
+
+        try:
+            colors = [
+                self._jitter([x / 255 for x in self.metadata.stuff_colors[c]]) for c in category_ids
+            ]
+        except AttributeError:
+            colors = None
+        self.overlay_instances(masks=masks, labels=labels, assigned_colors=colors, alpha=alpha)
+
+        return self.output
+
+
+class VisualizationDemo(object):
+    def __init__(self, cfg, instance_mode=ColorMode.IMAGE, parallel=False):
+        """
+        Args:
+            cfg (CfgNode):
+            instance_mode (ColorMode):
+            parallel (bool): whether to run the model in different processes from visualization.
+                Useful since the visualization logic can be slow.
+        """
+
+        coco_metadata = MetadataCatalog.get("openvocab_coco_2017_val_panoptic_with_sem_seg")
+        ade20k_metadata = MetadataCatalog.get("openvocab_ade20k_panoptic_val")
+        lvis_classes = open("./fcclip/data/datasets/lvis_1203_with_prompt_eng.txt", 'r').read().splitlines()
+        lvis_classes = [x[x.find(':')+1:] for x in lvis_classes]
+        lvis_colors = list(
+            itertools.islice(itertools.cycle(coco_metadata.stuff_colors), len(lvis_classes))
+        )
+        # rerrange to thing_classes, stuff_classes
+        coco_thing_classes = coco_metadata.thing_classes
+        coco_stuff_classes = [x for x in coco_metadata.stuff_classes if x not in coco_thing_classes]
+        coco_thing_colors = coco_metadata.thing_colors
+        coco_stuff_colors = [x for x in coco_metadata.stuff_colors if x not in coco_thing_colors]
+        ade20k_thing_classes = ade20k_metadata.thing_classes
+        ade20k_stuff_classes = [x for x in ade20k_metadata.stuff_classes if x not in ade20k_thing_classes]
+        ade20k_thing_colors = ade20k_metadata.thing_colors
+        ade20k_stuff_colors = [x for x in ade20k_metadata.stuff_colors if x not in ade20k_thing_colors]
+
+        user_classes = []
+        user_colors = [random_color(rgb=True, maximum=1) for _ in range(len(user_classes))]
+
+        stuff_classes = coco_stuff_classes + ade20k_stuff_classes
+        stuff_colors = coco_stuff_colors + ade20k_stuff_colors
+        thing_classes = user_classes + coco_thing_classes + ade20k_thing_classes + lvis_classes
+        thing_colors = user_colors + coco_thing_colors + ade20k_thing_colors + lvis_colors
+
+        thing_dataset_id_to_contiguous_id = {x: x for x in range(len(thing_classes))}
+        DatasetCatalog.register(
+            "openvocab_dataset", lambda x: []
+        )
+        self.metadata = MetadataCatalog.get("openvocab_dataset").set(
+            stuff_classes=thing_classes+stuff_classes,
+            stuff_colors=thing_colors+stuff_colors,
+            thing_dataset_id_to_contiguous_id=thing_dataset_id_to_contiguous_id,
+        )
+        #print("self.metadata:", self.metadata)
+        self.cpu_device = torch.device("cpu")
+        self.instance_mode = instance_mode
+
+        self.parallel = parallel
+        if parallel:
+            num_gpu = torch.cuda.device_count()
+            self.predictor = AsyncPredictor(cfg, num_gpus=num_gpu)
+        else:
+            self.predictor = DefaultPredictor(cfg)
+        self.predictor.set_metadata(self.metadata)
+
+    def run_on_image(self, image):
+        """
+        Args:
+            image (np.ndarray): an image of shape (H, W, C) (in BGR order).
+                This is the format used by OpenCV.
+        Returns:
+            predictions (dict): the output of the model.
+            vis_output (VisImage): the visualized image output.
+        """
+        vis_output = None
+        predictions = self.predictor(image)
+        # Convert image from OpenCV BGR format to Matplotlib RGB format.
+        image = image[:, :, ::-1]
+        visualizer = OpenVocabVisualizer(image, self.metadata, instance_mode=self.instance_mode)
+        if "panoptic_seg" in predictions:
+            panoptic_seg, segments_info = predictions["panoptic_seg"]
+            vis_output = visualizer.draw_panoptic_seg(
+                panoptic_seg.to(self.cpu_device), segments_info
+            )
+        else:
+            if "sem_seg" in predictions:
+                vis_output = visualizer.draw_sem_seg(
+                    predictions["sem_seg"].argmax(dim=0).to(self.cpu_device)
+                )
+            if "instances" in predictions:
+                instances = predictions["instances"].to(self.cpu_device)
+                vis_output = visualizer.draw_instance_predictions(predictions=instances)
+
+        return predictions, vis_output
+
+    def _frame_from_video(self, video):
+        while video.isOpened():
+            success, frame = video.read()
+            if success:
+                yield frame
+            else:
+                break
+
+
+class AsyncPredictor:
+    """
+    A predictor that runs the model asynchronously, possibly on >1 GPUs.
+    Because rendering the visualization takes considerably amount of time,
+    this helps improve throughput a little bit when rendering videos.
+    """
+
+    class _StopToken:
+        pass
+
+    class _PredictWorker(mp.Process):
+        def __init__(self, cfg, task_queue, result_queue):
+            self.cfg = cfg
+            self.task_queue = task_queue
+            self.result_queue = result_queue
+            super().__init__()
+
+        def run(self):
+            predictor = DefaultPredictor(self.cfg)
+
+            while True:
+                task = self.task_queue.get()
+                if isinstance(task, AsyncPredictor._StopToken):
+                    break
+                idx, data = task
+                result = predictor(data)
+                self.result_queue.put((idx, result))
+
+    def __init__(self, cfg, num_gpus: int = 1):
+        """
+        Args:
+            cfg (CfgNode):
+            num_gpus (int): if 0, will run on CPU
+        """
+        num_workers = max(num_gpus, 1)
+        self.task_queue = mp.Queue(maxsize=num_workers * 3)
+        self.result_queue = mp.Queue(maxsize=num_workers * 3)
+        self.procs = []
+        for gpuid in range(max(num_gpus, 1)):
+            cfg = cfg.clone()
+            cfg.defrost()
+            cfg.MODEL.DEVICE = "cuda:{}".format(gpuid) if num_gpus > 0 else "cpu"
+            self.procs.append(
+                AsyncPredictor._PredictWorker(cfg, self.task_queue, self.result_queue)
+            )
+
+        self.put_idx = 0
+        self.get_idx = 0
+        self.result_rank = []
+        self.result_data = []
+
+        for p in self.procs:
+            p.start()
+        atexit.register(self.shutdown)
+
+    def put(self, image):
+        self.put_idx += 1
+        self.task_queue.put((self.put_idx, image))
+
+    def get(self):
+        self.get_idx += 1  # the index needed for this request
+        if len(self.result_rank) and self.result_rank[0] == self.get_idx:
+            res = self.result_data[0]
+            del self.result_data[0], self.result_rank[0]
+            return res
+
+        while True:
+            # make sure the results are returned in the correct order
+            idx, res = self.result_queue.get()
+            if idx == self.get_idx:
+                return res
+            insert = bisect.bisect(self.result_rank, idx)
+            self.result_rank.insert(insert, idx)
+            self.result_data.insert(insert, res)
+
+    def __len__(self):
+        return self.put_idx - self.get_idx
+
+    def __call__(self, image):
+        self.put(image)
+        return self.get()
+
+    def shutdown(self):
+        for _ in self.procs:
+            self.task_queue.put(AsyncPredictor._StopToken())
+
+    @property
+    def default_buffer_size(self):
+        return len(self.procs) * 5

fcclip/__init__.py

@@ -0,0 +1,26 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+from . import data  # register all new datasets
+from . import modeling
+
+# config
+from .config import add_maskformer2_config, add_fcclip_config
+
+# dataset loading
+from .data.dataset_mappers.coco_instance_new_baseline_dataset_mapper import COCOInstanceNewBaselineDatasetMapper
+from .data.dataset_mappers.coco_panoptic_new_baseline_dataset_mapper import COCOPanopticNewBaselineDatasetMapper
+from .data.dataset_mappers.mask_former_instance_dataset_mapper import (
+    MaskFormerInstanceDatasetMapper,
+)
+from .data.dataset_mappers.mask_former_panoptic_dataset_mapper import (
+    MaskFormerPanopticDatasetMapper,
+)
+from .data.dataset_mappers.mask_former_semantic_dataset_mapper import (
+    MaskFormerSemanticDatasetMapper,
+)
+
+# models
+from .fcclip import FCCLIP
+from .test_time_augmentation import SemanticSegmentorWithTTA
+
+# evaluation
+from .evaluation.instance_evaluation import InstanceSegEvaluator

fcclip/config.py

@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) Facebook, Inc. and its affiliates.
+from detectron2.config import CfgNode as CN
+
+
+def add_maskformer2_config(cfg):
+    """
+    Add config for MASK_FORMER.
+    """
+    # NOTE: configs from original maskformer
+    # data config
+    # select the dataset mapper
+    cfg.INPUT.DATASET_MAPPER_NAME = "mask_former_semantic"
+    # Color augmentation
+    cfg.INPUT.COLOR_AUG_SSD = False
+    # We retry random cropping until no single category in semantic segmentation GT occupies more
+    # than `SINGLE_CATEGORY_MAX_AREA` part of the crop.
+    cfg.INPUT.CROP.SINGLE_CATEGORY_MAX_AREA = 1.0
+    # Pad image and segmentation GT in dataset mapper.
+    cfg.INPUT.SIZE_DIVISIBILITY = -1
+
+    # solver config
+    # weight decay on embedding
+    cfg.SOLVER.WEIGHT_DECAY_EMBED = 0.0
+    # optimizer
+    cfg.SOLVER.OPTIMIZER = "ADAMW"
+    cfg.SOLVER.BACKBONE_MULTIPLIER = 0.1
+
+    # mask_former model config
+    cfg.MODEL.MASK_FORMER = CN()
+
+    # loss
+    cfg.MODEL.MASK_FORMER.DEEP_SUPERVISION = True
+    cfg.MODEL.MASK_FORMER.NO_OBJECT_WEIGHT = 0.1
+    cfg.MODEL.MASK_FORMER.CLASS_WEIGHT = 1.0
+    cfg.MODEL.MASK_FORMER.DICE_WEIGHT = 1.0
+    cfg.MODEL.MASK_FORMER.MASK_WEIGHT = 20.0
+
+    # transformer config
+    cfg.MODEL.MASK_FORMER.NHEADS = 8
+    cfg.MODEL.MASK_FORMER.DROPOUT = 0.1
+    cfg.MODEL.MASK_FORMER.DIM_FEEDFORWARD = 2048
+    cfg.MODEL.MASK_FORMER.ENC_LAYERS = 0
+    cfg.MODEL.MASK_FORMER.DEC_LAYERS = 6
+    cfg.MODEL.MASK_FORMER.PRE_NORM = False
+
+    cfg.MODEL.MASK_FORMER.HIDDEN_DIM = 256
+    cfg.MODEL.MASK_FORMER.NUM_OBJECT_QUERIES = 100
+
+    cfg.MODEL.MASK_FORMER.TRANSFORMER_IN_FEATURE = "res5"
+    cfg.MODEL.MASK_FORMER.ENFORCE_INPUT_PROJ = False
+
+    # mask_former inference config
+    cfg.MODEL.MASK_FORMER.TEST = CN()
+    cfg.MODEL.MASK_FORMER.TEST.SEMANTIC_ON = True
+    cfg.MODEL.MASK_FORMER.TEST.INSTANCE_ON = False
+    cfg.MODEL.MASK_FORMER.TEST.PANOPTIC_ON = False
+    cfg.MODEL.MASK_FORMER.TEST.OBJECT_MASK_THRESHOLD = 0.0
+    cfg.MODEL.MASK_FORMER.TEST.OVERLAP_THRESHOLD = 0.0
+    cfg.MODEL.MASK_FORMER.TEST.SEM_SEG_POSTPROCESSING_BEFORE_INFERENCE = False
+
+    # Sometimes `backbone.size_divisibility` is set to 0 for some backbone (e.g. ResNet)
+    # you can use this config to override
+    cfg.MODEL.MASK_FORMER.SIZE_DIVISIBILITY = 32
+
+    # pixel decoder config
+    cfg.MODEL.SEM_SEG_HEAD.MASK_DIM = 256
+    # adding transformer in pixel decoder
+    cfg.MODEL.SEM_SEG_HEAD.TRANSFORMER_ENC_LAYERS = 0
+    # pixel decoder
+    cfg.MODEL.SEM_SEG_HEAD.PIXEL_DECODER_NAME = "BasePixelDecoder"
+
+    # swin transformer backbone
+    cfg.MODEL.SWIN = CN()
+    cfg.MODEL.SWIN.PRETRAIN_IMG_SIZE = 224
+    cfg.MODEL.SWIN.PATCH_SIZE = 4
+    cfg.MODEL.SWIN.EMBED_DIM = 96
+    cfg.MODEL.SWIN.DEPTHS = [2, 2, 6, 2]
+    cfg.MODEL.SWIN.NUM_HEADS = [3, 6, 12, 24]
+    cfg.MODEL.SWIN.WINDOW_SIZE = 7
+    cfg.MODEL.SWIN.MLP_RATIO = 4.0
+    cfg.MODEL.SWIN.QKV_BIAS = True
+    cfg.MODEL.SWIN.QK_SCALE = None
+    cfg.MODEL.SWIN.DROP_RATE = 0.0
+    cfg.MODEL.SWIN.ATTN_DROP_RATE = 0.0
+    cfg.MODEL.SWIN.DROP_PATH_RATE = 0.3
+    cfg.MODEL.SWIN.APE = False
+    cfg.MODEL.SWIN.PATCH_NORM = True
+    cfg.MODEL.SWIN.OUT_FEATURES = ["res2", "res3", "res4", "res5"]
+    cfg.MODEL.SWIN.USE_CHECKPOINT = False
+
+    # NOTE: maskformer2 extra configs
+    # transformer module
+    cfg.MODEL.MASK_FORMER.TRANSFORMER_DECODER_NAME = "MultiScaleMaskedTransformerDecoder"
+
+    # LSJ aug
+    cfg.INPUT.IMAGE_SIZE = 1024
+    cfg.INPUT.MIN_SCALE = 0.1
+    cfg.INPUT.MAX_SCALE = 2.0
+
+    # MSDeformAttn encoder configs
+    cfg.MODEL.SEM_SEG_HEAD.DEFORMABLE_TRANSFORMER_ENCODER_IN_FEATURES = ["res3", "res4", "res5"]
+    cfg.MODEL.SEM_SEG_HEAD.DEFORMABLE_TRANSFORMER_ENCODER_N_POINTS = 4
+    cfg.MODEL.SEM_SEG_HEAD.DEFORMABLE_TRANSFORMER_ENCODER_N_HEADS = 8
+
+    # point loss configs
+    # Number of points sampled during training for a mask point head.
+    cfg.MODEL.MASK_FORMER.TRAIN_NUM_POINTS = 112 * 112
+    # Oversampling parameter for PointRend point sampling during training. Parameter `k` in the
+    # original paper.
+    cfg.MODEL.MASK_FORMER.OVERSAMPLE_RATIO = 3.0
+    # Importance sampling parameter for PointRend point sampling during training. Parametr `beta` in
+    # the original paper.
+    cfg.MODEL.MASK_FORMER.IMPORTANCE_SAMPLE_RATIO = 0.75
+
+
+def add_fcclip_config(cfg):
+    # FC-CLIP model config
+    cfg.MODEL.FC_CLIP = CN()
+    cfg.MODEL.FC_CLIP.CLIP_MODEL_NAME = "convnext_large_d_320"
+    cfg.MODEL.FC_CLIP.CLIP_PRETRAINED_WEIGHTS = "laion2b_s29b_b131k_ft_soup"
+    cfg.MODEL.FC_CLIP.EMBED_DIM = 768
+    cfg.MODEL.FC_CLIP.GEOMETRIC_ENSEMBLE_ALPHA = 0.4
+    cfg.MODEL.FC_CLIP.GEOMETRIC_ENSEMBLE_BETA = 0.8

fcclip/data/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+from . import datasets

fcclip/data/dataset_mappers/__init__.py

@@ -0,0 +1 @@
+# Copyright (c) Facebook, Inc. and its affiliates.

fcclip/data/dataset_mappers/coco_instance_new_baseline_dataset_mapper.py

@@ -0,0 +1,189 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+# Modified by Bowen Cheng from https://github.com/facebookresearch/detr/blob/master/d2/detr/dataset_mapper.py
+import copy
+import logging
+
+import numpy as np
+import torch
+
+from detectron2.config import configurable
+from detectron2.data import detection_utils as utils
+from detectron2.data import transforms as T
+from detectron2.data.transforms import TransformGen
+from detectron2.structures import BitMasks, Instances
+
+from pycocotools import mask as coco_mask
+
+__all__ = ["COCOInstanceNewBaselineDatasetMapper"]
+
+
+def convert_coco_poly_to_mask(segmentations, height, width):
+    masks = []
+    for polygons in segmentations:
+        rles = coco_mask.frPyObjects(polygons, height, width)
+        mask = coco_mask.decode(rles)
+        if len(mask.shape) < 3:
+            mask = mask[..., None]
+        mask = torch.as_tensor(mask, dtype=torch.uint8)
+        mask = mask.any(dim=2)
+        masks.append(mask)
+    if masks:
+        masks = torch.stack(masks, dim=0)
+    else:
+        masks = torch.zeros((0, height, width), dtype=torch.uint8)
+    return masks
+
+
+def build_transform_gen(cfg, is_train):
+    """
+    Create a list of default :class:`Augmentation` from config.
+    Now it includes resizing and flipping.
+    Returns:
+        list[Augmentation]
+    """
+    assert is_train, "Only support training augmentation"
+    image_size = cfg.INPUT.IMAGE_SIZE
+    min_scale = cfg.INPUT.MIN_SCALE
+    max_scale = cfg.INPUT.MAX_SCALE
+
+    augmentation = []
+
+    if cfg.INPUT.RANDOM_FLIP != "none":
+        augmentation.append(
+            T.RandomFlip(
+                horizontal=cfg.INPUT.RANDOM_FLIP == "horizontal",
+                vertical=cfg.INPUT.RANDOM_FLIP == "vertical",
+            )
+        )
+
+    augmentation.extend([
+        T.ResizeScale(
+            min_scale=min_scale, max_scale=max_scale, target_height=image_size, target_width=image_size
+        ),
+        T.FixedSizeCrop(crop_size=(image_size, image_size)),
+    ])
+
+    return augmentation
+
+
+# This is specifically designed for the COCO dataset.
+class COCOInstanceNewBaselineDatasetMapper:
+    """
+    A callable which takes a dataset dict in Detectron2 Dataset format,
+    and map it into a format used by MaskFormer.
+
+    This dataset mapper applies the same transformation as DETR for COCO panoptic segmentation.
+
+    The callable currently does the following:
+
+    1. Read the image from "file_name"
+    2. Applies geometric transforms to the image and annotation
+    3. Find and applies suitable cropping to the image and annotation
+    4. Prepare image and annotation to Tensors
+    """
+
+    @configurable
+    def __init__(
+        self,
+        is_train=True,
+        *,
+        tfm_gens,
+        image_format,
+    ):
+        """
+        NOTE: this interface is experimental.
+        Args:
+            is_train: for training or inference
+            augmentations: a list of augmentations or deterministic transforms to apply
+            tfm_gens: data augmentation
+            image_format: an image format supported by :func:`detection_utils.read_image`.
+        """
+        self.tfm_gens = tfm_gens
+        logging.getLogger(__name__).info(
+            "[COCOInstanceNewBaselineDatasetMapper] Full TransformGens used in training: {}".format(str(self.tfm_gens))
+        )
+
+        self.img_format = image_format
+        self.is_train = is_train
+    
+    @classmethod
+    def from_config(cls, cfg, is_train=True):
+        # Build augmentation
+        tfm_gens = build_transform_gen(cfg, is_train)
+
+        ret = {
+            "is_train": is_train,
+            "tfm_gens": tfm_gens,
+            "image_format": cfg.INPUT.FORMAT,
+        }
+        return ret
+
+    def __call__(self, dataset_dict):
+        """
+        Args:
+            dataset_dict (dict): Metadata of one image, in Detectron2 Dataset format.
+
+        Returns:
+            dict: a format that builtin models in detectron2 accept
+        """
+        dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
+        image = utils.read_image(dataset_dict["file_name"], format=self.img_format)
+        utils.check_image_size(dataset_dict, image)
+
+        # TODO: get padding mask
+        # by feeding a "segmentation mask" to the same transforms
+        padding_mask = np.ones(image.shape[:2])
+
+        image, transforms = T.apply_transform_gens(self.tfm_gens, image)
+        # the crop transformation has default padding value 0 for segmentation
+        padding_mask = transforms.apply_segmentation(padding_mask)
+        padding_mask = ~ padding_mask.astype(bool)
+
+        image_shape = image.shape[:2]  # h, w
+
+        # Pytorch's dataloader is efficient on torch.Tensor due to shared-memory,
+        # but not efficient on large generic data structures due to the use of pickle & mp.Queue.
+        # Therefore it's important to use torch.Tensor.
+        dataset_dict["image"] = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))
+        dataset_dict["padding_mask"] = torch.as_tensor(np.ascontiguousarray(padding_mask))
+
+        if not self.is_train:
+            # USER: Modify this if you want to keep them for some reason.
+            dataset_dict.pop("annotations", None)
+            return dataset_dict
+
+        if "annotations" in dataset_dict:
+            # USER: Modify this if you want to keep them for some reason.
+            for anno in dataset_dict["annotations"]:
+                # Let's always keep mask
+                # if not self.mask_on:
+                #     anno.pop("segmentation", None)
+                anno.pop("keypoints", None)
+
+            # USER: Implement additional transformations if you have other types of data
+            annos = [
+                utils.transform_instance_annotations(obj, transforms, image_shape)
+                for obj in dataset_dict.pop("annotations")
+                if obj.get("iscrowd", 0) == 0
+            ]
+            # NOTE: does not support BitMask due to augmentation
+            # Current BitMask cannot handle empty objects
+            instances = utils.annotations_to_instances(annos, image_shape)
+            # After transforms such as cropping are applied, the bounding box may no longer
+            # tightly bound the object. As an example, imagine a triangle object
+            # [(0,0), (2,0), (0,2)] cropped by a box [(1,0),(2,2)] (XYXY format). The tight
+            # bounding box of the cropped triangle should be [(1,0),(2,1)], which is not equal to
+            # the intersection of original bounding box and the cropping box.
+            instances.gt_boxes = instances.gt_masks.get_bounding_boxes()
+            # Need to filter empty instances first (due to augmentation)
+            instances = utils.filter_empty_instances(instances)
+            # Generate masks from polygon
+            h, w = instances.image_size
+            # image_size_xyxy = torch.as_tensor([w, h, w, h], dtype=torch.float)
+            if hasattr(instances, 'gt_masks'):
+                gt_masks = instances.gt_masks
+                gt_masks = convert_coco_poly_to_mask(gt_masks.polygons, h, w)
+                instances.gt_masks = gt_masks
+            dataset_dict["instances"] = instances
+
+        return dataset_dict

fcclip/data/dataset_mappers/coco_panoptic_new_baseline_dataset_mapper.py

@@ -0,0 +1,165 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+# Modified by Bowen Cheng from https://github.com/facebookresearch/detr/blob/master/d2/detr/dataset_mapper.py
+import copy
+import logging
+
+import numpy as np
+import torch
+
+from detectron2.config import configurable
+from detectron2.data import detection_utils as utils
+from detectron2.data import transforms as T
+from detectron2.data.transforms import TransformGen
+from detectron2.structures import BitMasks, Boxes, Instances
+
+__all__ = ["COCOPanopticNewBaselineDatasetMapper"]
+
+
+def build_transform_gen(cfg, is_train):
+    """
+    Create a list of default :class:`Augmentation` from config.
+    Now it includes resizing and flipping.
+    Returns:
+        list[Augmentation]
+    """
+    assert is_train, "Only support training augmentation"
+    image_size = cfg.INPUT.IMAGE_SIZE
+    min_scale = cfg.INPUT.MIN_SCALE
+    max_scale = cfg.INPUT.MAX_SCALE
+
+    augmentation = []
+
+    if cfg.INPUT.RANDOM_FLIP != "none":
+        augmentation.append(
+            T.RandomFlip(
+                horizontal=cfg.INPUT.RANDOM_FLIP == "horizontal",
+                vertical=cfg.INPUT.RANDOM_FLIP == "vertical",
+            )
+        )
+
+    augmentation.extend([
+        T.ResizeScale(
+            min_scale=min_scale, max_scale=max_scale, target_height=image_size, target_width=image_size
+        ),
+        T.FixedSizeCrop(crop_size=(image_size, image_size)),
+    ])
+
+    return augmentation
+
+
+# This is specifically designed for the COCO dataset.
+class COCOPanopticNewBaselineDatasetMapper:
+    """
+    A callable which takes a dataset dict in Detectron2 Dataset format,
+    and map it into a format used by MaskFormer.
+
+    This dataset mapper applies the same transformation as DETR for COCO panoptic segmentation.
+
+    The callable currently does the following:
+
+    1. Read the image from "file_name"
+    2. Applies geometric transforms to the image and annotation
+    3. Find and applies suitable cropping to the image and annotation
+    4. Prepare image and annotation to Tensors
+    """
+
+    @configurable
+    def __init__(
+        self,
+        is_train=True,
+        *,
+        tfm_gens,
+        image_format,
+    ):
+        """
+        NOTE: this interface is experimental.
+        Args:
+            is_train: for training or inference
+            augmentations: a list of augmentations or deterministic transforms to apply
+            crop_gen: crop augmentation
+            tfm_gens: data augmentation
+            image_format: an image format supported by :func:`detection_utils.read_image`.
+        """
+        self.tfm_gens = tfm_gens
+        logging.getLogger(__name__).info(
+            "[COCOPanopticNewBaselineDatasetMapper] Full TransformGens used in training: {}".format(
+                str(self.tfm_gens)
+            )
+        )
+
+        self.img_format = image_format
+        self.is_train = is_train
+
+    @classmethod
+    def from_config(cls, cfg, is_train=True):
+        # Build augmentation
+        tfm_gens = build_transform_gen(cfg, is_train)
+
+        ret = {
+            "is_train": is_train,
+            "tfm_gens": tfm_gens,
+            "image_format": cfg.INPUT.FORMAT,
+        }
+        return ret
+
+    def __call__(self, dataset_dict):
+        """
+        Args:
+            dataset_dict (dict): Metadata of one image, in Detectron2 Dataset format.
+
+        Returns:
+            dict: a format that builtin models in detectron2 accept
+        """
+        dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
+        image = utils.read_image(dataset_dict["file_name"], format=self.img_format)
+        utils.check_image_size(dataset_dict, image)
+
+        image, transforms = T.apply_transform_gens(self.tfm_gens, image)
+        image_shape = image.shape[:2]  # h, w
+
+        # Pytorch's dataloader is efficient on torch.Tensor due to shared-memory,
+        # but not efficient on large generic data structures due to the use of pickle & mp.Queue.
+        # Therefore it's important to use torch.Tensor.
+        dataset_dict["image"] = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))
+
+        if not self.is_train:
+            # USER: Modify this if you want to keep them for some reason.
+            dataset_dict.pop("annotations", None)
+            return dataset_dict
+
+        if "pan_seg_file_name" in dataset_dict:
+            pan_seg_gt = utils.read_image(dataset_dict.pop("pan_seg_file_name"), "RGB")
+            segments_info = dataset_dict["segments_info"]
+
+            # apply the same transformation to panoptic segmentation
+            pan_seg_gt = transforms.apply_segmentation(pan_seg_gt)
+
+            from panopticapi.utils import rgb2id
+
+            pan_seg_gt = rgb2id(pan_seg_gt)
+
+            instances = Instances(image_shape)
+            classes = []
+            masks = []
+            for segment_info in segments_info:
+                class_id = segment_info["category_id"]
+                if not segment_info["iscrowd"]:
+                    classes.append(class_id)
+                    masks.append(pan_seg_gt == segment_info["id"])
+
+            classes = np.array(classes)
+            instances.gt_classes = torch.tensor(classes, dtype=torch.int64)
+            if len(masks) == 0:
+                # Some image does not have annotation (all ignored)
+                instances.gt_masks = torch.zeros((0, pan_seg_gt.shape[-2], pan_seg_gt.shape[-1]))
+                instances.gt_boxes = Boxes(torch.zeros((0, 4)))
+            else:
+                masks = BitMasks(
+                    torch.stack([torch.from_numpy(np.ascontiguousarray(x.copy())) for x in masks])
+                )
+                instances.gt_masks = masks.tensor
+                instances.gt_boxes = masks.get_bounding_boxes()
+
+            dataset_dict["instances"] = instances
+
+        return dataset_dict

fcclip/data/dataset_mappers/mask_former_instance_dataset_mapper.py

@@ -0,0 +1,180 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+import copy
+import logging
+
+import numpy as np
+import pycocotools.mask as mask_util
+import torch
+from torch.nn import functional as F
+
+from detectron2.config import configurable
+from detectron2.data import detection_utils as utils
+from detectron2.data import transforms as T
+from detectron2.projects.point_rend import ColorAugSSDTransform
+from detectron2.structures import BitMasks, Instances, polygons_to_bitmask
+
+__all__ = ["MaskFormerInstanceDatasetMapper"]
+
+
+class MaskFormerInstanceDatasetMapper:
+    """
+    A callable which takes a dataset dict in Detectron2 Dataset format,
+    and map it into a format used by MaskFormer for instance segmentation.
+
+    The callable currently does the following:
+
+    1. Read the image from "file_name"
+    2. Applies geometric transforms to the image and annotation
+    3. Find and applies suitable cropping to the image and annotation
+    4. Prepare image and annotation to Tensors
+    """
+
+    @configurable
+    def __init__(
+        self,
+        is_train=True,
+        *,
+        augmentations,
+        image_format,
+        size_divisibility,
+    ):
+        """
+        NOTE: this interface is experimental.
+        Args:
+            is_train: for training or inference
+            augmentations: a list of augmentations or deterministic transforms to apply
+            image_format: an image format supported by :func:`detection_utils.read_image`.
+            size_divisibility: pad image size to be divisible by this value
+        """
+        self.is_train = is_train
+        self.tfm_gens = augmentations
+        self.img_format = image_format
+        self.size_divisibility = size_divisibility
+
+        logger = logging.getLogger(__name__)
+        mode = "training" if is_train else "inference"
+        logger.info(f"[{self.__class__.__name__}] Augmentations used in {mode}: {augmentations}")
+
+    @classmethod
+    def from_config(cls, cfg, is_train=True):
+        # Build augmentation
+        augs = [
+            T.ResizeShortestEdge(
+                cfg.INPUT.MIN_SIZE_TRAIN,
+                cfg.INPUT.MAX_SIZE_TRAIN,
+                cfg.INPUT.MIN_SIZE_TRAIN_SAMPLING,
+            )
+        ]
+        if cfg.INPUT.CROP.ENABLED:
+            augs.append(
+                T.RandomCrop(
+                    cfg.INPUT.CROP.TYPE,
+                    cfg.INPUT.CROP.SIZE,
+                )
+            )
+        if cfg.INPUT.COLOR_AUG_SSD:
+            augs.append(ColorAugSSDTransform(img_format=cfg.INPUT.FORMAT))
+        augs.append(T.RandomFlip())
+
+        ret = {
+            "is_train": is_train,
+            "augmentations": augs,
+            "image_format": cfg.INPUT.FORMAT,
+            "size_divisibility": cfg.INPUT.SIZE_DIVISIBILITY,
+        }
+        return ret
+
+    def __call__(self, dataset_dict):
+        """
+        Args:
+            dataset_dict (dict): Metadata of one image, in Detectron2 Dataset format.
+
+        Returns:
+            dict: a format that builtin models in detectron2 accept
+        """
+        assert self.is_train, "MaskFormerPanopticDatasetMapper should only be used for training!"
+
+        dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
+        image = utils.read_image(dataset_dict["file_name"], format=self.img_format)
+        utils.check_image_size(dataset_dict, image)
+
+        aug_input = T.AugInput(image)
+        aug_input, transforms = T.apply_transform_gens(self.tfm_gens, aug_input)
+        image = aug_input.image
+
+        # transform instnace masks
+        assert "annotations" in dataset_dict
+        for anno in dataset_dict["annotations"]:
+            anno.pop("keypoints", None)
+
+        annos = [
+            utils.transform_instance_annotations(obj, transforms, image.shape[:2])
+            for obj in dataset_dict.pop("annotations")
+            if obj.get("iscrowd", 0) == 0
+        ]
+
+        if len(annos):
+            assert "segmentation" in annos[0]
+        segms = [obj["segmentation"] for obj in annos]
+        masks = []
+        for segm in segms:
+            if isinstance(segm, list):
+                # polygon
+                masks.append(polygons_to_bitmask(segm, *image.shape[:2]))
+            elif isinstance(segm, dict):
+                # COCO RLE
+                masks.append(mask_util.decode(segm))
+            elif isinstance(segm, np.ndarray):
+                assert segm.ndim == 2, "Expect segmentation of 2 dimensions, got {}.".format(
+                    segm.ndim
+                )
+                # mask array
+                masks.append(segm)
+            else:
+                raise ValueError(
+                    "Cannot convert segmentation of type '{}' to BitMasks!"
+                    "Supported types are: polygons as list[list[float] or ndarray],"
+                    " COCO-style RLE as a dict, or a binary segmentation mask "
+                    " in a 2D numpy array of shape HxW.".format(type(segm))
+                )
+
+        # Pad image and segmentation label here!
+        image = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))
+        masks = [torch.from_numpy(np.ascontiguousarray(x)) for x in masks]
+
+        classes = [int(obj["category_id"]) for obj in annos]
+        classes = torch.tensor(classes, dtype=torch.int64)
+
+        if self.size_divisibility > 0:
+            image_size = (image.shape[-2], image.shape[-1])
+            padding_size = [
+                0,
+                self.size_divisibility - image_size[1],
+                0,
+                self.size_divisibility - image_size[0],
+            ]
+            # pad image
+            image = F.pad(image, padding_size, value=128).contiguous()
+            # pad mask
+            masks = [F.pad(x, padding_size, value=0).contiguous() for x in masks]
+
+        image_shape = (image.shape[-2], image.shape[-1])  # h, w
+
+        # Pytorch's dataloader is efficient on torch.Tensor due to shared-memory,
+        # but not efficient on large generic data structures due to the use of pickle & mp.Queue.
+        # Therefore it's important to use torch.Tensor.
+        dataset_dict["image"] = image
+
+        # Prepare per-category binary masks
+        instances = Instances(image_shape)
+        instances.gt_classes = classes
+        if len(masks) == 0:
+            # Some image does not have annotation (all ignored)
+            instances.gt_masks = torch.zeros((0, image.shape[-2], image.shape[-1]))
+        else:
+            masks = BitMasks(torch.stack(masks))
+            instances.gt_masks = masks.tensor
+
+        dataset_dict["instances"] = instances
+
+        return dataset_dict

fcclip/data/dataset_mappers/mask_former_panoptic_dataset_mapper.py

@@ -0,0 +1,165 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+import copy
+import logging
+
+import numpy as np
+import torch
+from torch.nn import functional as F
+
+from detectron2.config import configurable
+from detectron2.data import detection_utils as utils
+from detectron2.data import transforms as T
+from detectron2.structures import BitMasks, Instances
+
+from .mask_former_semantic_dataset_mapper import MaskFormerSemanticDatasetMapper
+
+__all__ = ["MaskFormerPanopticDatasetMapper"]
+
+
+class MaskFormerPanopticDatasetMapper(MaskFormerSemanticDatasetMapper):
+    """
+    A callable which takes a dataset dict in Detectron2 Dataset format,
+    and map it into a format used by MaskFormer for panoptic segmentation.
+
+    The callable currently does the following:
+
+    1. Read the image from "file_name"
+    2. Applies geometric transforms to the image and annotation
+    3. Find and applies suitable cropping to the image and annotation
+    4. Prepare image and annotation to Tensors
+    """
+
+    @configurable
+    def __init__(
+        self,
+        is_train=True,
+        *,
+        augmentations,
+        image_format,
+        ignore_label,
+        size_divisibility,
+    ):
+        """
+        NOTE: this interface is experimental.
+        Args:
+            is_train: for training or inference
+            augmentations: a list of augmentations or deterministic transforms to apply
+            image_format: an image format supported by :func:`detection_utils.read_image`.
+            ignore_label: the label that is ignored to evaluation
+            size_divisibility: pad image size to be divisible by this value
+        """
+        super().__init__(
+            is_train,
+            augmentations=augmentations,
+            image_format=image_format,
+            ignore_label=ignore_label,
+            size_divisibility=size_divisibility,
+        )
+
+    def __call__(self, dataset_dict):
+        """
+        Args:
+            dataset_dict (dict): Metadata of one image, in Detectron2 Dataset format.
+
+        Returns:
+            dict: a format that builtin models in detectron2 accept
+        """
+        assert self.is_train, "MaskFormerPanopticDatasetMapper should only be used for training!"
+
+        dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
+        image = utils.read_image(dataset_dict["file_name"], format=self.img_format)
+        utils.check_image_size(dataset_dict, image)
+
+        # semantic segmentation
+        if "sem_seg_file_name" in dataset_dict:
+            # PyTorch transformation not implemented for uint16, so converting it to double first
+            sem_seg_gt = utils.read_image(dataset_dict.pop("sem_seg_file_name")).astype("double")
+        else:
+            sem_seg_gt = None
+
+        # panoptic segmentation
+        if "pan_seg_file_name" in dataset_dict:
+            pan_seg_gt = utils.read_image(dataset_dict.pop("pan_seg_file_name"), "RGB")
+            segments_info = dataset_dict["segments_info"]
+        else:
+            pan_seg_gt = None
+            segments_info = None
+
+        if pan_seg_gt is None:
+            raise ValueError(
+                "Cannot find 'pan_seg_file_name' for panoptic segmentation dataset {}.".format(
+                    dataset_dict["file_name"]
+                )
+            )
+
+        aug_input = T.AugInput(image, sem_seg=sem_seg_gt)
+        aug_input, transforms = T.apply_transform_gens(self.tfm_gens, aug_input)
+        image = aug_input.image
+        if sem_seg_gt is not None:
+            sem_seg_gt = aug_input.sem_seg
+
+        # apply the same transformation to panoptic segmentation
+        pan_seg_gt = transforms.apply_segmentation(pan_seg_gt)
+
+        from panopticapi.utils import rgb2id
+
+        pan_seg_gt = rgb2id(pan_seg_gt)
+
+        # Pad image and segmentation label here!
+        image = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))
+        if sem_seg_gt is not None:
+            sem_seg_gt = torch.as_tensor(sem_seg_gt.astype("long"))
+        pan_seg_gt = torch.as_tensor(pan_seg_gt.astype("long"))
+
+        if self.size_divisibility > 0:
+            image_size = (image.shape[-2], image.shape[-1])
+            padding_size = [
+                0,
+                self.size_divisibility - image_size[1],
+                0,
+                self.size_divisibility - image_size[0],
+            ]
+            image = F.pad(image, padding_size, value=128).contiguous()
+            if sem_seg_gt is not None:
+                sem_seg_gt = F.pad(sem_seg_gt, padding_size, value=self.ignore_label).contiguous()
+            pan_seg_gt = F.pad(
+                pan_seg_gt, padding_size, value=0
+            ).contiguous()  # 0 is the VOID panoptic label
+
+        image_shape = (image.shape[-2], image.shape[-1])  # h, w
+
+        # Pytorch's dataloader is efficient on torch.Tensor due to shared-memory,
+        # but not efficient on large generic data structures due to the use of pickle & mp.Queue.
+        # Therefore it's important to use torch.Tensor.
+        dataset_dict["image"] = image
+        if sem_seg_gt is not None:
+            dataset_dict["sem_seg"] = sem_seg_gt.long()
+
+        if "annotations" in dataset_dict:
+            raise ValueError("Pemantic segmentation dataset should not have 'annotations'.")
+
+        # Prepare per-category binary masks
+        pan_seg_gt = pan_seg_gt.numpy()
+        instances = Instances(image_shape)
+        classes = []
+        masks = []
+        for segment_info in segments_info:
+            class_id = segment_info["category_id"]
+            if not segment_info["iscrowd"]:
+                classes.append(class_id)
+                masks.append(pan_seg_gt == segment_info["id"])
+
+        classes = np.array(classes)
+        instances.gt_classes = torch.tensor(classes, dtype=torch.int64)
+        if len(masks) == 0:
+            # Some image does not have annotation (all ignored)
+            instances.gt_masks = torch.zeros((0, pan_seg_gt.shape[-2], pan_seg_gt.shape[-1]))
+        else:
+            masks = BitMasks(
+                torch.stack([torch.from_numpy(np.ascontiguousarray(x.copy())) for x in masks])
+            )
+            instances.gt_masks = masks.tensor
+
+        dataset_dict["instances"] = instances
+
+        return dataset_dict

fcclip/data/dataset_mappers/mask_former_semantic_dataset_mapper.py

@@ -0,0 +1,184 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+import copy
+import logging
+
+import numpy as np
+import torch
+from torch.nn import functional as F
+
+from detectron2.config import configurable
+from detectron2.data import MetadataCatalog
+from detectron2.data import detection_utils as utils
+from detectron2.data import transforms as T
+from detectron2.projects.point_rend import ColorAugSSDTransform
+from detectron2.structures import BitMasks, Instances
+
+__all__ = ["MaskFormerSemanticDatasetMapper"]
+
+
+class MaskFormerSemanticDatasetMapper:
+    """
+    A callable which takes a dataset dict in Detectron2 Dataset format,
+    and map it into a format used by MaskFormer for semantic segmentation.
+
+    The callable currently does the following:
+
+    1. Read the image from "file_name"
+    2. Applies geometric transforms to the image and annotation
+    3. Find and applies suitable cropping to the image and annotation
+    4. Prepare image and annotation to Tensors
+    """
+
+    @configurable
+    def __init__(
+        self,
+        is_train=True,
+        *,
+        augmentations,
+        image_format,
+        ignore_label,
+        size_divisibility,
+    ):
+        """
+        NOTE: this interface is experimental.
+        Args:
+            is_train: for training or inference
+            augmentations: a list of augmentations or deterministic transforms to apply
+            image_format: an image format supported by :func:`detection_utils.read_image`.
+            ignore_label: the label that is ignored to evaluation
+            size_divisibility: pad image size to be divisible by this value
+        """
+        self.is_train = is_train
+        self.tfm_gens = augmentations
+        self.img_format = image_format
+        self.ignore_label = ignore_label
+        self.size_divisibility = size_divisibility
+
+        logger = logging.getLogger(__name__)
+        mode = "training" if is_train else "inference"
+        logger.info(f"[{self.__class__.__name__}] Augmentations used in {mode}: {augmentations}")
+
+    @classmethod
+    def from_config(cls, cfg, is_train=True):
+        # Build augmentation
+        augs = [
+            T.ResizeShortestEdge(
+                cfg.INPUT.MIN_SIZE_TRAIN,
+                cfg.INPUT.MAX_SIZE_TRAIN,
+                cfg.INPUT.MIN_SIZE_TRAIN_SAMPLING,
+            )
+        ]
+        if cfg.INPUT.CROP.ENABLED:
+            augs.append(
+                T.RandomCrop_CategoryAreaConstraint(
+                    cfg.INPUT.CROP.TYPE,
+                    cfg.INPUT.CROP.SIZE,
+                    cfg.INPUT.CROP.SINGLE_CATEGORY_MAX_AREA,
+                    cfg.MODEL.SEM_SEG_HEAD.IGNORE_VALUE,
+                )
+            )
+        if cfg.INPUT.COLOR_AUG_SSD:
+            augs.append(ColorAugSSDTransform(img_format=cfg.INPUT.FORMAT))
+        augs.append(T.RandomFlip())
+
+        # Assume always applies to the training set.
+        dataset_names = cfg.DATASETS.TRAIN
+        meta = MetadataCatalog.get(dataset_names[0])
+        ignore_label = meta.ignore_label
+
+        ret = {
+            "is_train": is_train,
+            "augmentations": augs,
+            "image_format": cfg.INPUT.FORMAT,
+            "ignore_label": ignore_label,
+            "size_divisibility": cfg.INPUT.SIZE_DIVISIBILITY,
+        }
+        return ret
+
+    def __call__(self, dataset_dict):
+        """
+        Args:
+            dataset_dict (dict): Metadata of one image, in Detectron2 Dataset format.
+
+        Returns:
+            dict: a format that builtin models in detectron2 accept
+        """
+        assert self.is_train, "MaskFormerSemanticDatasetMapper should only be used for training!"
+
+        dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
+        image = utils.read_image(dataset_dict["file_name"], format=self.img_format)
+        utils.check_image_size(dataset_dict, image)
+
+        if "sem_seg_file_name" in dataset_dict:
+            # PyTorch transformation not implemented for uint16, so converting it to double first
+            sem_seg_gt = utils.read_image(dataset_dict.pop("sem_seg_file_name")).astype("double")
+        else:
+            sem_seg_gt = None
+
+        if sem_seg_gt is None:
+            raise ValueError(
+                "Cannot find 'sem_seg_file_name' for semantic segmentation dataset {}.".format(
+                    dataset_dict["file_name"]
+                )
+            )
+
+        aug_input = T.AugInput(image, sem_seg=sem_seg_gt)
+        aug_input, transforms = T.apply_transform_gens(self.tfm_gens, aug_input)
+        image = aug_input.image
+        sem_seg_gt = aug_input.sem_seg
+
+        # Pad image and segmentation label here!
+        image = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))
+        if sem_seg_gt is not None:
+            sem_seg_gt = torch.as_tensor(sem_seg_gt.astype("long"))
+
+        if self.size_divisibility > 0:
+            image_size = (image.shape[-2], image.shape[-1])
+            padding_size = [
+                0,
+                self.size_divisibility - image_size[1],
+                0,
+                self.size_divisibility - image_size[0],
+            ]
+            image = F.pad(image, padding_size, value=128).contiguous()
+            if sem_seg_gt is not None:
+                sem_seg_gt = F.pad(sem_seg_gt, padding_size, value=self.ignore_label).contiguous()
+
+        image_shape = (image.shape[-2], image.shape[-1])  # h, w
+
+        # Pytorch's dataloader is efficient on torch.Tensor due to shared-memory,
+        # but not efficient on large generic data structures due to the use of pickle & mp.Queue.
+        # Therefore it's important to use torch.Tensor.
+        dataset_dict["image"] = image
+
+        if sem_seg_gt is not None:
+            dataset_dict["sem_seg"] = sem_seg_gt.long()
+
+        if "annotations" in dataset_dict:
+            raise ValueError("Semantic segmentation dataset should not have 'annotations'.")
+
+        # Prepare per-category binary masks
+        if sem_seg_gt is not None:
+            sem_seg_gt = sem_seg_gt.numpy()
+            instances = Instances(image_shape)
+            classes = np.unique(sem_seg_gt)
+            # remove ignored region
+            classes = classes[classes != self.ignore_label]
+            instances.gt_classes = torch.tensor(classes, dtype=torch.int64)
+
+            masks = []
+            for class_id in classes:
+                masks.append(sem_seg_gt == class_id)
+
+            if len(masks) == 0:
+                # Some image does not have annotation (all ignored)
+                instances.gt_masks = torch.zeros((0, sem_seg_gt.shape[-2], sem_seg_gt.shape[-1]))
+            else:
+                masks = BitMasks(
+                    torch.stack([torch.from_numpy(np.ascontiguousarray(x.copy())) for x in masks])
+                )
+                instances.gt_masks = masks.tensor
+
+            dataset_dict["instances"] = instances
+
+        return dataset_dict

fcclip/data/datasets/__init__.py

@@ -0,0 +1,15 @@
+from . import (
+    register_coco_panoptic_annos_semseg,
+    register_ade20k_panoptic,
+    register_cityscapes_panoptic,
+    register_mapillary_vistas_panoptic,
+    register_ade20k_full,
+    register_pascal_voc_20_semantic,
+    register_pascal_voc_21_semantic,
+    register_pascal_ctx_59_sem_seg,
+    register_pascal_ctx_459_sem_seg,
+    register_coco_instance,
+    register_ade20k_instance,
+    register_coco_stuff_164k,
+    openseg_classes
+)

fcclip/data/datasets/ade20k_150_with_prompt_eng.txt

@@ -0,0 +1,151 @@
+0:invalid_class_id
+1:wall,walls,brick wall,stone wall,interior wall
+2:building,buildings,edifice,edifices
+3:sky,clouds
+4:floor,flooring
+5:tree,trees
+6:ceiling
+7:road,route,street,roads,streets,routes
+8:bed,beds
+9:windowpane,window,windows
+10:grass,grass field
+11:cabinet,cabinets,wall mounted cabine
+12:sidewalk,pavement
+13:person,child,girl,boy,woman,man,people,children,girls,boys,women,men
+14:earth,ground
+15:door,double door,doors
+16:table,tables,tablecloth
+17:mountain,mount,mountains
+18:plant,flora,plant life,plants,bushes
+19:curtain,drape,drapery,mantle,pall
+20:chair,chairs
+21:car,automobile,cars
+22:water
+23:painting,picture,paintings,pictures,wallart,framed canvas
+24:sofa,couch,sofas,couches
+25:shelf,shelves
+26:house exterior
+27:sea,ocean
+28:mirror,mirrors
+29:rug,carpet,carpeting
+30:field
+31:armchair,armchairs
+32:seat,seats
+33:fence,fencing
+34:desk,desks
+35:rock,stone,rocks,stones
+36:wardrobe,closet,press,wardrobes,closets
+37:lamp,lamps
+38:bathtub,bathing tub,bath,tub
+39:railing,rail
+40:cushion,cushions
+41:pedestal
+42:box,boxes
+43:column,pillar
+44:signboard,sign,signboards,signs
+45:chest of drawers,chest,bureau,dresser
+46:counter
+47:sand
+48:sink
+49:skyscraper,skyscrapers
+50:fireplace,hearth,open fireplace
+51:refrigerator,icebox
+52:grandstand,covered stand
+53:path
+54:stairs,steps
+55:runway
+56:case,display case,showcase,vitrine
+57:pool table,billiard table,snooker table
+58:pillow,pillows
+59:screen door,shower door
+60:stairway,staircase
+61:river
+62:bridge,span
+63:bookcase
+64:window screen,door screen
+65:coffee table,cocktail table
+66:toilet,commode,crapper,potty
+67:flower,flowers
+68:book,books
+69:hill
+70:bench,benches
+71:countertop,counter top,worktop
+72:stove,kitchen stove,kitchen range,kitchen range,cooking stove
+73:palm tree,palm trees
+74:kitchen island
+75:computer,computing machine,computing device,data processor,electronic computer,information processing system
+76:swivel chair
+77:boat
+78:bar
+79:arcade machine,arcade machines
+80:hovel,hut,hutch,shack,shanty
+81:bus,autobus,double-decker,jitney,motorbus,motorcoach,omnibus,passenger vehicle
+82:towel
+83:light bulb,lightbulb,bulb,incandescent lamp,electric light,electric-light bulb
+84:truck,motortruck
+85:tower,towers
+86:chandelier,pendant,pendent
+87:awning,sunshade,sunblind
+88:streetlight,street lamp
+89:booth,cubicle,stall,kiosk
+90:television receiver,television,television set,tv,tv set
+91:airplane,aeroplane,airplanes,aeroplanes
+92:dirt track
+93:apparel,wearing apparel,dress,clothes
+94:pole
+95:land,soil
+96:bannister,banister,balustrade,balusters,handrail
+97:escalator,moving staircase,moving stairway
+98:ottoman,pouf,pouffe,puff,hassock
+99:bottle,bottles,water bottle
+100:buffet,sideboard
+101:poster,posting,placard,notice,bill,card
+102:stage
+103:van
+104:ship
+105:fountain
+106:conveyer belt,conveyor belt,conveyer,conveyor,transporter
+107:canopy
+108:washer,automatic washer,washing machine
+109:plaything,toy,toys
+110:swimming pool,swimming bath
+111:stool,stools
+112:barrel,cask,barrels,casks
+113:basket,handbasket
+114:waterfall,falls
+115:tent,collapsible shelter
+116:bag,bags,gift bag,paper bag
+117:minibike,motorbike
+118:cradle
+119:oven
+120:ball,balls
+121:food,solid food
+122:step,stair
+123:tank,storage tank
+124:trade name,brand name,brand,marque
+125:microwave,microwave oven
+126:plant pots,plant pot,flower pot,flowerpot,planter
+127:animal,animate being,dog,cat,horse,cow,sheep,zebra,girraffe,bird
+128:bicycle,bike
+129:lake
+130:dishwasher,dish washer,dishwashing machine
+131:projection screen
+132:blanket,cover
+133:sculpture,sculptures
+134:exhaust hood
+135:sconce,sconce lamp,sconce light
+136:vase,vases
+137:traffic light,traffic signal,traffic lights
+138:tray,trays
+139:ashcan,trash can,garbage can,wastebin,ash bin,ash-bin,ashbin,dustbin,trash barrel,trash bin
+140:ceiling fan,floor fan
+141:pier,wharf,wharfage,dock
+142:crt screen
+143:plate,plates
+144:monitor,monitoring device,monitors
+145:bulletin board,notice board
+146:shower
+147:radiator
+148:cup,cups,drinking glass,drinking glasses
+149:clock
+150:flag,flags

fcclip/data/datasets/ade20k_847_with_prompt_eng.txt

@@ -0,0 +1,848 @@
+0:invalid_class_id
+1:wall,walls,interior wall,brick wall,stone wall
+2:building,buildings,edifice,edifices
+3:sky,clouds
+4:tree,trees
+5:road,route,street,roads,streets,routes
+6:floor,flooring
+7:ceiling
+8:bed,beds
+9:sidewalk,pavement
+10:earth,ground
+11:cabinet,cabinets,wall mounted cabine
+12:person,child,girl,boy,woman,man,people,children,girls,boys,women,men
+13:grass,grass field
+14:windowpane,window,windows
+15:car,automobile,cars
+16:mountain,mount,mountains
+17:plant,flora,plant life,plants,bushes
+18:table,tables,tablecloth
+19:chair,chairs
+20:curtain,drape,drapery,mantle,pall
+21:door,double door,doors
+22:sofa,couch,sofas,couches
+23:sea,ocean
+24:painting,picture,paintings,pictures,wallart,framed canvas
+25:water
+26:mirror,mirrors
+27:house exterior
+28:rug,carpet,carpeting
+29:shelf,shelves
+30:armchair,armchairs
+31:fence,fencing
+32:field
+33:lamp,lamps
+34:rock,stone,rocks,stones
+35:seat,seats
+36:river
+37:desk,desks
+38:bathtub,bathing tub,bath,tub
+39:railing,rail
+40:signboard,sign,signboards,signs
+41:cushion,cushions
+42:path
+43:work surface
+44:stairs,steps
+45:column,pillar
+46:sink
+47:wardrobe,closet,press,wardrobes,closets
+48:snow
+49:refrigerator,icebox
+50:pedestal
+51:bridge,span
+52:blind
+53:runway
+54:cliff,drop,drop-off
+55:sand
+56:fireplace,hearth,open fireplace
+57:pillow,pillows
+58:screen door,shower door
+59:toilet,commode,crapper,potty
+60:skyscraper,skyscrapers
+61:grandstand,covered stand
+62:box,boxes
+63:pool table,billiard table,snooker table
+64:palm tree,palm trees
+65:double door
+66:coffee table,cocktail table
+67:counter
+68:countertop,counter top,worktop
+69:chest of drawers,chest,bureau,dresser
+70:kitchen island
+71:boat
+72:waterfall,falls
+73:stove,kitchen stove,kitchen range,kitchen range,cooking stove
+74:flower,flowers
+75:bookcase
+76:controls
+77:book,books
+78:stairway,staircase
+79:streetlight,street lamp
+80:computer,computing machine,computing device,data processor,electronic computer,information processing system
+81:bus,autobus,double-decker,jitney,motorbus,motorcoach,omnibus,passenger vehicle
+82:swivel chair
+83:light,light source
+84:bench,benches
+85:case,display case,showcase,vitrine
+86:towel
+87:fountain
+88:embankment
+89:television receiver,television,television set,tv,tv set
+90:van
+91:hill
+92:awning,sunshade,sunblind
+93:poster,posting,placard,notice,bill,card
+94:truck,motortruck
+95:airplane,aeroplane,airplanes,aeroplanes
+96:pole
+97:tower,towers
+98:court
+99:ball,balls
+100:aircraft carrier,carrier,flattop,attack aircraft carrier
+101:buffet,sideboard
+102:hovel,hut,hutch,shack,shanty
+103:apparel,wearing apparel,dress,clothes
+104:minibike,motorbike
+105:animal,animate being,dog,cat,horse,cow,sheep,zebra,giraffe,bird
+106:chandelier,pendant,pendent
+107:step,stair
+108:booth,cubicle,stall,kiosk
+109:bicycle,bike
+110:doorframe,doorcase
+111:sconce,sconce lamp,sconce light
+112:pond
+113:trade name,brand name
+114:bannister,banister,balustrade,balusters,handrail
+115:bag,bags,gift bag,paper bag
+116:traffic light,traffic signal,traffic lights
+117:gazebo
+118:escalator,moving staircase,moving stairway
+119:land,soil
+120:board,plank
+121:arcade machine,arcade machines
+122:eiderdown,duvet,continental quilt
+123:bar
+124:stall,stand,sales booth
+125:playground
+126:ship
+127:ottoman,pouf,pouffe,puff,hassock
+128:ashcan,trash can,garbage can,wastebin,ash bin,ash-bin,ashbin,dustbin,trash barrel,trash bin
+129:bottle,bottles,water bottle
+130:cradle
+131:pot,flowerpot
+132:conveyer belt,conveyor belt,conveyer,conveyor,transporter
+133:train,railroad train
+134:stool,stools
+135:lake
+136:tank,storage tank
+137:ice,water ice
+138:basket,handbasket
+139:manhole
+140:tent,collapsible shelter
+141:canopy
+142:microwave,microwave oven
+143:barrel,cask,barrels,casks
+144:dirt track
+145:beam
+146:dishwasher,dish washer,dishwashing machine
+147:plate,plates
+148:crt screen
+149:ruins
+150:washer,automatic washer,washing machine
+151:blanket,cover
+152:plaything,toy,toys
+153:food,solid food
+154:projection screen
+155:oven
+156:stage
+157:beacon,lighthouse,beacon light,pharos
+158:umbrella
+159:sculpture,sculptures
+160:aqueduct
+161:container
+162:scaffolding,staging
+163:exhaust hood
+164:curb,curbing,kerb
+165:roller coaster
+166:horse,equus caballus
+167:catwalk
+168:glass,drinking glass
+169:vase,vases
+170:central reservation
+171:carousel
+172:radiator
+173:closet
+174:machine
+175:pier,wharf,wharfage,dock
+176:ceiling fan,floor fan
+177:inflatable bounce game
+178:pitch
+179:paper
+180:arcade,colonnade
+181:hot tub
+182:helicopter
+183:tray,trays
+184:partition,divider
+185:vineyard
+186:bowl
+187:bullring
+188:flag,flags
+189:pot
+190:footbridge,overcrossing,pedestrian bridge
+191:shower
+192:bag,traveling bag,travelling bag,grip,suitcase
+193:bulletin board,notice board
+194:confessional booth
+195:trunk,tree trunk,bole
+196:forest
+197:elevator door
+198:laptop,laptop computer
+199:instrument panel
+200:bucket,pail
+201:tapestry,tapis
+202:platform
+203:jacket
+204:gate
+205:monitor,monitoring device,monitors
+206:telephone booth,phone booth,call box,telephone box,telephone kiosk
+207:spotlight,spot
+208:ring
+209:control panel
+210:blackboard,chalkboard
+211:air conditioner,air conditioning
+212:chest
+213:clock
+214:sand dune
+215:pipe,pipage,piping
+216:vault
+217:table football
+218:cannon
+219:swimming pool,swimming bath
+220:fluorescent,fluorescent fixture
+221:statue
+222:loudspeaker,speaker,speaker unit,loudspeaker system,speaker system
+223:exhibitor
+224:ladder
+225:carport
+226:dam
+227:pulpit
+228:skylight,fanlight
+229:water tower
+230:grill,grille,grillwork
+231:display board
+232:pane,pane of glass,window glass
+233:rubbish,trash,scrap
+234:ice rink
+235:fruit
+236:patio
+237:vending machine
+238:telephone,phone,telephone set
+239:net
+240:backpack,back pack,knapsack,packsack,rucksack,haversack
+241:jar
+242:track
+243:magazine
+244:shutter
+245:roof
+246:banner,streamer
+247:landfill
+248:post
+249:altarpiece,reredos
+250:hat,chapeau,lid
+251:arch,archway
+252:table game
+253:bag,handbag,pocketbook,purse
+254:document,written document,papers
+255:dome
+256:pier
+257:shanties
+258:forecourt
+259:crane
+260:dog,domestic dog,canis familiaris
+261:piano,pianoforte,forte-piano
+262:drawing
+263:cabin
+264:ad,advertisement,advertizement,advertising,advertizing,advert
+265:amphitheater,amphitheatre,coliseum
+266:monument
+267:henhouse
+268:cockpit
+269:heater,warmer
+270:windmill,aerogenerator,wind generator
+271:pool
+272:elevator,lift
+273:decoration,ornament,ornamentation
+274:labyrinth
+275:text,textual matter
+276:printer
+277:mezzanine,first balcony
+278:mattress
+279:straw
+280:stalls
+281:patio,terrace
+282:billboard,hoarding
+283:bus stop
+284:trouser,pant
+285:console table,console
+286:rack
+287:notebook
+288:shrine
+289:pantry
+290:cart
+291:steam shovel
+292:porch
+293:postbox,mailbox,letter box
+294:figurine,statuette
+295:recycling bin
+296:folding screen
+297:telescope
+298:deck chair,beach chair
+299:kennel
+300:coffee maker
+301:altar,communion table,lord's table
+302:fish
+303:easel
+304:artificial golf green
+305:iceberg
+306:candlestick,candle holder
+307:shower stall,shower bath
+308:television stand
+309:wall socket,wall plug,electric outlet,electrical outlet,outlet,electric receptacle
+310:skeleton
+311:grand piano,grand
+312:candy,confect
+313:grille door
+314:pedestal,plinth,footstall
+315:jersey,t-shirt,tee shirt
+316:shoe
+317:gravestone,headstone,tombstone
+318:shanty
+319:structure
+320:rocking chair,rocker
+321:bird
+322:place mat
+323:tomb
+324:big top
+325:gas pump,gasoline pump,petrol pump,island dispenser
+326:lockers
+327:cage
+328:finger
+329:bleachers
+330:ferris wheel
+331:hairdresser chair
+332:mat
+333:stands
+334:aquarium,fish tank,marine museum
+335:streetcar,tram,tramcar,trolley,trolley car
+336:napkin,table napkin,serviette
+337:dummy
+338:booklet,brochure,folder,leaflet,pamphlet
+339:sand trap
+340:shop,store
+341:table cloth
+342:service station
+343:coffin
+344:drawer
+345:cages
+346:slot machine,coin machine
+347:balcony
+348:volleyball court
+349:table tennis
+350:control table
+351:shirt
+352:merchandise,ware,product
+353:railway
+354:parterre
+355:chimney
+356:can,tin,tin can
+357:tanks
+358:fabric,cloth,material,textile
+359:alga,algae
+360:system
+361:map
+362:greenhouse
+363:mug
+364:barbecue
+365:trailer
+366:toilet tissue,toilet paper,bathroom tissue
+367:organ
+368:dishrag,dishcloth
+369:island
+370:keyboard
+371:trench
+372:basket,basketball hoop,hoop
+373:steering wheel,wheel
+374:pitcher,ewer
+375:goal
+376:bread,breadstuff,staff of life
+377:beds
+378:wood
+379:file cabinet
+380:newspaper,paper
+381:motorboat
+382:rope
+383:guitar
+384:rubble
+385:scarf
+386:barrels
+387:cap
+388:leaves
+389:control tower
+390:dashboard
+391:bandstand
+392:lectern
+393:switch,electric switch,electrical switch
+394:baseboard,mopboard,skirting board
+395:shower room
+396:smoke
+397:faucet,spigot
+398:bulldozer
+399:saucepan
+400:shops
+401:meter
+402:crevasse
+403:gear
+404:candelabrum,candelabra
+405:sofa bed
+406:tunnel
+407:pallet
+408:wire,conducting wire
+409:kettle,boiler
+410:bidet
+411:baby buggy,baby carriage,carriage,perambulator,pram,stroller,go-cart,pushchair,pusher
+412:music stand
+413:pipe,tube
+414:cup,cups,drinking glass,drinking glasses
+415:parking meter
+416:ice hockey rink
+417:shelter
+418:weeds
+419:temple
+420:patty,cake
+421:ski slope
+422:panel
+423:wallet
+424:wheel
+425:towel rack,towel horse
+426:roundabout
+427:canister,cannister,tin
+428:rod
+429:soap dispenser
+430:bell
+431:canvas
+432:box office,ticket office,ticket booth
+433:teacup
+434:trellis
+435:workbench
+436:valley,vale
+437:toaster
+438:knife
+439:podium
+440:ramp
+441:tumble dryer
+442:fireplug,fire hydrant,plug
+443:gym shoe,sneaker,tennis shoe
+444:lab bench
+445:equipment
+446:rocky formation
+447:plastic
+448:calendar
+449:caravan
+450:check-in-desk
+451:ticket counter
+452:brush
+453:mill
+454:covered bridge
+455:bowling alley
+456:hanger
+457:excavator
+458:trestle
+459:revolving door
+460:blast furnace
+461:scale,weighing machine
+462:projector
+463:soap
+464:locker
+465:tractor
+466:stretcher
+467:frame
+468:grating
+469:alembic
+470:candle,taper,wax light
+471:barrier
+472:cardboard
+473:cave
+474:puddle
+475:tarp
+476:price tag
+477:watchtower
+478:meters
+479:light bulb,bulb,bulbs
+480:tracks
+481:hair dryer
+482:skirt
+483:viaduct
+484:paper towel
+485:coat
+486:sheet
+487:fire extinguisher,extinguisher,asphyxiator
+488:water wheel
+489:pottery,clayware
+490:magazine rack
+491:teapot
+492:microphone,mike
+493:support
+494:forklift
+495:canyon
+496:cash register,register
+497:leaf,leafage,foliage
+498:remote control,remote
+499:soap dish
+500:windshield,windscreen
+501:cat
+502:cue,cue stick,pool cue,pool stick
+503:vent,venthole,vent-hole,blowhole
+504:videos
+505:shovel
+506:eaves
+507:antenna,aerial,transmitting aerial
+508:shipyard
+509:hen,biddy
+510:traffic cone
+511:washing machines
+512:truck crane
+513:cds
+514:niche
+515:scoreboard
+516:briefcase
+517:boot
+518:sweater,jumper
+519:hay
+520:pack
+521:bottle rack
+522:glacier
+523:pergola
+524:building materials
+525:television camera
+526:first floor
+527:rifle
+528:tennis table
+529:stadium
+530:safety belt
+531:cover
+532:dish rack
+533:synthesizer
+534:pumpkin
+535:gutter
+536:fruit stand
+537:ice floe,floe
+538:handle,grip,handgrip,hold
+539:wheelchair
+540:mousepad,mouse mat
+541:diploma
+542:fairground ride
+543:radio
+544:hotplate
+545:junk
+546:wheelbarrow
+547:stream
+548:toll plaza
+549:punching bag
+550:trough
+551:throne
+552:chair desk
+553:weighbridge
+554:extractor fan
+555:hanging clothes
+556:dish,dish aerial,dish antenna,saucer
+557:alarm clock,alarm
+558:ski lift
+559:chain
+560:garage
+561:mechanical shovel
+562:wine rack
+563:tramway
+564:treadmill
+565:menu
+566:block
+567:well
+568:witness stand
+569:branch
+570:duck
+571:casserole
+572:frying pan
+573:desk organizer
+574:mast
+575:spectacles,specs,eyeglasses,glasses
+576:service elevator
+577:dollhouse
+578:hammock
+579:clothes hanging
+580:photocopier
+581:notepad
+582:golf cart
+583:footpath
+584:cross
+585:baptismal font
+586:boiler
+587:skip
+588:rotisserie
+589:tables
+590:water mill
+591:helmet
+592:cover curtain
+593:brick
+594:table runner
+595:ashtray
+596:street box
+597:stick
+598:hangers
+599:cells
+600:urinal
+601:centerpiece
+602:portable fridge
+603:dvds
+604:golf club
+605:skirting board
+606:water cooler
+607:clipboard
+608:camera,photographic camera
+609:pigeonhole
+610:chips
+611:food processor
+612:post box
+613:lid
+614:drum
+615:blender
+616:cave entrance
+617:dental chair
+618:obelisk
+619:canoe
+620:mobile
+621:monitors
+622:pool ball
+623:cue rack
+624:baggage carts
+625:shore
+626:fork
+627:paper filer
+628:bicycle rack
+629:coat rack
+630:garland
+631:sports bag
+632:fish tank
+633:towel dispenser
+634:carriage
+635:brochure
+636:plaque
+637:stringer
+638:iron
+639:spoon
+640:flag pole
+641:toilet brush
+642:book stand
+643:water faucet,water tap,tap,hydrant
+644:ticket office
+645:broom
+646:dvd
+647:ice bucket
+648:carapace,shell,cuticle,shield
+649:tureen
+650:folders
+651:chess
+652:root
+653:sewing machine
+654:model
+655:pen
+656:violin
+657:sweatshirt
+658:recycling materials
+659:mitten
+660:chopping board,cutting board
+661:mask
+662:log
+663:mouse,computer mouse
+664:grill
+665:hole
+666:target
+667:trash bag
+668:chalk
+669:sticks
+670:balloon
+671:score
+672:hair spray
+673:roll
+674:runner
+675:engine
+676:inflatable glove
+677:games
+678:pallets
+679:baskets
+680:coop
+681:dvd player
+682:rocking horse
+683:buckets
+684:bread rolls
+685:shawl
+686:watering can
+687:spotlights
+688:post-it
+689:bowls
+690:security camera
+691:runner cloth
+692:lock
+693:alarm,warning device,alarm system
+694:side
+695:roulette
+696:bone
+697:cutlery
+698:pool balls
+699:wheels
+700:spice rack
+701:plant pots,plant pot,flower pot,flowerpot,planter
+702:towel ring
+703:bread box
+704:video
+705:funfair
+706:breads
+707:tripod
+708:ironing board
+709:skimmer
+710:hollow
+711:scratching post
+712:tricycle
+713:file box
+714:mountain pass
+715:tombstones
+716:cooker
+717:card game,cards
+718:golf bag
+719:towel paper
+720:chaise lounge
+721:sun
+722:toilet paper holder
+723:rake
+724:key
+725:umbrella stand
+726:dartboard
+727:transformer
+728:fireplace utensils
+729:sweatshirts
+730:cellular telephone,cellular phone,cellphone,cell,mobile phone
+731:tallboy
+732:stapler
+733:sauna
+734:test tube
+735:palette
+736:shopping carts
+737:tools
+738:push button,push,button
+739:star
+740:roof rack
+741:barbed wire
+742:spray
+743:ear
+744:sponge
+745:racket
+746:tins
+747:eyeglasses
+748:file
+749:scarfs
+750:sugar bowl
+751:flip flop
+752:headstones
+753:laptop bag
+754:leash
+755:climbing frame
+756:suit hanger
+757:floor spotlight
+758:plate rack
+759:sewer
+760:hard drive
+761:sprinkler
+762:tools box
+763:necklace
+764:bulbs
+765:steel industry
+766:club
+767:jack
+768:door bars
+769:control panel,instrument panel,control board,board,panel
+770:hairbrush
+771:napkin holder
+772:office
+773:smoke detector
+774:utensils
+775:apron
+776:scissors
+777:terminal
+778:grinder
+779:entry phone
+780:newspaper stand
+781:pepper shaker
+782:onions
+783:central processing unit,cpu,central processor,processor,mainframe
+784:tape
+785:bat
+786:coaster
+787:calculator
+788:potatoes
+789:luggage rack
+790:salt
+791:street number
+792:viewpoint
+793:sword
+794:cd
+795:rowing machine
+796:plug
+797:andiron,firedog,dog,dog-iron
+798:pepper
+799:tongs
+800:bonfire
+801:dog dish
+802:belt
+803:dumbbells
+804:videocassette recorder,vcr
+805:hook
+806:envelopes
+807:shower faucet
+808:watch
+809:padlock
+810:swimming pool ladder
+811:spanners
+812:gravy boat
+813:notice board
+814:trash bags
+815:fire alarm
+816:ladle
+817:stethoscope
+818:rocket
+819:funnel
+820:bowling pins
+821:valve
+822:thermometer
+823:cups
+824:spice jar
+825:night light
+826:soaps
+827:games table
+828:slotted spoon
+829:reel
+830:scourer
+831:sleeping robe
+832:desk mat
+833:dumbbell
+834:hammer
+835:tie
+836:typewriter
+837:shaker
+838:cheese dish
+839:sea star
+840:racquet
+841:butane gas cylinder
+842:paper weight
+843:shaving brush
+844:sunglasses
+845:gear shift
+846:towel rail
+847:adding machine,totalizer,totaliser

fcclip/data/datasets/cityscapes_with_prompt_eng.txt

@@ -0,0 +1,19 @@
+0:road,railroad
+1:sidewalk,pavement
+2:building,buildings,edifice,edifices,house,ceiling
+3:wall,walls,brick wall,stone wall,tile wall,wood wall
+4:fence,fences
+5:pole,poles
+6:traffic light,traffic lights
+7:traffic sign,stop sign
+8:vegetation,tree,trees,palm tree,bushes
+9:terrain,river,sand,sea,snow,water,mountain,grass,dirt,rock
+10:sky,clouds
+11:person
+12:rider
+13:car,cars
+14:truck,trucks
+15:bus,buses
+16:train,trains,locomotive,locomotives,freight train
+17:motorcycle,motorcycles
+18:bicycle,bicycles,bike,bikes

fcclip/data/datasets/coco_panoptic_with_prompt_eng.txt

@@ -0,0 +1,201 @@
+0:invalid_class_id
+1:person,child,girl,boy,woman,man,people,children,girls,boys,women,men,lady,guy,ladies,guys,clothes
+2:bicycle,bicycles,bike,bikes
+3:car,cars
+4:motorcycle,motorcycles
+5:airplane,airplanes
+6:bus,buses
+7:train,trains,locomotive,locomotives,freight train
+8:truck,trucks
+9:boat,boats
+10:traffic light
+11:fire hydrant
+12:invalid_class_id
+13:stop sign
+14:parking meter
+15:bench,benches
+16:bird,birds
+17:cat,cats,kitties,kitty
+18:dog,dogs,puppy,puppies
+19:horse,horses,foal
+20:sheep
+21:cow,cows,calf
+22:elephant,elephants
+23:bear,bears
+24:zebra,zebras
+25:giraffe,giraffes
+26:invalid_class_id
+27:backpack,backpacks
+28:umbrella,umbrellas
+29:invalid_class_id
+30:invalid_class_id
+31:handbag,handbags
+32:tie
+33:suitcase,suitcases
+34:frisbee
+35:skis
+36:snowboard
+37:sports ball
+38:kite,kites
+39:baseball bat
+40:baseball glove
+41:skateboard
+42:surfboard
+43:tennis racket
+44:bottle,bottles,water bottle
+45:invalid_class_id
+46:wine glass,wine glasses,wineglass
+47:cup,cups,water cup,water glass
+48:fork,forks
+49:knife,knives
+50:spoon,spoons
+51:bowl,bowls
+52:banana,bananas
+53:apple,apples,apple fruit
+54:sandwich,sandwiches
+55:orange fruit
+56:broccoli
+57:carrot,carrots
+58:hot dog
+59:pizza
+60:donut,donuts
+61:cake,cakes
+62:chair,chairs
+63:couch,sofa,sofas
+64:potted plant,potted plants,pottedplant,pottedplants,planter,planters
+65:bed,beds
+66:invalid_class_id
+67:dining table,dining tables,diningtable,diningtables,plate,plates,diningtable tablecloth
+68:invalid_class_id
+69:invalid_class_id
+70:toilet
+71:invalid_class_id
+72:tv
+73:laptop
+74:mouse
+75:tv remote,remote control
+76:keyboard
+77:cell phone,mobile
+78:microwave
+79:oven,ovens
+80:toaster
+81:sink,sinks
+82:refrigerator,fridge
+83:invalid_class_id
+84:book,books
+85:clock
+86:vase,vases
+87:scissor,scissors
+88:teddy bear,teddy bears
+89:hair drier
+90:toothbrush,toothbrushes
+91:invalid_class_id
+92:banner,banners
+93:blanket,blankets
+94:invalid_class_id
+95:bridge
+96:invalid_class_id
+97:invalid_class_id
+98:invalid_class_id
+99:invalid_class_id
+100:cardboard
+101:invalid_class_id
+102:invalid_class_id
+103:invalid_class_id
+104:invalid_class_id
+105:invalid_class_id
+106:invalid_class_id
+107:counter
+108:invalid_class_id
+109:curtain,curtains
+110:invalid_class_id
+111:invalid_class_id
+112:door,doors
+113:invalid_class_id
+114:invalid_class_id
+115:invalid_class_id
+116:invalid_class_id
+117:invalid_class_id
+118:wood floor
+119:flower,flowers
+120:invalid_class_id
+121:invalid_class_id
+122:fruit,fruits
+123:invalid_class_id
+124:invalid_class_id
+125:gravel
+126:invalid_class_id
+127:invalid_class_id
+128:house
+129:invalid_class_id
+130:lamp,bulb,lamps,bulbs
+131:invalid_class_id
+132:invalid_class_id
+133:mirror
+134:invalid_class_id
+135:invalid_class_id
+136:invalid_class_id
+137:invalid_class_id
+138:tennis net
+139:invalid_class_id
+140:invalid_class_id
+141:pillow,pillows
+142:invalid_class_id
+143:invalid_class_id
+144:platform
+145:playingfield,tennis court,baseball field,soccer field,tennis field
+146:invalid_class_id
+147:railroad
+148:river
+149:road
+150:invalid_class_id
+151:roof
+152:invalid_class_id
+153:invalid_class_id
+154:sand
+155:sea,sea wave,wave,waves
+156:shelf
+157:invalid_class_id
+158:invalid_class_id
+159:snow
+160:invalid_class_id
+161:stairs
+162:invalid_class_id
+163:invalid_class_id
+164:invalid_class_id
+165:invalid_class_id
+166:tent
+167:invalid_class_id
+168:towel
+169:invalid_class_id
+170:invalid_class_id
+171:brick wall
+172:invalid_class_id
+173:invalid_class_id
+174:invalid_class_id
+175:stone wall
+176:tile wall
+177:wood wall
+178:water
+179:invalid_class_id
+180:window blind
+181:window
+182:invalid_class_id
+183:invalid_class_id
+184:tree,trees,palm tree,bushes
+185:fence,fences
+186:ceiling
+187:sky,clouds
+188:cabinet,cabinets
+189:table
+190:floor,flooring,tile floor
+191:pavement
+192:mountain,mountains
+193:grass
+194:dirt
+195:paper
+196:food
+197:building,buildings
+198:rock
+199:wall,walls
+200:rug

fcclip/data/datasets/coco_stuff_with_prompt_eng.txt

@@ -0,0 +1,183 @@
+0:invalid_class_id
+1:person,child,girl,boy,woman,man,people,children,girls,boys,women,men,lady,guy,ladies,guys
+2:bicycle,bicycles,bike,bikes
+3:car,cars
+4:motorcycle,motorcycles
+5:airplane,airplanes
+6:bus,buses
+7:train,trains,locomotive,locomotives,freight train
+8:truck,trucks
+9:boat,boats
+10:traffic light
+11:fire hydrant
+12:invalid_class_id
+13:stop sign
+14:parking meter
+15:bench,benches
+16:bird,birds
+17:cat,cats,kitties,kitty
+18:dog,dogs,puppy,puppies
+19:horse,horses,foal
+20:sheep
+21:cow,cows,calf
+22:elephant,elephants
+23:bear,bears
+24:zebra,zebras
+25:giraffe,giraffes
+26:invalid_class_id
+27:backpack,backpacks
+28:umbrella,umbrellas
+29:invalid_class_id
+30:invalid_class_id
+31:handbag,handbags
+32:tie
+33:suitcase,suitcases
+34:frisbee
+35:skis
+36:snowboard
+37:sports ball
+38:kite,kites
+39:baseball bat
+40:baseball glove
+41:skateboard
+42:surfboard
+43:tennis racket
+44:bottle,bottles,water bottle
+45:invalid_class_id
+46:wine glass,wine glasses,wineglass
+47:cup,cups,water cup,water glass
+48:fork,forks
+49:knife,knives
+50:spoon,spoons
+51:bowl,bowls
+52:banana,bananas
+53:apple,apples,apple fruit
+54:sandwich,sandwiches
+55:orange,oranges,orange fruit
+56:broccoli
+57:carrot,carrots
+58:hot dog
+59:pizza
+60:donut,donuts
+61:cake,cakes
+62:chair,chairs
+63:couch,sofa,sofas
+64:potted plant,potted plants,pottedplant,pottedplants,planter,planters
+65:bed,beds
+66:invalid_class_id
+67:dining table,dining tables,diningtable,diningtables,plate,plates,diningtable tablecloth
+68:invalid_class_id
+69:invalid_class_id
+70:toilet
+71:invalid_class_id
+72:tv
+73:laptop
+74:mouse
+75:remote,tv remote,remote control
+76:keyboard
+77:cell phone,mobile
+78:microwave
+79:oven,ovens
+80:toaster
+81:sink,sinks
+82:refrigerator,fridge
+83:invalid_class_id
+84:book,books
+85:clock
+86:vase,vases
+87:scissors,scissor
+88:teddy bear,teddy bears
+89:hair drier
+90:toothbrush,toothbrushes
+91:invalid_class_id
+92:banner,banners
+93:blanket,blankets
+94:branch
+95:bridge
+96:building,buildings
+97:bush,bushes
+98:cabinet,cabinets
+99:cage,cages
+100:cardboard
+101:carpet,carpets
+102:ceiling-other,ceiling
+103:ceiling-tile,ceiling tile
+104:cloth
+105:clothes
+106:clouds
+107:counter
+108:cupboard,cupboards
+109:curtain,curtains
+110:desk-stuff,desk,desks
+111:dirt
+112:door-stuff,door,doors
+113:fence,fences
+114:floor-marble,marble floor,floor marble
+115:floor-other,floor
+116:floor-stone,stone floor,floor stone
+117:floor-tile,tile floor,floor tile
+118:floor-wood,wood floor,floor wood
+119:flower,flowers
+120:fog
+121:food-other,food
+122:fruit,fruits
+123:furniture-other,furniture
+124:grass
+125:gravel
+126:ground-other,ground
+127:hill
+128:house
+129:leaves
+130:light
+131:mat
+132:metal
+133:mirror-stuff,mirror
+134:moss
+135:mountain,mountains
+136:mud
+137:napkin
+138:net
+139:paper
+140:pavement
+141:pillow,pillows
+142:plant-other
+143:plastic
+144:platform
+145:playingfield,tennis court,baseball field,soccer field,tennis field
+146:railing
+147:railroad
+148:river
+149:road
+150:rock
+151:roof
+152:rug
+153:salad
+154:sand
+155:sea,sea wave,wave,waves
+156:shelf
+157:sky-other,sky
+158:skyscraper
+159:snow
+160:solid-other,solid
+161:stairs
+162:stone
+163:straw
+164:structural-other,structural
+165:table
+166:tent
+167:textile-other,textile
+168:towel
+169:tree,trees,palm tree
+170:vegetable
+171:wall-brick,brick wall,wall brick
+172:wall-concrete,concrete wall,wall concrete
+173:wall-other,wall
+174:wall-panel,wall panel,panel wall
+175:wall-stone,stone wall,wall stone
+176:wall-tile,wall tile,tile wall
+177:wall-wood,wood wall, wall wood
+178:water-other,water
+179:waterdrops
+180:window-blind,window blind
+181:window-other,window
+182:wood

fcclip/data/datasets/lvis_1203_with_prompt_eng.txt

@@ -0,0 +1,1203 @@
+1:aerosol can,spray can
+2:air conditioner
+3:airplane,aeroplane
+4:alarm clock
+5:alcohol,alcoholic beverage
+6:alligator,gator
+7:almond
+8:ambulance
+9:amplifier
+10:anklet,ankle bracelet
+11:antenna,aerial,transmitting aerial
+12:apple
+13:applesauce
+14:apricot
+15:apron
+16:aquarium,fish tank
+17:arctic (type of shoe),galosh,golosh,rubber (type of shoe),gumshoe
+18:armband
+19:armchair
+20:armoire
+21:armor,armour
+22:artichoke
+23:trash can,garbage can,wastebin,dustbin,trash barrel,trash bin
+24:ashtray
+25:asparagus
+26:atomizer,atomiser,spray,sprayer,nebulizer,nebuliser
+27:avocado
+28:award,accolade
+29:awning
+30:ax,axe
+31:baboon
+32:baby buggy,baby carriage,perambulator,pram,stroller
+33:basketball backboard
+34:backpack,knapsack,packsack,rucksack,haversack
+35:handbag,purse,pocketbook
+36:suitcase,baggage,luggage
+37:bagel,beigel
+38:bagpipe
+39:baguet,baguette
+40:bait,lure
+41:ball
+42:ballet skirt,tutu
+43:balloon
+44:bamboo
+45:banana
+46:Band Aid
+47:bandage
+48:bandanna,bandana
+49:banjo
+50:banner,streamer
+51:barbell
+52:barge
+53:barrel,cask
+54:barrette
+55:barrow,garden cart,lawn cart,wheelbarrow
+56:baseball base
+57:baseball
+58:baseball bat
+59:baseball cap,jockey cap,golf cap
+60:baseball glove,baseball mitt
+61:basket,handbasket
+62:basketball
+63:bass horn,sousaphone,tuba
+64:bat (animal)
+65:bath mat
+66:bath towel
+67:bathrobe
+68:bathtub,bathing tub
+69:batter (food)
+70:battery
+71:beachball
+72:bead
+73:bean curd,tofu
+74:beanbag
+75:beanie,beany
+76:bear
+77:bed
+78:bedpan
+79:bedspread,bedcover,bed covering,counterpane,spread
+80:cow
+81:beef (food),boeuf (food)
+82:beeper,pager
+83:beer bottle
+84:beer can
+85:beetle
+86:bell
+87:bell pepper,capsicum
+88:belt
+89:belt buckle
+90:bench
+91:beret
+92:bib
+93:Bible
+94:bicycle,bike (bicycle)
+95:visor,vizor
+96:billboard
+97:binder,ring-binder
+98:binoculars,field glasses,opera glasses
+99:bird
+100:birdfeeder
+101:birdbath
+102:birdcage
+103:birdhouse
+104:birthday cake
+105:birthday card
+106:pirate flag
+107:black sheep
+108:blackberry
+109:blackboard,chalkboard
+110:blanket
+111:blazer,sport jacket,sport coat,sports jacket,sports coat
+112:blender,liquidizer,liquidiser
+113:blimp
+114:blinker,flasher
+115:blouse
+116:blueberry
+117:gameboard
+118:boat,ship (boat)
+119:bob,bobber,bobfloat
+120:bobbin,spool,reel
+121:bobby pin,hairgrip
+122:boiled egg,coddled egg
+123:bolo tie,bolo,bola tie,bola
+124:deadbolt
+125:bolt
+126:bonnet
+127:book
+128:bookcase
+129:booklet,brochure,leaflet,pamphlet
+130:bookmark,bookmarker
+131:boom microphone,microphone boom
+132:boot
+133:bottle
+134:bottle opener
+135:bouquet
+136:bow (weapon)
+137:bow (decorative ribbons)
+138:bow-tie,bowtie
+139:bowl
+140:pipe bowl
+141:bowler hat,bowler,derby hat,derby,plug hat
+142:bowling ball
+143:box
+144:boxing glove
+145:suspenders
+146:bracelet,bangle
+147:brass plaque
+148:brassiere,bra,bandeau
+149:bread-bin,breadbox
+150:bread
+151:breechcloth,breechclout,loincloth
+152:bridal gown,wedding gown,wedding dress
+153:briefcase
+154:broccoli
+155:broach
+156:broom
+157:brownie
+158:brussels sprouts
+159:bubble gum
+160:bucket,pail
+161:horse buggy
+162:horned cow
+163:bulldog
+164:bulldozer,dozer
+165:bullet train
+166:bulletin board,notice board
+167:bulletproof vest
+168:bullhorn,megaphone
+169:bun,roll
+170:bunk bed
+171:buoy
+172:burrito
+173:bus (vehicle),autobus,charabanc,double-decker,motorbus,motorcoach
+174:business card
+175:butter
+176:butterfly
+177:button
+178:cab (taxi),taxi,taxicab
+179:cabana
+180:cabin car,caboose
+181:cabinet
+182:locker,storage locker
+183:cake
+184:calculator
+185:calendar
+186:calf
+187:camcorder
+188:camel
+189:camera
+190:camera lens
+191:camper (vehicle),camping bus,motor home
+192:can,tin can
+193:can opener,tin opener
+194:candle,candlestick
+195:candle holder
+196:candy bar
+197:candy cane
+198:walking cane
+199:canister,cannister
+200:canoe
+201:cantaloup,cantaloupe
+202:canteen
+203:cap (headwear)
+204:bottle cap,cap (container lid)
+205:cape
+206:cappuccino,coffee cappuccino
+207:car (automobile),auto (automobile),automobile
+208:railcar (part of a train),railway car (part of a train),railroad car (part of a train)
+209:elevator car
+210:car battery,automobile battery
+211:identity card
+212:card
+213:cardigan
+214:cargo ship,cargo vessel
+215:carnation
+216:horse carriage
+217:carrot
+218:tote bag
+219:cart
+220:carton
+221:cash register,register (for cash transactions)
+222:casserole
+223:cassette
+224:cast,plaster cast,plaster bandage
+225:cat
+226:cauliflower
+227:cayenne (spice),cayenne pepper (spice),red pepper (spice)
+228:CD player
+229:celery
+230:cellular telephone,cellular phone,cellphone,mobile phone,smart phone
+231:chain mail,ring mail,chain armor,chain armour,ring armor,ring armour
+232:chair
+233:chaise longue,chaise,daybed
+234:chalice
+235:chandelier
+236:chap
+237:checkbook,chequebook
+238:checkerboard
+239:cherry
+240:chessboard
+241:chicken (animal)
+242:chickpea,garbanzo
+243:chili (vegetable),chili pepper (vegetable),chilli (vegetable),chilly (vegetable),chile (vegetable)
+244:chime,gong
+245:chinaware
+246:crisp (potato chip),potato chip
+247:poker chip
+248:chocolate bar
+249:chocolate cake
+250:chocolate milk
+251:chocolate mousse
+252:choker,collar,neckband
+253:chopping board,cutting board,chopping block
+254:chopstick
+255:Christmas tree
+256:slide
+257:cider,cyder
+258:cigar box
+259:cigarette
+260:cigarette case,cigarette pack
+261:cistern,water tank
+262:clarinet
+263:clasp
+264:cleansing agent,cleanser,cleaner
+265:cleat (for securing rope)
+266:clementine
+267:clip
+268:clipboard
+269:clippers (for plants)
+270:cloak
+271:clock,timepiece,timekeeper
+272:clock tower
+273:clothes hamper,laundry basket,clothes basket
+274:clothespin,clothes peg
+275:clutch bag
+276:coaster
+277:coat
+278:coat hanger,clothes hanger,dress hanger
+279:coatrack,hatrack
+280:cock,rooster
+281:cockroach
+282:cocoa (beverage),hot chocolate (beverage),drinking chocolate
+283:coconut,cocoanut
+284:coffee maker,coffee machine
+285:coffee table,cocktail table
+286:coffeepot
+287:coil
+288:coin
+289:colander,cullender
+290:coleslaw,slaw
+291:coloring material,colouring material
+292:combination lock
+293:pacifier,teething ring
+294:comic book
+295:compass
+296:computer keyboard,keyboard (computer)
+297:condiment
+298:cone,traffic cone
+299:control,controller
+300:convertible (automobile)
+301:sofa bed
+302:cooker
+303:cookie,cooky,biscuit (cookie)
+304:cooking utensil
+305:cooler (for food),ice chest
+306:cork (bottle plug),bottle cork
+307:corkboard
+308:corkscrew,bottle screw
+309:edible corn,corn,maize
+310:cornbread
+311:cornet,horn,trumpet
+312:cornice,valance,valance board,pelmet
+313:cornmeal
+314:corset,girdle
+315:costume
+316:cougar,puma,catamount,mountain lion,panther
+317:coverall
+318:cowbell
+319:cowboy hat,ten-gallon hat
+320:crab (animal)
+321:crabmeat
+322:cracker
+323:crape,crepe,French pancake
+324:crate
+325:crayon,wax crayon
+326:cream pitcher
+327:crescent roll,croissant
+328:crib,cot
+329:crock pot,earthenware jar
+330:crossbar
+331:crouton
+332:crow
+333:crowbar,wrecking bar,pry bar
+334:crown
+335:crucifix
+336:cruise ship,cruise liner
+337:police cruiser,patrol car,police car,squad car
+338:crumb
+339:crutch
+340:cub (animal)
+341:cube,square block
+342:cucumber,cuke
+343:cufflink
+344:cup
+345:trophy cup
+346:cupboard,closet
+347:cupcake
+348:hair curler,hair roller,hair crimper
+349:curling iron
+350:curtain,drapery
+351:cushion
+352:cylinder
+353:cymbal
+354:dagger
+355:dalmatian
+356:dartboard
+357:date (fruit)
+358:deck chair,beach chair
+359:deer,cervid
+360:dental floss,floss
+361:desk
+362:detergent
+363:diaper
+364:diary,journal
+365:die,dice
+366:dinghy,dory,rowboat
+367:dining table
+368:tux,tuxedo
+369:dish
+370:dish antenna
+371:dishrag,dishcloth
+372:dishtowel,tea towel
+373:dishwasher,dishwashing machine
+374:dishwasher detergent,dishwashing detergent,dishwashing liquid,dishsoap
+375:dispenser
+376:diving board
+377:Dixie cup,paper cup
+378:dog
+379:dog collar
+380:doll
+381:dollar,dollar bill,one dollar bill
+382:dollhouse,doll's house
+383:dolphin
+384:domestic ass,donkey
+385:doorknob,doorhandle
+386:doormat,welcome mat
+387:doughnut,donut
+388:dove
+389:dragonfly
+390:drawer
+391:underdrawers,boxers,boxershorts
+392:dress,frock
+393:dress hat,high hat,opera hat,silk hat,top hat
+394:dress suit
+395:dresser
+396:drill
+397:drone
+398:dropper,eye dropper
+399:drum (musical instrument)
+400:drumstick
+401:duck
+402:duckling
+403:duct tape
+404:duffel bag,duffle bag,duffel,duffle
+405:dumbbell
+406:dumpster
+407:dustpan
+408:eagle
+409:earphone,earpiece,headphone
+410:earplug
+411:earring
+412:easel
+413:eclair
+414:eel
+415:egg,eggs
+416:egg roll,spring roll
+417:egg yolk,yolk (egg)
+418:eggbeater,eggwhisk
+419:eggplant,aubergine
+420:electric chair
+421:refrigerator
+422:elephant
+423:elk,moose
+424:envelope
+425:eraser
+426:escargot
+427:eyepatch
+428:falcon
+429:fan
+430:faucet,spigot,tap
+431:fedora
+432:ferret
+433:Ferris wheel
+434:ferry,ferryboat
+435:fig (fruit)
+436:fighter jet,fighter aircraft,attack aircraft
+437:figurine
+438:file cabinet,filing cabinet
+439:file (tool)
+440:fire alarm,smoke alarm
+441:fire engine,fire truck
+442:fire extinguisher,extinguisher
+443:fire hose
+444:fireplace
+445:fireplug,fire hydrant,hydrant
+446:first-aid kit
+447:fish
+448:fish (food)
+449:fishbowl,goldfish bowl
+450:fishing rod,fishing pole
+451:flag
+452:flagpole,flagstaff
+453:flamingo
+454:flannel
+455:flap
+456:flash,flashbulb
+457:flashlight,torch
+458:fleece
+459:flip-flop (sandal)
+460:flipper (footwear),fin (footwear)
+461:flower arrangement,floral arrangement
+462:flute glass,champagne flute
+463:foal
+464:folding chair
+465:food processor
+466:football (American)
+467:football helmet
+468:footstool,footrest
+469:fork
+470:forklift
+471:freight car
+472:French toast
+473:freshener,air freshener
+474:frisbee
+475:frog,toad,toad frog
+476:fruit juice
+477:frying pan,frypan,skillet
+478:fudge
+479:funnel
+480:futon
+481:gag,muzzle
+482:garbage
+483:garbage truck
+484:garden hose
+485:gargle,mouthwash
+486:gargoyle
+487:garlic,ail
+488:gasmask,respirator,gas helmet
+489:gazelle
+490:gelatin,jelly
+491:gemstone
+492:generator
+493:giant panda,panda,panda bear
+494:gift wrap
+495:ginger,gingerroot
+496:giraffe
+497:cincture,sash,waistband,waistcloth
+498:glass (drink container),drinking glass
+499:globe
+500:glove
+501:goat
+502:goggles
+503:goldfish
+504:golf club,golf-club
+505:golfcart
+506:gondola (boat)
+507:goose
+508:gorilla
+509:gourd
+510:grape
+511:grater
+512:gravestone,headstone,tombstone
+513:gravy boat,gravy holder
+514:green bean
+515:green onion,spring onion,scallion
+516:griddle
+517:grill,grille,grillwork,radiator grille
+518:grits,hominy grits
+519:grizzly,grizzly bear
+520:grocery bag
+521:guitar
+522:gull,seagull
+523:gun
+524:hairbrush
+525:hairnet
+526:hairpin
+527:halter top
+528:ham,jambon,gammon
+529:hamburger,beefburger,burger
+530:hammer
+531:hammock
+532:hamper
+533:hamster
+534:hair dryer
+535:hand glass,hand mirror
+536:hand towel,face towel
+537:handcart,pushcart,hand truck
+538:handcuff
+539:handkerchief
+540:handle,grip,handgrip
+541:handsaw,carpenter's saw
+542:hardback book,hardcover book
+543:harmonium,organ (musical instrument),reed organ (musical instrument)
+544:hat
+545:hatbox
+546:veil
+547:headband
+548:headboard
+549:headlight,headlamp
+550:headscarf
+551:headset
+552:headstall (for horses),headpiece (for horses)
+553:heart
+554:heater,warmer
+555:helicopter
+556:helmet
+557:heron
+558:highchair,feeding chair
+559:hinge
+560:hippopotamus
+561:hockey stick
+562:hog,pig
+563:home plate (baseball),home base (baseball)
+564:honey
+565:fume hood,exhaust hood
+566:hook
+567:hookah,narghile,nargileh,sheesha,shisha,water pipe
+568:hornet
+569:horse
+570:hose,hosepipe
+571:hot-air balloon
+572:hotplate
+573:hot sauce
+574:hourglass
+575:houseboat
+576:hummingbird
+577:hummus,humus,hommos,hoummos,humous
+578:polar bear
+579:icecream
+580:popsicle
+581:ice maker
+582:ice pack,ice bag
+583:ice skate
+584:igniter,ignitor,lighter
+585:inhaler,inhalator
+586:iPod
+587:iron (for clothing),smoothing iron (for clothing)
+588:ironing board
+589:jacket
+590:jam
+591:jar
+592:jean,blue jean,denim
+593:jeep,landrover
+594:jelly bean,jelly egg
+595:jersey,T-shirt,tee shirt
+596:jet plane,jet-propelled plane
+597:jewel,gem,precious stone
+598:jewelry,jewellery
+599:joystick
+600:jumpsuit
+601:kayak
+602:keg
+603:kennel,doghouse
+604:kettle,boiler
+605:key
+606:keycard
+607:kilt
+608:kimono
+609:kitchen sink
+610:kitchen table
+611:kite
+612:kitten,kitty
+613:kiwi fruit
+614:knee pad
+615:knife
+616:knitting needle
+617:knob
+618:knocker (on a door),doorknocker
+619:koala,koala bear
+620:lab coat,laboratory coat
+621:ladder
+622:ladle
+623:ladybug,ladybeetle,ladybird beetle
+624:lamb (animal)
+625:lamb-chop,lambchop
+626:lamp
+627:lamppost
+628:lampshade
+629:lantern
+630:lanyard,laniard
+631:laptop computer,notebook computer
+632:lasagna,lasagne
+633:latch
+634:lawn mower
+635:leather
+636:legging (clothing),leging (clothing),leg covering
+637:Lego,Lego set
+638:legume
+639:lemon
+640:lemonade
+641:lettuce
+642:license plate,numberplate
+643:life buoy,lifesaver,life belt,life ring
+644:life jacket,life vest
+645:lightbulb
+646:lightning rod,lightning conductor
+647:lime
+648:limousine
+649:lion
+650:lip balm
+651:liquor,spirits,hard liquor,liqueur,cordial
+652:lizard
+653:log
+654:lollipop
+655:speaker (stero equipment)
+656:loveseat
+657:machine gun
+658:magazine
+659:magnet
+660:mail slot
+661:mailbox (at home),letter box (at home)
+662:mallard
+663:mallet
+664:mammoth
+665:manatee
+666:mandarin orange
+667:manger,trough
+668:manhole
+669:map
+670:marker
+671:martini
+672:mascot
+673:mashed potato
+674:masher
+675:mask,facemask
+676:mast
+677:mat (gym equipment),gym mat
+678:matchbox
+679:mattress
+680:measuring cup
+681:measuring stick,ruler (measuring stick),measuring rod
+682:meatball
+683:medicine
+684:melon
+685:microphone
+686:microscope
+687:microwave oven
+688:milestone,milepost
+689:milk
+690:milk can
+691:milkshake
+692:minivan
+693:mint candy
+694:mirror
+695:mitten
+696:mixer (kitchen tool),stand mixer
+697:money
+698:monitor (computer equipment) computer monitor
+699:monkey
+700:motor
+701:motor scooter,scooter
+702:motor vehicle,automotive vehicle
+703:motorcycle
+704:mound (baseball),pitcher's mound
+705:mouse (computer equipment),computer mouse
+706:mousepad
+707:muffin
+708:mug
+709:mushroom
+710:music stool,piano stool
+711:musical instrument,instrument (musical)
+712:nailfile
+713:napkin,table napkin,serviette
+714:neckerchief
+715:necklace
+716:necktie,tie (necktie)
+717:needle
+718:nest
+719:newspaper,paper (newspaper)
+720:newsstand
+721:nightshirt,nightwear,sleepwear,nightclothes
+722:nosebag (for animals),feedbag
+723:noseband (for animals),nosepiece (for animals)
+724:notebook
+725:notepad
+726:nut
+727:nutcracker
+728:oar
+729:octopus (food)
+730:octopus (animal)
+731:oil lamp,kerosene lamp,kerosine lamp
+732:olive oil
+733:omelet,omelette
+734:onion
+735:orange (fruit)
+736:orange juice
+737:ostrich
+738:ottoman,pouf,pouffe,hassock
+739:oven
+740:overalls (clothing)
+741:owl
+742:packet
+743:inkpad,inking pad,stamp pad
+744:pad
+745:paddle,boat paddle
+746:padlock
+747:paintbrush
+748:painting
+749:pajamas,pyjamas
+750:palette,pallet
+751:pan (for cooking),cooking pan
+752:pan (metal container)
+753:pancake
+754:pantyhose
+755:papaya
+756:paper plate
+757:paper towel
+758:paperback book,paper-back book,softback book,soft-cover book
+759:paperweight
+760:parachute
+761:parakeet,parrakeet,parroket,paraquet,paroquet,parroquet
+762:parasail (sports)
+763:parasol,sunshade
+764:parchment
+765:parka,anorak
+766:parking meter
+767:parrot
+768:passenger car (part of a train),coach (part of a train)
+769:passenger ship
+770:passport
+771:pastry
+772:patty (food)
+773:pea (food)
+774:peach
+775:peanut butter
+776:pear
+777:peeler (tool for fruit and vegetables)
+778:wooden leg,pegleg
+779:pegboard
+780:pelican
+781:pen
+782:pencil
+783:pencil box,pencil case
+784:pencil sharpener
+785:pendulum
+786:penguin
+787:pennant
+788:penny (coin)
+789:pepper,peppercorn
+790:pepper mill,pepper grinder
+791:perfume
+792:persimmon
+793:person,baby,child,boy,girl,man,woman,human
+794:pet
+795:pew (church bench),church bench
+796:phonebook,telephone book,telephone directory
+797:phonograph record,phonograph recording,record (phonograph recording)
+798:piano
+799:pickle
+800:pickup truck
+801:pie
+802:pigeon
+803:piggy bank,penny bank
+804:pillow
+805:pin (non jewelry)
+806:pineapple
+807:pinecone
+808:ping-pong ball
+809:pinwheel
+810:tobacco pipe
+811:pipe,piping
+812:pistol,handgun
+813:pita (bread),pocket bread
+814:pitcher (vessel for liquid),ewer
+815:pitchfork
+816:pizza
+817:place mat
+818:plate
+819:platter
+820:playpen
+821:pliers,plyers
+822:plow (farm equipment),plough (farm equipment)
+823:plume
+824:pocket watch
+825:pocketknife
+826:poker (fire stirring tool),stove poker,fire hook
+827:pole,post
+828:polo shirt,sport shirt
+829:poncho
+830:pony
+831:pool table,billiard table,snooker table
+832:pop (soda),soda (pop),tonic,soft drink
+833:postbox (public),mailbox (public)
+834:postcard,postal card,mailing-card
+835:poster,placard
+836:pot
+837:flowerpot
+838:potato
+839:potholder
+840:pottery,clayware
+841:pouch
+842:power shovel,excavator,digger
+843:prawn,shrimp
+844:pretzel
+845:printer,printing machine
+846:projectile (weapon),missile
+847:projector
+848:propeller,propellor
+849:prune
+850:pudding
+851:puffer (fish),pufferfish,blowfish,globefish
+852:puffin
+853:pug-dog
+854:pumpkin
+855:puncher
+856:puppet,marionette
+857:puppy
+858:quesadilla
+859:quiche
+860:quilt,comforter
+861:rabbit
+862:race car,racing car
+863:racket,racquet
+864:radar
+865:radiator
+866:radio receiver,radio set,radio,tuner (radio)
+867:radish,daikon
+868:raft
+869:rag doll
+870:raincoat,waterproof jacket
+871:ram (animal)
+872:raspberry
+873:rat
+874:razorblade
+875:reamer (juicer),juicer,juice reamer
+876:rearview mirror
+877:receipt
+878:recliner,reclining chair,lounger (chair)
+879:record player,phonograph (record player),turntable
+880:reflector
+881:remote control
+882:rhinoceros
+883:rib (food)
+884:rifle
+885:ring
+886:river boat
+887:road map
+888:robe
+889:rocking chair
+890:rodent
+891:roller skate
+892:Rollerblade
+893:rolling pin
+894:root beer
+895:router (computer equipment)
+896:rubber band,elastic band
+897:runner (carpet)
+898:plastic bag,paper bag
+899:saddle (on an animal)
+900:saddle blanket,saddlecloth,horse blanket
+901:saddlebag
+902:safety pin
+903:sail
+904:salad
+905:salad plate,salad bowl
+906:salami
+907:salmon (fish)
+908:salmon (food)
+909:salsa
+910:saltshaker
+911:sandal (type of shoe)
+912:sandwich
+913:satchel
+914:saucepan
+915:saucer
+916:sausage
+917:sawhorse,sawbuck
+918:saxophone
+919:scale (measuring instrument)
+920:scarecrow,strawman
+921:scarf
+922:school bus
+923:scissors
+924:scoreboard
+925:scraper
+926:screwdriver
+927:scrubbing brush
+928:sculpture
+929:seabird,seafowl
+930:seahorse
+931:seaplane,hydroplane
+932:seashell
+933:sewing machine
+934:shaker
+935:shampoo
+936:shark
+937:sharpener
+938:Sharpie
+939:shaver (electric),electric shaver,electric razor
+940:shaving cream,shaving soap
+941:shawl
+942:shears
+943:sheep
+944:shepherd dog,sheepdog
+945:sherbert,sherbet
+946:shield
+947:shirt
+948:shoe,sneaker (type of shoe),tennis shoe
+949:shopping bag
+950:shopping cart
+951:short pants,shorts (clothing),trunks (clothing)
+952:shot glass
+953:shoulder bag
+954:shovel
+955:shower head
+956:shower cap
+957:shower curtain
+958:shredder (for paper)
+959:signboard
+960:silo
+961:sink
+962:skateboard
+963:skewer
+964:ski
+965:ski boot
+966:ski parka,ski jacket
+967:ski pole
+968:skirt
+969:skullcap
+970:sled,sledge,sleigh
+971:sleeping bag
+972:sling (bandage),triangular bandage
+973:slipper (footwear),carpet slipper (footwear)
+974:smoothie
+975:snake,serpent
+976:snowboard
+977:snowman
+978:snowmobile
+979:soap
+980:soccer ball
+981:sock
+982:sofa,couch,lounge
+983:softball
+984:solar array,solar battery,solar panel
+985:sombrero
+986:soup
+987:soup bowl
+988:soupspoon
+989:sour cream,soured cream
+990:soya milk,soybean milk,soymilk
+991:space shuttle
+992:sparkler (fireworks)
+993:spatula
+994:spear,lance
+995:spectacles,specs,eyeglasses,glasses
+996:spice rack
+997:spider
+998:crawfish,crayfish
+999:sponge
+1000:spoon
+1001:sportswear,athletic wear,activewear
+1002:spotlight
+1003:squid (food),calamari,calamary
+1004:squirrel
+1005:stagecoach
+1006:stapler (stapling machine)
+1007:starfish,sea star
+1008:statue (sculpture)
+1009:steak (food)
+1010:steak knife
+1011:steering wheel
+1012:stepladder
+1013:step stool
+1014:stereo (sound system)
+1015:stew
+1016:stirrer
+1017:stirrup
+1018:stool
+1019:stop sign
+1020:brake light
+1021:stove,kitchen stove,range (kitchen appliance),kitchen range,cooking stove
+1022:strainer
+1023:strap
+1024:straw (for drinking),drinking straw
+1025:strawberry
+1026:street sign
+1027:streetlight,street lamp
+1028:string cheese
+1029:stylus
+1030:subwoofer
+1031:sugar bowl
+1032:sugarcane (plant)
+1033:suit (clothing)
+1034:sunflower
+1035:sunglasses
+1036:sunhat
+1037:surfboard
+1038:sushi
+1039:mop
+1040:sweat pants
+1041:sweatband
+1042:sweater
+1043:sweatshirt
+1044:sweet potato
+1045:swimsuit,swimwear,bathing suit,swimming costume,bathing costume,swimming trunks,bathing trunks
+1046:sword
+1047:syringe
+1048:Tabasco sauce
+1049:table-tennis table,ping-pong table
+1050:table
+1051:table lamp
+1052:tablecloth
+1053:tachometer
+1054:taco
+1055:tag
+1056:taillight,rear light
+1057:tambourine
+1058:army tank,armored combat vehicle,armoured combat vehicle
+1059:tank (storage vessel),storage tank
+1060:tank top (clothing)
+1061:tape (sticky cloth or paper)
+1062:tape measure,measuring tape
+1063:tapestry
+1064:tarp
+1065:tartan,plaid
+1066:tassel
+1067:tea bag
+1068:teacup
+1069:teakettle
+1070:teapot
+1071:teddy bear
+1072:telephone,phone,telephone set
+1073:telephone booth,phone booth,call box,telephone box,telephone kiosk
+1074:telephone pole,telegraph pole,telegraph post
+1075:telephoto lens,zoom lens
+1076:television camera,tv camera
+1077:television set,tv,tv set
+1078:tennis ball
+1079:tennis racket
+1080:tequila
+1081:thermometer
+1082:thermos bottle
+1083:thermostat
+1084:thimble
+1085:thread,yarn
+1086:thumbtack,drawing pin,pushpin
+1087:tiara
+1088:tiger
+1089:tights (clothing),leotards
+1090:timer,stopwatch
+1091:tinfoil
+1092:tinsel
+1093:tissue paper
+1094:toast (food)
+1095:toaster
+1096:toaster oven
+1097:toilet
+1098:toilet tissue,toilet paper,bathroom tissue
+1099:tomato
+1100:tongs
+1101:toolbox
+1102:toothbrush
+1103:toothpaste
+1104:toothpick
+1105:cover
+1106:tortilla
+1107:tow truck
+1108:towel
+1109:towel rack,towel rail,towel bar
+1110:toy
+1111:tractor (farm equipment)
+1112:traffic light
+1113:dirt bike
+1114:trailer truck,tractor trailer,trucking rig,articulated lorry,semi truck
+1115:train (railroad vehicle),railroad train
+1116:trampoline
+1117:tray
+1118:trench coat
+1119:triangle (musical instrument)
+1120:tricycle
+1121:tripod
+1122:trousers,pants (clothing)
+1123:truck
+1124:truffle (chocolate),chocolate truffle
+1125:trunk
+1126:vat
+1127:turban
+1128:turkey (food)
+1129:turnip
+1130:turtle
+1131:turtleneck (clothing),polo-neck
+1132:typewriter
+1133:umbrella
+1134:underwear,underclothes,underclothing,underpants
+1135:unicycle
+1136:urinal
+1137:urn
+1138:vacuum cleaner
+1139:vase
+1140:vending machine
+1141:vent,blowhole,air vent
+1142:vest,waistcoat
+1143:videotape
+1144:vinegar
+1145:violin,fiddle
+1146:vodka
+1147:volleyball
+1148:vulture
+1149:waffle
+1150:waffle iron
+1151:wagon
+1152:wagon wheel
+1153:walking stick
+1154:wall clock
+1155:wall socket,wall plug,electric outlet,electrical outlet,outlet,electric receptacle
+1156:wallet,billfold
+1157:walrus
+1158:wardrobe
+1159:washbasin,basin (for washing),washbowl,washstand,handbasin
+1160:automatic washer,washing machine
+1161:watch,wristwatch
+1162:water bottle
+1163:water cooler
+1164:water faucet,water tap,tap (water faucet)
+1165:water heater,hot-water heater
+1166:water jug
+1167:water gun,squirt gun
+1168:water scooter,sea scooter,jet ski
+1169:water ski
+1170:water tower
+1171:watering can
+1172:watermelon
+1173:weathervane,vane (weathervane),wind vane
+1174:webcam
+1175:wedding cake,bridecake
+1176:wedding ring,wedding band
+1177:wet suit
+1178:wheel
+1179:wheelchair
+1180:whipped cream
+1181:whistle
+1182:wig
+1183:wind chime
+1184:windmill
+1185:window box (for plants)
+1186:windshield wiper,windscreen wiper,wiper (for windshield/screen)
+1187:windsock,air sock,air-sleeve,wind sleeve,wind cone
+1188:wine bottle
+1189:wine bucket,wine cooler
+1190:wineglass
+1191:blinder (for horses)
+1192:wok
+1193:wolf
+1194:wooden spoon
+1195:wreath
+1196:wrench,spanner
+1197:wristband
+1198:wristlet,wrist band
+1199:yacht
+1200:yogurt,yoghurt,yoghourt
+1201:yoke (animal equipment)
+1202:zebra
+1203:zucchini,courgette