Initial commit

.gitignore

@@ -0,0 +1,62 @@
+third_party/detectron2
+./models
+configs-experimental
+experiments
+# output dir
+index.html
+data/*
+slurm/
+slurm
+slurm-output
+slurm-output/
+output
+instant_test_output
+inference_test_output
+
+
+*.png
+*.diff
+*.jpg
+!/projects/DensePose/doc/images/*.jpg
+
+# compilation and distribution
+__pycache__
+_ext
+*.pyc
+*.pyd
+*.so
+*.dll
+*.egg-info/
+build/
+dist/
+wheels/
+
+# pytorch/python/numpy formats
+*.pth
+*.pkl
+*.ts
+model_ts*.txt
+
+# ipython/jupyter notebooks
+*.ipynb
+**/.ipynb_checkpoints/
+
+# Editor temporaries
+*.swn
+*.swo
+*.swp
+*~
+
+# editor settings
+.idea
+.vscode
+_darcs
+
+# project dirs
+/detectron2/model_zoo/configs
+/datasets/*
+!/datasets/*.*
+!/datasets/metadata
+/projects/*/datasets
+/models
+/snippet

.gitmodules

@@ -0,0 +1,6 @@
+[submodule "third_party/Deformable-DETR"]
+	path = third_party/Deformable-DETR
+	url = https://github.com/fundamentalvision/Deformable-DETR.git
+[submodule "third_party/CenterNet2"]
+	path = third_party/CenterNet2
+	url = https://github.com/xingyizhou/CenterNet2.git

CODE_OF_CONDUCT.md

@@ -0,0 +1,5 @@
+# Code of Conduct
+
+Facebook has adopted a Code of Conduct that we expect project participants to adhere to.
+Please read the [full text](https://code.fb.com/codeofconduct/)
+so that you can understand what actions will and will not be tolerated.

CONTRIBUTING.md

@@ -0,0 +1,39 @@
+# Contributing to Detic
+We want to make contributing to this project as easy and transparent as
+possible.
+
+## Our Development Process
+Minor changes and improvements will be released on an ongoing basis. Larger changes (e.g., changesets implementing a new paper) will be released on a more periodic basis.
+
+## Pull Requests
+We actively welcome your pull requests.
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Facebook's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
+disclosure of security bugs. In those cases, please go through the process
+outlined on that page and do not file a public issue.
+
+## Coding Style  
+* 4 spaces for indentation rather than tabs
+* 80 character line length
+* PEP8 formatting following [Black](https://black.readthedocs.io/en/stable/)
+
+## License
+By contributing to Detic, you agree that your contributions will be licensed
+under the LICENSE file in the root directory of this source tree.

LICENSE

@@ -0,0 +1,202 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction,
+and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by
+the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all
+other entities that control, are controlled by, or are under common
+control with that entity. For the purposes of this definition,
+"control" means (i) the power, direct or indirect, to cause the
+direction or management of such entity, whether by contract or
+otherwise, or (ii) ownership of fifty percent (50%) or more of the
+outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity
+exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications,
+including but not limited to software source code, documentation
+source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical
+transformation or translation of a Source form, including but
+not limited to compiled object code, generated documentation,
+and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or
+Object form, made available under the License, as indicated by a
+copyright notice that is included in or attached to the work
+(an example is provided in the Appendix below).
+
+"Derivative Works" shall mean any work, whether in Source or Object
+form, that is based on (or derived from) the Work and for which the
+editorial revisions, annotations, elaborations, or other modifications
+represent, as a whole, an original work of authorship. For the purposes
+of this License, Derivative Works shall not include works that remain
+separable from, or merely link (or bind by name) to the interfaces of,
+the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including
+the original version of the Work and any modifications or additions
+to that Work or Derivative Works thereof, that is intentionally
+submitted to Licensor for inclusion in the Work by the copyright owner
+or by an individual or Legal Entity authorized to submit on behalf of
+the copyright owner. For the purposes of this definition, "submitted"
+means any form of electronic, verbal, or written communication sent
+to the Licensor or its representatives, including but not limited to
+communication on electronic mailing lists, source code control systems,
+and issue tracking systems that are managed by, or on behalf of, the
+Licensor for the purpose of discussing and improving the Work, but
+excluding communication that is conspicuously marked or otherwise
+designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity
+on behalf of whom a Contribution has been received by Licensor and
+subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+copyright license to reproduce, prepare Derivative Works of,
+publicly display, publicly perform, sublicense, and distribute the
+Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+(except as stated in this section) patent license to make, have made,
+use, offer to sell, sell, import, and otherwise transfer the Work,
+where such license applies only to those patent claims licensable
+by such Contributor that are necessarily infringed by their
+Contribution(s) alone or by combination of their Contribution(s)
+with the Work to which such Contribution(s) was submitted. If You
+institute patent litigation against any entity (including a
+cross-claim or counterclaim in a lawsuit) alleging that the Work
+or a Contribution incorporated within the Work constitutes direct
+or contributory patent infringement, then any patent licenses
+granted to You under this License for that Work shall terminate
+as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+Work or Derivative Works thereof in any medium, with or without
+modifications, and in Source or Object form, provided that You
+meet the following conditions:
+
+(a) You must give any other recipients of the Work or
+Derivative Works a copy of this License; and
+
+(b) You must cause any modified files to carry prominent notices
+stating that You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works
+that You distribute, all copyright, patent, trademark, and
+attribution notices from the Source form of the Work,
+excluding those notices that do not pertain to any part of
+the Derivative Works; and
+
+(d) If the Work includes a "NOTICE" text file as part of its
+distribution, then any Derivative Works that You distribute must
+include a readable copy of the attribution notices contained
+within such NOTICE file, excluding those notices that do not
+pertain to any part of the Derivative Works, in at least one
+of the following places: within a NOTICE text file distributed
+as part of the Derivative Works; within the Source form or
+documentation, if provided along with the Derivative Works; or,
+within a display generated by the Derivative Works, if and
+wherever such third-party notices normally appear. The contents
+of the NOTICE file are for informational purposes only and
+do not modify the License. You may add Your own attribution
+notices within Derivative Works that You distribute, alongside
+or as an addendum to the NOTICE text from the Work, provided
+that such additional attribution notices cannot be construed
+as modifying the License.
+
+You may add Your own copyright statement to Your modifications and
+may provide additional or different license terms and conditions
+for use, reproduction, or distribution of Your modifications, or
+for any such Derivative Works as a whole, provided Your use,
+reproduction, and distribution of the Work otherwise complies with
+the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+any Contribution intentionally submitted for inclusion in the Work
+by You to the Licensor shall be under the terms and conditions of
+this License, without any additional terms or conditions.
+Notwithstanding the above, nothing herein shall supersede or modify
+the terms of any separate license agreement you may have executed
+with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+names, trademarks, service marks, or product names of the Licensor,
+except as required for reasonable and customary use in describing the
+origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+agreed to in writing, Licensor provides the Work (and each
+Contributor provides its Contributions) on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+implied, including, without limitation, any warranties or conditions
+of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+PARTICULAR PURPOSE. You are solely responsible for determining the
+appropriateness of using or redistributing the Work and assume any
+risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+whether in tort (including negligence), contract, or otherwise,
+unless required by applicable law (such as deliberate and grossly
+negligent acts) or agreed to in writing, shall any Contributor be
+liable to You for damages, including any direct, indirect, special,
+incidental, or consequential damages of any character arising as a
+result of this License or out of the use or inability to use the
+Work (including but not limited to damages for loss of goodwill,
+work stoppage, computer failure or malfunction, or any and all
+other commercial damages or losses), even if such Contributor
+has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+the Work or Derivative Works thereof, You may choose to offer,
+and charge a fee for, acceptance of support, warranty, indemnity,
+or other liability obligations and/or rights consistent with this
+License. However, in accepting such obligations, You may act only
+on Your own behalf and on Your sole responsibility, not on behalf
+of any other Contributor, and only if You agree to indemnify,
+defend, and hold each Contributor harmless for any liability
+incurred by, or claims asserted against, such Contributor by reason
+of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+To apply the Apache License to your work, attach the following
+boilerplate notice, with the fields enclosed by brackets "[]"
+replaced with your own identifying information. (Don't include
+the brackets!)  The text should be enclosed in the appropriate
+comment syntax for the file format. We also recommend that a
+file or class name and description of purpose be included on the
+same "printed page" as the copyright notice for easier
+identification within third-party archives.
+
+Copyright [yyyy] [name of copyright owner]
+
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

README.md

@@ -0,0 +1,107 @@
+# Detecting Twenty-thousand Classes using Image-level Supervision
+
+**Detic**: A **Det**ector with **i**mage **c**lasses that can use image-level labels to easily train detectors.
+
+<p align="center"> <img src='docs/teaser.jpeg' align="center" height="300px"> </p>
+
+> [**Detecting Twenty-thousand Classes using Image-level Supervision**](http://arxiv.org/abs/xxxx.xxxxx),               
+> Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr&auml;henb&uuml;hl, Ishan Misra,                 
+> *arXiv technical report ([arXiv xxxx.xxxxx](http://arxiv.org/abs/xxxx.xxxxx))*         
+
+
+## Features
+
+- Detects **any** class given class names (using [CLIP](https://github.com/openai/CLIP)).
+
+- We train the detector on ImageNet-21K dataset with 21K classes.
+
+- Cross-dataset generalization to OpenImages and Objects365 **without finetuning**. 
+
+- State-of-the-art results on Open-vocabulary LVIS and Open-vocabulary COCO.
+
+- Works for DETR-style detectors.
+
+
+## Installation
+
+See [installation instructions](docs/INSTALL.md).
+
+## Demo
+
+We use the default detectron2 [demo interface](https://github.com/facebookresearch/detectron2/blob/main/GETTING_STARTED.md). 
+For example, to run our [21K model](docs/MODEL_ZOO#cross-dataset-evaluation) on a [messy desk image](https://web.eecs.umich.edu/~fouhey/fun/desk/desk.jpg) (image credit [David Fouhey](https://web.eecs.umich.edu/~fouhey)) with the lvis vocabulary, run
+
+~~~
+mkdir models
+wget https://dl.fbaipublicfiles.com/detic/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth -O models/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth
+wget https://web.eecs.umich.edu/~fouhey/fun/desk/desk.jpg
+python demo.py --config-file configs/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.yaml --input desk.jpg --output out.jpg --vocabulary lvis --opts MODEL.WEIGHTS models/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth
+~~~
+
+If setup correctly, the output should look like:
+
+<p align="center"> <img src='docs/example_output_lvis.jpeg' align="center" height="450px"> </p>
+
+The same model can run with other vocabularies (COCO, OpenImages, or Objects365), or a **custom vocabulary**. For example:
+
+~~~
+python demo.py --config-file configs/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.yaml --input desk.jpg --output out2.jpg --vocabulary custom --custom_vocabulary headphone,webcam,paper,coffe --confidence-threshold 0.3 --opts MODEL.WEIGHTS models/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth
+~~~
+
+The output should look like:
+
+<p align="center"> <img src='docs/example_output_custom.jpeg' align="center" height="450px"> </p>
+
+Note that `headphone`, `paper` and `coffe` (typo intended) are **not** LVIS classes. Despite the misspelled class name, our detector can produce a reasonable detection for `coffe`.
+
+## Benchmark evaluation and training
+
+Please first [prepare datasets](datasets/README.md), then check our   [MODEL ZOO](docs/MODEL_ZOO.md) to reproduce results in our paper. We highlight key results below:
+
+- Open-vocabulary LVIS
+
+    |                       |  mask mAP | mask mAP_novel  |
+    |-----------------------|-----------|-----------------|
+    |Box-Supervised         | 30.2      |       16.4      |
+    |Detic                  | 32.4      |       24.9      |
+
+- Standard LVIS
+
+    |                       | Detector/ Backbone |  mask mAP | mask mAP_rare  |
+    |-----------------------|----------|-----------|-----------------|
+    |Box-Supervised         | CenterNet2-ResNet50 | 31.5      |       25.6      |
+    |Detic                  | CenterNet2-ResNet50 | 33.2      |       29.7      |
+    |Box-Supervised         | CenterNet2-SwinB    | 40.7      |       35.9      |
+    |Detic                  | CenterNet2-SwinB    | 41.7      |       41.7      |
+
+    |                       | Detector/ Backbone |  box mAP | box mAP_rare  |
+    |-----------------------|----------|-----------|-----------------|
+    |Box-Supervised         | DeformableDETR-ResNet50 | 31.7      |       21.4      |
+    |Detic                  | DeformableDETR-ResNet50 | 32.5      |       26.2      |
+
+- Cross-dataset generalization
+
+    |                       | Backbone |  Objects365 box mAP | OpenImages box mAP50  |
+    |-----------------------|----------|-----------|-----------------|
+    |Box-Supervised         | SwinB    | 19.1      |       46.2      |
+    |Detic                  | SwinB    | 21.4      |       55.2      |
+
+
+## License
+
+The majority of Detic is licensed under the [Apache 2.0 license](LICENSE), however portions of the project are available under separate license terms: SWIN-Transformer, CLIP, and TensorFlow Object Detection API are licensed under the MIT license; UniDet is licensed under the Apache 2.0 license; and the LVIS API is licensed under a custom license (https://github.com/lvis-dataset/lvis-api/blob/master/LICENSE)” If you later add other third party code, please keep this license info updated, and please let us know if that component is licensed under something other than CC-BY-NC, MIT, or CC0
+
+## Ethical Considerations
+Detic's wide range of detection capabilities may introduce similar challenges to many other visual recognition and open-set recognition methods.
+As the user can define arbitrary detection classes, class design and semantics may impact the model output.
+
+## Citation
+
+If you find this project useful for your research, please use the following BibTeX entry.
+
+    @inproceedings{zhou2021detecting,
+      title={Detecting Twenty-thousand Classes using Image-level Supervision},
+      author={Zhou, Xingyi and Girdhar, Rohit and Joulin, Armand and Kr{\"a}henb{\"u}hl, Philipp and Misra, Ishan},
+      booktitle={arXiv preprint arXiv:xxxx.xxxxx},
+      year={2021}
+    }

configs/Base-C2_L_R5021k_640b64_4x.yaml

@@ -0,0 +1,82 @@
+MODEL:
+  META_ARCHITECTURE: "CustomRCNN"
+  MASK_ON: True
+  PROPOSAL_GENERATOR:
+    NAME: "CenterNet"
+  WEIGHTS: "models/resnet50_miil_21k.pkl"
+  BACKBONE:
+    NAME: build_p67_timm_fpn_backbone
+  TIMM:
+    BASE_NAME: resnet50_in21k
+  FPN:
+    IN_FEATURES: ["layer3", "layer4", "layer5"]
+  PIXEL_MEAN: [123.675, 116.280, 103.530]
+  PIXEL_STD: [58.395, 57.12, 57.375]
+  ROI_HEADS:
+    NAME: DeticCascadeROIHeads
+    IN_FEATURES: ["p3", "p4", "p5"]
+    IOU_THRESHOLDS: [0.6]
+    NUM_CLASSES: 1203
+    SCORE_THRESH_TEST: 0.02
+    NMS_THRESH_TEST: 0.5
+  ROI_BOX_CASCADE_HEAD:
+    IOUS: [0.6, 0.7, 0.8]
+  ROI_BOX_HEAD:
+    NAME: "FastRCNNConvFCHead"
+    NUM_FC: 2
+    POOLER_RESOLUTION: 7
+    CLS_AGNOSTIC_BBOX_REG: True
+    MULT_PROPOSAL_SCORE: True
+
+    USE_SIGMOID_CE: True
+    USE_FED_LOSS: True
+  ROI_MASK_HEAD:
+    NAME: "MaskRCNNConvUpsampleHead"
+    NUM_CONV: 4
+    POOLER_RESOLUTION: 14
+    CLS_AGNOSTIC_MASK: True
+  CENTERNET:
+    NUM_CLASSES: 1203
+    REG_WEIGHT: 1.
+    NOT_NORM_REG: True
+    ONLY_PROPOSAL: True
+    WITH_AGN_HM: True
+    INFERENCE_TH: 0.0001
+    PRE_NMS_TOPK_TRAIN: 4000
+    POST_NMS_TOPK_TRAIN: 2000
+    PRE_NMS_TOPK_TEST: 1000
+    POST_NMS_TOPK_TEST: 256
+    NMS_TH_TRAIN: 0.9
+    NMS_TH_TEST: 0.9
+    POS_WEIGHT: 0.5
+    NEG_WEIGHT: 0.5
+    IGNORE_HIGH_FP: 0.85
+DATASETS:
+  TRAIN: ("lvis_v1_train",)
+  TEST: ("lvis_v1_val",)
+DATALOADER:
+  SAMPLER_TRAIN: "RepeatFactorTrainingSampler"
+  REPEAT_THRESHOLD: 0.001
+  NUM_WORKERS: 8
+TEST:
+  DETECTIONS_PER_IMAGE: 300
+SOLVER:
+  LR_SCHEDULER_NAME: "WarmupCosineLR"
+  CHECKPOINT_PERIOD: 1000000000
+  WARMUP_ITERS: 10000
+  WARMUP_FACTOR: 0.0001
+  USE_CUSTOM_SOLVER: True
+  OPTIMIZER: "ADAMW"
+  MAX_ITER: 90000
+  IMS_PER_BATCH: 64
+  BASE_LR: 0.0002
+  CLIP_GRADIENTS:
+    ENABLED: True
+INPUT:
+  FORMAT: RGB
+  CUSTOM_AUG: EfficientDetResizeCrop
+  TRAIN_SIZE: 640
+OUTPUT_DIR: "./output/Detic/auto"
+EVAL_PROPOSAL_AR: False
+VERSION: 2
+FP16: True

configs/Base-DeformDETR_L_R50_4x.yaml

@@ -0,0 +1,59 @@
+MODEL:
+  META_ARCHITECTURE: "DeformableDetr"
+  WEIGHTS: "detectron2://ImageNetPretrained/torchvision/R-50.pkl"
+  PIXEL_MEAN: [123.675, 116.280, 103.530]
+  PIXEL_STD: [58.395, 57.120, 57.375]
+  MASK_ON: False
+  RESNETS:
+    DEPTH: 50
+    STRIDE_IN_1X1: False
+    OUT_FEATURES: ["res3", "res4", "res5"]
+  DETR:
+    CLS_WEIGHT: 2.0
+    GIOU_WEIGHT: 2.0
+    L1_WEIGHT: 5.0
+    NUM_OBJECT_QUERIES: 300
+    DIM_FEEDFORWARD: 1024
+    WITH_BOX_REFINE: True
+    TWO_STAGE: True
+    NUM_CLASSES: 1203
+    USE_FED_LOSS: True
+DATASETS:
+  TRAIN: ("lvis_v1_train",)
+  TEST: ("lvis_v1_val",)
+SOLVER:
+  CHECKPOINT_PERIOD: 10000000
+  USE_CUSTOM_SOLVER: True
+  IMS_PER_BATCH: 32
+  BASE_LR: 0.0002
+  STEPS: (150000,)
+  MAX_ITER: 180000
+  WARMUP_FACTOR: 1.0
+  WARMUP_ITERS: 10
+  WEIGHT_DECAY: 0.0001
+  OPTIMIZER: "ADAMW"
+  BACKBONE_MULTIPLIER: 0.1
+  CLIP_GRADIENTS:
+    ENABLED: True
+    CLIP_TYPE: "full_model"
+    CLIP_VALUE: 0.01
+    NORM_TYPE: 2.0
+  CUSTOM_MULTIPLIER: 0.1
+  CUSTOM_MULTIPLIER_NAME: ['reference_points', 'sampling_offsets']
+INPUT:
+  FORMAT: "RGB"
+  MIN_SIZE_TRAIN: (480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800)
+  CROP:
+    ENABLED: True
+    TYPE: "absolute_range"
+    SIZE: (384, 600)
+  CUSTOM_AUG: "DETR"
+TEST:
+  DETECTIONS_PER_IMAGE: 300
+DATALOADER:
+  FILTER_EMPTY_ANNOTATIONS: False
+  NUM_WORKERS: 4
+  SAMPLER_TRAIN: "RepeatFactorTrainingSampler"
+  REPEAT_THRESHOLD: 0.001
+OUTPUT_DIR: "output/Detic/auto"
+VERSION: 2

configs/Base_OVCOCO_C4_1x.yaml

@@ -0,0 +1,31 @@
+MODEL:
+  META_ARCHITECTURE: "CustomRCNN"
+  RPN:
+    PRE_NMS_TOPK_TEST: 6000
+    POST_NMS_TOPK_TEST: 1000
+  ROI_HEADS:
+    NAME: "CustomRes5ROIHeads"
+  WEIGHTS: "detectron2://ImageNetPretrained/MSRA/R-50.pkl"
+  RESNETS:
+    DEPTH: 50
+  ROI_BOX_HEAD:
+    CLS_AGNOSTIC_BBOX_REG: True
+    USE_SIGMOID_CE: True
+    USE_ZEROSHOT_CLS: True
+    ZEROSHOT_WEIGHT_PATH: 'datasets/metadata/coco_clip_a+cname.npy'
+    IGNORE_ZERO_CATS: True
+    CAT_FREQ_PATH: 'datasets/coco/zero-shot/instances_train2017_seen_2_oriorder_cat_info.json'
+DATASETS:
+  TRAIN: ("coco_zeroshot_train_oriorder",)
+  TEST: ("coco_generalized_zeroshot_val",)
+SOLVER:
+  IMS_PER_BATCH: 16
+  BASE_LR: 0.02
+  STEPS: (60000, 80000)
+  MAX_ITER: 90000
+  CHECKPOINT_PERIOD: 1000000000
+INPUT:
+  MIN_SIZE_TRAIN: (800,)
+VERSION: 2
+OUTPUT_DIR: output/Detic-COCO/auto
+FP16: True

configs/BoxSup-C2_LCOCO_CLIP_SwinB_896b32_4x.yaml

@@ -0,0 +1,19 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+  WEIGHTS: "models/swin_base_patch4_window7_224_22k.pkl"
+  BACKBONE:
+    NAME: build_swintransformer_fpn_backbone
+  SWIN:
+    SIZE: B-22k
+  FPN:
+    IN_FEATURES: ["swin1", "swin2", "swin3"]
+SOLVER:
+  MAX_ITER: 180000
+  IMS_PER_BATCH: 32
+  BASE_LR: 0.0001
+INPUT:
+  TRAIN_SIZE: 896
+DATASETS:
+  TRAIN: ("lvis_v1_train+coco",)

configs/BoxSup-C2_L_CLIP_R5021k_640b64_4x.yaml

@@ -0,0 +1,4 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True

configs/BoxSup-C2_L_CLIP_SwinB_896b32_4x.yaml

@@ -0,0 +1,17 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+  WEIGHTS: "models/swin_base_patch4_window7_224_22k.pkl"
+  BACKBONE:
+    NAME: build_swintransformer_fpn_backbone
+  SWIN:
+    SIZE: B-22k
+  FPN:
+    IN_FEATURES: ["swin1", "swin2", "swin3"]
+SOLVER:
+  MAX_ITER: 180000
+  IMS_PER_BATCH: 32
+  BASE_LR: 0.0001
+INPUT:
+  TRAIN_SIZE: 896

configs/BoxSup-C2_Lbase_CLIP_R5021k_640b64_4x.yaml

@@ -0,0 +1,6 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+DATASETS:
+  TRAIN: ("lvis_v1_train_norare",)

configs/BoxSup-C2_Lbase_CLIP_SwinB_896b32_4x.yaml

@@ -0,0 +1,19 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+  WEIGHTS: "models/swin_base_patch4_window7_224_22k.pkl"
+  BACKBONE:
+    NAME: build_swintransformer_fpn_backbone
+  SWIN:
+    SIZE: B-22k
+  FPN:
+    IN_FEATURES: ["swin1", "swin2", "swin3"]
+SOLVER:
+  MAX_ITER: 180000
+  IMS_PER_BATCH: 32
+  BASE_LR: 0.0001
+INPUT:
+  TRAIN_SIZE: 896
+DATASETS:
+  TRAIN: ("lvis_v1_train_norare",)

configs/BoxSup-DeformDETR_L_R50_2x.yaml

@@ -0,0 +1,3 @@
+_BASE_: "Base-DeformDETR_L_R50_4x.yaml"
+SOLVER:
+  IMS_PER_BATCH: 16

configs/BoxSup-DeformDETR_L_R50_4x.yaml

@@ -0,0 +1 @@
+_BASE_: "Base-DeformDETR_L_R50_4x.yaml"

configs/BoxSup_OVCOCO_CLIP_R50_1x.yaml

@@ -0,0 +1 @@
+_BASE_: "Base_OVCOCO_C4_1x.yaml"

configs/Detic_DeformDETR_LI_R50_4x_ft4x.yaml

@@ -0,0 +1,22 @@
+_BASE_: "Base-DeformDETR_L_R50_4x.yaml"
+MODEL:
+  WEIGHTS: "models/BoxSup-DeformDETR_L_R50_4x.pth"
+INPUT:
+  CUSTOM_AUG: ResizeShortestEdge
+  MIN_SIZE_TRAIN_SAMPLING: range
+  MIN_SIZE_TRAIN: [480, 800]
+DATASETS:
+  TRAIN: ("lvis_v1_train","imagenet_lvis_v1")
+  TEST: ("lvis_v1_val",)
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [4, 16]
+  USE_RFS: [True, False]
+  DATASET_MIN_SIZES: [[480, 800], [240, 400]]
+  DATASET_MAX_SIZES: [1333, 667]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+WITH_IMAGE_LABELS: True

configs/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.yaml

@@ -0,0 +1,43 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  WEIGHTS: "models/BoxSup-C2_LCOCO_CLIP_SwinB_896b32_4x.pth"
+  DYNAMIC_CLASSIFIER: True
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+    IMAGE_LABEL_LOSS: 'max_size'
+    ZEROSHOT_WEIGHT_PATH: 'datasets/metadata/lvis-21k_clip_a+cname.npy'
+    USE_FED_LOSS: False # Federated loss is enabled when DYNAMIC_CLASSIFIER is on
+  ROI_HEADS:
+    NUM_CLASSES: 22047
+  BACKBONE:
+    NAME: build_swintransformer_fpn_backbone
+  SWIN:
+    SIZE: B-22k
+  FPN:
+    IN_FEATURES: ["swin1", "swin2", "swin3"]
+  RESET_CLS_TESTS: True
+  TEST_CLASSIFIERS: ("datasets/metadata/oid_clip_a+cname.npy","datasets/metadata/o365_clip_a+cnamefix.npy")
+  TEST_NUM_CLASSES: [500, 365]
+SOLVER:
+  MAX_ITER: 180000
+  IMS_PER_BATCH: 32
+  BASE_LR: 0.0001
+  WARMUP_ITERS: 1000
+  WARMUP_FACTOR: 0.001
+DATASETS:
+  TRAIN: ("lvis_v1_train+coco","imagenet_lvis-22k")
+  TEST: ('oid_val_expanded', 'objects365_v2_val')
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 16]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [4, 16]
+  DATASET_INPUT_SIZE: [896, 448]
+  USE_RFS: [True, False]
+  DATASET_INPUT_SCALE: [[0.1, 2.0], [0.5, 1.5]]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+  NUM_WORKERS: 4
+  USE_TAR_DATASET: True
+WITH_IMAGE_LABELS: True

configs/Detic_LI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.yaml

@@ -0,0 +1,43 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  WEIGHTS: "models/BoxSup-C2_L_CLIP_SwinB_896b32_4x.pth"
+  DYNAMIC_CLASSIFIER: True
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+    IMAGE_LABEL_LOSS: 'max_size'
+    ZEROSHOT_WEIGHT_PATH: 'datasets/metadata/lvis-21k_clip_a+cname.npy'
+    USE_FED_LOSS: False # Federated loss is enabled when DYNAMIC_CLASSIFIER is on
+  ROI_HEADS:
+    NUM_CLASSES: 22047
+  BACKBONE:
+    NAME: build_swintransformer_fpn_backbone
+  SWIN:
+    SIZE: B-22k
+  FPN:
+    IN_FEATURES: ["swin1", "swin2", "swin3"]
+  RESET_CLS_TESTS: True
+  TEST_CLASSIFIERS: ("datasets/metadata/oid_clip_a+cname.npy","datasets/metadata/o365_clip_a+cnamefix.npy")
+  TEST_NUM_CLASSES: [500, 365]
+SOLVER:
+  MAX_ITER: 180000
+  IMS_PER_BATCH: 32
+  BASE_LR: 0.0001
+  WARMUP_ITERS: 1000
+  WARMUP_FACTOR: 0.001
+DATASETS:
+  TRAIN: ("lvis_v1_train","imagenet_lvis-22k")
+  TEST: ('oid_val_expanded', 'objects365_v2_val')
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 16]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [4, 16]
+  DATASET_INPUT_SIZE: [896, 448]
+  USE_RFS: [True, False]
+  DATASET_INPUT_SCALE: [[0.1, 2.0], [0.5, 1.5]]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+  NUM_WORKERS: 4
+  USE_TAR_DATASET: True
+WITH_IMAGE_LABELS: True

configs/Detic_LI_CLIP_R5021k_640b64_4x_ft4x_max-size.yaml

@@ -0,0 +1,27 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+    IMAGE_LABEL_LOSS: 'max_size'
+  WEIGHTS: "models/BoxSup-C2_L_CLIP_R5021k_640b64_4x.pth"
+SOLVER:
+  MAX_ITER: 90000
+  IMS_PER_BATCH: 64
+  BASE_LR: 0.0002
+  WARMUP_ITERS: 1000
+  WARMUP_FACTOR: 0.001
+DATASETS:
+  TRAIN: ("lvis_v1_train","imagenet_lvis_v1")
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [8, 32]
+  DATASET_INPUT_SIZE: [640, 320]
+  USE_RFS: [True, False]
+  DATASET_INPUT_SCALE: [[0.1, 2.0], [0.5, 1.5]]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

configs/Detic_LI_CLIP_SwinB_896b32_4x_ft4x_max-size.yaml

@@ -0,0 +1,33 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+    IMAGE_LABEL_LOSS: 'max_size'
+  BACKBONE:
+    NAME: build_swintransformer_fpn_backbone
+  SWIN:
+    SIZE: B-22k
+  FPN:
+    IN_FEATURES: ["swin1", "swin2", "swin3"]
+  WEIGHTS: "models/BoxSup-C2_L_CLIP_SwinB_896b32_4x.pth"
+SOLVER:
+  MAX_ITER: 180000
+  IMS_PER_BATCH: 32
+  BASE_LR: 0.0001
+  WARMUP_ITERS: 1000
+  WARMUP_FACTOR: 0.001
+DATASETS:
+  TRAIN: ("lvis_v1_train","imagenet_lvis_v1")
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [4, 16]
+  DATASET_INPUT_SIZE: [896, 448]
+  USE_RFS: [True, False]
+  DATASET_INPUT_SCALE: [[0.1, 2.0], [0.5, 1.5]]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

configs/Detic_LbaseCCcapimg_CLIP_R5021k_640b64_4x_ft4x_max-size.yaml

@@ -0,0 +1,30 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  WITH_CAPTION: True
+  SYNC_CAPTION_BATCH: True
+  ROI_BOX_HEAD:
+    ADD_IMAGE_BOX: True # caption loss is added to the image-box
+    USE_ZEROSHOT_CLS: True
+    IMAGE_LABEL_LOSS: 'max_size'
+  WEIGHTS: "models/BoxSup-C2_Lbase_CLIP_R5021k_640b64_4x.pth"
+SOLVER:
+  MAX_ITER: 90000
+  IMS_PER_BATCH: 64
+  BASE_LR: 0.0002
+  WARMUP_ITERS: 1000
+  WARMUP_FACTOR: 0.001
+DATASETS:
+  TRAIN: ("lvis_v1_train_norare","cc3m_v1_train_tags")
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [8, 32]
+  DATASET_INPUT_SIZE: [640, 320]
+  USE_RFS: [True, False]
+  DATASET_INPUT_SCALE: [[0.1, 2.0], [0.5, 1.5]]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'captiontag']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

configs/Detic_LbaseCCimg_CLIP_R5021k_640b64_4x_ft4x_max-size.yaml

@@ -0,0 +1,27 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+    IMAGE_LABEL_LOSS: 'max_size'
+  WEIGHTS: "models/BoxSup-C2_Lbase_CLIP_R5021k_640b64_4x.pth"
+SOLVER:
+  MAX_ITER: 90000
+  IMS_PER_BATCH: 64
+  BASE_LR: 0.0002
+  WARMUP_ITERS: 1000
+  WARMUP_FACTOR: 0.001
+DATASETS:
+  TRAIN: ("lvis_v1_train_norare","cc3m_v1_train_tags")
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [8, 32]
+  DATASET_INPUT_SIZE: [640, 320]
+  USE_RFS: [True, False]
+  DATASET_INPUT_SCALE: [[0.1, 2.0], [0.5, 1.5]]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

configs/Detic_LbaseI_CLIP_R5021k_640b64_4x_ft4x_max-size.yaml

@@ -0,0 +1,27 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+    IMAGE_LABEL_LOSS: 'max_size'
+  WEIGHTS: "models/BoxSup-C2_Lbase_CLIP_R5021k_640b64_4x.pth"
+SOLVER:
+  MAX_ITER: 90000
+  IMS_PER_BATCH: 64
+  BASE_LR: 0.0002
+  WARMUP_ITERS: 1000
+  WARMUP_FACTOR: 0.001
+DATASETS:
+  TRAIN: ("lvis_v1_train_norare","imagenet_lvis_v1")
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [8, 32]
+  DATASET_INPUT_SIZE: [640, 320]
+  USE_RFS: [True, False]
+  DATASET_INPUT_SCALE: [[0.1, 2.0], [0.5, 1.5]]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

configs/Detic_LbaseI_CLIP_R5021k_640b64_4x_ft4x_predicted.yaml

@@ -0,0 +1,27 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+    IMAGE_LABEL_LOSS: 'max_score'
+  WEIGHTS: "models/BoxSup-C2_Lbase_CLIP_R5021k_640b64_4x.pth"
+SOLVER:
+  MAX_ITER: 90000
+  IMS_PER_BATCH: 64
+  BASE_LR: 0.0002
+  WARMUP_ITERS: 1000
+  WARMUP_FACTOR: 0.001
+DATASETS:
+  TRAIN: ("lvis_v1_train_norare","imagenet_lvis_v1")
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [8, 32]
+  DATASET_INPUT_SIZE: [640, 320]
+  USE_RFS: [True, False]
+  DATASET_INPUT_SCALE: [[0.1, 2.0], [0.5, 1.5]]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

configs/Detic_LbaseI_CLIP_SwinB_896b32_4x_ft4x_max-size.yaml

@@ -0,0 +1,33 @@
+_BASE_: "Base-C2_L_R5021k_640b64_4x.yaml"
+MODEL:
+  ROI_BOX_HEAD:
+    USE_ZEROSHOT_CLS: True
+    IMAGE_LABEL_LOSS: 'max_size'
+  BACKBONE:
+    NAME: build_swintransformer_fpn_backbone
+  SWIN:
+    SIZE: B-22k
+  FPN:
+    IN_FEATURES: ["swin1", "swin2", "swin3"]
+  WEIGHTS: "models/BoxSup-C2_Lbase_CLIP_SwinB_896b32_4x.pth"
+SOLVER:
+  MAX_ITER: 180000
+  IMS_PER_BATCH: 32
+  BASE_LR: 0.0001
+  WARMUP_ITERS: 1000
+  WARMUP_FACTOR: 0.001
+DATASETS:
+  TRAIN: ("lvis_v1_train_norare","imagenet_lvis_v1")
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [4, 16]
+  DATASET_INPUT_SIZE: [896, 448]
+  USE_RFS: [True, False]
+  DATASET_INPUT_SCALE: [[0.1, 2.0], [0.5, 1.5]]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

configs/Detic_OVCOCO_CLIP_R50_1x_caption.yaml

@@ -0,0 +1,33 @@
+_BASE_: "Base_OVCOCO_C4_1x.yaml"
+MODEL:
+  WEIGHTS: "models/BoxSup_OVCOCO_CLIP_R50_1x.pth"
+  WITH_CAPTION: True
+  SYNC_CAPTION_BATCH: True
+  ROI_BOX_HEAD:
+    WS_NUM_PROPS: 1
+    ADD_IMAGE_BOX: True
+    NEG_CAP_WEIGHT: 1.0
+SOLVER:
+  IMS_PER_BATCH: 16
+  BASE_LR: 0.02
+  STEPS: (60000, 80000)
+  MAX_ITER: 90000
+DATASETS:
+  TRAIN: ("coco_zeroshot_train_oriorder", "coco_caption_train_tags")
+INPUT:
+  CUSTOM_AUG: ResizeShortestEdge
+  MIN_SIZE_TRAIN_SAMPLING: range
+  MIN_SIZE_TRAIN: (800, 800)
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [2, 8]
+  USE_RFS: [False, False]
+  DATASET_MIN_SIZES: [[800, 800], [400, 400]]
+  DATASET_MAX_SIZES: [1333, 667]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'caption']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

configs/Detic_OVCOCO_CLIP_R50_1x_max-size.yaml

@@ -0,0 +1,30 @@
+_BASE_: "Base_OVCOCO_C4_1x.yaml"
+MODEL:
+  WEIGHTS: "models/BoxSup_OVCOCO_CLIP_R50_1x.pth"
+  ROI_BOX_HEAD:
+    WS_NUM_PROPS: 32
+    IMAGE_LABEL_LOSS: 'max_size'
+SOLVER:
+  IMS_PER_BATCH: 16
+  BASE_LR: 0.02
+  STEPS: (60000, 80000)
+  MAX_ITER: 90000
+DATASETS:
+  TRAIN: ("coco_zeroshot_train_oriorder", "coco_caption_train_tags")
+INPUT:
+  CUSTOM_AUG: ResizeShortestEdge
+  MIN_SIZE_TRAIN_SAMPLING: range
+  MIN_SIZE_TRAIN: (800, 800)
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [2, 8]
+  USE_RFS: [False, False]
+  DATASET_MIN_SIZES: [[800, 800], [400, 400]]
+  DATASET_MAX_SIZES: [1333, 667]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'image']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

configs/Detic_OVCOCO_CLIP_R50_1x_max-size_caption.yaml

@@ -0,0 +1,35 @@
+_BASE_: "Base_OVCOCO_C4_1x.yaml"
+MODEL:
+  WEIGHTS: "models/BoxSup_OVCOCO_CLIP_R50_1x.pth"
+  WITH_CAPTION: True
+  SYNC_CAPTION_BATCH: True
+  ROI_BOX_HEAD:
+    WS_NUM_PROPS: 32
+    ADD_IMAGE_BOX: True # caption loss is added to the image-box
+    IMAGE_LABEL_LOSS: 'max_size'
+
+    NEG_CAP_WEIGHT: 1.0
+SOLVER:
+  IMS_PER_BATCH: 16
+  BASE_LR: 0.02
+  STEPS: (60000, 80000)
+  MAX_ITER: 90000
+DATASETS:
+  TRAIN: ("coco_zeroshot_train_oriorder", "coco_caption_train_tags")
+INPUT:
+  CUSTOM_AUG: ResizeShortestEdge
+  MIN_SIZE_TRAIN_SAMPLING: range
+  MIN_SIZE_TRAIN: (800, 800)
+DATALOADER:
+  SAMPLER_TRAIN: "MultiDatasetSampler"
+  DATASET_RATIO: [1, 4]
+  USE_DIFF_BS_SIZE: True
+  DATASET_BS: [2, 8]
+  USE_RFS: [False, False]
+  DATASET_MIN_SIZES: [[800, 800], [400, 400]]
+  DATASET_MAX_SIZES: [1333, 667]
+  FILTER_EMPTY_ANNOTATIONS: False
+  MULTI_DATASET_GROUPING: True
+  DATASET_ANN: ['box', 'captiontag']
+  NUM_WORKERS: 8
+WITH_IMAGE_LABELS: True

datasets/README.md

@@ -0,0 +1,207 @@
+# Prepare datasets for Detic
+
+The basic training of our model uses [LVIS](https://www.lvisdataset.org/) (which uses [COCO](https://cocodataset.org/) images) and [ImageNet-21K](https://www.image-net.org/download.php). 
+Some models are trained on [Conceptual Caption (CC3M)](https://ai.google.com/research/ConceptualCaptions/).
+Optionally, we use [Objects365](https://www.objects365.org/) and [OpenImages (Challenge 2019 version)](https://storage.googleapis.com/openimages/web/challenge2019.html) for cross-dataset evaluation. 
+Before starting processing, please download the (selected) datasets from the official websites and place or sim-link them under `$Detic_ROOT/datasets/`. 
+
+```
+$Detic_ROOT/datasets/
+    metadata/
+    lvis/
+    coco/
+    imagenet/
+    cc3m/
+    objects365/
+    oid/
+```
+`metadata/` is our preprocessed meta-data (included in the repo). See the below [section](#Metadata) for details.
+Please follow the following instruction to pre-process individual datasets.
+
+### COCO and LVIS
+
+First, download COCO and LVIS data place them in the following way:
+
+```
+lvis/
+    lvis_v1_train.json
+    lvis_v1_val.json
+coco/
+    train2017/
+    val2017/
+    annotations/
+        captions_train2017.json
+        instances_train2017.json 
+        instances_val2017.json
+```
+
+Next, prepare the open-vocabulary LVIS training set using 
+
+```
+python tools/remove_lvis_rare.py --ann datasets/lvis/lvis_v1_train.json
+```
+
+This will generate `datasets/lvis/lvis_v1_train_norare.json`.
+
+### ImageNet-21K
+
+The ImageNet-21K folder should look like:
+```
+imagenet/
+    ImageNet-21K/
+        n01593028.tar
+        n01593282.tar
+        ...
+```
+
+We first unzip the overlapping classes of LVIS (we will directly work with the .tar file for the rest classes) and convert them into LVIS annotation format.
+
+~~~
+mkdir imagenet/annotations
+python tools/unzip_imagenet_lvis.py --dst_path datasets/imagenet/ImageNet-LVIS
+python tools/create_imagenetlvis_json.py --imagenet_path datasets/imagenet/ImageNet-LVIS --out_path datasets/imagenet/annotations/imagenet_lvis_image_info.json
+~~~
+This creates `datasets/imagenet/annotations/imagenet_lvis_image_info.json`.
+
+[Optional] To train with all the 21K classes, run
+
+~~~
+python tools/get_imagenet_21k_full_tar_json.py
+python tools/create_lvis_21k.py
+~~~
+This creates `datasets/imagenet/annotations/imagenet-21k_image_info_lvis-21k.json` and `datasets/lvis/lvis_v1_train_lvis-21k.json` (combined LVIS and ImageNet-21K classes in `categories`).
+
+[Optional] To train on combined LVIS and COCO, run
+
+~~~
+python tools/merge_lvis_coco.py
+~~~
+This creates `datasets/lvis/lvis_v1_train+coco_mask.json`
+
+### Conceptual Caption
+
+
+Download the dataset from [this](https://ai.google.com/research/ConceptualCaptions/download) page and place them as:
+```
+cc3m/
+    GCC-training.tsv
+```
+
+Run the following command to download the images and convert the annotations to LVIS format (Note: download images takes long).
+
+~~~
+python tools/download_cc.py --ann datasets/cc3m/GCC-training.tsv --save_image_path datasets/cc3m/training/ --out_path datasets/cc3m/train_image_info.json
+python tools/get_cc_tags.py
+~~~
+
+This creates `datasets/cc3m/train_image_info_tags.json`.
+
+### Objects365
+Download Objects365 (v2) from the website. We only need the validation set in this project:
+```
+objects365/
+    annotations/
+        zhiyuan_objv2_val.json
+    val/
+        images/
+            v1/
+                patch0/
+                ...
+                patch15/
+            v2/
+                patch16/
+                ...
+                patch49/
+
+```
+
+The original annotation has typos in the class names, we first fix them for our following use of language embeddings.
+
+```
+python tools/fix_o365_names.py --ann datasets/objects365/annotations/zhiyuan_objv2_val.json
+```
+This creates `datasets/objects365/zhiyuan_objv2_val_fixname.json`.
+
+To train on Objects365, download the training images and use the command above.  We note some images in the training annotation do not exist.
+We use the following command to filter the missing images.
+~~~
+python tools/fix_0365_path.py
+~~~
+This creates `datasets/objects365/zhiyuan_objv2_train_fixname_fixmiss.json`.
+
+### OpenImages
+
+We followed the instructions in [UniDet](https://github.com/xingyizhou/UniDet/blob/master/projects/UniDet/unidet_docs/DATASETS.md#openimages) to convert the metadata for OpenImages.
+
+The converted folder should look like
+
+```
+oid/
+    annotations/
+        oid_challenge_2019_train_bbox.json
+        oid_challenge_2019_val_expanded.json
+    images/
+        0/
+        1/
+        2/
+        ...
+```
+
+### Open-vocabulary COCO
+
+We first follow [OVR-CNN](https://github.com/alirezazareian/ovr-cnn/blob/master/ipynb/003.ipynb) to create the open-vocabulary COCO split. The converted files should be like 
+
+```
+coco/
+    zero-shot/
+        instances_train2017_seen_2.json
+        instances_val2017_all_2.json
+```
+
+We further pre-process the annotation format for easier evaluation:
+
+```
+python tools/get_coco_zeroshot_oriorder.py --data_path datasets/coco/zero-shot/instances_train2017_seen_2.json
+python tools/get_coco_zeroshot_oriorder.py --data_path datasets/coco/zero-shot/instances_val2017_all_2.json
+```
+
+Next, we preprocess the COCO caption data:
+
+```
+python tools/get_cc_tags.py --cc_ann datasets/coco/annotations/captions_train2017.json --out_path datasets/coco/captions_train2017_tags_allcaps.json --allcaps --convert_caption
+```
+This creates `datasets/coco/captions_train2017_tags_allcaps.json`.
+
+### Metadata
+
+```
+metadata/
+    lvis_v1_train_cat_info.json
+    coco_clip_a+cname.npy
+    lvis_v1_clip_a+cname.npy
+    o365_clip_a+cnamefix.npy
+    oid_clip_a+cname.npy
+    imagenet_lvis_wnid.txt
+    Objects365_names_fix.csv
+```
+
+`lvis_v1_train_cat_info.json` is used by the Federated loss.
+This is created by 
+~~~
+python tools/get_lvis_cat_info.py --ann datasets/lvis/lvis_v1_train.json
+~~~
+
+`*_clip_a+cname.npy` is the pre-computed CLIP embeddings for each datasets.
+They are created by (taking LVIS as an example)
+~~~
+python tools/dump_clip_features.py --ann datasets/lvis/lvis_v1_val.json --out_path metadata/lvis_v1_clip_a+cname.npy
+~~~
+Note we do not include the 21K class embeddings due to the large file size.
+To create it, run
+~~~
+python tools/dump_clip_features.py --ann datasets/lvis/lvis_v1_val_lvis-21k.json --out_path datasets/metadata/lvis-21k_clip_a+cname.npy
+~~~
+
+`imagenet_lvis_wnid.txt` is the list of matched classes between ImageNet-21K and LVIS.
+
+`Objects365_names_fix.csv` is our manual fix of the Objects365 names.

datasets/metadata/Objects365_names_fix.csv

@@ -0,0 +1,365 @@
+1,Person,Person
+2,Sneakers,Sneakers
+3,Chair,Chair
+4,Other Shoes,Other Shoes
+5,Hat,Hat
+6,Car,Car
+7,Lamp,Lamp
+8,Glasses,Glasses
+9,Bottle,Bottle
+10,Desk,Desk
+11,Cup,Cup
+12,Street Lights,Street Lights
+13,Cabinet/shelf,Cabinet/shelf
+14,Handbag/Satchel,Handbag/Satchel
+15,Bracelet,Bracelet
+16,Plate,Plate
+17,Picture/Frame,Picture/Frame
+18,Helmet,Helmet
+19,Book,Book
+20,Gloves,Gloves
+21,Storage box,Storage box
+22,Boat,Boat
+23,Leather Shoes,Leather Shoes
+24,Flower,Flower
+25,Bench,Bench
+26,Potted Plant,Potted Plant
+27,Bowl/Basin,Bowl/Basin
+28,Flag,Flag
+29,Pillow,Pillow
+30,Boots,Boots
+31,Vase,Vase
+32,Microphone,Microphone
+33,Necklace,Necklace
+34,Ring,Ring
+35,SUV,SUV
+36,Wine Glass,Wine Glass
+37,Belt,Belt
+38,Moniter/TV,Monitor/TV
+39,Backpack,Backpack
+40,Umbrella,Umbrella
+41,Traffic Light,Traffic Light
+42,Speaker,Speaker
+43,Watch,Watch
+44,Tie,Tie
+45,Trash bin Can,Trash bin Can
+46,Slippers,Slippers
+47,Bicycle,Bicycle
+48,Stool,Stool
+49,Barrel/bucket,Barrel/bucket
+50,Van,Van
+51,Couch,Couch
+52,Sandals,Sandals
+53,Bakset,Basket
+54,Drum,Drum
+55,Pen/Pencil,Pen/Pencil
+56,Bus,Bus
+57,Wild Bird,Wild Bird
+58,High Heels,High Heels
+59,Motorcycle,Motorcycle
+60,Guitar,Guitar
+61,Carpet,Carpet
+62,Cell Phone,Cell Phone
+63,Bread,Bread
+64,Camera,Camera
+65,Canned,Canned
+66,Truck,Truck
+67,Traffic cone,Traffic cone
+68,Cymbal,Cymbal
+69,Lifesaver,Lifesaver
+70,Towel,Towel
+71,Stuffed Toy,Stuffed Toy
+72,Candle,Candle
+73,Sailboat,Sailboat
+74,Laptop,Laptop
+75,Awning,Awning
+76,Bed,Bed
+77,Faucet,Faucet
+78,Tent,Tent
+79,Horse,Horse
+80,Mirror,Mirror
+81,Power outlet,Power outlet
+82,Sink,Sink
+83,Apple,Apple
+84,Air Conditioner,Air Conditioner
+85,Knife,Knife
+86,Hockey Stick,Hockey Stick
+87,Paddle,Paddle
+88,Pickup Truck,Pickup Truck
+89,Fork,Fork
+90,Traffic Sign,Traffic Sign
+91,Ballon,Ballon
+92,Tripod,Tripod
+93,Dog,Dog
+94,Spoon,Spoon
+95,Clock,Clock
+96,Pot,Pot
+97,Cow,Cow
+98,Cake,Cake
+99,Dinning Table,Dining Table
+100,Sheep,Sheep
+101,Hanger,Hanger
+102,Blackboard/Whiteboard,Blackboard/Whiteboard
+103,Napkin,Napkin
+104,Other Fish,Other Fish
+105,Orange/Tangerine,Orange/Tangerine
+106,Toiletry,Toiletry
+107,Keyboard,Keyboard
+108,Tomato,Tomato
+109,Lantern,Lantern
+110,Machinery Vehicle,Machinery Vehicle
+111,Fan,Fan
+112,Green Vegetables,Green Vegetables
+113,Banana,Banana
+114,Baseball Glove,Baseball Glove
+115,Airplane,Airplane
+116,Mouse,Mouse
+117,Train,Train
+118,Pumpkin,Pumpkin
+119,Soccer,Soccer
+120,Skiboard,Skiboard
+121,Luggage,Luggage
+122,Nightstand,Nightstand
+123,Tea pot,Teapot
+124,Telephone,Telephone
+125,Trolley,Trolley
+126,Head Phone,Head Phone
+127,Sports Car,Sports Car
+128,Stop Sign,Stop Sign
+129,Dessert,Dessert
+130,Scooter,Scooter
+131,Stroller,Stroller
+132,Crane,Crane
+133,Remote,Remote
+134,Refrigerator,Refrigerator
+135,Oven,Oven
+136,Lemon,Lemon
+137,Duck,Duck
+138,Baseball Bat,Baseball Bat
+139,Surveillance Camera,Surveillance Camera
+140,Cat,Cat
+141,Jug,Jug
+142,Broccoli,Broccoli
+143,Piano,Piano
+144,Pizza,Pizza
+145,Elephant,Elephant
+146,Skateboard,Skateboard
+147,Surfboard,Surfboard
+148,Gun,Gun
+149,Skating and Skiing shoes,Skating and Skiing shoes
+150,Gas stove,Gas stove
+151,Donut,Donut
+152,Bow Tie,Bow Tie
+153,Carrot,Carrot
+154,Toilet,Toilet
+155,Kite,Kite
+156,Strawberry,Strawberry
+157,Other Balls,Other Balls
+158,Shovel,Shovel
+159,Pepper,Pepper
+160,Computer Box,Computer Box
+161,Toilet Paper,Toilet Paper
+162,Cleaning Products,Cleaning Products
+163,Chopsticks,Chopsticks
+164,Microwave,Microwave
+165,Pigeon,Pigeon
+166,Baseball,Baseball
+167,Cutting/chopping Board,Cutting/chopping Board
+168,Coffee Table,Coffee Table
+169,Side Table,Side Table
+170,Scissors,Scissors
+171,Marker,Marker
+172,Pie,Pie
+173,Ladder,Ladder
+174,Snowboard,Snowboard
+175,Cookies,Cookies
+176,Radiator,Radiator
+177,Fire Hydrant,Fire Hydrant
+178,Basketball,Basketball
+179,Zebra,Zebra
+180,Grape,Grape
+181,Giraffe,Giraffe
+182,Potato,Potato
+183,Sausage,Sausage
+184,Tricycle,Tricycle
+185,Violin,Violin
+186,Egg,Egg
+187,Fire Extinguisher,Fire Extinguisher
+188,Candy,Candy
+189,Fire Truck,Fire Truck
+190,Billards,Billards
+191,Converter,Converter
+192,Bathtub,Bathtub
+193,Wheelchair,Wheelchair
+194,Golf Club,Golf Club
+195,Briefcase,Briefcase
+196,Cucumber,Cucumber
+197,Cigar/Cigarette,Cigar/Cigarette
+198,Paint Brush,Paint Brush
+199,Pear,Pear
+200,Heavy Truck,Heavy Truck
+201,Hamburger,Hamburger
+202,Extractor,Extractor
+203,Extention Cord,Extension Cord
+204,Tong,Tong
+205,Tennis Racket,Tennis Racket
+206,Folder,Folder
+207,American Football,American Football
+208,earphone,earphone
+209,Mask,Mask
+210,Kettle,Kettle
+211,Tennis,Tennis
+212,Ship,Ship
+213,Swing,Swing
+214,Coffee Machine,Coffee Machine
+215,Slide,Slide
+216,Carriage,Carriage
+217,Onion,Onion
+218,Green beans,Green beans
+219,Projector,Projector
+220,Frisbee,Frisbee
+221,Washing Machine/Drying Machine,Washing Machine/Drying Machine
+222,Chicken,Chicken
+223,Printer,Printer
+224,Watermelon,Watermelon
+225,Saxophone,Saxophone
+226,Tissue,Tissue
+227,Toothbrush,Toothbrush
+228,Ice cream,Ice cream
+229,Hotair ballon,Hot air balloon
+230,Cello,Cello
+231,French Fries,French Fries
+232,Scale,Scale
+233,Trophy,Trophy
+234,Cabbage,Cabbage
+235,Hot dog,Hot dog
+236,Blender,Blender
+237,Peach,Peach
+238,Rice,Rice
+239,Wallet/Purse,Wallet/Purse
+240,Volleyball,Volleyball
+241,Deer,Deer
+242,Goose,Goose
+243,Tape,Tape
+244,Tablet,Tablet
+245,Cosmetics,Cosmetics
+246,Trumpet,Trumpet
+247,Pineapple,Pineapple
+248,Golf Ball,Golf Ball
+249,Ambulance,Ambulance
+250,Parking meter,Parking meter
+251,Mango,Mango
+252,Key,Key
+253,Hurdle,Hurdle
+254,Fishing Rod,Fishing Rod
+255,Medal,Medal
+256,Flute,Flute
+257,Brush,Brush
+258,Penguin,Penguin
+259,Megaphone,Megaphone
+260,Corn,Corn
+261,Lettuce,Lettuce
+262,Garlic,Garlic
+263,Swan,Swan
+264,Helicopter,Helicopter
+265,Green Onion,Green Onion
+266,Sandwich,Sandwich
+267,Nuts,Nuts
+268,Speed Limit Sign,Speed Limit Sign
+269,Induction Cooker,Induction Cooker
+270,Broom,Broom
+271,Trombone,Trombone
+272,Plum,Plum
+273,Rickshaw,Rickshaw
+274,Goldfish,Goldfish
+275,Kiwi fruit,Kiwi fruit
+276,Router/modem,Router/modem
+277,Poker Card,Poker Card
+278,Toaster,Toaster
+279,Shrimp,Shrimp
+280,Sushi,Sushi
+281,Cheese,Cheese
+282,Notepaper,Notepaper
+283,Cherry,Cherry
+284,Pliers,Pliers
+285,CD,CD
+286,Pasta,Pasta
+287,Hammer,Hammer
+288,Cue,Cue
+289,Avocado,Avocado
+290,Hamimelon,Hami melon
+291,Flask,Flask
+292,Mushroon,Mushroom
+293,Screwdriver,Screwdriver
+294,Soap,Soap
+295,Recorder,Recorder
+296,Bear,Bear
+297,Eggplant,Eggplant
+298,Board Eraser,Board Eraser
+299,Coconut,Coconut
+300,Tape Measur/ Ruler,Tape Measure/ Ruler
+301,Pig,Pig
+302,Showerhead,Showerhead
+303,Globe,Globe
+304,Chips,Chips
+305,Steak,Steak
+306,Crosswalk Sign,Crosswalk Sign
+307,Stapler,Stapler
+308,Campel,Camel
+309,Formula 1,Formula 1
+310,Pomegranate,Pomegranate
+311,Dishwasher,Dishwasher
+312,Crab,Crab
+313,Hoverboard,Hoverboard
+314,Meat ball,Meatball
+315,Rice Cooker,Rice Cooker
+316,Tuba,Tuba
+317,Calculator,Calculator
+318,Papaya,Papaya
+319,Antelope,Antelope
+320,Parrot,Parrot
+321,Seal,Seal
+322,Buttefly,Butterfly
+323,Dumbbell,Dumbbell
+324,Donkey,Donkey
+325,Lion,Lion
+326,Urinal,Urinal
+327,Dolphin,Dolphin
+328,Electric Drill,Electric Drill
+329,Hair Dryer,Hair Dryer
+330,Egg tart,Egg tart
+331,Jellyfish,Jellyfish
+332,Treadmill,Treadmill
+333,Lighter,Lighter
+334,Grapefruit,Grapefruit
+335,Game board,Game board
+336,Mop,Mop
+337,Radish,Radish
+338,Baozi,Baozi
+339,Target,Target
+340,French,French
+341,Spring Rolls,Spring Rolls
+342,Monkey,Monkey
+343,Rabbit,Rabbit
+344,Pencil Case,Pencil Case
+345,Yak,Yak
+346,Red Cabbage,Red Cabbage
+347,Binoculars,Binoculars
+348,Asparagus,Asparagus
+349,Barbell,Barbell
+350,Scallop,Scallop
+351,Noddles,Noddles
+352,Comb,Comb
+353,Dumpling,Dumpling
+354,Oyster,Oyster
+355,Table Teniis paddle,Table Tennis paddle
+356,Cosmetics Brush/Eyeliner Pencil,Cosmetics Brush/Eyeliner Pencil
+357,Chainsaw,Chainsaw
+358,Eraser,Eraser
+359,Lobster,Lobster
+360,Durian,Durian
+361,Okra,Okra
+362,Lipstick,Lipstick
+363,Cosmetics Mirror,Cosmetics Mirror
+364,Curling,Curling
+365,Table Tennis,Table Tennis

demo.py

@@ -0,0 +1,204 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+import argparse
+import glob
+import multiprocessing as mp
+import numpy as np
+import os
+import tempfile
+import time
+import warnings
+import cv2
+import tqdm
+import sys
+
+from detectron2.config import get_cfg
+from detectron2.data.detection_utils import read_image
+from detectron2.utils.logger import setup_logger
+
+sys.path.insert(0, 'third_party/CenterNet2/projects/CenterNet2/')
+from centernet.config import add_centernet_config
+from detic.config import add_detic_config
+
+from detic.predictor import VisualizationDemo
+
+
+# constants
+WINDOW_NAME = "Detic"
+
+def setup_cfg(args):
+    cfg = get_cfg()
+    add_centernet_config(cfg)
+    add_detic_config(cfg)
+    cfg.merge_from_file(args.config_file)
+    cfg.merge_from_list(args.opts)
+    # Set score_threshold for builtin models
+    cfg.MODEL.RETINANET.SCORE_THRESH_TEST = args.confidence_threshold
+    cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = args.confidence_threshold
+    cfg.MODEL.PANOPTIC_FPN.COMBINE.INSTANCES_CONFIDENCE_THRESH = args.confidence_threshold
+    cfg.MODEL.ROI_BOX_HEAD.ZEROSHOT_WEIGHT_PATH = 'rand' # load later
+    if not args.pred_all_class:
+        cfg.MODEL.ROI_HEADS.ONE_CLASS_PER_PROPOSAL = True
+    cfg.freeze()
+    return cfg
+
+
+def get_parser():
+    parser = argparse.ArgumentParser(description="Detectron2 demo for builtin configs")
+    parser.add_argument(
+        "--config-file",
+        default="configs/quick_schedules/mask_rcnn_R_50_FPN_inference_acc_test.yaml",
+        metavar="FILE",
+        help="path to config file",
+    )
+    parser.add_argument("--webcam", action="store_true", help="Take inputs from webcam.")
+    parser.add_argument("--video-input", help="Path to video file.")
+    parser.add_argument(
+        "--input",
+        nargs="+",
+        help="A list of space separated input images; "
+        "or a single glob pattern such as 'directory/*.jpg'",
+    )
+    parser.add_argument(
+        "--output",
+        help="A file or directory to save output visualizations. "
+        "If not given, will show output in an OpenCV window.",
+    )
+    parser.add_argument(
+        "--vocabulary",
+        default="lvis",
+        choices=['lvis', 'openimages', 'objects365', 'coco', 'custom'],
+        help="",
+    )
+    parser.add_argument(
+        "--custom_vocabulary",
+        default="",
+        help="",
+    )
+    parser.add_argument("--pred_all_class", action='store_true')
+    parser.add_argument(
+        "--confidence-threshold",
+        type=float,
+        default=0.5,
+        help="Minimum score for instance predictions to be shown",
+    )
+    parser.add_argument(
+        "--opts",
+        help="Modify config options using the command-line 'KEY VALUE' pairs",
+        default=[],
+        nargs=argparse.REMAINDER,
+    )
+    return parser
+
+
+def test_opencv_video_format(codec, file_ext):
+    with tempfile.TemporaryDirectory(prefix="video_format_test") as dir:
+        filename = os.path.join(dir, "test_file" + file_ext)
+        writer = cv2.VideoWriter(
+            filename=filename,
+            fourcc=cv2.VideoWriter_fourcc(*codec),
+            fps=float(30),
+            frameSize=(10, 10),
+            isColor=True,
+        )
+        [writer.write(np.zeros((10, 10, 3), np.uint8)) for _ in range(30)]
+        writer.release()
+        if os.path.isfile(filename):
+            return True
+        return False
+
+
+if __name__ == "__main__":
+    mp.set_start_method("spawn", force=True)
+    args = get_parser().parse_args()
+    setup_logger(name="fvcore")
+    logger = setup_logger()
+    logger.info("Arguments: " + str(args))
+
+    cfg = setup_cfg(args)
+
+    demo = VisualizationDemo(cfg, args)
+
+    if args.input:
+        if len(args.input) == 1:
+            args.input = glob.glob(os.path.expanduser(args.input[0]))
+            assert args.input, "The input path(s) was not found"
+        for path in tqdm.tqdm(args.input, disable=not args.output):
+            img = read_image(path, format="BGR")
+            start_time = time.time()
+            predictions, visualized_output = demo.run_on_image(img)
+            logger.info(
+                "{}: {} in {:.2f}s".format(
+                    path,
+                    "detected {} instances".format(len(predictions["instances"]))
+                    if "instances" in predictions
+                    else "finished",
+                    time.time() - start_time,
+                )
+            )
+
+            if args.output:
+                if os.path.isdir(args.output):
+                    assert os.path.isdir(args.output), args.output
+                    out_filename = os.path.join(args.output, os.path.basename(path))
+                else:
+                    assert len(args.input) == 1, "Please specify a directory with args.output"
+                    out_filename = args.output
+                visualized_output.save(out_filename)
+            else:
+                cv2.namedWindow(WINDOW_NAME, cv2.WINDOW_NORMAL)
+                cv2.imshow(WINDOW_NAME, visualized_output.get_image()[:, :, ::-1])
+                if cv2.waitKey(0) == 27:
+                    break  # esc to quit
+    elif args.webcam:
+        assert args.input is None, "Cannot have both --input and --webcam!"
+        assert args.output is None, "output not yet supported with --webcam!"
+        cam = cv2.VideoCapture(0)
+        for vis in tqdm.tqdm(demo.run_on_video(cam)):
+            cv2.namedWindow(WINDOW_NAME, cv2.WINDOW_NORMAL)
+            cv2.imshow(WINDOW_NAME, vis)
+            if cv2.waitKey(1) == 27:
+                break  # esc to quit
+        cam.release()
+        cv2.destroyAllWindows()
+    elif args.video_input:
+        video = cv2.VideoCapture(args.video_input)
+        width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH))
+        height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))
+        frames_per_second = video.get(cv2.CAP_PROP_FPS)
+        num_frames = int(video.get(cv2.CAP_PROP_FRAME_COUNT))
+        basename = os.path.basename(args.video_input)
+        codec, file_ext = (
+            ("x264", ".mkv") if test_opencv_video_format("x264", ".mkv") else ("mp4v", ".mp4")
+        )
+        if codec == ".mp4v":
+            warnings.warn("x264 codec not available, switching to mp4v")
+        if args.output:
+            if os.path.isdir(args.output):
+                output_fname = os.path.join(args.output, basename)
+                output_fname = os.path.splitext(output_fname)[0] + file_ext
+            else:
+                output_fname = args.output
+            assert not os.path.isfile(output_fname), output_fname
+            output_file = cv2.VideoWriter(
+                filename=output_fname,
+                # some installation of opencv may not support x264 (due to its license),
+                # you can try other format (e.g. MPEG)
+                fourcc=cv2.VideoWriter_fourcc(*codec),
+                fps=float(frames_per_second),
+                frameSize=(width, height),
+                isColor=True,
+            )
+        assert os.path.isfile(args.video_input)
+        for vis_frame in tqdm.tqdm(demo.run_on_video(video), total=num_frames):
+            if args.output:
+                output_file.write(vis_frame)
+            else:
+                cv2.namedWindow(basename, cv2.WINDOW_NORMAL)
+                cv2.imshow(basename, vis_frame)
+                if cv2.waitKey(1) == 27:
+                    break  # esc to quit
+        video.release()
+        if args.output:
+            output_file.release()
+        else:
+            cv2.destroyAllWindows()

detic/__init__.py

@@ -0,0 +1,19 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+from .modeling.meta_arch import custom_rcnn
+from .modeling.roi_heads import detic_roi_heads
+from .modeling.roi_heads import res5_roi_heads
+from .modeling.backbone import swintransformer
+from .modeling.backbone import timm
+
+
+from .data.datasets import lvis_v1
+from .data.datasets import imagenet
+from .data.datasets import cc
+from .data.datasets import objects365
+from .data.datasets import oid
+from .data.datasets import coco_zeroshot
+
+try:
+    from .modeling.meta_arch import d2_deformable_detr
+except:
+    pass

detic/config.py

@@ -0,0 +1,131 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+from detectron2.config import CfgNode as CN
+
+def add_detic_config(cfg):
+    _C = cfg
+
+    _C.WITH_IMAGE_LABELS = False # Turn on co-training with classification data
+
+    # Open-vocabulary classifier
+    _C.MODEL.ROI_BOX_HEAD.USE_ZEROSHOT_CLS = False # Use fixed classifier for open-vocabulary detection
+    _C.MODEL.ROI_BOX_HEAD.ZEROSHOT_WEIGHT_PATH = 'datasets/metadata/lvis_v1_clip_a+cname.npy'
+    _C.MODEL.ROI_BOX_HEAD.ZEROSHOT_WEIGHT_DIM = 512
+    _C.MODEL.ROI_BOX_HEAD.NORM_WEIGHT = True
+    _C.MODEL.ROI_BOX_HEAD.NORM_TEMP = 50.0
+    _C.MODEL.ROI_BOX_HEAD.IGNORE_ZERO_CATS = False
+    _C.MODEL.ROI_BOX_HEAD.USE_BIAS = 0.0 # >= 0: not use
+    
+    _C.MODEL.ROI_BOX_HEAD.MULT_PROPOSAL_SCORE = False # CenterNet2
+    _C.MODEL.ROI_BOX_HEAD.USE_SIGMOID_CE = False
+    _C.MODEL.ROI_BOX_HEAD.PRIOR_PROB = 0.01
+    _C.MODEL.ROI_BOX_HEAD.USE_FED_LOSS = False # Federated Loss
+    _C.MODEL.ROI_BOX_HEAD.CAT_FREQ_PATH = \
+        'datasets/metadata/lvis_v1_train_cat_info.json'
+    _C.MODEL.ROI_BOX_HEAD.FED_LOSS_NUM_CAT = 50
+    _C.MODEL.ROI_BOX_HEAD.FED_LOSS_FREQ_WEIGHT = 0.5
+
+    # Classification data configs
+    _C.MODEL.ROI_BOX_HEAD.IMAGE_LABEL_LOSS = 'max_size' # max, softmax, sum
+    _C.MODEL.ROI_BOX_HEAD.IMAGE_LOSS_WEIGHT = 0.1
+    _C.MODEL.ROI_BOX_HEAD.IMAGE_BOX_SIZE = 1.0
+    _C.MODEL.ROI_BOX_HEAD.ADD_IMAGE_BOX = False # Used for image-box loss and caption loss
+    _C.MODEL.ROI_BOX_HEAD.WS_NUM_PROPS = 128 # num proposals for image-labeled data
+    _C.MODEL.ROI_BOX_HEAD.WITH_SOFTMAX_PROP = False # Used for WSDDN
+    _C.MODEL.ROI_BOX_HEAD.CAPTION_WEIGHT = 1.0 # Caption loss weight
+    _C.MODEL.ROI_BOX_HEAD.NEG_CAP_WEIGHT = 0.125 # Caption loss hyper-parameter
+    _C.MODEL.ROI_BOX_HEAD.ADD_FEATURE_TO_PROP = False # Used for WSDDN
+    _C.MODEL.ROI_BOX_HEAD.SOFTMAX_WEAK_LOSS = False # Used when USE_SIGMOID_CE is False
+
+    _C.MODEL.ROI_HEADS.MASK_WEIGHT = 1.0
+    _C.MODEL.ROI_HEADS.ONE_CLASS_PER_PROPOSAL = False # For demo only
+
+    # Caption losses
+    _C.MODEL.CAP_BATCH_RATIO = 4 # Ratio between detection data and caption data
+    _C.MODEL.WITH_CAPTION = False
+    _C.MODEL.SYNC_CAPTION_BATCH = False # synchronize across GPUs to enlarge # "classes"
+
+    # dynamic class sampling when training with 21K classes
+    _C.MODEL.DYNAMIC_CLASSIFIER = False
+    _C.MODEL.NUM_SAMPLE_CATS = 50
+
+    # Different classifiers in testing, used in cross-dataset evaluation
+    _C.MODEL.RESET_CLS_TESTS = False
+    _C.MODEL.TEST_CLASSIFIERS = []
+    _C.MODEL.TEST_NUM_CLASSES = []
+
+    # Backbones
+    _C.MODEL.SWIN = CN()
+    _C.MODEL.SWIN.SIZE = 'T' # 'T', 'S', 'B'
+    _C.MODEL.SWIN.USE_CHECKPOINT = False
+    _C.MODEL.SWIN.OUT_FEATURES = (1, 2, 3) # FPN stride 8 - 32
+
+    _C.MODEL.TIMM = CN()
+    _C.MODEL.TIMM.BASE_NAME = 'resnet50'
+    _C.MODEL.TIMM.OUT_LEVELS = (3, 4, 5)
+    _C.MODEL.TIMM.NORM = 'FrozenBN'
+    _C.MODEL.TIMM.FREEZE_AT = 0
+    _C.MODEL.DATASET_LOSS_WEIGHT = []
+    
+    # Multi-dataset dataloader
+    _C.DATALOADER.DATASET_RATIO = [1, 1] # sample ratio
+    _C.DATALOADER.USE_RFS = [False, False]
+    _C.DATALOADER.MULTI_DATASET_GROUPING = False # Always true when multi-dataset is enabled
+    _C.DATALOADER.DATASET_ANN = ['box', 'box'] # Annotation type of each dataset
+    _C.DATALOADER.USE_DIFF_BS_SIZE = False # Use different batchsize for each dataset
+    _C.DATALOADER.DATASET_BS = [8, 32] # Used when USE_DIFF_BS_SIZE is on
+    _C.DATALOADER.DATASET_INPUT_SIZE = [896, 384] # Used when USE_DIFF_BS_SIZE is on
+    _C.DATALOADER.DATASET_INPUT_SCALE = [(0.1, 2.0), (0.5, 1.5)] # Used when USE_DIFF_BS_SIZE is on 
+    _C.DATALOADER.DATASET_MIN_SIZES = [(640, 800), (320, 400)] # Used when USE_DIFF_BS_SIZE is on
+    _C.DATALOADER.DATASET_MAX_SIZES = [1333, 667] # Used when USE_DIFF_BS_SIZE is on
+    _C.DATALOADER.USE_TAR_DATASET = False # for ImageNet-21K, directly reading from unziped files
+    _C.DATALOADER.TARFILE_PATH = 'datasets/imagenet/metadata-22k/tar_files.npy'
+    _C.DATALOADER.TAR_INDEX_DIR = 'datasets/imagenet/metadata-22k/tarindex_npy'
+    
+    _C.SOLVER.USE_CUSTOM_SOLVER = False
+    _C.SOLVER.OPTIMIZER = 'SGD'
+    _C.SOLVER.BACKBONE_MULTIPLIER = 1.0 # Used in DETR
+    _C.SOLVER.CUSTOM_MULTIPLIER = 1.0 # Used in DETR
+    _C.SOLVER.CUSTOM_MULTIPLIER_NAME = [] # Used in DETR
+
+    # Deformable DETR
+    _C.MODEL.DETR = CN()
+    _C.MODEL.DETR.NUM_CLASSES = 80
+    _C.MODEL.DETR.FROZEN_WEIGHTS = '' # For Segmentation
+    _C.MODEL.DETR.GIOU_WEIGHT = 2.0
+    _C.MODEL.DETR.L1_WEIGHT = 5.0
+    _C.MODEL.DETR.DEEP_SUPERVISION = True
+    _C.MODEL.DETR.NO_OBJECT_WEIGHT = 0.1
+    _C.MODEL.DETR.CLS_WEIGHT = 2.0
+    _C.MODEL.DETR.NUM_FEATURE_LEVELS = 4
+    _C.MODEL.DETR.TWO_STAGE = False
+    _C.MODEL.DETR.WITH_BOX_REFINE = False
+    _C.MODEL.DETR.FOCAL_ALPHA = 0.25
+    _C.MODEL.DETR.NHEADS = 8
+    _C.MODEL.DETR.DROPOUT = 0.1
+    _C.MODEL.DETR.DIM_FEEDFORWARD = 2048
+    _C.MODEL.DETR.ENC_LAYERS = 6
+    _C.MODEL.DETR.DEC_LAYERS = 6
+    _C.MODEL.DETR.PRE_NORM = False
+    _C.MODEL.DETR.HIDDEN_DIM = 256
+    _C.MODEL.DETR.NUM_OBJECT_QUERIES = 100
+
+    _C.MODEL.DETR.USE_FED_LOSS = False
+    _C.MODEL.DETR.WEAK_WEIGHT = 0.1
+
+    _C.INPUT.CUSTOM_AUG = ''
+    _C.INPUT.TRAIN_SIZE = 640
+    _C.INPUT.TEST_SIZE = 640
+    _C.INPUT.SCALE_RANGE = (0.1, 2.)
+    # 'default' for fixed short/ long edge, 'square' for max size=INPUT.SIZE
+    _C.INPUT.TEST_INPUT_TYPE = 'default' 
+
+    _C.FIND_UNUSED_PARAM = True
+    _C.EVAL_PRED_AR = False
+    _C.EVAL_PROPOSAL_AR = False
+    _C.EVAL_CAT_SPEC_AR = False
+    _C.IS_DEBUG = False
+    _C.QUICK_DEBUG = False
+    _C.FP16 = False
+    _C.EVAL_AP_FIX = False
+    _C.GEN_PSEDO_LABELS = False
+    _C.SAVE_DEBUG_PATH = 'output/save_debug/'

detic/custom_solver.py

@@ -0,0 +1,78 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
+from enum import Enum
+import itertools
+from typing import Any, Callable, Dict, Iterable, List, Set, Type, Union
+import torch
+
+from detectron2.config import CfgNode
+
+from detectron2.solver.build import maybe_add_gradient_clipping
+
+def match_name_keywords(n, name_keywords):
+    out = False
+    for b in name_keywords:
+        if b in n:
+            out = True
+            break
+    return out
+
+def build_custom_optimizer(cfg: CfgNode, model: torch.nn.Module) -> torch.optim.Optimizer:
+    """
+    Build an optimizer from config.
+    """
+    params: List[Dict[str, Any]] = []
+    memo: Set[torch.nn.parameter.Parameter] = set()
+    custom_multiplier_name = cfg.SOLVER.CUSTOM_MULTIPLIER_NAME
+    optimizer_type = cfg.SOLVER.OPTIMIZER
+    for key, value in model.named_parameters(recurse=True):
+        if not value.requires_grad:
+            continue
+        # Avoid duplicating parameters
+        if value in memo:
+            continue
+        memo.add(value)
+        lr = cfg.SOLVER.BASE_LR
+        weight_decay = cfg.SOLVER.WEIGHT_DECAY
+        if "backbone" in key:
+            lr = lr * cfg.SOLVER.BACKBONE_MULTIPLIER
+        if match_name_keywords(key, custom_multiplier_name):
+            lr = lr * cfg.SOLVER.CUSTOM_MULTIPLIER
+            print('Costum LR', key, lr)
+        param = {"params": [value], "lr": lr}
+        if optimizer_type != 'ADAMW':
+            param['weight_decay'] = weight_decay
+        params += [param]
+
+    def maybe_add_full_model_gradient_clipping(optim):  # optim: the optimizer class
+        # detectron2 doesn't have full model gradient clipping now
+        clip_norm_val = cfg.SOLVER.CLIP_GRADIENTS.CLIP_VALUE
+        enable = (
+            cfg.SOLVER.CLIP_GRADIENTS.ENABLED
+            and cfg.SOLVER.CLIP_GRADIENTS.CLIP_TYPE == "full_model"
+            and clip_norm_val > 0.0
+        )
+
+        class FullModelGradientClippingOptimizer(optim):
+            def step(self, closure=None):
+                all_params = itertools.chain(*[x["params"] for x in self.param_groups])
+                torch.nn.utils.clip_grad_norm_(all_params, clip_norm_val)
+                super().step(closure=closure)
+
+        return FullModelGradientClippingOptimizer if enable else optim
+
+    
+    if optimizer_type == 'SGD':
+        optimizer = maybe_add_full_model_gradient_clipping(torch.optim.SGD)(
+            params, cfg.SOLVER.BASE_LR, momentum=cfg.SOLVER.MOMENTUM, 
+            nesterov=cfg.SOLVER.NESTEROV
+        )
+    elif optimizer_type == 'ADAMW':
+        optimizer = maybe_add_full_model_gradient_clipping(torch.optim.AdamW)(
+            params, cfg.SOLVER.BASE_LR, 
+            weight_decay=cfg.SOLVER.WEIGHT_DECAY
+        )
+    else:
+        raise NotImplementedError(f"no optimizer type {optimizer_type}")
+    if not cfg.SOLVER.CLIP_GRADIENTS.CLIP_TYPE == "full_model":
+        optimizer = maybe_add_gradient_clipping(cfg, optimizer)
+    return optimizer

detic/data/custom_build_augmentation.py

@@ -0,0 +1,51 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+import logging
+import numpy as np
+import pycocotools.mask as mask_util
+import torch
+from fvcore.common.file_io import PathManager
+from PIL import Image
+
+
+from detectron2.data import transforms as T
+from .transforms.custom_augmentation_impl import EfficientDetResizeCrop
+
+def build_custom_augmentation(cfg, is_train, scale=None, size=None, \
+    min_size=None, max_size=None):
+    """
+    Create a list of default :class:`Augmentation` from config.
+    Now it includes resizing and flipping.
+
+    Returns:
+        list[Augmentation]
+    """
+    if cfg.INPUT.CUSTOM_AUG == 'ResizeShortestEdge':
+        if is_train:
+            min_size = cfg.INPUT.MIN_SIZE_TRAIN if min_size is None else min_size
+            max_size = cfg.INPUT.MAX_SIZE_TRAIN if max_size is None else max_size
+            sample_style = cfg.INPUT.MIN_SIZE_TRAIN_SAMPLING
+        else:
+            min_size = cfg.INPUT.MIN_SIZE_TEST
+            max_size = cfg.INPUT.MAX_SIZE_TEST
+            sample_style = "choice"
+        augmentation = [T.ResizeShortestEdge(min_size, max_size, sample_style)]
+    elif cfg.INPUT.CUSTOM_AUG == 'EfficientDetResizeCrop':
+        if is_train:
+            scale = cfg.INPUT.SCALE_RANGE if scale is None else scale
+            size = cfg.INPUT.TRAIN_SIZE if size is None else size
+        else:
+            scale = (1, 1)
+            size = cfg.INPUT.TEST_SIZE
+        augmentation = [EfficientDetResizeCrop(size, scale)]
+    else:
+        assert 0, cfg.INPUT.CUSTOM_AUG
+
+    if is_train:
+        augmentation.append(T.RandomFlip())
+    return augmentation
+
+
+build_custom_transform_gen = build_custom_augmentation
+"""
+Alias for backward-compatibility.
+"""

detic/data/custom_dataset_dataloader.py

@@ -0,0 +1,331 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+# Part of the code is from https://github.com/xingyizhou/UniDet/blob/master/projects/UniDet/unidet/data/multi_dataset_dataloader.py (Apache-2.0 License)
+import copy
+import logging
+import numpy as np
+import operator
+import torch
+import torch.utils.data
+import json
+from detectron2.utils.comm import get_world_size
+from detectron2.utils.logger import _log_api_usage, log_first_n
+
+from detectron2.config import configurable
+from detectron2.data import samplers
+from torch.utils.data.sampler import BatchSampler, Sampler
+from detectron2.data.common import DatasetFromList, MapDataset
+from detectron2.data.dataset_mapper import DatasetMapper
+from detectron2.data.build import get_detection_dataset_dicts, build_batch_data_loader
+from detectron2.data.samplers import TrainingSampler, RepeatFactorTrainingSampler
+from detectron2.data.build import worker_init_reset_seed, print_instances_class_histogram
+from detectron2.data.build import filter_images_with_only_crowd_annotations
+from detectron2.data.build import filter_images_with_few_keypoints
+from detectron2.data.build import check_metadata_consistency
+from detectron2.data.catalog import MetadataCatalog, DatasetCatalog
+from detectron2.utils import comm
+import itertools
+import math
+from collections import defaultdict
+from typing import Optional
+
+
+def _custom_train_loader_from_config(cfg, mapper=None, *, dataset=None, sampler=None):
+    sampler_name = cfg.DATALOADER.SAMPLER_TRAIN
+    if 'MultiDataset' in sampler_name:
+        dataset_dicts = get_detection_dataset_dicts_with_source(
+            cfg.DATASETS.TRAIN,
+            filter_empty=cfg.DATALOADER.FILTER_EMPTY_ANNOTATIONS,
+            min_keypoints=cfg.MODEL.ROI_KEYPOINT_HEAD.MIN_KEYPOINTS_PER_IMAGE
+            if cfg.MODEL.KEYPOINT_ON else 0,
+            proposal_files=cfg.DATASETS.PROPOSAL_FILES_TRAIN if cfg.MODEL.LOAD_PROPOSALS else None,
+        )
+    else:
+        dataset_dicts = get_detection_dataset_dicts(
+            cfg.DATASETS.TRAIN,
+            filter_empty=cfg.DATALOADER.FILTER_EMPTY_ANNOTATIONS,
+            min_keypoints=cfg.MODEL.ROI_KEYPOINT_HEAD.MIN_KEYPOINTS_PER_IMAGE
+            if cfg.MODEL.KEYPOINT_ON else 0,
+            proposal_files=cfg.DATASETS.PROPOSAL_FILES_TRAIN if cfg.MODEL.LOAD_PROPOSALS else None,
+        )
+
+    if mapper is None:
+        mapper = DatasetMapper(cfg, True)
+
+    if sampler is not None:
+        pass
+    elif sampler_name == "TrainingSampler":
+        sampler = TrainingSampler(len(dataset))
+    elif sampler_name == "MultiDatasetSampler":
+        sampler = MultiDatasetSampler(
+            dataset_dicts,
+            dataset_ratio = cfg.DATALOADER.DATASET_RATIO,
+            use_rfs = cfg.DATALOADER.USE_RFS,
+            dataset_ann = cfg.DATALOADER.DATASET_ANN,
+            repeat_threshold = cfg.DATALOADER.REPEAT_THRESHOLD,
+        )
+    elif sampler_name == "RepeatFactorTrainingSampler":
+        repeat_factors = RepeatFactorTrainingSampler.repeat_factors_from_category_frequency(
+            dataset_dicts, cfg.DATALOADER.REPEAT_THRESHOLD
+        )
+        sampler = RepeatFactorTrainingSampler(repeat_factors)
+    else:
+        raise ValueError("Unknown training sampler: {}".format(sampler_name))
+
+    return {
+        "dataset": dataset_dicts,
+        "sampler": sampler,
+        "mapper": mapper,
+        "total_batch_size": cfg.SOLVER.IMS_PER_BATCH,
+        "aspect_ratio_grouping": cfg.DATALOADER.ASPECT_RATIO_GROUPING,
+        "num_workers": cfg.DATALOADER.NUM_WORKERS,
+        'multi_dataset_grouping': cfg.DATALOADER.MULTI_DATASET_GROUPING,
+        'use_diff_bs_size': cfg.DATALOADER.USE_DIFF_BS_SIZE,
+        'dataset_bs': cfg.DATALOADER.DATASET_BS,
+        'num_datasets': len(cfg.DATASETS.TRAIN)
+    }
+
+
+@configurable(from_config=_custom_train_loader_from_config)
+def build_custom_train_loader(
+        dataset, *, mapper, sampler, 
+        total_batch_size=16,
+        aspect_ratio_grouping=True, 
+        num_workers=0,
+        num_datasets=1,
+        multi_dataset_grouping=False,
+        use_diff_bs_size=False,
+        dataset_bs=[]
+    ):
+    """
+    Modified from detectron2.data.build.build_custom_train_loader, but supports
+    different samplers
+    """
+    if isinstance(dataset, list):
+        dataset = DatasetFromList(dataset, copy=False)
+    if mapper is not None:
+        dataset = MapDataset(dataset, mapper)
+    if sampler is None:
+        sampler = TrainingSampler(len(dataset))
+    assert isinstance(sampler, torch.utils.data.sampler.Sampler)
+    if multi_dataset_grouping:
+        return build_multi_dataset_batch_data_loader(
+            use_diff_bs_size,
+            dataset_bs,
+            dataset,
+            sampler,
+            total_batch_size,
+            num_datasets=num_datasets,
+            num_workers=num_workers,
+        )
+    else:
+        return build_batch_data_loader(
+            dataset,
+            sampler,
+            total_batch_size,
+            aspect_ratio_grouping=aspect_ratio_grouping,
+            num_workers=num_workers,
+        )
+
+
+def build_multi_dataset_batch_data_loader(
+    use_diff_bs_size, dataset_bs,
+    dataset, sampler, total_batch_size, num_datasets, num_workers=0
+):
+    """
+    """
+    world_size = get_world_size()
+    assert (
+        total_batch_size > 0 and total_batch_size % world_size == 0
+    ), "Total batch size ({}) must be divisible by the number of gpus ({}).".format(
+        total_batch_size, world_size
+    )
+
+    batch_size = total_batch_size // world_size
+    data_loader = torch.utils.data.DataLoader(
+        dataset,
+        sampler=sampler,
+        num_workers=num_workers,
+        batch_sampler=None,
+        collate_fn=operator.itemgetter(0),  # don't batch, but yield individual elements
+        worker_init_fn=worker_init_reset_seed,
+    )  # yield individual mapped dict
+    if use_diff_bs_size:
+        return DIFFMDAspectRatioGroupedDataset(
+            data_loader, dataset_bs, num_datasets)
+    else:
+        return MDAspectRatioGroupedDataset(
+            data_loader, batch_size, num_datasets)
+
+
+def get_detection_dataset_dicts_with_source(
+    dataset_names, filter_empty=True, min_keypoints=0, proposal_files=None
+):
+    assert len(dataset_names)
+    dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in dataset_names]
+    for dataset_name, dicts in zip(dataset_names, dataset_dicts):
+        assert len(dicts), "Dataset '{}' is empty!".format(dataset_name)
+    
+    for source_id, (dataset_name, dicts) in \
+        enumerate(zip(dataset_names, dataset_dicts)):
+        assert len(dicts), "Dataset '{}' is empty!".format(dataset_name)
+        for d in dicts:
+            d['dataset_source'] = source_id
+
+        if "annotations" in dicts[0]:
+            try:
+                class_names = MetadataCatalog.get(dataset_name).thing_classes
+                check_metadata_consistency("thing_classes", dataset_name)
+                print_instances_class_histogram(dicts, class_names)
+            except AttributeError:  # class names are not available for this dataset
+                pass
+
+    assert proposal_files is None
+
+    dataset_dicts = list(itertools.chain.from_iterable(dataset_dicts))
+
+    has_instances = "annotations" in dataset_dicts[0]
+    if filter_empty and has_instances:
+        dataset_dicts = filter_images_with_only_crowd_annotations(dataset_dicts)
+    if min_keypoints > 0 and has_instances:
+        dataset_dicts = filter_images_with_few_keypoints(dataset_dicts, min_keypoints)
+
+    return dataset_dicts
+
+
+class MultiDatasetSampler(Sampler):
+    def __init__(
+        self, 
+        dataset_dicts, 
+        dataset_ratio,
+        use_rfs,
+        dataset_ann,
+        repeat_threshold=0.001,
+        seed: Optional[int] = None,
+        ):
+        """
+        """
+        sizes = [0 for _ in range(len(dataset_ratio))]
+        for d in dataset_dicts:
+            sizes[d['dataset_source']] += 1
+        print('dataset sizes', sizes)
+        self.sizes = sizes
+        assert len(dataset_ratio) == len(sizes), \
+            'length of dataset ratio {} should be equal to number if dataset {}'.format(
+                len(dataset_ratio), len(sizes)
+            )
+        if seed is None:
+            seed = comm.shared_random_seed()
+        self._seed = int(seed)
+        self._rank = comm.get_rank()
+        self._world_size = comm.get_world_size()
+        
+        self.dataset_ids =  torch.tensor(
+            [d['dataset_source'] for d in dataset_dicts], dtype=torch.long)
+
+        dataset_weight = [torch.ones(s) * max(sizes) / s * r / sum(dataset_ratio) \
+            for i, (r, s) in enumerate(zip(dataset_ratio, sizes))]
+        dataset_weight = torch.cat(dataset_weight)
+        
+        rfs_factors = []
+        st = 0
+        for i, s in enumerate(sizes):
+            if use_rfs[i]:
+                if dataset_ann[i] == 'box':
+                    rfs_func = RepeatFactorTrainingSampler.repeat_factors_from_category_frequency
+                else:
+                    rfs_func = repeat_factors_from_tag_frequency
+                rfs_factor = rfs_func(
+                    dataset_dicts[st: st + s],
+                    repeat_thresh=repeat_threshold)
+                rfs_factor = rfs_factor * (s / rfs_factor.sum())
+            else:
+                rfs_factor = torch.ones(s)
+            rfs_factors.append(rfs_factor)
+            st = st + s
+        rfs_factors = torch.cat(rfs_factors)
+
+        self.weights = dataset_weight * rfs_factors
+        self.sample_epoch_size = len(self.weights)
+
+    def __iter__(self):
+        start = self._rank
+        yield from itertools.islice(
+            self._infinite_indices(), start, None, self._world_size)
+
+
+    def _infinite_indices(self):
+        g = torch.Generator()
+        g.manual_seed(self._seed)
+        while True:
+            ids = torch.multinomial(
+                self.weights, self.sample_epoch_size, generator=g, 
+                replacement=True)
+            nums = [(self.dataset_ids[ids] == i).sum().int().item() \
+                for i in range(len(self.sizes))]
+            yield from ids
+
+
+class MDAspectRatioGroupedDataset(torch.utils.data.IterableDataset):
+    def __init__(self, dataset, batch_size, num_datasets):
+        """
+        """
+        self.dataset = dataset
+        self.batch_size = batch_size
+        self._buckets = [[] for _ in range(2 * num_datasets)]
+
+    def __iter__(self):
+        for d in self.dataset:
+            w, h = d["width"], d["height"]
+            aspect_ratio_bucket_id = 0 if w > h else 1
+            bucket_id = d['dataset_source'] * 2 + aspect_ratio_bucket_id
+            bucket = self._buckets[bucket_id]
+            bucket.append(d)
+            if len(bucket) == self.batch_size:
+                yield bucket[:]
+                del bucket[:]
+
+
+class DIFFMDAspectRatioGroupedDataset(torch.utils.data.IterableDataset):
+    def __init__(self, dataset, batch_sizes, num_datasets):
+        """
+        """
+        self.dataset = dataset
+        self.batch_sizes = batch_sizes
+        self._buckets = [[] for _ in range(2 * num_datasets)]
+
+    def __iter__(self):
+        for d in self.dataset:
+            w, h = d["width"], d["height"]
+            aspect_ratio_bucket_id = 0 if w > h else 1
+            bucket_id = d['dataset_source'] * 2 + aspect_ratio_bucket_id
+            bucket = self._buckets[bucket_id]
+            bucket.append(d)
+            if len(bucket) == self.batch_sizes[d['dataset_source']]:
+                yield bucket[:]
+                del bucket[:]
+
+
+def repeat_factors_from_tag_frequency(dataset_dicts, repeat_thresh):
+    """
+    """
+    category_freq = defaultdict(int)
+    for dataset_dict in dataset_dicts:
+        cat_ids = dataset_dict['pos_category_ids']
+        for cat_id in cat_ids:
+            category_freq[cat_id] += 1
+    num_images = len(dataset_dicts)
+    for k, v in category_freq.items():
+        category_freq[k] = v / num_images
+
+    category_rep = {
+        cat_id: max(1.0, math.sqrt(repeat_thresh / cat_freq))
+        for cat_id, cat_freq in category_freq.items()
+    }
+
+    rep_factors = []
+    for dataset_dict in dataset_dicts:
+        cat_ids = dataset_dict['pos_category_ids']
+        rep_factor = max({category_rep[cat_id] for cat_id in cat_ids}, default=1.0)
+        rep_factors.append(rep_factor)
+
+    return torch.tensor(rep_factors, dtype=torch.float32)

detic/data/custom_dataset_mapper.py

@@ -0,0 +1,280 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
+import copy
+import logging
+import numpy as np
+from typing import List, Optional, Union
+import torch
+import pycocotools.mask as mask_util
+
+from detectron2.config import configurable
+
+from detectron2.data import detection_utils as utils
+from detectron2.data.detection_utils import transform_keypoint_annotations
+from detectron2.data import transforms as T
+from detectron2.data.dataset_mapper import DatasetMapper
+from detectron2.structures import Boxes, BoxMode, Instances
+from detectron2.structures import Keypoints, PolygonMasks, BitMasks
+from fvcore.transforms.transform import TransformList
+from .custom_build_augmentation import build_custom_augmentation
+from .tar_dataset import DiskTarDataset
+
+__all__ = ["CustomDatasetMapper"]
+
+class CustomDatasetMapper(DatasetMapper):
+    @configurable
+    def __init__(self, is_train: bool, 
+        with_ann_type=False,
+        dataset_ann=[],
+        use_diff_bs_size=False,
+        dataset_augs=[],
+        is_debug=False,
+        use_tar_dataset=False,
+        tarfile_path='',
+        tar_index_dir='',
+        **kwargs):
+        """
+        add image labels
+        """
+        self.with_ann_type = with_ann_type
+        self.dataset_ann = dataset_ann
+        self.use_diff_bs_size = use_diff_bs_size
+        if self.use_diff_bs_size and is_train:
+            self.dataset_augs = [T.AugmentationList(x) for x in dataset_augs]
+        self.is_debug = is_debug
+        self.use_tar_dataset = use_tar_dataset
+        if self.use_tar_dataset:
+            print('Using tar dataset')
+            self.tar_dataset = DiskTarDataset(tarfile_path, tar_index_dir)
+        super().__init__(is_train, **kwargs)
+ 
+
+    @classmethod
+    def from_config(cls, cfg, is_train: bool = True):
+        ret = super().from_config(cfg, is_train)
+        ret.update({
+            'with_ann_type': cfg.WITH_IMAGE_LABELS,
+            'dataset_ann': cfg.DATALOADER.DATASET_ANN,
+            'use_diff_bs_size': cfg.DATALOADER.USE_DIFF_BS_SIZE,
+            'is_debug': cfg.IS_DEBUG,
+            'use_tar_dataset': cfg.DATALOADER.USE_TAR_DATASET,
+            'tarfile_path': cfg.DATALOADER.TARFILE_PATH,
+            'tar_index_dir': cfg.DATALOADER.TAR_INDEX_DIR,
+        })
+        if ret['use_diff_bs_size'] and is_train:
+            if cfg.INPUT.CUSTOM_AUG == 'EfficientDetResizeCrop':
+                dataset_scales = cfg.DATALOADER.DATASET_INPUT_SCALE
+                dataset_sizes = cfg.DATALOADER.DATASET_INPUT_SIZE
+                ret['dataset_augs'] = [
+                    build_custom_augmentation(cfg, True, scale, size) \
+                        for scale, size in zip(dataset_scales, dataset_sizes)]
+            else:
+                assert cfg.INPUT.CUSTOM_AUG == 'ResizeShortestEdge'
+                min_sizes = cfg.DATALOADER.DATASET_MIN_SIZES
+                max_sizes = cfg.DATALOADER.DATASET_MAX_SIZES
+                ret['dataset_augs'] = [
+                    build_custom_augmentation(
+                        cfg, True, min_size=mi, max_size=ma) \
+                        for mi, ma in zip(min_sizes, max_sizes)]
+        else:
+            ret['dataset_augs'] = []
+
+        return ret
+
+    def __call__(self, dataset_dict):
+        """
+        include image labels
+        """
+        dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
+        # USER: Write your own image loading if it's not from a file
+        if 'file_name' in dataset_dict:
+            ori_image = utils.read_image(
+                dataset_dict["file_name"], format=self.image_format)
+        else:
+            ori_image, _, _ = self.tar_dataset[dataset_dict["tar_index"]]
+            ori_image = utils._apply_exif_orientation(ori_image)
+            ori_image = utils.convert_PIL_to_numpy(ori_image, self.image_format)
+        utils.check_image_size(dataset_dict, ori_image)
+
+        # USER: Remove if you don't do semantic/panoptic segmentation.
+        if "sem_seg_file_name" in dataset_dict:
+            sem_seg_gt = utils.read_image(
+                dataset_dict.pop("sem_seg_file_name"), "L").squeeze(2)
+        else:
+            sem_seg_gt = None
+
+        if self.is_debug:
+            dataset_dict['dataset_source'] = 0
+
+        not_full_labeled = 'dataset_source' in dataset_dict and \
+            self.with_ann_type and \
+                self.dataset_ann[dataset_dict['dataset_source']] != 'box'
+
+        aug_input = T.AugInput(copy.deepcopy(ori_image), sem_seg=sem_seg_gt)
+        if self.use_diff_bs_size and self.is_train:
+            transforms = \
+                self.dataset_augs[dataset_dict['dataset_source']](aug_input)
+        else:
+            transforms = self.augmentations(aug_input)
+        image, sem_seg_gt = aug_input.image, aug_input.sem_seg
+
+        image_shape = image.shape[:2]  # h, w
+        dataset_dict["image"] = torch.as_tensor(
+            np.ascontiguousarray(image.transpose(2, 0, 1)))
+        
+        if sem_seg_gt is not None:
+            dataset_dict["sem_seg"] = torch.as_tensor(sem_seg_gt.astype("long"))
+
+        # USER: Remove if you don't use pre-computed proposals.
+        # Most users would not need this feature.
+        if self.proposal_topk is not None:
+            utils.transform_proposals(
+                dataset_dict, image_shape, transforms, 
+                proposal_topk=self.proposal_topk
+            )
+
+        if not self.is_train:
+            # USER: Modify this if you want to keep them for some reason.
+            dataset_dict.pop("annotations", None)
+            dataset_dict.pop("sem_seg_file_name", None)
+            return dataset_dict
+
+        if "annotations" in dataset_dict:
+            # USER: Modify this if you want to keep them for some reason.
+            for anno in dataset_dict["annotations"]:
+                if not self.use_instance_mask:
+                    anno.pop("segmentation", None)
+                if not self.use_keypoint:
+                    anno.pop("keypoints", None)
+
+            # USER: Implement additional transformations if you have other types of data
+            all_annos = [
+                (utils.transform_instance_annotations(
+                    obj, transforms, image_shape, 
+                    keypoint_hflip_indices=self.keypoint_hflip_indices,
+                ),  obj.get("iscrowd", 0))
+                for obj in dataset_dict.pop("annotations")
+            ]
+            annos = [ann[0] for ann in all_annos if ann[1] == 0]
+            instances = utils.annotations_to_instances(
+                annos, image_shape, mask_format=self.instance_mask_format
+            )
+            
+            del all_annos
+            if self.recompute_boxes:
+                instances.gt_boxes = instances.gt_masks.get_bounding_boxes()
+            dataset_dict["instances"] = utils.filter_empty_instances(instances)
+        if self.with_ann_type:
+            dataset_dict["pos_category_ids"] = dataset_dict.get(
+                'pos_category_ids', [])
+            dataset_dict["ann_type"] = \
+                self.dataset_ann[dataset_dict['dataset_source']]
+        if self.is_debug and (('pos_category_ids' not in dataset_dict) or \
+            (dataset_dict['pos_category_ids'] == [])):
+            dataset_dict['pos_category_ids'] = [x for x in sorted(set(
+                dataset_dict['instances'].gt_classes.tolist()
+            ))]
+        return dataset_dict
+
+# DETR augmentation
+def build_transform_gen(cfg, is_train):
+    """
+    """
+    if is_train:
+        min_size = cfg.INPUT.MIN_SIZE_TRAIN
+        max_size = cfg.INPUT.MAX_SIZE_TRAIN
+        sample_style = cfg.INPUT.MIN_SIZE_TRAIN_SAMPLING
+    else:
+        min_size = cfg.INPUT.MIN_SIZE_TEST
+        max_size = cfg.INPUT.MAX_SIZE_TEST
+        sample_style = "choice"
+    if sample_style == "range":
+        assert len(min_size) == 2, "more than 2 ({}) min_size(s) are provided for ranges".format(len(min_size))
+
+    logger = logging.getLogger(__name__)
+    tfm_gens = []
+    if is_train:
+        tfm_gens.append(T.RandomFlip())
+    tfm_gens.append(T.ResizeShortestEdge(min_size, max_size, sample_style))
+    if is_train:
+        logger.info("TransformGens used in training: " + str(tfm_gens))
+    return tfm_gens
+
+
+class DetrDatasetMapper:
+    """
+    A callable which takes a dataset dict in Detectron2 Dataset format,
+    and map it into a format used by DETR.
+    The callable currently does the following:
+    1. Read the image from "file_name"
+    2. Applies geometric transforms to the image and annotation
+    3. Find and applies suitable cropping to the image and annotation
+    4. Prepare image and annotation to Tensors
+    """
+
+    def __init__(self, cfg, is_train=True):
+        if cfg.INPUT.CROP.ENABLED and is_train:
+            self.crop_gen = [
+                T.ResizeShortestEdge([400, 500, 600], sample_style="choice"),
+                T.RandomCrop(cfg.INPUT.CROP.TYPE, cfg.INPUT.CROP.SIZE),
+            ]
+        else:
+            self.crop_gen = None
+
+        self.mask_on = cfg.MODEL.MASK_ON
+        self.tfm_gens = build_transform_gen(cfg, is_train)
+        logging.getLogger(__name__).info(
+            "Full TransformGens used in training: {}, crop: {}".format(str(self.tfm_gens), str(self.crop_gen))
+        )
+
+        self.img_format = cfg.INPUT.FORMAT
+        self.is_train = is_train
+
+    def __call__(self, dataset_dict):
+        """
+        Args:
+            dataset_dict (dict): Metadata of one image, in Detectron2 Dataset format.
+        Returns:
+            dict: a format that builtin models in detectron2 accept
+        """
+        dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
+        image = utils.read_image(dataset_dict["file_name"], format=self.img_format)
+        utils.check_image_size(dataset_dict, image)
+
+        if self.crop_gen is None:
+            image, transforms = T.apply_transform_gens(self.tfm_gens, image)
+        else:
+            if np.random.rand() > 0.5:
+                image, transforms = T.apply_transform_gens(self.tfm_gens, image)
+            else:
+                image, transforms = T.apply_transform_gens(
+                    self.tfm_gens[:-1] + self.crop_gen + self.tfm_gens[-1:], image
+                )
+
+        image_shape = image.shape[:2]  # h, w
+
+        # Pytorch's dataloader is efficient on torch.Tensor due to shared-memory,
+        # but not efficient on large generic data structures due to the use of pickle & mp.Queue.
+        # Therefore it's important to use torch.Tensor.
+        dataset_dict["image"] = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))
+
+        if not self.is_train:
+            # USER: Modify this if you want to keep them for some reason.
+            dataset_dict.pop("annotations", None)
+            return dataset_dict
+
+        if "annotations" in dataset_dict:
+            # USER: Modify this if you want to keep them for some reason.
+            for anno in dataset_dict["annotations"]:
+                if not self.mask_on:
+                    anno.pop("segmentation", None)
+                anno.pop("keypoints", None)
+
+            # USER: Implement additional transformations if you have other types of data
+            annos = [
+                utils.transform_instance_annotations(obj, transforms, image_shape)
+                for obj in dataset_dict.pop("annotations")
+                if obj.get("iscrowd", 0) == 0
+            ]
+            instances = utils.annotations_to_instances(annos, image_shape)
+            dataset_dict["instances"] = utils.filter_empty_instances(instances)
+        return dataset_dict

detic/data/datasets/cc.py

@@ -0,0 +1,23 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+import logging
+import os
+
+from detectron2.data.datasets.builtin_meta import _get_builtin_metadata
+from detectron2.data.datasets.lvis import get_lvis_instances_meta
+from .lvis_v1 import custom_register_lvis_instances
+
+_CUSTOM_SPLITS = {
+    "cc3m_v1_val": ("cc3m/validation/", "cc3m/val_image_info.json"),
+    "cc3m_v1_train": ("cc3m/training/", "cc3m/train_image_info.json"),
+    "cc3m_v1_train_tags": ("cc3m/training/", "cc3m/train_image_info_tags.json"),
+
+}
+
+for key, (image_root, json_file) in _CUSTOM_SPLITS.items():
+    custom_register_lvis_instances(
+        key,
+        get_lvis_instances_meta('lvis_v1'),
+        os.path.join("datasets", json_file) if "://" not in json_file else json_file,
+        os.path.join("datasets", image_root),
+    )
+

detic/data/datasets/coco_zeroshot.py

@@ -0,0 +1,121 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+import os
+
+from detectron2.data.datasets.register_coco import register_coco_instances
+from detectron2.data.datasets.builtin_meta import _get_builtin_metadata
+from .lvis_v1 import custom_register_lvis_instances
+
+categories_seen = [
+    {'id': 1, 'name': 'person'},
+    {'id': 2, 'name': 'bicycle'},
+    {'id': 3, 'name': 'car'},
+    {'id': 4, 'name': 'motorcycle'},
+    {'id': 7, 'name': 'train'},
+    {'id': 8, 'name': 'truck'},
+    {'id': 9, 'name': 'boat'},
+    {'id': 15, 'name': 'bench'},
+    {'id': 16, 'name': 'bird'},
+    {'id': 19, 'name': 'horse'},
+    {'id': 20, 'name': 'sheep'},
+    {'id': 23, 'name': 'bear'},
+    {'id': 24, 'name': 'zebra'},
+    {'id': 25, 'name': 'giraffe'},
+    {'id': 27, 'name': 'backpack'},
+    {'id': 31, 'name': 'handbag'},
+    {'id': 33, 'name': 'suitcase'},
+    {'id': 34, 'name': 'frisbee'},
+    {'id': 35, 'name': 'skis'},
+    {'id': 38, 'name': 'kite'},
+    {'id': 42, 'name': 'surfboard'},
+    {'id': 44, 'name': 'bottle'},
+    {'id': 48, 'name': 'fork'},
+    {'id': 50, 'name': 'spoon'},
+    {'id': 51, 'name': 'bowl'},
+    {'id': 52, 'name': 'banana'},
+    {'id': 53, 'name': 'apple'},
+    {'id': 54, 'name': 'sandwich'},
+    {'id': 55, 'name': 'orange'},
+    {'id': 56, 'name': 'broccoli'},
+    {'id': 57, 'name': 'carrot'},
+    {'id': 59, 'name': 'pizza'},
+    {'id': 60, 'name': 'donut'},
+    {'id': 62, 'name': 'chair'},
+    {'id': 65, 'name': 'bed'},
+    {'id': 70, 'name': 'toilet'},
+    {'id': 72, 'name': 'tv'},
+    {'id': 73, 'name': 'laptop'},
+    {'id': 74, 'name': 'mouse'},
+    {'id': 75, 'name': 'remote'},
+    {'id': 78, 'name': 'microwave'},
+    {'id': 79, 'name': 'oven'},
+    {'id': 80, 'name': 'toaster'},
+    {'id': 82, 'name': 'refrigerator'},
+    {'id': 84, 'name': 'book'},
+    {'id': 85, 'name': 'clock'},
+    {'id': 86, 'name': 'vase'},
+    {'id': 90, 'name': 'toothbrush'},
+]
+
+categories_unseen = [
+    {'id': 5, 'name': 'airplane'},
+    {'id': 6, 'name': 'bus'},
+    {'id': 17, 'name': 'cat'},
+    {'id': 18, 'name': 'dog'},
+    {'id': 21, 'name': 'cow'},
+    {'id': 22, 'name': 'elephant'},
+    {'id': 28, 'name': 'umbrella'},
+    {'id': 32, 'name': 'tie'},
+    {'id': 36, 'name': 'snowboard'},
+    {'id': 41, 'name': 'skateboard'},
+    {'id': 47, 'name': 'cup'},
+    {'id': 49, 'name': 'knife'},
+    {'id': 61, 'name': 'cake'},
+    {'id': 63, 'name': 'couch'},
+    {'id': 76, 'name': 'keyboard'},
+    {'id': 81, 'name': 'sink'},
+    {'id': 87, 'name': 'scissors'},
+]
+
+def _get_metadata(cat):
+    if cat == 'all':
+        return _get_builtin_metadata('coco')
+    elif cat == 'seen':
+        id_to_name = {x['id']: x['name'] for x in categories_seen}
+    else:
+        assert cat == 'unseen'
+        id_to_name = {x['id']: x['name'] for x in categories_unseen}
+
+    thing_dataset_id_to_contiguous_id = {
+        x: i for i, x in enumerate(sorted(id_to_name))}
+    thing_classes = [id_to_name[k] for k in sorted(id_to_name)]
+    return {
+        "thing_dataset_id_to_contiguous_id": thing_dataset_id_to_contiguous_id,
+        "thing_classes": thing_classes}
+
+_PREDEFINED_SPLITS_COCO = {
+    "coco_zeroshot_train": ("coco/train2017", "coco/zero-shot/instances_train2017_seen_2.json", 'seen'),
+    "coco_zeroshot_val": ("coco/val2017", "coco/zero-shot/instances_val2017_unseen_2.json", 'unseen'),
+    "coco_not_zeroshot_val": ("coco/val2017", "coco/zero-shot/instances_val2017_seen_2.json", 'seen'),
+    "coco_generalized_zeroshot_val": ("coco/val2017", "coco/zero-shot/instances_val2017_all_2_oriorder.json", 'all'),
+    "coco_zeroshot_train_oriorder": ("coco/train2017", "coco/zero-shot/instances_train2017_seen_2_oriorder.json", 'all'),
+}
+
+for key, (image_root, json_file, cat) in _PREDEFINED_SPLITS_COCO.items():
+    register_coco_instances(
+        key,
+        _get_metadata(cat),
+        os.path.join("datasets", json_file) if "://" not in json_file else json_file,
+        os.path.join("datasets", image_root),
+    )
+
+_CUSTOM_SPLITS_COCO = {
+    "cc3m_coco_train_tags": ("cc3m/training/", "cc3m/coco_train_image_info_tags.json"),
+    "coco_caption_train_tags": ("coco/train2017/", "coco/annotations/captions_train2017_tags_allcaps.json"),}
+
+for key, (image_root, json_file) in _CUSTOM_SPLITS_COCO.items():
+    custom_register_lvis_instances(
+        key,
+        _get_builtin_metadata('coco'),
+        os.path.join("datasets", json_file) if "://" not in json_file else json_file,
+        os.path.join("datasets", image_root),
+    )

detic/data/datasets/imagenet.py

@@ -0,0 +1,41 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+import logging
+import os
+
+from detectron2.data import DatasetCatalog, MetadataCatalog
+from detectron2.data.datasets.lvis import get_lvis_instances_meta
+from .lvis_v1 import custom_load_lvis_json, get_lvis_22k_meta
+def custom_register_imagenet_instances(name, metadata, json_file, image_root):
+    """
+    """
+    DatasetCatalog.register(name, lambda: custom_load_lvis_json(
+        json_file, image_root, name))
+    MetadataCatalog.get(name).set(
+        json_file=json_file, image_root=image_root, 
+        evaluator_type="imagenet", **metadata
+    )
+
+_CUSTOM_SPLITS_IMAGENET = {
+    "imagenet_lvis_v1": ("imagenet/ImageNet-LVIS/", "imagenet/annotations/imagenet_lvis_image_info.json"),
+}
+
+for key, (image_root, json_file) in _CUSTOM_SPLITS_IMAGENET.items():
+    custom_register_imagenet_instances(
+        key,
+        get_lvis_instances_meta('lvis_v1'),
+        os.path.join("datasets", json_file) if "://" not in json_file else json_file,
+        os.path.join("datasets", image_root),
+    )
+
+
+_CUSTOM_SPLITS_IMAGENET_22K = {
+    "imagenet_lvis-22k": ("imagenet/ImageNet-LVIS/", "imagenet/annotations/imagenet-22k_image_info_lvis-22k.json"),
+}
+
+for key, (image_root, json_file) in _CUSTOM_SPLITS_IMAGENET_22K.items():
+    custom_register_imagenet_instances(
+        key,
+        get_lvis_22k_meta(),
+        os.path.join("datasets", json_file) if "://" not in json_file else json_file,
+        os.path.join("datasets", image_root),
+    )

detic/data/datasets/lvis_v1.py

@@ -0,0 +1,155 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+import logging
+import os
+
+from fvcore.common.timer import Timer
+from detectron2.structures import BoxMode
+from fvcore.common.file_io import PathManager
+from detectron2.data import DatasetCatalog, MetadataCatalog
+from detectron2.data.datasets.lvis import get_lvis_instances_meta
+
+logger = logging.getLogger(__name__)
+
+__all__ = ["custom_load_lvis_json", "custom_register_lvis_instances"]
+
+
+def custom_register_lvis_instances(name, metadata, json_file, image_root):
+    """
+    """
+    DatasetCatalog.register(name, lambda: custom_load_lvis_json(
+        json_file, image_root, name))
+    MetadataCatalog.get(name).set(
+        json_file=json_file, image_root=image_root, 
+        evaluator_type="lvis", **metadata
+    )
+
+
+def custom_load_lvis_json(json_file, image_root, dataset_name=None):
+    '''
+    Modifications:
+      use `file_name`
+      convert neg_category_ids
+      add pos_category_ids
+    '''
+    from lvis import LVIS
+
+    json_file = PathManager.get_local_path(json_file)
+
+    timer = Timer()
+    lvis_api = LVIS(json_file)
+    if timer.seconds() > 1:
+        logger.info("Loading {} takes {:.2f} seconds.".format(
+            json_file, timer.seconds()))
+
+    catid2contid = {x['id']: i for i, x in enumerate(
+        sorted(lvis_api.dataset['categories'], key=lambda x: x['id']))}
+    if len(lvis_api.dataset['categories']) == 1203:
+        for x in lvis_api.dataset['categories']:
+            assert catid2contid[x['id']] == x['id'] - 1
+    img_ids = sorted(lvis_api.imgs.keys())
+    imgs = lvis_api.load_imgs(img_ids)
+    anns = [lvis_api.img_ann_map[img_id] for img_id in img_ids]
+
+    ann_ids = [ann["id"] for anns_per_image in anns for ann in anns_per_image]
+    assert len(set(ann_ids)) == len(ann_ids), \
+        "Annotation ids in '{}' are not unique".format(json_file)
+
+    imgs_anns = list(zip(imgs, anns))
+    logger.info("Loaded {} images in the LVIS v1 format from {}".format(
+        len(imgs_anns), json_file))
+
+    dataset_dicts = []
+
+    for (img_dict, anno_dict_list) in imgs_anns:
+        record = {}
+        if "file_name" in img_dict:
+            file_name = img_dict["file_name"]
+            if img_dict["file_name"].startswith("COCO"):
+                file_name = file_name[-16:]
+            record["file_name"] = os.path.join(image_root, file_name)
+        elif 'coco_url' in img_dict:
+            # e.g., http://images.cocodataset.org/train2017/000000391895.jpg
+            file_name = img_dict["coco_url"][30:]
+            record["file_name"] = os.path.join(image_root, file_name)
+        elif 'tar_index' in img_dict:
+            record['tar_index'] = img_dict['tar_index']
+        
+        record["height"] = img_dict["height"]
+        record["width"] = img_dict["width"]
+        record["not_exhaustive_category_ids"] = img_dict.get(
+            "not_exhaustive_category_ids", [])
+        record["neg_category_ids"] = img_dict.get("neg_category_ids", [])
+        # NOTE: modified by Xingyi: convert to 0-based
+        record["neg_category_ids"] = [
+            catid2contid[x] for x in record["neg_category_ids"]]
+        if 'pos_category_ids' in img_dict:
+            record['pos_category_ids'] = [
+                catid2contid[x] for x in img_dict.get("pos_category_ids", [])]
+        if 'captions' in img_dict:
+            record['captions'] = img_dict['captions']
+        if 'caption_features' in img_dict:
+            record['caption_features'] = img_dict['caption_features']
+        image_id = record["image_id"] = img_dict["id"]
+
+        objs = []
+        for anno in anno_dict_list:
+            assert anno["image_id"] == image_id
+            if anno.get('iscrowd', 0) > 0:
+                continue
+            obj = {"bbox": anno["bbox"], "bbox_mode": BoxMode.XYWH_ABS}
+            obj["category_id"] = catid2contid[anno['category_id']] 
+            if 'segmentation' in anno:
+                segm = anno["segmentation"]
+                valid_segm = [poly for poly in segm \
+                    if len(poly) % 2 == 0 and len(poly) >= 6]
+                # assert len(segm) == len(
+                #     valid_segm
+                # ), "Annotation contains an invalid polygon with < 3 points"
+                if not len(segm) == len(valid_segm):
+                    print('Annotation contains an invalid polygon with < 3 points')
+                assert len(segm) > 0
+                obj["segmentation"] = segm
+            objs.append(obj)
+        record["annotations"] = objs
+        dataset_dicts.append(record)
+
+    return dataset_dicts
+
+_CUSTOM_SPLITS_LVIS = {
+    "lvis_v1_train+coco": ("coco/", "lvis/lvis_v1_train+coco_mask.json"),
+    "lvis_v1_train_norare": ("coco/", "lvis/lvis_v1_train_norare.json"),
+}
+
+
+for key, (image_root, json_file) in _CUSTOM_SPLITS_LVIS.items():
+    custom_register_lvis_instances(
+        key,
+        get_lvis_instances_meta(key),
+        os.path.join("datasets", json_file) if "://" not in json_file else json_file,
+        os.path.join("datasets", image_root),
+    )
+
+
+def get_lvis_22k_meta():
+    from .lvis_22k_categories import CATEGORIES
+    cat_ids = [k["id"] for k in CATEGORIES]
+    assert min(cat_ids) == 1 and max(cat_ids) == len(
+        cat_ids
+    ), "Category ids are not in [1, #categories], as expected"
+    # Ensure that the category list is sorted by id
+    lvis_categories = sorted(CATEGORIES, key=lambda x: x["id"])
+    thing_classes = [k["name"] for k in lvis_categories]
+    meta = {"thing_classes": thing_classes}
+    return meta
+
+_CUSTOM_SPLITS_LVIS_22K = {
+    "lvis_v1_train_22k": ("coco/", "lvis/lvis_v1_train_lvis-22k.json"),
+}
+
+for key, (image_root, json_file) in _CUSTOM_SPLITS_LVIS_22K.items():
+    custom_register_lvis_instances(
+        key,
+        get_lvis_22k_meta(),
+        os.path.join("datasets", json_file) if "://" not in json_file else json_file,
+        os.path.join("datasets", image_root),
+    )

detic/data/datasets/objects365.py

@@ -0,0 +1,770 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+from detectron2.data.datasets.register_coco import register_coco_instances
+import os
+
+# categories_v2 = [
+#     {'id': 1, 'name': 'Person'},
+#     {'id': 2, 'name': 'Sneakers'},
+#     {'id': 3, 'name': 'Chair'},
+#     {'id': 4, 'name': 'Other Shoes'},
+#     {'id': 5, 'name': 'Hat'},
+#     {'id': 6, 'name': 'Car'},
+#     {'id': 7, 'name': 'Lamp'},
+#     {'id': 8, 'name': 'Glasses'},
+#     {'id': 9, 'name': 'Bottle'},
+#     {'id': 10, 'name': 'Desk'},
+#     {'id': 11, 'name': 'Cup'},
+#     {'id': 12, 'name': 'Street Lights'},
+#     {'id': 13, 'name': 'Cabinet/shelf'},
+#     {'id': 14, 'name': 'Handbag/Satchel'},
+#     {'id': 15, 'name': 'Bracelet'},
+#     {'id': 16, 'name': 'Plate'},
+#     {'id': 17, 'name': 'Picture/Frame'},
+#     {'id': 18, 'name': 'Helmet'},
+#     {'id': 19, 'name': 'Book'},
+#     {'id': 20, 'name': 'Gloves'},
+#     {'id': 21, 'name': 'Storage box'},
+#     {'id': 22, 'name': 'Boat'},
+#     {'id': 23, 'name': 'Leather Shoes'},
+#     {'id': 24, 'name': 'Flower'},
+#     {'id': 25, 'name': 'Bench'},
+#     {'id': 26, 'name': 'Potted Plant'},
+#     {'id': 27, 'name': 'Bowl/Basin'},
+#     {'id': 28, 'name': 'Flag'},
+#     {'id': 29, 'name': 'Pillow'},
+#     {'id': 30, 'name': 'Boots'},
+#     {'id': 31, 'name': 'Vase'},
+#     {'id': 32, 'name': 'Microphone'},
+#     {'id': 33, 'name': 'Necklace'},
+#     {'id': 34, 'name': 'Ring'},
+#     {'id': 35, 'name': 'SUV'},
+#     {'id': 36, 'name': 'Wine Glass'},
+#     {'id': 37, 'name': 'Belt'},
+#     {'id': 38, 'name': 'Moniter/TV'},
+#     {'id': 39, 'name': 'Backpack'},
+#     {'id': 40, 'name': 'Umbrella'},
+#     {'id': 41, 'name': 'Traffic Light'},
+#     {'id': 42, 'name': 'Speaker'},
+#     {'id': 43, 'name': 'Watch'},
+#     {'id': 44, 'name': 'Tie'},
+#     {'id': 45, 'name': 'Trash bin Can'},
+#     {'id': 46, 'name': 'Slippers'},
+#     {'id': 47, 'name': 'Bicycle'},
+#     {'id': 48, 'name': 'Stool'},
+#     {'id': 49, 'name': 'Barrel/bucket'},
+#     {'id': 50, 'name': 'Van'},
+#     {'id': 51, 'name': 'Couch'},
+#     {'id': 52, 'name': 'Sandals'},
+#     {'id': 53, 'name': 'Bakset'},
+#     {'id': 54, 'name': 'Drum'},
+#     {'id': 55, 'name': 'Pen/Pencil'},
+#     {'id': 56, 'name': 'Bus'},
+#     {'id': 57, 'name': 'Wild Bird'},
+#     {'id': 58, 'name': 'High Heels'},
+#     {'id': 59, 'name': 'Motorcycle'},
+#     {'id': 60, 'name': 'Guitar'},
+#     {'id': 61, 'name': 'Carpet'},
+#     {'id': 62, 'name': 'Cell Phone'},
+#     {'id': 63, 'name': 'Bread'},
+#     {'id': 64, 'name': 'Camera'},
+#     {'id': 65, 'name': 'Canned'},
+#     {'id': 66, 'name': 'Truck'},
+#     {'id': 67, 'name': 'Traffic cone'},
+#     {'id': 68, 'name': 'Cymbal'},
+#     {'id': 69, 'name': 'Lifesaver'},
+#     {'id': 70, 'name': 'Towel'},
+#     {'id': 71, 'name': 'Stuffed Toy'},
+#     {'id': 72, 'name': 'Candle'},
+#     {'id': 73, 'name': 'Sailboat'},
+#     {'id': 74, 'name': 'Laptop'},
+#     {'id': 75, 'name': 'Awning'},
+#     {'id': 76, 'name': 'Bed'},
+#     {'id': 77, 'name': 'Faucet'},
+#     {'id': 78, 'name': 'Tent'},
+#     {'id': 79, 'name': 'Horse'},
+#     {'id': 80, 'name': 'Mirror'},
+#     {'id': 81, 'name': 'Power outlet'},
+#     {'id': 82, 'name': 'Sink'},
+#     {'id': 83, 'name': 'Apple'},
+#     {'id': 84, 'name': 'Air Conditioner'},
+#     {'id': 85, 'name': 'Knife'},
+#     {'id': 86, 'name': 'Hockey Stick'},
+#     {'id': 87, 'name': 'Paddle'},
+#     {'id': 88, 'name': 'Pickup Truck'},
+#     {'id': 89, 'name': 'Fork'},
+#     {'id': 90, 'name': 'Traffic Sign'},
+#     {'id': 91, 'name': 'Ballon'},
+#     {'id': 92, 'name': 'Tripod'},
+#     {'id': 93, 'name': 'Dog'},
+#     {'id': 94, 'name': 'Spoon'},
+#     {'id': 95, 'name': 'Clock'},
+#     {'id': 96, 'name': 'Pot'},
+#     {'id': 97, 'name': 'Cow'},
+#     {'id': 98, 'name': 'Cake'},
+#     {'id': 99, 'name': 'Dinning Table'},
+#     {'id': 100, 'name': 'Sheep'},
+#     {'id': 101, 'name': 'Hanger'},
+#     {'id': 102, 'name': 'Blackboard/Whiteboard'},
+#     {'id': 103, 'name': 'Napkin'},
+#     {'id': 104, 'name': 'Other Fish'},
+#     {'id': 105, 'name': 'Orange/Tangerine'},
+#     {'id': 106, 'name': 'Toiletry'},
+#     {'id': 107, 'name': 'Keyboard'},
+#     {'id': 108, 'name': 'Tomato'},
+#     {'id': 109, 'name': 'Lantern'},
+#     {'id': 110, 'name': 'Machinery Vehicle'},
+#     {'id': 111, 'name': 'Fan'},
+#     {'id': 112, 'name': 'Green Vegetables'},
+#     {'id': 113, 'name': 'Banana'},
+#     {'id': 114, 'name': 'Baseball Glove'},
+#     {'id': 115, 'name': 'Airplane'},
+#     {'id': 116, 'name': 'Mouse'},
+#     {'id': 117, 'name': 'Train'},
+#     {'id': 118, 'name': 'Pumpkin'},
+#     {'id': 119, 'name': 'Soccer'},
+#     {'id': 120, 'name': 'Skiboard'},
+#     {'id': 121, 'name': 'Luggage'},
+#     {'id': 122, 'name': 'Nightstand'},
+#     {'id': 123, 'name': 'Tea pot'},
+#     {'id': 124, 'name': 'Telephone'},
+#     {'id': 125, 'name': 'Trolley'},
+#     {'id': 126, 'name': 'Head Phone'},
+#     {'id': 127, 'name': 'Sports Car'},
+#     {'id': 128, 'name': 'Stop Sign'},
+#     {'id': 129, 'name': 'Dessert'},
+#     {'id': 130, 'name': 'Scooter'},
+#     {'id': 131, 'name': 'Stroller'},
+#     {'id': 132, 'name': 'Crane'},
+#     {'id': 133, 'name': 'Remote'},
+#     {'id': 134, 'name': 'Refrigerator'},
+#     {'id': 135, 'name': 'Oven'},
+#     {'id': 136, 'name': 'Lemon'},
+#     {'id': 137, 'name': 'Duck'},
+#     {'id': 138, 'name': 'Baseball Bat'},
+#     {'id': 139, 'name': 'Surveillance Camera'},
+#     {'id': 140, 'name': 'Cat'},
+#     {'id': 141, 'name': 'Jug'},
+#     {'id': 142, 'name': 'Broccoli'},
+#     {'id': 143, 'name': 'Piano'},
+#     {'id': 144, 'name': 'Pizza'},
+#     {'id': 145, 'name': 'Elephant'},
+#     {'id': 146, 'name': 'Skateboard'},
+#     {'id': 147, 'name': 'Surfboard'},
+#     {'id': 148, 'name': 'Gun'},
+#     {'id': 149, 'name': 'Skating and Skiing shoes'},
+#     {'id': 150, 'name': 'Gas stove'},
+#     {'id': 151, 'name': 'Donut'},
+#     {'id': 152, 'name': 'Bow Tie'},
+#     {'id': 153, 'name': 'Carrot'},
+#     {'id': 154, 'name': 'Toilet'},
+#     {'id': 155, 'name': 'Kite'},
+#     {'id': 156, 'name': 'Strawberry'},
+#     {'id': 157, 'name': 'Other Balls'},
+#     {'id': 158, 'name': 'Shovel'},
+#     {'id': 159, 'name': 'Pepper'},
+#     {'id': 160, 'name': 'Computer Box'},
+#     {'id': 161, 'name': 'Toilet Paper'},
+#     {'id': 162, 'name': 'Cleaning Products'},
+#     {'id': 163, 'name': 'Chopsticks'},
+#     {'id': 164, 'name': 'Microwave'},
+#     {'id': 165, 'name': 'Pigeon'},
+#     {'id': 166, 'name': 'Baseball'},
+#     {'id': 167, 'name': 'Cutting/chopping Board'},
+#     {'id': 168, 'name': 'Coffee Table'},
+#     {'id': 169, 'name': 'Side Table'},
+#     {'id': 170, 'name': 'Scissors'},
+#     {'id': 171, 'name': 'Marker'},
+#     {'id': 172, 'name': 'Pie'},
+#     {'id': 173, 'name': 'Ladder'},
+#     {'id': 174, 'name': 'Snowboard'},
+#     {'id': 175, 'name': 'Cookies'},
+#     {'id': 176, 'name': 'Radiator'},
+#     {'id': 177, 'name': 'Fire Hydrant'},
+#     {'id': 178, 'name': 'Basketball'},
+#     {'id': 179, 'name': 'Zebra'},
+#     {'id': 180, 'name': 'Grape'},
+#     {'id': 181, 'name': 'Giraffe'},
+#     {'id': 182, 'name': 'Potato'},
+#     {'id': 183, 'name': 'Sausage'},
+#     {'id': 184, 'name': 'Tricycle'},
+#     {'id': 185, 'name': 'Violin'},
+#     {'id': 186, 'name': 'Egg'},
+#     {'id': 187, 'name': 'Fire Extinguisher'},
+#     {'id': 188, 'name': 'Candy'},
+#     {'id': 189, 'name': 'Fire Truck'},
+#     {'id': 190, 'name': 'Billards'},
+#     {'id': 191, 'name': 'Converter'},
+#     {'id': 192, 'name': 'Bathtub'},
+#     {'id': 193, 'name': 'Wheelchair'},
+#     {'id': 194, 'name': 'Golf Club'},
+#     {'id': 195, 'name': 'Briefcase'},
+#     {'id': 196, 'name': 'Cucumber'},
+#     {'id': 197, 'name': 'Cigar/Cigarette '},
+#     {'id': 198, 'name': 'Paint Brush'},
+#     {'id': 199, 'name': 'Pear'},
+#     {'id': 200, 'name': 'Heavy Truck'},
+#     {'id': 201, 'name': 'Hamburger'},
+#     {'id': 202, 'name': 'Extractor'},
+#     {'id': 203, 'name': 'Extention Cord'},
+#     {'id': 204, 'name': 'Tong'},
+#     {'id': 205, 'name': 'Tennis Racket'},
+#     {'id': 206, 'name': 'Folder'},
+#     {'id': 207, 'name': 'American Football'},
+#     {'id': 208, 'name': 'earphone'},
+#     {'id': 209, 'name': 'Mask'},
+#     {'id': 210, 'name': 'Kettle'},
+#     {'id': 211, 'name': 'Tennis'},
+#     {'id': 212, 'name': 'Ship'},
+#     {'id': 213, 'name': 'Swing'},
+#     {'id': 214, 'name': 'Coffee Machine'},
+#     {'id': 215, 'name': 'Slide'},
+#     {'id': 216, 'name': 'Carriage'},
+#     {'id': 217, 'name': 'Onion'},
+#     {'id': 218, 'name': 'Green beans'},
+#     {'id': 219, 'name': 'Projector'},
+#     {'id': 220, 'name': 'Frisbee'},
+#     {'id': 221, 'name': 'Washing Machine/Drying Machine'},
+#     {'id': 222, 'name': 'Chicken'},
+#     {'id': 223, 'name': 'Printer'},
+#     {'id': 224, 'name': 'Watermelon'},
+#     {'id': 225, 'name': 'Saxophone'},
+#     {'id': 226, 'name': 'Tissue'},
+#     {'id': 227, 'name': 'Toothbrush'},
+#     {'id': 228, 'name': 'Ice cream'},
+#     {'id': 229, 'name': 'Hotair ballon'},
+#     {'id': 230, 'name': 'Cello'},
+#     {'id': 231, 'name': 'French Fries'},
+#     {'id': 232, 'name': 'Scale'},
+#     {'id': 233, 'name': 'Trophy'},
+#     {'id': 234, 'name': 'Cabbage'},
+#     {'id': 235, 'name': 'Hot dog'},
+#     {'id': 236, 'name': 'Blender'},
+#     {'id': 237, 'name': 'Peach'},
+#     {'id': 238, 'name': 'Rice'},
+#     {'id': 239, 'name': 'Wallet/Purse'},
+#     {'id': 240, 'name': 'Volleyball'},
+#     {'id': 241, 'name': 'Deer'},
+#     {'id': 242, 'name': 'Goose'},
+#     {'id': 243, 'name': 'Tape'},
+#     {'id': 244, 'name': 'Tablet'},
+#     {'id': 245, 'name': 'Cosmetics'},
+#     {'id': 246, 'name': 'Trumpet'},
+#     {'id': 247, 'name': 'Pineapple'},
+#     {'id': 248, 'name': 'Golf Ball'},
+#     {'id': 249, 'name': 'Ambulance'},
+#     {'id': 250, 'name': 'Parking meter'},
+#     {'id': 251, 'name': 'Mango'},
+#     {'id': 252, 'name': 'Key'},
+#     {'id': 253, 'name': 'Hurdle'},
+#     {'id': 254, 'name': 'Fishing Rod'},
+#     {'id': 255, 'name': 'Medal'},
+#     {'id': 256, 'name': 'Flute'},
+#     {'id': 257, 'name': 'Brush'},
+#     {'id': 258, 'name': 'Penguin'},
+#     {'id': 259, 'name': 'Megaphone'},
+#     {'id': 260, 'name': 'Corn'},
+#     {'id': 261, 'name': 'Lettuce'},
+#     {'id': 262, 'name': 'Garlic'},
+#     {'id': 263, 'name': 'Swan'},
+#     {'id': 264, 'name': 'Helicopter'},
+#     {'id': 265, 'name': 'Green Onion'},
+#     {'id': 266, 'name': 'Sandwich'},
+#     {'id': 267, 'name': 'Nuts'},
+#     {'id': 268, 'name': 'Speed Limit Sign'},
+#     {'id': 269, 'name': 'Induction Cooker'},
+#     {'id': 270, 'name': 'Broom'},
+#     {'id': 271, 'name': 'Trombone'},
+#     {'id': 272, 'name': 'Plum'},
+#     {'id': 273, 'name': 'Rickshaw'},
+#     {'id': 274, 'name': 'Goldfish'},
+#     {'id': 275, 'name': 'Kiwi fruit'},
+#     {'id': 276, 'name': 'Router/modem'},
+#     {'id': 277, 'name': 'Poker Card'},
+#     {'id': 278, 'name': 'Toaster'},
+#     {'id': 279, 'name': 'Shrimp'},
+#     {'id': 280, 'name': 'Sushi'},
+#     {'id': 281, 'name': 'Cheese'},
+#     {'id': 282, 'name': 'Notepaper'},
+#     {'id': 283, 'name': 'Cherry'},
+#     {'id': 284, 'name': 'Pliers'},
+#     {'id': 285, 'name': 'CD'},
+#     {'id': 286, 'name': 'Pasta'},
+#     {'id': 287, 'name': 'Hammer'},
+#     {'id': 288, 'name': 'Cue'},
+#     {'id': 289, 'name': 'Avocado'},
+#     {'id': 290, 'name': 'Hamimelon'},
+#     {'id': 291, 'name': 'Flask'},
+#     {'id': 292, 'name': 'Mushroon'},
+#     {'id': 293, 'name': 'Screwdriver'},
+#     {'id': 294, 'name': 'Soap'},
+#     {'id': 295, 'name': 'Recorder'},
+#     {'id': 296, 'name': 'Bear'},
+#     {'id': 297, 'name': 'Eggplant'},
+#     {'id': 298, 'name': 'Board Eraser'},
+#     {'id': 299, 'name': 'Coconut'},
+#     {'id': 300, 'name': 'Tape Measur/ Ruler'},
+#     {'id': 301, 'name': 'Pig'},
+#     {'id': 302, 'name': 'Showerhead'},
+#     {'id': 303, 'name': 'Globe'},
+#     {'id': 304, 'name': 'Chips'},
+#     {'id': 305, 'name': 'Steak'},
+#     {'id': 306, 'name': 'Crosswalk Sign'},
+#     {'id': 307, 'name': 'Stapler'},
+#     {'id': 308, 'name': 'Campel'},
+#     {'id': 309, 'name': 'Formula 1 '},
+#     {'id': 310, 'name': 'Pomegranate'},
+#     {'id': 311, 'name': 'Dishwasher'},
+#     {'id': 312, 'name': 'Crab'},
+#     {'id': 313, 'name': 'Hoverboard'},
+#     {'id': 314, 'name': 'Meat ball'},
+#     {'id': 315, 'name': 'Rice Cooker'},
+#     {'id': 316, 'name': 'Tuba'},
+#     {'id': 317, 'name': 'Calculator'},
+#     {'id': 318, 'name': 'Papaya'},
+#     {'id': 319, 'name': 'Antelope'},
+#     {'id': 320, 'name': 'Parrot'},
+#     {'id': 321, 'name': 'Seal'},
+#     {'id': 322, 'name': 'Buttefly'},
+#     {'id': 323, 'name': 'Dumbbell'},
+#     {'id': 324, 'name': 'Donkey'},
+#     {'id': 325, 'name': 'Lion'},
+#     {'id': 326, 'name': 'Urinal'},
+#     {'id': 327, 'name': 'Dolphin'},
+#     {'id': 328, 'name': 'Electric Drill'},
+#     {'id': 329, 'name': 'Hair Dryer'},
+#     {'id': 330, 'name': 'Egg tart'},
+#     {'id': 331, 'name': 'Jellyfish'},
+#     {'id': 332, 'name': 'Treadmill'},
+#     {'id': 333, 'name': 'Lighter'},
+#     {'id': 334, 'name': 'Grapefruit'},
+#     {'id': 335, 'name': 'Game board'},
+#     {'id': 336, 'name': 'Mop'},
+#     {'id': 337, 'name': 'Radish'},
+#     {'id': 338, 'name': 'Baozi'},
+#     {'id': 339, 'name': 'Target'},
+#     {'id': 340, 'name': 'French'},
+#     {'id': 341, 'name': 'Spring Rolls'},
+#     {'id': 342, 'name': 'Monkey'},
+#     {'id': 343, 'name': 'Rabbit'},
+#     {'id': 344, 'name': 'Pencil Case'},
+#     {'id': 345, 'name': 'Yak'},
+#     {'id': 346, 'name': 'Red Cabbage'},
+#     {'id': 347, 'name': 'Binoculars'},
+#     {'id': 348, 'name': 'Asparagus'},
+#     {'id': 349, 'name': 'Barbell'},
+#     {'id': 350, 'name': 'Scallop'},
+#     {'id': 351, 'name': 'Noddles'},
+#     {'id': 352, 'name': 'Comb'},
+#     {'id': 353, 'name': 'Dumpling'},
+#     {'id': 354, 'name': 'Oyster'},
+#     {'id': 355, 'name': 'Table Teniis paddle'},
+#     {'id': 356, 'name': 'Cosmetics Brush/Eyeliner Pencil'},
+#     {'id': 357, 'name': 'Chainsaw'},
+#     {'id': 358, 'name': 'Eraser'},
+#     {'id': 359, 'name': 'Lobster'},
+#     {'id': 360, 'name': 'Durian'},
+#     {'id': 361, 'name': 'Okra'},
+#     {'id': 362, 'name': 'Lipstick'},
+#     {'id': 363, 'name': 'Cosmetics Mirror'},
+#     {'id': 364, 'name': 'Curling'},
+#     {'id': 365, 'name': 'Table Tennis '},
+# ]
+
+'''
+The official Objects365 category names contains typos.
+Below is a manual fix.
+'''
+categories_v2_fix = [
+  {'id': 1, 'name': 'Person'},
+  {'id': 2, 'name': 'Sneakers'},
+  {'id': 3, 'name': 'Chair'},
+  {'id': 4, 'name': 'Other Shoes'},
+  {'id': 5, 'name': 'Hat'},
+  {'id': 6, 'name': 'Car'},
+  {'id': 7, 'name': 'Lamp'},
+  {'id': 8, 'name': 'Glasses'},
+  {'id': 9, 'name': 'Bottle'},
+  {'id': 10, 'name': 'Desk'},
+  {'id': 11, 'name': 'Cup'},
+  {'id': 12, 'name': 'Street Lights'},
+  {'id': 13, 'name': 'Cabinet/shelf'},
+  {'id': 14, 'name': 'Handbag/Satchel'},
+  {'id': 15, 'name': 'Bracelet'},
+  {'id': 16, 'name': 'Plate'},
+  {'id': 17, 'name': 'Picture/Frame'},
+  {'id': 18, 'name': 'Helmet'},
+  {'id': 19, 'name': 'Book'},
+  {'id': 20, 'name': 'Gloves'},
+  {'id': 21, 'name': 'Storage box'},
+  {'id': 22, 'name': 'Boat'},
+  {'id': 23, 'name': 'Leather Shoes'},
+  {'id': 24, 'name': 'Flower'},
+  {'id': 25, 'name': 'Bench'},
+  {'id': 26, 'name': 'Potted Plant'},
+  {'id': 27, 'name': 'Bowl/Basin'},
+  {'id': 28, 'name': 'Flag'},
+  {'id': 29, 'name': 'Pillow'},
+  {'id': 30, 'name': 'Boots'},
+  {'id': 31, 'name': 'Vase'},
+  {'id': 32, 'name': 'Microphone'},
+  {'id': 33, 'name': 'Necklace'},
+  {'id': 34, 'name': 'Ring'},
+  {'id': 35, 'name': 'SUV'},
+  {'id': 36, 'name': 'Wine Glass'},
+  {'id': 37, 'name': 'Belt'},
+  {'id': 38, 'name': 'Monitor/TV'},
+  {'id': 39, 'name': 'Backpack'},
+  {'id': 40, 'name': 'Umbrella'},
+  {'id': 41, 'name': 'Traffic Light'},
+  {'id': 42, 'name': 'Speaker'},
+  {'id': 43, 'name': 'Watch'},
+  {'id': 44, 'name': 'Tie'},
+  {'id': 45, 'name': 'Trash bin Can'},
+  {'id': 46, 'name': 'Slippers'},
+  {'id': 47, 'name': 'Bicycle'},
+  {'id': 48, 'name': 'Stool'},
+  {'id': 49, 'name': 'Barrel/bucket'},
+  {'id': 50, 'name': 'Van'},
+  {'id': 51, 'name': 'Couch'},
+  {'id': 52, 'name': 'Sandals'},
+  {'id': 53, 'name': 'Basket'},
+  {'id': 54, 'name': 'Drum'},
+  {'id': 55, 'name': 'Pen/Pencil'},
+  {'id': 56, 'name': 'Bus'},
+  {'id': 57, 'name': 'Wild Bird'},
+  {'id': 58, 'name': 'High Heels'},
+  {'id': 59, 'name': 'Motorcycle'},
+  {'id': 60, 'name': 'Guitar'},
+  {'id': 61, 'name': 'Carpet'},
+  {'id': 62, 'name': 'Cell Phone'},
+  {'id': 63, 'name': 'Bread'},
+  {'id': 64, 'name': 'Camera'},
+  {'id': 65, 'name': 'Canned'},
+  {'id': 66, 'name': 'Truck'},
+  {'id': 67, 'name': 'Traffic cone'},
+  {'id': 68, 'name': 'Cymbal'},
+  {'id': 69, 'name': 'Lifesaver'},
+  {'id': 70, 'name': 'Towel'},
+  {'id': 71, 'name': 'Stuffed Toy'},
+  {'id': 72, 'name': 'Candle'},
+  {'id': 73, 'name': 'Sailboat'},
+  {'id': 74, 'name': 'Laptop'},
+  {'id': 75, 'name': 'Awning'},
+  {'id': 76, 'name': 'Bed'},
+  {'id': 77, 'name': 'Faucet'},
+  {'id': 78, 'name': 'Tent'},
+  {'id': 79, 'name': 'Horse'},
+  {'id': 80, 'name': 'Mirror'},
+  {'id': 81, 'name': 'Power outlet'},
+  {'id': 82, 'name': 'Sink'},
+  {'id': 83, 'name': 'Apple'},
+  {'id': 84, 'name': 'Air Conditioner'},
+  {'id': 85, 'name': 'Knife'},
+  {'id': 86, 'name': 'Hockey Stick'},
+  {'id': 87, 'name': 'Paddle'},
+  {'id': 88, 'name': 'Pickup Truck'},
+  {'id': 89, 'name': 'Fork'},
+  {'id': 90, 'name': 'Traffic Sign'},
+  {'id': 91, 'name': 'Ballon'},
+  {'id': 92, 'name': 'Tripod'},
+  {'id': 93, 'name': 'Dog'},
+  {'id': 94, 'name': 'Spoon'},
+  {'id': 95, 'name': 'Clock'},
+  {'id': 96, 'name': 'Pot'},
+  {'id': 97, 'name': 'Cow'},
+  {'id': 98, 'name': 'Cake'},
+  {'id': 99, 'name': 'Dining Table'},
+  {'id': 100, 'name': 'Sheep'},
+  {'id': 101, 'name': 'Hanger'},
+  {'id': 102, 'name': 'Blackboard/Whiteboard'},
+  {'id': 103, 'name': 'Napkin'},
+  {'id': 104, 'name': 'Other Fish'},
+  {'id': 105, 'name': 'Orange/Tangerine'},
+  {'id': 106, 'name': 'Toiletry'},
+  {'id': 107, 'name': 'Keyboard'},
+  {'id': 108, 'name': 'Tomato'},
+  {'id': 109, 'name': 'Lantern'},
+  {'id': 110, 'name': 'Machinery Vehicle'},
+  {'id': 111, 'name': 'Fan'},
+  {'id': 112, 'name': 'Green Vegetables'},
+  {'id': 113, 'name': 'Banana'},
+  {'id': 114, 'name': 'Baseball Glove'},
+  {'id': 115, 'name': 'Airplane'},
+  {'id': 116, 'name': 'Mouse'},
+  {'id': 117, 'name': 'Train'},
+  {'id': 118, 'name': 'Pumpkin'},
+  {'id': 119, 'name': 'Soccer'},
+  {'id': 120, 'name': 'Skiboard'},
+  {'id': 121, 'name': 'Luggage'},
+  {'id': 122, 'name': 'Nightstand'},
+  {'id': 123, 'name': 'Teapot'},
+  {'id': 124, 'name': 'Telephone'},
+  {'id': 125, 'name': 'Trolley'},
+  {'id': 126, 'name': 'Head Phone'},
+  {'id': 127, 'name': 'Sports Car'},
+  {'id': 128, 'name': 'Stop Sign'},
+  {'id': 129, 'name': 'Dessert'},
+  {'id': 130, 'name': 'Scooter'},
+  {'id': 131, 'name': 'Stroller'},
+  {'id': 132, 'name': 'Crane'},
+  {'id': 133, 'name': 'Remote'},
+  {'id': 134, 'name': 'Refrigerator'},
+  {'id': 135, 'name': 'Oven'},
+  {'id': 136, 'name': 'Lemon'},
+  {'id': 137, 'name': 'Duck'},
+  {'id': 138, 'name': 'Baseball Bat'},
+  {'id': 139, 'name': 'Surveillance Camera'},
+  {'id': 140, 'name': 'Cat'},
+  {'id': 141, 'name': 'Jug'},
+  {'id': 142, 'name': 'Broccoli'},
+  {'id': 143, 'name': 'Piano'},
+  {'id': 144, 'name': 'Pizza'},
+  {'id': 145, 'name': 'Elephant'},
+  {'id': 146, 'name': 'Skateboard'},
+  {'id': 147, 'name': 'Surfboard'},
+  {'id': 148, 'name': 'Gun'},
+  {'id': 149, 'name': 'Skating and Skiing shoes'},
+  {'id': 150, 'name': 'Gas stove'},
+  {'id': 151, 'name': 'Donut'},
+  {'id': 152, 'name': 'Bow Tie'},
+  {'id': 153, 'name': 'Carrot'},
+  {'id': 154, 'name': 'Toilet'},
+  {'id': 155, 'name': 'Kite'},
+  {'id': 156, 'name': 'Strawberry'},
+  {'id': 157, 'name': 'Other Balls'},
+  {'id': 158, 'name': 'Shovel'},
+  {'id': 159, 'name': 'Pepper'},
+  {'id': 160, 'name': 'Computer Box'},
+  {'id': 161, 'name': 'Toilet Paper'},
+  {'id': 162, 'name': 'Cleaning Products'},
+  {'id': 163, 'name': 'Chopsticks'},
+  {'id': 164, 'name': 'Microwave'},
+  {'id': 165, 'name': 'Pigeon'},
+  {'id': 166, 'name': 'Baseball'},
+  {'id': 167, 'name': 'Cutting/chopping Board'},
+  {'id': 168, 'name': 'Coffee Table'},
+  {'id': 169, 'name': 'Side Table'},
+  {'id': 170, 'name': 'Scissors'},
+  {'id': 171, 'name': 'Marker'},
+  {'id': 172, 'name': 'Pie'},
+  {'id': 173, 'name': 'Ladder'},
+  {'id': 174, 'name': 'Snowboard'},
+  {'id': 175, 'name': 'Cookies'},
+  {'id': 176, 'name': 'Radiator'},
+  {'id': 177, 'name': 'Fire Hydrant'},
+  {'id': 178, 'name': 'Basketball'},
+  {'id': 179, 'name': 'Zebra'},
+  {'id': 180, 'name': 'Grape'},
+  {'id': 181, 'name': 'Giraffe'},
+  {'id': 182, 'name': 'Potato'},
+  {'id': 183, 'name': 'Sausage'},
+  {'id': 184, 'name': 'Tricycle'},
+  {'id': 185, 'name': 'Violin'},
+  {'id': 186, 'name': 'Egg'},
+  {'id': 187, 'name': 'Fire Extinguisher'},
+  {'id': 188, 'name': 'Candy'},
+  {'id': 189, 'name': 'Fire Truck'},
+  {'id': 190, 'name': 'Billards'},
+  {'id': 191, 'name': 'Converter'},
+  {'id': 192, 'name': 'Bathtub'},
+  {'id': 193, 'name': 'Wheelchair'},
+  {'id': 194, 'name': 'Golf Club'},
+  {'id': 195, 'name': 'Briefcase'},
+  {'id': 196, 'name': 'Cucumber'},
+  {'id': 197, 'name': 'Cigar/Cigarette '},
+  {'id': 198, 'name': 'Paint Brush'},
+  {'id': 199, 'name': 'Pear'},
+  {'id': 200, 'name': 'Heavy Truck'},
+  {'id': 201, 'name': 'Hamburger'},
+  {'id': 202, 'name': 'Extractor'},
+  {'id': 203, 'name': 'Extension Cord'},
+  {'id': 204, 'name': 'Tong'},
+  {'id': 205, 'name': 'Tennis Racket'},
+  {'id': 206, 'name': 'Folder'},
+  {'id': 207, 'name': 'American Football'},
+  {'id': 208, 'name': 'earphone'},
+  {'id': 209, 'name': 'Mask'},
+  {'id': 210, 'name': 'Kettle'},
+  {'id': 211, 'name': 'Tennis'},
+  {'id': 212, 'name': 'Ship'},
+  {'id': 213, 'name': 'Swing'},
+  {'id': 214, 'name': 'Coffee Machine'},
+  {'id': 215, 'name': 'Slide'},
+  {'id': 216, 'name': 'Carriage'},
+  {'id': 217, 'name': 'Onion'},
+  {'id': 218, 'name': 'Green beans'},
+  {'id': 219, 'name': 'Projector'},
+  {'id': 220, 'name': 'Frisbee'},
+  {'id': 221, 'name': 'Washing Machine/Drying Machine'},
+  {'id': 222, 'name': 'Chicken'},
+  {'id': 223, 'name': 'Printer'},
+  {'id': 224, 'name': 'Watermelon'},
+  {'id': 225, 'name': 'Saxophone'},
+  {'id': 226, 'name': 'Tissue'},
+  {'id': 227, 'name': 'Toothbrush'},
+  {'id': 228, 'name': 'Ice cream'},
+  {'id': 229, 'name': 'Hot air balloon'},
+  {'id': 230, 'name': 'Cello'},
+  {'id': 231, 'name': 'French Fries'},
+  {'id': 232, 'name': 'Scale'},
+  {'id': 233, 'name': 'Trophy'},
+  {'id': 234, 'name': 'Cabbage'},
+  {'id': 235, 'name': 'Hot dog'},
+  {'id': 236, 'name': 'Blender'},
+  {'id': 237, 'name': 'Peach'},
+  {'id': 238, 'name': 'Rice'},
+  {'id': 239, 'name': 'Wallet/Purse'},
+  {'id': 240, 'name': 'Volleyball'},
+  {'id': 241, 'name': 'Deer'},
+  {'id': 242, 'name': 'Goose'},
+  {'id': 243, 'name': 'Tape'},
+  {'id': 244, 'name': 'Tablet'},
+  {'id': 245, 'name': 'Cosmetics'},
+  {'id': 246, 'name': 'Trumpet'},
+  {'id': 247, 'name': 'Pineapple'},
+  {'id': 248, 'name': 'Golf Ball'},
+  {'id': 249, 'name': 'Ambulance'},
+  {'id': 250, 'name': 'Parking meter'},
+  {'id': 251, 'name': 'Mango'},
+  {'id': 252, 'name': 'Key'},
+  {'id': 253, 'name': 'Hurdle'},
+  {'id': 254, 'name': 'Fishing Rod'},
+  {'id': 255, 'name': 'Medal'},
+  {'id': 256, 'name': 'Flute'},
+  {'id': 257, 'name': 'Brush'},
+  {'id': 258, 'name': 'Penguin'},
+  {'id': 259, 'name': 'Megaphone'},
+  {'id': 260, 'name': 'Corn'},
+  {'id': 261, 'name': 'Lettuce'},
+  {'id': 262, 'name': 'Garlic'},
+  {'id': 263, 'name': 'Swan'},
+  {'id': 264, 'name': 'Helicopter'},
+  {'id': 265, 'name': 'Green Onion'},
+  {'id': 266, 'name': 'Sandwich'},
+  {'id': 267, 'name': 'Nuts'},
+  {'id': 268, 'name': 'Speed Limit Sign'},
+  {'id': 269, 'name': 'Induction Cooker'},
+  {'id': 270, 'name': 'Broom'},
+  {'id': 271, 'name': 'Trombone'},
+  {'id': 272, 'name': 'Plum'},
+  {'id': 273, 'name': 'Rickshaw'},
+  {'id': 274, 'name': 'Goldfish'},
+  {'id': 275, 'name': 'Kiwi fruit'},
+  {'id': 276, 'name': 'Router/modem'},
+  {'id': 277, 'name': 'Poker Card'},
+  {'id': 278, 'name': 'Toaster'},
+  {'id': 279, 'name': 'Shrimp'},
+  {'id': 280, 'name': 'Sushi'},
+  {'id': 281, 'name': 'Cheese'},
+  {'id': 282, 'name': 'Notepaper'},
+  {'id': 283, 'name': 'Cherry'},
+  {'id': 284, 'name': 'Pliers'},
+  {'id': 285, 'name': 'CD'},
+  {'id': 286, 'name': 'Pasta'},
+  {'id': 287, 'name': 'Hammer'},
+  {'id': 288, 'name': 'Cue'},
+  {'id': 289, 'name': 'Avocado'},
+  {'id': 290, 'name': 'Hami melon'},
+  {'id': 291, 'name': 'Flask'},
+  {'id': 292, 'name': 'Mushroom'},
+  {'id': 293, 'name': 'Screwdriver'},
+  {'id': 294, 'name': 'Soap'},
+  {'id': 295, 'name': 'Recorder'},
+  {'id': 296, 'name': 'Bear'},
+  {'id': 297, 'name': 'Eggplant'},
+  {'id': 298, 'name': 'Board Eraser'},
+  {'id': 299, 'name': 'Coconut'},
+  {'id': 300, 'name': 'Tape Measure/ Ruler'},
+  {'id': 301, 'name': 'Pig'},
+  {'id': 302, 'name': 'Showerhead'},
+  {'id': 303, 'name': 'Globe'},
+  {'id': 304, 'name': 'Chips'},
+  {'id': 305, 'name': 'Steak'},
+  {'id': 306, 'name': 'Crosswalk Sign'},
+  {'id': 307, 'name': 'Stapler'},
+  {'id': 308, 'name': 'Camel'},
+  {'id': 309, 'name': 'Formula 1 '},
+  {'id': 310, 'name': 'Pomegranate'},
+  {'id': 311, 'name': 'Dishwasher'},
+  {'id': 312, 'name': 'Crab'},
+  {'id': 313, 'name': 'Hoverboard'},
+  {'id': 314, 'name': 'Meatball'},
+  {'id': 315, 'name': 'Rice Cooker'},
+  {'id': 316, 'name': 'Tuba'},
+  {'id': 317, 'name': 'Calculator'},
+  {'id': 318, 'name': 'Papaya'},
+  {'id': 319, 'name': 'Antelope'},
+  {'id': 320, 'name': 'Parrot'},
+  {'id': 321, 'name': 'Seal'},
+  {'id': 322, 'name': 'Butterfly'},
+  {'id': 323, 'name': 'Dumbbell'},
+  {'id': 324, 'name': 'Donkey'},
+  {'id': 325, 'name': 'Lion'},
+  {'id': 326, 'name': 'Urinal'},
+  {'id': 327, 'name': 'Dolphin'},
+  {'id': 328, 'name': 'Electric Drill'},
+  {'id': 329, 'name': 'Hair Dryer'},
+  {'id': 330, 'name': 'Egg tart'},
+  {'id': 331, 'name': 'Jellyfish'},
+  {'id': 332, 'name': 'Treadmill'},
+  {'id': 333, 'name': 'Lighter'},
+  {'id': 334, 'name': 'Grapefruit'},
+  {'id': 335, 'name': 'Game board'},
+  {'id': 336, 'name': 'Mop'},
+  {'id': 337, 'name': 'Radish'},
+  {'id': 338, 'name': 'Baozi'},
+  {'id': 339, 'name': 'Target'},
+  {'id': 340, 'name': 'French'},
+  {'id': 341, 'name': 'Spring Rolls'},
+  {'id': 342, 'name': 'Monkey'},
+  {'id': 343, 'name': 'Rabbit'},
+  {'id': 344, 'name': 'Pencil Case'},
+  {'id': 345, 'name': 'Yak'},
+  {'id': 346, 'name': 'Red Cabbage'},
+  {'id': 347, 'name': 'Binoculars'},
+  {'id': 348, 'name': 'Asparagus'},
+  {'id': 349, 'name': 'Barbell'},
+  {'id': 350, 'name': 'Scallop'},
+  {'id': 351, 'name': 'Noddles'},
+  {'id': 352, 'name': 'Comb'},
+  {'id': 353, 'name': 'Dumpling'},
+  {'id': 354, 'name': 'Oyster'},
+  {'id': 355, 'name': 'Table Tennis paddle'},
+  {'id': 356, 'name': 'Cosmetics Brush/Eyeliner Pencil'},
+  {'id': 357, 'name': 'Chainsaw'},
+  {'id': 358, 'name': 'Eraser'},
+  {'id': 359, 'name': 'Lobster'},
+  {'id': 360, 'name': 'Durian'},
+  {'id': 361, 'name': 'Okra'},
+  {'id': 362, 'name': 'Lipstick'},
+  {'id': 363, 'name': 'Cosmetics Mirror'},
+  {'id': 364, 'name': 'Curling'},
+  {'id': 365, 'name': 'Table Tennis '},
+]
+
+
+def _get_builtin_metadata():
+    id_to_name = {x['id']: x['name'] for x in categories_v2_fix}
+    thing_dataset_id_to_contiguous_id = {
+        x['id']: i for i, x in enumerate(
+            sorted(categories_v2_fix, key=lambda x: x['id']))}
+    thing_classes = [id_to_name[k] for k in sorted(id_to_name)]
+    return {
+        "thing_dataset_id_to_contiguous_id": thing_dataset_id_to_contiguous_id,
+        "thing_classes": thing_classes}
+
+
+_PREDEFINED_SPLITS_OBJECTS365 = {
+    "objects365_v2_train": ("objects365/train", "objects365/annotations/zhiyuan_objv2_train_fixname_fixmiss.json"),
+    # 80,000 images, 1,240,587 annotations
+    "objects365_v2_val": ("objects365/val", "objects365/annotations/zhiyuan_objv2_val_fixname.json"),
+    "objects365_v2_val_rare": ("objects365/val", "objects365/annotations/zhiyuan_objv2_val_fixname_rare.json"),
+}
+
+for key, (image_root, json_file) in _PREDEFINED_SPLITS_OBJECTS365.items():
+    register_coco_instances(
+        key,
+        _get_builtin_metadata(),
+        os.path.join("datasets", json_file) if "://" not in json_file else json_file,
+        os.path.join("datasets", image_root),
+    )