| # Prepare datasets for Detic | |
| The basic training of our model uses [LVIS](https://www.lvisdataset.org/) (which uses [COCO](https://cocodataset.org/) images) and [ImageNet-21K](https://www.image-net.org/download.php). | |
| Some models are trained on [Conceptual Caption (CC3M)](https://ai.google.com/research/ConceptualCaptions/). | |
| Optionally, we use [Objects365](https://www.objects365.org/) and [OpenImages (Challenge 2019 version)](https://storage.googleapis.com/openimages/web/challenge2019.html) for cross-dataset evaluation. | |
| Before starting processing, please download the (selected) datasets from the official websites and place or sim-link them under `$Detic_ROOT/datasets/`. | |
| ``` | |
| $Detic_ROOT/datasets/ | |
| metadata/ | |
| lvis/ | |
| coco/ | |
| imagenet/ | |
| cc3m/ | |
| objects365/ | |
| oid/ | |
| ``` | |
| `metadata/` is our preprocessed meta-data (included in the repo). See the below [section](#Metadata) for details. | |
| Please follow the following instruction to pre-process individual datasets. | |
| ### COCO and LVIS | |
| First, download COCO and LVIS data place them in the following way: | |
| ``` | |
| lvis/ | |
| lvis_v1_train.json | |
| lvis_v1_val.json | |
| coco/ | |
| train2017/ | |
| val2017/ | |
| annotations/ | |
| captions_train2017.json | |
| instances_train2017.json | |
| instances_val2017.json | |
| ``` | |
| Next, prepare the open-vocabulary LVIS training set using | |
| ``` | |
| python tools/remove_lvis_rare.py --ann datasets/lvis/lvis_v1_train.json | |
| ``` | |
| This will generate `datasets/lvis/lvis_v1_train_norare.json`. | |
| ### ImageNet-21K | |
| The ImageNet-21K folder should look like: | |
| ``` | |
| imagenet/ | |
| ImageNet-21K/ | |
| n01593028.tar | |
| n01593282.tar | |
| ... | |
| ``` | |
| We first unzip the overlapping classes of LVIS (we will directly work with the .tar file for the rest classes) and convert them into LVIS annotation format. | |
| ~~~ | |
| mkdir imagenet/annotations | |
| python tools/unzip_imagenet_lvis.py --dst_path datasets/imagenet/ImageNet-LVIS | |
| python tools/create_imagenetlvis_json.py --imagenet_path datasets/imagenet/ImageNet-LVIS --out_path datasets/imagenet/annotations/imagenet_lvis_image_info.json | |
| ~~~ | |
| This creates `datasets/imagenet/annotations/imagenet_lvis_image_info.json`. | |
| [Optional] To train with all the 21K classes, run | |
| ~~~ | |
| python tools/get_imagenet_21k_full_tar_json.py | |
| python tools/create_lvis_21k.py | |
| ~~~ | |
| This creates `datasets/imagenet/annotations/imagenet-21k_image_info_lvis-21k.json` and `datasets/lvis/lvis_v1_train_lvis-21k.json` (combined LVIS and ImageNet-21K classes in `categories`). | |
| [Optional] To train on combined LVIS and COCO, run | |
| ~~~ | |
| python tools/merge_lvis_coco.py | |
| ~~~ | |
| This creates `datasets/lvis/lvis_v1_train+coco_mask.json` | |
| ### Conceptual Caption | |
| Download the dataset from [this](https://ai.google.com/research/ConceptualCaptions/download) page and place them as: | |
| ``` | |
| cc3m/ | |
| GCC-training.tsv | |
| ``` | |
| Run the following command to download the images and convert the annotations to LVIS format (Note: download images takes long). | |
| ~~~ | |
| python tools/download_cc.py --ann datasets/cc3m/GCC-training.tsv --save_image_path datasets/cc3m/training/ --out_path datasets/cc3m/train_image_info.json | |
| python tools/get_cc_tags.py | |
| ~~~ | |
| This creates `datasets/cc3m/train_image_info_tags.json`. | |
| ### Objects365 | |
| Download Objects365 (v2) from the website. We only need the validation set in this project: | |
| ``` | |
| objects365/ | |
| annotations/ | |
| zhiyuan_objv2_val.json | |
| val/ | |
| images/ | |
| v1/ | |
| patch0/ | |
| ... | |
| patch15/ | |
| v2/ | |
| patch16/ | |
| ... | |
| patch49/ | |
| ``` | |
| The original annotation has typos in the class names, we first fix them for our following use of language embeddings. | |
| ``` | |
| python tools/fix_o365_names.py --ann datasets/objects365/annotations/zhiyuan_objv2_val.json | |
| ``` | |
| This creates `datasets/objects365/zhiyuan_objv2_val_fixname.json`. | |
| To train on Objects365, download the training images and use the command above. We note some images in the training annotation do not exist. | |
| We use the following command to filter the missing images. | |
| ~~~ | |
| python tools/fix_0365_path.py | |
| ~~~ | |
| This creates `datasets/objects365/zhiyuan_objv2_train_fixname_fixmiss.json`. | |
| ### OpenImages | |
| We followed the instructions in [UniDet](https://github.com/xingyizhou/UniDet/blob/master/projects/UniDet/unidet_docs/DATASETS.md#openimages) to convert the metadata for OpenImages. | |
| The converted folder should look like | |
| ``` | |
| oid/ | |
| annotations/ | |
| oid_challenge_2019_train_bbox.json | |
| oid_challenge_2019_val_expanded.json | |
| images/ | |
| 0/ | |
| 1/ | |
| 2/ | |
| ... | |
| ``` | |
| ### Open-vocabulary COCO | |
| We first follow [OVR-CNN](https://github.com/alirezazareian/ovr-cnn/blob/master/ipynb/003.ipynb) to create the open-vocabulary COCO split. The converted files should be like | |
| ``` | |
| coco/ | |
| zero-shot/ | |
| instances_train2017_seen_2.json | |
| instances_val2017_all_2.json | |
| ``` | |
| We further pre-process the annotation format for easier evaluation: | |
| ``` | |
| python tools/get_coco_zeroshot_oriorder.py --data_path datasets/coco/zero-shot/instances_train2017_seen_2.json | |
| python tools/get_coco_zeroshot_oriorder.py --data_path datasets/coco/zero-shot/instances_val2017_all_2.json | |
| ``` | |
| Next, we preprocess the COCO caption data: | |
| ``` | |
| python tools/get_cc_tags.py --cc_ann datasets/coco/annotations/captions_train2017.json --out_path datasets/coco/captions_train2017_tags_allcaps.json --allcaps --convert_caption | |
| ``` | |
| This creates `datasets/coco/captions_train2017_tags_allcaps.json`. | |
| ### Metadata | |
| ``` | |
| metadata/ | |
| lvis_v1_train_cat_info.json | |
| coco_clip_a+cname.npy | |
| lvis_v1_clip_a+cname.npy | |
| o365_clip_a+cnamefix.npy | |
| oid_clip_a+cname.npy | |
| imagenet_lvis_wnid.txt | |
| Objects365_names_fix.csv | |
| ``` | |
| `lvis_v1_train_cat_info.json` is used by the Federated loss. | |
| This is created by | |
| ~~~ | |
| python tools/get_lvis_cat_info.py --ann datasets/lvis/lvis_v1_train.json | |
| ~~~ | |
| `*_clip_a+cname.npy` is the pre-computed CLIP embeddings for each datasets. | |
| They are created by (taking LVIS as an example) | |
| ~~~ | |
| python tools/dump_clip_features.py --ann datasets/lvis/lvis_v1_val.json --out_path metadata/lvis_v1_clip_a+cname.npy | |
| ~~~ | |
| Note we do not include the 21K class embeddings due to the large file size. | |
| To create it, run | |
| ~~~ | |
| python tools/dump_clip_features.py --ann datasets/lvis/lvis_v1_val_lvis-21k.json --out_path datasets/metadata/lvis-21k_clip_a+cname.npy | |
| ~~~ | |
| `imagenet_lvis_wnid.txt` is the list of matched classes between ImageNet-21K and LVIS. | |
| `Objects365_names_fix.csv` is our manual fix of the Objects365 names. |