added datasets and virtex

.gitignore

@@ -0,0 +1 @@
+.ipynb_checkpoints/*

virtex/CHANGELOG.md

@@ -0,0 +1,41 @@
+ArXiv v1 -> v2 CHANGELOG
+=========================
+
+[ArXiv v1](https://arxiv.org/abs/2006.06666v1) was our ECCV 2020 submission (reject). [ArXiv v2](https://arxiv.org/abs/2006.06666v2) is out CVPR 2021 submission (accept). The repository snapshots for these two versions are tagged at [`v0.9`](https://github.com/kdexd/virtex/releases/tag/v0.9) and [`v1.0`](https://github.com/kdexd/virtex/releases/tag/v1.0).
+
+While the core motivation and approach is the same, we have made some minor changes in our experiments and evaluation setup. These slightly improve model performances across the board (within decimals). New models are available in [`v1.0` model zoo](http://kdexd.github.io/virtex/virtex/usage/model_zoo.html), however links to old models in `v0.9` will be active till June 30, 2021. We encourage you to use the new models!
+
+We have updated the experiment config files for all changes described below.
+
+Experiment Changes
+------------------
+
+### New Feature:
+
+Add a new pretraining task for BERT-style _Masked Language Modeling_. Pre-trained model released in Model Zoo.
+
+### Pre-training:
+
+- The only change during pre-training is that we do not apply weight decay to LayerNorm and biases in input embedding and transformer layers. We apply weight decay to the biases in output linear layer (before softmax).
+
+- Other factors that could affect results:
+  - Use official [albumentations.ColorJitter transform](https://albumentations.ai/docs/api_reference/augmentations/transforms/#albumentations.augmentations.transforms.ColorJitter) that mimics torchvision ColorJitter transform. Earlier I implemented [my own ColorJitter](https://github.com/kdexd/virtex/blob/c19e7fc9b98e98af82286ed1537b6f588eaeac44/virtex/data/transforms.py#L156) because albumentations didn't have one.
+  - Use PyTorch Native AMP (Automatic Mixed Precision) instead of NVIDIA Apex.
+
+### Downstream Evaluations:
+
+1. **PASCAL VOC 2007 Linear Classification:** [[diff]](https://github.com/kdexd/virtex/compare/57889ca9829f27b932e92b9e6b51f50f20f2d546..7645cc0d1e3e49f00e347e9873fd020faa2ec62e#diff-b4405dd4879a48ef1e5b1e2801035909584a5f1f32f63d5e793fb50dee077b97)
+   - Instead of training linear SVMs on 8192-dimensional average pooled features from ResNet-50 (7x7x2048 —> 2x2x2048), like [(Misra et al. 2019)](https://arxiv.org/abs/1905.01235), we directly train SVMs on 2048-dimensional global average pooled features, following recent works like [SwAV (Caron et al. 2020)](https://arxiv.org/abs/2006.09882).
+   - We change the pre-processing: resize shortest edge to 256 pixels, and take center crop of 224 pixels.
+   - These improve VOC mAP by 1-2 points everywhere, and makes SVM training faster. Since we select best checkpoint based on this metric, all results on other downstream tasks also change in `ArXiv v2` (But the trends remain same.)
+
+2. **ImageNet Linear Evaluation:** [[diff]](https://github.com/kdexd/virtex/compare/57889ca9829f27b932e92b9e6b51f50f20f2d546..7645cc0d1e3e49f00e347e9873fd020faa2ec62e#diff-d3dea1e7bf97d0cfca4b59a47c0a9bb81e78b8827654fe0258df9ce2c3f5f41c)
+   - Changed random resized crop scale from (20-100%) to (8-100%) for consistency with evaluations in SSL works like MoCo and SwAV.
+   - Use cosine LR decay instead of step decay, following SwAV. Improves accuracy by up to 1%.
+
+3. **iNaturalist Fine-tuning:** [[diff]](https://github.com/kdexd/virtex/compare/57889ca9829f27b932e92b9e6b51f50f20f2d546..7645cc0d1e3e49f00e347e9873fd020faa2ec62e#diff-09096da78cfcde3a604ce22d80313f0800225d928cce5ef7334b89a382adfe4d)
+   - This evaluation is left unchanged across ArXiv versions, but we fixd a typo in image pre-processing step, present in publicly released config.
+
+4. **Detectron2 tasks (COCO and LVIS Instance Segmentation, VOC Detection):**
+   - Heavily simplified the script. Updated Detectron2 uses a more memory-efficient SyncBatchNorm and supports AMP.
+

virtex/LICENSE

@@ -0,0 +1,16 @@
+Copyright (c) 2020, Karan Desai.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+associated documentation files (the "Software"), to deal in the Software without restriction,
+including without limitation the rights to use, copy, modify, merge, publish, distribute,
+sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial
+portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
+NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES
+OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

virtex/README.md

@@ -0,0 +1,92 @@
+VirTex: Learning Visual Representations from Textual Annotations
+================================================================
+
+<h4>
+Karan Desai and Justin Johnson
+</br>
+<span style="font-size: 14pt; color: #555555">
+University of Michigan
+</span>
+</h4>
+<hr>
+
+**CVPR 2021** [arxiv.org/abs/2006.06666][1]
+
+**Model Zoo, Usage Instructions and API docs:** [kdexd.github.io/virtex](https://kdexd.github.io/virtex)
+
+VirTex is a pretraining approach which uses semantically dense captions to
+learn visual representations. We train CNN + Transformers from scratch on
+COCO Captions, and transfer the CNN to downstream vision tasks including
+image classification, object detection, and instance segmentation.
+VirTex matches or outperforms models which use ImageNet for pretraining -- 
+both supervised or unsupervised -- despite using up to 10x fewer images.
+
+![virtex-model](docs/_static/system_figure.jpg)
+
+
+Get the pretrained ResNet-50 visual backbone from our best performing VirTex
+model in one line *without any installation*!
+
+```python
+import torch
+
+# That's it, this one line only requires PyTorch.
+model = torch.hub.load("kdexd/virtex", "resnet50", pretrained=True)
+```
+
+### Note (For returning users before January 2021):
+
+The pretrained models in our model zoo have changed from [`v1.0`](https://github.com/kdexd/virtex/releases/tag/v1.0) onwards.
+They are slightly better tuned than older models, and reproduce the results in our
+CVPR 2021 accepted paper ([arXiv v2](https://arxiv.org/abs/2006.06666v2)). 
+Some training and evaluation hyperparams are changed since [`v0.9`](https://github.com/kdexd/virtex/releases/tag/v0.9).
+Please refer [`CHANGELOG.md`](https://github.com/kdexd/virtex/blob/master/CHANGELOG.md)
+
+
+Usage Instructions
+------------------
+
+1. [How to setup this codebase?][2]  
+2. [VirTex Model Zoo][3]  
+3. [How to train your VirTex model?][4]  
+4. [How to evaluate on downstream tasks?][5]  
+
+Full documentation is available at [kdexd.github.io/virtex](https://kdexd.github.io/virtex).
+
+
+Citation
+--------
+
+If you find this code useful, please consider citing:
+
+```text
+@inproceedings{desai2021virtex,
+    title={{VirTex: Learning Visual Representations from Textual Annotations}},
+    author={Karan Desai and Justin Johnson},
+    booktitle={CVPR},
+    year={2021}
+}
+```
+
+Acknowledgments
+---------------
+
+We thank Harsh Agrawal, Mohamed El Banani, Richard  Higgins, Nilesh Kulkarni
+and Chris Rockwell for helpful discussions and feedback on the paper. We thank
+Ishan Misra for discussions regarding PIRL evaluation protocol; Saining Xie for
+discussions about replicating iNaturalist evaluation as MoCo; Ross Girshick and
+Yuxin Wu for help with Detectron2 model zoo; Georgia Gkioxari for suggesting
+the Instance Segmentation pretraining task ablation; and Stefan Lee for
+suggestions on figure aesthetics. We thank Jia Deng for access to extra GPUs
+during project development; and UMich ARC-TS team for support with GPU cluster
+management. Finally, we thank all the Starbucks outlets in Ann Arbor for many
+hours of free WiFi. This work was partially supported by the Toyota Research
+Institute (TRI). However, note that this article solely reflects the opinions
+and conclusions of its authors and not TRI or any other Toyota entity.
+
+
+[1]: https://arxiv.org/abs/2006.06666
+[2]: https://kdexd.github.io/virtex/virtex/usage/setup_dependencies.html
+[3]: https://kdexd.github.io/virtex/virtex/usage/model_zoo.html
+[4]: https://kdexd.github.io/virtex/virtex/usage/pretrain.html
+[5]: https://kdexd.github.io/virtex/virtex/usage/downstream.html

virtex/configs/_base_bicaptioning_R_50_L1_H1024.yaml

@@ -0,0 +1,66 @@
+# -----------------------------------------------------------------------------
+# Base config: VirTex pretraining for our "base" bicaptioning model:
+# ResNet-50 + (L = 1, H = 1024) transformer trained for 500K iterations.
+# -----------------------------------------------------------------------------
+RANDOM_SEED: 0
+AMP: true
+CUDNN_BENCHMARK: true
+CUDNN_DETERMINISTIC: false
+
+DATA:
+  ROOT: "datasets/coco"
+  TOKENIZER_MODEL: "datasets/vocab/coco_10k.model"
+  VOCAB_SIZE: 10000
+  UNK_INDEX: 0
+  SOS_INDEX: 1
+  EOS_INDEX: 2
+  MASK_INDEX: 3
+
+  IMAGE_CROP_SIZE: 224
+  MAX_CAPTION_LENGTH: 30
+
+  IMAGE_TRANSFORM_TRAIN:
+    - "random_resized_crop"
+    - "horizontal_flip"
+    - "color_jitter"
+    - "normalize"
+
+  IMAGE_TRANSFORM_VAL:
+    - "smallest_resize"
+    - "center_crop"
+    - "normalize"
+
+  USE_PERCENTAGE: 100.0
+  USE_SINGLE_CAPTION: false
+
+MODEL:
+  NAME: "virtex"
+  VISUAL:
+    NAME: "torchvision::resnet50"
+    PRETRAINED: false
+    FROZEN: false
+  TEXTUAL:
+    NAME: "transdec_postnorm::L1_H1024_A16_F4096"
+    DROPOUT: 0.1
+
+OPTIM:
+  OPTIMIZER_NAME: "sgd"
+  SGD_MOMENTUM: 0.9
+  WEIGHT_DECAY: 0.0001
+
+  LOOKAHEAD:
+    USE: true
+    ALPHA: 0.5
+    STEPS: 5
+
+  BATCH_SIZE: 256
+  CNN_LR: 0.2
+  LR: 0.001
+  NUM_ITERATIONS: 500000
+
+  WARMUP_STEPS: 10000
+  LR_DECAY_NAME: "cosine"
+
+  NO_DECAY: ".*textual.(embedding|transformer).*(norm.*|bias)"
+  CLIP_GRAD_NORM: 10.0
+

virtex/configs/backbone_ablations/bicaptioning_R_101_L1_H1024.yaml

@@ -0,0 +1,5 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  VISUAL:
+    NAME: "torchvision::resnet101"

virtex/configs/backbone_ablations/bicaptioning_R_50W2X_L1_H1024.yaml

@@ -0,0 +1,5 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  VISUAL:
+    NAME: "torchvision::wide_resnet50_2"

virtex/configs/backbone_ablations/bicaptioning_R_50_L1_H1024.yaml

@@ -0,0 +1 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"

virtex/configs/depth_ablations/bicaptioning_R_50_L1_H1024.yaml

@@ -0,0 +1 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"

virtex/configs/depth_ablations/bicaptioning_R_50_L2_H1024.yaml

@@ -0,0 +1,5 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  TEXTUAL:
+    NAME: "transdec_postnorm::L2_H1024_A16_F4096"

virtex/configs/depth_ablations/bicaptioning_R_50_L3_H1024.yaml

@@ -0,0 +1,5 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  TEXTUAL:
+    NAME: "transdec_postnorm::L3_H1024_A16_F4096"

virtex/configs/depth_ablations/bicaptioning_R_50_L4_H1024.yaml

@@ -0,0 +1,5 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  TEXTUAL:
+    NAME: "transdec_postnorm::L4_H1024_A16_F4096"

virtex/configs/detectron2/_base_faster_rcnn_R_50_C4_BN.yaml

@@ -0,0 +1,49 @@
+# ----------------------------------------------------------------------------
+# Train a Faster R-CNN with ResNet-50 and C4 backbone. This config follows
+# Detectron2 format; and is unrelated with our VirTex configs. Params here
+# replicate evaluation protocol as per MoCo (https://arxiv.org/abs/1911.05722).
+# ----------------------------------------------------------------------------
+
+INPUT:
+  # Input format will always be RGB, consistent with torchvision.
+  FORMAT: "RGB"
+  MIN_SIZE_TRAIN: (640, 672, 704, 736, 768, 800)
+  MIN_SIZE_TEST: 800
+
+MODEL:
+  META_ARCHITECTURE: "GeneralizedRCNN"
+
+  # Train all layers end-to-end by default.
+  BACKBONE:
+    NAME: build_resnet_backbone
+    FREEZE_AT: 0
+
+  # Fine-tune with SyncBN.
+  # STRIDE_IN_1X1 is False for torchvision-like models.
+  RESNETS:
+    DEPTH: 50
+    NORM: SyncBN
+    STRIDE_IN_1X1: False
+
+  RPN:
+    PRE_NMS_TOPK_TEST: 6000
+    POST_NMS_TOPK_TEST: 1000
+
+  # ROI head with extra BN layer after res5 stage.
+  ROI_HEADS:
+    NAME: "Res5ROIHeadsExtraNorm"
+
+  # ImageNet color mean for torchvision-like models (RGB order).
+  PIXEL_MEAN: [123.675, 116.280, 103.530]
+  PIXEL_STD: [58.395, 57.120, 57.375]
+
+SOLVER:
+  # This is for 8 GPUs, apply linear scaling for 4 GPUs.
+  IMS_PER_BATCH: 16
+  BASE_LR: 0.02
+
+TEST:
+  PRECISE_BN:
+    ENABLED: True
+
+VERSION: 2

virtex/configs/detectron2/_base_mask_rcnn_R_50_FPN.yaml

@@ -0,0 +1,75 @@
+# ----------------------------------------------------------------------------
+# Train a Mask R-CNN with ResNet-50 and FPN backbone. This config follows
+# Detectron2 format; and is unrelated with our VirTex configs. Params here
+# replicate evaluation protocol as per MoCo (https://arxiv.org/abs/1911.05722).
+# ----------------------------------------------------------------------------
+
+INPUT:
+  # Input format will always be RGB, consistent with torchvision.
+  FORMAT: "RGB"
+  MIN_SIZE_TRAIN: (640, 672, 704, 736, 768, 800)
+  MIN_SIZE_TEST: 800
+
+MODEL:
+  META_ARCHITECTURE: "GeneralizedRCNN"
+
+  # Train all layers end-to-end by default.
+  BACKBONE:
+    NAME: "build_resnet_fpn_backbone"
+    FREEZE_AT: 0
+
+  # Fine-tune with SyncBN.
+  # STRIDE_IN_1X1 is False for torchvision-like models.
+  RESNETS:
+    DEPTH: 50
+    NORM: "SyncBN"
+    STRIDE_IN_1X1: False
+    OUT_FEATURES: ["res2", "res3", "res4", "res5"]
+
+  FPN:
+    IN_FEATURES: ["res2", "res3", "res4", "res5"]
+
+  ANCHOR_GENERATOR:
+    # One size for each in feature map
+    SIZES: [[32], [64], [128], [256], [512]]
+    # Three aspect ratios (same for all in feature maps)
+    ASPECT_RATIOS: [[0.5, 1.0, 2.0]]
+
+  RPN:
+    IN_FEATURES: ["p2", "p3", "p4", "p5", "p6"]
+    PRE_NMS_TOPK_TRAIN: 2000
+    PRE_NMS_TOPK_TEST: 1000
+
+    POST_NMS_TOPK_TRAIN: 1000
+    POST_NMS_TOPK_TEST: 1000
+
+  ROI_HEADS:
+    NAME: "StandardROIHeads"
+    IN_FEATURES: ["p2", "p3", "p4", "p5"]
+
+  ROI_BOX_HEAD:
+    NAME: "FastRCNNConvFCHead"
+    NUM_FC: 2
+    POOLER_RESOLUTION: 7
+
+  ROI_MASK_HEAD:
+    NAME: "MaskRCNNConvUpsampleHead"
+    NUM_CONV: 4
+    POOLER_RESOLUTION: 14
+
+  # ImageNet color mean for torchvision-like models (RGB order).
+  # These are in [0-255] range as expected by Detectron2. Rest of our codebase
+  # uses [0-1] range; but both are equivalent and consistent.
+  PIXEL_MEAN: [123.675, 116.280, 103.530]
+  PIXEL_STD: [58.395, 57.120, 57.375]
+
+SOLVER:
+  # This is for 8 GPUs, apply linear scaling for 4 GPUs.
+  IMS_PER_BATCH: 16
+  BASE_LR: 0.02
+
+TEST:
+  PRECISE_BN:
+    ENABLED: True
+
+VERSION: 2

virtex/configs/detectron2/coco_segm_default_init_2x.yaml

@@ -0,0 +1,24 @@
+# -----------------------------------------------------------------------------
+# Train a Mask R-CNN R50-FPN backbone on LVIS instance segmentation with any of
+# these weight init: random, imagenet (torchvision), virtex or MoCo.
+# -----------------------------------------------------------------------------
+_BASE_: "_base_mask_rcnn_R_50_FPN.yaml"
+
+DATASETS:
+  TRAIN: ("coco_2017_train",)
+  TEST: ("coco_2017_val",)
+
+MODEL:
+  MASK_ON: True
+  # FPN also has SyncBN, as opposed to no norm (usually).
+  FPN:
+    NORM: "SyncBN"
+  
+  # This will be ignored, weights will be loaded manually in the script.
+  WEIGHTS: ""
+  
+SOLVER:
+  STEPS: (120000, 160000)
+  MAX_ITER: 180000
+  
+VERSION: 2

virtex/configs/detectron2/lvis_segm_default_init_2x.yaml

@@ -0,0 +1,36 @@
+# -----------------------------------------------------------------------------
+# Train a Mask R-CNN R50-FPN backbone on LVIS instance segmentation with any of
+# these weight init: random, virtex or MoCo. (ImageNet init config is separate)
+# -----------------------------------------------------------------------------
+_BASE_: "_base_mask_rcnn_R_50_FPN.yaml"
+
+DATASETS:
+  TRAIN: ("lvis_v1_train",)
+  TEST: ("lvis_v1_val",)
+
+DATALOADER:
+  SAMPLER_TRAIN: "RepeatFactorTrainingSampler"
+  REPEAT_THRESHOLD: 0.001
+
+TEST:
+  DETECTIONS_PER_IMAGE: 300  # LVIS allows up to 300.
+
+MODEL:
+  MASK_ON: True
+  # FPN also has SyncBN, as opposed to no norm (usually).
+  FPN:
+    NORM: "SyncBN"
+
+  ROI_HEADS:
+    NUM_CLASSES: 1203
+    SCORE_THRESH_TEST: 0.0001
+
+  # This will be ignored, weights will be loaded manually in the script.
+  WEIGHTS: ""
+
+SOLVER:
+  STEPS: (120000, 160000)
+  MAX_ITER: 180000
+
+VERSION: 2
+

virtex/configs/detectron2/lvis_segm_imagenet_init_2x.yaml

@@ -0,0 +1,38 @@
+# -----------------------------------------------------------------------------
+# Train a Mask R-CNN R50-FPN backbone on LVIS instance segmentation
+# with weights initialized from supervised ImageNet pretraining (torchvision).
+# Key difference is that fine-tuning here happens with BN frozen.
+# -----------------------------------------------------------------------------
+_BASE_: "_base_mask_rcnn_R_50_FPN.yaml"
+
+DATASETS:
+  TRAIN: ("lvis_v1_train",)
+  TEST: ("lvis_v1_val",)
+
+DATALOADER:
+  SAMPLER_TRAIN: "RepeatFactorTrainingSampler"
+  REPEAT_THRESHOLD: 0.001
+
+TEST:
+  DETECTIONS_PER_IMAGE: 300  # LVIS allows up to 300.
+
+MODEL:
+  MASK_ON: True
+  RESNETS:
+    NORM: "FrozenBN"
+
+  # Do not tune with SyncBN for ImageNet init from LVIS.
+  ROI_HEADS:
+    NUM_CLASSES: 1203
+    SCORE_THRESH_TEST: 0.0001
+
+  # This will be ignored, weights will be loaded manually in the script.
+  WEIGHTS: ""
+
+SOLVER:
+  STEPS: (120000, 160000)
+  MAX_ITER: 180000
+
+VERSION: 2
+
+

virtex/configs/detectron2/voc_det_default_init_24k.yaml

@@ -0,0 +1,28 @@
+# -----------------------------------------------------------------------------
+# Train a Faster R-CNN with R50-C4 backbone on VOC07+12 detection with any of
+# these weight init: random, imagenet (torchvision), virtex or MoCo.
+# -----------------------------------------------------------------------------
+_BASE_: "_base_faster_rcnn_R_50_C4_BN.yaml"
+
+DATASETS:
+  TRAIN: ("voc_2007_trainval", "voc_2012_trainval")
+  TEST: ("voc_2007_test",)
+
+INPUT:
+  MIN_SIZE_TRAIN: (480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800)
+  MIN_SIZE_TEST: 800
+
+MODEL:
+  MASK_ON: False
+  ROI_HEADS:
+    NUM_CLASSES: 20
+
+  # This will be ignored, weights will be loaded manually in the script.
+  WEIGHTS: ""
+
+SOLVER:
+  STEPS: (18000, 22000)
+  MAX_ITER: 24000
+  WARMUP_ITERS: 100
+
+VERSION: 2

virtex/configs/downstream/imagenet_clf.yaml

@@ -0,0 +1,33 @@
+RANDOM_SEED: 0
+# Don't need AMP to train a tiny linear layer.
+AMP: false
+CUDNN_BENCHMARK: true
+CUDNN_DETERMINISTIC: false
+
+DATA:
+  ROOT: "datasets/imagenet"
+  IMAGE_TRANSFORM_TRAIN:
+    - "random_resized_crop::{'scale': (0.08, 1.0)}"
+    - "horizontal_flip"
+    - "normalize"
+  IMAGE_TRANSFORM_VAL:
+    - "smallest_resize"
+    - "center_crop"
+    - "normalize"
+
+MODEL:
+  VISUAL:
+    FROZEN: true
+
+OPTIM:
+  BATCH_SIZE: 256
+  SGD_MOMENTUM: 0.9
+  WEIGHT_DECAY: 0.0
+  NO_DECAY: "none"
+  LOOKAHEAD:
+    USE: false
+
+  LR: 0.3
+  WARMUP_STEPS: 0
+  LR_DECAY_NAME: "cosine"
+  NUM_ITERATIONS: 500500  # 100 epochs

virtex/configs/downstream/inaturalist_clf.yaml

@@ -0,0 +1,36 @@
+RANDOM_SEED: 0
+AMP: true
+CUDNN_BENCHMARK: true
+CUDNN_DETERMINISTIC: false
+
+DATA:
+  ROOT: "datasets/inaturalist"
+  IMAGE_TRANSFORM_TRAIN:
+    - "random_resized_crop::{'scale': (0.08, 1.0)}"
+    - "horizontal_flip"
+    - "normalize"
+  IMAGE_TRANSFORM_VAL:
+    - "smallest_resize"
+    - "center_crop"
+    - "normalize"
+
+MODEL:
+  VISUAL:
+    FROZEN: false
+    
+OPTIM:
+  BATCH_SIZE: 256
+  SGD_MOMENTUM: 0.9
+  WEIGHT_DECAY: 0.0001
+  NO_DECAY: "none"
+  LOOKAHEAD:
+    USE: false
+
+  LR: 0.025
+  WARMUP_STEPS: 0
+  LR_DECAY_NAME: multistep
+  LR_GAMMA: 0.1
+  LR_STEPS:
+    - 119700  # 70 epochs
+    - 153900  # 90 epochs
+  NUM_ITERATIONS: 171000  # 100 epochs

virtex/configs/downstream/voc07_clf.yaml

@@ -0,0 +1,15 @@
+RANDOM_SEED: 0
+DATA:
+  ROOT: datasets/VOC2007
+  IMAGE_TRANSFORM_TRAIN:
+    - smallest_resize
+    - center_crop
+    - normalize
+  IMAGE_TRANSFORM_VAL:
+    - smallest_resize
+    - center_crop
+    - normalize
+
+OPTIM:
+  # Only used for feature extraction, doesn't mean much.
+  BATCH_SIZE: 128

virtex/configs/redcaps/gcc_R_50_L6_H512.yaml

@@ -0,0 +1,35 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+AMP: True
+
+DATA:
+  ROOT: "datasets/gcc/tarfiles/*.tar"
+  TOKENIZER_MODEL: "datasets/vocab/common_30k.model"
+  VOCAB_SIZE: 30000
+  UNK_INDEX: 0
+  SOS_INDEX: 1
+  EOS_INDEX: 2
+  MASK_INDEX: 3
+
+  MAX_CAPTION_LENGTH: 50
+
+MODEL:
+  NAME: "virtex_web"
+  TEXTUAL:
+    NAME: "transdec_prenorm::L6_H512_A8_F2048"
+
+  LABEL_SMOOTHING: 0.1
+
+OPTIM:
+  OPTIMIZER_NAME: "adamw"
+  WEIGHT_DECAY: 0.01
+  LOOKAHEAD:
+    USE: false
+
+  BATCH_SIZE: 256
+  CNN_LR: 0.0005
+  LR: 0.0005
+  NUM_ITERATIONS: 1500000
+
+  WARMUP_STEPS: 10000
+  LR_DECAY_NAME: "cosine"

virtex/configs/redcaps/miniclip_sbu_R_50_L12_H512.yaml

@@ -0,0 +1,35 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+AMP: True
+
+DATA:
+  ROOT: "datasets/sbu/tarfiles/*.tar"
+  TOKENIZER_MODEL: "datasets/vocab/common_30k.model"
+  VOCAB_SIZE: 30000
+  UNK_INDEX: 0
+  SOS_INDEX: 1
+  EOS_INDEX: 2
+  MASK_INDEX: 3
+
+  MAX_CAPTION_LENGTH: 50
+
+MODEL:
+  NAME: "miniclip_web"
+  TEXTUAL:
+    NAME: "transenc_prenorm::L12_H512_A8_F2048"
+  LABEL_SMOOTHING: 0.1
+
+OPTIM:
+  OPTIMIZER_NAME: "adamw"
+  WEIGHT_DECAY: 0.01
+
+  LOOKAHEAD:
+    USE: false
+
+  BATCH_SIZE: 256
+  CNN_LR: 0.0005
+  LR: 0.0005
+  NUM_ITERATIONS: 1500000
+
+  WARMUP_STEPS: 10000
+  LR_DECAY_NAME: "cosine"

virtex/configs/redcaps/redcaps_2020_R_50_L6_H512.yaml

@@ -0,0 +1,35 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+AMP: True
+
+DATA:
+  ROOT: "datasets/redcaps/tarfiles/*_2020_*.tar"
+  TOKENIZER_MODEL: "datasets/vocab/common_30k.model"
+  VOCAB_SIZE: 30000
+  UNK_INDEX: 0
+  SOS_INDEX: 1
+  EOS_INDEX: 2
+  MASK_INDEX: 3
+
+  MAX_CAPTION_LENGTH: 50
+
+MODEL:
+  NAME: "virtex_web"
+  TEXTUAL:
+    NAME: "transdec_prenorm::L6_H512_A8_F2048"
+  LABEL_SMOOTHING: 0.1
+
+OPTIM:
+  OPTIMIZER_NAME: "adamw"
+  WEIGHT_DECAY: 0.01
+
+  LOOKAHEAD:
+    USE: false
+
+  BATCH_SIZE: 256
+  CNN_LR: 0.0005
+  LR: 0.0005
+  NUM_ITERATIONS: 1500000
+
+  WARMUP_STEPS: 10000
+  LR_DECAY_NAME: "cosine"

virtex/configs/redcaps/redcaps_all_R_50_L6_H512.yaml

@@ -0,0 +1,35 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+AMP: True
+
+DATA:
+  ROOT: "datasets/redcaps/tarfiles/*.tar"
+  TOKENIZER_MODEL: "datasets/vocab/common_30k.model"
+  VOCAB_SIZE: 30000
+  UNK_INDEX: 0
+  SOS_INDEX: 1
+  EOS_INDEX: 2
+  MASK_INDEX: 3
+
+  MAX_CAPTION_LENGTH: 50
+
+MODEL:
+  NAME: "virtex_web"
+  TEXTUAL:
+    NAME: "transdec_prenorm::L6_H512_A8_F2048"
+  LABEL_SMOOTHING: 0.1
+
+OPTIM:
+  OPTIMIZER_NAME: "adamw"
+  WEIGHT_DECAY: 0.01
+
+  LOOKAHEAD:
+    USE: false
+
+  BATCH_SIZE: 256
+  CNN_LR: 0.0005
+  LR: 0.0005
+  NUM_ITERATIONS: 1500000
+
+  WARMUP_STEPS: 10000
+  LR_DECAY_NAME: "cosine"

virtex/configs/redcaps/sbu_R_50_L6_H512.yaml

@@ -0,0 +1,35 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+AMP: True
+
+DATA:
+  ROOT: "datasets/sbu/tarfiles/*.tar"
+  TOKENIZER_MODEL: "datasets/vocab/common_30k.model"
+  VOCAB_SIZE: 30000
+  UNK_INDEX: 0
+  SOS_INDEX: 1
+  EOS_INDEX: 2
+  MASK_INDEX: 3
+
+  MAX_CAPTION_LENGTH: 50
+
+MODEL:
+  NAME: "virtex_web"
+  TEXTUAL:
+    NAME: "transdec_prenorm::L6_H512_A8_F2048"
+  LABEL_SMOOTHING: 0.1
+
+OPTIM:
+  OPTIMIZER_NAME: "adamw"
+  WEIGHT_DECAY: 0.01
+
+  LOOKAHEAD:
+    USE: false
+
+  BATCH_SIZE: 256
+  CNN_LR: 0.0005
+  LR: 0.0005
+  NUM_ITERATIONS: 1500000
+
+  WARMUP_STEPS: 10000
+  LR_DECAY_NAME: "cosine"

virtex/configs/task_ablations/bicaptioning_R_50_L1_H2048.yaml

@@ -0,0 +1,5 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  TEXTUAL:
+    NAME: "transdec_postnorm::L1_H2048_A32_F8192"

virtex/configs/task_ablations/captioning_R_50_L1_H2048.yaml

@@ -0,0 +1,6 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  NAME: "captioning"
+  TEXTUAL:
+    NAME: "transdec_postnorm::L1_H2048_A32_F8192"

virtex/configs/task_ablations/masked_lm_R_50_L1_H2048.yaml

@@ -0,0 +1,6 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  NAME: "masked_lm"
+  TEXTUAL:
+    NAME: "transdec_postnorm::L1_H2048_A32_F8192"

virtex/configs/task_ablations/multilabel_classification_R_50.yaml

@@ -0,0 +1,12 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+DATA:
+  VOCAB_SIZE: 81
+
+MODEL:
+  NAME: "multilabel_classification"
+  TEXTUAL:
+    NAME: "none"
+
+OPTIM:
+  NO_DECAY: "none"

virtex/configs/task_ablations/token_classification_R_50.yaml

@@ -0,0 +1,9 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  NAME: "token_classification"
+  TEXTUAL:
+    NAME: "none"
+
+OPTIM:
+  NO_DECAY: "none"

virtex/configs/width_ablations/bicaptioning_R_50_L1_H1024.yaml

@@ -0,0 +1 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"

virtex/configs/width_ablations/bicaptioning_R_50_L1_H2048.yaml

@@ -0,0 +1,5 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  TEXTUAL:
+    NAME: "transdec_postnorm::L1_H2048_A32_F8192"

virtex/configs/width_ablations/bicaptioning_R_50_L1_H512.yaml

@@ -0,0 +1,5 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  TEXTUAL:
+    NAME: "transdec_postnorm::L1_H512_A8_F2048"

virtex/configs/width_ablations/bicaptioning_R_50_L1_H768.yaml

@@ -0,0 +1,5 @@
+_BASE_: "../_base_bicaptioning_R_50_L1_H1024.yaml"
+
+MODEL:
+  TEXTUAL:
+    NAME: "transdec_postnorm::L1_H768_A12_F3072"

virtex/docs/Makefile

@@ -0,0 +1,19 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = ../../virtex-sphinx
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

virtex/docs/_static/custom.css

@@ -0,0 +1,115 @@
+body {
+    padding: 40px 0 0 0;
+    font-size: 12pt;
+    font-family: Inconsolata !important;
+}
+
+/* Monospace everywhere */
+h1, h2, h3, h4, div.sphinxsidebar h1, div.sphinxsidebar h2,
+div.sphinxsidebar h3, div.sphinxsidebar h4, div.body h1,
+div.body h2, div.body h3, div.body h4, .admonition-title {
+    font-family: monospace !important;
+}
+
+/* Make main content wider */
+div.document {
+    margin: auto;
+    width: 65%;
+}
+
+/* Make sidebar slightly wider. */
+div.sphinxsidebar {
+    width: 250px;
+}
+
+div.bodywrapper {
+    margin: 0 0 0 250px;
+}
+
+div.body {
+    color: black;
+    max-width: 100%
+}
+
+/* Darker headings */
+h1, h2, h3, h4, div.sphinxsidebar h1, div.sphinxsidebar h2,
+div.sphinxsidebar h3, div.sphinxsidebar h4, div.body h1,
+div.body h2, div.body h3, div.body h4 {
+    color: black;
+}
+
+@media screen and (max-width: 875px) {
+    div.sphinxsidebar {
+        background-color: white;
+    }
+}
+
+/* Darker bold words */
+strong {
+    color: #252525;
+}
+
+/* TOC tree tag, view source link & permalink anchor styling. */
+div.sphinxsidebar a, .viewcode-link, a.reference {
+    color: darkgreen;
+    text-decoration: none;
+    border-bottom: 1px dashed green;
+    text-underline-position: under;
+}
+a.headerlink {
+    color: black;
+}
+
+/* TOC tree tag, view source link & permalink anchor styling. */
+div.sphinxsidebar a:hover, .viewcode-link:hover, a.reference:hover,
+a.headerlink:hover {
+    font-weight: 700;
+    border-bottom: 1px solid green;
+}
+
+/* Add a light background to class signatures. */
+dl.class > dt:first-of-type, dl.function > dt:first-of-type,
+dl.method > dt:first-of-type, dl.classmethod > dt:first-of-type,
+dl.attribute > dt:first-of-type, dl.data > dt:first-of-type {
+    font-size: 14pt;
+    background-color: #d8f6e9;
+    padding: 10px 20px 10px 10px;
+    border: 1px solid #1b5e20;
+}
+
+/* Add lightgrey background to code snippets. */
+pre {
+    background-color: #eeeeee !important;
+    border: 1pt solid #999999;
+    border-radius: 5px;
+}
+
+/* Dark orange-red comments in code snippets. */
+.highlight .c1 {
+    color: #dd4533;
+}
+
+.admonition, .note {
+    background-color: #fed8b1 !important;
+    border: 1pt solid #ff7700;
+    border-radius: 5px;
+}
+
+/* Make "Parameters" subsection wider - display heading and content vertically. */
+dl.field-list {
+    display: block;
+}
+
+/* Increase font size of subsection headings ("Parameters", "Examples" etc.) */
+.rubric, dl.field-list > dt.field-odd, dl.field-list > dt.field-even {
+    color: black;
+    font-size: 18pt;
+    font-weight: bold;
+    padding: 0px;
+    margin: 20px 0px 20px 0px;
+}
+
+/* Add margins around methods and properties. */
+.py {
+    margin: 20px 0px 20px 0px;
+}

virtex/docs/_templates/layout.html

@@ -0,0 +1,19 @@
+{% extends "!layout.html" %}
+
+{% block htmltitle %}
+
+    <!-- Global site tag (gtag.js) - Google Analytics -->
+    <script async src="https://www.googletagmanager.com/gtag/js?id=UA-120523111-2"></script>
+    <script>
+    window.dataLayer = window.dataLayer || [];
+    function gtag(){dataLayer.push(arguments);}
+    gtag('js', new Date());
+
+    gtag('config', 'UA-120523111-2');
+    </script>
+
+    <link href="https://fonts.googleapis.com/css?family=Inconsolata&display=swap" rel="stylesheet">
+    <link href="https://fonts.googleapis.com/css?family=Ubuntu+Mono&display=swap" rel="stylesheet">
+
+{{ super() }}
+{% endblock %}

virtex/docs/conf.py

@@ -0,0 +1,173 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import inspect
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath("../"))
+
+
+# -- Project information -----------------------------------------------------
+
+project = "virtex"
+copyright = "2021, Karan Desai and Justin Johnson"
+author = "Karan Desai"
+
+# The full version, including alpha/beta/rc tags
+release = "1.1"
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.coverage",
+    "sphinx.ext.doctest",
+    "sphinx.ext.linkcode",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.coverage",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.mathjax",
+    "sphinx_copybutton",
+    "numpydoc",
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = ".rst"
+
+# The master toctree document.
+master_doc = "index"
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# This version is used underneath the title on the index page.
+version = "1.1"
+# The following is used if you need to also include a more detailed version.
+release = "1.1"
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = "en"
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This patterns also effect to html_static_path and html_extra_path
+exclude_patterns = ["_build"]
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = False
+
+numpydoc_show_class_members = False
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = "alabaster"
+
+# html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {"collapse_navigation": False, "display_version": True}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["_static"]
+
+
+# -- Autodoc configuration ------------------------------------------------
+
+autodoc_default_options = {
+    "members": True,
+    "member-order": "bysource",
+    "private-members": True,
+    "show-inheritance": True,
+}
+
+
+# -- Intersphinx configuration --------------------------------------------
+
+intersphinx_mapping = {
+    "torch": ("https://pytorch.org/docs/stable/", None),
+    "albumentations": ("https://albumentations.readthedocs.io/en/latest/", None),
+}
+
+# -- Miscellaneous Extra Tweaks -------------------------------------------
+
+# make github links resolve
+def linkcode_resolve(domain, info):
+    """
+    Determine the URL corresponding to Python object
+    This code is from
+    https://github.com/numpy/numpy/blob/master/doc/source/conf.py#L290
+    and https://github.com/Lasagne/Lasagne/pull/262
+    """
+    if domain != "py":
+        return None
+
+    modname = info["module"]
+    fullname = info["fullname"]
+
+    submod = sys.modules.get(modname)
+    if submod is None:
+        return None
+
+    obj = submod
+    for part in fullname.split("."):
+        try:
+            obj = getattr(obj, part)
+        except:  # noqa: E722
+            return None
+
+    try:
+        fn = inspect.getsourcefile(obj)
+    except:  # noqa: E722
+        fn = None
+    if not fn:
+        return None
+
+    try:
+        source, lineno = inspect.getsourcelines(obj)
+    except:  # noqa: E722
+        lineno = None
+
+    if lineno:
+        linespec = "#L%d-L%d" % (lineno, lineno + len(source) - 1)
+    else:
+        linespec = ""
+
+    filename = info["module"].replace(".", "/")
+    return f"https://github.com/kdexd/virtex/blob/master/{filename}.py{linespec}"

virtex/docs/index.rst

@@ -0,0 +1,122 @@
+.. raw:: html
+
+    <h1 style="text-align: center">
+    VirTex: Learning Visual Representations from Textual Annotations
+    </h1>
+    <h4 style="text-align: center">
+    Karan Desai and Justin Johnson
+    </br>
+    <span style="font-size: 14pt; color: #555555">
+    University of Michigan
+    </span>
+    </h4>
+    <hr>
+
+    <h4 style="text-align: center">
+    Abstract
+    </h4>
+
+    <p style="text-align: justify">
+    The de-facto approach to many vision tasks is to start from pretrained
+    visual representations, typically learned via supervised training on
+    ImageNet. Recent methods have explored unsupervised pretraining to scale to
+    vast quantities of unlabeled images. In contrast, we aim to learn
+    high-quality visual representations from fewer images. To this end we
+    revisit supervised pretraining, and seek data-efficient alternatives to
+    classification-based pretraining. We propose VirTex -- a pretraining
+    approach using semantically dense captions to learn visual representations.
+    We train convolutional networks from scratch on COCO Captions, and transfer
+    them to downstream recognition tasks including image classification, object
+    detection, and instance segmentation. On all tasks, VirTex yields features
+    that match or exceed those learned on ImageNet -- supervised or unsupervised
+    -- despite using up to ten times fewer images.
+    </p>
+
+**CVPR 2021. Paper available at:** `arxiv.org/abs/2006.06666 <https://arxiv.org/abs/2006.06666>`_.
+
+**Code available at:** `github.com/kdexd/virtex <https://github.com/kdexd/virtex>`_.
+
+.. image:: _static/system_figure.jpg
+
+
+Get the pretrained ResNet-50 visual backbone from our best performing VirTex
+model in one line *without any installation*!
+
+.. code-block:: python
+
+    import torch
+
+    # That's it, this one line only requires PyTorch.
+    model = torch.hub.load("kdexd/virtex", "resnet50", pretrained=True)
+
+
+More details in :doc:`virtex/usage/model_zoo`. Next, dive deeper into our
+code with User Guide and API References!
+
+
+User Guide
+----------
+
+.. toctree::
+    :maxdepth: 2
+
+    virtex/usage/setup_dependencies
+    virtex/usage/model_zoo
+    virtex/usage/pretrain
+    virtex/usage/downstream
+
+
+API Reference
+-------------
+
+.. toctree::
+    :maxdepth: 2
+
+    virtex/config
+    virtex/factories
+    virtex/data
+    virtex/models
+    virtex/modules
+    virtex/optim
+    virtex/utils
+    virtex/model_zoo
+
+
+Citation
+--------
+
+If you find this code useful, please consider citing:
+
+.. code-block:: text
+
+    @inproceedings{desai2021virtex,
+        title={{VirTex: Learning Visual Representations from Textual Annotations}},
+        author={Karan Desai and Justin Johnson},
+        booktitle={CVPR},
+        year={2021}
+    }
+
+
+Acknowledgments
+---------------
+
+We thank Harsh Agrawal, Mohamed El Banani, Richard  Higgins, Nilesh Kulkarni
+and Chris Rockwell for helpful discussions and feedback on the paper. We thank
+Ishan Misra for discussions regarding PIRL evaluation protocol; Saining Xie for
+discussions about replicating iNaturalist evaluation as MoCo; Ross Girshick and
+Yuxin Wu for help with Detectron2 model zoo; Georgia Gkioxari for suggesting
+the Instance Segmentation pretraining task ablation; and Stefan Lee for
+suggestions on figure aesthetics. We thank Jia Deng for access to extra GPUs
+during project development; and UMich ARC-TS team for support with GPU cluster
+management. Finally, we thank all the Starbucks outlets in Ann Arbor for many
+hours of free WiFi. This work was partially supported by the Toyota Research
+Institute (TRI). However, note that this article solely reflects the opinions
+and conclusions of its authors and not TRI or any other Toyota entity.
+
+
+Indices and Tables
+------------------
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

virtex/docs/virtex/config.rst

@@ -0,0 +1,18 @@
+virtex.config
+=============
+
+.. raw:: html
+
+    <hr>
+
+.. automodule:: virtex.config
+
+
+Config References
+-----------------
+
+.. literalinclude:: ../../virtex/config.py
+  :language: python
+  :linenos:
+  :lines: 46-206
+  :dedent: 8

virtex/docs/virtex/data.datasets.rst

@@ -0,0 +1,20 @@
+virtex.data.datasets
+====================
+
+.. raw:: html
+
+    <hr>
+
+Pretraining Datasets
+--------------------
+
+.. automodule:: virtex.data.datasets.captioning
+
+.. automodule:: virtex.data.datasets.classification
+
+------------------------------------------------------------------------------
+
+Downstream Datasets
+-------------------
+
+.. automodule:: virtex.data.datasets.downstream

virtex/docs/virtex/data.readers.rst

@@ -0,0 +1,8 @@
+virtex.data.readers
+===================
+
+.. raw:: html
+
+    <hr>
+
+.. automodule:: virtex.data.readers

virtex/docs/virtex/data.rst

@@ -0,0 +1,14 @@
+virtex.data
+===========
+
+.. raw:: html
+
+    <hr>
+
+
+.. toctree::
+
+    data.readers
+    data.datasets
+    data.tokenizers
+    data.transforms

virtex/docs/virtex/data.tokenizers.rst

@@ -0,0 +1,8 @@
+virtex.data.tokenizers
+======================
+
+.. raw:: html
+
+    <hr>
+
+.. automodule:: virtex.data.tokenizers

virtex/docs/virtex/data.transforms.rst

@@ -0,0 +1,8 @@
+virtex.data.transforms
+======================
+
+.. raw:: html
+
+    <hr>
+
+.. automodule:: virtex.data.transforms

virtex/docs/virtex/factories.rst

@@ -0,0 +1,56 @@
+virtex.factories
+================
+
+.. raw:: html
+
+    <hr>
+
+.. First only include the top-level module, and base class docstrings.
+
+.. automodule:: virtex.factories
+    :no-members:
+
+.. autoclass:: virtex.factories.Factory
+
+
+------------------------------------------------------------------------------
+
+Dataloading-related Factories
+-----------------------------
+
+.. autoclass:: virtex.factories.TokenizerFactory
+    :members: from_config
+
+.. autoclass:: virtex.factories.ImageTransformsFactory
+    :members: from_config
+
+.. autoclass:: virtex.factories.PretrainingDatasetFactory
+    :members: from_config
+
+.. autoclass:: virtex.factories.DownstreamDatasetFactory
+    :members: from_config
+
+------------------------------------------------------------------------------
+
+Modeling-related Factories
+--------------------------
+
+.. autoclass:: virtex.factories.VisualBackboneFactory
+    :members: from_config
+
+.. autoclass:: virtex.factories.TextualHeadFactory
+    :members: from_config
+
+.. autoclass:: virtex.factories.PretrainingModelFactory
+    :members: from_config
+
+------------------------------------------------------------------------------
+
+Optimization-related Factories
+------------------------------
+
+.. autoclass:: virtex.factories.OptimizerFactory
+    :members: from_config
+
+.. autoclass:: virtex.factories.LRSchedulerFactory
+    :members: from_config

virtex/docs/virtex/model_zoo.rst

@@ -0,0 +1,8 @@
+virtex.model_zoo
+================
+
+.. raw:: html
+
+    <hr>
+
+.. automodule:: virtex.model_zoo.model_zoo