Upload 704 files

CITATION.cff

@@ -0,0 +1,9 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+title: "OpenMMLab Text Detection, Recognition and Understanding Toolbox"
+authors:
+  - name: "MMOCR Contributors"
+version: 0.3.0
+date-released: 2020-08-15
+repository-code: "https://github.com/open-mmlab/mmocr"
+license: Apache-2.0

configs/backbone/oclip/README.md

@@ -0,0 +1,41 @@
+# oCLIP
+
+> [Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting](https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136880282.pdf)
+
+<!-- [ALGORITHM] -->
+
+## Abstract
+
+Recently, Vision-Language Pre-training (VLP) techniques have greatly benefited various vision-language tasks by jointly learning visual and textual representations, which intuitively helps in Optical Character Recognition (OCR) tasks due to the rich visual and textual information in scene text images. However, these methods cannot well cope with OCR tasks because of the difficulty in both instance-level text encoding and image-text pair acquisition (i.e. images and captured texts in them). This paper presents a weakly supervised pre-training method, oCLIP, which can acquire effective scene text representations by jointly learning and aligning visual and textual information. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features, respectively, as well as a visual-textual decoder that models the interaction among textual and visual features for learning effective scene text representations. With the learning of textual features, the pre-trained model can attend texts in images well with character awareness. Besides, these designs enable the learning from weakly annotated texts (i.e. partial texts in images without text bounding boxes) which mitigates the data annotation constraint greatly. Experiments over the weakly annotated images in ICDAR2019-LSVT show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks, respectively. In addition, the proposed method outperforms existing pre-training techniques consistently across multiple public datasets (e.g., +3.2% and +1.3% for Total-Text and CTW1500).
+
+<div align=center>
+<img src="https://user-images.githubusercontent.com/24622904/199475057-aa688422-518d-4d7a-86fc-1be0cc1b5dc6.png"/>
+</div>
+
+## Models
+
+| Backbone  | Pre-train Data |                                       Model                                       |
+| :-------: | :------------: | :-------------------------------------------------------------------------------: |
+| ResNet-50 |   SynthText    | [Link](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |
+
+```{note}
+The model is converted from the official [oCLIP](https://github.com/bytedance/oclip.git).
+```
+
+## Supported Text Detection Models
+
+|           | [DBNet](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#dbnet) | [DBNet++](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#dbnetpp) | [FCENet](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#fcenet) | [TextSnake](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#fcenet) | [PSENet](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#psenet) | [DRRG](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#drrg) | [Mask R-CNN](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#mask-r-cnn) |
+| :-------: | :------------------------------------------------------------------------: | :----------------------------------------------------------------------------: | :--------------------------------------------------------------------------: | :-----------------------------------------------------------------------------: | :--------------------------------------------------------------------------: | :----------------------------------------------------------------------: | :----------------------------------------------------------------------------------: |
+| ICDAR2015 |                                     ✓                                      |                                       ✓                                        |                                      ✓                                       |                                                                                 |                                      ✓                                       |                                                                          |                                          ✓                                           |
+|  CTW1500  |                                                                            |                                                                                |                                      ✓                                       |                                        ✓                                        |                                      ✓                                       |                                    ✓                                     |                                          ✓                                           |
+
+## Citation
+
+```bibtex
+@article{xue2022language,
+  title={Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting},
+  author={Xue, Chuhui and Zhang, Wenqing and Hao, Yu and Lu, Shijian and Torr, Philip and Bai, Song},
+  journal={Proceedings of the European Conference on Computer Vision (ECCV)},
+  year={2022}
+}
+```

configs/backbone/oclip/metafile.yml

@@ -0,0 +1,13 @@
+Collections:
+- Name: oCLIP
+  Metadata:
+    Training Data: SynthText
+    Architecture:
+      - CLIPResNet
+  Paper:
+    URL: https://arxiv.org/abs/2203.03911
+    Title: 'Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting'
+  README: configs/backbone/oclip/README.md
+
+Models:
+  Weights: https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth

configs/kie/_base_/datasets/wildreceipt-openset.py

@@ -0,0 +1,26 @@
+wildreceipt_openset_data_root = 'data/wildreceipt/'
+
+wildreceipt_openset_train = dict(
+    type='WildReceiptDataset',
+    data_root=wildreceipt_openset_data_root,
+    metainfo=dict(category=[
+        dict(id=0, name='bg'),
+        dict(id=1, name='key'),
+        dict(id=2, name='value'),
+        dict(id=3, name='other')
+    ]),
+    ann_file='openset_train.txt',
+    pipeline=None)
+
+wildreceipt_openset_test = dict(
+    type='WildReceiptDataset',
+    data_root=wildreceipt_openset_data_root,
+    metainfo=dict(category=[
+        dict(id=0, name='bg'),
+        dict(id=1, name='key'),
+        dict(id=2, name='value'),
+        dict(id=3, name='other')
+    ]),
+    ann_file='openset_test.txt',
+    test_mode=True,
+    pipeline=None)

configs/kie/_base_/datasets/wildreceipt.py

@@ -0,0 +1,16 @@
+wildreceipt_data_root = 'data/wildreceipt/'
+
+wildreceipt_train = dict(
+    type='WildReceiptDataset',
+    data_root=wildreceipt_data_root,
+    metainfo=wildreceipt_data_root + 'class_list.txt',
+    ann_file='train.txt',
+    pipeline=None)
+
+wildreceipt_test = dict(
+    type='WildReceiptDataset',
+    data_root=wildreceipt_data_root,
+    metainfo=wildreceipt_data_root + 'class_list.txt',
+    ann_file='test.txt',
+    test_mode=True,
+    pipeline=None)

configs/kie/_base_/default_runtime.py

@@ -0,0 +1,33 @@
+default_scope = 'mmocr'
+env_cfg = dict(
+    cudnn_benchmark=False,
+    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
+    dist_cfg=dict(backend='nccl'),
+)
+randomness = dict(seed=None)
+
+default_hooks = dict(
+    timer=dict(type='IterTimerHook'),
+    logger=dict(type='LoggerHook', interval=100),
+    param_scheduler=dict(type='ParamSchedulerHook'),
+    checkpoint=dict(type='CheckpointHook', interval=1),
+    sampler_seed=dict(type='DistSamplerSeedHook'),
+    sync_buffer=dict(type='SyncBuffersHook'),
+    visualization=dict(
+        type='VisualizationHook',
+        interval=1,
+        enable=False,
+        show=False,
+        draw_gt=False,
+        draw_pred=False),
+)
+
+# Logging
+log_level = 'INFO'
+log_processor = dict(type='LogProcessor', window_size=10, by_epoch=True)
+
+load_from = None
+resume = False
+
+visualizer = dict(
+    type='KIELocalVisualizer', name='visualizer', is_openset=False)

configs/kie/_base_/schedules/schedule_adam_60e.py

@@ -0,0 +1,10 @@
+# optimizer
+optim_wrapper = dict(
+    type='OptimWrapper', optimizer=dict(type='Adam', weight_decay=0.0001))
+train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=60, val_interval=1)
+val_cfg = dict(type='ValLoop')
+test_cfg = dict(type='TestLoop')
+# learning rate
+param_scheduler = [
+    dict(type='MultiStepLR', milestones=[40, 50], end=60),
+]

configs/kie/sdmgr/README.md

@@ -0,0 +1,41 @@
+# SDMGR
+
+> [Spatial Dual-Modality Graph Reasoning for Key Information Extraction](https://arxiv.org/abs/2103.14470)
+
+<!-- [ALGORITHM] -->
+
+## Abstract
+
+Key information extraction from document images is of paramount importance in office automation. Conventional template matching based approaches fail to generalize well to document images of unseen templates, and are not robust against text recognition errors. In this paper, we propose an end-to-end Spatial Dual-Modality Graph Reasoning method (SDMG-R) to extract key information from unstructured document images. We model document images as dual-modality graphs, nodes of which encode both the visual and textual features of detected text regions, and edges of which represent the spatial relations between neighboring text regions. The key information extraction is solved by iteratively propagating messages along graph edges and reasoning the categories of graph nodes. In order to roundly evaluate our proposed method as well as boost the future research, we release a new dataset named WildReceipt, which is collected and annotated tailored for the evaluation of key information extraction from document images of unseen templates in the wild. It contains 25 key information categories, a total of about 69000 text boxes, and is about 2 times larger than the existing public datasets. Extensive experiments validate that all information including visual features, textual features and spatial relations can benefit key information extraction. It has been shown that SDMG-R can effectively extract key information from document images of unseen templates, and obtain new state-of-the-art results on the recent popular benchmark SROIE and our WildReceipt. Our code and dataset will be publicly released.
+
+<div align=center>
+<img src="https://user-images.githubusercontent.com/22607038/142580689-18edb4d7-f716-475c-b1c1-e2b934658cee.png"/>
+</div>
+
+## Results and models
+
+### WildReceipt
+
+|                                 Method                                 |     Modality     | Macro F1-Score |                                               Download                                               |
+| :--------------------------------------------------------------------: | :--------------: | :------------: | :--------------------------------------------------------------------------------------------------: |
+|   [sdmgr_unet16](/configs/kie/sdmgr/sdmgr_unet16_60e_wildreceipt.py)   | Visual + Textual |     0.890      | [model](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_unet16_60e_wildreceipt/sdmgr_unet16_60e_wildreceipt_20220825_151648-22419f37.pth) \| [log](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_unet16_60e_wildreceipt/20220825_151648.log) |
+| [sdmgr_novisual](/configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt.py) |     Textual      |     0.873      | [model](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt/sdmgr_novisual_60e_wildreceipt_20220831_193317-827649d8.pth) \| [log](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt/20220831_193317.log) |
+
+### WildReceiptOpenset
+
+|                                Method                                 | Modality | Edge F1-Score | Node Macro F1-Score | Node Micro F1-Score |                                 Download                                 |
+| :-------------------------------------------------------------------: | :------: | :-----------: | :-----------------: | :-----------------: | :----------------------------------------------------------------------: |
+| [sdmgr_novisual_openset](/configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt-openset.py) | Textual  |     0.792     |        0.931        |        0.940        | [model](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt-openset/sdmgr_novisual_60e_wildreceipt-openset_20220831_200807-dedf15ec.pth) \| [log](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt-openset/20220831_200807.log) |
+
+## Citation
+
+```bibtex
+@misc{sun2021spatial,
+      title={Spatial Dual-Modality Graph Reasoning for Key Information Extraction},
+      author={Hongbin Sun and Zhanghui Kuang and Xiaoyu Yue and Chenhao Lin and Wayne Zhang},
+      year={2021},
+      eprint={2103.14470},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```

configs/kie/sdmgr/_base_sdmgr_novisual.py

@@ -0,0 +1,35 @@
+num_classes = 26
+
+model = dict(
+    type='SDMGR',
+    kie_head=dict(
+        type='SDMGRHead',
+        visual_dim=16,
+        num_classes=num_classes,
+        module_loss=dict(type='SDMGRModuleLoss'),
+        postprocessor=dict(type='SDMGRPostProcessor')),
+    dictionary=dict(
+        type='Dictionary',
+        dict_file='{{ fileDirname }}/../../../dicts/sdmgr_dict.txt',
+        with_padding=True,
+        with_unknown=True,
+        unknown_token=None),
+)
+
+train_pipeline = [
+    dict(type='LoadKIEAnnotations'),
+    dict(type='Resize', scale=(1024, 512), keep_ratio=True),
+    dict(type='PackKIEInputs')
+]
+test_pipeline = [
+    dict(type='LoadKIEAnnotations'),
+    dict(type='Resize', scale=(1024, 512), keep_ratio=True),
+    dict(type='PackKIEInputs'),
+]
+
+val_evaluator = dict(
+    type='F1Metric',
+    mode='macro',
+    num_classes=num_classes,
+    ignored_classes=[0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 25])
+test_evaluator = val_evaluator

configs/kie/sdmgr/_base_sdmgr_unet16.py

@@ -0,0 +1,28 @@
+_base_ = '_base_sdmgr_novisual.py'
+
+model = dict(
+    backbone=dict(type='UNet', base_channels=16),
+    roi_extractor=dict(
+        type='mmdet.SingleRoIExtractor',
+        roi_layer=dict(type='RoIAlign', output_size=7),
+        featmap_strides=[1]),
+    data_preprocessor=dict(
+        type='ImgDataPreprocessor',
+        mean=[123.675, 116.28, 103.53],
+        std=[58.395, 57.12, 57.375],
+        bgr_to_rgb=True,
+        pad_size_divisor=32),
+)
+
+train_pipeline = [
+    dict(type='LoadImageFromFile'),
+    dict(type='LoadKIEAnnotations'),
+    dict(type='Resize', scale=(1024, 512), keep_ratio=True),
+    dict(type='PackKIEInputs')
+]
+test_pipeline = [
+    dict(type='LoadImageFromFile'),
+    dict(type='LoadKIEAnnotations'),
+    dict(type='Resize', scale=(1024, 512), keep_ratio=True),
+    dict(type='PackKIEInputs', meta_keys=('img_path', )),
+]

configs/kie/sdmgr/metafile.yml

@@ -0,0 +1,52 @@
+Collections:
+- Name: SDMGR
+  Metadata:
+    Training Data: KIEDataset
+    Training Techniques:
+      - Adam
+    Training Resources: 1x NVIDIA A100-SXM4-80GB
+    Architecture:
+      - UNet
+      - SDMGRHead
+  Paper:
+    URL: https://arxiv.org/abs/2103.14470.pdf
+    Title: 'Spatial Dual-Modality Graph Reasoning for Key Information Extraction'
+  README: configs/kie/sdmgr/README.md
+
+Models:
+  - Name: sdmgr_unet16_60e_wildreceipt
+    Alias: SDMGR
+    In Collection: SDMGR
+    Config: configs/kie/sdmgr/sdmgr_unet16_60e_wildreceipt.py
+    Metadata:
+      Training Data: wildreceipt
+    Results:
+      - Task: Key Information Extraction
+        Dataset: wildreceipt
+        Metrics:
+          macro_f1: 0.890
+    Weights: https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_unet16_60e_wildreceipt/sdmgr_unet16_60e_wildreceipt_20220825_151648-22419f37.pth
+  - Name: sdmgr_novisual_60e_wildreceipt
+    In Collection: SDMGR
+    Config: configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt.py
+    Metadata:
+      Training Data: wildreceipt
+    Results:
+      - Task: Key Information Extraction
+        Dataset: wildreceipt
+        Metrics:
+          macro_f1: 0.873
+    Weights: https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt/sdmgr_novisual_60e_wildreceipt_20220831_193317-827649d8.pth
+  - Name: sdmgr_novisual_60e_wildreceipt_openset
+    In Collection: SDMGR
+    Config: configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt-openset.py
+    Metadata:
+      Training Data: wildreceipt-openset
+    Results:
+      - Task: Key Information Extraction
+        Dataset: wildreceipt
+        Metrics:
+          macro_f1: 0.931
+          micro_f1: 0.940
+          edge_micro_f1: 0.792
+    Weights: https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt-openset/sdmgr_novisual_60e_wildreceipt-openset_20220831_200807-dedf15ec.pth

configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt-openset.py

@@ -0,0 +1,71 @@
+_base_ = [
+    '../_base_/default_runtime.py',
+    '../_base_/datasets/wildreceipt-openset.py',
+    '../_base_/schedules/schedule_adam_60e.py',
+    '_base_sdmgr_novisual.py',
+]
+
+node_num_classes = 4  # 4 classes: bg, key, value and other
+edge_num_classes = 2  # edge connectivity
+key_node_idx = 1
+value_node_idx = 2
+
+model = dict(
+    type='SDMGR',
+    kie_head=dict(
+        num_classes=node_num_classes,
+        postprocessor=dict(
+            link_type='one-to-many',
+            key_node_idx=key_node_idx,
+            value_node_idx=value_node_idx)),
+)
+
+test_pipeline = [
+    dict(
+        type='LoadKIEAnnotations',
+        key_node_idx=key_node_idx,
+        value_node_idx=value_node_idx),  # Keep key->value edges for evaluation
+    dict(type='Resize', scale=(1024, 512), keep_ratio=True),
+    dict(type='PackKIEInputs'),
+]
+
+wildreceipt_openset_train = _base_.wildreceipt_openset_train
+wildreceipt_openset_train.pipeline = _base_.train_pipeline
+wildreceipt_openset_test = _base_.wildreceipt_openset_test
+wildreceipt_openset_test.pipeline = test_pipeline
+
+train_dataloader = dict(
+    batch_size=4,
+    num_workers=1,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=wildreceipt_openset_train)
+val_dataloader = dict(
+    batch_size=1,
+    num_workers=1,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=False),
+    dataset=wildreceipt_openset_test)
+test_dataloader = val_dataloader
+
+val_evaluator = [
+    dict(
+        type='F1Metric',
+        prefix='node',
+        key='labels',
+        mode=['micro', 'macro'],
+        num_classes=node_num_classes,
+        cared_classes=[key_node_idx, value_node_idx]),
+    dict(
+        type='F1Metric',
+        prefix='edge',
+        mode='micro',
+        key='edge_labels',
+        cared_classes=[1],  # Collapse to binary F1 score
+        num_classes=edge_num_classes)
+]
+test_evaluator = val_evaluator
+
+visualizer = dict(
+    type='KIELocalVisualizer', name='visualizer', is_openset=True)
+auto_scale_lr = dict(base_batch_size=4)

configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt.py

@@ -0,0 +1,28 @@
+_base_ = [
+    '../_base_/default_runtime.py',
+    '../_base_/datasets/wildreceipt.py',
+    '../_base_/schedules/schedule_adam_60e.py',
+    '_base_sdmgr_novisual.py',
+]
+
+wildreceipt_train = _base_.wildreceipt_train
+wildreceipt_train.pipeline = _base_.train_pipeline
+wildreceipt_test = _base_.wildreceipt_test
+wildreceipt_test.pipeline = _base_.test_pipeline
+
+train_dataloader = dict(
+    batch_size=4,
+    num_workers=1,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=wildreceipt_train)
+
+val_dataloader = dict(
+    batch_size=1,
+    num_workers=1,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=False),
+    dataset=wildreceipt_test)
+test_dataloader = val_dataloader
+
+auto_scale_lr = dict(base_batch_size=4)

configs/kie/sdmgr/sdmgr_unet16_60e_wildreceipt.py

@@ -0,0 +1,29 @@
+_base_ = [
+    '../_base_/default_runtime.py',
+    '../_base_/datasets/wildreceipt.py',
+    '../_base_/schedules/schedule_adam_60e.py',
+    '_base_sdmgr_unet16.py',
+]
+
+wildreceipt_train = _base_.wildreceipt_train
+wildreceipt_train.pipeline = _base_.train_pipeline
+wildreceipt_test = _base_.wildreceipt_test
+wildreceipt_test.pipeline = _base_.test_pipeline
+
+train_dataloader = dict(
+    batch_size=4,
+    num_workers=4,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=wildreceipt_train)
+
+val_dataloader = dict(
+    batch_size=1,
+    num_workers=1,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=False),
+    dataset=wildreceipt_test)
+
+test_dataloader = val_dataloader
+
+auto_scale_lr = dict(base_batch_size=4)

configs/textdet/_base_/datasets/ctw1500.py

@@ -0,0 +1,15 @@
+ctw1500_textdet_data_root = 'data/ctw1500'
+
+ctw1500_textdet_train = dict(
+    type='OCRDataset',
+    data_root=ctw1500_textdet_data_root,
+    ann_file='textdet_train.json',
+    filter_cfg=dict(filter_empty_gt=True, min_size=32),
+    pipeline=None)
+
+ctw1500_textdet_test = dict(
+    type='OCRDataset',
+    data_root=ctw1500_textdet_data_root,
+    ann_file='textdet_test.json',
+    test_mode=True,
+    pipeline=None)

configs/textdet/_base_/datasets/icdar2015.py

@@ -0,0 +1,15 @@
+icdar2015_textdet_data_root = 'data/icdar2015'
+
+icdar2015_textdet_train = dict(
+    type='OCRDataset',
+    data_root=icdar2015_textdet_data_root,
+    ann_file='textdet_train.json',
+    filter_cfg=dict(filter_empty_gt=True, min_size=32),
+    pipeline=None)
+
+icdar2015_textdet_test = dict(
+    type='OCRDataset',
+    data_root=icdar2015_textdet_data_root,
+    ann_file='textdet_test.json',
+    test_mode=True,
+    pipeline=None)

configs/textdet/_base_/datasets/icdar2017.py

@@ -0,0 +1,17 @@
+icdar2017_textdet_data_root = 'data/det/icdar_2017'
+
+icdar2017_textdet_train = dict(
+    type='OCRDataset',
+    data_root=icdar2017_textdet_data_root,
+    ann_file='instances_training.json',
+    data_prefix=dict(img_path='imgs/'),
+    filter_cfg=dict(filter_empty_gt=True, min_size=32),
+    pipeline=None)
+
+icdar2017_textdet_test = dict(
+    type='OCRDataset',
+    data_root=icdar2017_textdet_data_root,
+    ann_file='instances_test.json',
+    data_prefix=dict(img_path='imgs/'),
+    test_mode=True,
+    pipeline=None)

configs/textdet/_base_/datasets/synthtext.py

@@ -0,0 +1,8 @@
+synthtext_textdet_data_root = 'data/synthtext'
+
+synthtext_textdet_train = dict(
+    type='OCRDataset',
+    data_root=synthtext_textdet_data_root,
+    ann_file='textdet_train.json',
+    filter_cfg=dict(filter_empty_gt=True, min_size=32),
+    pipeline=None)

configs/textdet/_base_/datasets/totaltext.py

@@ -0,0 +1,15 @@
+totaltext_textdet_data_root = 'data/totaltext'
+
+totaltext_textdet_train = dict(
+    type='OCRDataset',
+    data_root=totaltext_textdet_data_root,
+    ann_file='textdet_train.json',
+    filter_cfg=dict(filter_empty_gt=True, min_size=32),
+    pipeline=None)
+
+totaltext_textdet_test = dict(
+    type='OCRDataset',
+    data_root=totaltext_textdet_data_root,
+    ann_file='textdet_test.json',
+    test_mode=True,
+    pipeline=None)

configs/textdet/_base_/datasets/toy_data.py

@@ -0,0 +1,17 @@
+toy_det_data_root = 'tests/data/det_toy_dataset'
+
+toy_det_train = dict(
+    type='OCRDataset',
+    data_root=toy_det_data_root,
+    ann_file='instances_training.json',
+    data_prefix=dict(img_path='imgs/'),
+    filter_cfg=dict(filter_empty_gt=True, min_size=32),
+    pipeline=None)
+
+toy_det_test = dict(
+    type='OCRDataset',
+    data_root=toy_det_data_root,
+    ann_file='instances_test.json',
+    data_prefix=dict(img_path='imgs/'),
+    test_mode=True,
+    pipeline=None)

configs/textdet/_base_/default_runtime.py

@@ -0,0 +1,41 @@
+default_scope = 'mmocr'
+env_cfg = dict(
+    cudnn_benchmark=False,
+    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
+    dist_cfg=dict(backend='nccl'),
+)
+randomness = dict(seed=None)
+
+default_hooks = dict(
+    timer=dict(type='IterTimerHook'),
+    logger=dict(type='LoggerHook', interval=5),
+    param_scheduler=dict(type='ParamSchedulerHook'),
+    checkpoint=dict(type='CheckpointHook', interval=20),
+    sampler_seed=dict(type='DistSamplerSeedHook'),
+    sync_buffer=dict(type='SyncBuffersHook'),
+    visualization=dict(
+        type='VisualizationHook',
+        interval=1,
+        enable=False,
+        show=False,
+        draw_gt=False,
+        draw_pred=False),
+)
+
+# Logging
+log_level = 'INFO'
+log_processor = dict(type='LogProcessor', window_size=10, by_epoch=True)
+
+load_from = None
+resume = False
+
+# Evaluation
+val_evaluator = dict(type='HmeanIOUMetric')
+test_evaluator = val_evaluator
+
+# Visualization
+vis_backends = [dict(type='LocalVisBackend')]
+visualizer = dict(
+    type='TextDetLocalVisualizer',
+    name='visualizer',
+    vis_backends=vis_backends)

configs/textdet/_base_/pretrain_runtime.py

@@ -0,0 +1,14 @@
+_base_ = 'default_runtime.py'
+
+default_hooks = dict(
+    logger=dict(type='LoggerHook', interval=1000),
+    checkpoint=dict(
+        type='CheckpointHook',
+        interval=10000,
+        by_epoch=False,
+        max_keep_ckpts=1),
+)
+
+# Evaluation
+val_evaluator = None
+test_evaluator = None

configs/textdet/_base_/schedules/schedule_adam_600e.py

@@ -0,0 +1,9 @@
+# optimizer
+optim_wrapper = dict(type='OptimWrapper', optimizer=dict(type='Adam', lr=1e-3))
+train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=600, val_interval=20)
+val_cfg = dict(type='ValLoop')
+test_cfg = dict(type='TestLoop')
+# learning rate
+param_scheduler = [
+    dict(type='PolyLR', power=0.9, end=600),
+]

configs/textdet/_base_/schedules/schedule_sgd_100k.py

@@ -0,0 +1,12 @@
+# optimizer
+optim_wrapper = dict(
+    type='OptimWrapper',
+    optimizer=dict(type='SGD', lr=0.007, momentum=0.9, weight_decay=0.0001))
+
+train_cfg = dict(type='IterBasedTrainLoop', max_iters=100000)
+test_cfg = None
+val_cfg = None
+# learning policy
+param_scheduler = [
+    dict(type='PolyLR', power=0.9, eta_min=1e-7, by_epoch=False, end=100000),
+]

configs/textdet/_base_/schedules/schedule_sgd_1200e.py

@@ -0,0 +1,11 @@
+# optimizer
+optim_wrapper = dict(
+    type='OptimWrapper',
+    optimizer=dict(type='SGD', lr=0.007, momentum=0.9, weight_decay=0.0001))
+train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=1200, val_interval=20)
+val_cfg = dict(type='ValLoop')
+test_cfg = dict(type='TestLoop')
+# learning policy
+param_scheduler = [
+    dict(type='PolyLR', power=0.9, eta_min=1e-7, end=1200),
+]

configs/textdet/_base_/schedules/schedule_sgd_base.py

@@ -0,0 +1,15 @@
+# Note: This schedule config serves as a base config for other schedules.
+# Users would have to at least fill in "max_epochs" and "val_interval"
+# in order to use this config in their experiments.
+
+# optimizer
+optim_wrapper = dict(
+    type='OptimWrapper',
+    optimizer=dict(type='SGD', lr=0.007, momentum=0.9, weight_decay=0.0001))
+train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=None, val_interval=20)
+val_cfg = dict(type='ValLoop')
+test_cfg = dict(type='TestLoop')
+# learning policy
+param_scheduler = [
+    dict(type='ConstantLR', factor=1.0),
+]

configs/textdet/dbnet/README.md

@@ -0,0 +1,47 @@
+# DBNet
+
+> [Real-time Scene Text Detection with Differentiable Binarization](https://arxiv.org/abs/1911.08947)
+
+<!-- [ALGORITHM] -->
+
+## Abstract
+
+Recently, segmentation-based methods are quite popular in scene text detection, as the segmentation results can more accurately describe scene text of various shapes such as curve text. However, the post-processing of binarization is essential for segmentation-based detection, which converts probability maps produced by a segmentation method into bounding boxes/regions of text. In this paper, we propose a module named Differentiable Binarization (DB), which can perform the binarization process in a segmentation network. Optimized along with a DB module, a segmentation network can adaptively set the thresholds for binarization, which not only simplifies the post-processing but also enhances the performance of text detection. Based on a simple segmentation network, we validate the performance improvements of DB on five benchmark datasets, which consistently achieves state-of-the-art results, in terms of both detection accuracy and speed. In particular, with a light-weight backbone, the performance improvements by DB are significant so that we can look for an ideal tradeoff between detection accuracy and efficiency. Specifically, with a backbone of ResNet-18, our detector achieves an F-measure of 82.8, running at 62 FPS, on the MSRA-TD500 dataset.
+
+<div align=center>
+<img src="https://user-images.githubusercontent.com/22607038/142791306-0da6db2a-20a6-4a68-b228-64ff275f67b3.png"/>
+</div>
+
+## Results and models
+
+### SynthText
+
+|                                  Method                                   | Backbone | Training set | #iters  |                                               Download                                               |
+| :-----------------------------------------------------------------------: | :------: | :----------: | :-----: | :--------------------------------------------------------------------------------------------------: |
+| [DBNet_r18](/configs/textdet/dbnet/dbnet_resnet18_fpnc_100k_synthtext.py) | ResNet18 |  SynthText   | 100,000 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_100k_synthtext/dbnet_resnet18_fpnc_100k_synthtext-2e9bf392.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_100k_synthtext/20221214_150351.log) |
+
+### ICDAR2015
+
+|             Method             |             Backbone             |             Pretrained Model             |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |             Download             |
+| :----------------------------: | :------------------------------: | :--------------------------------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------: |
+| [DBNet_r18](/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py) |             ResNet18             |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |    736    |  0.8853   | 0.7583 | 0.8169 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/20220825_221614.log) |
+| [DBNet_r50](/configs/textdet/dbnet/dbnet_resnet50_1200e_icdar2015.py) |             ResNet50             |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.8744   | 0.8276 | 0.8504 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50_1200e_icdar2015/dbnet_resnet50_1200e_icdar2015_20221102_115917-54f50589.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50_1200e_icdar2015/20221102_115917.log) |
+| [DBNet_r50dcn](/configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py) |           ResNet50-DCN           | [Synthtext](https://download.openmmlab.com/mmocr/textdet/dbnet/tmp_1.0_pretrain/dbnet_r50dcnv2_fpnc_sbn_2e_synthtext_20210325-ed322016.pth) | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.8784   | 0.8315 | 0.8543 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015_20220828_124917-452c443c.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/20220828_124917.log) |
+| [DBNet_r50-oclip](/configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.9052   | 0.8272 | 0.8644 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015/dbnet_resnet50-oclip_1200e_icdar2015_20221102_115917-bde8c87a.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015/20221102_115917.log) |
+
+### Total Text
+
+|                         Method                         | Backbone | Pretrained Model |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |                         Download                         |
+| :----------------------------------------------------: | :------: | :--------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------------------------------: |
+| [DBNet_r18](/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext.py) | ResNet18 |        -         | Totaltext Train | Totaltext Test |  1200   |    736    |  0.8640   | 0.7770 | 0.8182 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext/dbnet_resnet18_fpnc_1200e_totaltext-3ed3233c.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext/20221219_201038.log) |
+
+## Citation
+
+```bibtex
+@article{Liao_Wan_Yao_Chen_Bai_2020,
+    title={Real-Time Scene Text Detection with Differentiable Binarization},
+    journal={Proceedings of the AAAI Conference on Artificial Intelligence},
+    author={Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang},
+    year={2020},
+    pages={11474-11481}}
+```

configs/textdet/dbnet/_base_dbnet_resnet18_fpnc.py

@@ -0,0 +1,64 @@
+model = dict(
+    type='DBNet',
+    backbone=dict(
+        type='mmdet.ResNet',
+        depth=18,
+        num_stages=4,
+        out_indices=(0, 1, 2, 3),
+        frozen_stages=-1,
+        norm_cfg=dict(type='BN', requires_grad=True),
+        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet18'),
+        norm_eval=False,
+        style='caffe'),
+    neck=dict(
+        type='FPNC', in_channels=[64, 128, 256, 512], lateral_channels=256),
+    det_head=dict(
+        type='DBHead',
+        in_channels=256,
+        module_loss=dict(type='DBModuleLoss'),
+        postprocessor=dict(type='DBPostprocessor', text_repr_type='quad')),
+    data_preprocessor=dict(
+        type='TextDetDataPreprocessor',
+        mean=[123.675, 116.28, 103.53],
+        std=[58.395, 57.12, 57.375],
+        bgr_to_rgb=True,
+        pad_size_divisor=32))
+
+train_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(
+        type='LoadOCRAnnotations',
+        with_polygon=True,
+        with_bbox=True,
+        with_label=True,
+    ),
+    dict(
+        type='TorchVisionWrapper',
+        op='ColorJitter',
+        brightness=32.0 / 255,
+        saturation=0.5),
+    dict(
+        type='ImgAugWrapper',
+        args=[['Fliplr', 0.5],
+              dict(cls='Affine', rotate=[-10, 10]), ['Resize', [0.5, 3.0]]]),
+    dict(type='RandomCrop', min_side_ratio=0.1),
+    dict(type='Resize', scale=(640, 640), keep_ratio=True),
+    dict(type='Pad', size=(640, 640)),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape'))
+]
+
+test_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(type='Resize', scale=(1333, 736), keep_ratio=True),
+    dict(
+        type='LoadOCRAnnotations',
+        with_polygon=True,
+        with_bbox=True,
+        with_label=True,
+    ),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor'))
+]

configs/textdet/dbnet/_base_dbnet_resnet50-dcnv2_fpnc.py

@@ -0,0 +1,66 @@
+model = dict(
+    type='DBNet',
+    backbone=dict(
+        type='mmdet.ResNet',
+        depth=50,
+        num_stages=4,
+        out_indices=(0, 1, 2, 3),
+        frozen_stages=-1,
+        norm_cfg=dict(type='BN', requires_grad=True),
+        norm_eval=False,
+        style='pytorch',
+        dcn=dict(type='DCNv2', deform_groups=1, fallback_on_stride=False),
+        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50'),
+        stage_with_dcn=(False, True, True, True)),
+    neck=dict(
+        type='FPNC', in_channels=[256, 512, 1024, 2048], lateral_channels=256),
+    det_head=dict(
+        type='DBHead',
+        in_channels=256,
+        module_loss=dict(type='DBModuleLoss'),
+        postprocessor=dict(type='DBPostprocessor', text_repr_type='quad')),
+    data_preprocessor=dict(
+        type='TextDetDataPreprocessor',
+        mean=[123.675, 116.28, 103.53],
+        std=[58.395, 57.12, 57.375],
+        bgr_to_rgb=True,
+        pad_size_divisor=32))
+
+train_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(
+        type='LoadOCRAnnotations',
+        with_bbox=True,
+        with_polygon=True,
+        with_label=True,
+    ),
+    dict(
+        type='TorchVisionWrapper',
+        op='ColorJitter',
+        brightness=32.0 / 255,
+        saturation=0.5),
+    dict(
+        type='ImgAugWrapper',
+        args=[['Fliplr', 0.5],
+              dict(cls='Affine', rotate=[-10, 10]), ['Resize', [0.5, 3.0]]]),
+    dict(type='RandomCrop', min_side_ratio=0.1),
+    dict(type='Resize', scale=(640, 640), keep_ratio=True),
+    dict(type='Pad', size=(640, 640)),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape'))
+]
+
+test_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(type='Resize', scale=(4068, 1024), keep_ratio=True),
+    dict(
+        type='LoadOCRAnnotations',
+        with_polygon=True,
+        with_bbox=True,
+        with_label=True,
+    ),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor'))
+]

configs/textdet/dbnet/dbnet_resnet18_fpnc_100k_synthtext.py

@@ -0,0 +1,45 @@
+_base_ = [
+    '_base_dbnet_resnet18_fpnc.py',
+    '../_base_/datasets/synthtext.py',
+    '../_base_/pretrain_runtime.py',
+    '../_base_/schedules/schedule_sgd_100k.py',
+]
+
+train_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(
+        type='LoadOCRAnnotations',
+        with_polygon=True,
+        with_bbox=True,
+        with_label=True,
+    ),
+    dict(type='FixInvalidPolygon'),
+    dict(
+        type='TorchVisionWrapper',
+        op='ColorJitter',
+        brightness=32.0 / 255,
+        saturation=0.5),
+    dict(
+        type='ImgAugWrapper',
+        args=[['Fliplr', 0.5],
+              dict(cls='Affine', rotate=[-10, 10]), ['Resize', [0.5, 3.0]]]),
+    dict(type='RandomCrop', min_side_ratio=0.1),
+    dict(type='Resize', scale=(640, 640), keep_ratio=True),
+    dict(type='Pad', size=(640, 640)),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape'))
+]
+
+# dataset settings
+synthtext_textdet_train = _base_.synthtext_textdet_train
+synthtext_textdet_train.pipeline = train_pipeline
+
+train_dataloader = dict(
+    batch_size=16,
+    num_workers=8,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=synthtext_textdet_train)
+
+auto_scale_lr = dict(base_batch_size=16)

configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py

@@ -0,0 +1,30 @@
+_base_ = [
+    '_base_dbnet_resnet18_fpnc.py',
+    '../_base_/datasets/icdar2015.py',
+    '../_base_/default_runtime.py',
+    '../_base_/schedules/schedule_sgd_1200e.py',
+]
+
+# dataset settings
+icdar2015_textdet_train = _base_.icdar2015_textdet_train
+icdar2015_textdet_train.pipeline = _base_.train_pipeline
+icdar2015_textdet_test = _base_.icdar2015_textdet_test
+icdar2015_textdet_test.pipeline = _base_.test_pipeline
+
+train_dataloader = dict(
+    batch_size=16,
+    num_workers=8,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=icdar2015_textdet_train)
+
+val_dataloader = dict(
+    batch_size=1,
+    num_workers=4,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=False),
+    dataset=icdar2015_textdet_test)
+
+test_dataloader = val_dataloader
+
+auto_scale_lr = dict(base_batch_size=16)

configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext.py

@@ -0,0 +1,73 @@
+_base_ = [
+    '_base_dbnet_resnet18_fpnc.py',
+    '../_base_/datasets/totaltext.py',
+    '../_base_/default_runtime.py',
+    '../_base_/schedules/schedule_sgd_1200e.py',
+]
+
+train_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(
+        type='LoadOCRAnnotations',
+        with_polygon=True,
+        with_bbox=True,
+        with_label=True,
+    ),
+    dict(type='FixInvalidPolygon', min_poly_points=4),
+    dict(
+        type='TorchVisionWrapper',
+        op='ColorJitter',
+        brightness=32.0 / 255,
+        saturation=0.5),
+    dict(
+        type='ImgAugWrapper',
+        args=[['Fliplr', 0.5],
+              dict(cls='Affine', rotate=[-10, 10]), ['Resize', [0.5, 3.0]]]),
+    dict(type='RandomCrop', min_side_ratio=0.1),
+    dict(type='Resize', scale=(640, 640), keep_ratio=True),
+    dict(type='Pad', size=(640, 640)),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape'))
+]
+
+test_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(type='Resize', scale=(1333, 736), keep_ratio=True),
+    dict(
+        type='LoadOCRAnnotations',
+        with_polygon=True,
+        with_bbox=True,
+        with_label=True,
+    ),
+    dict(type='FixInvalidPolygon', min_poly_points=4),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor'))
+]
+
+# dataset settings
+totaltext_textdet_train = _base_.totaltext_textdet_train
+totaltext_textdet_test = _base_.totaltext_textdet_test
+totaltext_textdet_train.pipeline = train_pipeline
+totaltext_textdet_test.pipeline = test_pipeline
+
+train_dataloader = dict(
+    batch_size=16,
+    num_workers=16,
+    pin_memory=True,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=totaltext_textdet_train)
+
+val_dataloader = dict(
+    batch_size=1,
+    num_workers=1,
+    pin_memory=True,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=False),
+    dataset=totaltext_textdet_test)
+
+test_dataloader = val_dataloader
+
+auto_scale_lr = dict(base_batch_size=16)

configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_100k_synthtext.py

@@ -0,0 +1,30 @@
+_base_ = [
+    '_base_dbnet_resnet50-dcnv2_fpnc.py',
+    '../_base_/default_runtime.py',
+    '../_base_/datasets/synthtext.py',
+    '../_base_/schedules/schedule_sgd_100k.py',
+]
+
+# dataset settings
+synthtext_textdet_train = _base_.synthtext_textdet_train
+synthtext_textdet_train.pipeline = _base_.train_pipeline
+synthtext_textdet_test = _base_.synthtext_textdet_test
+synthtext_textdet_test.pipeline = _base_.test_pipeline
+
+train_dataloader = dict(
+    batch_size=16,
+    num_workers=8,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=synthtext_textdet_train)
+
+val_dataloader = dict(
+    batch_size=1,
+    num_workers=4,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=False),
+    dataset=synthtext_textdet_test)
+
+test_dataloader = val_dataloader
+
+auto_scale_lr = dict(base_batch_size=16)

configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py

@@ -0,0 +1,33 @@
+_base_ = [
+    '_base_dbnet_resnet50-dcnv2_fpnc.py',
+    '../_base_/datasets/icdar2015.py',
+    '../_base_/default_runtime.py',
+    '../_base_/schedules/schedule_sgd_1200e.py',
+]
+
+# TODO: Replace the link
+load_from = 'https://download.openmmlab.com/mmocr/textdet/dbnet/tmp_1.0_pretrain/dbnet_r50dcnv2_fpnc_sbn_2e_synthtext_20210325-ed322016.pth'  # noqa
+
+# dataset settings
+icdar2015_textdet_train = _base_.icdar2015_textdet_train
+icdar2015_textdet_train.pipeline = _base_.train_pipeline
+icdar2015_textdet_test = _base_.icdar2015_textdet_test
+icdar2015_textdet_test.pipeline = _base_.test_pipeline
+
+train_dataloader = dict(
+    batch_size=16,
+    num_workers=8,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=icdar2015_textdet_train)
+
+val_dataloader = dict(
+    batch_size=1,
+    num_workers=4,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=False),
+    dataset=icdar2015_textdet_test)
+
+test_dataloader = val_dataloader
+
+auto_scale_lr = dict(base_batch_size=16)

configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015.py

@@ -0,0 +1,20 @@
+_base_ = [
+    'dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py',
+]
+
+load_from = None
+
+_base_.model.backbone = dict(
+    type='CLIPResNet',
+    init_cfg=dict(
+        type='Pretrained',
+        checkpoint='https://download.openmmlab.com/'
+        'mmocr/backbone/resnet50-oclip-7ba0c533.pth'))
+
+_base_.train_dataloader.num_workers = 24
+_base_.optim_wrapper.optimizer.lr = 0.002
+
+param_scheduler = [
+    dict(type='LinearLR', end=100, start_factor=0.001),
+    dict(type='PolyLR', power=0.9, eta_min=1e-7, begin=100, end=1200),
+]

configs/textdet/dbnet/dbnet_resnet50_1200e_icdar2015.py

@@ -0,0 +1,24 @@
+_base_ = [
+    'dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py',
+]
+
+load_from = None
+
+_base_.model.backbone = dict(
+    type='mmdet.ResNet',
+    depth=50,
+    num_stages=4,
+    out_indices=(0, 1, 2, 3),
+    frozen_stages=-1,
+    norm_cfg=dict(type='BN', requires_grad=True),
+    norm_eval=True,
+    style='pytorch',
+    init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50'))
+
+_base_.train_dataloader.num_workers = 24
+_base_.optim_wrapper.optimizer.lr = 0.002
+
+param_scheduler = [
+    dict(type='LinearLR', end=100, start_factor=0.001),
+    dict(type='PolyLR', power=0.9, eta_min=1e-7, begin=100, end=1200),
+]

configs/textdet/dbnet/metafile.yml

@@ -0,0 +1,80 @@
+Collections:
+- Name: DBNet
+  Metadata:
+    Training Data: ICDAR2015
+    Training Techniques:
+      - SGD with Momentum
+      - Weight Decay
+    Training Resources: 1x NVIDIA A100-SXM4-80GB
+    Architecture:
+      - ResNet
+      - FPNC
+  Paper:
+    URL: https://arxiv.org/pdf/1911.08947.pdf
+    Title: 'Real-time Scene Text Detection with Differentiable Binarization'
+  README: configs/textdet/dbnet/README.md
+
+Models:
+  - Name: dbnet_resnet18_fpnc_1200e_icdar2015
+    Alias: DB_r18
+    In Collection: DBNet
+    Config: configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou: 0.8169
+    Weights: https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth
+
+  - Name: dbnet_resnet50_fpnc_1200e_icdar2015
+    In Collection: DBNet
+    Config: configs/textdet/dbnet/dbnet_resnet50_fpnc_1200e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou: 0.8504
+    Weights: https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50_1200e_icdar2015/dbnet_resnet50_1200e_icdar2015_20221102_115917-54f50589.pth
+
+  - Name: dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015
+    In Collection: DBNet
+    Config: configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou: 0.8543
+    Weights: https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015_20220828_124917-452c443c.pth
+
+  - Name: dbnet_resnet50-oclip_fpnc_1200e_icdar2015
+    In Collection: DBNet
+    Alias:
+      - DB_r50
+      - DBNet
+    Config: configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou: 0.8644
+    Weights: https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015/dbnet_resnet50-oclip_1200e_icdar2015_20221102_115917-bde8c87a.pth
+
+  - Name: dbnet_resnet18_fpnc_1200e_totaltext
+    In Collection: DBNet
+    Config: configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext.py
+    Metadata:
+      Training Data: Totaltext
+    Results:
+      - Task: Text Detection
+        Dataset: Totaltext
+        Metrics:
+          hmean-iou: 0.8182
+    Weights: https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext/dbnet_resnet18_fpnc_1200e_totaltext-3ed3233c.pth

configs/textdet/dbnetpp/README.md

@@ -0,0 +1,41 @@
+# DBNetpp
+
+> [Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion](https://arxiv.org/abs/2202.10304)
+
+<!-- [ALGORITHM] -->
+
+## Abstract
+
+Recently, segmentation-based scene text detection methods have drawn extensive attention in the scene text detection field, because of their superiority in detecting the text instances of arbitrary shapes and extreme aspect ratios, profiting from the pixel-level descriptions. However, the vast majority of the existing segmentation-based approaches are limited to their complex post-processing algorithms and the scale robustness of their segmentation models, where the post-processing algorithms are not only isolated to the model optimization but also time-consuming and the scale robustness is usually strengthened by fusing multi-scale feature maps directly. In this paper, we propose a Differentiable Binarization (DB) module that integrates the binarization process, one of the most important steps in the post-processing procedure, into a segmentation network. Optimized along with the proposed DB module, the segmentation network can produce more accurate results, which enhances the accuracy of text detection with a simple pipeline. Furthermore, an efficient Adaptive Scale Fusion (ASF) module is proposed to improve the scale robustness by fusing features of different scales adaptively. By incorporating the proposed DB and ASF with the segmentation network, our proposed scene text detector consistently achieves state-of-the-art results, in terms of both detection accuracy and speed, on five standard benchmarks.
+
+<div align=center>
+<img src="https://user-images.githubusercontent.com/45810070/166850828-f1e48c25-4a0f-429d-ae54-6997ed25c062.png"/>
+</div>
+
+## Results and models
+
+### SynthText
+
+|                                       Method                                       |    BackBone    | Training set | #iters  |                                       Download                                        |
+| :--------------------------------------------------------------------------------: | :------------: | :----------: | :-----: | :-----------------------------------------------------------------------------------: |
+| [DBNetpp_r50dcn](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext.py) | ResNet50-dcnv2 |  SynthText   | 100,000 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext-00f0a80b.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext/20221215_013531.log) |
+
+### ICDAR2015
+
+|             Method             |             BackBone             |             Pretrained Model             |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |             Download             |
+| :----------------------------: | :------------------------------: | :--------------------------------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------: |
+| [DBNetpp_r50](/configs/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015.py) |             ResNet50             |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.9079   | 0.8209 | 0.8622 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015/dbnetpp_resnet50_fpnc_1200e_icdar2015_20221025_185550-013730aa.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015/20221025_185550.log) |
+| [DBNetpp_r50dcn](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py) |          ResNet50-dcnv2          | [Synthtext](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext.py) ([model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/tmp_1.0_pretrain/dbnetpp_r50dcnv2_fpnc_100k_iter_synthtext-20220502-352fec8a.pth)) | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.9116   | 0.8291 | 0.8684 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015_20220829_230108-f289bd20.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/20220829_230108.log) |
+| [DBNetpp_r50-oclip](/configs/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.9174   | 0.8609 | 0.8882 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015_20221101_124139-4ecb39ac.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/20221101_124139.log) |
+
+## Citation
+
+```bibtex
+@article{liao2022real,
+    title={Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion},
+    author={Liao, Minghui and Zou, Zhisheng and Wan, Zhaoyi and Yao, Cong and Bai, Xiang},
+    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
+    year={2022},
+    publisher={IEEE}
+}
+```

configs/textdet/dbnetpp/_base_dbnetpp_resnet50-dcnv2_fpnc.py

@@ -0,0 +1,72 @@
+model = dict(
+    type='DBNet',
+    backbone=dict(
+        type='mmdet.ResNet',
+        depth=50,
+        num_stages=4,
+        out_indices=(0, 1, 2, 3),
+        frozen_stages=-1,
+        norm_cfg=dict(type='BN', requires_grad=True),
+        norm_eval=False,
+        style='pytorch',
+        dcn=dict(type='DCNv2', deform_groups=1, fallback_on_stride=False),
+        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50'),
+        stage_with_dcn=(False, True, True, True)),
+    neck=dict(
+        type='FPNC',
+        in_channels=[256, 512, 1024, 2048],
+        lateral_channels=256,
+        asf_cfg=dict(attention_type='ScaleChannelSpatial')),
+    det_head=dict(
+        type='DBHead',
+        in_channels=256,
+        module_loss=dict(type='DBModuleLoss'),
+        postprocessor=dict(
+            type='DBPostprocessor', text_repr_type='quad',
+            epsilon_ratio=0.002)),
+    data_preprocessor=dict(
+        type='TextDetDataPreprocessor',
+        mean=[123.675, 116.28, 103.53],
+        std=[58.395, 57.12, 57.375],
+        bgr_to_rgb=True,
+        pad_size_divisor=32))
+
+train_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(
+        type='LoadOCRAnnotations',
+        with_bbox=True,
+        with_polygon=True,
+        with_label=True,
+    ),
+    dict(
+        type='TorchVisionWrapper',
+        op='ColorJitter',
+        brightness=32.0 / 255,
+        saturation=0.5),
+    dict(
+        type='ImgAugWrapper',
+        args=[['Fliplr', 0.5],
+              dict(cls='Affine', rotate=[-10, 10]), ['Resize', [0.5, 3.0]]]),
+    dict(type='RandomCrop', min_side_ratio=0.1),
+    dict(type='Resize', scale=(640, 640), keep_ratio=True),
+    dict(type='Pad', size=(640, 640)),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape'))
+]
+
+test_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(type='Resize', scale=(4068, 1024), keep_ratio=True),
+    dict(
+        type='LoadOCRAnnotations',
+        with_polygon=True,
+        with_bbox=True,
+        with_label=True,
+    ),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor',
+                   'instances'))
+]

configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext.py

@@ -0,0 +1,44 @@
+_base_ = [
+    '_base_dbnetpp_resnet50-dcnv2_fpnc.py',
+    '../_base_/pretrain_runtime.py',
+    '../_base_/datasets/synthtext.py',
+    '../_base_/schedules/schedule_sgd_100k.py',
+]
+
+train_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(
+        type='LoadOCRAnnotations',
+        with_bbox=True,
+        with_polygon=True,
+        with_label=True,
+    ),
+    dict(type='FixInvalidPolygon'),
+    dict(
+        type='TorchVisionWrapper',
+        op='ColorJitter',
+        brightness=32.0 / 255,
+        saturation=0.5),
+    dict(
+        type='ImgAugWrapper',
+        args=[['Fliplr', 0.5],
+              dict(cls='Affine', rotate=[-10, 10]), ['Resize', [0.5, 3.0]]]),
+    dict(type='RandomCrop', min_side_ratio=0.1),
+    dict(type='Resize', scale=(640, 640), keep_ratio=True),
+    dict(type='Pad', size=(640, 640)),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape'))
+]
+
+synthtext_textdet_train = _base_.synthtext_textdet_train
+synthtext_textdet_train.pipeline = train_pipeline
+
+train_dataloader = dict(
+    batch_size=16,
+    num_workers=8,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=synthtext_textdet_train)
+
+auto_scale_lr = dict(base_batch_size=16)

configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py

@@ -0,0 +1,36 @@
+_base_ = [
+    '_base_dbnetpp_resnet50-dcnv2_fpnc.py',
+    '../_base_/default_runtime.py',
+    '../_base_/datasets/icdar2015.py',
+    '../_base_/schedules/schedule_sgd_1200e.py',
+]
+
+load_from = 'https://download.openmmlab.com/mmocr/textdet/dbnetpp/tmp_1.0_pretrain/dbnetpp_r50dcnv2_fpnc_100k_iter_synthtext-20220502-352fec8a.pth'  # noqa
+
+# dataset settings
+train_list = [_base_.icdar2015_textdet_train]
+test_list = [_base_.icdar2015_textdet_test]
+
+train_dataloader = dict(
+    batch_size=16,
+    num_workers=8,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=dict(
+        type='ConcatDataset',
+        datasets=train_list,
+        pipeline=_base_.train_pipeline))
+
+val_dataloader = dict(
+    batch_size=16,
+    num_workers=8,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=False),
+    dataset=dict(
+        type='ConcatDataset',
+        datasets=test_list,
+        pipeline=_base_.test_pipeline))
+
+test_dataloader = val_dataloader
+
+auto_scale_lr = dict(base_batch_size=16)

configs/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015.py

@@ -0,0 +1,20 @@
+_base_ = [
+    'dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py',
+]
+
+load_from = None
+
+_base_.model.backbone = dict(
+    type='CLIPResNet',
+    init_cfg=dict(
+        type='Pretrained',
+        checkpoint='https://download.openmmlab.com/'
+        'mmocr/backbone/resnet50-oclip-7ba0c533.pth'))
+
+_base_.train_dataloader.num_workers = 24
+_base_.optim_wrapper.optimizer.lr = 0.002
+
+param_scheduler = [
+    dict(type='LinearLR', end=200, start_factor=0.001),
+    dict(type='PolyLR', power=0.9, eta_min=1e-7, begin=200, end=1200),
+]

configs/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015.py

@@ -0,0 +1,24 @@
+_base_ = [
+    'dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py',
+]
+
+load_from = None
+
+_base_.model.backbone = dict(
+    type='mmdet.ResNet',
+    depth=50,
+    num_stages=4,
+    out_indices=(0, 1, 2, 3),
+    frozen_stages=-1,
+    norm_cfg=dict(type='BN', requires_grad=True),
+    norm_eval=True,
+    style='pytorch',
+    init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50'))
+
+_base_.train_dataloader.num_workers = 24
+_base_.optim_wrapper.optimizer.lr = 0.003
+
+param_scheduler = [
+    dict(type='LinearLR', end=200, start_factor=0.001),
+    dict(type='PolyLR', power=0.9, eta_min=1e-7, begin=200, end=1200),
+]

configs/textdet/dbnetpp/metafile.yml

@@ -0,0 +1,56 @@
+Collections:
+- Name: DBNetpp
+  Metadata:
+    Training Data: ICDAR2015
+    Training Techniques:
+      - SGD with Momentum
+      - Weight Decay
+    Training Resources: 1x NVIDIA A100-SXM4-80GB
+    Architecture:
+      - ResNet
+      - FPNC
+  Paper:
+    URL: https://arxiv.org/abs/2202.10304
+    Title: 'Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion'
+  README: configs/textdet/dbnetpp/README.md
+
+Models:
+  - Name: dbnetpp_resnet50_fpnc_1200e_icdar2015
+    In Collection: DBNetpp
+    Alias:
+      - DBPP_r50
+    Config: configs/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou: 0.8622
+    Weights: https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015/dbnetpp_resnet50_fpnc_1200e_icdar2015_20221025_185550-013730aa.pth
+
+  - Name: dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015
+    In Collection: DBNetpp
+    Config: configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou: 0.8684
+    Weights: https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015_20220829_230108-f289bd20.pth
+
+  - Name: dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015
+    Alias:
+      - DBNetpp
+    In Collection: DBNetpp
+    Config: configs/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou: 0.8882
+    Weights: https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015_20221101_124139-4ecb39ac.pth

configs/textdet/drrg/README.md

@@ -0,0 +1,34 @@
+# DRRG
+
+> [Deep relational reasoning graph network for arbitrary shape text detection](https://arxiv.org/abs/2003.07493)
+
+<!-- [ALGORITHM] -->
+
+## Abstract
+
+Arbitrary shape text detection is a challenging task due to the high variety and complexity of scenes texts. In this paper, we propose a novel unified relational reasoning graph network for arbitrary shape text detection. In our method, an innovative local graph bridges a text proposal model via Convolutional Neural Network (CNN) and a deep relational reasoning network via Graph Convolutional Network (GCN), making our network end-to-end trainable. To be concrete, every text instance will be divided into a series of small rectangular components, and the geometry attributes (e.g., height, width, and orientation) of the small components will be estimated by our text proposal model. Given the geometry attributes, the local graph construction model can roughly establish linkages between different text components. For further reasoning and deducing the likelihood of linkages between the component and its neighbors, we adopt a graph-based network to perform deep relational reasoning on local graphs. Experiments on public available datasets demonstrate the state-of-the-art performance of our method.
+
+<div align=center>
+<img src="https://user-images.githubusercontent.com/22607038/142791777-f282300a-fb83-4b5a-a7d4-29f308949f11.png"/>
+</div>
+
+## Results and models
+
+### CTW1500
+
+|                 Method                  |                 BackBone                  | Pretrained Model | Training set  |   Test set   | #epochs | Test size | Precision | Recall | Hmean  |                  Download                  |
+| :-------------------------------------: | :---------------------------------------: | :--------------: | :-----------: | :----------: | :-----: | :-------: | :-------: | :----: | :----: | :----------------------------------------: |
+| [DRRG](/configs/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500.py) |                 ResNet50                  |        -         | CTW1500 Train | CTW1500 Test |  1200   |    640    |  0.8775   | 0.8179 | 0.8467 | [model](https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/drrg_resnet50_fpn-unet_1200e_ctw1500_20220827_105233-d5c702dd.pth) \\ [log](https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/20220827_105233.log) |
+| [DRRG_r50-oclip](/configs/textdet/drrg/drrg_resnet50-oclip_fpn-unet_1200e_ctw1500.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |        -         | CTW1500 Train | CTW1500 Test |  1200   |           |           |        |        |          [model](<>) \\ [log](<>)          |
+
+## Citation
+
+```bibtex
+@article{zhang2020drrg,
+  title={Deep relational reasoning graph network for arbitrary shape text detection},
+  author={Zhang, Shi-Xue and Zhu, Xiaobin and Hou, Jie-Bo and Liu, Chang and Yang, Chun and Wang, Hongfa and Yin, Xu-Cheng},
+  booktitle={CVPR},
+  pages={9699-9708},
+  year={2020}
+}
+```

configs/textdet/drrg/_base_drrg_resnet50_fpn-unet.py

@@ -0,0 +1,92 @@
+model = dict(
+    type='DRRG',
+    backbone=dict(
+        type='mmdet.ResNet',
+        depth=50,
+        num_stages=4,
+        out_indices=(0, 1, 2, 3),
+        frozen_stages=-1,
+        norm_cfg=dict(type='BN', requires_grad=True),
+        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50'),
+        norm_eval=True,
+        style='caffe'),
+    neck=dict(
+        type='FPN_UNet', in_channels=[256, 512, 1024, 2048], out_channels=32),
+    det_head=dict(
+        type='DRRGHead',
+        in_channels=32,
+        text_region_thr=0.3,
+        center_region_thr=0.4,
+        module_loss=dict(type='DRRGModuleLoss'),
+        postprocessor=dict(type='DRRGPostprocessor', link_thr=0.80)),
+    data_preprocessor=dict(
+        type='TextDetDataPreprocessor',
+        mean=[123.675, 116.28, 103.53],
+        std=[58.395, 57.12, 57.375],
+        bgr_to_rgb=True,
+        pad_size_divisor=32))
+
+train_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(
+        type='LoadOCRAnnotations',
+        with_bbox=True,
+        with_polygon=True,
+        with_label=True),
+    dict(
+        type='TorchVisionWrapper',
+        op='ColorJitter',
+        brightness=32.0 / 255,
+        saturation=0.5),
+    dict(
+        type='RandomResize',
+        scale=(800, 800),
+        ratio_range=(0.75, 2.5),
+        keep_ratio=True),
+    dict(
+        type='TextDetRandomCropFlip',
+        crop_ratio=0.5,
+        iter_num=1,
+        min_area_ratio=0.2),
+    dict(
+        type='RandomApply',
+        transforms=[dict(type='RandomCrop', min_side_ratio=0.3)],
+        prob=0.8),
+    dict(
+        type='RandomApply',
+        transforms=[
+            dict(
+                type='RandomRotate',
+                max_angle=60,
+                use_canvas=True,
+                pad_with_fixed_color=False)
+        ],
+        prob=0.5),
+    dict(
+        type='RandomChoice',
+        transforms=[[
+            dict(type='Resize', scale=800, keep_ratio=True),
+            dict(type='SourceImagePad', target_scale=800)
+        ],
+                    dict(type='Resize', scale=800, keep_ratio=False)],
+        prob=[0.4, 0.6]),
+    dict(type='RandomFlip', prob=0.5, direction='horizontal'),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape'))
+]
+
+test_pipeline = [
+    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
+    dict(type='Resize', scale=(1024, 640), keep_ratio=True),
+    # add loading annotation after ``Resize`` because ground truth
+    # does not need to do resize data transform
+    dict(
+        type='LoadOCRAnnotations',
+        with_polygon=True,
+        with_bbox=True,
+        with_label=True),
+    dict(
+        type='PackTextDetInputs',
+        meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor'))
+]

configs/textdet/drrg/drrg_resnet50-oclip_fpn-unet_1200e_ctw1500.py

@@ -0,0 +1,17 @@
+_base_ = [
+    'drrg_resnet50_fpn-unet_1200e_ctw1500.py',
+]
+
+load_from = None
+
+_base_.model.backbone = dict(
+    type='CLIPResNet',
+    init_cfg=dict(
+        type='Pretrained',
+        checkpoint='https://download.openmmlab.com/'
+        'mmocr/backbone/resnet50-oclip-7ba0c533.pth'))
+
+param_scheduler = [
+    dict(type='LinearLR', end=100, start_factor=0.001),
+    dict(type='PolyLR', power=0.9, eta_min=1e-7, begin=100, end=1200),
+]

configs/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500.py

@@ -0,0 +1,30 @@
+_base_ = [
+    '_base_drrg_resnet50_fpn-unet.py',
+    '../_base_/datasets/ctw1500.py',
+    '../_base_/default_runtime.py',
+    '../_base_/schedules/schedule_sgd_1200e.py',
+]
+
+# dataset settings
+ctw1500_textdet_train = _base_.ctw1500_textdet_train
+ctw1500_textdet_train.pipeline = _base_.train_pipeline
+ctw1500_textdet_test = _base_.ctw1500_textdet_test
+ctw1500_textdet_test.pipeline = _base_.test_pipeline
+
+train_dataloader = dict(
+    batch_size=4,
+    num_workers=4,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=True),
+    dataset=ctw1500_textdet_train)
+
+val_dataloader = dict(
+    batch_size=1,
+    num_workers=1,
+    persistent_workers=True,
+    sampler=dict(type='DefaultSampler', shuffle=False),
+    dataset=ctw1500_textdet_test)
+
+test_dataloader = val_dataloader
+
+auto_scale_lr = dict(base_batch_size=16)

configs/textdet/drrg/metafile.yml

@@ -0,0 +1,28 @@
+Collections:
+- Name: DRRG
+  Metadata:
+    Training Data: SCUT-CTW1500
+    Training Techniques:
+      - SGD with Momentum
+    Training Resources: 4x NVIDIA A100-SXM4-80GB
+    Architecture:
+      - ResNet
+      - FPN_UNet
+  Paper:
+    URL: https://arxiv.org/abs/2003.07493.pdf
+    Title: 'Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detection'
+  README: configs/textdet/drrg/README.md
+
+Models:
+  - Name: drrg_resnet50_fpn-unet_1200e_ctw1500
+    Alias: DRRG
+    In Collection: DRRG
+    Config: configs/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500.py
+    Metadata:
+      Training Data: CTW1500
+    Results:
+      - Task: Text Detection
+        Dataset: CTW1500
+        Metrics:
+          hmean-iou: 0.8467
+    Weights: https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/drrg_resnet50_fpn-unet_1200e_ctw1500_20220827_105233-d5c702dd.pth

configs/textdet/fcenet/README.md

@@ -0,0 +1,46 @@
+# FCENet
+
+> [Fourier Contour Embedding for Arbitrary-Shaped Text Detection](https://arxiv.org/abs/2104.10442)
+
+<!-- [ALGORITHM] -->
+
+## Abstract
+
+One of the main challenges for arbitrary-shaped text detection is to design a good text instance representation that allows networks to learn diverse text geometry variances. Most of existing methods model text instances in image spatial domain via masks or contour point sequences in the Cartesian or the polar coordinate system. However, the mask representation might lead to expensive post-processing, while the point sequence one may have limited capability to model texts with highly-curved shapes. To tackle these problems, we model text instances in the Fourier domain and propose one novel Fourier Contour Embedding (FCE) method to represent arbitrary shaped text contours as compact signatures. We further construct FCENet with a backbone, feature pyramid networks (FPN) and a simple post-processing with the Inverse Fourier Transformation (IFT) and Non-Maximum Suppression (NMS). Different from previous methods, FCENet first predicts compact Fourier signatures of text instances, and then reconstructs text contours via IFT and NMS during test. Extensive experiments demonstrate that FCE is accurate and robust to fit contours of scene texts even with highly-curved shapes, and also validate the effectiveness and the good generalization of FCENet for arbitrary-shaped text detection. Furthermore, experimental results show that our FCENet is superior to the state-of-the-art (SOTA) methods on CTW1500 and Total-Text, especially on challenging highly-curved text subset.
+
+<div align=center>
+<img src="https://user-images.githubusercontent.com/22607038/142791859-1b0ebde4-b151-4c25-ba1b-f354bd8ddc8c.png"/>
+</div>
+
+## Results and models
+
+### CTW1500
+
+|                 Method                 |                 Backbone                  | Pretrained Model | Training set  |   Test set   | #epochs |  Test size  | Precision | Recall | Hmean  |                 Download                  |
+| :------------------------------------: | :---------------------------------------: | :--------------: | :-----------: | :----------: | :-----: | :---------: | :-------: | :----: | :----: | :---------------------------------------: |
+| [FCENet_r50dcn](/configs/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500.py) |             ResNet50 + DCNv2              |        -         | CTW1500 Train | CTW1500 Test |  1500   | (736, 1080) |  0.8689   | 0.8296 | 0.8488 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500_20220825_221510-4d705392.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/20220825_221510.log) |
+| [FCENet_r50-oclip](/configs/textdet/fcenet/fcenet_resnet50-oclip-dcnv2_fpn_1500e_ctw1500.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |        -         | CTW1500 Train | CTW1500 Test |  1500   | (736, 1080) |  0.8383   | 0.801  | 0.8192 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_ctw1500/fcenet_resnet50-oclip_fpn_1500e_ctw1500_20221102_121909-101df7e6.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_ctw1500/20221102_121909.log) |
+
+### ICDAR2015
+
+|                        Method                         |    Backbone    | Pretrained Model | Training set | Test set  | #epochs |  Test size   | Precision | Recall | Hmean  |                         Download                         |
+| :---------------------------------------------------: | :------------: | :--------------: | :----------: | :-------: | :-----: | :----------: | :-------: | :----: | :----: | :------------------------------------------------------: |
+| [FCENet_r50](/configs/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015.py) |    ResNet50    |        -         |  IC15 Train  | IC15 Test |  1500   | (2260, 2260) |  0.8243   | 0.8834 | 0.8528 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/fcenet_resnet50_fpn_1500e_icdar2015_20220826_140941-167d9042.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/20220826_140941.log) |
+| [FCENet_r50-oclip](/configs/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_icdar2015.py) | ResNet50-oCLIP |        -         |  IC15 Train  | IC15 Test |  1500   | (2260, 2260) |  0.9176   | 0.8098 | 0.8604 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_icdar2015/fcenet_resnet50-oclip_fpn_1500e_icdar2015_20221101_150145-5a6fc412.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_icdar2015/20221101_150145.log) |
+
+### Total Text
+
+|                        Method                         | Backbone | Pretrained Model |  Training set   |    Test set    | #epochs |  Test size  | Precision | Recall | Hmean  |                        Download                         |
+| :---------------------------------------------------: | :------: | :--------------: | :-------------: | :------------: | :-----: | :---------: | :-------: | :----: | :----: | :-----------------------------------------------------: |
+| [FCENet_r50](/configs/textdet/fcenet/fcenet_resnet50_fpn_1500e_totaltext.py) | ResNet50 |        -         | Totaltext Train | Totaltext Test |  1500   | (1280, 960) |  0.8485   | 0.7810 | 0.8134 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_totaltext/fcenet_resnet50_fpn_1500e_totaltext-91bd37af.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_totaltext/20221219_201107.log) |
+
+## Citation
+
+```bibtex
+@InProceedings{zhu2021fourier,
+      title={Fourier Contour Embedding for Arbitrary-Shaped Text Detection},
+      author={Yiqin Zhu and Jianyong Chen and Lingyu Liang and Zhanghui Kuang and Lianwen Jin and Wayne Zhang},
+      year={2021},
+      booktitle = {CVPR}
+      }
+```