init

README.md

@@ -0,0 +1,183 @@
+# Video Feature Enhancement with PyTorch
+
+[![License](https://img.shields.io/badge/license-BSD-blue.svg)](LICENSE)
+
+This repo contains the code for the paper:
+[MAMBA](https://arxiv.org/abs/2401.09923), STPN, TDViT, EOVOD
+
+
+Additionally, we provide archive files of two widely-used datasets, ImageNetVID and GOT-10K. The official links of these datasets are not accessible or deleted. We hope these resources can help future research.
+
+## Progress
+
+- [x] [MAMBA](https://arxiv.org/abs/2401.09923)
+- [ ] STPN
+- [ ] TDViT
+- [ ] EOVOD
+
+## Main Results
+
+|       Model        |  Backbone  | AP50 | AP (fast) | AP (med) | AP (slow) |                                             Link                                             |
+| :----------------: | :--------: | :--: | :-------: | :------: | :-------: | :------------------------------------------------------------------------------------------: |
+|     FasterRCNN     | ResNet-101 | 76.7 |   52.3    |   74.1   |   84.9    | [model](https://drive.google.com/file/d/1W17f9GC60rHU47lUeOEfU--Ra-LTw3Tq/view?usp=sharing), [reference](https://github.com/Scalsol/mega.pytorch/tree/master?tab=readme-ov-file#main-results)|
+|     SELSA          | ResNet-101 |  81.5  |    --     |    --    |    --     | [model](https://download.openmmlab.com/mmtracking/vid/selsa/selsa_faster_rcnn_r101_dc5_1x_imagenetvid/selsa_faster_rcnn_r101_dc5_1x_imagenetvid_20201218_172724-aa961bcc.pth), [reference](https://github.com/open-mmlab/mmtracking/tree/master/configs/vid/selsa) |
+|     MEGA     |   ResNet-101  |  82.9	|62.7	|81.6	|89.4  | [model](https://drive.google.com/file/d/1ZnAdFafF1vW9Lnpw-RPF1AD_csw61lBY/view?usp=sharing), [reference](https://github.com/Scalsol/mega.pytorch/tree/master) |
+|     **MAMBA**      | ResNet-101 |  83.8 | 65.3 | 83.8 | 89.5 | [model](), [paper](https://arxiv.org/abs/2401.09923)|
+
+## Installation
+The code are tested with the following environments:
+
+### Tested environments:
+
+- python 3.8
+- pytorch 1.10.1
+- cuda 11.3
+- mmcv-full 1.3.17
+
+### Option 1: Step-by-step installation
+
+```bash
+conda create --name vfe -y python=3.8
+conda activate vfe
+
+# install PyTorch with cuda support
+conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
+
+# install mmcv-full 1.3.17
+pip install mmcv-full==1.3.17 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10/index.html
+
+# install other requirements
+pip install -r requirements.txt
+
+# install mmpycocotools
+pip install mmpycocotools
+```
+
+See [here](https://github.com/open-mmlab/mmcv#installation) for different versions of MMCV compatible to different PyTorch and CUDA versions.
+
+## Data preparation
+
+### Download Datasets
+
+The original links of ImageNetVID dataset are either broken or unavailible. Here, we provide the new link to download the file for the furture reference of the community. Please download ILSVRC2015 DET and ILSVRC2015 VID datasets from [here](). 
+
+**Note:** The links expire every 30 days. If they expire, please contact me to renew it.
+
+After that, we recommend to symlink the path to the datasets to `datasets/`. And the path structure should be as follows:
+
+    ./data/ILSVRC/
+    ./data/ILSVRC/Annotations/DET
+    ./data/ILSVRC/Annotations/VID
+    ./data/ILSVRC/Data/DET
+    ./data/ILSVRC/Data/VID
+    ./data/ILSVRC/ImageSets
+
+**Note**: List txt files under `ImageSets` folder can be obtained from
+[here](https://github.com/msracver/Flow-Guided-Feature-Aggregation/tree/master/data/ILSVRC2015/ImageSets).
+
+### Convert Annotations
+
+We use [CocoVID](mmdet/datasets/parsers/coco_video_parser.py) to maintain all datasets in this codebase. In this case, you need to convert the official annotations to this style. We provide scripts and the usages are as following:
+
+```bash
+# ImageNet DET
+python ./tools/convert_datasets/ilsvrc/imagenet2coco_det.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
+
+# ImageNet VID
+python ./tools/convert_datasets/ilsvrc/imagenet2coco_vid.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
+
+```
+
+## Usage
+
+### Inference
+
+This section will show how to test existing models on supported datasets.
+The following testing environments are supported:
+
+- single GPU
+- single node multiple GPU
+
+During testing, different tasks share the same API and we only support `samples_per_gpu = 1`.
+
+You can use the following commands for testing:
+
+```shell
+# single-gpu testing
+python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}]
+
+# multi-gpu testing
+./tools/dist_test.sh ${CONFIG_FILE} ${GPU_NUM} [--checkpoint ${CHECKPOINT_FILE}] [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}]
+```
+
+Optional arguments:
+
+- `CHECKPOINT_FILE`: Filename of the checkpoint. You do not need to define it when applying some MOT methods but specify the checkpoints in the config.
+- `RESULT_FILE`: Filename of the output results in pickle format. If not specified, the results will not be saved to a file.
+- `EVAL_METRICS`: Items to be evaluated on the results. Allowed values depend on the dataset, e.g., `bbox` is available for ImageNet VID, `track` is available for LaSOT, `bbox` and `track` are both suitable for MOT17.
+- `--cfg-options`: If specified, the key-value pair optional cfg will be merged into config file
+- `--eval-options`: If specified, the key-value pair optional eval cfg will be kwargs for dataset.evaluate() function, it’s only for evaluation
+- `--format-only`: If specified, the results will be formatted to the official format.
+
+#### Examples of testing VID model
+
+Assume that you have already downloaded the checkpoints to the directory `work_dirs/`.
+
+1. Test MAMBA on ImageNet VID, and evaluate the bbox mAP.
+
+   ```shell
+   python tools/test.py configs/vid/mamba/mamba_r101_dc5_6x.py \
+       --checkpoint work_dirs/mamba_r101_dc5_6x/epoch_6_model.pth \
+       --out results.pkl \
+       --eval bbox
+   ```
+
+2. Test MAMBA with 8 GPUs on ImageNet VID, and evaluate the bbox mAP.
+
+   ```shell
+   ./tools/dist_test.sh configs/vid/mamba/mamba_r101_dc5_6x.py 8 \
+       --checkpoint work_dirs/mamba_r101_dc5_6x/epoch_6_model.pth \
+       --out results.pkl \
+       --eval bbox
+   ```
+
+### Training
+
+#### Training on a single GPU
+
+```shell
+python tools/train.py ${CONFIG_FILE} [optional arguments]
+```
+
+During training, log files and checkpoints will be saved to the working directory, which is specified by `work_dir` in the config file or via CLI argument `--work-dir`.
+
+#### Training on multiple GPUs
+
+We provide `tools/dist_train.sh` to launch training on multiple GPUs.
+The basic usage is as follows.
+
+```shell
+bash ./tools/dist_train.sh \
+    ${CONFIG_FILE} \
+    ${GPU_NUM} \
+    [optional arguments]
+```
+
+#### Examples of training VID model
+
+1. Train MAMBA on ImageNet VID and ImageNet DET with single GPU, then evaluate the bbox mAP at the last epoch.
+
+   ```shell
+   python tools/train.py configs/vid/mamba/mamba_r101_dc5_6x.py 
+   ```
+
+2. Train MAMBA on ImageNet VID and ImageNet DET with 8 GPUs, then evaluate the bbox mAP at the last epoch.
+
+   ```shell
+   ./tools/dist_train.sh configs/vid/mamba/mamba_r101_dc5_6x.py 8
+   ```
+
+## Reference
+
+The codebase is implemented based on two popular open-source repos:
+ [mmdetection](https://github.com/open-mmlab/mmdetection) and [mmtracking](https://github.com/open-mmlab/mmtracking) in [PyTorch](https://pytorch.org/).

work_dirs/mamba_r101_dc5_6x/epoch_6_model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d652da944645d3f2f5cfcb1a88b2d29df8f02d9269e5fa4f940bf32bd44a0881
+size 359129121

work_dirs/mamba_r101_dc5_6x/mamba_r101_dc5_6x.py

@@ -0,0 +1,298 @@
+model = dict(
+    detector=dict(
+        type='FasterRCNN',
+        backbone=dict(
+            type='ResNet',
+            depth=101,
+            num_stages=4,
+            out_indices=(3, ),
+            strides=(1, 2, 2, 1),
+            dilations=(1, 1, 1, 2),
+            frozen_stages=1,
+            norm_cfg=dict(type='BN', requires_grad=True),
+            norm_eval=True,
+            style='pytorch',
+            init_cfg=dict(
+                type='Pretrained', checkpoint='torchvision://resnet101')),
+        neck=dict(
+            type='ChannelMapper',
+            in_channels=[2048],
+            out_channels=512,
+            kernel_size=3),
+        rpn_head=dict(
+            type='RPNHead',
+            in_channels=512,
+            feat_channels=512,
+            anchor_generator=dict(
+                type='AnchorGenerator',
+                scales=[4, 8, 16, 32],
+                ratios=[0.5, 1.0, 2.0],
+                strides=[16]),
+            bbox_coder=dict(
+                type='DeltaXYWHBBoxCoder',
+                target_means=[0.0, 0.0, 0.0, 0.0],
+                target_stds=[1.0, 1.0, 1.0, 1.0]),
+            loss_cls=dict(
+                type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
+            loss_bbox=dict(
+                type='SmoothL1Loss', beta=0.1111111111111111,
+                loss_weight=1.0)),
+        roi_head=dict(
+            type='MambaRoIHead',
+            bbox_roi_extractor=dict(
+                type='SingleRoIExtractor',
+                roi_layer=dict(
+                    type='RoIAlign', output_size=7, sampling_ratio=2),
+                out_channels=512,
+                featmap_strides=[16]),
+            bbox_head=dict(
+                type='MambaBBoxHead',
+                in_channels=512,
+                fc_out_channels=1024,
+                roi_feat_size=7,
+                num_classes=30,
+                bbox_coder=dict(
+                    type='DeltaXYWHBBoxCoder',
+                    target_means=[0.0, 0.0, 0.0, 0.0],
+                    target_stds=[0.2, 0.2, 0.2, 0.2]),
+                reg_class_agnostic=False,
+                loss_cls=dict(
+                    type='CrossEntropyLoss',
+                    use_sigmoid=False,
+                    loss_weight=1.0),
+                loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0),
+                num_shared_fcs=2,
+                topk=75,
+                aggregator=dict(
+                    type='MambaAggregator',
+                    in_channels=1024,
+                    num_attention_blocks=16))),
+        train_cfg=dict(
+            rpn=dict(
+                assigner=dict(
+                    type='MaxIoUAssigner',
+                    pos_iou_thr=0.7,
+                    neg_iou_thr=0.3,
+                    min_pos_iou=0.3,
+                    ignore_iof_thr=-1),
+                sampler=dict(
+                    type='RandomSampler',
+                    num=256,
+                    pos_fraction=0.5,
+                    neg_pos_ub=-1,
+                    add_gt_as_proposals=False),
+                allowed_border=0,
+                pos_weight=-1,
+                debug=False),
+            rpn_proposal=dict(
+                nms_pre=6000,
+                max_per_img=600,
+                nms=dict(type='nms', iou_threshold=0.7),
+                min_bbox_size=0),
+            rcnn=dict(
+                assigner=dict(
+                    type='MaxIoUAssigner',
+                    pos_iou_thr=0.5,
+                    neg_iou_thr=0.5,
+                    min_pos_iou=0.5,
+                    ignore_iof_thr=-1),
+                sampler=dict(
+                    type='RandomSampler',
+                    num=256,
+                    pos_fraction=0.25,
+                    neg_pos_ub=-1,
+                    add_gt_as_proposals=True),
+                pos_weight=-1,
+                debug=False)),
+        test_cfg=dict(
+            rpn=dict(
+                nms_pre=6000,
+                max_per_img=300,
+                nms=dict(type='nms', iou_threshold=0.7),
+                min_bbox_size=0),
+            rcnn=dict(
+                score_thr=0.0001,
+                nms=dict(type='nms', iou_threshold=0.5),
+                max_per_img=100))),
+    type='MAMBA')
+dataset_type = 'ImagenetVIDDataset'
+data_root = 'data/ILSVRC/'
+img_norm_cfg = dict(
+    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
+train_pipeline = [
+    dict(type='LoadMultiImagesFromFile'),
+    dict(type='SeqLoadAnnotations', with_bbox=True, with_track=True),
+    dict(type='SeqResize', img_scale=(1000, 600), keep_ratio=True),
+    dict(type='SeqRandomFlip', share_params=True, flip_ratio=0.5),
+    dict(
+        type='SeqNormalize',
+        mean=[123.675, 116.28, 103.53],
+        std=[58.395, 57.12, 57.375],
+        to_rgb=True),
+    dict(type='SeqPad', size_divisor=16),
+    dict(
+        type='VideoCollect',
+        keys=['img', 'gt_bboxes', 'gt_labels', 'gt_instance_ids']),
+    dict(type='ConcatVideoReferences'),
+    dict(type='SeqDefaultFormatBundle', ref_prefix='ref')
+]
+test_pipeline = [
+    dict(type='LoadMultiImagesFromFile'),
+    dict(type='SeqResize', img_scale=(1000, 600), keep_ratio=True),
+    dict(type='SeqRandomFlip', share_params=True, flip_ratio=0.0),
+    dict(
+        type='SeqNormalize',
+        mean=[123.675, 116.28, 103.53],
+        std=[58.395, 57.12, 57.375],
+        to_rgb=True),
+    dict(type='SeqPad', size_divisor=16),
+    dict(
+        type='VideoCollect',
+        keys=['img'],
+        meta_keys=('num_left_ref_imgs', 'frame_stride')),
+    dict(type='ConcatVideoReferences'),
+    dict(type='MultiImagesToTensor', ref_prefix='ref'),
+    dict(type='ToList')
+]
+data = dict(
+    samples_per_gpu=1,
+    workers_per_gpu=4,
+    train=[
+        dict(
+            type='ImagenetVIDDataset',
+            ann_file='data/ILSVRC/annotations/imagenet_vid_train.json',
+            img_prefix='data/ILSVRC/Data/VID',
+            ref_img_sampler=dict(
+                num_ref_imgs=2,
+                frame_range=1000,
+                filter_key_img=True,
+                method='bilateral_uniform'),
+            pipeline=[
+                dict(type='LoadMultiImagesFromFile'),
+                dict(
+                    type='SeqLoadAnnotations', with_bbox=True,
+                    with_track=True),
+                dict(type='SeqResize', img_scale=(1000, 600), keep_ratio=True),
+                dict(type='SeqRandomFlip', share_params=True, flip_ratio=0.5),
+                dict(
+                    type='SeqNormalize',
+                    mean=[123.675, 116.28, 103.53],
+                    std=[58.395, 57.12, 57.375],
+                    to_rgb=True),
+                dict(type='SeqPad', size_divisor=16),
+                dict(
+                    type='VideoCollect',
+                    keys=['img', 'gt_bboxes', 'gt_labels', 'gt_instance_ids']),
+                dict(type='ConcatVideoReferences'),
+                dict(type='SeqDefaultFormatBundle', ref_prefix='ref')
+            ]),
+        dict(
+            type='ImagenetVIDDataset',
+            load_as_video=False,
+            ann_file='data/ILSVRC/annotations/imagenet_det_30plus1cls.json',
+            img_prefix='data/ILSVRC/Data/DET',
+            ref_img_sampler=dict(
+                num_ref_imgs=2,
+                frame_range=0,
+                filter_key_img=False,
+                method='bilateral_uniform'),
+            pipeline=[
+                dict(type='LoadMultiImagesFromFile'),
+                dict(
+                    type='SeqLoadAnnotations', with_bbox=True,
+                    with_track=True),
+                dict(type='SeqResize', img_scale=(1000, 600), keep_ratio=True),
+                dict(type='SeqRandomFlip', share_params=True, flip_ratio=0.5),
+                dict(
+                    type='SeqNormalize',
+                    mean=[123.675, 116.28, 103.53],
+                    std=[58.395, 57.12, 57.375],
+                    to_rgb=True),
+                dict(type='SeqPad', size_divisor=16),
+                dict(
+                    type='VideoCollect',
+                    keys=['img', 'gt_bboxes', 'gt_labels', 'gt_instance_ids']),
+                dict(type='ConcatVideoReferences'),
+                dict(type='SeqDefaultFormatBundle', ref_prefix='ref')
+            ])
+    ],
+    val=dict(
+        type='ImagenetVIDDataset',
+        ann_file='data/ILSVRC/annotations/imagenet_vid_val.json',
+        img_prefix='data/ILSVRC/Data/VID',
+        ref_img_sampler=dict(
+            num_ref_imgs=14,
+            frame_range=[-7, 7],
+            stride=1,
+            method='test_with_adaptive_stride'),
+        pipeline=[
+            dict(type='LoadMultiImagesFromFile'),
+            dict(type='SeqResize', img_scale=(1000, 600), keep_ratio=True),
+            dict(type='SeqRandomFlip', share_params=True, flip_ratio=0.0),
+            dict(
+                type='SeqNormalize',
+                mean=[123.675, 116.28, 103.53],
+                std=[58.395, 57.12, 57.375],
+                to_rgb=True),
+            dict(type='SeqPad', size_divisor=16),
+            dict(
+                type='VideoCollect',
+                keys=['img'],
+                meta_keys=('num_left_ref_imgs', 'frame_stride')),
+            dict(type='ConcatVideoReferences'),
+            dict(type='MultiImagesToTensor', ref_prefix='ref'),
+            dict(type='ToList')
+        ],
+        test_mode=True,
+        shuffle_video_frames=True),
+    test=dict(
+        type='ImagenetVIDDataset',
+        ann_file='data/ILSVRC/annotations/imagenet_vid_val.json',
+        img_prefix='data/ILSVRC/Data/VID',
+        ref_img_sampler=dict(
+            num_ref_imgs=14,
+            frame_range=[-7, 7],
+            stride=1,
+            method='test_with_adaptive_stride'),
+        pipeline=[
+            dict(type='LoadMultiImagesFromFile'),
+            dict(type='SeqResize', img_scale=(1000, 600), keep_ratio=True),
+            dict(type='SeqRandomFlip', share_params=True, flip_ratio=0.0),
+            dict(
+                type='SeqNormalize',
+                mean=[123.675, 116.28, 103.53],
+                std=[58.395, 57.12, 57.375],
+                to_rgb=True),
+            dict(type='SeqPad', size_divisor=16),
+            dict(
+                type='VideoCollect',
+                keys=['img'],
+                meta_keys=('num_left_ref_imgs', 'frame_stride')),
+            dict(type='ConcatVideoReferences'),
+            dict(type='MultiImagesToTensor', ref_prefix='ref'),
+            dict(type='ToList')
+        ],
+        test_mode=True,
+        shuffle_video_frames=True))
+checkpoint_config = dict(interval=3)
+log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')])
+custom_hooks = [dict(type='NumClassCheckHook')]
+dist_params = dict(backend='nccl')
+log_level = 'INFO'
+load_from = None
+resume_from = 'work_dirs/mamba_r101_dc5_6x/epoch_3.pth'
+workflow = [('train', 1)]
+optimizer = dict(type='SGD', lr=0.001, momentum=0.9, weight_decay=0.0001)
+optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
+lr_config = dict(
+    policy='step',
+    warmup='linear',
+    warmup_iters=500,
+    warmup_ratio=0.3333333333333333,
+    step=[4])
+runner = dict(type='EpochBasedRunner', max_epochs=6)
+is_video_model = True
+total_epochs = 6
+evaluation = dict(metric=['bbox'], vid_style=True, interval=1)
+work_dir = './work_dirs/mamba_r101_dc5_6x'
+gpu_ids = range(0, 8)