Eugene Siow commited on
Commit
3002dd6
1 Parent(s): d946649

Initial commit.

Browse files
README.md ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - super-image
5
+ - image-super-resolution
6
+ datasets:
7
+ - div2k
8
+ metrics:
9
+ - pnsr
10
+ - ssim
11
+ ---
12
+ # Multi-scale Residual Network for Image Super-Resolution (MSRN)
13
+ MSRN model pre-trained on DIV2K (800 images training, augmented to 4000 images, 100 images validation) for 2x, 3x and 4x image super resolution. It was introduced in the paper [Multi-scale Residual Network for Image Super-Resolution](https://openaccess.thecvf.com/content_ECCV_2018/html/Juncheng_Li_Multi-scale_Residual_Network_ECCV_2018_paper.html) by Li et al. (2018) and first released in [this repository](https://github.com/MIVRC/MSRN-PyTorch).
14
+
15
+ The goal of image super resolution is to restore a high resolution (HR) image from a single low resolution (LR) image. The image below shows the ground truth (HR), the bicubic upscaling x2 and model upscaling x2.
16
+
17
+ ![Comparing Bicubic upscaling against the models x2 upscaling on Set5 Image 4](images/msrn_4_4_compare.png "Comparing Bicubic upscaling against the models x2 upscaling on Set5 Image 4")
18
+ ## Model description
19
+ The MSRN model proposes a feature extraction structure called the multi-scale residual block. This module can "adaptively detect image features at different scales" and "exploit the potential features of the image".
20
+ ## Intended uses & limitations
21
+ You can use the pre-trained models for upscaling your images 2x, 3x and 4x. You can also use the trainer to train a model on your own dataset.
22
+ ### How to use
23
+ The model can be used with the [super_image](https://github.com/eugenesiow/super-image) library:
24
+ ```bash
25
+ pip install super-image
26
+ ```
27
+ Here is how to use a pre-trained model to upscale your image:
28
+ ```python
29
+ from super_image import MsrnModel, ImageLoader
30
+ from PIL import Image
31
+ import requests
32
+
33
+ url = 'https://paperswithcode.com/media/datasets/Set5-0000002728-07a9793f_zA3bDjj.jpg'
34
+ image = Image.open(requests.get(url, stream=True).raw)
35
+
36
+ model = MsrnModel.from_pretrained('eugenesiow/msrn', scale=2) # scale 2, 3 and 4 models available
37
+ inputs = ImageLoader.load_image(image)
38
+ preds = model(inputs)
39
+
40
+ ImageLoader.save_image(preds, './scaled_2x.png') # save the output 2x scaled image to `./scaled_2x.png`
41
+ ImageLoader.save_compare(inputs, preds, './scaled_2x_compare.png') # save an output comparing the super-image with a bicubic scaling
42
+ ```
43
+ ## Training data
44
+ The models for 2x, 3x and 4x image super resolution were pretrained on [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/), a dataset of 800 high-quality (2K resolution) images for training, augmented to 4000 images and uses a dev set of 100 validation images (images numbered 801 to 900).
45
+ ## Training procedure
46
+ ### Preprocessing
47
+ We follow the pre-processing and training method of [Wang et al.](https://arxiv.org/abs/2104.07566).
48
+ Low Resolution (LR) images are created by using bicubic interpolation as the resizing method to reduce the size of the High Resolution (HR) images by x2, x3 and x4 times.
49
+ During training, RGB patches with size of 64×64 from the LR input are used together with their corresponding HR patches.
50
+ Data augmentation is applied to the training set in the pre-processing stage where five images are created from the four corners and center of the original image.
51
+
52
+ The following code provides some helper functions to preprocess the data.
53
+ ```python
54
+ from super_image.data import EvalDataset, TrainAugmentDataset, DatasetBuilder
55
+
56
+ DatasetBuilder.prepare(
57
+ base_path='./DIV2K/DIV2K_train_HR',
58
+ output_path='./div2k_4x_train.h5',
59
+ scale=4,
60
+ do_augmentation=True
61
+ )
62
+ DatasetBuilder.prepare(
63
+ base_path='./DIV2K/DIV2K_val_HR',
64
+ output_path='./div2k_4x_val.h5',
65
+ scale=4,
66
+ do_augmentation=False
67
+ )
68
+ train_dataset = TrainAugmentDataset('./div2k_4x_train.h5', scale=4)
69
+ val_dataset = EvalDataset('./div2k_4x_val.h5')
70
+ ```
71
+ ### Pretraining
72
+ The model was trained on GPU. The training code is provided below:
73
+ ```python
74
+ from super_image import Trainer, TrainingArguments, MsrnModel, MsrnConfig
75
+
76
+ training_args = TrainingArguments(
77
+ output_dir='./results', # output directory
78
+ num_train_epochs=1000, # total number of training epochs
79
+ )
80
+
81
+ config = MsrnConfig(
82
+ scale=4, # train a model to upscale 4x
83
+ )
84
+ model = MsrnModel(config)
85
+
86
+ trainer = Trainer(
87
+ model=model, # the instantiated model to be trained
88
+ args=training_args, # training arguments, defined above
89
+ train_dataset=train_dataset, # training dataset
90
+ eval_dataset=val_dataset # evaluation dataset
91
+ )
92
+
93
+ trainer.train()
94
+ ```
95
+ ## Evaluation results
96
+ The evaluation metrics include [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio#Quality_estimation_with_PSNR) and [SSIM](https://en.wikipedia.org/wiki/Structural_similarity#Algorithm).
97
+
98
+ Evaluation datasets include:
99
+ - Set5 - [Bevilacqua et al. (2012)](http://people.rennes.inria.fr/Aline.Roumy/results/SR_BMVC12.html)
100
+ - Set14 - [Zeyde et al. (2010)](https://sites.google.com/site/romanzeyde/research-interests)
101
+ - BSD100 - [Martin et al. (2001)](https://www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/)
102
+ - Urban100 - [Huang et al. (2015)](https://sites.google.com/site/jbhuang0604/publications/struct_sr)
103
+
104
+ The results columns below are represented below as `PSNR/SSIM`. They are compared against a Bicubic baseline.
105
+
106
+ |Dataset |Scale |Bicubic |msrn-bam |
107
+ |--- |--- |--- |--- |
108
+ |Set5 |2x |33.64/0.9292 | |
109
+ |Set5 |3x |30.39/0.8678 | |
110
+ |Set5 |4x |28.42/0.8101 |**32.19/0.8951** |
111
+ |Set14 |2x |30.22/0.8683 | |
112
+ |Set14 |3x |27.53/0.7737 | |
113
+ |Set14 |4x |25.99/0.7023 |**28.67/0.7833** |
114
+ |BSD100 |2x |29.55/0.8425 | |
115
+ |BSD100 |3x |27.20/0.7382 | |
116
+ |BSD100 |4x |25.96/0.6672 |**27.63/0.7374** |
117
+ |Urban100 |2x |26.66/0.8408 | |
118
+ |Urban100 |3x | | |
119
+ |Urban100 |4x |23.14/0.6573 |**26.12/0.7866** |
120
+
121
+ ![Comparing Bicubic upscaling against the models x2 upscaling on Set5 Image 2](images/msrn_2_4_compare.png "Comparing Bicubic upscaling against the models x2 upscaling on Set5 Image 2")
122
+
123
+ ## BibTeX entry and citation info
124
+ ```bibtex
125
+ @InProceedings{Li_2018_ECCV,
126
+ author = {Li, Juncheng and Fang, Faming and Mei, Kangfu and Zhang, Guixu},
127
+ title = {Multi-scale Residual Network for Image Super-Resolution},
128
+ booktitle = {The European Conference on Computer Vision (ECCV)},
129
+ month = {September},
130
+ year = {2018}
131
+ }
132
+ ```
config.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "eugenesiow/msrn",
3
+ "data_parallel": false,
4
+ "model_type": "MSRN",
5
+ "n_feats": 64,
6
+ "n_blocks": 8,
7
+ "rgb_range": 255
8
+ }
images/msrn_2_4_compare.png ADDED
Binary file
images/msrn_4_4_compare.png ADDED
Binary file
pytorch_model_4x.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:040478be5e5c0f8d3a12fcab7e3402715b2de9eec5fd39660886b41834bee4ec
3
+ size 24343877