|
--- |
|
license: apache-2.0 |
|
tags: |
|
- super-image |
|
- image-super-resolution |
|
datasets: |
|
- div2k |
|
metrics: |
|
- pnsr |
|
- ssim |
|
--- |
|
# Multi-scale Residual Network for Image Super-Resolution (MSRN) |
|
MSRN model pre-trained on DIV2K (800 images training, augmented to 4000 images, 100 images validation) for 2x, 3x and 4x image super resolution. It was introduced in the paper [Multi-scale Residual Network for Image Super-Resolution](https://openaccess.thecvf.com/content_ECCV_2018/html/Juncheng_Li_Multi-scale_Residual_Network_ECCV_2018_paper.html) by Li et al. (2018) and first released in [this repository](https://github.com/MIVRC/MSRN-PyTorch). |
|
|
|
The goal of image super resolution is to restore a high resolution (HR) image from a single low resolution (LR) image. The image below shows the ground truth (HR), the bicubic upscaling x2 and model upscaling x2. |
|
|
|
 |
|
## Model description |
|
The MSRN model proposes a feature extraction structure called the multi-scale residual block. This module can "adaptively detect image features at different scales" and "exploit the potential features of the image". |
|
## Intended uses & limitations |
|
You can use the pre-trained models for upscaling your images 2x, 3x and 4x. You can also use the trainer to train a model on your own dataset. |
|
### How to use |
|
The model can be used with the [super_image](https://github.com/eugenesiow/super-image) library: |
|
```bash |
|
pip install super-image |
|
``` |
|
Here is how to use a pre-trained model to upscale your image: |
|
```python |
|
from super_image import MsrnModel, ImageLoader |
|
from PIL import Image |
|
import requests |
|
|
|
url = 'https://paperswithcode.com/media/datasets/Set5-0000002728-07a9793f_zA3bDjj.jpg' |
|
image = Image.open(requests.get(url, stream=True).raw) |
|
|
|
model = MsrnModel.from_pretrained('eugenesiow/msrn', scale=2) # scale 2, 3 and 4 models available |
|
inputs = ImageLoader.load_image(image) |
|
preds = model(inputs) |
|
|
|
ImageLoader.save_image(preds, './scaled_2x.png') # save the output 2x scaled image to `./scaled_2x.png` |
|
ImageLoader.save_compare(inputs, preds, './scaled_2x_compare.png') # save an output comparing the super-image with a bicubic scaling |
|
``` |
|
## Training data |
|
The models for 2x, 3x and 4x image super resolution were pretrained on [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/), a dataset of 800 high-quality (2K resolution) images for training, augmented to 4000 images and uses a dev set of 100 validation images (images numbered 801 to 900). |
|
## Training procedure |
|
### Preprocessing |
|
We follow the pre-processing and training method of [Wang et al.](https://arxiv.org/abs/2104.07566). |
|
Low Resolution (LR) images are created by using bicubic interpolation as the resizing method to reduce the size of the High Resolution (HR) images by x2, x3 and x4 times. |
|
During training, RGB patches with size of 64×64 from the LR input are used together with their corresponding HR patches. |
|
Data augmentation is applied to the training set in the pre-processing stage where five images are created from the four corners and center of the original image. |
|
|
|
The following code provides some helper functions to preprocess the data. |
|
```python |
|
from super_image.data import EvalDataset, TrainAugmentDataset, DatasetBuilder |
|
|
|
DatasetBuilder.prepare( |
|
base_path='./DIV2K/DIV2K_train_HR', |
|
output_path='./div2k_4x_train.h5', |
|
scale=4, |
|
do_augmentation=True |
|
) |
|
DatasetBuilder.prepare( |
|
base_path='./DIV2K/DIV2K_val_HR', |
|
output_path='./div2k_4x_val.h5', |
|
scale=4, |
|
do_augmentation=False |
|
) |
|
train_dataset = TrainAugmentDataset('./div2k_4x_train.h5', scale=4) |
|
val_dataset = EvalDataset('./div2k_4x_val.h5') |
|
``` |
|
### Pretraining |
|
The model was trained on GPU. The training code is provided below: |
|
```python |
|
from super_image import Trainer, TrainingArguments, MsrnModel, MsrnConfig |
|
|
|
training_args = TrainingArguments( |
|
output_dir='./results', # output directory |
|
num_train_epochs=1000, # total number of training epochs |
|
) |
|
|
|
config = MsrnConfig( |
|
scale=4, # train a model to upscale 4x |
|
) |
|
model = MsrnModel(config) |
|
|
|
trainer = Trainer( |
|
model=model, # the instantiated model to be trained |
|
args=training_args, # training arguments, defined above |
|
train_dataset=train_dataset, # training dataset |
|
eval_dataset=val_dataset # evaluation dataset |
|
) |
|
|
|
trainer.train() |
|
``` |
|
## Evaluation results |
|
The evaluation metrics include [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio#Quality_estimation_with_PSNR) and [SSIM](https://en.wikipedia.org/wiki/Structural_similarity#Algorithm). |
|
|
|
Evaluation datasets include: |
|
- Set5 - [Bevilacqua et al. (2012)](http://people.rennes.inria.fr/Aline.Roumy/results/SR_BMVC12.html) |
|
- Set14 - [Zeyde et al. (2010)](https://sites.google.com/site/romanzeyde/research-interests) |
|
- BSD100 - [Martin et al. (2001)](https://www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/) |
|
- Urban100 - [Huang et al. (2015)](https://sites.google.com/site/jbhuang0604/publications/struct_sr) |
|
|
|
The results columns below are represented below as `PSNR/SSIM`. They are compared against a Bicubic baseline. |
|
|
|
|Dataset |Scale |Bicubic |msrn-bam | |
|
|--- |--- |--- |--- | |
|
|Set5 |2x |33.64/0.9292 | | |
|
|Set5 |3x |30.39/0.8678 | | |
|
|Set5 |4x |28.42/0.8101 |**32.19/0.8951** | |
|
|Set14 |2x |30.22/0.8683 | | |
|
|Set14 |3x |27.53/0.7737 | | |
|
|Set14 |4x |25.99/0.7023 |**28.67/0.7833** | |
|
|BSD100 |2x |29.55/0.8425 | | |
|
|BSD100 |3x |27.20/0.7382 | | |
|
|BSD100 |4x |25.96/0.6672 |**27.63/0.7374** | |
|
|Urban100 |2x |26.66/0.8408 | | |
|
|Urban100 |3x | | | |
|
|Urban100 |4x |23.14/0.6573 |**26.12/0.7866** | |
|
|
|
 |
|
|
|
## BibTeX entry and citation info |
|
```bibtex |
|
@InProceedings{Li_2018_ECCV, |
|
author = {Li, Juncheng and Fang, Faming and Mei, Kangfu and Zhang, Guixu}, |
|
title = {Multi-scale Residual Network for Image Super-Resolution}, |
|
booktitle = {The European Conference on Computer Vision (ECCV)}, |
|
month = {September}, |
|
year = {2018} |
|
} |
|
``` |
