Eugene Siow
commited on
Commit
•
5de2c6f
1
Parent(s):
9bbd588
Initial commit.
Browse files- README.md +152 -0
- config.json +15 -0
- images/carn_2_4_compare.png +0 -0
- images/carn_4_4_compare.png +0 -0
- pytorch_model_4x.pt +3 -0
README.md
ADDED
@@ -0,0 +1,152 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
tags:
|
4 |
+
- super-image
|
5 |
+
- image-super-resolution
|
6 |
+
datasets:
|
7 |
+
- eugenesiow/Div2k
|
8 |
+
- eugenesiow/Set5
|
9 |
+
- eugenesiow/Set14
|
10 |
+
- eugenesiow/BSD100
|
11 |
+
- eugenesiow/Urban100
|
12 |
+
metrics:
|
13 |
+
- pnsr
|
14 |
+
- ssim
|
15 |
+
---
|
16 |
+
# Cascading Residual Network (CARN)
|
17 |
+
CARN model pre-trained on DIV2K (800 images training, augmented to 4000 images, 100 images validation) for 2x, 3x and 4x image super resolution. It was introduced in the paper [Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network](https://arxiv.org/abs/1803.08664) by Ahn et al. (2018) and first released in [this repository](https://github.com/nmhkahn/CARN-pytorch).
|
18 |
+
|
19 |
+
The goal of image super resolution is to restore a high resolution (HR) image from a single low resolution (LR) image. The image below shows the ground truth (HR), the bicubic upscaling and model upscaling.
|
20 |
+
|
21 |
+
![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4](images/carn_4_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4")
|
22 |
+
## Model description
|
23 |
+
The CARN model proposes an architecture that implements a cascading mechanism upon a residual network for accurate and lightweight image super-resolution.
|
24 |
+
|
25 |
+
This model also applies the balanced attention (BAM) method invented by [Wang et al. (2021)](https://arxiv.org/abs/2104.07566) to further improve the results.
|
26 |
+
## Intended uses & limitations
|
27 |
+
You can use the pre-trained models for upscaling your images 2x, 3x and 4x. You can also use the trainer to train a model on your own dataset.
|
28 |
+
### How to use
|
29 |
+
The model can be used with the [super_image](https://github.com/eugenesiow/super-image) library:
|
30 |
+
```bash
|
31 |
+
pip install super-image
|
32 |
+
```
|
33 |
+
Here is how to use a pre-trained model to upscale your image:
|
34 |
+
```python
|
35 |
+
from super_image import CarnModel, ImageLoader
|
36 |
+
from PIL import Image
|
37 |
+
import requests
|
38 |
+
|
39 |
+
url = 'https://paperswithcode.com/media/datasets/Set5-0000002728-07a9793f_zA3bDjj.jpg'
|
40 |
+
image = Image.open(requests.get(url, stream=True).raw)
|
41 |
+
|
42 |
+
model = CarnModel.from_pretrained('eugenesiow/carn-bam', scale=2) # scale 2, 3 and 4 models available
|
43 |
+
inputs = ImageLoader.load_image(image)
|
44 |
+
preds = model(inputs)
|
45 |
+
|
46 |
+
ImageLoader.save_image(preds, './scaled_2x.png') # save the output 2x scaled image to `./scaled_2x.png`
|
47 |
+
ImageLoader.save_compare(inputs, preds, './scaled_2x_compare.png') # save an output comparing the super-image with a bicubic scaling
|
48 |
+
```
|
49 |
+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Upscale_Images_with_Pretrained_super_image_Models.ipynb "Open in Colab")
|
50 |
+
## Training data
|
51 |
+
The models for 2x, 3x and 4x image super resolution were pretrained on [DIV2K](https://huggingface.co/datasets/eugenesiow/Div2k), a dataset of 800 high-quality (2K resolution) images for training, augmented to 4000 images and uses a dev set of 100 validation images (images numbered 801 to 900).
|
52 |
+
## Training procedure
|
53 |
+
### Preprocessing
|
54 |
+
We follow the pre-processing and training method of [Wang et al.](https://arxiv.org/abs/2104.07566).
|
55 |
+
Low Resolution (LR) images are created by using bicubic interpolation as the resizing method to reduce the size of the High Resolution (HR) images by x2, x3 and x4 times.
|
56 |
+
During training, RGB patches with size of 64×64 from the LR input are used together with their corresponding HR patches.
|
57 |
+
Data augmentation is applied to the training set in the pre-processing stage where five images are created from the four corners and center of the original image.
|
58 |
+
|
59 |
+
We need the huggingface [datasets](https://huggingface.co/datasets?filter=task_ids:other-other-image-super-resolution) library to download the data:
|
60 |
+
```bash
|
61 |
+
pip install datasets
|
62 |
+
```
|
63 |
+
The following code gets the data and preprocesses/augments the data.
|
64 |
+
|
65 |
+
```python
|
66 |
+
from datasets import load_dataset
|
67 |
+
from super_image.data import EvalDataset, TrainDataset, augment_five_crop
|
68 |
+
|
69 |
+
augmented_dataset = load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='train')\
|
70 |
+
.map(augment_five_crop, batched=True, desc="Augmenting Dataset") # download and augment the data with the five_crop method
|
71 |
+
train_dataset = TrainDataset(augmented_dataset) # prepare the train dataset for loading PyTorch DataLoader
|
72 |
+
eval_dataset = EvalDataset(load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='validation')) # prepare the eval dataset for the PyTorch DataLoader
|
73 |
+
```
|
74 |
+
### Pretraining
|
75 |
+
The model was trained on GPU. The training code is provided below:
|
76 |
+
```python
|
77 |
+
from super_image import Trainer, TrainingArguments, CarnModel, CarnConfig
|
78 |
+
|
79 |
+
training_args = TrainingArguments(
|
80 |
+
output_dir='./results', # output directory
|
81 |
+
num_train_epochs=1000, # total number of training epochs
|
82 |
+
)
|
83 |
+
|
84 |
+
config = CarnConfig(
|
85 |
+
scale=4, # train a model to upscale 4x
|
86 |
+
bam=True, # apply balanced attention to the network
|
87 |
+
)
|
88 |
+
model = CarnModel(config)
|
89 |
+
|
90 |
+
trainer = Trainer(
|
91 |
+
model=model, # the instantiated model to be trained
|
92 |
+
args=training_args, # training arguments, defined above
|
93 |
+
train_dataset=train_dataset, # training dataset
|
94 |
+
eval_dataset=eval_dataset # evaluation dataset
|
95 |
+
)
|
96 |
+
|
97 |
+
trainer.train()
|
98 |
+
```
|
99 |
+
|
100 |
+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Train_super_image_Models.ipynb "Open in Colab")
|
101 |
+
## Evaluation results
|
102 |
+
The evaluation metrics include [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio#Quality_estimation_with_PSNR) and [SSIM](https://en.wikipedia.org/wiki/Structural_similarity#Algorithm).
|
103 |
+
|
104 |
+
Evaluation datasets include:
|
105 |
+
- Set5 - [Bevilacqua et al. (2012)](https://huggingface.co/datasets/eugenesiow/Set5)
|
106 |
+
- Set14 - [Zeyde et al. (2010)](https://huggingface.co/datasets/eugenesiow/Set14)
|
107 |
+
- BSD100 - [Martin et al. (2001)](https://huggingface.co/datasets/eugenesiow/BSD100)
|
108 |
+
- Urban100 - [Huang et al. (2015)](https://huggingface.co/datasets/eugenesiow/Urban100)
|
109 |
+
|
110 |
+
The results columns below are represented below as `PSNR/SSIM`. They are compared against a Bicubic baseline.
|
111 |
+
|
112 |
+
|Dataset |Scale |Bicubic |carn-bam |
|
113 |
+
|--- |--- |--- |--- |
|
114 |
+
|Set5 |2x |33.64/0.9292 |**** |
|
115 |
+
|Set5 |3x |30.39/0.8678 |**** |
|
116 |
+
|Set5 |4x |28.42/0.8101 |**32.0/0.8923** |
|
117 |
+
|Set14 |2x |30.22/0.8683 |**** |
|
118 |
+
|Set14 |3x |27.53/0.7737 |**** |
|
119 |
+
|Set14 |4x |25.99/0.7023 |**28.62/0.7822** |
|
120 |
+
|BSD100 |2x |29.55/0.8425 |**** |
|
121 |
+
|BSD100 |3x |27.20/0.7382 |**** |
|
122 |
+
|BSD100 |4x |25.96/0.6672 |**28.41/0.7614** |
|
123 |
+
|Urban100 |2x |26.66/0.8408 |**** |
|
124 |
+
|Urban100 |3x | |**** |
|
125 |
+
|Urban100 |4x |23.14/0.6573 |**25.77/0.7741** |
|
126 |
+
|
127 |
+
![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2](images/carn_2_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2")
|
128 |
+
|
129 |
+
You can find a notebook to easily run evaluation on pretrained models below:
|
130 |
+
|
131 |
+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Evaluate_Pretrained_super_image_Models.ipynb "Open in Colab")
|
132 |
+
|
133 |
+
## BibTeX entry and citation info
|
134 |
+
```bibtex
|
135 |
+
@misc{wang2021bam,
|
136 |
+
title={BAM: A Lightweight and Efficient Balanced Attention Mechanism for Single Image Super Resolution},
|
137 |
+
author={Fanyi Wang and Haotian Hu and Cheng Shen},
|
138 |
+
year={2021},
|
139 |
+
eprint={2104.07566},
|
140 |
+
archivePrefix={arXiv},
|
141 |
+
primaryClass={eess.IV}
|
142 |
+
}
|
143 |
+
```
|
144 |
+
|
145 |
+
```bibtex
|
146 |
+
@article{ahn2018fast,
|
147 |
+
title={Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network},
|
148 |
+
author={Ahn, Namhyuk and Kang, Byungkon and Sohn, Kyung-Ah},
|
149 |
+
journal={arXiv preprint arXiv:1803.08664},
|
150 |
+
year={2018}
|
151 |
+
}
|
152 |
+
```
|
config.json
ADDED
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"bam": true,
|
3 |
+
"data_parallel": false,
|
4 |
+
"model_type": "CARN",
|
5 |
+
"rgb_mean": [
|
6 |
+
0.4488,
|
7 |
+
0.4371,
|
8 |
+
0.404
|
9 |
+
],
|
10 |
+
"rgb_std": [
|
11 |
+
1.0,
|
12 |
+
1.0,
|
13 |
+
1.0
|
14 |
+
]
|
15 |
+
}
|
images/carn_2_4_compare.png
ADDED
![]() |
images/carn_4_4_compare.png
ADDED
![]() |
pytorch_model_4x.pt
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:374799ff654d4242a6439592aaa29a676b5c43a2bcf93ef19d275637042503e8
|
3 |
+
size 6406493
|