Transformers
HAN
super-image
image-super-resolution
Inference Endpoints
Eugene Siow commited on
Commit
0cdc0d6
1 Parent(s): 02bef2c

Initial commit.

Browse files
README.md ADDED
@@ -0,0 +1,140 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - super-image
5
+ - image-super-resolution
6
+ datasets:
7
+ - eugenesiow/Div2k
8
+ - eugenesiow/Set5
9
+ - eugenesiow/Set14
10
+ - eugenesiow/BSD100
11
+ - eugenesiow/Urban100
12
+ metrics:
13
+ - pnsr
14
+ - ssim
15
+ ---
16
+ # Holistic Attention Network (HAN)
17
+ HAN model pre-trained on DIV2K (800 images training, augmented to 4000 images, 100 images validation) for 2x, 3x and 4x image super resolution. It was introduced in the paper [Single Image Super-Resolution via a Holistic Attention Network](https://arxiv.org/abs/2008.08767) by Niu et al. (2020) and first released in [this repository](https://github.com/wwlCape/HAN).
18
+
19
+ The goal of image super resolution is to restore a high resolution (HR) image from a single low resolution (LR) image. The image below shows the ground truth (HR), the bicubic upscaling and model upscaling.
20
+
21
+ ![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4](images/han_4_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4")
22
+ ## Model description
23
+ Informative features play a crucial role in the single image super-resolution task. Channel attention has been demonstrated to be effective for preserving information-rich features in each layer. However, channel attention treats each convolution layer as a separate process that misses the correlation among different layers. To address this problem, we propose a new holistic attention network (HAN), which consists of a layer attention module (LAM) and a channel-spatial attention module (CSAM), to model the holistic interdependencies among layers, channels, and positions. Specifically, the proposed LAM adaptively emphasizes hierarchical features by considering correlations among layers. Meanwhile, CSAM learns the confidence at all the positions of each channel to selectively capture more informative features. Extensive experiments demonstrate that the proposed HAN performs favorably against the state-of-the-art single image super- resolution approaches.
24
+ ## Intended uses & limitations
25
+ You can use the pre-trained models for upscaling your images 2x, 3x and 4x. You can also use the trainer to train a model on your own dataset.
26
+ ### How to use
27
+ The model can be used with the [super_image](https://github.com/eugenesiow/super-image) library:
28
+ ```bash
29
+ pip install super-image
30
+ ```
31
+ Here is how to use a pre-trained model to upscale your image:
32
+ ```python
33
+ from super_image import HanModel, ImageLoader
34
+ from PIL import Image
35
+ import requests
36
+
37
+ url = 'https://paperswithcode.com/media/datasets/Set5-0000002728-07a9793f_zA3bDjj.jpg'
38
+ image = Image.open(requests.get(url, stream=True).raw)
39
+
40
+ model = HanModel.from_pretrained('eugenesiow/han', scale=2) # scale 2, 3 and 4 models available
41
+ inputs = ImageLoader.load_image(image)
42
+ preds = model(inputs)
43
+
44
+ ImageLoader.save_image(preds, './scaled_2x.png') # save the output 2x scaled image to `./scaled_2x.png`
45
+ ImageLoader.save_compare(inputs, preds, './scaled_2x_compare.png') # save an output comparing the super-image with a bicubic scaling
46
+ ```
47
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Upscale_Images_with_Pretrained_super_image_Models.ipynb "Open in Colab")
48
+ ## Training data
49
+ The models for 2x, 3x and 4x image super resolution were pretrained on [DIV2K](https://huggingface.co/datasets/eugenesiow/Div2k), a dataset of 800 high-quality (2K resolution) images for training, augmented to 4000 images and uses a dev set of 100 validation images (images numbered 801 to 900).
50
+ ## Training procedure
51
+ ### Preprocessing
52
+ We follow the pre-processing and training method of [Wang et al.](https://arxiv.org/abs/2104.07566).
53
+ Low Resolution (LR) images are created by using bicubic interpolation as the resizing method to reduce the size of the High Resolution (HR) images by x2, x3 and x4 times.
54
+ During training, RGB patches with size of 64×64 from the LR input are used together with their corresponding HR patches.
55
+ Data augmentation is applied to the training set in the pre-processing stage where five images are created from the four corners and center of the original image.
56
+
57
+ We need the huggingface [datasets](https://huggingface.co/datasets?filter=task_ids:other-other-image-super-resolution) library to download the data:
58
+ ```bash
59
+ pip install datasets
60
+ ```
61
+ The following code gets the data and preprocesses/augments the data.
62
+
63
+ ```python
64
+ from datasets import load_dataset
65
+ from super_image.data import EvalDataset, TrainDataset, augment_five_crop
66
+
67
+ augmented_dataset = load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='train')\
68
+ .map(augment_five_crop, batched=True, desc="Augmenting Dataset") # download and augment the data with the five_crop method
69
+ train_dataset = TrainDataset(augmented_dataset) # prepare the train dataset for loading PyTorch DataLoader
70
+ eval_dataset = EvalDataset(load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='validation')) # prepare the eval dataset for the PyTorch DataLoader
71
+ ```
72
+ ### Pretraining
73
+ The model was trained on GPU. The training code is provided below:
74
+ ```python
75
+ from super_image import Trainer, TrainingArguments, HanModel, HanConfig
76
+
77
+ training_args = TrainingArguments(
78
+ output_dir='./results', # output directory
79
+ num_train_epochs=1000, # total number of training epochs
80
+ )
81
+
82
+ config = HanConfig(
83
+ scale=4, # train a model to upscale 4x
84
+ )
85
+ model = HanModel(config)
86
+
87
+ trainer = Trainer(
88
+ model=model, # the instantiated model to be trained
89
+ args=training_args, # training arguments, defined above
90
+ train_dataset=train_dataset, # training dataset
91
+ eval_dataset=eval_dataset # evaluation dataset
92
+ )
93
+
94
+ trainer.train()
95
+ ```
96
+
97
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Train_super_image_Models.ipynb "Open in Colab")
98
+ ## Evaluation results
99
+ The evaluation metrics include [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio#Quality_estimation_with_PSNR) and [SSIM](https://en.wikipedia.org/wiki/Structural_similarity#Algorithm).
100
+
101
+ Evaluation datasets include:
102
+ - Set5 - [Bevilacqua et al. (2012)](https://huggingface.co/datasets/eugenesiow/Set5)
103
+ - Set14 - [Zeyde et al. (2010)](https://huggingface.co/datasets/eugenesiow/Set14)
104
+ - BSD100 - [Martin et al. (2001)](https://huggingface.co/datasets/eugenesiow/BSD100)
105
+ - Urban100 - [Huang et al. (2015)](https://huggingface.co/datasets/eugenesiow/Urban100)
106
+
107
+ The results columns below are represented below as `PSNR/SSIM`. They are compared against a Bicubic baseline.
108
+
109
+ |Dataset |Scale |Bicubic |han |
110
+ |--- |--- |--- |--- |
111
+ |Set5 |2x |33.64/0.9292 |**** |
112
+ |Set5 |3x |30.39/0.8678 |**** |
113
+ |Set5 |4x |28.42/0.8101 |**31.21/0.8778** |
114
+ |Set14 |2x |30.22/0.8683 |**** |
115
+ |Set14 |3x |27.53/0.7737 |**** |
116
+ |Set14 |4x |25.99/0.7023 |**28.18/0.7712** |
117
+ |BSD100 |2x |29.55/0.8425 |**** |
118
+ |BSD100 |3x |27.20/0.7382 |**** |
119
+ |BSD100 |4x |25.96/0.6672 |**28.09/0.7533** |
120
+ |Urban100 |2x |26.66/0.8408 |**** |
121
+ |Urban100 |3x | |**** |
122
+ |Urban100 |4x |23.14/0.6573 |**25.1/0.7497** |
123
+
124
+ ![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2](images/han_2_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2")
125
+
126
+ You can find a notebook to easily run evaluation on pretrained models below:
127
+
128
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Evaluate_Pretrained_super_image_Models.ipynb "Open in Colab")
129
+
130
+ ## BibTeX entry and citation info
131
+ ```bibtex
132
+ @misc{niu2020single,
133
+ title={Single Image Super-Resolution via a Holistic Attention Network},
134
+ author={Ben Niu and Weilei Wen and Wenqi Ren and Xiangde Zhang and Lianping Yang and Shuzhen Wang and Kaihao Zhang and Xiaochun Cao and Haifeng Shen},
135
+ year={2020},
136
+ eprint={2008.08767},
137
+ archivePrefix={arXiv},
138
+ primaryClass={eess.IV}
139
+ }
140
+ ```
config.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "data_parallel": false,
3
+ "model_type": "HAN",
4
+ "n_colors": 3,
5
+ "n_feats": 64,
6
+ "n_resblocks": 20,
7
+ "n_resgroups": 10,
8
+ "reduction": 16,
9
+ "res_scale": 1,
10
+ "rgb_mean": [
11
+ 0.4488,
12
+ 0.4371,
13
+ 0.404
14
+ ],
15
+ "rgb_range": 255,
16
+ "rgb_std": [
17
+ 1.0,
18
+ 1.0,
19
+ 1.0
20
+ ]
21
+ }
images/han_2_4_compare.png ADDED
images/han_4_4_compare.png ADDED
pytorch_model_4x.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2d6294fbae3a343100c135eeebbf9f585a54ac0924cb25330b7008e4a6e2bd51
3
+ size 64935663