# eugenesiow /drln-bam

 1 --- 2 license: apache-2.0 3 tags: 4 - super-image 5 - image-super-resolution 6 datasets: 7 - eugenesiow/Div2k 8 - eugenesiow/Set5 9 - eugenesiow/Set14 10 - eugenesiow/BSD100 11 - eugenesiow/Urban100 12 metrics: 13 - pnsr 14 - ssim 15 --- 16 # Densely Residual Laplacian Super-Resolution (DRLN) 17 DRLN model pre-trained on DIV2K (800 images training, augmented to 4000 images, 100 images validation) for 2x, 3x and 4x image super resolution. It was introduced in the paper [Densely Residual Laplacian Super-resolution](https://arxiv.org/abs/1906.12021) by Anwar et al. (2020) and first released in [this repository](https://github.com/saeed-anwar/DRLN). 18 19 The goal of image super resolution is to restore a high resolution (HR) image from a single low resolution (LR) image. The image below shows the ground truth (HR), the bicubic upscaling and model upscaling. 20 21 ![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4](images/drln_4_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4") 22 ## Model description 23 Super-Resolution convolutional neural networks have recently demonstrated high-quality restoration for single images. However, existing algorithms often require very deep architectures and long training times. Furthermore, current convolutional neural networks for super-resolution are unable to exploit features at multiple scales and weigh them equally, limiting their learning capability. In this exposition, we present a compact and accurate super-resolution algorithm namely, Densely Residual Laplacian Network (DRLN). The proposed network employs cascading residual on the residual structure to allow the flow of low-frequency information to focus on learning high and mid-level features. In addition, deep supervision is achieved via the densely concatenated residual blocks settings, which also helps in learning from high-level complex features. Moreover, we propose Laplacian attention to model the crucial features to learn the inter and intra-level dependencies between the feature maps. Furthermore, comprehensive quantitative and qualitative evaluations on low-resolution, noisy low-resolution, and real historical image benchmark datasets illustrate that our DRLN algorithm performs favorably against the state-of-the-art methods visually and accurately. 24 25 This model also applies the balanced attention (BAM) method invented by [Wang et al. (2021)](https://arxiv.org/abs/2104.07566) to further improve the results. 26 ## Intended uses & limitations 27 You can use the pre-trained models for upscaling your images 2x, 3x and 4x. You can also use the trainer to train a model on your own dataset. 28 ### How to use 29 The model can be used with the [super_image](https://github.com/eugenesiow/super-image) library: 30 bash 31 pip install super-image 32  33 Here is how to use a pre-trained model to upscale your image: 34 python 35 from super_image import DrlnModel, ImageLoader 36 from PIL import Image 37 import requests 38 39 url = 'https://paperswithcode.com/media/datasets/Set5-0000002728-07a9793f_zA3bDjj.jpg' 40 image = Image.open(requests.get(url, stream=True).raw) 41 42 model = DrlnModel.from_pretrained('eugenesiow/drln-bam', scale=2) # scale 2, 3 and 4 models available 43 inputs = ImageLoader.load_image(image) 44 preds = model(inputs) 45 46 ImageLoader.save_image(preds, './scaled_2x.png') # save the output 2x scaled image to ./scaled_2x.png 47 ImageLoader.save_compare(inputs, preds, './scaled_2x_compare.png') # save an output comparing the super-image with a bicubic scaling 48  49 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Upscale_Images_with_Pretrained_super_image_Models.ipynb "Open in Colab") 50 ## Training data 51 The models for 2x, 3x and 4x image super resolution were pretrained on [DIV2K](https://huggingface.co/datasets/eugenesiow/Div2k), a dataset of 800 high-quality (2K resolution) images for training, augmented to 4000 images and uses a dev set of 100 validation images (images numbered 801 to 900). 52 ## Training procedure 53 ### Preprocessing 54 We follow the pre-processing and training method of [Wang et al.](https://arxiv.org/abs/2104.07566). 55 Low Resolution (LR) images are created by using bicubic interpolation as the resizing method to reduce the size of the High Resolution (HR) images by x2, x3 and x4 times. 56 During training, RGB patches with size of 64×64 from the LR input are used together with their corresponding HR patches. 57 Data augmentation is applied to the training set in the pre-processing stage where five images are created from the four corners and center of the original image. 58 59 We need the huggingface [datasets](https://huggingface.co/datasets?filter=task_ids:other-other-image-super-resolution) library to download the data: 60 bash 61 pip install datasets 62  63 The following code gets the data and preprocesses/augments the data. 64 65 python 66 from datasets import load_dataset 67 from super_image.data import EvalDataset, TrainDataset, augment_five_crop 68 69 augmented_dataset = load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='train')\ 70 .map(augment_five_crop, batched=True, desc="Augmenting Dataset") # download and augment the data with the five_crop method 71 train_dataset = TrainDataset(augmented_dataset) # prepare the train dataset for loading PyTorch DataLoader 72 eval_dataset = EvalDataset(load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='validation')) # prepare the eval dataset for the PyTorch DataLoader 73  74 ### Pretraining 75 The model was trained on GPU. The training code is provided below: 76 python 77 from super_image import Trainer, TrainingArguments, DrlnModel, DrlnConfig 78 79 training_args = TrainingArguments( 80 output_dir='./results', # output directory 81 num_train_epochs=1000, # total number of training epochs 82 ) 83 84 config = DrlnConfig( 85 scale=4, # train a model to upscale 4x 86 bam=True, # apply balanced attention to the network 87 ) 88 model = DrlnModel(config) 89 90 trainer = Trainer( 91 model=model, # the instantiated model to be trained 92 args=training_args, # training arguments, defined above 93 train_dataset=train_dataset, # training dataset 94 eval_dataset=eval_dataset # evaluation dataset 95 ) 96 97 trainer.train() 98  99 100 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Train_super_image_Models.ipynb "Open in Colab") 101 ## Evaluation results 102 The evaluation metrics include [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio#Quality_estimation_with_PSNR) and [SSIM](https://en.wikipedia.org/wiki/Structural_similarity#Algorithm). 103 104 Evaluation datasets include: 105 - Set5 - [Bevilacqua et al. (2012)](https://huggingface.co/datasets/eugenesiow/Set5) 106 - Set14 - [Zeyde et al. (2010)](https://huggingface.co/datasets/eugenesiow/Set14) 107 - BSD100 - [Martin et al. (2001)](https://huggingface.co/datasets/eugenesiow/BSD100) 108 - Urban100 - [Huang et al. (2015)](https://huggingface.co/datasets/eugenesiow/Urban100) 109 110 The results columns below are represented below as PSNR/SSIM. They are compared against a Bicubic baseline. 111 112 |Dataset |Scale |Bicubic |drln-bam | 113 |--- |--- |--- |--- | 114 |Set5 |2x |33.64/0.9292 |**38.23/0.9614** | 115 |Set5 |3x |30.39/0.8678 |**35.3/0.9422** | 116 |Set5 |4x |28.42/0.8101 |**32.49/0.8986** | 117 |Set14 |2x |30.22/0.8683 |**33.95/0.9206** | 118 |Set14 |3x |27.53/0.7737 |**31.27/0.8624** | 119 |Set14 |4x |25.99/0.7023 |**28.94/0.7899** | 120 |BSD100 |2x |29.55/0.8425 |**33.95/0.9269** | 121 |BSD100 |3x |27.20/0.7382 |**29.78/0.8224** | 122 |BSD100 |4x |25.96/0.6672 |**28.63/0.7686** | 123 |Urban100 |2x |26.66/0.8408 |**32.81/0.9339** | 124 |Urban100 |3x | |**29.82/0.8828** | 125 |Urban100 |4x |23.14/0.6573 |**26.53/0.7991** | 126 127 ![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2](images/drln_2_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2") 128 129 You can find a notebook to easily run evaluation on pretrained models below: 130 131 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Evaluate_Pretrained_super_image_Models.ipynb "Open in Colab") 132 133 ## BibTeX entry and citation info 134 bibtex 135 @misc{wang2021bam, 136 title={BAM: A Lightweight and Efficient Balanced Attention Mechanism for Single Image Super Resolution}, 137 author={Fanyi Wang and Haotian Hu and Cheng Shen}, 138 year={2021}, 139 eprint={2104.07566}, 140 archivePrefix={arXiv}, 141 primaryClass={eess.IV} 142 } 143  144 145 bibtex 146 @misc{anwar2019densely, 147 title={Densely Residual Laplacian Super-Resolution}, 148 author={Saeed Anwar and Nick Barnes}, 149 year={2019}, 150 eprint={1906.12021}, 151 archivePrefix={arXiv}, 152 primaryClass={eess.IV} 153 } 154