Super-Resolution Using Autoencoders and TF2.0
Published: 2020-05-20 07:21:58

Through this post, I will be showing how to make an autoencoder network that can super resolve an image to 4x and also ways to increase it to 10x and maybe even further. This was a project landed me the first prize in a competition organized by my high school.
<br>
<hr/>
<h1>Why use an Autoencoder?</h1>
There are many reasons you might want to use this type of network, one example is transferring image in a chat app. Chat apps like WhatsApp and Instagram use this type of compression, although more complex and trained for a longer time and with a lot more data, they use because due to compression of the image the amount of data to upload goes down, making the upload speeds go up and also the amount of data to store in servers go down without any loss of image quality.

<a href="https://colab.research.google.com/drive/1YXy_9B4h-NC7KU_40aP5U2SgVCVw4H_D">Link to the Project Notebook</a>
<br>
<hr/>
<h1>Overview</h1>
First of all, I will be walking you through building the encoder part of the network individually and then making both encoder and decoder, connecting it and training it with custom data and throughout this, I will also be explaining the autoencoder neural network and ways to improve it.
<hr/>
<h2>Libraries to Import</h2>
We will be using Tensorflow 2.0 for building the network, Urllib module for downloading the images, Numpy for processing image array, Open-Cv2 for resizing image, Os module for loading images and Matplotlib for showing images.
<hr/>
<h2>What are Autoencoders?</h2>
An Autoencoder is just an unsupervised neural network that by design learns how to reduce data and reconstruct it losing as little data as possible.

It consists of an encoder and a decoder as you might have guessed. The role of the encoder is to compress the data to a lower dimension in a way that only the most influential features persist like PCA and the role of the decoder is to reconstruct the data from the lower-dimensional representation to as close it can get to the original.
<hr/>
<h2>Building the Encoder</h2>

<hr/>
<div style="background-color:black;color:white;padding:20px;">
import tensoflow as tf

input_img = tf.keras.layers.Input(shape=(256, 256, 3))

l1 = tf.keras.layers.Conv2D(64, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(input_img)
l2 = tf.keras.layers.Conv2D(64, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l1)
l3 = tf.keras.layers.MaxPool2D(padding='same')(l2)

l4 = tf.keras.layers.Conv2D(128, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l3)
l5 = tf.keras.layers.Conv2D(128, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l4)
l6 = tf.keras.layers.MaxPool2D(padding='same')(l5)

l7 = tf.keras.layers.Conv2D(256, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l6)

encoder_model = tf.keras.models.Model(inputs=(input_img), outputs=l7)
</div>
<hr/>
Here we have 2 Conv2D blocks as we have images of only 256 pixels of height and width. However, when using images of 1920 X 1080 pixels I would suggest using more than 4 Conv2D blocks with filter increasing in 2 folds. This will ensure that there are enough parameters to change and enough filters that can ultimately be used to super resolve an image up to 10x.
Here is the model summary.

<img src="https://user-images.githubusercontent.com/44964331/82416031-0900ad80-9a97-11ea-9b77-448728334fe8.png"/>

As you can see, the number of filters we have is 256, we could have stopped at 128 or 64 but we do not have a lot of images to train with, also the real dimensions of the image are just 256X 256 and due to this, we have to compensate for it by using a lot of filters. Though in real life, the image will be 1920 X 1080 and thus the compression to 192 X 108 will be much more apparent.
Image to better understand the flow of encoder

<img src="https://miro.medium.com/max/381/1*XEV1KsKANbjJ5Z_X3XwOCA.png"/>

<hr/>

<h2> Whole Model </h2>
<hr/>
<div style="background-color:black;color:white;padding:20px;">
import tensorflow as tf

input_img = tf.keras.layers.Input(shape=(256, 256, 3))

l1 = tf.keras.layers.Conv2D(64, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(input_img)
l2 = tf.keras.layers.Conv2D(64, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l1)
l3 = tf.keras.layers.MaxPool2D(padding='same')(l2)

l4 = tf.keras.layers.Conv2D(128, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l3)
l5 = tf.keras.layers.Conv2D(128, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l4)
l6 = tf.keras.layers.MaxPool2D(padding='same')(l5)

l7 = tf.keras.layers.Conv2D(256, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l6)

l8 = tf.keras.layers.UpSampling2D()(l7)
l9 = tf.keras.layers.Conv2D(128, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l8)
l10 = tf.keras.layers.Conv2D(128, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l9)

l11 = tf.keras.layers.add([l10, l5])

l12 = tf.keras.layers.UpSampling2D()(l11)
l13 = tf.keras.layers.Conv2D(64, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l12)
l14 = tf.keras.layers.Conv2D(64, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l13)

l15 = tf.keras.layers.add([l14, l2])

decoded_image = tf.keras.layers.Conv2D(3, (3, 3), padding='same', kernel_initializer='he_uniform', activation='relu', activity_regularizer=tf.keras.regularizers.l1(10e-10))(l15)

auto_encoder = tf.keras.models.Model(inputs=(input_img), outputs=decoded_image)

auto_encoder.compile(optimizer='adadelta', loss='mean_squared_error')
</div>
<hr/>
As you can see, we are using upsampling instead of Conv2D Transpose, this is since Conv2D transpose has trainable parameters while being beneficial for our task, it increases training time and the little amount of data we have, it wasn’t needed but surely for production-level models, Conv2D Transpose layer instead of Upsampling2D layer is a must.

Here is the summary,
<img src="https://user-images.githubusercontent.com/44964331/82416299-6d237180-9a97-11ea-82f7-5d8459ad5f04.png"/>

As you might have noticed, we are using the residual connection between encoder and decoder, this is to diminish the lossy results of the final reconstructed image, this may or may not be needed as per your training data amount, making it optional.

The residual connection is more apparent in this diagram:
<img src="https://miro.medium.com/max/594/1*PI_S9eFtjmjFcAUrDYHV6w.png"/>

As you might have noticed, we are using mean squared error as loss function which is maybe the worst but the fastest and this is why I have used it here.
However I suggest not to use it for production, instead use Perpetual loss, it is leaps better than MSE. Research paper explaining the loss can be found here.

<hr/>

<h2> Getting Data and Training Method </h2>
I have retrieved car images from image net using Urllib and then used OpenCV to resize the image to 1/4th of its size. Using the resized image with a lot of data loss as input and super high-resolution original image as the image to compare loss with.

This way the model will slowly learn how to super resolve an image to 4x its native resolution by optimizing the loss between the reconstructed and the original image.
<hr/>

<h2>Final Result</h2>

Input Image:
<img src="https://miro.medium.com/max/328/1*N3OrTRbENme50kMFj8oGyg.png"/>

Reconstructed Image:
<img src="https://miro.medium.com/max/328/1*F9GTK-F5oAz3JIrzHvlj1w.png"/>

<b>As you can see, our Autoencoder is doing an amazing job!</b>

<hr/>

<h1>Places to Improve</h1>
The following are the ways you can use to increase the super-resolution to 10x and maybe even more!

<ol>
<li>The most important point would be to use Full HD images to train the model, also use more than 5000 images.</li>
<l1>Use Conv2D Transpose layers rather than Upscaling layers.</li>
<li>Use <a href="https://arxiv.org/abs/1603.08155">Perceptual Loss</a> instead of MSE.</li>
<li>Use Adam optimizer instead of Adadelta.</li>
<li>Use <a href="https://arxiv.org/abs/1909.11862">Convolutional Layers with Dynamic Regularization</a>.</li>