Assorted Neural Style Transfer: An extension of Vanilla NST
Tags: python, deeplearning, tutorial, programming
Published: 2022-07-02 15:27:58

**_Neural Style Transfer_** explores methods for artistic style transfer based on Convolutional Neural Networks. The core idea proposed by [Gatys et al.](https://arxiv.org/abs/1508.06576) became very popular, and with further research, [Johnson et al.](https://arxiv.org/abs/1603.08155) overcame a significant limitation to achieve style transfer in real-time.

This article uses the VGG16 model for [manual implementation](https://harishnarayanan.org/writing/artistic-style-transfer/) of Neural Style Transfer. Another implementation is based on an [article](https://www.tensorflow.org/tutorials/generative/style_transfer) by TensorFlow, which uses a pre-trained model for NST.

## So, what is Assorted Neural Style Transfer? (Yes, I came up with that name myself 😅)

Well, as we know, NST combines the style of Style Image with the content of Content Image as follows:

![Vanilla NST](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/geep25gatek3h81q1g96.png)

We propose the **Assorted NST**, which combines the style of 3 Style Images with the content of Content Image. Below are a few examples:

![Input Images to Assorted NST](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/masjk4xetp6uh4l8oc0x.png)

![The output of Assorted NST (Style Weights = [0.3, 0.3, 0.4])](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/91bc833l87y21hwaq0sa.png)

We not only combine the three styles, but we can also control how much weight to give to which style. The above output was generated with weights _[0.3, 0.3, 0.4]_. The weights _[0.1, 0.1, 0.8]_ (where style 3 has more weight) will give the following output:
 
![The output of Assorted NST (Style Weights = [0.1, 0.1, 0.8])](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7o2hef3y3bzsnbo5sec7.png)

## So, how does the Assorted NST work??

It's pretty straightforward. Instead of giving the model a single style image as input, we take the weighted combination of all three style images and feed that to the model. Before taking the weighted combination, we resize the style images to have the exact dimensions.

{% embed https://gist.github.com/ManishAradwad/233b678fd8a0af34c2f1f2ead1585ff0 %}

In this way, the model can extract the style of the corresponding final image, which can be used for final image generation.

The above Assorted NST example is based on the [TF Hub's model](https://colab.research.google.com/drive/1zTUF0uKeAIv51V9cWxu8EAbKjBAlNkyv?usp=sharing), while below are some examples of [manual Assorted NST implementation](https://colab.research.google.com/drive/1fHHzQOzaO_F7nsJsq9qPCqofv-pFv5Wn?usp=sharing):

![Input Images to Assorted NST](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m5tcqnn3bfkiv6ho93yc.png)

![The output of Assorted NST (Style Weights = [0.35, 0.1, 0.45])](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k77jeojvo4t5nwhs6e6i.png)

![The output of Assorted NST (Style Weights = [0.75, 0.1, 0.15])](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hg6s07c9uynjvfu4iwyf.png)

## Few Limitations of this method:

- For each set of content and style images, we have to do fine variations in weight values for the output to be better. It is impossible to have a fixed set of weights that work on all images. If the weights are not proper, then the outcome might be invalid like below:

![Not Good Outputs!](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rb32b3q1mqqa76a7mqa7.png)

- The time taken for output image generation is almost 8 seconds per iteration in manual implementation. We need at least ten iterations to get valid output. This can further be reduced using an end-to-end CNN model explicitly built for NST as introduced in Johnson et al. (which is used in TFHub implementation).

## _Thanks for reading!_

 
**References:**

1. [Mixed Neural Style Transfer](https://towardsdatascience.com/mixed-neural-style-transfer-with-two-style-images-9469b2681b54)
2. [Manual Implementation of NST](https://harishnarayanan.org/writing/artistic-style-transfer/)
3. [TFHub's NST Tutorial](https://www.tensorflow.org/tutorials/generative/style_transfer)