A reduced explanation on neural networks
Tags: deeplearning, machinelearning, python, beginners
Published: 2021-03-25 19:26:57

<!--
We may compare deep learning and neural networks to <b>a linear fit</b>:

 <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/syrr9emb11d4wgm3j61s.jpg" alt="drawing" width="300"/> -->

In a linear fit, data is fitted to a linear equation, typically to the slope-intercept form:

{% katex %}  y = w_1 x + w_0 {% endkatex %}

<figure>
 <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h1o96pmqaz0gaip7e8so.png">
  <figcaption>A linear fit (red) of the data (in blue) with the above equation. Such equation is better known as y = m x + b, but it has defined as above for context.</figcaption>
</figure>
<br>
<br>

Likewise, for a multi-linear equation with a total number of {% katex inline %} D {% endkatex %} independent variables, the expression is:

{% katex %}  y = w_1 x_1 + w_2 x_2 + \cdots + w_D x_D + w_0 {% endkatex %}

Either of the above equations can be fitted by changing the {% katex inline %} w_i {% endkatex %} weights in a way that <i>minimizes</i> the error between the data points and the theoretical points generated by the equation. This error metric is typically known as a <a href="https://en.wikipedia.org/wiki/Loss_function"><b>loss function</b> or <b>cost function</b></a>. 

In the case of deep learning, the equation is more complex. The multilayer perceptron (MLP) model is used as an example here for simplicity, as it may be one of the simplest neural networks to explain. Its architecture is shown below:

<!--
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bcw1jtikg6cvi536to7x.png" alt="drawing" width="400"/>
-->

<figure>
 <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/58on25580qjxa6x6ds2x.png">
  <figcaption>A multilayer perceptron (MLP) neural network.</figcaption>
</figure>
<br>
<br>

<!--
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/58on25580qjxa6x6ds2x.png)
-->

In here, each column is a <i>layer</i> of vertical nodes, where the first layer from the left has nodes containing the  input {% katex inline %} x_i {% endkatex %} variables containing the information of each {% katex inline %} i {% endkatex %} pixel in this example. The layers residing between the first and last layers in the network are known as hidden layers as the flow of operations in these sectors is typically not disclosed by the machine and is thus hidden to the user. The nodes in these hidden layers are also typically known as neurons as they handle most of the {% katex inline %} w_i {% endkatex %} coefficients adjustments during the training phase. Finally, the last layer contains the output nodes, which give rise to the dependent variables of the model. 

<!-- 
In short, a neural network can be seen as <b>a path</b> to the output variables from an optimization of multiple {% katex inline %} w_i {% endkatex %} weights correlated to the inputs.
-->

It is helpful to get a notion of the connectivity of this topology to understand how the network operates. Here, the connections to 2 different <b>nodes</b> of the above network will be explored. First, let's look at the first node of the first hidden layer:

<!--
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jhv6vmosr287jvlxad7d.jpg" alt="drawing" width="500"/>
-->

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nzqrte7hs5fyh3yzyrek.png)

The connections of the 4 input variables to the first node of the first hidden layer is a linear combination of the first four variables to such node, and can be expressed as:

<!--
n<sub>1-</sub> = w<sub>11-</sub> x<sub>1</sub> + w<sub>21-</sub> x<sub>2</sub> + w<sub>31-</sub> x<sub>3</sub> + w<sub>41-</sub> x<sub>4</sub> 
-->

{% katex %}  n_1 l_1 = w_{1,n_1 l_1}x_1 + w_{2,n_1 l_1}x_2 + w_{3,n_1 l_1}x_3 + w_{4,n_1 l_1}x_4 {% endkatex %}

Now let's look at the connections for the following node in the second hidden layer:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n7fzju1snjcs2jwtmhe4.png)

The connections of the 6 hidden nodes from the first hidden layer to the first node of the second hidden layer is a linear combination of the latter 6 hidden nodes. This new node can then be defined by the below equation. The node previously defined in the above equation has been bolded for context.  

{% katex %}  n_1 l_2 = w_{n1,n_1 l_2} \bm{n_1 l_1} + w_{n2,n_1 l_2} n_2 l_1 + w_{n3,n_1 l_2} n_3 l_1  + w_{n4,n_1 l_2} n_4 l_1 + w_{n5,n_1 l_2} n_5 l_1 + w_{n6,n_1 l_2} n_6 l_1 {% endkatex %}

<!--
n<sub>1=</sub> = w<sub>1-1=</sub> n<sub>1</sub> + w<sub>2-1=</sub> n<sub>2-</sub> + w<sub>3-1=</sub> n<sub>3-</sub> + w<sub>4-1=</sub> n<sub>4-</sub> + w<sub>5-1=</sub> n<sub>5-</sub> + w<sub>6-1=</sub> n<sub>6-</sub> 
-->

> _”MLP neural networks are linear combinations of recursively-nested linear combinations.”_

The number of nested operations depend on the depth [the number of layers of the network] and the number of variables and nodes in the system.
			
In a similar manner as with the linear equation fit, deep learning can be seen as <i> the process done to minimize the cost or loss function of a deep learning neural network </i> given the inputs and the shape of the network.


## Images

Image classification and image recognition problems are widely taught in deep learning. The popular image recognition problem of "cat and dog" identification from a library of these animals occurs through a process analogous to the following heuristic.  

For this example, let's use the following figure of a puppy as a "shiba.png" file (right-click Save Image As...):


<figure>
  <a href="https://shibaimage.herokuapp.com/"><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cosv5tbzelpezy1snwn4.png"></a>
  <figcaption>Left-click above image for a graphical representation of the <i>"dog"</i> array</figcaption>
</figure>
<br>
<br>

The .png image may be imported to Python in the following way:

```ruby 
import numpy as np
import cv2 as vision       #import computer vision module 
#read color image and declare it as (LxWx(rgb_color_channels(3))) array 
dog = vision.imload("shiba.png")   
# we can also decrease the complexity of the image by looking at the pixel intensities of the first channel, the "red" channel: 
dog = dog[:,:,0] 
dog = dog.reshape((*dog.shape[:2]))  # L X W array
```

The resulting image is now a 180 x 180 pixels array. To correlate the image in an analogous manner to the linear regressions made with the slope intercept form equation, each of these pixels must be made dependent to the inputs in the neural network. This can be done by vectorizing the above image array with the "flatten()" operation:

```ruby 
X = dog.flatten()
```

The {% katex inline %} \mathbf{X} {% endkatex %} variable above is thus a 1-D vector with a length of 32,400 pixels. 

<figure>
  <a href="https://shibaimage-flat.herokuapp.com/"><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/km41bkd1c4jaqtgd2wuk.png"></a>
  <figcaption>Click above image for a graphical representation of the vectorized <i>"dog"</i> array</figcaption>
</figure>
<br>
<br>

After this transformation, each pixel can be more easily assigned to each of the nodes on the dense layer(s) of the neural network, which is fitted with a weight {% katex inline %} w_i {% endkatex %}. The vector {% katex inline %} \mathbf{W} {% endkatex %} containing these weights must be of the same size as the number of pixels of this vectorized image. As an example, let's make these weights very small random numbers: 

```ruby 
W = np.random.random(dog.shape[0])/10000
```
The value of a <b>single</b> {% katex inline %} y {% endkatex %} data point is then the dot product between this vector and the vectorized image:
```ruby 
y = np.sum(W*X) 
print(y)
```
This value is analogous to calculating a singular {% katex inline %} y {% endkatex %}  value with the multi-linear equation shown at the beginning of this post with a random set of {% katex inline %} w_i {% endkatex %} weights. As this value is only a single data point, it is evident that more than a single image is needed to make a proper fit for a cat-dog identification model.

Having the network properly identify this image as the drawing of a dog or cat will require passing the neural network through many images containing dogs as well as cats, allowing the network to adjust the weights associated with each pixel in the images to generate an adequate {% katex inline %} y {% endkatex %} variable . In order to classify qualitatively, a logistic function (a "sigmoid") is typically used with the regression. 

This is a reduced explanation on a simple neural network model. For a deeper understanding on this concept, here are some good online references:

* Ian Goodfellow et. al. <a href="https://www.deeplearningbook.org/">Deep Learning</a> 

* Michael Nielsen <a href="http://neuralnetworksanddeeplearning.com/">Neural Networks and Deep Learning</a>  

* Adrian Rosebrock <a href="https://www.deeplearningbook.org/">pyimagesearch</a> 

-Andrew