The layers for Neural Network in PyTorch
Tags: pytorch, layer, neuralnetwork, function
Published: 2024-06-10 11:30:55

A layer is a collection of nodes to do a specific task.

Basically, a Neural Network(NN) consists of 3 layers as shown below:

- Input Layer:
  - is the 1st layer which accepts data and pass it to a hidden layer.

- Hidden Layer:
  - is the layer between an input and output layer.
  - can be zero or more hidden layers in a neural network.

- Output layer:
  - is the last layer which holds a result.

*There is the Single Layer Neural Network or Perceptron which only has an input and output layer without hidden layer.

And, there are popular kinds of layers as shown below:

*Memos:
- These layers below can be used as an input, hidden or output layer.
- Some layers are also called Neural Network.

(1) Fully-connected Layer:
- connects every neuron in one layer to every neuron in the next layer.
- is also called Linear Layer, Dense Layer or Affine Layer.
- is [Linear()](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear) in PyTorch.

(2) Convolutional Layer:
- can make data stand out by extracting the features from the data with filters(kernels). *Extracting the features from the data also downsamples and reduce the data to reduce computation.
- is used for image processing.
- is [Conv1d()](https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html#torch.nn.Conv1d), [Conv2d()](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html) or [Conv3d()](https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html#torch.nn.Conv3d) in PyTorch.

*The Neural Network with Convolutional Layer is called Convolutional Neural Network(CNN).

(3) Transposed Convolutional Layer:
- can upsample data.
- is used for image processing.
- is similar to Deconvolutional Layer but not the same.
- is [ConvTranspose1d()](https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose1d.html#torch.nn.ConvTranspose1d), [ConvTranspose2d()](https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html#torch.nn.ConvTranspose2d) or [ConvTranspose3d()](https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose3d.html#torch.nn.ConvTranspose3d) in PyTorch.

(4) Pooling Layer:
- can downsample and reduce data to reduce overfitting and computation. *The way of downsampling data is different from Convolutional Layer.
- is used for image processing.
- is [MaxPool1d()](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool1d.html#torch.nn.MaxPool1d), [MaxPool2d()](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html) or [MaxPool3d()](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool3d.html#torch.nn.MaxPool3d) in PyTorch.

(5) Recurrent Layer:
- can remember the output of the previous step to use it in the next step.
- is used for NLP(Natural Language Processing). *NLP is the technology which enables computers to understand and communicate with human language.
- is also called Recurrent Neural Network(RNN).
- is [RNN()](https://pytorch.org/docs/stable/generated/torch.nn.RNN.html) in PyTorch.

(6) LSTM(Long Short-Term Memory) layer:
  - can remember the output of the previous steps to use them in the next step while Recurrent Layer can remember the output of only the previous step.
- is the improved version of Recurrent Layer.
- has more ability than Recurrent Layer.
- is slower to train data than Recurrent Layer because the computation is more complex.
- is used for NLP.
- is [LSTM()](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM) in PyTorch.

(7) Batch Normalization layer:
- can normalize each feature within a mini-batch of samples(layers) to be similar scale to accelerate(speed up) training. 
- is unstable with small batch sizes, then it leads to increased train time.
- is not good with RNN.
- is [BatchNorm1d()](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html#torch.nn.BatchNorm1d), [BatchNorm2d()](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d) or [BatchNorm3d()](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm3d.html#torch.nn.BatchNorm3d) in PyTorch.

(8) Layer Normalization:
- can normalize all features within each sample(layer) to be similar scale to accelerate(speed up) training.
- is the improved version of Batch Normalization layer.
- is stable with small batch sizes, then it doesn't lead to increased train time.
- is good with RNN.
- is [LayerNorm()](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html#torch.nn.LayerNorm) in PyTorch.

(9) Dropout Layer:
- can reduce overfitting by randomly dropping out nodes during training.
- is [Dropout()](https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html#torch.nn.Dropout) in PyTorch.

(10) Embedding Layer:
- can convert categorical data to numerical data.
- is used for NLP.
- is [Embedding()](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#torch.nn.Embedding) in PyTorch.