The activation functions for Neural Network in PyTorch
Tags: python, activationfunction, deeplearning, machinelearning
Published: 2024-06-14 04:42:36

An activation function is the function or layer which enables neural network to learn complex(non-linear) relationships by transforming the output of the previous layer. *Without activation functions, neural network can only learn linear relationships.

There are popular activation functions as shown below:

(1) Step Function:
- can convert input values to 0 or 1. *If `input` < 0, then 0 while if `input` >= 0, then 1.
- is also called Binary Step Function, Unit Step Function, Binary Threshold Function, Threshold Function, Heaviside Step Function or Heaviside Function.
- is [heaviside()](https://pytorch.org/docs/stable/generated/torch.heaviside.html) in PyTorch.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/upex28ivo643lcgrxrjw.png)

(2) ReLU(Rectified Linear Unit) Function:
- can convert input values to the output values between 0 and `input`. *If `input` <= 0, then 0 while if `input` > 0, then `input`.
- is [ReLU()](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html#torch.nn.ReLU) in PyTorch.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f9s1vtqekki2relzoxpk.png)

(3) Leaky ReLU(Leaky Rectified Linear Unit) Function:
- can convert input values to the output values between `input` * `slope` and `input`.
  *Memos:
  - If `input` <= 0, then `input` * `slope` while if `input` > 0, then `input`.
  - `slope` is 0.01 basically.
- is the improved version of ReLU Function.
- is [LeakyReLU()](https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU.html#torch.nn.LeakyReLU) in PyTorch.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rbnfrnjo41vv4d40geu9.png)

(4) ELU(Exponential Linear Unit) Function:
- can convert input values to the output values between `a` * (e<sup>`input`</sup> - 1) and `input`.
  *Memos:
  - If `input` <= 0, then `a` * (e<sup>`input`</sup> - 1) while if `input` > 0, then `input`.
  - `a` is 1.0 basically.
- is the improved version of ReLU Function.
- is [ELU()](https://pytorch.org/docs/stable/generated/torch.nn.ELU.html#torch.nn.ELU) in PyTorch.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mdrns9tq6oab7nreldeq.png)

(5) Sigmoid Function:
- can convert input values to the output values between 0 and 1.
- 's formula is y = 1 / (1 + e<sup>-x</sup>).
- is [Sigmoid()](https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html) in PyTorch.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9s4ddptkwlfl64mimuxm.png)

(6) Tanh Function:
- can convert input values to the output values between -1 and 1.
- 's formula is y = (e<sup>x</sup> - e<sup>-x</sup>) / (e<sup>x</sup> + e<sup>-x</sup>).
- is also called Hyperbolic Tangent Function.
- is [Tanh()](https://pytorch.org/docs/stable/generated/torch.nn.Tanh.html#torch.nn.Tanh) in PyTorch.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h5qaljkj9xayyll1lceu.png)

(7) Softmax Function:
- can convert input values to the output values between 0 and 1 each and whose sum is 1(100%). *If input values are [5, 4, -1], then the output values are [0.730, 0.268, 0.002] which is 0.730(73%) + 0.268(26.8%) + 0.002(0.2%) = 1(100%).
- 's formula is:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1uvqkizh1bpzmga1ylog.png)
- is [Softmax()](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html#torch.nn.Softmax) in PyTorch.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3bkweibyvz15yz1eka32.png)