The loss functions for Neural Network in PyTorch
Tags: pytorch, lossfunction, neuralnetwork, function
Published: 2024-06-17 00:55:05

A loss function is the function which can get the mean(average) of the sum of the losses(differences) between a model's predictions and true values(train or test data) to optimize a model during training or to evaluate how good a model is during testing. *Loss function is also called Cost Function or Error Function.

There are popular loss functions as shown below:

(1) L1 Loss:
- can compute the mean(average) of the sum of the absolute losses(differences) between a model's predictions and true values(train and test data).
- 's formula is as shown below:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/20qsszmsf0bjexmh8p16.png)
- 's pros are as shown below:
  - Less sensitive to outliers.
  - We can easily complare the losses because they are just made absolute so the range of them is not big.
- 's cons are as shown below:
  - The absolute loss of |0| cannot be differentiable according to [this post](https://math.stackexchange.com/questions/991475/why-is-the-absolute-value-function-not-differentiable-at-x-0) and [this post](https://datascience.stackexchange.com/questions/71380/if-mean-absolute-loss-is-not-differentiable-how-it-can-be-used-in-neural-networ).
- is used for a regression model.
- is also called Mean Absolute Error(MAE).
- is [L1Loss()](https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html#torch.nn.L1Loss) in PyTorch.

(2) L2 Loss:
- can compute the mean(average) of the sum of the squared losses(differences) between a model's predictions and true values(train and test data).
- 's formula is as shown below:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/63wr94q5w1dkr132hiwq.png)
- 's pros are as shown below:
  - All squared losses can be differentiable.
- 's cons are as shown below:
  - Sensitive to outliers.
  - We cannot easily complare the losses because they are squared so the range of them is big.
- is used for a regression model.
- is also called Mean Squared Error(MSE).
- is [MSELoss()](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss) in PyTorch

(3) Huber Loss:
- can do the similar computation of either L1 Loss or L2 Loss depending on the absolute losses(differences) between a model's predictions and true values(train and test data) compared with `delta` which you set.
  *Memos:
  - `delta` is 1.0 basically.
  - Be careful, the computation is not exactly same as L1 Loss or L2 Loss according to the formulas below.
- 's formula is as shown below. *The 1st one is L2 Loss-like one and the 2nd one is L1 Loss-like one:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l90k9w1cswr768d9mzor.png)
- 's pros are as shown below:
  - Less sensitive to outliers.
  - All losses can be differentiable.
  - We can more easily complare the losses than L2 Loss because only small losses are squared so the range of them is smaller than L2 Loss.
- 's cons are as shown below:
  - The computation is more than L1 Loss and L2 Loss because the formula is more complex than them.
- is used for a regression model.
- is [HuberLoss()](https://pytorch.org/docs/stable/generated/torch.nn.HuberLoss.html#torch.nn.HuberLoss) in PyTorch.
- with `delta` of 1.0 is same as Smooth L1 Loss which is [SmoothL1Loss()](https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html) in PyTorch.

(4) BCE(Binary Cross Entropy) Loss:
- can compute the mean(average) of the sum of the losses(differences) between a model's binary predictions and true binary values(train and test data).
- s' formula is as shown below:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jigp72l8v0qakuwvp1km.png)
- is used for Binary Classification. *Binary Classification is the technology to classify data into two classes.
- is also called Binary Cross Entropy or Log(Logarithmic) Loss.
- is [BCELoss()](https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss) in PyTorch. 
  *Memos:
  - There is also [BCEWithLogitsLoss()](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html) which is the combination of [BCE Loss](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html) and [Sigmoid Activation Function](https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html) in PyTorch.
  - Sigmoid Activation Function suits BCE Loss to be more stable.

(5) Cross Entropy Loss:
- can compute the mean(average) of the sum of the losses(differences) between a model's predictions and true values(train and test data). *A loss is between 0 and 1.
- s' formula is as shown below:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xhp1a7wp6jbxwjir8l9i.png)
- is used for Multiclass Classification and Computer Vision.
  *Memos:
  - Multiclass Classification is the technology to classify data into multiple classes.
  - Computer vision is the technology which enables a computer to understand objects.
- is [CrossEntropyLoss()](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) in PyTorch.