PyTorch :: Performing Derivatives on Tensors
Tags: ai, pytorch, python, neuralnetworks
Published: 2021-01-06 17:28:57

Hello everyone, 

Especially to those who fear from maths :P

In this you will learn how to perform derivatives on tensors using pytorch. Let me tell you, once you will learn, it will be fun for you.

Basically the derivatives are used for generating the parameters in the neural network. For now you will learn the basics of derivatives.

Suppose you have a function {% katex inline %} f(x) = x^2 {% endkatex %}, so according to calculus, the derivative of {% katex inline %} f(x) {% endkatex %} would be {% katex inline %}\frac{d f(x)}{x} = 2x^1{% endkatex %}

Let's see how can we do this and find the value of where $x=9$

Read more about it's usage: https://medium.com/@alfonsollanes/why-do-we-use-the-derivatives-of-activation-functions-in-a-neural-network-563f2886f2ab

```python
import torch

x = torch.tensor(9.0, requires_grad=True)
print(x)
```
```
tensor(9., requires_grad=True)
```

This time I am setting `requires_grad=True`, this means we are using pytorch autograd engine to compute the derivatives. Derivatives are calculated by tracing the graph (enabled when `requires_grad=True`) from root to leaf by using chain rule (back propagation).

**Note** Only Tensors of floating point dtype can require gradients

For more information you can read this wonderful post: https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95

Also you can read the official documentation regarding autograd on pytorch's website: https://pytorch.org/docs/stable/notes/autograd.html

Moving on, let's calculate {% katex inline %}x^2{% endkatex %}
```py
y = x**2
print(y)
```
```
tensor(81., grad_fn=<PowBackward0>)
```
If you would see, this time the pytorch has attached a gradient function with the tensor. 

To calculate the derivative, you need to call [`Tensor.backward()`](https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward) method. 

And use [`.grad`](https://pytorch.org/docs/stable/autograd.html#torch.Tensor.grad) property on the leaf tensor (here it's $x$)
```py
y.backward()
print(x.grad)
```
```
tensor(18.)
```
Isn't it look like a magic to you? Let's calculate the derivative of another complex equation. 

{% katex %}

f(x) = \sin ( \frac{x}{2 \sqrt 2} )

{% endkatex %}

Where {% katex inline %}x = 100 {% endkatex %}

If you would simplify the denominator {% katex inline %}2 \sqrt 2{% endkatex %}, it would be {% katex inline %}2^{\frac{3}{2}}{% endkatex %}

```py
x = torch.tensor(100., requires_grad=True)

y = torch.sin(x / (2 ** 1.5) )

print(x)
print(y)

y.backward()
print(x.grad)
```
```
tensor(100., requires_grad=True)
tensor(-0.7158, grad_fn=<SinBackward>)
tensor(-0.2469)
```

If you want to test it, you can go to https://www.derivative-calculator.net/ and type in the above expression.


You can also calculate the partial derivatives in pytorch. For example you have {% katex inline %} f(u,v) = uv + u^2 + v^3{% endkatex %} , the derivative {% katex inline %} \frac{d f(u,v)}{du} = 2u + v{% endkatex %} and {% katex inline %} \frac{d f(u,v)}{dv} = 3v^2 + u{% endkatex %}

Let's calculate the partial derivative using pytorch where {% katex inline %}u = 2, v = 4{% endkatex %}

```py
u = torch.tensor(2., requires_grad=True)
v = torch.tensor(4., requires_grad=True)

f = u**2 + v**3 + u*v

print(u)
print(v)
print(f)

f.backward()
print(u.grad)
print(v.grad)
```
```
tensor(2., requires_grad=True)
tensor(4., requires_grad=True)
tensor(76., grad_fn=<AddBackward0>)
tensor(8.)
tensor(50.)
```

If you have doubts or any idea reach me out via following sources
+ Email: tbhaxor@gmail.com (Recommended)
+ Twitter: [@tbhaxor](https://twitter.com/tbhaxor)
+ GitHub: [@tbhaxor](https://github.com/tbhaxor)
+ LinkedIn: [@gurkirat--singh](https://www.linkedin.com/in/gurkirat--singh)
+ Instagram: [@_tbhaxor_](https://instagram.com/_tbhaxor_)