Linear Algebra Operations with PyTorch
Published: 2020-05-27 11:15:55

Linear algebra is a core mathematical concept in machine learning, especially deep learning, a sub-field of ML. There is a number of instances where linear algebra comes in handy when implementing neural network in deep learning. One of it is when dealing with unstructured data like images. An image consists of pixels which is commonly represented as a tensor or a matrix. In this blog post, I will briefly talk about basic linear algebra operations in PyTorch that are used in deep learning.

### 1. `torch.dot()`

This function allows us to perform dot product aka inner product between two vectors of the same size. The first element of `t1` is multiplied with the first element of `t2` and the second element of `t1` is multiplied with the second element of `t2` and so on and so forth. These products are then summed together. Note that a dot product between 2 vectors always returns a scalar value.

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_2_dot.ipynb %}

This operation is also commutative, in which `t1 . t2 = t2 . t1`. If we pass in `t2` as the first argument and `t1` as the second argument, we will get the same answer. One thing to keep in mind is that the dot product only works if the vectors are of the same size. If not, it will spit out an error complaining about inconsistent number of elements.

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_3_dot.ipynb %}

### 2. `torch.mm()`

`torch.mm()` is responsible for multiplication between 2 matrices. Similar to vector multiplication, matrix multiplication makes use of dot product and requires the matrices to have certain sizes. The number of columns of the first matrix must be equal to the number of rows of the second matrix. Each row of the first matrix will be transposed and multiplied against each column in the second matrix. This is basically a vector multiplication where each row in the first matrix is transposed to make sure it has the same dimension as each column in the second matrix.

For example, the dot product is valid if the first matrix has a dimension of (3, 2) and the second matrix has a dimension of (2, 2). But not the other way around. A bunch of words might not help much, so let's look at a couple of examples.

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_1_mm.ipynb %}

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_2_mm.ipynb %}

Example 2 uses the same arguments as example 1, except that the order of the arguments is swapped. We can see that swapping the order results in a completely different outcome. So, unlike the dot product between 2 vectors, matrix multiplication is not commutative; `t1 x t2 != t2 x t1`.

Example below shows that it is important to make sure the rows of the first matrix have the same number of entries as the columns of the second matrix.

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_3_mm.ipynb %}

### 3. `torch.matmul()`

This function performs multiplication, but it is not limited to certain shapes of tensors. `torch.matmul()` allows us to do multiplication for different ranks of tensors. Based on PyTorch's official [documentation](https://pytorch.org/docs/stable/torch.html#torch.matmul), this function behaves according to the dimensionality of the input tensors. For instance, if both arguments are vectors of the same size, it will behave exactly like `torch.dot()`. If both arguments are matrices, it will perform matrix multiplication similar to `torch.mm()`. It also supports multiplication between a scalar and a matrix, by converting the scalar value into a rank-2 tensor so these 2 tensors will be compatible. In other words, it supports broadcasting. Check out this [blogpost](https://deeplizard.com/learn/video/6_33ulFDuCg) for a detailed explanation on broadcasting.

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_2_matmul.ipynb %}

In above example, `t1` is a scalar value and `t2` is a matrix. The way `matmul` handles this is by pre-pending a `1` to the dimension of `t1` so the new dimension becomes `(1, 2)`. It is now compatible with `t2` that has a dimension of `(2, 3)`. The pre-pended `1` is removed after multiplication is performed.

### 4. `torch.transpose()`

Sometimes the tensor that we have is not the shape or dimension that we desire, and this happens a lot. So this is where transposing an array or a matrix comes in handy. One of the applications is when doing an operation within matrices itself, which I mentioned earlier in `torch.mm()` section. In terms of a matrix, transposing can be thought of as flipping the elements over the diagonal axis. `torch.transpose()` accepts 3 arguments; first argument being the tensor, second and third arguments being the dimensions that we want to swap.

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_1_transpose.ipynb %}

The above example shows that we swap between dimension 0 and dimension 1, so the row of the output is the column of the input and vice versa.

`torch.transpose()` allows us to swap between the same dimension, but it is the same as not swapping. So the output will be just the same as the input. Example below demonstrates this behavior:

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_2_transpose.ipynb %}

### 5. `torch.add()`

So far, we haven't mentioned element-wise operation between tensors. There are a number of functions in PyTorch that allows us to do that and one of them is `torch.add()`. In order to sum two matrices together, they must have the same size.

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_1_add.ipynb %}

`t1` and `t2` are both (2, 2) matrices allowing values at the same position of the 2 matrices to be added together resulting in also a (2, 2) matrix.

{% gist https://gist.github.com/aishahsofea/2bd91937128b9c90a8dda852f1eaaaae file=example_2_add.ipynb %}

Example above shows that the function supports broadcasting in which it modifies the dimension of `t1` so it becomes compatible with the second argument, `t2`.


### Conclusion
It is certainly important to have a good understanding and know when to implement a particular linear algebra operation if we want to delve into the world of deep learning. That being said, the list of functions above is far from exhaustive. Fret not, as we dive deeper, we are likely to discover more functions and the list will only grow from here on out!