Matrix and Element-wise multiplication in PyTorch
Tags: pytorch, matrix, elementwise, multiplication
Published: 2024-05-10 06:11:33

*Memos:
- [My post](https://dev.to/hyperkai/dot-and-matrix-vector-multiplication-in-pytorch-3859) explains Dot and Matrix-vector multiplication in PyTorch.
- [My post](https://dev.to/hyperkai/functions-and-operators-for-dot-and-matrix-multiplication-and-element-wise-calculation-in-pytorch-3co9) explains the functions and operators for Dot and Matrix multiplication and Element-wise calculation in PyTorch.
- [My post](https://dev.to/hyperkai/matmul-and-dot-in-pytorch-4e8p) explains [matmul()](https://pytorch.org/docs/stable/generated/torch.matmul.html) and [dot()](https://pytorch.org/docs/stable/generated/torch.dot.html)
- [My post](https://dev.to/hyperkai/mv-mm-and-bmm-in-pytorch-57nh) explains [mv()](https://pytorch.org/docs/stable/generated/torch.mv.html), [mm()](https://pytorch.org/docs/stable/generated/torch.mm.html) and [bmm()](https://pytorch.org/docs/stable/generated/torch.bmm.html).
- [My post](https://dev.to/hyperkai/add-sub-and-mul-in-pytorch-1m1f) explains [add()](https://pytorch.org/docs/stable/generated/torch.add.html), [sub()](https://pytorch.org/docs/stable/generated/torch.sub.html) and [mul()](https://pytorch.org/docs/stable/generated/torch.mul.html).
- [My post](https://dev.to/hyperkai/div-remainder-and-fmod-in-pytorch-1eh1) explains [div()](https://pytorch.org/docs/stable/generated/torch.div.html), [remainder()](https://pytorch.org/docs/stable/generated/torch.remainder.html) and [fmod()](https://pytorch.org/docs/stable/generated/torch.fmod.html).

## <**Matrix multiplication(product)**>
- Matrix multiplication is the multiplication of 2D or more tensors(arrays). *The 1st operand can be a 1D tensor(array) but not the last operand.
- The rule which you must follow to do matrix multiplication is the number of the columns of `A` tensor(array) must match the number of the rows of `B` tensor(array).

### 2D tensors(arrays):

```lang-none
    <A>            <B>
[[a, b, c], x [[g, h, i, j], = [[ag+bk+co, ah+bl+cp, ai+bm+cq, aj+bn+cr],
 [d, e, f]]    [k, l, m, n],    [dg+ek+fo, dh+el+fp, di+em+fq, dj+en+fr]]
               [o, p, q, r]]
2 rows         (3) rows
(3) columns    4 columns

[[2, -7, 4], x [[-5, 0, 8, -4], = [[-73, 18, 22, -37],
 [6, 3, -1]]    [9, -2, -6, 7],    [-3, -7, 39, -8]]              
                [0, 1, -9, 5]]   
[[2x(-5)-7x9+4x0, 2x0-7x(-2)+4x1, 2x8-7x(-6)+4x(-9), 2x(-4)-7x7+4x5]
 [6x(-5)+3x9-1x0, 6x0+3x(-2)-1x1, 6x8+3x(-6)-1x(-9), 6x(-4)+3x7-1x5]]
```

In PyTorch with [matmul()](https://pytorch.org/docs/stable/generated/torch.matmul.html), [mm()](https://pytorch.org/docs/stable/generated/torch.mm.html) or `@`. 
*Memos:
- `matmul()` or `@` can do dot, matrix-vector or matrix multiplication with two of 1D or more D tensors.
-`mm()` can do matrix multiplication with two of 2D tensors:

```python
import torch

tensor1 = torch.tensor([[2, -7, 4], [6, 3, -1]])
tensor2 = torch.tensor([[-5, 0, 8, -4], [9, -2, -6, 7], [0, 1, -9, 5]])

torch.matmul(input=tensor1, other=tensor2)
tensor1.matmul(other=tensor2)
torch.mm(input=tensor1, mat2=tensor2)
tensor1.mm(mat2=tensor2)
tensor1 @ tensor2
# tensor([[-73, 18, 22, -37], [-3, -7, 39, -8]])
```

In NumPy with [matmul()](https://numpy.org/doc/stable/reference/generated/numpy.matmul.html), [dot()](https://numpy.org/doc/stable/reference/generated/numpy.dot.html) or `@`:

*Memos:
- `matmul()` or `@` can do dot, matrix-vector or matrix multiplication with two of 1D or more D arrays.
- `dot()` can do dot, matrix-vector or matrix multiplication with two of 0D or more D arrays. *dot() is basically used to multiply two of 1D arrays.

```python
import numpy

array1 = numpy.array([[2, -7, 4], [6, 3, -1]])
array2 = numpy.array([[-5, 0, 8, -4], [9, -2, -6, 7], [0, 1, -9, 5]])

numpy.matmul(array1, array2)
numpy.dot(array1, array2)
array1.dot(array2)
array1 @ array2
# array([[-73, 18, 22, -37], [-3, -7, 39, -8]])
```

### A 1D and 3D tensor(array):

*The 3D tensor(array) of `B` has three of 2D tensors(arrays) which have 2 rows and 4 columns each.

```lang-none
  <A>             <B>
[a, b] x [[[c, d, e, f], = [[(ac+bg), (ad+bh), (ae+bi), (af+bj)],
           [g, h, i, j]],   [(ak+bo), (al+bp), (am+bq), (an+br)],
          [[k, l, m, n],    [(as+bw), (at+bx), (au+by), (av+bz)]]
           [o, p, q, r]],
          [[s, t, u, v],
           [w, x, y, z]]]
1 row         (2) rows
(2) columns   4 columns

[2, -7] x [[[-5, 0, 8, -4], = [[-73, 14, 58, -57],
            [9, -2, -6, 7]],   [21, -26, 31, -4],
           [[0, 1, -9, 5],     [40, 60, -1, -30]]
            [-3, 4, -7, 2]],
           [[6, 2, 3, -1],
            [-4, -8, 1, 4]]]
[[2x(-5)-7x9, 2x0-7x(-2), 2x8-7x(-6), 2x(-4)-7x7],
 [2x0-7x(-3), 2x1-7x4, 2x(-9)-7x(-7), 2x5-7x2],
 [2x6-7x(-4), 2x2-7x(-8), 2x3-7x1, 2x(-1)-7x4]]
```

In PyTorch with `matmul()` or `@`:

```python
import torch

tensor1 = torch.tensor([2, -7])
tensor2 = torch.tensor([[[-5, 0, 8, -4], [9, -2, -6, 7]],
                        [[0, 1, -9, 5], [-3, 4, -7, 2]],
                        [[6, 2, 3, -1], [-4, -8, 1, 4]]])
torch.matmul(input=tensor1, other=tensor2)
tensor1.matmul(other=tensor2)
tensor1 @ tensor2
# tensor([[-73, 14, 58, -57],
#         [21, -26, 31, -4], 
#         [40, 60, -1, -30]])
```

In NumPy with `matmul()`, `dot()` or `@`:

```python
import numpy

array1 = numpy.array([2, -7])
array2 = numpy.array([[[-5, 0, 8, -4], [9, -2, -6, 7]],
                        [[0, 1, -9, 5], [-3, 4, -7, 2]],
                        [[6, 2, 3, -1], [-4, -8, 1, 4]]])
numpy.matmul(array1, array2)
numpy.dot(array1, array2)
array1.dot(array2)
array1 @ array2
# array([[-73, 14, 58, -57],
#        [21, -26, 31, -4],
#        [40, 60, -1, -30]])
```

## <**Element-wise multiplication(product)**>
- Element-wise multiplication is the multiplication of 0D or more D tensors(arrays).
- The rule which you must follow to do element-wise multiplication is 2 tensors(arrays) must have the same number of rows and columns.

### 1D tensors(arrays):

```lang-none
[a, b, c] x [d, e, f] = [ad, be, cf]
1 row       1 row 
3 columns   3 columns

[2, -7, 4] x [6, 3, -1] = [12, -21, -4]
                          [2x6, -7x3, 4x(-1)]
 [2, -7,  4]
  x   x   x 
 [6,  3, -1]
     ||
[12, -21, -4]
```

In PyTorch with [mul()](https://pytorch.org/docs/stable/generated/torch.mul.html) or `*`. *`mul()` or <code>&#42;</code> can do element-wise multiplication with two of 0D or more D tensors:

```python
import torch

tensor1 = torch.tensor([2, -7, 4])
tensor2 = torch.tensor([6, 3, -1])

torch.mul(input=tensor1, other=tensor2)
tensor1.mul(other=tensor2)
tensor1 * tensor2
# tensor([12, -21, -4])
```

In NumPy with [multiply()](https://numpy.org/doc/stable/reference/generated/numpy.multiply.html) or `*`. *`multiply()` or <code>&#42;</code> can do element-wise multiplication with two of 0D or more D arrays.

```python
import numpy

array1 = numpy.array([2, -7, 4])
array2 = numpy.array([6, 3, -1])

numpy.multiply(array1, array2)
array1 * array2
# array([12, -21, -4])
```

2D tensors(arrays):

```lang-none
[[a, b, c], x [[g, h, i], = [[ag, bh, ci],
 [d, e, f]]    [j, k, l]]    [dj, ek, fl]]
2 rows        2 rows
3 columns     3 columns

[[2, -7, 4], x [[-5, 0, 8], = [[10, 0, 32],
 [6, 3, -1]]    [9, -2, -6]]   [18, 18, 5]]
                              [[2x(-5), -7x0, 4x8] 
                               [6x9, 3x(-2) -1x(-6)]]
  [[2, -7, 4], [6,  3,  5]]
    x   x  x    x   x   x
 [[-5,  0, 8], [9, -2, -6]]
             ||
 [[10, 0, 32], [18, 18, 5]]
```

In PyTorch with `*` or `mul()`:

```python
import torch

tensor1 = torch.tensor([[2, -7, 4], [6, 3, -1]])
tensor2 = torch.tensor([[-5, 0, 8], [9, -2, -6]])

torch.mul(input=tensor1, other=tensor2)
tensor1.mul(other=tensor2)
tensor1 * tensor2
# tensor([[-10, 0, 32], [54, -6, 6]])
```

In NumPy with `*` or `multiply()`:

```python
import numpy

array1 = numpy.array([[2, -7, 4], [6, 3, -1]])
array2 = numpy.array([[-5, 0, 8], [9, -2, -6]])

numpy.multiply(array1, array2)
array1 * array2
# array([[-10, 0, 32], [54, -6, 6]])
```