Dot and Matrix-vector multiplication in PyTorch
Tags: pytorch, dot, matrixvector, multiplication
Published: 2024-05-09 23:22:06

*Memos:
- [My post](https://dev.to/hyperkai/matrix-and-element-wise-multiplication-in-pytorch-25le) explains Matrix and Element-wise multiplication in PyTorch.
- [My post](https://dev.to/hyperkai/functions-and-operators-for-dot-and-matrix-multiplication-and-element-wise-calculation-in-pytorch-3co9) explains the functions and operators for Dot and Matrix multiplication and Element-wise calculation in [PyTorch](https://pytorch.org/).

## <**Dot multiplication(product)**>

- Dot multiplication is the multiplication of 1D tensors(arrays).
- The rule which you must follow to do dot multiplication is the number of the rows of `A` and `B` tensor(array) must be 1 and the number of the columns must be the same.

```lang-none
   <A>         <B>
[a, b, c] x [d, e, f] = ad+be+cf
1 row       1 row
3 columns   3 columns

[2, -7, 4] x [-5, 0, 8] = 22
                     2x(-5)-7x0+4x8
  [2, -7, 4]
   x   x  x
 [-5,  0, 8]
      ||
 [-10, 0, 32]
   -10+0+32
      ||
      22
```

In PyTorch with [dot()](https://pytorch.org/docs/stable/generated/torch.dot.html), [matmul()](https://pytorch.org/docs/stable/generated/torch.matmul.html) or `@`:

*Memos:
- `dot()` can do dot multiplication with two of 1D tensors.
- `matmul()` or `@` can do dot, matrix-vector or matrix multiplication with two of 1D or more D tensors.

```python
import torch

tensor1 = torch.tensor([2, -7, 4])
tensor2 = torch.tensor([-5, 0, 8])

torch.dot(input=tensor1, tensor=tensor2)
tensor1.dot(tensor=tensor2)
torch.matmul(input=tensor1, other=tensor2)
tensor1.matmul(other=tensor2)
tensor1 @ tensor2
# tensor(22)
```

In [NumPy](https://numpy.org/) with [dot()](https://numpy.org/doc/stable/reference/generated/numpy.dot.html), [matmul()](https://numpy.org/doc/stable/reference/generated/numpy.matmul.html) or `@`:

*Memos:
- `dot()` can do dot, matrix-vector or matrix multiplication with two of 0D or more D arrays. *`dot()` is basically used to multiply 1D arrays.
- `matmul()` or `@` can do dot, matrix-vector or matrix multiplication with two of 1D or more D arrays.

```python
import numpy

array1 = numpy.array([2, -7, 4])
array2 = numpy.array([-5, 0, 8])

numpy.dot(array1, array2)
array1.dot(array2)
numpy.matmul(array1, array2)
array1 @ array2
# 22
```

## <**Matrix-vector multiplication(product)**>

- Matrix-vector multiplication is the multiplication of a 2D or more D tensor(array) and 1D tensor(array). *The order must be a 2D or more D tensor and 1D tensor but not a 1D tensor and 2D or more D tensor(array).
- The rule which you must follow to do matrix-vector multiplication is the number of the columns of `A` and `B` tensor(array) must be the same.

### A 2D and 1D tensor(array):

```lang-none
    <A>          <B>
[[a, b, c], [d, e, f]] x [g, h, i] = [ag+bh+ci, dg+eh+fi]
2 rows                   1 row
(3) columns              (3) columns

[[2, -7, 4], [6, 3, -1]] x [-5, 0, 8] = [22, -38]
                            [2x(-5)-7x0+4x8, 6x(-5)+3x0-1x8]
 [[2, -7, 4], [6, 3, -1]]
   x   x  x    x  x   x
 [-5,  0, 8] [-5, 0,  8]
      ||          ||
[-10, 0, 32] [-30, 0, -8]
  -10+0+32     -30+0-8
      ||          ||
     [22,        -38]
```

In PyTorch with `matmul()`, [mv()](https://pytorch.org/docs/stable/generated/torch.mv.html) or `@`. *`mv()` can do matrix-vector multiplication with a 2D tensor and 1D tensor:

```python
import torch

tensor1 = torch.tensor([[2, -7, 4], [6, 3, -1]])
tensor2 = torch.tensor([-5, 0, 8])

torch.matmul(input=tensor1, other=tensor2)
tensor1.matmul(other=tensor2)
torch.mv(input=tensor1, vec=tensor2)
tensor1.mv(vec=tensor2)
tensor1 @ tensor2
# tensor([22, -38])
```

In NumPy with `dot()`, `matmul()` or `@`:

```python
import numpy

array1 = numpy.array([[2, -7, 4], [6, 3, -1]])
array2 = numpy.array([-5, 0, 8])

numpy.dot(array1, array2)
array1.dot(array2)
numpy.matmul(array1, array2)
array1 @ array2
# array([22, -38])
```

### A 3D and 1D tensor(array):

*The 3D tensor(array) of `A` has three of 2D tensors(arrays) which have 2 rows and 3 columns each.

```lang-none
     <A>                      <B>
[[[a, b, c], [d, e, f]], x [s, t, u] = [[[as+bt+cu, ds+et+fu]],
 [[g, h, i], [j, k, l]],                [[gs+ht+iu, js+kt+lu]],
 [[m, n, o], [p, q, r]]]                [[ms+nt+ou, ps+qt+ru]]]
2 rows                     1 row
(3) columns                (3) columns

[[[2, -7, 4], [6, 3, -1]] x [-5, 0, 8] = [[22, -38],
 [[-4, 9, 0], [5, 8, -2]],                [20, -41],
 [[-6, 7, 1], [0, -9, 5]]]                [38, 40]])
                             [[2x(-5)-7x0+4x8, 6x(-5)+3x0-1x8],
                              [-4x(-5)+9x0+0x8, 5x(-5)+8x0-2x8],
                              [-6x(-5)+7x0+1x8, 0x(-5)-9x0+5x8]]
```

In PyTorch with `matmul()` or `@`:

```python
import torch

tensor1 = torch.tensor([[[2, -7, 4], [6, 3, -1]],
                        [[-4, 9, 0], [5, 8, -2]],
                        [[-6, 7, 1], [0, -9, 5]]])
tensor2 = torch.tensor([-5, 0, 8])

torch.matmul(input=tensor1, other=tensor2)
tensor1.matmul(other=tensor2)
tensor1 @ tensor2
# tensor([[22, -38],
#         [20, -41],
#         [38, 40]])
```

In NumPy with `dot()`, `matmul()` or `@`:

```python
import numpy

array1 = numpy.array([[[2, -7, 4], [6, 3, -1]],
                      [[-4, 9, 0], [5, 8, -2]],
                      [[-6, 7, 1], [0, -9, 5]]])
array2 = numpy.array([-5, 0, 8])

numpy.dot(array1, array2)
array1.dot(array2)
numpy.matmul(array1, array2)
array1 @ array2
# array([[22, -38],
#        [20, -41],
#        [38, 40]])
```