Repository: pytorch
Issue: filtering on CPU slower in pytorch than scipy
Labels: module: performance,module: cpu,triaged

### 🐛 Describe the bug

I'm comparing filtering performance on CPU vs CUDA vs scipy and I found that pytorch on CPU was significantly slower than scipy.

I'm testing filtering a large image with a `median` filter, a `Gaussian` filter, and a `Laplacian` filter. I got the filter kernels from `kornia`, but other than that it's basic pytorch.

Here's my code I used for testing:

Note, the reason why in `torch_gauss` I'm changing axis based on cpu vs cuda is because of [this issue](https://discuss.pytorch.org/t/performance-issue-for-conv2d-with-1d-filter-along-a-dim/201734).

```py
import numpy as np
from scipy.ndimage import gaussian_filter, laplace
from scipy.signal import medfilt2d
import torch
from numpy import random
import torch.nn.functional as F
from torch.utils.benchmark import Timer, Compare
from itertools import product
from math import ceil
from kornia.filters.kernels import (
    get_binary_kernel2d,
    get_gaussian_kernel1d,
    get_gaussian_kernel2d,
    get_laplacian_kernel2d,
    normalize_kernel2d,
)


W = 3848
H = 3222

np_data = random.rand(W, H).astype(np.float32)
torch_cpu = torch.from_numpy(np_data).unsqueeze(0).unsqueeze(0)
torch_cuda = torch_cpu.to(device="cuda")


bin_kernel_cpu = get_binary_kernel2d(3, device="cpu", dtype=torch.float32)
bin_kernel_cuda = get_binary_kernel2d(3, device="cuda", dtype=torch.float32)

def torch_med(device):
    data = torch_cpu if device == "cpu" else torch_cuda
    kernel = bin_kernel_cpu if device == "cpu" else bin_kernel_cuda
    data = F.conv2d(data, kernel, padding="same")
    data = data.median(dim=1, keepdim=True)[0]
    return data

def np_med():
    return medfilt2d(np_data)


laplace_gaussian_sigma = 0.2 * 30
def get_gauss_kernel(device):
    kernel_size = 2 * int(round(4 * laplace_gaussian_sigma)) + 1
    kernel = get_gaussian_kernel1d(
        kernel_size,
        laplace_gaussian_sigma,
        device=device,
        dtype=torch.float32,
    ).view(1, 1, -1, 1)
    if device == "cpu":
        kernel = kernel.view(1, 1, -1, 1)
    else:
        kernel = kernel.view(1, 1, 1, -1)

    return kernel

gauss_kernel_cpu = get_gauss_kernel("cpu")
gauss_kernel_cuda = get_gauss_kernel("cuda")

def torch_gauss(device):
    data = torch_cpu if device == "cpu" else torch_cuda
    if device == "cpu":
        data = F.conv2d(data, gauss_kernel_cpu, padding="same")
        data = F.conv2d(
            data.moveaxis(-1, -2), gauss_kernel_cpu, padding="same"
        ).moveaxis(-1, -2)
    else:
        data = F.conv2d(
            data.moveaxis(-1, -2), gauss_kernel_cuda, padding="same"
        ).moveaxis(-1, -2)
        data = F.conv2d(data, gauss_kernel_cuda, padding="same")
    return data

def np_gauss():
    return gaussian_filter(np_data, laplace_gaussian_sigma)


def get_lap_kernel(device):
    lap_kernel = get_laplacian_kernel2d(3, device=device, dtype=torch.float32)
    lap_kernel = normalize_kernel2d(lap_kernel)
    lap_kernel = lap_kernel.view(
        1, 1, lap_kernel.shape[0], lap_kernel.shape[1]
    )
    return lap_kernel

lap_kernel_cpu = get_lap_kernel("cpu")
lap_kernel_cuda = get_lap_kernel("cuda")

def torch_lap(device):
    data = torch_cpu if device == "cpu" else torch_cuda
    kernel = lap_kernel_cpu if device == "cpu" else lap_kernel_cuda
    return F.conv2d(data, kernel, padding="same")

def np_lap():
    return laplace(np_data)


n = 5
threads = [1, 2, 4, 12, 20]
results = []

for name in ["lap", "gauss", "med"]:
    results.append(
        Timer(setup=f"from __main__ import np_{name}", stmt=f"np_{name}()", label=name, sub_label="np", description="").timeit(n)
    )
    results.append(
        Timer(setup=f"from __main__ import torch_{name}", stmt=f"torch_{name}('cuda')", label=name, sub_label="cuda", description="").timeit(n)
    )
    for i in threads:
        results.append(
            Timer(setup=f"from __main__ import torch_{name}", stmt=f"torch_{name}('cpu')", label=name, sub_label=f"cpu_{i}",
                  description="", num_threads=i).timeit(n)
        )
compare = Compare(results)
compare.print()
```

I tested it on two computers.

Computer 1 with specs:
```
Windows
Processor	Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz   3.00 GHz
Installed RAM	128 GB (128 GB usable)
Core: 36
GPU: GTX-1060 6GB
```

It printed:
```
[------- lap --------]
              |
1 threads: -----------
      np      |  193.9
      cuda    |    3.0
      cpu_1   |  499.6
2 threads: -----------
      cpu_2   |  354.1
4 threads: -----------
      cpu_4   |  248.1
12 threads: ----------
      cpu_12  |  213.1
20 threads: ----------
      cpu_20  |  222.1

Times are in milliseconds (ms).

[------- gauss -------]
              |
1 threads: ------------
      np      |   560.2
      cuda    |    38.5
      cpu_1   |  1131.6
2 threads: ------------
      cpu_2   |   726.1
4 threads: ------------
      cpu_4   |   488.9
12 threads: -----------
      cpu_12  |   375.3
20 threads: -----------
      cpu_20  |   392.4

Times are in milliseconds (ms).

[-------- med --------]
              |
1 threads: ------------
      np      |  1162.5
      cuda    |   227.6
      cpu_1   |  2855.2
2 threads: ------------
      cpu_2   |  1658.3
4 threads: ------------
      cpu_4   |   969.1
12 threads: -----------
      cpu_12  |   447.8
20 threads: -----------
      cpu_20  |   488.0

Times are in milliseconds (ms).
```

Computer 2 with specs:
```
Windows
Processor	Intel(R) Core(TM) i7-7820X CPU @ 3.60GHz   3.60 GHz
Installed RAM	128 GB (128 GB usable)
Core: 16
GPU: RTX-2080 SUPER 8GB

```

It printed:
```
[--------- lap ---------]
              |
1 threads: --------------
      np      |  180645.7
      cuda    |     476.7
      cpu_1   |  365684.4
2 threads: --------------
      cpu_2   |  314580.5
4 threads: --------------
      cpu_4   |  218351.7
12 threads: -------------
      cpu_12  |  197216.8

Times are in microseconds (us).

[------- gauss -------]
              |
1 threads: ------------
      np      |   551.8
      cuda    |    17.5
      cpu_1   |  1008.1
2 threads: ------------
      cpu_2   |   641.2
4 threads: ------------
      cpu_4   |   438.4
12 threads: -----------
      cpu_12  |   395.1

Times are in milliseconds (ms).

[-------- med --------]
              |
1 threads: ------------
      np      |  1136.8
      cuda    |   101.7
      cpu_1   |  2743.0
2 threads: ------------
      cpu_2   |  1531.0
4 threads: ------------
      cpu_4   |   902.6
12 threads: -----------
      cpu_12  |   585.7

Times are in milliseconds (ms).
```

In both, but one more than the other, Pytorch CPU is significantly slower than scipy on single, or even multiple threads.

### Versions

Computer 1:

```
Collecting environment information...
PyTorch version: 2.3.0+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.19 (main, Mar 21 2024, 17:21:27) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1060 6GB
Nvidia driver version: 536.23
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3000
DeviceID=CPU0
Family=179
L2CacheSize=18432
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=3000
Name=Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz
ProcessorType=3
Revision=21764

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] numpydoc==1.7.0
[pip3] torch==2.3.0+cu118
[pip3] torchaudio==2.3.0+cu118
[pip3] torchvision==0.18.0+cu118
[conda] cudatoolkit               11.2.2              h7d7167e_13    conda-forge
[conda] mkl                       2021.4.0                 pypi_0    pypi
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] numpydoc                  1.7.0                    pypi_0    pypi
[conda] torch                     2.3.0+cu118              pypi_0    pypi
[conda] torchaudio                2.3.0+cu118              pypi_0    pypi
[conda] torchvision               0.18.0+cu118             pypi_0    pypi
```

Computer 2:

```
Collecting environment information...
PyTorch version: 2.3.0+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.19 (main, Mar 21 2024, 17:21:27) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22631-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2080 SUPER
Nvidia driver version: 536.23
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3600
DeviceID=CPU0
Family=179
L2CacheSize=8192
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=3600
Name=Intel(R) Core(TM) i7-7820X CPU @ 3.60GHz
ProcessorType=3
Revision=21764

Versions of relevant libraries:
[pip3] numpy==1.26.3
[pip3] numpydoc==1.7.0
[pip3] torch==2.3.0+cu118
[pip3] torchaudio==2.3.0+cu118
[pip3] torchvision==0.18.0+cu118
[conda] cudatoolkit               11.2.2              h7d7167e_13    conda-forge
[conda] mkl                       2021.4.0                 pypi_0    pypi
[conda] numpy                     1.26.3                   pypi_0    pypi
[conda] numpydoc                  1.7.0                    pypi_0    pypi
[conda] torch                     2.3.0+cu118              pypi_0    pypi
[conda] torchaudio                2.3.0+cu118              pypi_0    pypi
[conda] torchvision               0.18.0+cu118             pypi_0    pypi
```

cc @msaroufim @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10