Repository: pytorch
Issue: Speed Degradation from cu118 to cu121
Labels: module: performance,module: cuda,triaged

### 🐛 Describe the bug

When running our internal models we've notice a 2x slowdown in training times when going from Pytorch2.0.1+cu118 to Pytorch 2.1.2+cu121 and this extends to Pytorch 2.2.0+cu121.

We're not 100% sure what is causing this. But it seems cuda related. If we run the below minimal script using the Resnet18 model on an L4 GPU we can replicate a good deal of the slowdown. This slowdown seems to be dependent on cuda version bundled with python:
Running resnet18 10.000 times

2.2.0+cu121
time elapsed 48.99877489299979

2.2.0+cu118
time elapsed 38.23451283498434

2.1.2+cu121
time elapsed 47.73117630800698

2.1.2+cu118
time elapsed 37.863490240997635

2.0.1+cu118
time elapsed 35.008950340998126

```
import torch
import torchvision.models as models
from torch.profiler import profile, record_function, ProfilerActivity

print("Profiling resnet18 using torch:", torch.__version__)

model = models.resnet18().cuda()
inputs = torch.randn(5, 3, 224, 224).cuda()

with profile(activities=[
        ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
    with record_function("model_inference"):
        model(inputs)

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))

from time import perf_counter

print("Running resnet18 10.000 times")

it_start = perf_counter()

for i in range(10000):
    model(inputs)

it_end = perf_counter()

print("time elapsed", it_end-it_start)
```

While the above is on an L4 we also see similar training slowdowns on A100's on our internal models.

Full output of above script run with the different torch version:
```
On L4

Profiling resnet18 using torch: 2.0.1+cu118
WARNING:2024-02-08 11:14:30 6718:6718 init.cpp:146] function cbapi->getCuptiStatus() failed with error CUPTI_ERROR_INVALID_DEVICE (2)
WARNING:2024-02-08 11:14:30 6718:6718 init.cpp:147] CUPTI initialization failed - CUDA profiler activities will be missing
INFO:2024-02-08 11:14:30 6718:6718 init.cpp:149] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti
STAGE:2024-02-08 11:14:30 6718:6718 ActivityProfilerController.cpp:311] Completed Stage: Warm Up
STAGE:2024-02-08 11:14:31 6718:6718 ActivityProfilerController.cpp:317] Completed Stage: Collection
STAGE:2024-02-08 11:14:31 6718:6718 ActivityProfilerController.cpp:321] Completed Stage: Post Processing
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                             Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                  model_inference         1.20%       3.258ms       100.00%     270.901ms     270.901ms             1  
                     aten::conv2d         0.04%     101.000us        83.23%     225.473ms      11.274ms            20  
                aten::convolution         0.10%     284.000us        83.19%     225.372ms      11.269ms            20  
               aten::_convolution         0.06%     171.000us        83.09%     225.088ms      11.254ms            20  
          aten::cudnn_convolution        83.03%     224.917ms        83.03%     224.917ms      11.246ms            20  
                       aten::add_         3.49%       9.449ms         3.49%       9.449ms     337.464us            28  
                 aten::batch_norm         0.02%      66.000us         5.00%      13.551ms     677.550us            20  
     aten::_batch_norm_impl_index         0.04%     110.000us         4.98%      13.485ms     674.250us            20  
           aten::cudnn_batch_norm         4.55%      12.334ms         4.94%      13.375ms     668.750us            20  
                 aten::empty_like         0.04%      98.000us         0.34%     920.000us      46.000us            20  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 270.901ms

Running resnet18 10.000 times
time elapsed 35.008950340998126

Profiling resnet18 using torch: 2.1.2+cu121
STAGE:2024-02-08 10:54:00 6241:6241 ActivityProfilerController.cpp:312] Completed Stage: Warm Up
STAGE:2024-02-08 10:54:01 6241:6241 ActivityProfilerController.cpp:318] Completed Stage: Collection
STAGE:2024-02-08 10:54:01 6241:6241 ActivityProfilerController.cpp:322] Completed Stage: Post Processing
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                        model_inference         1.20%       3.980ms       100.00%     332.660ms     332.660ms       0.000us         0.00%       1.543ms       1.543ms             1  
                                      aten::convolution         0.12%     403.000us        76.35%     253.985ms      12.699ms       0.000us         0.00%       1.199ms      59.950us            20  
                                     aten::_convolution         0.07%     230.000us        76.23%     253.582ms      12.679ms       0.000us         0.00%       1.199ms      59.950us            20  
                                aten::cudnn_convolution        40.99%     136.358ms        76.16%     253.352ms      12.668ms       1.198ms        77.69%       1.199ms      59.950us            20  
                                           aten::conv2d         0.08%     265.000us        76.40%     254.145ms      12.707ms       0.000us         0.00%       1.137ms      56.850us            20  
void cudnn::ops::nchwToNhwcKernel<float, float, floa...         0.00%       0.000us         0.00%       0.000us       0.000us     278.000us        18.03%     278.000us       8.688us            32  
                                       aten::batch_norm         0.02%      69.000us         5.30%      17.643ms     882.150us       0.000us         0.00%     193.000us       9.650us            20  
                           aten::_batch_norm_impl_index         0.03%     115.000us         5.28%      17.574ms     878.700us       0.000us         0.00%     193.000us       9.650us            20  
                                 aten::cudnn_batch_norm         1.30%       4.326ms         5.25%      17.459ms     872.950us     193.000us        12.52%     193.000us       9.650us            20  
sm86_xmma_fprop_implicit_gemm_indexed_tf32f32_tf32f3...         0.00%       0.000us         0.00%       0.000us       0.000us     188.000us        12.19%     188.000us      47.000us             4  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 332.672ms
Self CUDA time total: 1.542ms

Running resnet18 10.000 times
time elapsed 47.73117630800698

Profiling resnet18 using torch: 2.1.2+cu118
WARNING:2024-02-08 11:23:55 6753:6753 init.cpp:155] function cbapi->getCuptiStatus() failed with error CUPTI_ERROR_INVALID_DEVICE (2)
WARNING:2024-02-08 11:23:55 6753:6753 init.cpp:156] CUPTI initialization failed - CUDA profiler activities will be missing
INFO:2024-02-08 11:23:55 6753:6753 init.cpp:158] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti
STAGE:2024-02-08 11:23:55 6753:6753 ActivityProfilerController.cpp:312] Completed Stage: Warm Up
STAGE:2024-02-08 11:23:55 6753:6753 ActivityProfilerController.cpp:318] Completed Stage: Collection
STAGE:2024-02-08 11:23:55 6753:6753 ActivityProfilerController.cpp:322] Completed Stage: Post Processing
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                             Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                  model_inference         1.42%       4.034ms       100.00%     284.097ms     284.097ms             1  
                     aten::conv2d         0.07%     208.000us        82.24%     233.651ms      11.683ms            20  
                aten::convolution         0.13%     377.000us        82.19%     233.513ms      11.676ms            20  
               aten::_convolution         0.09%     245.000us        82.06%     233.136ms      11.657ms            20  
          aten::cudnn_convolution        81.98%     232.891ms        81.98%     232.891ms      11.645ms            20  
                       aten::add_         3.67%      10.422ms         3.67%      10.422ms     372.214us            28  
                 aten::batch_norm         0.03%      97.000us         4.80%      13.645ms     682.250us            20  
     aten::_batch_norm_impl_index         0.05%     131.000us         4.77%      13.548ms     677.400us            20  
           aten::cudnn_batch_norm         4.32%      12.264ms         4.72%      13.417ms     670.850us            20  
                 aten::empty_like         0.06%     165.000us         0.36%       1.011ms      50.550us            20  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 284.097ms

Running resnet18 10.000 times
time elapsed 37.863490240997635

Profiling resnet18 using torch: 2.2.0+cu121
STAGE:2024-02-08 11:28:50 6789:6789 ActivityProfilerController.cpp:314] Completed Stage: Warm Up
STAGE:2024-02-08 11:28:51 6789:6789 ActivityProfilerController.cpp:320] Completed Stage: Collection
STAGE:2024-02-08 11:28:51 6789:6789 ActivityProfilerController.cpp:324] Completed Stage: Post Processing
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                        model_inference         0.55%       4.761ms       100.00%     862.176ms     862.176ms       0.000us         0.00%       1.546ms       1.546ms             1  
                                           aten::conv2d         0.02%     215.000us        83.60%     720.756ms      36.038ms       0.000us         0.00%       1.198ms      59.900us            20  
                                      aten::convolution         0.07%     597.000us        83.57%     720.541ms      36.027ms       0.000us         0.00%       1.198ms      59.900us            20  
                                     aten::_convolution         0.04%     311.000us        83.50%     719.944ms      35.997ms       0.000us         0.00%       1.198ms      59.900us            20  
                                aten::cudnn_convolution        52.19%     450.009ms        83.47%     719.633ms      35.982ms       1.197ms        77.48%       1.198ms      59.900us            20  
void cudnn::ops::nchwToNhwcKernel<float, float, floa...         0.00%       0.000us         0.00%       0.000us       0.000us     278.000us        17.99%     278.000us       8.688us            32  
                                       aten::batch_norm         0.01%     112.000us         5.50%      47.440ms       2.372ms       0.000us         0.00%     194.000us       9.700us            20  
                           aten::_batch_norm_impl_index         0.02%     144.000us         5.49%      47.328ms       2.366ms       0.000us         0.00%     194.000us       9.700us            20  
                                 aten::cudnn_batch_norm         1.74%      14.960ms         5.47%      47.184ms       2.359ms     194.000us        12.56%     194.000us       9.700us            20  
sm86_xmma_fprop_implicit_gemm_indexed_tf32f32_tf32f3...         0.00%       0.000us         0.00%       0.000us       0.000us     187.000us        12.10%     187.000us      46.750us             4  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 862.191ms
Self CUDA time total: 1.545ms

Running resnet18 10.000 times
time elapsed 48.99877489299979

Profiling resnet18 using torch: 2.2.0+cu118
WARNING:2024-02-08 10:47:36 6204:6204 init.cpp:155] function cbapi->getCuptiStatus() failed with error CUPTI_ERROR_INVALID_DEVICE (2)
WARNING:2024-02-08 10:47:36 6204:6204 init.cpp:156] CUPTI initialization failed - CUDA profiler activities will be missing
INFO:2024-02-08 10:47:36 6204:6204 init.cpp:158] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti
STAGE:2024-02-08 10:47:36 6204:6204 ActivityProfilerController.cpp:314] Completed Stage: Warm Up
STAGE:2024-02-08 10:47:37 6204:6204 ActivityProfilerController.cpp:320] Completed Stage: Collection
STAGE:2024-02-08 10:47:37 6204:6204 ActivityProfilerController.cpp:324] Completed Stage: Post Processing
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                             Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                  model_inference         1.12%       3.039ms       100.00%     271.268ms     271.268ms             1  
                     aten::conv2d         0.04%     114.000us        80.80%     219.173ms      10.959ms            20  
                aten::convolution         0.12%     313.000us        80.75%     219.059ms      10.953ms            20  
               aten::_convolution         0.07%     185.000us        80.64%     218.746ms      10.937ms            20  
          aten::cudnn_convolution        80.57%     218.561ms        80.57%     218.561ms      10.928ms            20  
                       aten::add_         4.01%      10.890ms         4.01%      10.890ms     388.929us            28  
                 aten::batch_norm         0.08%     229.000us         4.78%      12.960ms     648.000us            20  
     aten::_batch_norm_impl_index         0.04%     119.000us         4.76%      12.903ms     645.150us            20  
           aten::cudnn_batch_norm         4.34%      11.760ms         4.71%      12.784ms     639.200us            20  
                 aten::empty_like         0.04%     105.000us         0.33%     897.000us      44.850us            20  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 271.268ms

Running resnet18 10.000 times
time elapsed 38.23451283498434
```

### Versions

Collecting environment information...
PyTorch version: 2.2.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.2.0-1019-gcp-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA L4
Nvidia driver version: 530.30.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             4
On-line CPU(s) list:                0-3
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family:                         6
Model:                              85
Thread(s) per core:                 2
Core(s) per socket:                 2
Socket(s):                          1
Stepping:                           7
BogoMIPS:                           4400.38
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          64 KiB (2 instances)
L1i cache:                          64 KiB (2 instances)
L2 cache:                           2 MiB (2 instances)
L3 cache:                           38.5 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-3
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:             Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; Clear CPU buffers; SMT Host state unknown

Versions of relevant libraries:
[pip3] audio-diffusion-pytorch==0.0.96
[pip3] audio-encoders-pytorch==0.0.22
[pip3] flake8==4.0.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] torch==2.2.0
[pip3] torch-stoi==0.2.1
[pip3] torchaudio==2.2.0
[pip3] torchdata==0.7.1
[pip3] torchvision==0.17.0
[pip3] triton==2.2.0
[conda] Could not collect


cc @ptrblck