Repository: pytorch
Issue: Significantly difference in execution time when convolution is run as nn.Conv2d and as nn.Sequential
Labels: module: performance,module: cuda,module: convolution,triaged

## 🐛 Bug

Hi all, 
I'm experiencing a huge difference in execution runtime between a convolution run as nn.Conv2d and nn.Sequential() on  `aarch64` (Nvidia Jetson AGX). 
I think this issue is showing up due to the lack of any optimized kernels for the specific convolution layers I'm trying. This issue is only reprodcible on `aarch64` processor. But still that couldn't explain this huge difference in execution times. 

## To Reproduce

Steps to reproduce the behavior:

1. Run a Depthwise Convolution (groups = in_channels) and Pointwise Convolution (kernel size =1) individually as nn.Conv2d() and measure their runtime. 
1. Run the same set of convolution as a single nn.Sequential
```python
import torch
import time
x = torch.rand([1,78, 56, 56])
#### nn.Conv2d
dconv = torch.nn.Conv2d(78, 78, 3, stride=2, padding=1,groups=78)
pconv = torch.nn.Conv2d(78, 78, 1)
delay = []
for i in range(30):
    start = time.perf_counter()
    y = dconv(x)
    end = time.perf_counter()
    delay.append((end-start)*1000)
print(sum(delay)/len(delay))
delay = []
for i in range(30):
    start = time.perf_counter()
    z = pconv(y)
    end = time.perf_counter()
    delay.append((end-start)*1000)
print(sum(delay)/len(delay))
##### nn.Module
model = torch.nn.Sequential(torch.nn.Conv2d(78, 78, 3, stride=2, padding=1,groups=78, device='cpu'), torch.nn.Conv2d(78, 78, 1, device='cpu'))
delay = []
for i in range(30):
    start = time.perf_counter()
    z = model(x)
    end = time.perf_counter()
    delay.append((end-start)*1000)
print(sum(delay)/len(delay))
```
Output
```
5.974839165961991
4.580187029205263
141.68961447236748
```
## Expected behavior
There's almost **15x** difference in runtime between convolution executing as individual nn.Conv2Ds and nn.Sequential. The expected output would be to have time(dconv) + time(pconv) almost similar to time(model) in the above example. 

## Environment

Collecting environment information...
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (aarch64)
GCC version: (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.17.5
Libc version: glibc-2.25

Python version: 3.6.9 (default, Jan 26 2021, 15:33:00)  [GCC 8.4.0] (64-bit runtime)
Python platform: Linux-4.9.201-tegra-aarch64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.0.0
/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.0.0
/usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.0.0
/usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.0.0
/usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.0.0
/usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.0.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.4
[pip3] torch==1.9.0
[pip3] torchvision==0.9.0
[conda] Could not collect

More Details: This error was run on Jetson AGX with PyTorch built from source. I followed the instructions from this [website](https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-9-0-now-available/72048).

These were the PyTorch build flags used 
export USE_NCCL=0
export USE_DISTRIBUTED=0
export USE_QNNPACK=0
export USE_PYTORCH_QNNPACK=0
export TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
export PYTORCH_BUILD_VERSION=1.9.0
export PYTORCH_BUILD_NUMBER=1
export USE_KINETO=1
export USE_CUDA=1
export USE_CUPTI_SO=1


## Additional context

<!-- Add any other context about the problem here. -->


cc @VitalyFedyunin @ngimel